gpml-matlab-v3.6-2015-07-07/000755 000765 000024 00000000000 12550665747 015107 5ustar00carlstaff000000 000000 gpml-matlab-v3.6-2015-07-07/.octaverc000644 000765 000024 00000000010 11422240325 016657 0ustar00carlstaff000000 000000 startup gpml-matlab-v3.6-2015-07-07/priorDistributions.m000644 000765 000024 00000003767 12550665732 021212 0ustar00carlstaff000000 000000 % prior distributions to be used for hyperparameters of Gaussian processes % using infPrior. % There are two different kinds of prior distributions: simple and composite: % % simple prior distributions: % % priorGauss - univariate Gaussian % priorLaplace - univariate Laplace % priorT - univariate Student's t % % priorSmoothBox1 - univariate interval (linear decay in log domain) % priorSmoothBox2 - univariate interval (quadr. decay in log domain) % % priorGamma - univariate Gamma, IR+ % priorWeibull - univariate Weibull, IR+ % priorInvGauss - univariate Inverse Gaussian, IR+ % priorLogNormal - univariate Log-normal, IR+ % % priorClamped or - fix hyperparameter to its current value by setting % priorDelta derivatives to zero, no effect on marginal likelihood % % priorGaussMulti - multivariate Gauss % priorLaplaceMulti - multivariate Laplace % priorTMulti - multivariate Student's t % % priorClampedMulti or - fix hyperparameter to its current value by setting % priorDeltaMulti derivatives to zero, no effect on marginal likelihood % % composite prior distributions (see explanation at the bottom): % % priorMix - nonnegative mixture of priors % priorTransform - prior on g(t) rather than t % % Naming convention: all prior distributions are named "prior/prior*.m". % % % 1) With only a fixed input arguments: % % r = priorNAME(par1,par2,parN) % % The function returns a random sample from the distribution for e.g. % random restarts, simulations or optimisation initialisation. % % 2) With one additional input arguments: % % [lp,dlp] = priorNAME(par1,par2,parN, t) % % The function returns the log density at location t along with its first % derivative. % % See also doc/usagePrior.m, inf/infPrior.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-07-13. % File automatically generated using noweb. gpml-matlab-v3.6-2015-07-07/gp.m000644 000765 000024 00000023062 12550665731 015667 0ustar00carlstaff000000 000000 function [varargout] = gp(hyp, inf, mean, cov, lik, x, y, xs, ys) % Gaussian Process inference and prediction. The gp function provides a % flexible framework for Bayesian inference and prediction with Gaussian % processes for scalar targets, i.e. both regression and binary % classification. The prior is Gaussian process, defined through specification % of its mean and covariance function. The likelihood function is also % specified. Both the prior and the likelihood may have hyperparameters % associated with them. % % Two modes are possible: training or prediction: if no test cases are % supplied, then the negative log marginal likelihood and its partial % derivatives w.r.t. the hyperparameters is computed; this mode is used to fit % the hyperparameters. If test cases are given, then the test set predictive % probabilities are returned. Usage: % % training: [nlZ dnlZ ] = gp(hyp, inf, mean, cov, lik, x, y); % prediction: [ymu ys2 fmu fs2 ] = gp(hyp, inf, mean, cov, lik, x, y, xs); % or: [ymu ys2 fmu fs2 lp] = gp(hyp, inf, mean, cov, lik, x, y, xs, ys); % % where: % % hyp struct of column vectors of mean/cov/lik hyperparameters % inf function specifying the inference method % mean prior mean function % cov prior covariance function % lik likelihood function % x n by D matrix of training inputs % y column vector of length n of training targets % xs ns by D matrix of test inputs % ys column vector of length nn of test targets % % nlZ returned value of the negative log marginal likelihood % dnlZ struct of column vectors of partial derivatives of the negative % log marginal likelihood w.r.t. mean/cov/lik hyperparameters % ymu column vector (of length ns) of predictive output means % ys2 column vector (of length ns) of predictive output variances % fmu column vector (of length ns) of predictive latent means % fs2 column vector (of length ns) of predictive latent variances % lp column vector (of length ns) of log predictive probabilities % % post struct representation of the (approximate) posterior % 3rd output in training mode or 6th output in prediction mode % can be reused in prediction mode gp(.., cov, lik, x, post, xs,..) % % See also infMethods.m, meanFunctions.m, covFunctions.m, likFunctions.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-07-13. % File automatically generated using noweb. if nargin<7 || nargin>9 disp('Usage: [nlZ dnlZ ] = gp(hyp, inf, mean, cov, lik, x, y);') disp(' or: [ymu ys2 fmu fs2 ] = gp(hyp, inf, mean, cov, lik, x, y, xs);') disp(' or: [ymu ys2 fmu fs2 lp] = gp(hyp, inf, mean, cov, lik, x, y, xs, ys);') return end if isempty(mean), mean = {@meanZero}; end % set default mean if ischar(mean) || isa(mean, 'function_handle'), mean = {mean}; end % make cell if isempty(cov), error('Covariance function cannot be empty'); end % no default if ischar(cov) || isa(cov,'function_handle'), cov = {cov}; end % make cell cstr = cov{1}; if isa(cstr,'function_handle'), cstr = func2str(cstr); end if strcmp(cstr,'covFITC') && isfield(hyp,'xu'), cov{3} = hyp.xu; end %use hyp.xu if isempty(inf) % set default inference method if strcmp(cstr,'covFITC'), inf = {@infFITC}; else inf = {@infExact}; end end if ischar(inf), inf = str2func(inf); end % convert into function handle if ischar(inf) || isa(inf,'function_handle'), inf = {inf}; end % make cell istr = inf{1}; if isa(istr,'function_handle'), istr = func2str(istr); end if strcmp(istr,'infPrior') istr = inf{2}; if isa(istr,'function_handle'), istr = func2str(istr); end end if strcmp(cstr,'covFITC') % only infFITC* are possible if isempty(strfind(istr,'infFITC')==1) error('Only infFITC* are possible inference algorithms') end end % only one possible class of inference algorithms if isempty(lik), lik = {@likGauss}; end % set default lik if ischar(lik) || isa(lik,'function_handle'), lik = {lik}; end % make cell lstr = lik{1}; if isa(lstr,'function_handle'), lstr = func2str(lstr); end D = size(x,2); if strcmp(cstr,'covGrid') % only infGrid* are possible if isempty(strfind(istr,'infGrid')==1) error('Only infGrid* are possible inference algorithms') end D = 0; xg = cov{3}; p = numel(xg); for i=1:p, D = D+size(xg{i},2); end % dims end if ~isfield(hyp,'mean'), hyp.mean = []; end % check the hyp specification if eval(feval(mean{:})) ~= numel(hyp.mean) error('Number of mean function hyperparameters disagree with mean function') end if ~isfield(hyp,'cov'), hyp.cov = []; end if eval(feval(cov{:})) ~= numel(hyp.cov) error('Number of cov function hyperparameters disagree with cov function') end if ~isfield(hyp,'lik'), hyp.lik = []; end if eval(feval(lik{:})) ~= numel(hyp.lik) error('Number of lik function hyperparameters disagree with lik function') end try % call the inference method % issue a warning if a classification likelihood is used in conjunction with % labels different from +1 and -1 if strcmp(lstr,'likErf') || strcmp(lstr,'likLogistic') if ~isstruct(y) uy = unique(y); if any( uy~=+1 & uy~=-1 ) warning('You try classification with labels different from {+1,-1}') end end end if nargin>7 % compute marginal likelihood and its derivatives only if needed if isstruct(y) post = y; % reuse a previously computed posterior approximation else post = feval(inf{:}, hyp, mean, cov, lik, x, y); end else if nargout<=1 [post nlZ] = feval(inf{:}, hyp, mean, cov, lik, x, y); dnlZ = {}; else [post nlZ dnlZ] = feval(inf{:}, hyp, mean, cov, lik, x, y); end end catch msgstr = lasterr; if nargin>7, error('Inference method failed [%s]', msgstr); else warning('Inference method failed [%s] .. attempting to continue',msgstr) dnlZ = struct('cov',0*hyp.cov, 'mean',0*hyp.mean, 'lik',0*hyp.lik); varargout = {NaN, dnlZ}; return % continue with a warning end end if nargin==7 % if no test cases are provided varargout = {nlZ, dnlZ, post}; % report -log marg lik, derivatives and post else alpha = post.alpha; L = post.L; sW = post.sW; if issparse(alpha) % handle things for sparse representations nz = alpha ~= 0; % determine nonzero indices if issparse(L), L = full(L(nz,nz)); end % convert L and sW if necessary if issparse(sW), sW = full(sW(nz)); end else nz = true(size(alpha,1),1); end % non-sparse representation if isempty(L) % in case L is not provided, we compute it K = feval(cov{:}, hyp.cov, x(nz,:)); L = chol(eye(sum(nz))+sW*sW'.*K); end %verify whether L contains valid Cholesky decomposition or something different Lchol = isnumeric(L) && all(all(tril(L,-1)==0)&diag(L)'>0&isreal(diag(L))'); ns = size(xs,1); % number of data points if strcmp(cstr,'covGrid'), xs = covGrid('idx2dat',cov{3},xs); end % expand xs nperbatch = 1000; % number of data points per mini batch nact = 0; % number of already processed test data points ymu = zeros(ns,1); ys2 = ymu; fmu = ymu; fs2 = ymu; lp = ymu; % allocate mem while nact use Cholesky parameters (alpha,sW,L) V = L'\(repmat(sW,1,length(id)).*Ks); fs2(id) = kss - sum(V.*V,1)'; % predictive variances else % L is not triangular => use alternative parametrisation if isnumeric(L), LKs = L*Ks; else LKs = L(Ks); end % matrix or callback fs2(id) = kss + sum(Ks.*LKs,1)'; % predictive variances end fs2(id) = max(fs2(id),0); % remove numerical noise i.e. negative variances Fs2 = repmat(fs2(id),1,N); % we have multiple values in case of sampling if nargin<9 [Lp, Ymu, Ys2] = feval(lik{:},hyp.lik,[], Fmu(:),Fs2(:)); else Ys = repmat(ys(id),1,N); [Lp, Ymu, Ys2] = feval(lik{:},hyp.lik,Ys(:),Fmu(:),Fs2(:)); end lp(id) = sum(reshape(Lp, [],N),2)/N; % log probability; sample averaging ymu(id) = sum(reshape(Ymu,[],N),2)/N; % predictive mean ys|y and .. ys2(id) = sum(reshape(Ys2,[],N),2)/N; % .. variance nact = id(end); % set counter to index of last processed data point end if nargin<9 varargout = {ymu, ys2, fmu, fs2, [], post}; % assign output arguments else varargout = {ymu, ys2, fmu, fs2, lp, post}; end end gpml-matlab-v3.6-2015-07-07/mean/000755 000765 000024 00000000000 12550665747 016027 5ustar00carlstaff000000 000000 gpml-matlab-v3.6-2015-07-07/util/000755 000765 000024 00000000000 12550665747 016064 5ustar00carlstaff000000 000000 gpml-matlab-v3.6-2015-07-07/likFunctions.m000644 000765 000024 00000010312 12550665732 017724 0ustar00carlstaff000000 000000 % likelihood functions are provided to be used by the gp.m function: % % likErf (Error function, classification, probit regression) % likLogistic (Logistic, classification, logit regression) % likUni (Uniform likelihood, classification) % % likGauss (Gaussian, regression) % likGaussWarp (Warped Gaussian, regression) % likGumbel (Gumbel likelihood for extremal values) % likLaplace (Laplacian or double exponential, regression) % likSech2 (Sech-square, regression) % likT (Student's t, regression) % % likPoisson (Poisson regression, count data) % likGamma (Nonnegative regression, positive data) % likExp (Nonnegative regression, positive data) % likInvGauss (Nonnegative regression, positive data) % likBeta (Beta regression, interval data) % % likMix (Mixture of individual covariance functions) % % The likelihood functions have three possible modes, the mode being selected % as follows (where "lik" stands for any likelihood function in "lik/lik*.m".): % % 1) With one or no input arguments: [REPORT NUMBER OF HYPERPARAMETERS] % % s = lik OR s = lik(hyp) % % The likelihood function returns a string telling how many hyperparameters it % expects, using the convention that "D" is the dimension of the input space. % For example, calling "likLogistic" returns the string '0'. % % % 2) With three or four input arguments: [PREDICTION MODE] % % lp = lik(hyp, y, mu) OR [lp, ymu, ys2] = lik(hyp, y, mu, s2) % % This allows to evaluate the predictive distribution. Let p(y_*|f_*) be the % likelihood of a test point and N(f_*|mu,s2) an approximation to the posterior % marginal p(f_*|x_*,x,y) as returned by an inference method. The predictive % distribution p(y_*|x_*,x,y) is approximated by. % q(y_*) = \int N(f_*|mu,s2) p(y_*|f_*) df_* % % lp = log( q(y) ) for a particular value of y, if s2 is [] or 0, this % corresponds to log( p(y|mu) ) % ymu and ys2 the mean and variance of the predictive marginal q(y) % note that these two numbers do not depend on a particular % value of y % All vectors have the same size. % % % 3) With five or six input arguments, the fifth being a string [INFERENCE MODE] % % [varargout] = lik(hyp, y, mu, s2, inf) OR % [varargout] = lik(hyp, y, mu, s2, inf, i) % % There are three cases for inf, namely a) infLaplace, b) infEP and c) infVB. % The last input i, refers to derivatives w.r.t. the ith hyperparameter. % % a1) [lp,dlp,d2lp,d3lp] = lik(hyp, y, f, [], 'infLaplace') % lp, dlp, d2lp and d3lp correspond to derivatives of the log likelihood % log(p(y|f)) w.r.t. to the latent location f. % lp = log( p(y|f) ) % dlp = d log( p(y|f) ) / df % d2lp = d^2 log( p(y|f) ) / df^2 % d3lp = d^3 log( p(y|f) ) / df^3 % % a2) [lp_dhyp,dlp_dhyp,d2lp_dhyp] = lik(hyp, y, f, [], 'infLaplace', i) % returns derivatives w.r.t. to the ith hyperparameter % lp_dhyp = d log( p(y|f) ) / ( dhyp_i) % dlp_dhyp = d^2 log( p(y|f) ) / (df dhyp_i) % d2lp_dhyp = d^3 log( p(y|f) ) / (df^2 dhyp_i) % % % b1) [lZ,dlZ,d2lZ] = lik(hyp, y, mu, s2, 'infEP') % let Z = \int p(y|f) N(f|mu,s2) df then % lZ = log(Z) % dlZ = d log(Z) / dmu % d2lZ = d^2 log(Z) / dmu^2 % % b2) [dlZhyp] = lik(hyp, y, mu, s2, 'infEP', i) % returns derivatives w.r.t. to the ith hyperparameter % dlZhyp = d log(Z) / dhyp_i % % % c1) [b,z] = lik(hyp, y, [], ga, 'infVB') % ga is the variance of a Gaussian lower bound to the likelihood p(y|f). % p(y|f) \ge exp( b*(f+z) - (f+z).^2/(2*ga) - h(ga)/2 ) \propto N(f|b*ga-z,ga) % The function returns the linear part b and z. % % Cumulative likelihoods are designed for binary classification. Therefore, they % only look at the sign of the targets y; zero values are treated as +1. % % Some examples for valid likelihood functions: % lik = @likLogistic; % lik = {'likMix',{'likUni',@likErf}} % lik = {@likPoisson,'logistic'}; % % See the help for the individual likelihood for the computations specific to % each likelihood function. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-07-13. % File automatically generated using noweb. gpml-matlab-v3.6-2015-07-07/covFunctions.m000644 000765 000024 00000013571 12550665732 017746 0ustar00carlstaff000000 000000 % covariance functions to be use by Gaussian process functions. There are two % different kinds of covariance functions: simple and composite: % % simple covariance functions: % covConst - covariance for constant functions % covCos - sine periodic covariance function (1d) with unit period % covLIN - linear covariance function without parameters % covLINard - linear covariance function with ARD % covLINiso - linear covariance function % covLINone - linear covariance function with bias % covMaternard - Matern covariance function with nu=1/2, 3/2 or 5/2 with ARD % covMaterniso - Matern covariance function with nu=1/2, 3/2 or 5/2 % covNNone - neural network covariance function % covNoise - independent covariance function (i.e. white noise) % covPeriodic - smooth periodic covariance function (1d) % covPeriodicNoDC - as above but with zero DC component and properly scaled % covPoly - polynomial covariance function % covPPard - piecewise polynomial covariance function (compact support) % covPPiso - piecewise polynomial covariance function (compact support) % covRQard - rational quadratic covariance function with ARD % covRQiso - isotropic rational quadratic covariance function % covSEard - squared exponential covariance function with ARD % covSEiso - isotropic squared exponential covariance function % covSEisoU - same as above but without latent scale % covSEvlen - spatially varying lengthscale squared exponential % covSEfact - factor analysis squared exponential covariance function % covSM - spectral mixture covariance function % covGaborard - Gabor covariance function with ARD % covGaborsio - isotropic Gabor covariance function % covDiscrete - precomputed covariance for discrete data % % composite (meta) covariance functions (see explanation at the bottom): % covScale - scaled version of a covariance function % covProd - products of covariance functions % covSum - sums of covariance functions % covADD - additive covariance function % covMask - mask some dimensions of the data % covPERard - make ARD stationary covariance periodic % covPERiso - make isotropic stationary covariance periodic % covPref - difference covariance for preference learning % % special purpose (wrapper) covariance functions % covFITC - to be used in conjunction with infFITC* for large scale % inference problems; any covariance can be wrapped by % covFITC such that the FITC approximation is applicable % covGrid - to be used in conjunction with infGrid* for large scale % inference problems on grids resulting Kronecker structure % % Naming convention: all covariance functions are named "cov/cov*.m". A trailing % "iso" means isotropic, "ard" means Automatic Relevance Determination, and % "one" means that the distance measure is parameterized by a single parameter. % % The covariance functions are written according to a special convention where % the exact behaviour depends on the number of input and output arguments % passed to the function. If you want to add new covariance functions, you % should follow this convention if you want them to work with the function gp. % There are four different ways of calling the covariance functions: % % 1) With no (or one) input argument(s): % % s = cov % % The covariance function returns a string s telling how many hyperparameters it % expects, using the convention that "D" is the dimension of the input space. % For example, calling "covRQard" returns the string '(D+2)'. % % 2) With two input arguments: % % K = cov(hyp, x) equivalent to K = cov(hyp, x, []) % % The function computes and returns the covariance matrix where hyp are % the hyperparameters and x is an n by D matrix of cases, where % D is the dimension of the input space. The returned covariance matrix is of % size n by n. % % 3) With three input arguments: % % Ks = cov(hyp, x, xs) % kss = cov(hyp, xs, 'diag') % % The function computes test set covariances; kss is a vector of self covariances % for the test cases in xs (of length ns) and Ks is an (n by ns) matrix of cross % covariances between training cases x and test cases xs. % % 4) With four input arguments: % % dKi = cov(hyp, x, [], i) % dKsi = cov(hyp, x, xs, i) % dkssi = cov(hyp, xs, 'diag', i) % % The function computes and returns the partial derivatives of the % covariance matrices with respect to hyp(i), i.e. with % respect to the hyperparameter number i. % % Covariance functions can be specified in two ways: either as a string % containing the name of the covariance function or using a cell array. For % example: % % cov = 'covRQard'; % cov = {'covRQard'}; % cov = {@covRQard}; % % are supported. Only the second and third form using the cell array can be used % for specifying composite covariance functions, made up of several % contributions. For example: % % cov = {'covScale', {'covRQiso'}}; % cov = {'covSum', {'covRQiso','covSEard','covNoise'}}; % cov = {'covProd',{'covRQiso','covSEard','covNoise'}}; % cov = {'covMask',{mask,'covSEiso'}} % q=1; cov = {'covPPiso',q}; % d=3; cov = {'covPoly',d}; % cov = {'covADD',{[1,2],'covSEiso'}}; % cov = {@covFITC, {@covSEiso}, u}; where u are the inducing inputs % % specifies a covariance function which is the sum of three contributions. To % find out how many hyperparameters this covariance function requires, we do: % % feval(cov{:}) % % which returns the string '3+(D+1)+1' (i.e. the 'covRQiso' contribution uses % 3 parameters, the 'covSEard' uses D+1 and 'covNoise' a single parameter). % % See also doc/usageCov.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-07-13. % File automatically generated using noweb. gpml-matlab-v3.6-2015-07-07/README000644 000765 000024 00000040637 12550665746 016000 0ustar00carlstaff000000 000000 GAUSSIAN PROCESS REGRESSION AND CLASSIFICATION Toolbox version 3.6 for GNU Octave 3.2.x and Matlab 7.x Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-07-07. 0) HOW TO READ ============== If you want to get started without further delay, then read section 1) below and jump right to the examples in doc/index.html. 1) ABOUT THESE PROGRAMS ======================= This collection of matlab programs implements and demonstrates some of the algorithms described in a) the book by Rasmussen and Williams: "Gaussian Processes for Machine Learning", the MIT Press 2006, in b) the article by Nickisch and Rasmussen: "Approximations for Binary Gaussian Process Classification", JMLR 2008, in c) the article by Candela and Rasmussen: "A Unifying View of Sparse Approximate Gaussian Process Regression", JMLR 2005, in d) the paper by Murray, Adams and Mackay: "Elliptical slice sampling", AISTATS 2010, in e) the report by Neal: "Annealed Importance Sampling", UToronto 1998, in f) the paper by Naish-Guzman and Holden: "The Generalized FITC Approximation", NIPS, 2007, in g) the paper by Duvenaud, Nickisch and Rasmussen: "Additive Gaussian Processes", NIPS, 2011, in h) the paper by Wilson and Adams: "Gaussian Process Kernels for Pattern Discovery and Extrapolation", ICML, 2013, and in i) the paper Snelson, Rasmussen and Ghahramani: "Warped Gaussian Processes", NIPS, 2003. There are 7 code subdirectories: cov, doc, inf, lik, mean, prior and util. cov/ contains covariance functions cov*.m => see covFunctions.m doc/ contains an index.html file providing documentation. This information is also available from http://www.GaussianProcess.org/gpml/code. Usage of mean, cov, classification and regression is demonstrated in usage*.m. Further details can be found in the developer documentation manual.pdf. inf/ contains the inference algorithms inf*.m => see infMethods.m lik/ contains the likelihood functions lik*.m => see likFunctions.m mean/ contains the mean functions mean*.m => see meanFunctions.m prior/ contains the hyperparameter prior distributions prior*.m => see priorDistributions.m util/ contains optimisation routines, backward compatibility programs and small auxiliary scripts Before running the demos or any other part of the code, you should execute startup.m to add all necessary directories to your path. In Octave, you can also use the .octaverc file. 2) ABOUT MEX FILES ================== Some of the programs make use of the MEX facility in matlab for more efficient implementation. However, if you don't know about how to compile MEX files, you do not need to worry about this -- the code runs anyway. If you do compile the MEX files, this is automatically detected, and the program will run more efficiently. If you do not have a compiler installed on your system, you might want to consult [3]. Two components of the toolbox can be accelerated by MEX. First, there is an auxiliary functions solve_chol.m having a MEX equivalent. This can be compiled by either executing util/make.m from the Matlab/Octave command line. In addition to that, we provide a Makefile for Matlab in util/ that you can run by simply typing make in your shell after having supplied your Matlab path and your operating system. We recommend the make.m script because it works both under Matlab and Octave. Second, in order to use the L-BFGS minimiser for hyperparameter optimisation as an equivalent to minimize.m, you have to compile Peter Carbonetto's "Matlab interface for L-BFGS-B" [1] which was extended to also work for Octave by José Vallet. The challenge here is the Fortran 77 code. We provide a Makefile suitable for Linux 32/64 bit and Mac whenever you have 1) g77 or 2) gfortran properly set up. The latter is prefered on current Ubuntu systems. Under Ubuntu, you can achieve this by installing the packages fort77 and gfortran, respectively. Note that the code is GNU licensed. Compilation is done by editing util/lbfgsb/Makefile. In any case, you need to provide $MATLAB_HOME which can be found by the commands 'locate matlab' or 'find / -name "matlab"'. You can choose between two compilation modes: a) without mex utility by Matlab [default] provide $MEX_SUFFIX and $MATLAB_LIB, then type 'make nomex' b) using the mex utility by Matlab provide $MEX, then type 'make mex' c) compilation for Octave provide Octave variables and type 'make oct' In Ubuntu 10.04 LTS, the libg2c library needed for both 1)+a) and 1)+b) is not included per default. If 'ls /usr/lib/libg2c.*' does not list anything this is the case on your machine. You then whant to install the packages gcc-3.4-base and libg2c0 e.g. from http://packages.ubuntu.com/hardy/gcc-3.4-base and http://packages.ubuntu.com/hardy/libg2c0. After installation, you have to create a symbolic link by 'cd /usr/lib' and 'ln -s libg2c.so.0 libg2c.so'. There were reports of problems running the mex files on recent 64bit Ubuntu since the libgfortran.3.0.0 that comes with matlab ($(MATLAB_HOME)/sys/os/glnxa64/libgfortran.so.3.0.0) is not the same as the libgfortran.3.0.0 that comes with the ubuntu package libgfortran3 and that is used when linking. So running the mex file results in a version mismatch error. The issue can be solved by changing the name of to the matlab file and linking to the ubuntu file instead: in $(MATLAB_HOME)/sys/os/glnxa64 run 'sudo mv libgfortran.3.0.0 libgfortran.3.0.0_matlab' 'sudo ln -s /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0 libgfortran.3.0.0' Thanks to Giampiero Salvi for posting the workaround. [1] https://github.com/pcarbo/lbfgsb-matlab [2] http://www.mathworks.com/support/compilers/R2010a 3) CURRENT VERSION ================== The current version of the programs is 3.6. Previous versions of the code are available at http://www.gaussianprocess.org/gpml/code/oldcode.html. 4) DIFFERENCES TO PREVIOUS VERSIONS =================================== NEW in version 3.6, 2015-07-07 ------------------------------ - added a new inference function infGrid_Laplace allowing to use non-Gaussian likelihoods for large grids - fixed a bug due to Octave evaluating norm([]) to a tiny nonzero value, modified all lik/lik*.m functions reported by Philipp Richter - small bugfixes in covGrid and infGrid - bugfix in predictive variance of likNegBinom due to Seth Flaxman - bugfix in infFITC_Laplace as suggested by Wu Lin - bugfix in covPP{iso,ard} NEW in version 3.5, 2014-12-08 ------------------------------ - mechanism for specifying hyperparameter priors (together with Roman Garnett and José Vallet) - new inference method inf/infGrid allowing efficient inference for data defined on a Cartesian grid (together with Andrew Wilson) - new mean/cov functions for preference learning: meanPref/covPref - new mean/cov functions for non-vectorial data: meanDiscrete/covDiscrete - new piecewise constant nearest neighbor mean function: meanNN - new mean functions being predictions from GPs: meanGP and meanGPexact - new covariance function for standard additive noise: covEye - new covariance function for factor analysis: covSEfact - new covariance function with varying length scale : covSEvlen - make covScale more general to scaling with a function instead of a scalar - bugfix in covGabor* and covSM (due to Andrew Gordon Wilson) - bugfix in lik/likBeta.m (suggested by Dali Wei) - bugfix in solve_chol.c (due to Todd Small) - bugfix in FITC inference mode (due to Joris Mooij) where the wrong mode for post.L was chosen when using infFITC and post.L being a diagonal matrix - bugfix in infVB marginal likelihood for likLogistic with nonzero mean function (reported by James Lloyd) - removed the combination likErf/infVB as it yields a bad posterior approximation and lacks theoretical justification - Matlab and Octave compilation for L-BFGS-B v2.4 and the more recent L-BFGS-B v3.0 (contributed by José Vallet) - smaller bugfixes in gp.m (due to Joris Mooij and Ernst Kloppenburg) - bugfix in lik/likBeta.m (due to Dali Wei) - updated use of logphi in lik/likErf - bugfix in util/solve_chol.c where a typing issue occured on OS X (due to Todd Small) - bugfix due to Bjørn Sand Jensen noticing that cov_deriv_sq_dist.m was missing in the distribution - bugfix in infFITC_EP for ttau->inf (suggested by Ryan Turner) NEW in version 3.4, 2013-11-11 ------------------------------ - new likelihood function for warped GPs: likGaussWarp - new exponential likelihood function: likExp - new Weibull likelihood function: likWeibull - new Gumbel likelihood function for extremal value regression: likGumbel - new polynomial mean function: meanPoly - included derivatives w.r.t. inducing points xu in infFITC, infFITC_Laplace, infFITC_EP - allow for zero noise variance for infExact NEW in version 3.3, 2013-10-19 ------------------------------ - new variational inference algorithm based on direct KL minimisation: infKL - improved inf/infVB double loop scheme so that only very few likelihood properties are required; infVB is now internally a sequence of infLaplace runs - improved inf/infLaplace to be more generic so that optimisers other than scaled Newton can be used - improved inf/infEP so that the internal variables (mu,Sigma) now represent the current posterior approximation - added three new generalised linear model likelihoods: gamma, beta, inverse Gaussian - new covariance functions: spectral mixture covSM, covGaboriso and covGaborard - new meta covariance function to turn a stationary covariance into a periodic covariance function covPERard, covPERiso - new periodic covariance function with zero DC component and correct scaling covPeriodicNoDC New likelihood functions in lik/: - likGamma.m, likBeta.m, likInvGauss.m New inference functions in inf/: - infKL.m New covariance functions in cov/: - covSM.m, covGaboriso.m, covGaborard.m (contributed by Andrew Gordon Wilson) - covPPard.m, covMaternard.m, covLINiso.m - covCos.m, covPeriodicNoDC.m (contributed by James Robert Lloyd) - covPERiso.m, covPERard.m NEW in version 3.2, 2013-01-15 ------------------------------ - added inf/infFITC_Laplace.m and inf/infFITC_EP.m for large scale approximate inference - added inf/infMCMC.m to allow for Markov Chain Monte Carlo sampling and Annealed Importance Sampling; we currently support two samplers: Elliptical Slice Sampling (ESS) and Hybrid Monte Carlo (HMC) - improved accuracy of the digamma function lik/likT.m/dloggamma from three to ten digits precision (thanks to Edward Challis) - added gfortran support to util/lbfgsb/Makefile (thanks to Ernst Kloppenburg) - fixed a stupid bug in cov/covFITC.m (thanks to Krzysztof Chalupka) - fixed a severe stability bug in the derivatives of inf/infFITC.m (thanks to Andrew McHutchon) - changed scalars in util/solve_chol.c to long to provide 64bit compatibility (thanks to Ernst Kloppenburg) - added infLOO to allow pseudo-likelihood optimisation instead of the marginal likelihood - fixed a performance issue in util/sq_dist.m (thanks to Krzysztof Chalupka) - fixed a bug causing numerical problems in infFITC and covFITC whenever Kuu is close to singular (thanks to Joris M. Mooij) - fixed an issue in covNoise (thanks to Daniel Marthaler) - fixed implicit derivative w.r.t. likelihood in infLaplace* - added two more likelihood functions likMix.m for mixtures and likUni for classification noise - slightly changed the interface for the likelihood functions - added likPoisson for Poisson regression of count data NEW in version 3.1, 2010-10-10 ------------------------------ - following a suggestion by Ed Snelson we now support FITC regression - cov/covFITC.m and inf/infFITC.m have been added with Ed Snelson's help - the covariance interface was slightly changed to make that possible - changed parametrisation in cov/covPeriodic.m (thanks to Philipp Hennig) NEW in version 3.0, 2010-07-23 ------------------------------ A major code reorganisation effort did take place in the current release. First, classification and regression are now done by a single file gp.m which is completely generic in the likelihood. The previous regression program gpr.m corresponds to gp.m with Gaussian likelihood. Several other likelihoods for robust regresssion were added. Further, the code now supports mean functions with a similar specification mechanism as already used by the covariance functions. Previous implementations correspond to using meanZero.m. We merged the covariance functions covMatern3iso.m and covMatern5.iso into a single covMaterniso.m and added a covariance function for additive functions as well as the possibility to use only certain components of the data. Finally, we included covPPiso.m a piecewise polynomial covariance function with compact support. New likelihood functions in lik/: likGauss.m, likLaplace.m, likLogistic.m and likT.m New mean functions in mean/: meanConst.m, meanLinear.m, meanPow.m, meanProd.m, meanSum.m and meanZero.m New covariance functions in cov/: covADD.m, covMask.m, covPPiso.m The gprSRPP.m function which previously provided "Subset of Regressors" and the "Projected Process" approximation has now been removed. NEW in version 2.1, 2007-07-25 ------------------------------ covConst.m: fixed a bug which caused an error in the derivative of the log marginal likelihood for certain combinations of covariance functions and approximation methods. (Thanks to Antonio Eleuteri for reporting the problem) gauher.m: added the function "gauher.m" which was mistakenly missing from the previous release. This caused an error for certain combinations of approximation method and likelihood function. logistic.m: modified the approximation of moments calculation to use a mixture of cumulative Gaussian, rather than Gauss-Hermite quadrature, as the former turns out to be more accurate. NEW in version 2.0, 2007-06-25 ------------------------------ Some code restructuring has taken place for the classification code to make it more modular, to facilitate addition of new likelihood functions and approximations methods. Now, all classification is done using the binaryGP function, which (among other things) takes an approximation method and a likelihood function as an arguments. Thus, binaryGP replaces both binaryEPGP and binaryLapaceGP, although wrapper functions are still provided for backward compatibility. This gives added flexibility: now EP can also be used wth the logistic likelihood function (implemented using Gauss-Hermite quadrature). approxEP.m: New file, containing the Expectation Propagation approximation method, which was previously contained in binaryEPGP.m approxLA.m: New file, containing Laplaces approximation method, which was previously contained in binaryLaplace.m approximations.m: New file, help for the approximation methods. binaryEPGP.m: This file has been replaced by a wrapper (for backward compatibility) which calls the more general binaryGP function. binaryGP.m: New general function to do binary classification. binaryLaplaceGP.m: This file has been replaced by a wrapper (for backward compatibility) which calls the more general binaryGP function. covMatern3iso.m, covMatern5iso.m, covNNone.m, covRQard.m, covRQiso.m, cosSEard, covSEiso: now check more carefully, that persistent variables have the correct sizes, and some variable names have been modified. cumGauss.m: New file, containing code for the cumulative Gaussian likelihood function likelihoods.m: New file, help for likelihood functions logistic.m: New file, logistic likelihood NEW in version 1.3, 2006-09-08 ------------------------------ covRQard.m: bugfix: replaced x with x' and z with z' in line 36 covRQiso.m: bugfix: replaced x with x' and z with z' in line 28 minimize.m: correction: replaced "error()" with "error('')", and made a few cosmetic changes binaryEPGP.m: added the line "lml = -n*log(2);" in line 77. This change should be largely inconsequential, but occationally may save things when the covariance matrix is exceptionally badly conditioned. NEW in version 1.2, 2006-05-10 ------------------------------ added the "erfint" function to "binaryLaplaceGP.m". The erfint function was missing by mistake, preventing the use of the "logistic" likelihood. NEW in version 1.1, 2006-04-12 ------------------------------ added files: "covProd.m" and "covPeriodic.m" changes: "covSEiso.m" was changed slightly to avoid the use of persistent variables NEW in version 1.0, 2006-03-29 ------------------------------ initial version shipped with the book gpml-matlab-v3.6-2015-07-07/inf/000755 000765 000024 00000000000 12550665747 015663 5ustar00carlstaff000000 000000 gpml-matlab-v3.6-2015-07-07/prior/000755 000765 000024 00000000000 12550665747 016242 5ustar00carlstaff000000 000000 gpml-matlab-v3.6-2015-07-07/infMethods.m000644 000765 000024 00000005264 12550665732 017366 0ustar00carlstaff000000 000000 % Inference methods: Compute the (approximate) posterior for a Gaussian process. % Methods currently implemented include: % % infExact Exact inference (only possible with Gaussian likelihood) % infLaplace Laplace's Approximation % infEP Expectation Propagation % infVB Variational Bayes Approximation % infKL Kullback-Leibler optimal Approximation % % infFITC Large scale regression with approximate covariance matrix % infFITC_Laplace Large scale inference with approximate covariance matrix % infFITC_EP Large scale inference with approximate covariance matrix % % infMCMC Markov Chain Monte Carlo and Annealed Importance Sampling % We offer two samplers. % - hmc: Hybrid Monte Carlo % - ess: Elliptical Slice Sampling % No derivatives w.r.t. to hyperparameters are provided. % % infLOO Leave-One-Out predictive probability and Least-Squares Approxim. % infPrior Perform inference with hyperparameter prior. % % The interface to the approximation methods is the following: % % function [post nlZ dnlZ] = inf..(hyp, cov, lik, x, y) % % where: % % hyp is a struct of hyperparameters % cov is the name of the covariance function (see covFunctions.m) % lik is the name of the likelihood function (see likFunctions.m) % x is a n by D matrix of training inputs % y is a (column) vector (of size n) of targets % % nlZ is the returned value of the negative log marginal likelihood % dnlZ is a (column) vector of partial derivatives of the negative % log marginal likelihood w.r.t. each hyperparameter % post struct representation of the (approximate) posterior containing % alpha is a (sparse or full column vector) containing inv(K)*(mu-m), % where K is the prior covariance matrix, m the prior mean, % and mu the approx posterior mean % sW is a (sparse or full column) vector containing diagonal of sqrt(W) % the approximate posterior covariance matrix is inv(inv(K)+W) % L is a (sparse or full) matrix, L = chol(sW*K*sW+eye(n)) % % Usually, the approximate posterior to be returned admits the form % N(mu=m+K*alpha, V=inv(inv(K)+W)), where alpha is a vector and W is diagonal; % if not, then L contains instead -inv(K+inv(W)), and sW is unused. % % For more information on the individual approximation methods and their % implementations, see the separate inf??.m files. See also gp.m % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-07-13. % File automatically generated using noweb. gpml-matlab-v3.6-2015-07-07/meanFunctions.m000644 000765 000024 00000003727 12550665732 020101 0ustar00carlstaff000000 000000 % mean functions to be use by Gaussian process functions. There are two % different kinds of mean functions: simple and composite: % % simple mean functions: % % meanZero - zero mean function % meanOne - one mean function % meanConst - constant mean function % meanLinear - linear mean function % meanPoly - polynomial mean function % meanDiscrete - precomputed mean for discrete data % meanGP - predictive mean of another GP % meanGPexact - predictive mean of a regression GP % meanNN - nearest neighbor mean function % % composite covariance functions (see explanation at the bottom): % % meanScale - scaled version of a mean function % meanPow - power of a mean function % meanProd - products of mean functions % meanSum - sums of mean functions % meanMask - mask some dimensions of the data % meanPref - difference mean for preference learning % % Naming convention: all mean functions are named "mean/mean*.m". % % % 1) With no or only a single input argument: % % s = meanNAME or s = meanNAME(hyp) % % The mean function returns a string s telling how many hyperparameters hyp it % expects, using the convention that "D" is the dimension of the input space. % For example, calling "meanLinear" returns the string 'D'. % % 2) With two input arguments: % % m = meanNAME(hyp, x) % % The function computes and returns the mean vector where hyp are the % hyperparameters and x is an n by D matrix of cases, where D is the dimension % of the input space. The returned mean vector is of size n by 1. % % 3) With three input arguments: % % dm = meanNAME(hyp, x, i) % % The function computes and returns the n by 1 vector of partial derivatives % of the mean vector w.r.t. hyp(i) i.e. hyperparameter number i. % % See also doc/usageMean.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-07-13. % File automatically generated using noweb. gpml-matlab-v3.6-2015-07-07/lik/000755 000765 000024 00000000000 12550665747 015666 5ustar00carlstaff000000 000000 gpml-matlab-v3.6-2015-07-07/doc/000755 000765 000024 00000000000 12550665747 015654 5ustar00carlstaff000000 000000 gpml-matlab-v3.6-2015-07-07/Copyright000644 000765 000024 00000003456 12546444650 017003 0ustar00carlstaff000000 000000 GAUSSIAN PROCESS REGRESSION AND CLASSIFICATION Toolbox version 3.6 for GNU Octave 3.2.x and Matlab 7.x The code is released under the FreeBSD License. Copyright (c) 2005-2015 Carl Edward Rasmussen & Hannes Nickisch. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY CARL EDWARD RASMUSSEN & HANNES NICKISCH ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL CARL EDWARD RASMUSSEN & HANNES NICKISCH OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. The views and conclusions contained in the software and documentation are those of the authors and should not be interpreted as representing official policies, either expressed or implied, of Carl Edward Rasmussen & Hannes Nickisch. The code and associated documentation is available from http://gaussianprocess.org/gpml/code. gpml-matlab-v3.6-2015-07-07/cov/000755 000765 000024 00000000000 12550665747 015676 5ustar00carlstaff000000 000000 gpml-matlab-v3.6-2015-07-07/startup.m000644 000765 000024 00000001440 12441263152 016745 0ustar00carlstaff000000 000000 % startup script to make Octave/Matlab aware of the GPML package % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2014-11-30. disp(['executing gpml startup script...']); OCT = exist('OCTAVE_VERSION') ~= 0; % check if we run Matlab or Octave me = mfilename; % what is my filename mydir = which(me); mydir = mydir(1:end-2-numel(me)); % where am I located if OCT && numel(mydir)==2 if strcmp(mydir,'./'), mydir = [pwd,mydir(2:end)]; end end % OCTAVE 3.0.x relative, MATLAB and newer have absolute path addpath(mydir(1:end-1)) addpath([mydir,'cov']) addpath([mydir,'doc']) addpath([mydir,'inf']) addpath([mydir,'lik']) addpath([mydir,'mean']) addpath([mydir,'prior']) addpath([mydir,'util']) clear me mydirgpml-matlab-v3.6-2015-07-07/cov/covPERiso.m000644 000765 000024 00000006111 12441263146 017706 0ustar00carlstaff000000 000000 function K = covPERiso(cov, hyp, x, z, i) % Stationary periodic covariance function for an isotropic stationary covariance % function k0 such as covMaterniso, covPPiso, covRQiso and covSEiso. % Isotropic stationary means that the covariance function k0(x,z) depends on the % data points x,z only through the squared distance % dxz = (x-z)'*inv(P)*(x-z) where the P matrix is ell^2 times the unit matrix. % The covariance function is parameterized as: % % k(x,z) = k0(u(x),u(z)), u(x) = [sin(pi*x/p); cos(pi*x/p)] % % where the period p belongs to covPERiso and hyp0 belong to k0: % % hyp = [ log(p) % hyp0 ] % % The first hyperparameter of k0 is the log lengthscale hyp0(1) = log(ell). % Note that for k0 = covSEiso and D = 1, a faster alternative is covPeriodic. % % Copyright (c) by Hannes Nickisch, 2013-10-15. % % See also COVFUNCTIONS.M. nocov = false; % default case when no cov argument is provided if nargin==0, cov = {@covSEiso}; nocov = true; end % default case if isnumeric(cov) % detect old version where the cov parameter was missing % i <- z, z <- x, x <- hyp, hyp <- cov if nargin>3, i = z; end if nargin>2, z = x; end if nargin>1, x = hyp; end hyp = cov; cov = {@covSEiso}; nocov = true; end if nocov && nargin<2 || ~nocov && nargin<3 % report number of parameters K = ['(1+',feval(cov{:}),')']; return end if nocov && nargin<3 || ~nocov && nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); p = exp(hyp(1)); if nocov && nargin<4 || ~nocov && nargin<5 [x,z] = u(x,z,p,dg); % apply the embedding u:IR^D->IR^2*D K = feval(cov{:},hyp(2:end),x,z); else if i==1 if dg % compute distance d d = zeros([n,1,D]); else if xeqz % symmetric matrix Kxx d = repmat(reshape(x,n,1,D),[1,n, 1])-repmat(reshape(x,1,n, D),[n,1,1]); else % cross covariances Kxz nz = size(z,1); d = repmat(reshape(x,n,1,D),[1,nz,1])-repmat(reshape(z,1,nz,D),[n,1,1]); end end d = 2*pi*d/p; dD2_dlp = -2*sum(sin(d).*d,3); % derivative dD2/dlog(p) [x,z] = u(x,z,p,dg); % apply the embedding u:IR^D->IR^2*D if dg % compute squared distances D2 = zeros(n,1); else if xeqz, D2 = sq_dist(x'); else D2 = sq_dist(x',z'); end end % reconstruct derivative w.r.t. D2 from derivative w.r.t. log(ell) dK_dD2 = feval(cov{:},hyp(2:end),x,z,1)./(-2*D2); dK_dD2(D2<1e-12) = 0; K = dK_dD2.*dD2_dlp; % apply chain rule else [x,z] = u(x,z,p,dg); % apply the embedding u:IR^D->IR^2*D K = feval(cov{:},hyp(2:end),x,z,i-1); end end function [x,z] = u(x,z,p,dg) % apply the embedding u:IR^D->IR^2*D x = 2*pi*x/p; x = [sin(x), cos(x)]; if numel(z)>0 && ~dg, z = 2*pi*z/p; z = [sin(z), cos(z)]; endgpml-matlab-v3.6-2015-07-07/cov/covPref.m000644 000765 000024 00000002233 12441263146 017442 0ustar00carlstaff000000 000000 function K = covPref(cov, hyp, x, z, varargin) % covPref - covariance function for preference learning. The covariance % function corresponds to a prior on f(x1) - f(x2). % % k(x,z) = k_0(x1,z1) + k_0(x2,z2) - k_0(x1,z2) - k_0(x2,z1). % % The hyperparameters are: % % hyp = [ hyp_k0 ] % % For more help on design of covariance functions, try "help covFunctions". % % See Collaborative Gaussian Processes for Preference Learning, NIPS 2014. % % Copyright (c) by Hannes Nickisch and Roman Garnett, 2014-11-01. % % See also COVFUNCTIONS.M. if nargin<3, K = strrep(feval(cov{:}),'D','D/2'); return; end % no of params x1 = x(:,1:end/2); x2 = x(:,1+end/2:end); if nargin<4 || isempty(z), z = x; end if strcmp(z,'diag') n = size(x,1); diag12 = zeros(n,1); for i=1:n, diag12(i) = feval(cov{:},hyp,x1(i,:),x2(i,:),varargin{:}); end K = feval(cov{:},hyp,x1,z,varargin{:}) ... + feval(cov{:},hyp,x2,z,varargin{:}) - 2*diag12; else z1 = z(:,1:end/2); z2 = z(:,1+end/2:end); K = feval(cov{:},hyp,x1,z1,varargin{:}) ... + feval(cov{:},hyp,x2,z2,varargin{:}) ... - feval(cov{:},hyp,x1,z2,varargin{:}) ... - feval(cov{:},hyp,x2,z1,varargin{:}); end gpml-matlab-v3.6-2015-07-07/cov/covLIN.m000644 000765 000024 00000002036 12441263146 017171 0ustar00carlstaff000000 000000 function K = covLIN(hyp, x, z, i) % Linear covariance function. The covariance function is parameterized as: % % k(x^p,x^q) = x^p'*x^q % % The are no hyperparameters: % % hyp = [ ] % % Note that there is no bias or scale term; use covConst to add these. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. if nargin<2, K = '0'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode % compute inner products if dg % vector kxx K = sum(x.*x,2); else if xeqz % symmetric matrix Kxx K = x*x'; else % cross covariances Kxz K = x*z'; end end if nargin>3 % derivatives error('Unknown hyperparameter') endgpml-matlab-v3.6-2015-07-07/cov/covLINard.m000644 000765 000024 00000003011 12441263146 017652 0ustar00carlstaff000000 000000 function K = covLINard(hyp, x, z, i) % Linear covariance function with Automatic Relevance Determination (ARD). The % covariance function is parameterized as: % % k(x^p,x^q) = x^p'*inv(P)*x^q % % where the P matrix is diagonal with ARD parameters ell_1^2,...,ell_D^2, where % D is the dimension of the input space. The hyperparameters are: % % hyp = [ log(ell_1) % log(ell_2) % .. % log(ell_D) ] % % Note that there is no bias term; use covConst to add a bias. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. if nargin<2, K = 'D'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode ell = exp(hyp); [n,D] = size(x); x = x*diag(1./ell); % precompute inner products if dg % vector kxx K = sum(x.*x,2); else if xeqz % symmetric matrix Kxx K = x*x'; else % cross covariances Kxz z = z*diag(1./ell); K = x*z'; end end if nargin>3 % derivatives if i<=D if dg K = -2*x(:,i).*x(:,i); else if xeqz K = -2*x(:,i)*x(:,i)'; else K = -2*x(:,i)*z(:,i)'; end end else error('Unknown hyperparameter') end endgpml-matlab-v3.6-2015-07-07/cov/covPoly.m000644 000765 000024 00000002717 12441263146 017500 0ustar00carlstaff000000 000000 function K = covPoly(d, hyp, x, z, i) % Polynomial covariance function. The covariance function is parameterized as: % % k(x^p,x^q) = sf^2 * ( c + (x^p)'*(x^q) )^d % % The hyperparameters are: % % hyp = [ log(c) % log(sf) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. if nargin<3, K = '2'; return; end % report number of parameters if nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode c = exp(hyp(1)); % inhomogeneous offset sf2 = exp(2*hyp(2)); % signal variance if d~=max(1,fix(d)), error('only nonzero integers allowed for d'), end % degree % precompute inner products if dg % vector kxx K = sum(x.*x,2); else if xeqz % symmetric matrix Kxx K = x*x'; else % cross covariances Kxz K = x*z'; end end if nargin<5 % covariances K = sf2*( c + K ).^d; else % derivatives if i==1 K = c*d*sf2*( c + K ).^(d-1); elseif i==2 K = 2*sf2*( c + K ).^d; else error('Unknown hyperparameter') end endgpml-matlab-v3.6-2015-07-07/cov/covLINone.m000644 000765 000024 00000002511 12441263146 017671 0ustar00carlstaff000000 000000 function K = covLINone(hyp, x, z, i) % Linear covariance function with a single hyperparameter. The covariance % function is parameterized as: % % k(x^p,x^q) = (x^p'*x^q + 1)/t^2; % % where the P matrix is t2 times the unit matrix. The second term plays the % role of the bias. The hyperparameter is: % % hyp = [ log(t) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. if nargin<2, K = '1'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode it2 = exp(-2*hyp); % t2 inverse % precompute inner products if dg % vector kxx K = sum(x.*x,2); else if xeqz % symmetric matrix Kxx K = x*x'; else % cross covariances Kxz K = x*z'; end end if nargin<4 % covariances K = it2*(1+K); else % derivatives if i==1 K = -2*it2*(1+K); else error('Unknown hyperparameter') end end gpml-matlab-v3.6-2015-07-07/cov/covLINiso.m000644 000765 000024 00000002524 12441263146 017706 0ustar00carlstaff000000 000000 function K = covLINiso(hyp, x, z, i) % Linear covariance function with Automatic Relevance Determination (ARD). The % covariance function is parameterized as: % % k(x^p,x^q) = x^p'*inv(P)*x^q % % where the P matrix is ell^2 times the unit matrix. The hyperparameters are: % % hyp = [ log(ell) ] % % Note that there is no bias term; use covConst to add a bias. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-10-13. % % See also COVFUNCTIONS.M. if nargin<2, K = '1'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode ell = exp(hyp(1)); x = x/ell; % precompute inner products if dg % vector kxx K = sum(x.*x,2); else if xeqz % symmetric matrix Kxx K = x*x'; else % cross covariances Kxz z = z/ell; K = x*z'; end end if nargin>3 % derivatives if i==1 if dg K = -2*sum(x.*x,2); else if xeqz K = -2*x*x'; else K = -2*x*z'; end end else error('Unknown hyperparameter') end endgpml-matlab-v3.6-2015-07-07/cov/covPERard.m000644 000765 000024 00000007004 12441263146 017664 0ustar00carlstaff000000 000000 function K = covPERard(cov, hyp, x, z, i) % Stationary periodic covariance function for a stationary covariance function % k0 such as covMaternard, covPPard, covRQard and covSEard. % Stationary means that the covariance function k0(x,z) depends on the % data points x,z only through the squared distance % dxz = (x-z)'*inv(P)*(x-z) where the P matrix is diagonal with ARD parameters % ell_1^2,...,ell_D^2, where D is the dimension of the input space. % The covariance function is parameterized as: % % k(x,z) = k0(u(x),u(z)), u(x) = [sin(pi*x/p); cos(pi*x/p)] % % where the period p belongs to covPERiso and hyp0 belong to k0: % % hyp = [ log(p_1) % log(p_2) % . % log(p_D) % hyp0 ] % % The first D hyperparameters of k0 are the log lengthscales such that % hyp0(i) = log(ell(i)) for i=1..D. % Note that for k0 = covSEard and D = 1, a faster alternative is covPeriodic. % % Copyright (c) by Hannes Nickisch, 2013-10-16. % % See also COVFUNCTIONS.M. nocov = false; % default case when no cov argument is provided if nargin==0, cov = {@covSEard}; nocov = true; end % default case if isnumeric(cov) % detect old version where the cov parameter was missing % i <- z, z <- x, x <- hyp, hyp <- cov if nargin>3, i = z; end if nargin>2, z = x; end if nargin>1, x = hyp; end hyp = cov; cov = {@covSEard}; nocov = true; end if nocov && nargin<2 || ~nocov && nargin<3 % report number of parameters K = ['(D+',feval(cov{:}),')']; return end if nocov && nargin<3 || ~nocov && nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); p = exp(hyp(1:D)); lell = hyp(D+(1:D)); hyp0 = [lell; lell; hyp(2*D+1:end)]; if nocov && nargin<4 || ~nocov && nargin<5 [x,z] = u(x,z,p,dg); % apply the embedding u:IR^D->IR^2*D K = feval(cov{:},hyp0,x,z); else if i<=D if dg % compute distance d di = zeros([n,1]); else if xeqz % symmetric matrix Kxx di = repmat(reshape(x(:,i),n, 1),[1, n])... -repmat(reshape(x(:,i),1, n),[n, 1]); else % cross covariances Kxz nz = size(z,1); di = repmat(reshape(x(:,i),n, 1),[1,nz])... -repmat(reshape(z(:,i),1,nz),[n, 1]); end end di = 2*pi*di/p(i); dD2_dlpi = -2*sin(di).*di; % derivative dD2i/dlog(p(i)) [x,z] = u(x,z,p,dg); % apply the embedding u:IR^D->IR^2*D if dg % compute squared distances D2 = zeros(n,1); else if xeqz D2 = sq_dist(x(:,[i,i+D])'); else D2 = sq_dist(x(:,[i,i+D])',z(:,[i,i+D])'); end end % reconstruct derivative w.r.t. D2i from derivative w.r.t. log(ell(i)) dK_dD2 = feval(cov{:},hyp0,x,z,i) + feval(cov{:},hyp0,x,z,i+D); dK_dD2 = dK_dD2./(-2*D2); dK_dD2(D2<1e-12) = 0; K = dK_dD2.*dD2_dlpi; % apply chain rule else [x,z] = u(x,z,p,dg); % apply the embedding u:IR^D->IR^2*D if i<=2*D K = feval(cov{:},hyp0,x,z,i-D) + feval(cov{:},hyp0,x,z,i); else K = feval(cov{:},hyp0,x,z,i); end end end function [x,z] = u(x,z,p,dg) % apply the embedding u:IR^D->IR^2*D x = 2*pi*x*diag(1./p); x = [sin(x), cos(x)]; if numel(z)>0 && ~dg, z = 2*pi*z*diag(1./p); z = [sin(z), cos(z)]; endgpml-matlab-v3.6-2015-07-07/cov/covSEisoU.m000644 000765 000024 00000002760 12441263146 017722 0ustar00carlstaff000000 000000 function K = covSEisoU(hyp, x, z, i) % Squared Exponential covariance function with isotropic distance measure with % unit magnitude. The covariance function is parameterized as: % % k(x^p,x^q) = exp(-(x^p - x^q)'*inv(P)*(x^p - x^q)/2) % % where the P matrix is ell^2 times the unit matrix and sf2 is the signal % variance. The hyperparameters are: % % hyp = [ log(ell) ] % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. if nargin<2, K = '1'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode ell = exp(hyp(1)); % characteristic length scale % precompute squared distances if dg % vector kxx K = zeros(size(x,1),1); else if xeqz % symmetric matrix Kxx K = sq_dist(x'/ell); else % cross covariances Kxz K = sq_dist(x'/ell,z'/ell); end end if nargin<4 % covariances K = exp(-K/2); else % derivatives if i==1 K = exp(-K/2).*K; else error('Unknown hyperparameter') end endgpml-matlab-v3.6-2015-07-07/cov/covConst.m000644 000765 000024 00000002172 12441263146 017636 0ustar00carlstaff000000 000000 function K = covConst(hyp, x, z, i) % covariance function for a constant function. The covariance function is % parameterized as: % % k(x^p,x^q) = s2; % % The scalar hyperparameter is: % % hyp = [ log(sqrt(s2)) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. if nargin<2, K = '1'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode s2 = exp(2*hyp); % s2 n = size(x,1); if dg % vector kxx K = s2*ones(n,1); else if xeqz % symmetric matrix Kxx K = s2*ones(n); else % cross covariances Kxz K = s2*ones(n,size(z,1)); end end if nargin>3 % derivatives if i==1 K = 2*K; else error('Unknown hyperparameter') end endgpml-matlab-v3.6-2015-07-07/cov/covADD.m000644 000765 000024 00000007060 12441263146 017141 0ustar00carlstaff000000 000000 function K = covADD(cov, hyp, x, z, i) % Additive covariance function using a 1d base covariance function % cov(x^p,x^q;hyp) with individual hyperparameters hyp. % % k(x^p,x^q) = \sum_{r \in R} sf_r \sum_{|I|=r} % \prod_{i \in I} cov(x^p_i,x^q_i;hyp_i) % % hyp = [ hyp_1 % hyp_2 % ... % hyp_D % log(sf_R(1)) % ... % log(sf_R(end)) ] % % where hyp_d are the parameters of the 1d covariance function which are shared % over the different values of R(1) to R(end). % % Please see the paper Additive Gaussian Processes by Duvenaud, Nickisch and % Rasmussen, NIPS, 2011 for details. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. R = cov{1}; nh = eval(feval(cov{2})); % number of hypers per individual covariance nr = numel(R); % number of different degrees of interaction if nargin<3 % report number of hyper parameters K = ['D*', int2str(nh), '+', int2str(nr)]; return end if nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); % dimensionality sf2 = exp( 2*hyp(D*nh+(1:nr)) ); % signal variances of individual degrees Kd = Kdim(cov{2},hyp,x,z); % evaluate dimensionwise covariances K if nargin<5 % covariances EE = elsympol(Kd,max(R)); % Rth elementary symmetric polynomials K = 0; for ii=1:nr, K = K + sf2(ii)*EE(:,:,R(ii)+1); end % sf2 weighted sum else % derivatives if i <= D*nh % individual covariance function parameters j = fix(1+(i-1)/nh); % j is the dimension of the hyperparameter if dg, zj='diag'; else if xeqz, zj=[]; else zj=z(:,j); end, end dKj = feval(cov{2},hyp(nh*(j-1)+(1:nh)),x(:,j),zj,i-(j-1)*nh); % other dK=0 % the final derivative is a sum of multilinear terms, so if only one term % depends on the hyperparameter under consideration, we can factorise it % out and compute the sum with one degree less E = elsympol(Kd(:,:,[1:j-1,j+1:D]),max(R)-1); % R-1th elementary sym polyn K = 0; for ii=1:nr, K = K + sf2(ii)*E(:,:,R(ii)); end % sf2 weighted sum K = dKj.*K; elseif i <= D*nh+nr EE = elsympol(Kd,max(R)); % Rth elementary symmetric polynomials j = i-D*nh; K = 2*sf2(j)*EE(:,:,R(j)+1); % rest of the sf2 weighted sum else error('Unknown hyperparameter') end end % evaluate dimensionwise covariances K function K = Kdim(cov,hyp,x,z) [n,D] = size(x); % dimensionality nh = eval(feval(cov)); % number of hypers per individual covariance if nargin<4, z = []; end % make sure, z exists xeqz = numel(z)==0; dg = strcmp(z,'diag') && numel(z)>0; % determine mode if dg % allocate memory K = zeros(n,1,D); else if xeqz, K = zeros(n,n,D); else K = zeros(n,size(z,1),D); end end for d=1:D hyp_d = hyp(nh*(d-1)+(1:nh)); % hyperparamter of dimension d if dg K(:,:,d) = feval(cov,hyp_d,x(:,d),'diag'); else if xeqz K(:,:,d) = feval(cov,hyp_d,x(:,d)); else K(:,:,d) = feval(cov,hyp_d,x(:,d),z(:,d)); end end endgpml-matlab-v3.6-2015-07-07/cov/covGaboriso.m000644 000765 000024 00000004431 12441263146 020315 0ustar00carlstaff000000 000000 function K = covGaboriso(hyp, x, z, i) % Gabor covariance function with length scale ell and period p. The % covariance function is parameterized as: % % k(x,z) = h(x-z) with h(t) = exp(-t'*t/(2*ell^2))*cos(2*pi*sum(t)/p). % % The hyperparameters are: % % hyp = [ log(ell) % log(p) ] % % Note that covSM implements a weighted sum of Gabor covariance functions, but % using an alternative (spectral) parameterization. % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Hannes Nickisch, 2014-09-26. % % See also COVFUNCTIONS.M, COVGABORARD.M, COVSM.M. if nargin<2, K = '2'; return; end % report no of params if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); % dimensionality ell = exp(hyp(1)); % length scale p = exp(hyp(2)); % period if dg % compute squared distance d2 d2 = zeros(n,1); else if xeqz % symmetric matrix Kxx d2 = sq_dist(x'/ell); else % cross covariances Kxz d2 = sq_dist(x'/ell,z'/ell); end end dp = zeros(size(d2)); % init sum(t)/p computation if ~dg if xeqz % symmetric matrix Kxx for d=1:D, dp = dp + (x(:,d)*ones(1,size(x,1))-ones(n,1)*x(:,d)')/p; end else % cross covariances Kxz for d=1:D, dp = dp + (x(:,d)*ones(1,size(z,1))-ones(n,1)*z(:,d)')/p; end end end K = exp(-d2/2); if nargin<4 % covariances K = cos(2*pi*dp).*K; else % derivatives if i==1 % length scale K = d2.*cos(2*pi*dp).*K; elseif i==2 % period K = 2*pi*dp.*sin(2*pi*dp).*K; else error('Unknown hyperparameter') end endgpml-matlab-v3.6-2015-07-07/cov/covGaborard.m000644 000765 000024 00000005473 12441263146 020300 0ustar00carlstaff000000 000000 function K = covGaborard(hyp, x, z, i) % Gabor covariance function with length scales ell and periods p. The covariance % function is parameterized as: % % k(x,z) = h(x-z), h(t) = exp(-sum(t.^2./(2*ell.^2)))*cos(2*pi*sum(t./p)). % % The hyperparameters are: % % hyp = [ log(ell_1) % log(ell_2) % .. % log(ell_D) % log(p_1) % log(p_2) % .. % log(p_D) ] % % For more help on design of covariance functions, try "help covFunctions". % % Note that covSM implements a weighted sum of Gabor covariance functions, but % using an alternative (spectral) parameterization. % % Copyright (c) by Hannes Nickisch, 2014-09-26. % % See also COVFUNCTIONS.M, COVGABORISO, COVSM.M. if nargin<2, K = '2*D'; return; end % report no of params if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); % dimensionality ell = exp(hyp(1:D)); % length scale p = exp(hyp(D+1:2*D)); % period if dg % compute squared distance d2 d2 = zeros(size(x,1),1); else if xeqz % symmetric matrix Kxx d2 = sq_dist(diag(1./ell)*x'); else % cross covariances Kxz d2 = sq_dist(diag(1./ell)*x',diag(1./ell)*z'); end end dp = zeros(size(d2)); % init sum(t)/p computation if ~dg if xeqz % symmetric matrix Kxx for d=1:D, dp = dp + (x(:,d)*ones(1,size(x,1))-ones(n,1)*x(:,d)')/p(d); end else % cross covariances Kxz for d=1:D, dp = dp + (x(:,d)*ones(1,size(z,1))-ones(n,1)*z(:,d)')/p(d); end end end K = exp(-d2/2); if nargin<4 % covariance K = cos(2*pi*dp).*K; else % derivatives d = mod(i-1,D)+1; if dg, K = K*0; end % handle diagonal case if xeqz % symmetric matrix Kxx dd = x(:,d)*ones(1,size(x,1))-ones(n,1)*x(:,d)'; else % cross covariances Kxz dd = x(:,d)*ones(1,size(z,1))-ones(n,1)*z(:,d)'; end if i<=D % length scale K = (dd/ell(d)).^2.*cos(2*pi*dp).*K; elseif i<=2*D % period K = 2*pi*dd/p(d).*sin(2*pi*dp).*K; else error('Unknown hyperparameter') end endgpml-matlab-v3.6-2015-07-07/cov/covSM.m000644 000765 000024 00000011263 12441263146 017070 0ustar00carlstaff000000 000000 function K = covSM(Q, hyp, x, z, i) % Gaussian Spectral Mixture covariance function. The covariance function % parametrization depends on the sign of Q. % % Let t(Dx1) be an offset vector in dataspace e.g. t = x_i - z_j. Then w(DxP) % are the weights and m(Dx|Q|) = 1/p, v(Dx|Q|) = (2*pi*ell)^-2 are spectral % means (frequencies) and variances, where p is the period and ell the length % scale of the Gabor function h(t2v,tm) given by the expression % h(t2v,tm) = exp(-2*pi^2*t2v).*cos(2*pi*tm) % % Then, the two covariances are obtained as follows: % % SM, spectral mixture: Q>0 => P = 1 % k(x_i,z_j) = w'*h(v'*(t.*t),m'*t) % % SMP, spectral mixture product: Q<0 => P = D % k(x_i,z_j) = prod(w'*h(T*T*v,T*m)), T = diag(t) % % The hyperparameters are: % % hyp = [ log(w(:)) % log(m(:)) % log(sqrt(v(:))) ] % % For more help on design of covariance functions, try "help covFunctions". % % Note that the spectral density H(s) = F[ h(t) ] of covGaboriso is given by % H(s) = N(s|m,v)/2 + N(s|-m,v)/2 where m=1/p is the mean and v=(2*pi*ell)^-2 % is the variance of a symmetric Gaussian mixture. Hence the covGaboriso % covariance forms a basis for the class of stationary covariances since a % weighted sum of covGaboriso covariances corresponds to an isotropic % location-scale mixture of a symmetric Gaussian mixture in the spectral domain. % % Internally, covSM constructs a weighted sum of products of 1d covGaboriso % covariances using covMask, covProd, covScale and covSum. % % For more details, see % 1) Gaussian Process Kernels for Pattern Discovery and Extrapolation, % ICML, 2013, by Andrew Gordon Wilson and Ryan Prescott Adams. % 2) GPatt: Fast Multidimensional Pattern Extrapolation with Gaussian % Processes, arXiv 1310.5288, 2013, by Andrew Gordon Wilson, Elad Gilboa, % Arye Nehorai and John P. Cunningham, and % http://mlg.eng.cam.ac.uk/andrew/pattern % % For Q>0, covSM corresponds to Eq. 12 in Ref (1) % For Q<0, covSM corresponds to Eq. 14 in Ref (2) (but w here = w^2 in (14)) % % Copyright (c) by Andrew Gordon Wilson and Hannes Nickisch, 2014-09-24. % % See also COVFUNCTIONS.M, COVGABORISO.M, COVGABORARD.M. smp = Q<0; Q = abs(Q); % switch between covSM and covSMP mode if nargin<3 % report no of parameters if smp, K = '3*D*'; else K = '(1+2*D)*'; end, K = [K,sprintf('%d',Q)]; return end if nargin<4, z = []; end % make sure, z exists D = size(x,2); P = smp*D+(1-smp); % dimensionality, P=D or P=1 lw = reshape(hyp( 1:P*Q) ,P,Q); % log mixture weights lm = reshape(hyp(P*Q+ (1:D*Q)),D,Q); % log spectral means ls = reshape(hyp(P*Q+D*Q+(1:D*Q)),D,Q); % log spectral standard deviations % In the following, we construct nested cell arrays to finally obtain either if smp % 1) the product of weighted sums of 1d covGabor covariance functions or fac = cell(1,D); for d=1:D add = cell(1,Q); % a) addends for weighted sum of univariate Gabor functions for q=1:Q, add{q} = {'covScale',{'covMask',{d,{'covGaboriso'}}}}; end fac{d} = {'covSum',add}; % b) combine addends into sum end cov = {'covProd',fac}; % c) combine factors into product else % 2) the weighted sum of multivariate covGaborard covariance functions. % weighted sum of multivariate Gabor functions add = cell(1,Q); for q=1:Q, add{q} = {'covScale',{'covGaborard'}}; end cov = {'covSum',add}; % combine into sum end if smp % assemble hyp; covGabor is parametrised using -ls-log(2*pi) and -lm hyp = [lw(:)'/2; -ls(:)'-log(2*pi); -lm(:)']; else hyp = [lw/2; -ls-log(2*pi); -lm ]; end if nargin<5 % evaluation of the covariance K = feval(cov{:},hyp(:),x,z); else % We compute the indices j in the new hyperparameter vector hyp. The % correction constants c are needed because some hyperparameters in hyp are % powers of hyperparameters accepted by covSM. if i<=P*Q % derivatives w.r.t. w c = 0.5; if smp, j = 1+3*(i-1); else j = 1+(i-1)*(2*D+1); end elseif i<=(P+ D)*Q % derivatives w.r.t. m c = -1.0; j = i- P *Q; [j1,j2] = ind2sub([D,Q],j); if smp, j = 3+3*(j-1); else j = 1+j1+D+(j2-1)*(2*D+1); end elseif i<=(P+2*D)*Q % derivatives w.r.t. v c = -1.0; j = i-(P+D)*Q; [j1,j2] = ind2sub([D,Q],j); if smp, j = 2+3*(j-1); else j = 1+j1+ (j2-1)*(2*D+1); end else error('Unknown hyperparameter') end K = c*feval(cov{:},hyp(:),x,z,j); endgpml-matlab-v3.6-2015-07-07/cov/covFITC.m000644 000765 000024 00000003341 12441263146 017274 0ustar00carlstaff000000 000000 function [K,Kuu,Ku] = covFITC(cov, xu, hyp, x, z, i) % Covariance function to be used together with the FITC approximation. % % The function allows for more than one output argument and does not respect the % interface of a proper covariance function. In fact, it wraps a proper % covariance function such that it can be used together with infFITC.m. % Instead of outputing the full covariance, it returns cross-covariances between % the inputs x, z and the inducing inputs xu as needed by infFITC.m % % Copyright (c) by Ed Snelson, Carl Edward Rasmussen % and Hannes Nickisch, 2011-11-02. % % See also COVFUNCTIONS.M, INFFITC.M. if nargin<4, K = feval(cov{:}); return, end if nargin<5, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode if size(xu,2) ~= size(x,2) error('Dimensionality of inducing inputs must match training inputs'); end if nargin<6 % covariances if dg K = feval(cov{:},hyp,x,'diag'); else if xeqz K = feval(cov{:},hyp,x,'diag'); if nargout>1, Kuu = feval(cov{:},hyp,xu); end if nargout>2, Ku = feval(cov{:},hyp,xu,x); end else K = feval(cov{:},hyp,xu,z); end end else % derivatives if dg K = feval(cov{:},hyp,x,'diag',i); else if xeqz K = feval(cov{:},hyp,x,'diag',i); if nargout>1, Kuu = feval(cov{:},hyp,xu,[],i); end if nargout>2, Ku = feval(cov{:},hyp,xu,x,i); end else K = feval(cov{:},hyp,xu,z,i); end end endgpml-matlab-v3.6-2015-07-07/cov/covPeriodic.m000644 000765 000024 00000003204 12441263146 020303 0ustar00carlstaff000000 000000 function K = covPeriodic(hyp, x, z, i) % Stationary covariance function for a smooth periodic function, with period p % in 1d (see covPERiso and covPERard for multivariate data): % % k(x,z) = sf^2 * exp( -2*sin^2( pi*||x-z||/p )/ell^2 ) % % where the hyperparameters are: % % hyp = [ log(ell) % log(p) % log(sf) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2011-01-05. % % See also COVFUNCTIONS.M. if nargin<2, K = '3'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); if D>1, error('Covariance is defined for 1d data only.'), end ell = exp(hyp(1)); p = exp(hyp(2)); sf2 = exp(2*hyp(3)); % precompute distances if dg % vector kxx K = zeros(size(x,1),1); else if xeqz % symmetric matrix Kxx K = sqrt(sq_dist(x')); else % cross covariances Kxz K = sqrt(sq_dist(x',z')); end end K = pi*K/p; if nargin<4 % covariances K = sin(K)/ell; K = K.*K; K = sf2*exp(-2*K); else % derivatives if i==1 K = sin(K)/ell; K = K.*K; K = 4*sf2*exp(-2*K).*K; elseif i==2 R = sin(K)/ell; K = 4*sf2/ell*exp(-2*R.*R).*R.*cos(K).*K; elseif i==3 K = sin(K)/ell; K = K.*K; K = 2*sf2*exp(-2*K); else error('Unknown hyperparameter') end endgpml-matlab-v3.6-2015-07-07/cov/covNNone.m000644 000765 000024 00000004016 12441263146 017564 0ustar00carlstaff000000 000000 function K = covNNone(hyp, x, z, i) % Neural network covariance function with a single parameter for the distance % measure. The covariance function is parameterized as: % % k(x^p,x^q) = sf2 * asin(x^p'*P*x^q / sqrt[(1+x^p'*P*x^p)*(1+x^q'*P*x^q)]) % % where the x^p and x^q vectors on the right hand side have an added extra bias % entry with unit value. P is ell^-2 times the unit matrix and sf2 controls the % signal variance. The hyperparameters are: % % hyp = [ log(ell) % log(sqrt(sf2) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. if nargin<2, K = '2'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode n = size(x,1); ell2 = exp(2*hyp(1)); sf2 = exp(2*hyp(2)); sx = 1 + sum(x.*x,2); if dg % vector kxx K = sx./(sx+ell2); else if xeqz % symmetric matrix Kxx S = 1 + x*x'; K = S./(sqrt(ell2+sx)*sqrt(ell2+sx)'); else % cross covariances Kxz S = 1 + x*z'; sz = 1 + sum(z.*z,2); K = S./(sqrt(ell2+sx)*sqrt(ell2+sz)'); end end if nargin<4 % covariances K = sf2*asin(K); else % derivatives if i==1 % lengthscale if dg V = K; else vx = sx./(ell2+sx); if xeqz V = repmat(vx/2,1,n) + repmat(vx'/2,n,1); else vz = sz./(ell2+sz); nz = size(z,1); V = repmat(vx/2,1,nz) + repmat(vz'/2,n,1); end end K = -2*sf2*(K-K.*V)./sqrt(1-K.*K); elseif i==2 % magnitude K = 2*sf2*asin(K); else error('Unknown hyperparameter') end endgpml-matlab-v3.6-2015-07-07/cov/covEye.m000644 000765 000024 00000002576 12441263146 017302 0ustar00carlstaff000000 000000 function K = covEye(hyp, x, z, i) % Independent covariance function, i.e. "white noise", with unit variance. % The covariance function is specified as: % % k(x^p,x^q) = \delta(p,q) % % \delta(p,q) is a Kronecker delta function which is 1 iff p=q and zero % otherwise in mode 1). % In cross covariance mode 2) two data points x_p and z_q are considered equal % if their difference norm |x_p-z_q| is less than eps, the machine precision. % The hyperparameters are: % % hyp = [ ] % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Hannes Nickisch, 2014-09-05. % % See also COVFUNCTIONS.M. tol = eps; % threshold on the norm when two vectors are considered to be equal if nargin<2, K = '0'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists dg = strcmp(z,'diag'); % determine mode n = size(x,1); % precompute raw if dg % vector kxx K = ones(n,1); else if isempty(z) % symmetric matrix Kxx K = eye(n); else % cross covariances Kxz K = double(sq_dist(x',z')1, mask = find(mask); end % convert 1/0->index D = length(mask); % masked dimension if nargin<3, K = num2str(eval(nh_string)); return, end % number of parameters if nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode if eval(nh_string)~=length(hyp) % check hyperparameters error('number of hyperparameters does not match size of masked data') end if nargin<5 % covariances if dg K = feval(cov{:}, hyp, x(:,mask), 'diag'); else if xeqz K = feval(cov{:}, hyp, x(:,mask)); else K = feval(cov{:}, hyp, x(:,mask), z(:,mask)); end end else % derivatives if i <= eval(nh_string) if dg K = feval(cov{:}, hyp, x(:,mask), 'diag', i); else if xeqz K = feval(cov{:}, hyp, x(:,mask), [], i); else K = feval(cov{:}, hyp, x(:,mask), z(:,mask), i); end end else error('Unknown hyperparameter') end endgpml-matlab-v3.6-2015-07-07/cov/covSEfact.m000644 000765 000024 00000006042 12504606111 017706 0ustar00carlstaff000000 000000 function K = covSEfact(d, hyp, x, z, i) % Factor analysis squared exponential covariance. % % The covariance function is parameterized as: % % k(x,z) = sf^2 * covSEiso(x*L',z*L'), % % where x of size (n,D) and z of size (nz,D) are the inputs and, L is an % embedding matrix of size (dxD), and sf is the signal standard deviation. % The low-rank matrix L'*L is the metric used to measure distances between % data inputs x an z; dist(x,z) = ||x-z||_{L'*L} % % The hyperparameters are: % hyp = [ hypL; % log(sf) ], % % where hypL is the vectorized L matrix. % The conversion between hypL and L is achieved by the following code: % % conversion L -> hypL % >> if d==1, diagL = L(1); else diagL = diag(L); end % >> L(1:d+1:d*d) = log(diagL); hyp = L(triu(true(d,D))); % % conversion hypL -> L % >> L = zeros(d,D); L(triu(true(d,D))) = hyp(:); % >> if d==1, diagL = L(1); else diagL = diag(L); end % >> L(1:d+1:d*d) = exp(diagL); % % hypL = [ log(L_11) % L_21 % log(L_22) % L_31 % L_32 % .. % log(L_dd) % .. % L_dD] % % Copyright (c) by Roman Garnett & Hannes Nickisch, 2014-08-15. % % See also COVFUNCTIONS.M. if nargin==0, error('d must be specified.'), end nh = sprintf('(D*%d - %d*(%d-1)/2 + 1)',d,d,d); % number of hyperparam string if nargin<3, K = nh; return; end % report number of hyperparameters if nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode if xeqz, z = x; end % make sure we have a valid z [n,D] = size(x); % dimensionality if d>D, error('We need d<=D.'), end nh = eval(nh); sf = exp(hyp(nh)); hyp = hyp(1:nh-1); % bring hypers in correct shape L = zeros(d,D); L(triu(true(d,D))) = hyp(:); % embedding matrix if d==1, diagL = L(1); else diagL = diag(L); end % properly handle limit case L(1:d+1:d*d) = exp(diagL); xl = x*L'; % transform input data if dg, zl = 'diag'; else if xeqz, zl = xl; else zl = z*L'; end end if nargin<5 % covariance, call covSEiso on the embedded points K = covSEiso([0;log(sf)],xl,zl); else if i==nh % derivative w.r.t. log signal standard deviation K = covSEiso([0;log(sf)],xl,zl,2); else td = d*(d+1)/2; % determine indices in L if i>td col = ceil((i-td)/d); col = col+ceil((sqrt(8*(i-d*col)+1)-1)/2); row = mod(i-d*(d+1)/2-1,d)+1; else col = ceil((sqrt(8*i+1)-1)/2); row = i-col*(col-1)/2; end if dg K = zeros(n,1); else K = covSEiso([0;log(sf)],xl,zl); % derivative w.r.t. log lengthscale m = size(z,1); F = x(:,col)*ones(1,m)-ones(n,1)*z(:,col)'; K = -K.*F.*(xl(:,row)*ones(1,m)-ones(n,1)*zl(:,row)'); if row==col, K = L(row,col)*K; end % diagonal is in the log domain end end end gpml-matlab-v3.6-2015-07-07/cov/covPPard.m000644 000765 000024 00000006022 12504606111 017545 0ustar00carlstaff000000 000000 function K = covPPard(v, hyp, x, z, i) % Piecewise polynomial covariance function with compact support, v = 0,1,2,3. % The covariance functions are 2v times contin. diff'ble and the corresponding % processes are hence v times mean-square diffble. The covariance function is: % % k(x^p,x^q) = s2f * max(1-r,0)^(j+v) * f(r,j) with j = floor(D/2)+v+1 % % where r is the distance sqrt((x^p-x^q)'*inv(P)*(x^p-x^q)), and the P matrix % is diagonal with ARD parameters ell_1^2,...,ell_D^2, where D is the dimension % of the input space and sf2 is the signal variance. The hyperparameters are: % % hyp = [ log(ell_1) % log(ell_2) % .. % log(ell_D) % log(sf) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-02-09. % % See also COVFUNCTIONS.M. if nargin<3, K = '(D+1)'; return; end % report number of parameters if nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); ell = exp(hyp(1:D)); sf2 = exp(2*hyp(D+1)); if all(v~=[0,1,2,3]), error('only 0,1,2 and 3 allowed for v'), end % degree j = floor(D/2)+v+1; % exponent switch v case 0, f = @(r,j) 1; df = @(r,j) 0; case 1, f = @(r,j) 1 + (j+1)*r; df = @(r,j) (j+1); case 2, f = @(r,j) 1 + (j+2)*r + ( j^2+ 4*j+ 3)/ 3*r.^2; df = @(r,j) (j+2) + 2*( j^2+ 4*j+ 3)/ 3*r; case 3, f = @(r,j) 1 + (j+3)*r + (6*j^2+36*j+45)/15*r.^2 ... + (j^3+9*j^2+23*j+15)/15*r.^3; df = @(r,j) (j+3) + 2*(6*j^2+36*j+45)/15*r ... + (j^3+9*j^2+23*j+15)/ 5*r.^2; end cs = @(r,e) (r<1).*max(1-r,0).^e; pp = @(r,j,v,f) cs(r,j+v ).* f(r,j); dpp = @(r,j,v,f) r.*cs(r,j+v-1).*( f(r,j)*(j+v) - max(1-r,0).*df(r,j) ); % precompute squared distances if dg % vector kxx K = zeros(size(x,1),1); else if xeqz % symmetric matrix Kxx K = sqrt( sq_dist(diag(1./ell)*x') ); else % cross covariances Kxz K = sqrt( sq_dist(diag(1./ell)*x',diag(1./ell)*z') ); end end if nargin<5 % covariances K = sf2*pp( K, j, v, f ); else % derivatives if i<=D % length scale parameter if dg Ki = zeros(size(x,1),1); else if xeqz Ki = sq_dist(1/ell(i)*x(:,i)'); else Ki = sq_dist(1/ell(i)*x(:,i)',1/ell(i)*z(:,i)'); end end K = sf2*dpp( K, j, v, f ).*Ki./K.^2; K(Ki<1e-12) = 0; % fix limit case elseif i==D+1 % magnitude parameter K = 2*sf2*pp( K, j, v, f ); else error('Unknown hyperparameter') end endgpml-matlab-v3.6-2015-07-07/cov/covScale.m000644 000765 000024 00000005743 12441263146 017606 0ustar00carlstaff000000 000000 function K = covScale(cov, lsf, hyp, x, z, i) % covScale - compose a covariance function as a scaled version of another % one to model functions of the form f(x) = sf(x) f0(x), where sf(x) is a % scaling function determining the function's standard deviation given f0(x) % is normalised. % % The covariance function is parameterized as: % k(x,z) = sf(x) * k_0(x,z) * sf(z) % with an important special case being % k(x,z) = sf^2 * k_0(x,z). % % You can either use K = covScale(cov, lsf, hyp, x, z, i) where the log scaling % function lsf is a GPML mean function with hyperparameters hyp_sf yielding % hyp = [ hyp_lsf % hyp_cov ] % as hyperparameters % or you can use covScale(cov, hyp, x, z, i) to perform % rescaling by a scalar value sf specified as an additional variable yielding % hyp = [ log(sf) % hyp_cov ] % as hyperparameters. % % Copyright (c) by Carl Edward Rasmussen, Hannes Nickisch & Roman Garnett % 2014-09-05. % % See also COVFUNCTIONS.M. if nargin==0, error('cov function must be specified'), end if nargin<=1, lsf = []; end, narg = nargin; % set a default value if isnumeric(lsf)&&~isempty(lsf) % shift parameters if sf contains actually hyp if nargin>4, i = z; end if nargin>3, z = x; end if nargin>2, x = hyp; end if nargin>1, hyp = lsf; end narg = nargin+1; lsf = []; end % below we us narg instead of nargin to be independent of the parameter shift if isempty(lsf), ssf = '1'; else ssf = feval(lsf{:}); end % number of hypers if narg<4, K = ['(',feval(cov{:}),'+',ssf,')']; return, end if narg<5, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); % dimension of input data nsf = eval(ssf); hyp_lsf = hyp(1:nsf); % number of params, split hyp ncov = eval(feval(cov{:})); hyp_cov = hyp(nsf+(1:ncov)); scalar = isempty(lsf); if scalar, sf = exp(hyp_lsf); end if scalar, sfx = sf; else sfx = exp(feval(lsf{:},hyp_lsf,x)); end if dg K = sfx.*sfx; else if xeqz, sfz = sfx; else if scalar, sfz = sf; else sfz = exp(feval(lsf{:},hyp_lsf,z)); end, end K = sfx*sfz'; end if narg<6 % covariances K = K.*feval(cov{:},hyp_cov,x,z); else % derivatives if i>nsf % wrt covariance function hyperparameters K = K.*feval(cov{:},hyp_cov,x,z,i-nsf); else % wrt scaling function hyperparameters K = feval(cov{:},hyp_cov,x,z); if scalar, dsfx = sfx; else dsfx = sfx.*feval(lsf{:},hyp_lsf,x,i); end if dg K = (2*sfx.*dsfx).*K; else if xeqz, dsfz = dsfx; else if scalar, dsfz = sfz; else dsfz = sfz.*feval(lsf{:},hyp_lsf,z,i); end end K = (sfx*dsfz'+dsfx*sfz').*K; end end endgpml-matlab-v3.6-2015-07-07/cov/covRQiso.m000644 000765 000024 00000003473 12441263146 017612 0ustar00carlstaff000000 000000 function K = covRQiso(hyp, x, z, i) % Rational Quadratic covariance function with isotropic distance measure. The % covariance function is parameterized as: % % k(x^p,x^q) = sf^2 * [1 + (x^p - x^q)'*inv(P)*(x^p - x^q)/(2*alpha)]^(-alpha) % % where the P matrix is ell^2 times the unit matrix, sf2 is the signal % variance and alpha is the shape parameter for the RQ covariance. The % hyperparameters are: % % hyp = [ log(ell) % log(sf) % log(alpha) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. if nargin<2, K = '3'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode ell = exp(hyp(1)); sf2 = exp(2*hyp(2)); alpha = exp(hyp(3)); % precompute squared distances if dg % vector kxx D2 = zeros(size(x,1),1); else if xeqz % symmetric matrix Kxx D2 = sq_dist(x'/ell); else % cross covariances Kxz D2 = sq_dist(x'/ell,z'/ell); end end if nargin<4 % covariances K = sf2*(1+0.5*D2/alpha).^(-alpha); else % derivatives if i==1 % length scale parameter K = sf2*(1+0.5*D2/alpha).^(-alpha-1).*D2; elseif i==2 % magnitude parameter K = 2*sf2*(1+0.5*D2/alpha).^(-alpha); elseif i==3 K = (1+0.5*D2/alpha); K = sf2*K.^(-alpha).*(0.5*D2./K - alpha*log(K)); else error('Unknown hyperparameter') end endgpml-matlab-v3.6-2015-07-07/cov/covNoise.m000644 000765 000024 00000003321 12441263146 017622 0ustar00carlstaff000000 000000 function K = covNoise(hyp, x, z, i) % Independent covariance function, i.e. "white noise", with specified variance. % The covariance function is specified as: % % k(x^p,x^q) = s2 * \delta(p,q) % % where s2 is the noise variance and \delta(p,q) is a Kronecker delta function % which is 1 iff p=q and zero otherwise in mode 1). % In cross covariance mode 2) two data points x_p and z_q are considered equal % if their difference norm |x_p-z_q| is less than eps, the machine precision. % The hyperparameter is: % % hyp = [ log(sqrt(s2)) ] % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-05-15. % % See also COVFUNCTIONS.M. tol = eps; % threshold on the norm when two vectors are considered to be equal if nargin<2, K = '1'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists dg = strcmp(z,'diag'); % determine mode n = size(x,1); s2 = exp(2*hyp); % noise variance % precompute raw if dg % vector kxx K = ones(n,1); else if numel(z)==0 % symmetric matrix Kxx K = eye(n); else % cross covariances Kxz K = double(sq_dist(x',z')1, error('Covariance is defined for 1d data only.'), end p = exp(hyp(1)); sf2 = exp(2*hyp(2)); % precompute distances if dg % vector kxx K = zeros(size(x,1),1); else if xeqz % symmetric matrix Kxx K = repmat(x,1,n) - repmat(x',n,1); else % cross covariances Kxz K = repmat(x,1,size(z,1)) - repmat(z',n,1); end end K = 2*pi*K/p; if nargin<4 % covariances K = sf2*cos(K); else % derivatives if i==1 K = sf2*sin(K).*K; elseif i==2 K = 2*sf2*cos(K); else error('Unknown hyperparameter') end endgpml-matlab-v3.6-2015-07-07/cov/covMaternard.m000644 000765 000024 00000005144 12441263146 020467 0ustar00carlstaff000000 000000 function K = covMaternard(d, hyp, x, z, i) % Matern covariance function with nu = d/2 and with Automatic Relevance % Determination (ARD) distance measure. For d=1 the function is also known as % the exponential covariance function or the Ornstein-Uhlenbeck covariance % in 1d. The covariance function is: % % k(x^p,x^q) = sf^2 * f( sqrt(d)*r ) * exp(-sqrt(d)*r) % % with f(t)=1 for d=1, f(t)=1+t for d=3 and f(t)=1+t+t²/3 for d=5. % Here r is the distance sqrt((x^p-x^q)'*inv(P)*(x^p-x^q)), where the P matrix % is diagonal with ARD parameters ell_1^2,...,ell_D^2, where D is the dimension % of the input space and sf2 is the signal variance. The hyperparameters are: % % hyp = [ log(ell_1) % log(ell_2) % .. % log(ell_D) % log(sf) ] % % Copyright (c) by Hannes Nickisch, 2013-10-13. % % See also COVFUNCTIONS.M. if nargin<3, K = '(D+1)'; return; end % report number of parameters if nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); ell = exp(hyp(1:D)); sf2 = exp(2*hyp(D+1)); if all(d~=[1,3,5]), error('only 1, 3 and 5 allowed for d'), end % degree switch d case 1, f = @(t) 1; df = @(t) 1./t; % df(t) = (f(t)-f'(t))/t case 3, f = @(t) 1 + t; df = @(t) 1; case 5, f = @(t) 1 + t.*(1+t/3); df = @(t) (1+t)/3; end m = @(t,f) f(t).*exp(-t); dm = @(t,f) df(t).*exp(-t); % precompute distances if dg % vector kxx K = zeros(size(x,1),1); else if xeqz % symmetric matrix Kxx K = sq_dist(diag(sqrt(d)./ell)*x'); else % cross covariances Kxz K = sq_dist(diag(sqrt(d)./ell)*x',diag(sqrt(d)./ell)*z'); end end if nargin<5 % covariances K = sf2*m(sqrt(K),f); else % derivatives if i<=D % length scale parameter if dg Ki = zeros(size(x,1),1); else if xeqz Ki = sq_dist(sqrt(d)/ell(i)*x(:,i)'); else Ki = sq_dist(sqrt(d)/ell(i)*x(:,i)',sqrt(d)/ell(i)*z(:,i)'); end end K = sf2*dm(sqrt(K),f).*Ki; K(Ki<1e-12) = 0; % fix limit case for d=1 elseif i==D+1 % magnitude parameter K = 2*sf2*m(sqrt(K),f); else error('Unknown hyperparameter') end endgpml-matlab-v3.6-2015-07-07/cov/covDiscrete.m000644 000765 000024 00000004541 12441263146 020314 0ustar00carlstaff000000 000000 function K = covDiscrete(s, hyp, x, z, i) % Covariance function for discrete inputs. Given a function defined on the % integers 1,2,3,..,s, the covariance function is parameterized as: % % k(x,z) = K_{xz}, % % where K is a matrix of size (s x s). % % This implementation assumes that the inputs x and z are given as integers % between 1 and s, which simply index the matrix K. % % The hyperparameters specify the upper-triangular part of the Cholesky factor % of K, where the (positive) diagonal elements are specified by their logarithm. % If L = chol(K), so K = L'*L, then the hyperparameters are: % % hyp = [ log(L_11) % L_21 % log(L_22) % L_31 % L_32 % .. % log(L_ss) ] % % The hyperparameters hyp can be generated from K using: % L = chol(K); L(1:(s+1):end) = log(diag(L)); % hyp = L(triu(true(s))); % % The covariance matrix K is obtained from the hyperparameters hyp by: % L = zeros(s); L(triu(true(s))) = hyp(:); % L(1:(s+1):end) = exp(diag(L)); K = L'*L; % % This parametrization allows unconstrained optimization of K. % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Roman Garnett, 2014-08-14. % % See also MEANDISCRETE.M, COVFUNCTIONS.M. if nargin==0, error('s must be specified.'), end % check for dimension if nargin<3, K = num2str(s*(s+1)/2); return; end % report number of parameters if nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode if xeqz, z = x; end % make sure we have a valid z L = zeros(s); L(triu(true(s))) = hyp(:); % build Cholesky factor L(1:(s+1):end) = exp(diag(L)); if nargin<5 A = L'*L; % A is a placeholder for K to avoid a name clash with the return arg if dg K = A(sub2ind(size(A),x,x)); else K = A(x,z); end else col = ceil((sqrt(8*i+1)-1)/2); row = i-col*(col-1)/2; % indices by tri-root dL = zeros(s); % derivative of Cholesky factor if (row == col) dL(row,col) = L(row,col); % diagonal entries have exp transformation else dL(row,col) = 1; end dK = dL'*L; dK = dK+dK'; % derivative of sxs covariance matrix if dg K = dK(sub2ind(size(dK),x,x)); else K = dK(x,z); end endgpml-matlab-v3.6-2015-07-07/cov/covSEiso.m000644 000765 000024 00000003166 12441263146 017576 0ustar00carlstaff000000 000000 function K = covSEiso(hyp, x, z, i) % Squared Exponential covariance function with isotropic distance measure. The % covariance function is parameterized as: % % k(x^p,x^q) = sf^2 * exp(-(x^p - x^q)'*inv(P)*(x^p - x^q)/2) % % where the P matrix is ell^2 times the unit matrix and sf^2 is the signal % variance. The hyperparameters are: % % hyp = [ log(ell) % log(sf) ] % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. if nargin<2, K = '2'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode ell = exp(hyp(1)); % characteristic length scale sf2 = exp(2*hyp(2)); % signal variance % precompute squared distances if dg % vector kxx K = zeros(size(x,1),1); else if xeqz % symmetric matrix Kxx K = sq_dist(x'/ell); else % cross covariances Kxz K = sq_dist(x'/ell,z'/ell); end end if nargin<4 % covariances K = sf2*exp(-K/2); else % derivatives if i==1 K = sf2*exp(-K/2).*K; elseif i==2 K = 2*sf2*exp(-K/2); else error('Unknown hyperparameter') end endgpml-matlab-v3.6-2015-07-07/cov/covGrid.m000644 000765 000024 00000023525 12504606111 017433 0ustar00carlstaff000000 000000 function [K,Mx,xe] = covGrid(cov, xg, hyp, x, z, i) % covGrid - Kronecker covariance function based on a grid. % % The grid g is represented by its p axes xg = {x1,x2,..xp}. An axis xi is of % size (ni,di) and the grid g has size (n1,n2,..,np,D), where D=d1+d2+..+dp. % Hence, the grid contains N=n1*n2*..*np data points. The axes do neither need % to be sorted in any way nor do they need to be 1d i.e. ni>=1. % % The covGrid function can be used to expand the cell array xg into an expanded % multivariate grid xe of size (N,D) via: % [xe,nx,Dx] = covGrid('expand',xg); => mode 1) % The operation can be reverted by: % xg = covGrid('factor',{xe,ng,Dg}); => mode 2) % % Given scattered data x of size (n,D), we can create a grid xg covering % the support of x using: % xg = covGrid('create',x,equi,k); => mode 3) % The flag equi (default value 1) can be used to enforce an equispaced % grid. The integer k, indicates the number of grid points per dimension. If % k is a real number from (0,1], then the number of grid points equals % k*numel(unique(x(:,1))). % We require at least two different components per dimension. % % The variables v={x,z} can either be a) grid indices or b) data points. % a) The variable v has size (nv,1) and contains integers from [1,N]. Then % the datapoints are obtained as g2 = reshape(g,N,D); v = g2(v,:). % b) The variable v has size (nv,D) and directly represents the data points. % At the moment, this is only supported for z. % % An arbitrary data point x -- be it an index vector of size (n,1) or a data % point of size (n,D) -- is converted into a regular data point xx of % size (n,D) by: % xx = covGrid('idx2dat',xg,x); => mode 4) % If x is already of size (n,D), xx will simply equal x. % % The resulting covariance matrix is given by: % K = kron( kron(...,K{2}), K{1} ) = K_p x .. x K_2 x K_1. % % The hyperparameters are: % hyp = [ hyp_1 % hyp_2 % .. % hyp_p ], % % Copyright (c) by Hannes Nickisch and Andrew Wilson 2015-03-05. % % See also COVFUNCTIONS.M, INFGRID.M. if nargin<2, error('Not enough parameters provided.'), end % mode 1) expand axes xg representation into full grid x if strcmp(cov,'expand') % call: [xe,nx,Dx] = covGrid('expand',xg); [K,Mx,xe] = expandgrid(xg); return % mode 2) factor full x grid into axes representation xg elseif strcmp(cov,'factor') % call: xg = covGrid('factor',{xe,ng,Dg}); K = factorgrid(xg{:}); return % mode 3) create axes representation xg from scattered data elseif strcmp(cov,'create') % call: xg = covGrid('create',x,equi,k); if nargin<3, equi = 1; else equi = hyp; end % set default values if nargin<4, k = 1; else k = x; end, x = xg; % set input params p = size(x,2); xg = cell(p,1); % allocate result if numel(k)>0, k = ones(p,1).*k(:); end % enforce vector-valued k for j=1:p % iterate over dimensions u = sort(unique(x(:,j))); if numel(u)<2, error('Two few unique points.'),end if isempty(k) % determine number of grid points if equi ngj = ceil( (u(end)-u(1))/min(abs(diff(u))) ); % use minimum spacing else ngj = numel(u); end elseif 0<=k(j) && k(j)<=1 ngj = ceil(k(j)*numel(u)); else ngj = k(j); end du = (u(end)-u(1))/ngj; bu = [u(1)-5*du, u(end)+5*du]; if equi % equispaced grid xg{j} = linspace(bu(1),bu(2),max(ngj,5))'; % at least 5 grid points else % non-equispaced grid [idx,xgj] = kmeans(u,min(numel(u),ngj-2)); xgj = sort(xgj(:))'; % cluster nb = ngj-numel(xgj); nb1 = floor(nb/2); nb2 = nb - nb1; % size of boundary xg1 = linspace(bu(1),xgj(1),nb1+1); xg2 = linspace(xgj(end),bu(2),nb2+1); xg{j} = [xg1(1:nb1), xgj, xg2(1+(1:nb2))]'; end end K = xg; return % mode 4) convert possible index vector into data space elseif strcmp(cov,'idx2dat') % call: xx = covGrid('idx2dat',xg,x); N = 1; for i=1:numel(xg), N = N*size(xg{i},1); end if isidx(hyp,N), xe = expandgrid(xg); K = xe(hyp,:); else K = hyp; end, return end % mode 0) regular covariance function computations p = numel(xg); ng = zeros(p,1); Dg = zeros(p,1); % number of Kronecker factors if numel(cov)~=p, error('We require p factors.'), end for ii = 1:p % iterate over covariance functions [ng(ii),Dg(ii)] = size(xg{ii}); f = cov(ii); if iscell(f{:}), f = f{:}; end % expand cell array if necessary D = Dg(ii); j(ii) = cellstr(num2str(eval(feval(f{:})))); % collect nbr hypers end if nargin<4 % report number of parameters K = char(j(1)); for ii=2:length(cov), K = [K, '+', char(j(ii))]; end, return end if nargin<5, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode v = []; % v vector indicates to which covariance parameters belong for ii = 1:p, v = [v repmat(ii, 1, eval(char(j(ii))))]; end if nargin==6 && i>length(v), error('Unknown hyperparameter'), end equi = true(p,1); % grid along dimension is equispaced -> Toeplitz structure for ii=1:p if ng(ii)>1 % diagnose Toeplitz structure if data is linearly increasing dev = abs(diff(xg{ii})-ones(ng(ii)-1,1)*(xg{ii}(2,:)-xg{ii}(1,:))); equi(ii) = max(dev(:))<1e-9; end end N = prod(ng); n = size(x,1); D = sum(Dg); % expanded grid and data dimension ix = isidx(x,N); % determine whether x is an index or a data array if ~ix && nargout>1 % off-grid interpolation error('No off-grid values for x allowed.') end if dg % evaluate as full dense vector for diagonal covariance case K = 1; % xg is not assumed to form a grid for z = 'diag' for ii = 1:length(cov) % iteration over factor functions f = cov(ii); if iscell(f{:}), f = f{:}; end % expand cell array if necessary d = sum(Dg(1:ii-1))+(1:Dg(ii)); % dimensions of interest if nargin<6, i = 0; vi = 0; else vi = v(i); end; % which covariance function if i<=length(v) if ix, xii = xg{ii}; else xii = x(:,d); end % switch Kronecker/plain prod if ii==vi j = sum(v(1:i)==vi); % which parameter in that covariance Kj = feval(f{:}, hyp(v==ii), xii, z, j); % deriv Kronecker factor else Kj = feval(f{:}, hyp(v==ii), xii, z); % plain Kronecker factor end if ix, K = kron(K,Kj); else K = K.*Kj; end % switch Kronecker/plain prod else error('Unknown hyperparameter') end end if ix, K = K(x); end, return end if isidx(z,N), iz = z; z = covGrid('expand',xg); z = z(iz,:); end % expand z K = cell(p,1); % covariance Kronecker factors for ii = 1:length(cov) % iteration over factor functions f = cov(ii); if iscell(f{:}), f = f{:}; end % expand cell array if necessary d = sum(Dg(1:ii-1))+(1:Dg(ii)); % dimensions of interest if isnumeric(z) && ~isempty(z) % cross terms zd = z(:,d); elseif xeqz && equi(ii) % we have Toeplitz structure zd = xg{ii}(1,:); else % symmetric matrix zd = z; end if nargin<6, i = 0; vi = 0; else vi = v(i); end; % which covariance function if i<=length(v) if ii==vi j = sum(v(1:i)==vi); % which parameter in that covariance K{ii} = feval(f{:}, hyp(v==ii), xg{ii}, zd, j); % deriv Kronecker factor else K{ii} = feval(f{:}, hyp(v==ii), xg{ii}, zd); % plain Kronecker factor end else error('Unknown hyperparameter') end if xeqz && equi(ii), K{ii} = {'toep',K{ii}}; end % make Toeplitz explicit end if ~xeqz % expand cross terms Ks = K; K = Ks{1}; for ii = 2:p, K = kron1(Ks{ii},K); end if ix, if numel(x)~=N || max(abs(x-(1:N)'))>0, K = K(x,:); end else K = Mx*K; end end if nargout>1, if ix, Mx = sparse(1:n,x,1,n,N); end, end if nargout>2, xe = covGrid('expand',xg); end % perform kron along first dimension only % the code is equivalent to the following loop % z = zeros(size(x,1)*size(y,1),size(x,2)); % for i=1:size(z,2), z(:,i) = kron(x(:,i),y(:,i)); end function z = kron1(x,y) nx = size(x,1); ny = size(y,1); z = repmat(reshape(x,1,nx,[]),[ny,1,1]).*repmat(reshape(y,ny,1,[]),[1,nx,1]); z = reshape(z,nx*ny,[]); function r = isidx(i,N) % check whether i represents an integer index vector r = false; if numel(i)>0 && ~strcmp(i,'diag') && size(i,2)==1 && ndims(i)==2 if max(abs(i-floor(i)))<1e-13 if 03 % derivatives if i<=D % length scale parameters if dg K = K*0; else if xeqz K = K.*sq_dist(x(:,i)'/ell(i)); else K = K.*sq_dist(x(:,i)'/ell(i),z(:,i)'/ell(i)); end end elseif i==D+1 % magnitude parameter K = 2*K; else error('Unknown hyperparameter') end endgpml-matlab-v3.6-2015-07-07/cov/covPeriodicNoDC.m000644 000765 000024 00000007056 12231455452 021020 0ustar00carlstaff000000 000000 function K = covPeriodicNoDC(hyp, x, z, i) % Stationary covariance function for a smooth periodic function, with period p: % % k(x,x') = sf^2 * [k0(pi*(x-x')/p) - f(ell)] / [1 - f(ell)] % with k0(t) = exp( -2*sin^2(t)/ell^2 ) and f(ell) = \int 0..pi k0(t) dt. % % The constant (DC component) has been removed and marginal variance is sf^2. % The hyperparameters are: % % hyp = [ log(ell) % log(p) % log(sf) ] % % Note that covPeriodicNoDC converges to covCos as ell goes to infinity. % % Copyright (c) by James Robert Lloyd and Hannes Nickisch 2013-10-21. % % See also COVFUNCTIONS.M, COVCOS.M. if nargin<2, K = '3'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = numel(z)==0; dg = strcmp(z,'diag') && numel(z)>0; % determine mode [n,D] = size(x); if D>1, error('Covariance is defined for 1d data only.'), end ell = exp(hyp(1)); p = exp(hyp(2)); sf2 = exp(2*hyp(3)); % precompute distances if dg % vector kxx K = zeros(size(x,1),1); else if xeqz % symmetric matrix Kxx K = sqrt(sq_dist(x')); else % cross covariances Kxz K = sqrt(sq_dist(x',z')); end end K = 2*pi*K/p; if nargin<4 % covariances K = sf2*covD(K,ell); else if i==1 if ell>1e4 % limit for ell->infty K = zeros(size(K)); % no further progress in ell possible elseif 1/ell^2<3.75 cK = cos(K); ecK = exp(cK/ell^2); b0 = besseli(0,1/ell^2); b1 = besseli(1,1/ell^2); K = 2*(exp(1/ell^2)-ecK )*b1 ... - 2*(exp(1/ell^2)-ecK.*cK)*b0 ... + 4*exp(2*(cos(K/2)/ell).^2).*sin(K/2).^2; K = sf2/(ell*(exp(1/ell^2)-b0))^2 * K; else cK = cos(K); ecK = exp((cK-1)/ell^2); b0 = embi0(1/ell^2); b1 = embi1(1/ell^2); K = 2*(1-ecK)*b1 - 2*(1-ecK.*cK)*b0 ... + 4*exp(2*(cos(K/2).^2-1)/ell^2).*sin(K/2).^2; K = sf2/(ell*(1-b0))^2 * K; end elseif i==2 if ell>1e4 % limit for ell->infty K = sf2*sin(K).*K; elseif 1/ell^2<3.75 K = exp(cos(K)/ell^2).*sin(K).*K; K = sf2/ell^2/(exp(1/ell^2)-besseli(0,1/ell^2))*K; else K = exp((cos(K)-1)/ell^2).*sin(K).*K; K = sf2/ell^2/(1-embi0(1/ell^2))*K; end elseif i==3 K = 2*sf2*covD(K,ell); else error('Unknown hyperparameter') end end function K = covD(D,ell) % evaluate covariances from distances if ell>1e4 % limit for ell->infty K = cos(D); elseif 1/ell^2<3.75 K = exp(cos(D)/ell^2); b0 = besseli(0,1/ell^2); K = (K-b0)/(exp(1/ell^2)-b0); else K = exp((cos(D)-1)/ell^2); b0 = embi0(1/ell^2); K = (K-b0)/(1-b0); end function y = embi0(x) % = exp(-x)*besseli(0,x) => 9.8.2 Abramowitz & Stegun y = 3.75/x; y = 0.39894228 + 0.01328592*y + 0.00225319*y^2 - 0.00157565*y^3 ... + 0.00916281*y^4 - 0.02057706*y^5 + 0.02635537*y^6 - 0.01647633*y^7 ... + 0.00392377*y^8; y = y/sqrt(x); function y = embi1(x) % = exp(-x)*besseli(1,x) => 9.8.4 Abramowitz & Stegun y = 3.75/x; y = 0.39894228 - 0.03988024*y - 0.00362018*y^2 + 0.00163801*y^3 ... - 0.01031555*y^4 + 0.02282967*y^5 - 0.02895312*y^6 + 0.01787654*y^7 ... - 0.00420059*y^8; y = y/sqrt(x);gpml-matlab-v3.6-2015-07-07/cov/covMaterniso.m000644 000765 000024 00000003765 12441263146 020522 0ustar00carlstaff000000 000000 function K = covMaterniso(d, hyp, x, z, i) % Matern covariance function with nu = d/2 and isotropic distance measure. For % d=1 the function is also known as the exponential covariance function or the % Ornstein-Uhlenbeck covariance in 1d. The covariance function is: % % k(x^p,x^q) = sf^2 * f( sqrt(d)*r ) * exp(-sqrt(d)*r) % % with f(t)=1 for d=1, f(t)=1+t for d=3 and f(t)=1+t+t²/3 for d=5. % Here r is the distance sqrt((x^p-x^q)'*inv(P)*(x^p-x^q)), P is ell times % the unit matrix and sf2 is the signal variance. The hyperparameters are: % % hyp = [ log(ell) % log(sf) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. if nargin<3, K = '2'; return; end % report number of parameters if nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode ell = exp(hyp(1)); sf2 = exp(2*hyp(2)); if all(d~=[1,3,5]), error('only 1, 3 and 5 allowed for d'), end % degree switch d case 1, f = @(t) 1; df = @(t) 1; % df(t) = f(t)-f'(t) case 3, f = @(t) 1 + t; df = @(t) t; case 5, f = @(t) 1 + t.*(1+t/3); df = @(t) t.*(1+t)/3; end m = @(t,f) f(t).*exp(-t); dm = @(t,f) df(t).*exp(-t).*t; % precompute distances if dg % vector kxx K = zeros(size(x,1),1); else if xeqz % symmetric matrix Kxx K = sq_dist(sqrt(d)/ell*x'); else % cross covariances Kxz K = sq_dist(sqrt(d)/ell*x',sqrt(d)/ell*z'); end end if nargin<5 % covariances K = sf2*m(sqrt(K),f); else % derivatives if i==1 K = sf2*dm(sqrt(K),f); elseif i==2 K = 2*sf2*m(sqrt(K),f); else error('Unknown hyperparameter') end endgpml-matlab-v3.6-2015-07-07/doc/demoGrid.m000644 000765 000024 00000002424 12441263151 017544 0ustar00carlstaff000000 000000 disp('See http://www.gaussianprocess.org/gpml/code/matlab/doc/ for details.') disp('Hit any key to continue...'); pause clear all, close all write_fig = 0; xg = {linspace(-2,2,120)',linspace(-3,3,120)'}; % construct a grid [xe,ng] = covGrid('expand',xg); y = sin(xe(:,2)) + xe(:,1); x = (1:prod(ng))'; % generate training data xgs = {linspace(-4,4,100)',linspace(-6,6,110)'}; y = y + 0.01*gpml_randn(1,size(y)); % add some noise [xs,ns] = covGrid('expand',xgs); cov = {{@covSEiso},{@covSEiso}}; covg = {@covGrid,cov,xg}; % set up a GP hyp.cov = zeros(4,1); hyp.mean = []; hyp.lik = log(0.1); hyp = minimize(hyp,@gp,-50,@infGrid,[],covg,[],x,y); % optimise hyperparameters opt.cg_maxit = 200; opt.cg_tol = 5e-3; % perform inference and prediction post = infGrid(hyp,{@meanZero},covg,'likGauss',x,y,opt); post.L = @(a) a; ym = gp(hyp, @infGrid,[],covg,[],x,post,xs); cl = [min(ym(:)), max(ym(:))]; subplot(121), imagesc(xg{1}, xg{2}, reshape(y, ng'),cl), title('data y') grid on, set(gca,'xtick',-2:2), set(gca,'ytick',-3:3) subplot(122), imagesc(xgs{1},xgs{2},reshape(ym,ns'),cl), title('prediction ym') grid on, set(gca,'xtick',-4:4), set(gca,'ytick',-6:6) rectangle('Position',[-2,-3,4,6]) if write_fig, print -depsc f9.eps; endgpml-matlab-v3.6-2015-07-07/doc/f8.gif000644 000765 000024 00000034552 12076017514 016653 0ustar00carlstaff000000 000000 GIF89aA/?O_/o?O_o/?O_oo_O?/!Created with GIMP,A H*\ȰÇ#JHEI\VƎ CQɓHjDɲ%&+]ʜfL8s*YRO2 [5 FgJf5׏a;S%ŸeF(ʷkXy( 蛃X(p艇"bpad "8h$:fbIF(@FP9G[W;WoWUQpz /,m%.Ye7E/ϼי87ʼ׋7]wX̠_O\޲V$E(F/)8 g!oH%{Ԙ`,ЁHEnCD߀QP` :.D4HC-‰"!:"yA)1`xD#!cg&fqP a;bQ 7)FAE/;5ưBq,d%!1I^ҍP9IKO/޻_1)b%V&m)4)a^2Ō"1'JA[ēIrdfxN5:c)LMlSKۣ/*~SCKu92xS Y8cx=<?r~$AaPF"Ԙ$C(IBE @JGwjOQ"wzS K9Nt3"l&D< [8uDJֲZRaJP@­)TiѩFª ׾|j@aDC "cGe+!4I7)+ hW(X"v_ _,U1VI4Me0JYIVL_fuF%yzpJӶ=;[ZJ4lVK nZA﷉pNj3ѸD%Z5Z]A\wͭ׶+ ih-y!v^/3,b|)@bWSmuƭaݔ`dE(NW#=3ar!TAh7{E>ib>.XJ1'0Iwq!| k #SKNOʘʛ&L^v` B'\$lE[9O?t-eI G>82FAYG?:Œ4})MK?!+"tWv^׼^FQjT:7l\I@[Z-;<8v'LmeNӰij٪iM֠;km޾v.Vqۦc=t[ӕk_;QY9o\4جxWc3 #+d`<&.7Ϸ\oIݼK؆_B9F<@H/:m~B2sjC 1@Q`peN/q OX`V uEԽlk=Y)~vZp;wgc{;YTc?{8|)v.|!$a}~MXzYwʦ@)kE_r&=#[XϾ{yԿ~p#KQyM4&qv\]wY6~E Gys$Wgzٗ2 bD5!#h%apw"x w !@XeP GA#&4סlTI.80Ȅg'p^wU:eЃsvA8`DZ&VńX&%"gNn8l[~ס7!eqA$gQPjX2*R tXshMx\gUnA4H|tUhc_'88/1H)Xhb`R(}Ds|7{֖8 AP4,&B]ƷzzE苘Sx֘g?b菼Z+1x$Iɋ8P1/4ny @\u^؅'u<ba0R ɋ\PXR,6=QHsk!Y&֎)U&ov}YA`b9dA15PlYPK9uyw~/k)y^Ypؤ _\I{ey0d=H\K؄n(YDu)bdF5^(*ɍD9ٚcɘ03/E(O樊T7wVYPyWx@};!&隮 ؆H 'SxAz9j9swyv}Ie~&pi pr'&5V737p婎 8yYQb U"꟔.y=p`x^2[iŢZ҉":I0Dvzi_7ؐ~C`jF=&ڋ&e L N wGE"E5 A@ea:cZ'.hI٠Siv金7yn5H< YککB~.J 'a+LIyR 5juG $txJ79 ZZ5 ǶI/o]Z[|ꫥz@U)ZiZڇneE$0*JXI : 8P[&*r^H{ ե: v S~zsZu|e &IY4[6{8[j;1Mep)z)I5_`09;TJ2>vXV_ zI)cu8KzO2T7k%-^iGzQWPEHsvEVsPTp۸* /O|0'BC|U=[=gҴjů<붎S+`rd6UT.u5gJ՗7!0,P{țƻg0{ [:(}%B1wv[5[HPq J{˼(ZzQ;_rTEY{}Yq)s[ȫ%*?h+V]s«qָP:7>g*~Fa"о[ 1)"UOLv*Eu_;;[q̪ \ѫ-0P5:E~5~Uuܖ=\^`B0Dfr6AÈ`s)s8Hr Q'a~<;,(@YY47Xi,A>RqUElGxG8™7ʌ6) ۶ʼL\L gY9V  }Yܼ<V!u#'O[2)[Hhm rC]MDӦ#*.Bs:,=]?-fdMRR:c7".,g:6D0!hZ] `zlUf$ݏ4 IHK3,;85YdY ԯ \Y1ɲ=u`:X Вݣ XUD8RAقp MBqt^b3]E;yjg__IE{H;88vQmJ-o]ߍԷTg$w7/C<5esR0n]SU#}CZ٩]+2[ST&dnBq\70$Tt^ f@?R皉(㮴0>څOB?FB/}D}J} @z|zQz\ Znި$ge_Q/a6А[@L=~Mud~x6Vn&'JIsr @.FnC%`e5遌'|ND!JP)'P=됌[P|enTDF{tc&^Hsz>N?Mʉۃ6סMmE+C>7!ol>>xqȝpݢn۸\\Y[@0> <#(+$vx@B?D~UΘ(Fl'';IG{imwiޕ18 Ei0#"-%Me-5atcq.^ݣ?..U Or8\nQ5_a p9/߼g?@Iΰ~EWsb UjB6r\;ah죌L?Ugq?~-σ3}Plu׮~ c'>_ۀ@(hH`8Ȩ)9IYiyYYyZ"êC)"++Hi(2jH*l,`L-h .>N޽Xx~ؾ/?H?=m/<,H+zA%… 4W Ä,2rG<mV.eJܱt$ȥL4k^bI6~7)4 P"[45)U'Vhg$%qqHy:ᴄRZsfBȥ͹t RV֝5QZBmBBT {oI|3i}KlK>軬[o[=rM~ JѬJkʵL2V^,QAjn^mE ]{>;u5zPy-)|j"nEa+? XIfIuބ]^1HyR{!&nDCqaV\&1W }! CXH8YMFBQ{HdN~o, `W${GY<]:Ј c 9a$EbW`UJ5$ 5C3 !s_UTDd,(&YdZC^if&(4iܶaѤ@)ڢgj)N8j٢ziri0[ =Ŋڏe!jLFoTZ2D`/]HVۚ)K/:/%ΘH*rw*oF:PBzkj느r2mRZoKr=J+( r7\x 0ȷ i@ }` jϣ.~l\rTKz|F6!5e+\5Yy%4OCzw~}uu<CocTWҙ3IO]uor8P5ߦNYuˌ綏${ ;eLJzn B!Go>AƓ5ZN}߬"3LІ43P8=n+0_ G7wCz=9(ǣƴGovSQ[q[g a.R9o{oL*"F7Q 8K*J/gŇBt*JhSYBHJGϱ(JqTYB`F @q]T ;%;PBT( j'I@b1f vEaB. *-А2 nK;M܁уtl%IH- Jdcx6S;R|Q89C(c#)1!Lt7a˓,%SRh B6ժgI p2$?lpFX֙LQ^Jo\PDdL;yKba`2bja/@""#ٟh&+7]MG&D-QYAъ%8I90PnxO`#2)% *=[A]͔MАfA;" ET9uFMR"luSgV~SS.j3!LzҌ3OK^ fð)eL،byt}]WE_JnSO*MJWJ*`%T$DS}D,)PE~}(XGx͛ E&ቛۅRGbA&69%M1.c2: ђԄGu F#Y>kwс8 @:f,99&;9\qx/ f$) UM0 (KyP6w1 L%88a ٰa9kW{y!{BZc9vЭh\1&wubvoB@ ʯr~/苻"YO)Q1$XI+)Z޹J8iiCM?qn WC/B%SEs=v㞲37Ԑ}`|.̀nY`$,5Ю6* o|ٵ2`Alm̥l&Aa&0_1hJmj_)۵EI)*%x| 5c lҔASDB\#^Ay 5ɗ䒄4o v8mk6 ecd|oc2Cy^*rlMۥ= rvh룴;۔y7E1ocbo~6H|,, uq31e3//~'tD"F6]jwûe7EGg:;߷ϲoOZu}'})ͅt5-~앚}1k'}~QR P_çW.br Lw7x '#Y4NKX HTF!y -O@$ FCzdTu7 UpH#W&(bFH2*,hwR8}*?c~vwK,w6azuzϒkXK({/Ph W1x'w&c%|;1{0Q`A|I膲kxQ5n$F%iRS814aHZ̧?h20(H(xV60!08%0xisA;NQ9$r7Ê|/ȍ݈h]x t8>悺*8WyC"8ONC(38uэIߨp~5s苍Qh gA{X I~X"A .6-W4aJ]A)9ybI@HRtxi2W9l(9h"Ψ QUzv$ɒW  Y'C;C !B}Ϩ9!>xh8"o q)s OYh Uv6*8yyI ['(tm%u6Wb(dtiiRTقxce;:-!ls(Mq_8&95%2ry Tɗ2yyyv#6q22bgyJ)s i™\[3'𘚈xh)xP|HޙIv Imb ˀI|A6b3%z"Y2بy1@%j'%*!`/ ."`mwwgơ^9#I+J}-ʋZ*X5jbRjC"1e.<"`<8z)LTJ'ʤ&.ڌ06 4cCoY(`yFblֱmo:P Fi;cy$zU 9#bCh"j0ꪯ ʨt=pJs*WFF<>_0|8$z?sdE"),&myJ:S > M 81+4>Uv%kSvIPת:c:6o7*"@$R~J1j^J4MF{ !d!TM trjtxSl'jeQFKzf`RdYПy!+5%[)Ȳ _pLaS %4<+R]H롎PP{HukvTnQ':Q*V49VB1fKնXEw Ke1Rۭ:QY6_yrx ]x껣CJy[ ^{˷}p4vFc oK?D \uYݓFa˾kzg ,:FzcqTBԻH6q )ڻ[3(P[MY˶ bKi,{ f[Ppyєm%羧e)TʋD-`GIlijԴ+[v"s} Y80EY G %=Ӷ(*sLG,$@Q\0VW YLԥѥKc#~0ՑYy'Tۏ<Ձ}yr'L5Gܷ Ԩ9Ke s,-KLkا~+lł,ƶyQ]vm=0a7 - ;[Hfԩ]jvKؓڒ]~#]Ew,M=mvEhڧ0njQ ݧUMbܣh-}ޢm-f=]IM- b]MěUٞ})${Lԉ ؍Mݕt^RBw߼ݙ>4]== lxvW`T߱ܽ$='[][.̼2[܇1&57 H_y想z+->g(7<*Xny.]2Eh1~[}nVPl)MPoD #`>yTZl ,\棘x %.~r~^30R6@t;,^ G쾵3m>;R`B=*vﭻjRkEo2l.[V.wA:/1#3촮UR=4>(H#_a/_d.Ν Ό~<K* I\c{_ +/ nT_D꙰}_g }3nFfD[o"?B@:}ތdV/OA:/?ϝN-X_ٯݯ_4 ȿ!? ]hj$HO#~A/8:A"A Aʢ !㽉A8ƋAX jDZgÔ[T_A.wG5)ݨ}!tnP@B#$澚Kd; !d$2eJhFsLdF}(ԫX-ӇRE-UBIRF*ڛR@WbRgBTӫV}LÄg]LҤ/ s,YSy̶ZFr*楮~6ٌsOʵs^_^u<}FH:HB}zS}s0n9fC_NphF"%)JB9}DW`owZ ڧ`(I`8Oi8al݈߅gn eeL FTdcup)O2ig9Ne!,ZV3m e\1YgqUǨ4]nGi饋ީVejȞYg/fi7V w(FbYff]КlRsqzHt*N"#V'=" VTHFV}FB^H*l:!r[;S#͞HlXMk(Mcj%0w շ" 7 \]G,/K(,*Κ.Vg$!5,SZ%N-,2,Ll4Rتf4 8/xg7Gy $GO,Z!\VӍvWu`֮ŭ]J(L! +GqLpV gHp̠b>HpФgu!GfFԣY t 0L_zP *!0ѓ**MF5ʻ5)pЬ`qOCU#zF٩G*L) 8*ՑP5+Q"R pզnj91IU;+;5ն"Ս\*쮀^W8౐d' ٷ4}AX]!4X6u$ʚ() N&WC,h5i?:,JGu f8^ O]iJַx*\rZC%5'/ԲgFs[ޖb]RuٖwM Gj(ͯ~߆N Y+Ki/+ߖH8ɻdy 7WG$LbR1MT !n07۸eFH\ٶ\9d66rckӔU!yV^$R,.*/r̶#y,߷NY%PsYr΀žL1G|ν9o3dZɨM'ͻ6fz=hVS#gMZZޜ)dîn}mw$[ظFqo|գ:~cAbڳsUc變 Ӕl?S> n{^lКaZFҿev?_`7'98N~;y@gL ri(OyO)K[㑨n蒻mOC|s̫T xpL6G|Nu،h2(XxؘڸHdk;Gt#Th'GզHu7t߈^(> (莕ӊW,%`Qh6x[Qxhfl&gȏhׄdbɐɐV 9HjxX .i!Y9wh'}=9?:)E5הH<0HW9HY=TVdX9敐 *n8:`[_f]GX\9Xky؍2ɒhli3[uVXșG{kIٗɘpAZIyٛ;gpml-matlab-v3.6-2015-07-07/doc/index.html000644 000765 000024 00000165110 12546703465 017650 0ustar00carlstaff000000 000000 Documentation for GPML Matlab Code

Documentation for GPML Matlab Code version 3.6

1) What?

The code provided here originally demonstrated the main algorithms from Rasmussen and Williams: Gaussian Processes for Machine Learning. It has since grown to allow more likelihood functions, further inference methods and a flexible framework for specifying GPs. Other GP packages can be found here.

The code is written by Carl Edward Rasmussen and Hannes Nickisch; it runs on both Octave 3.2.x and Matlab® 7.x and later. The code is based on previous versions written by Carl Edward Rasmussen and Chris Williams.

2) Download, Install and Documentation

All the code including demonstrations and html documentation can be downloaded in a tar or zip archive file.

Minor changes and incremental bugfixes to the current version are documented in the changelog, changes from previous versions are documented in README.

Please read the copyright notice.

After unpacking the tar or zip file you will find 7 subdirectories: cov, doc, inf, lik, mean, prior and util. It is not necessary to install anything to get started, just run the startup script to set your path.

Details about the directory contents and on how to compile mex files can be found in the README. The getting started guide is the remainder of the html file you are currently reading (also available at http://gaussianprocess.org/gpml/code/matlab/doc). A Developer's Guide containing technical documentation is found in manual.pdf, but for the casual user, the guide is below.

3) Theory

Gaussian Processes (GPs) can conveniently be used for Bayesian supervised learning, such as regression and classification. In its simplest form, GP inference can be implemented in a few lines of code. However, in practice, things typically get a little more complicated: you might want to use complicated covariance functions and mean functions, learn good values for hyperparameters, use non-Gaussian likelihood functions (rendering exact inference intractable), use approximate inference algorithms, or combinations of many or all of the above. This is what the GPML software package does.

Before going straight to the examples, just a brief note about the organization of the package. There are four types of objects which you need to know about:

Gaussian Process
A Gaussian Process is fully specified by a mean function and a covariance function. These functions are specified separately, and consist of a specification of a functional form as well as a set of parameters called hyperparameters, see below.
Mean functions
Several mean functions are available, all start with the four letters mean and reside in the mean directory. An overview is provided by the meanFunctions help function (type help meanFunctions to get help), and an example is the meanLinear function.
Covariance functions
There are many covariance functions available, all start with the three letters cov and reside in the cov directory. An overview is provided by the covFunctions help function (type help covFunctions to get help), and an example is the covSEard "Squared Exponential with Automatic Relevance Determination" covariance function.
For both mean functions and covariance functions, two types exist: simple and composite. Whereas simple types are specified by the function name (or function pointer), composite functions join together several components using cell arrays. Composite functions can be composed of other composite functions, allowing for very flexible and interesting structures.
Examples are given below and in the usageMean and usageCov functions.
Hyperparameters
GPs are typically specified using mean and covariance functions which have free parameters called hyperparameters. Also likelihood functions may have such parameters. These are encoded in a struct with the fields mean, cov and lik (some of which may be empty). When specifying hyperparameters, it is important that the number of elements in each of these struct fields, precisely match the number of parameters expected by the mean function, the covariance function and the likelihood functions respectively (otherwise an error will result). Hyperparameters whose natural domain is positive are represented by their logarithms. When working with FITC approximate inference, the inducing inputs xu can also be treated as hyperparameters for some common stationary covariances.
Hyperparameter Prior Distributions
When optimising the marginal likelihood w.r.t. hyperparameters, it is sometimes useful to softly constrain the hyperparameters by means of prior knowledge. A hyperprior is a probability distribution over individual or a group of hyperparameters. All hyperpriors begin with letters prior and reside in the prior directory. An overview is provided by the priorDistributions help function (type help priorDistributions to get help). Some examples are priorGauss and priorGaussMulti the uni- and multivariate Gaussian prior, priorSmoothBox1 and priorSmoothBox2 the univariate (smoothed) box prior with linear or quadratic decay, priorMix the mixture prior, and priorTransform allowing to place a prior on a transformation g(x) rather than on x directly. Further, we have the two equivalents priorDelta and priorClamp to exclude a parameter from optimisation.
Examples are given below and in the usagePrior function.
Likelihood Functions
The likelihood function specifies the probability of the observations given the latent function, i.e. the GP (and the hyperparameters). All likelihood functions begin with the three letters lik and reside in the lik directory. An overview is provided by the likFunctions help function (type help likFunctions to get help). Some examples are likGauss the Gaussian likelihood for regression, likPoisson the Poisson likelihood for count data, likGamma the Gamma likelihood for positive data, likBeta the Beta likelihood for interval data, and likLogistic the logistic function used in classification (a.k.a. logistic regression). Output dependent noise can be modeled using warped GPs implemented by likGaussWarp. As for the mean and covariance functions there is also a composite likelihood likMix that can be used to generate mixtures of other likelihood functions.
Examples are given below and in the usageLik function.
Inference Methods
The inference methods specify how to compute with the model, i.e. how to infer the (approximate) posterior process, how to find hyperparameters, evaluate the log marginal likelihood and how to make predictions. Inference methods all begin with the three letters inf and reside in the inf directory. An overview is provided by the infMethods help file (type help infMethods to get help). Some examples are infExact for exact inference (regression with Gaussian likelihood), infFITC and infGrid for large scale Gaussian regression, infEP for the Expectation Propagation algorithm, infVB and infKL for two variational approximations based on a lower bound to the marginal likelihood, or infFITC_Laplace for large scale Laplace's approximation.
Further usage examples are provided for both regression and classification. However, not all combinations of likelihood function and inference method are possible (e.g. you cannot do exact inference with a Laplace likelihood). An exhaustive compatibility matrix between likelihoods (rows) and inference methods (columns) is given in the table below:
Plain Regression Likelihoods
Inference GPML name Exact
infExact		 EP
infEP		 Laplace
infLaplace		 Variational Bayes
infVB		 Kullback Leibler
infKL		 Sampling
infMCMC		 LOO
infLOO		 type, output domain
 		 alternative names
 
Likelihood FITC
infFITC		 FITC-EP
infFITC_EP		 FITC-Laplace
infFITC_Laplace
Grid
infGrid		   Grid-Laplace
infGrid_Laplace
Gaussian likGauss $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ regression, IR  
Warped Gaussian likGaussWarp   $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ regression, IR  
Gumbel likGumbel     $\checkmark$   $\checkmark$ $\checkmark$ $\checkmark$ regression, IR extremal value regression
Sech squared likSech2   $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ regression, IR logistic distribution
Laplacian likLaplace   $\checkmark$   $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ regression, IR double exponential
Student's t likT     $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ regression, IR  
Composite Likelihoods
Inference GPML name Exact
infExact		 EP
infEP		 Laplace
infLaplace		 Variational Bayes
infVB		 Kullback Leibler
infKL		 Sampling
infMCMC		 LOO
infLOO		 type, output domain
 		 alternative names
 
Likelihood FITC
infFITC		 FITC-EP
infFITC_EP		 FITC-Laplace
infFITC_Laplace
Grid
infGrid		   Grid-Laplace
infGrid_Laplace
Mixture likMix   $\checkmark$ $\checkmark$   $\checkmark$ $\checkmark$ $\checkmark$ classification, ±1 and regression, IR mixing meta likelihood
Classification Likelihoods
Inference GPML name Exact
infExact		 EP
infEP		 Laplace
infLaplace		 Variational Bayes
infVB		 Kullback Leibler
infKL		 Sampling
infMCMC		 LOO
infLOO		 type, output domain
 		 alternative names
 
Likelihood FITC
infFITC		 FITC-EP
infFITC_EP		 FITC-Laplace
infFITC_Laplace
Grid
infGrid		   Grid-Laplace
infGrid_Laplace
Uniform likUni   $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ classification, ±1 label noise
Error function likErf   $\checkmark$ $\checkmark$   $\checkmark$ $\checkmark$ $\checkmark$ classification, ±1 probit regression
Logistic function likLogistic   $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ classification, ±1 logistic regression
logit regression
Generalized Linear Model Likelihoods
Inference GPML name Exact
infExact		 EP
infEP		 Laplace
infLaplace		 Variational Bayes
infVB		 Kullback Leibler
infKL		 Sampling
infMCMC		 LOO
infLOO		 type, output domain
 		 alternative names
 
Likelihood FITC
infFITC		 FITC-EP
infFITC_EP		 FITC-Laplace
infFITC_Laplace
Grid
infGrid		   Grid-Laplace
infGrid_Laplace
Weibull likWeibull     $\checkmark$     $\checkmark$ $\checkmark$ positive data, IR+\{0} nonnegative regression
Gamma likGamma     $\checkmark$     $\checkmark$ $\checkmark$ positive data, IR+\{0} nonnegative regression
Exponential likExp     $\checkmark$     $\checkmark$ $\checkmark$ positive data, IR+\{0} nonnegative regression
Inverse Gaussian likInvGauss     $\checkmark$     $\checkmark$ $\checkmark$ positive data, IR+\{0} nonnegative regression
Poisson likPoisson   ($\checkmark$)* $\checkmark$   $\checkmark$ $\checkmark$ $\checkmark$ count data, IN Poisson regression
Beta likBeta     $\checkmark$     $\checkmark$ $\checkmark$ interval data, [0,1] range regression

* EP might not converge in some cases since quadrature is used.

All of the objects described above are written in a modular way, so you can add functionality if you feel constrained despite the considerable flexibility provided. Details about how to do this are provided in the developer documentation.

Inference by MCMC sampling is the only inference method that cannot be used as a black box. Also gradient-based marginal likelihood optimisation is not possible with MCMC. Please see usageSampling for a toy example illustrating the usage of the implemented samplers.

4) Practice

Using the GPML package is simple. There is only one single function to call: gp, it does posterior inference, learns hyperparameters, computes the marginal likelihood and makes predictions. Generally, the gp function takes the following arguments: a hyperparameter struct, an inference method, a mean function, a covariance function, a likelihood function, training inputs, training targets, and possibly test cases. The exact computations done by the function is controlled by the number of input and output arguments in the call. Here is part of the help message for the gp function (follow the link to see the whole thing):

  function [varargout] = gp(hyp, inf, mean, cov, lik, x, y, xs, ys)

  [ ... snip ...]

  Two modes are possible: training or prediction: if no test cases are
  supplied, then the negative log marginal likelihood and its partial
  derivatives wrt the hyperparameters is computed; this mode is used to fit the
  hyperparameters. If test cases are given, then the test set predictive
  probabilities are returned. Usage:

    training: [nlZ dnlZ          ] = gp(hyp, inf, mean, cov, lik, x, y);
  prediction: [ymu ys2 fmu fs2   ] = gp(hyp, inf, mean, cov, lik, x, y, xs);
          or: [ymu ys2 fmu fs2 lp] = gp(hyp, inf, mean, cov, lik, x, y, xs, ys);

  [ .. snip ...]

Here x and y are training inputs and outputs, and xs and ys are test set inputs and outputs, nlZ is the negative log marginal likelihood and dnlZ its partial derivatives wrt the hyperparameters (which are used for training the hyperparameters). The prediction outputs are ymu and ys2 for test output mean and covariance, and fmu and fs2 are the equivalent quenteties for the corresponding latent variables. Finally, lp are the test output log probabilities.

Instead of exhaustively explaining all the possibilities, we will give two illustrative examples to give you the idea; one for regression and one for classification. You can either follow the example here on this page, or using the two scripts demoRegression and demoClassification (using the scripts, you still need to follow the explanation on this page).

3a) Regression

You can either follow the example here on this page, or use the script demoRegression.

This is a simple example, where we first generate n=20 data points from a GP, where the inputs are scalar (so that it is easy to plot what is going on). We then use various other GPs to make inferences about the underlying function.

First, generate some data from a Gaussian process (it is not essential to understand the details of this):

  clear all, close all
 
  meanfunc = {@meanSum, {@meanLinear, @meanConst}}; hyp.mean = [0.5; 1];
  covfunc = {@covMaterniso, 3}; ell = 1/4; sf = 1; hyp.cov = log([ell; sf]);
  likfunc = @likGauss; sn = 0.1; hyp.lik = log(sn);
 
  n = 20;
  x = gpml_randn(0.3, n, 1);
  K = feval(covfunc{:}, hyp.cov, x);
  mu = feval(meanfunc{:}, hyp.mean, x);
  y = chol(K)'*gpml_randn(0.15, n, 1) + mu + exp(hyp.lik)*gpml_randn(0.2, n, 1);

  plot(x, y, '+')

Above, we first specify the mean function meanfunc, covariance function covfunc of a GP and a likelihood function, likfunc. The corresponding hyperparameters are specified in the hyp structure:

The mean function is composite, adding (using meanSum function) a linear (meanLinear) and a constant (meanConst) to get an affine function. Note, how the different components are composed using cell arrays. The hyperparameters for the mean are given in hyp.mean and consists of a single (because the input will one dimensional, i.e. D=1) slope (set to 0.5) and an off-set (set to 1). The number and the order of these hyperparameters conform to the mean function specification. You can find out how many hyperparameters a mean (or covariance or likelihood function) expects by calling it without arguments, such as feval(meanfunc{:}). For more information on mean functions see meanFunctions and the directory mean/.

The covariance function is of the Matérn form with isotropic distance measure covMaterniso. This covariance function is also composite, as it takes a constant (related to the smoothness of the GP), which in this case is set to 3. The covariance function takes two hyperparameters, a characteristic length-scale ell and the standard deviation of the signal sf. Note, that these positive parameters are represented in hyp.cov using their logarithms. For more information on covariance functions see covFunctions and cov/.

Finally, the likelihood function is specified to be Gaussian. The standard deviation of the noise sn is set to 0.1. Again, the representation in the hyp.lik is given in terms of its logarithm. For more information about likelihood functions, see likFunctions and lik/.

Then, we generate a dataset with n=20 examples. The inputs x are drawn from a unit Gaussian (using the gpml_randn utility, which generates unit Gaussian pseudo random numbers with a specified seed). We then evaluate the covariance matrix K and the mean vector m by calling the corresponding functions with the hyperparameters and the input locations x. Finally, the targets y are computed by drawing randomly from a Gaussian with the desired covariance and mean and adding Gaussian noise with standard deviation exp(hyp.lik). The above code is a bit special because we explicitly call the mean and covariance functions (in order to generate samples from a GP); ordinarily, we would only directly call the gp function.

f1.gif

Let's ask the model to compute the (joint) negative log probability (density) nlml (also called marginal likelihood or evidence) and to generalize from the training data to other (test) inputs z:

  nlml = gp(hyp, @infExact, meanfunc, covfunc, likfunc, x, y)

  z = linspace(-1.9, 1.9, 101)';
  [m s2] = gp(hyp, @infExact, meanfunc, covfunc, likfunc, x, y, z);

  f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)]; 
  fill([z; flipdim(z,1)], f, [7 7 7]/8)
  hold on; plot(z, m); plot(x, y, '+')

The gp function is called with a struct of hyperparameters hyp, and inference method, in this case @infExact for exact inference and the mean, covariance and likelihood functions, as well as the inputs and outputs of the training data. With no test inputs, gp returns the negative log probability of the training data, in this example nlml=11.97.

To compute the predictions at test locations we add the test inputs z as a final argument, and gp returns the mean m variance s2 at the test location. The program is using algorithm 2.1 from the GPML book. Plotting the mean function plus/minus two standard deviations (corresponding to a 95% confidence interval):

f2.gif

Typically, we would not a priori know the values of the hyperparameters hyp, let alone the form of the mean, covariance or likelihood functions. So, let's pretend we didn't know any of this. We assume a particular structure and learn suitable hyperparameters:

  covfunc = @covSEiso; hyp2.cov = [0; 0]; hyp2.lik = log(0.1);

  hyp2 = minimize(hyp2, @gp, -100, @infExact, [], covfunc, likfunc, x, y);
  exp(hyp2.lik)
  nlml2 = gp(hyp2, @infExact, [], covfunc, likfunc, x, y)

  [m s2] = gp(hyp2, @infExact, [], covfunc, likfunc, x, y, z);
  f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)];
  fill([z; flipdim(z,1)], f, [7 7 7]/8)
  hold on; plot(z, m); plot(x, y, '+')

First, we guess that a squared exponential covariance function covSEiso may be suitable. This covariance function takes two hyperparameters: a characteristic length-scale and a signal standard deviation (magnitude). These hyperparameters are non-negative and represented by their logarithms; thus, initializing hyp2.cov to zero, correspond to unit characteristic length-scale and unit signal standard deviation. The likelihood hyperparameter in hyp2.lik is also initialized. We assume that the mean function is zero, so we simply ignore it (and when in the following we call gp, we give an empty argument for the mean function).

In the following line, we optimize over the hyperparameters, by minimizing the negative log marginal likelihood w.r.t. the hyperparameters. The third parameter in the call to minimize limits the number of function evaluations to a maximum of 100. The inferred noise standard deviation is exp(hyp2.lik)=0.15, somewhat larger than the one used to generate the data (0.1). The final negative log marginal likelihood is nlml2=14.13, showing that the joint probability (density) of the training data is about exp(14.13-11.97)=8.7 times smaller than for the setup actually generating the data. Finally, we plot the predictive distribution.

f3.gif

This plot shows clearly, that the model is indeed quite different from the generating process. This is due to the different specifications of both the mean and covariance functions. Below we'll try to do a better job, by allowing more flexibility in the specification.

Note that the confidence interval in this plot is the confidence for the distribution of the (noisy) data. If instead you want the confidence region for the underlying function, you should use the 3rd and 4th output arguments from gp as these refer to the latent process, rather than the data points.

  hyp.cov = [0; 0]; hyp.mean = [0; 0]; hyp.lik = log(0.1);
  hyp = minimize(hyp, @gp, -100, @infExact, meanfunc, covfunc, likfunc, x, y);
  [m s2] = gp(hyp, @infExact, meanfunc, covfunc, likfunc, x, y, z);
 
  f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)];
  fill([z; flipdim(z,1)], f, [7 7 7]/8)
  hold on; plot(z, m); plot(x, y, '+');

Here, we have changed the specification by adding the affine mean function. All the hyperparameters are learnt by optimizing the marginal likelihood.

f4.gif

This shows that a much better fit is achieved when allowing a mean function (although the covariance function is still different from that of the generating process).

Guiding marginal likelihood optimisation with a hyperprior

It can be usefull to put a prior distribution on (a part of) the hyperparameters. Sometimes, one may want to exclude some hyperparameters from the optimisation i.e. fix their values beforehand and treat them as constants.

In these cases, a hyperprior comes to bear. A hyperprior is specified by augmenting the inf parameter of gp.m In the regression before, we had inf = @infExact;. To put a Gaussian prior on the first mean hyperparameter hyp.mean(1) and a Laplacian prior on the second mean hyperparameter hyp.mean(2) and wished to fix the noise variance hyperparameter hyp.lik, we simple need to set up the corresponding prior structure as detailed below. 

  mu = 1.0; s2 = 0.01^2;
  prior.mean = {{@priorGauss,mu,s2}; {'priorLaplace',mu,s2}};
  prior.lik = {{@priorDelta}};
  inf = {@infPrior,@infExact,prior};
  hyp = minimize(hyp, @gp, -100, inf, meanfunc, covfunc, likfunc, x, y);

Further examples are provided in usagePrior.

Large scale regression using the FITC approximation

In case the number of training inputs x exceeds a few thousands, exact inference using infExact.m takes too long. We offer the FITC approximation based on a low-rank plus diagonal approximation to the exact covariance to deal with these cases. The general idea is to use inducing points u and to base the computations on cross-covariances between training, test and inducing points only.

Using the FITC approximation is very simple, we just have to wrap the covariance function covfunc into covFITC.m and call gp.m with the inference method infFITC.m as demonstrated by the following lines of code. 

  nu = fix(n/2); u = linspace(-1.3,1.3,nu)';
  covfuncF = {@covFITC, {covfunc}, u};
  [mF s2F] = gp(hyp, @infFITC, meanfunc, covfuncF, likfunc, x, y, z);

We define equispaced inducing points u that are shown in the figure as black circles. Note that the predictive variance is overestimated outside the support of the inducing inputs. In a multivariate example where densely sampled inducing inputs are infeasible, one can simply use a random subset of the training points. 

  nu = fix(n/2); iu = randperm(n); iu = iu(1:nu); u = x(iu,:);
f5.gif

Large scale regression exploiting the grid structure

A covariance function factorising over coordinates axes leads to a covariance matrix with Kronecker structure. This can be exploited to scale GPs beyond the O(n³) limit. If there are observations for all points of the grid, all computations are exact. The example below contains the most relevant code from the script demoGrid, where we extrapolate a pixel image beyond its boundaries.

For a comprehensive set of examples and more resources, see a website by Andrew Wilson.

We start off by setting up the training data and the GP on a [-2,2]x[-3,3] lattice with 14400 pixels -- a size where a usual dens GP would be computationally infeasible. Then we wrap the covariance functions for the axes into covGrid and specify the initial values for the hyperparameters. Note that covGrid can only be used in conjunction with infGrid. 

  xg = {linspace(-2,2,120)',linspace(-3,3,120)'}; xe = covGrid('expand',xg);
  x = (1:size(xe,1))'; y = sin(xe(:,2)) + xe(:,1) + 0.01*gpml_randn(1,size(x));
  cov = {{@covSEiso},{@covSEiso}}; covg = {@covGrid,cov,xg};
  hyp.cov = zeros(4,1); hyp.mean = []; hyp.lik = log(0.1);

As a next step, we perform hyperparameter optimisation as with a usual GP model. The inference method inGrid only supports likGauss, hence we can leave the field empty. 

  hyp = minimize(hyp,@gp,-50,@infGrid,@meanZero,covg,[],x,y);

Finally, we predict on a [-4,4]x[-6,6] lattice to see the result of the extrapolation. The figure shows on the left the training data and on the right the prediction. Note that, we also support partial observations. 

  xs = covGrid('expand',{linspace(-4,4,100)',linspace(-6,6,110)'});
  opt.cg_maxit = 200; opt.cg_tol = 5e-3;
  post = infGrid(hyp,{@meanZero},covg,'likGauss',x,y,opt); post.L = @(a) a;
  ym = gp(hyp, @infGrid,[],covg,[],x,post,xs);
f9.png

Exercises for the reader

Inference Methods
Try using Expectation Propagation instead of exact inference in the above, by exchanging @infExact with @infEP. You get exactly identical results, why?
Mean or Covariance
Try training a GP where the affine part of the function is captured by the covariance function instead of the mean function. That is, use a GP with no explicit mean function, but further additive contributions to the covariance. How would you expect the marginal likelihood to compare to the previous case?

3b) Classification

You can either follow the example here on this page, or use the script demoClassification.

The difference between regression and classification isn't of fundamental nature. We can use a Gaussian process latent function in essentially the same way, it is just that the Gaussian likelihood function often used for regression is inappropriate for classification. And since exact inference is only possible for Gaussian likelihood, we also need an alternative, approximate, inference method.

Here, we will demonstrate binary classification, using two partially overlapping Gaussian sources of data in two dimensions. First we generate the data:

  clear all, close all
 
  n1 = 80; n2 = 40;                   % number of data points from each class
  S1 = eye(2); S2 = [1 0.95; 0.95 1];           % the two covariance matrices
  m1 = [0.75; 0]; m2 = [-0.75; 0];                            % the two means
 
  x1 = bsxfun(@plus, chol(S1)'*gpml_randn(0.2, 2, n1), m1);
  x2 = bsxfun(@plus, chol(S2)'*gpml_randn(0.3, 2, n2), m2);
 
  x = [x1 x2]'; y = [-ones(1,n1) ones(1,n2)]';
  plot(x1(1,:), x1(2,:), 'b+'); hold on;
  plot(x2(1,:), x2(2,:), 'r+');

120 data points are generated from two Gaussians with different means and covariances. One Gaussian is isotropic and contains 2/3 of the data (blue), the other is highly correlated and contains 1/3 of the points (red). Note, that the labels for the targets are ±1 (and not 0/1).

In the plot, we superimpose the data points with the posterior equi-probability contour lines for the probability of class two given complete information about the generating mechanism

  [t1 t2] = meshgrid(-4:0.1:4,-4:0.1:4);
  t = [t1(:) t2(:)]; n = length(t);                 % these are the test inputs
  tmm = bsxfun(@minus, t, m1');
  p1 = n1*exp(-sum(tmm*inv(S1).*tmm/2,2))/sqrt(det(S1));
  tmm = bsxfun(@minus, t, m2');
  p2 = n2*exp(-sum(tmm*inv(S2).*tmm/2,2))/sqrt(det(S2));
  contour(t1, t2, reshape(p2./(p1+p2), size(t1)), [0.1:0.1:0.9]);
f6.gif

We specify a Gaussian process model as follows: a constant mean function, with initial parameter set to 0, a squared exponential with automatic relevance determination (ARD) covariance function covSEard. This covariance function has one characteristic length-scale parameter for each dimension of the input space, and a signal magnitude parameter, for a total of 3 parameters (as the input dimension is D=2). ARD with separate length-scales for each input dimension is a very powerful tool to learn which inputs are important for predictions: if length-scales are short, inputs are very important, and when they grow very long (compared to the spread of the data), the corresponding inputs will be largely ignored. Both length-scales and the signal magnitude are initialized to 1 (and represented in the log space). Finally, the likelihood function likErf has the shape of the error-function (or cumulative Gaussian), which doesn't take any hyperparameters (so hyp.lik does not exist).

  meanfunc = @meanConst; hyp.mean = 0;
  covfunc = @covSEard; ell = 1.0; sf = 1.0; hyp.cov = log([ell ell sf]);
  likfunc = @likErf;

  hyp = minimize(hyp, @gp, -40, @infEP, meanfunc, covfunc, likfunc, x, y);
  [a b c d lp] = gp(hyp, @infEP, meanfunc, covfunc, likfunc, x, y, t, ones(n, 1));

  plot(x1(1,:), x1(2,:), 'b+'); hold on; plot(x2(1,:), x2(2,:), 'r+')
  contour(t1, t2, reshape(exp(lp), size(t1)), [0.1:0.1:0.9]);

We train the hyperparameters using minimize, to minimize the negative log marginal likelihood. We allow for 40 function evaluations, and specify that inference should be done with the Expectation Propagation (EP) inference method @infEP, and pass the usual parameters. Training is done using algorithm 3.5 and 5.2 from the gpml book. When computing test probabilities, we call gp with additional test inputs, and as the last argument a vector of targets for which the log probabilities lp should be computed. The fist four output arguments of the function are mean and variance for the targets and corresponding latent variables respectively. The test set predictions are computed using algorithm 3.6 from the GPML book. The contour plot for the predictive distribution is shown below. Note, that the predictive probability is fairly close to the probabilities of the generating process in regions of high data density. Note also, that as you move away from the data, the probability approaches 1/3, the overall class probability.

f7.gif

Examining the two ARD characteristic length-scale parameters after learning, you will find that they are fairly similar, reflecting the fact that for this data set, both inputs important.

Large scale classification using the FITC approximation

In case the number of training inputs x exceeds a few hundreds, approximate inference using infLaplace.m, infEP.m and infVB.m takes too long. As in regression, we offer the FITC approximation based on a low-rank plus diagonal approximation to the exact covariance to deal with these cases. The general idea is to use inducing points u and to base the computations on cross-covariances between training, test and inducing points only.

Using the FITC approximation is very simple, we just have to wrap the covariance function covfunc into covFITC.m and call gp.m with the inference methods infFITC_Laplace.m and infFITC_EP.m as demonstrated by the following lines of code. 

  [u1,u2] = meshgrid(linspace(-2,2,5)); u = [u1(:),u2(:)]; clear u1; clear u2
  nu = size(u,1);
  covfuncF = {@covFITC, {covfunc}, u};
  inffunc = @infFITC_EP;                       % also @infFITC_Laplace is possible
  hyp = minimize(hyp, @gp, -40, inffunc, meanfunc, covfuncF, likfunc, x, y);
  [a b c d lp] = gp(hyp, inffunc, meanfunc, covfuncF, likfunc, x, y, t, ones(n,1));

We define equispaced inducing points u that are shown in the figure as black circles. Alternatively, a random subset of the training points can be used as inducing points.

f8.gif

Large scale classification exploiting the grid structure

Similar to regression using inGrid, we can perform approximate inference using inGrid_Laplace to scale GPs beyond the O(n³) limit. Please visit the website by Seth Flaxman for an extended example and related datasets.

Exercise for the reader

Inference Methods
Use the Laplace Approximation for inference @infLaplace (or @infFITC_Laplace in the large scale example), and compare the approximate marginal likelihood for the two methods. Which approximation is best?
Covariance Function
Try using the squared exponential with isotropic distance measure covSEiso instead of ARD distance measure covSEard. Which is best?

5) Acknowledgements

Innumerable colleagues have helped to improve this software. Some of these are: John Cunningham, Máté Lengyel, Joris Mooij, Iain Murray, David Duvenaud, Andrew McHutchon, Rowan McAllister, Daniel Marthaler, Giampiero Salvi, Mark van der Wilk, Marco Fraccaro, Dali Wei, Ernst Kloppenburg, Ryan Turner, Seth Flaxman and Chris Williams. Especially Ed Snelson helped to improve the code and to include sparse approximations and Roman Garnett and José Vallet helped to include hyperparameter priors. The spectral mixture covariance function and the grid-based inference were contributed by Andrew Gordon Wilson and periodic covariances were added by James Robert Lloyd.

Last modified: July 7th 2015 gpml-matlab-v3.6-2015-07-07/doc/changelog000644 000765 000024 00000000512 12550665722 017515 0ustar00carlstaff000000 000000 GAUSSIAN PROCESS REGRESSION AND CLASSIFICATION Toolbox version 3.6 for GNU Octave 3.2.x and Matlab 7.x Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-07-07. Changes and small incremental bug fixes applied since the last major release from 2015-07-07. - bugfix in likGaussWarp as reported by Obaid Malik gpml-matlab-v3.6-2015-07-07/doc/usageRegression.m000644 000765 000024 00000005271 12262621331 021161 0ustar00carlstaff000000 000000 % demonstrate usage of regression % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2013-10-16. clear all, close all %% SAY WHICH CODE WE WISH TO EXERCISE id = [1,1]; % use Gauss/Exact id = [1,2; 3,2; 4,2]; % compare Laplace id = [1,3; 2,3; 3,3]; % study EP id = [1,5; 2,5]; % look into KL (takes quite a while) id = [1,4; 2,4; 3,4; 4,4]; % deal with VB seed = 197; randn('seed',seed), rand('seed',seed) ntr = 50; nte = 1e4; % number of training and test points xtr = 10*sort(rand(ntr,1)); % sample dataset f = @(x) sin(x)+sqrt(x); % "true" underlying function sn = 0.2; ytr = f(xtr) + randn(ntr,1)*sn; % add Gaussian noise i = randperm(ntr); nout = 3; % add outliers ytr(i(1:nout)) = 5; xte = linspace(0,10,1e4)'; % support, we test our function on cov = {@covSEiso}; sf = 1; ell = 0.4; % setup the GP hyp0.cov = log([ell;sf]); mean = {@meanSum,{@meanLinear,@meanConst}}; a = 1/5; b = 1; % m(x) = a*x+b hyp0.mean = [a;b]; lik_list = {'likGauss','likLaplace','likSech2','likT'}; % possible likelihoods inf_list = {'infExact','infLaplace','infEP','infVB','infKL'}; % inference algs Ncg = 50; % number of conjugate gradient steps sdscale = 0.5; % how many sd wide should the error bars become? col = {'k',[.8,0,0],[0,.5,0],'b',[0,.75,.75],[.7,0,.5]}; % colors ymu{1} = f(xte); ys2{1} = sn^2; nlZ(1) = -Inf; for i=1:size(id,1) lik = lik_list{id(i,1)}; % setup the likelihood if strcmp(lik,'likT') nu = 4; hyp0.lik = log([nu-1;sqrt((nu-2)/nu)*sn]); else hyp0.lik = log(sn); end inf = inf_list{id(i,2)}; fprintf('OPT: %s/%s\n',lik_list{id(i,1)},inf_list{id(i,2)}) if Ncg==0 hyp = hyp0; else hyp = minimize(hyp0,'gp', -Ncg, inf, mean, cov, lik, xtr, ytr); % opt hypers end [ymu{i+1}, ys2{i+1}] = gp(hyp, inf, mean, cov, lik, xtr, ytr, xte); % predict [nlZ(i+1)] = gp(hyp, inf, mean, cov, lik, xtr, ytr); end figure, hold on for i=1:size(id,1)+1 plot(xte,ymu{i},'Color',col{i},'LineWidth',2) if i==1 leg = {'function'}; else leg{end+1} = sprintf('%s/%s -lZ=%1.2f',... lik_list{id(i-1,1)},inf_list{id(i-1,2)},nlZ(i)); end end for i=1:size(id,1)+1 ysd = sdscale*sqrt(ys2{i}); fill([xte;flipud(xte)],[ymu{i}+ysd;flipud(ymu{i}-ysd)],... col{i},'EdgeColor',col{i},'FaceAlpha',0.1,'EdgeAlpha',0.3); end for i=1:size(id,1)+1, plot(xte,ymu{i},'Color',col{i},'LineWidth',2), end plot(xtr,ytr,'k+'), plot(xtr,ytr,'ko'), legend(leg) gpml-matlab-v3.6-2015-07-07/doc/gpml_randn.m000644 000765 000024 00000002125 11422240325 020125 0ustar00carlstaff000000 000000 function x = gpml_randn(seed,varargin) % Generate pseudo-random numbers in a quick and dirty way. % The function makes sure, we obtain the same random numbers using Octave and % Matlab for the demo scripts. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2010-07-07. if nargin==2 sz = varargin{1}; else sz = zeros(1,nargin-1); for i=1:nargin-1 sz(i) = varargin{i}; end end n = prod(sz); N = ceil(n/2)*2; % minimal uniform random number generator for uniform deviates from [0,1] % by Park and Miller a = 7^5; m = 2^31-1; % using Schrage's algorithm q = fix(m/a); r = mod(m,a); % m = a*q+r u = zeros(N+1,1); u(1) = fix(seed*2^31); for i=2:N+1 % Schrage's algorithm for mod(a*u(i),m) u(i) = a*mod(u(i-1),q) - r*fix(u(i-1)/q); if u(i)<0, u(i) = u(i)+m; end end u = u(2:N+1)/2^31; % Box-Muller transform: Numerical Recipies, 2nd Edition, $7.2.8 % http://en.wikipedia.org/wiki/Box-Muller_transform w = sqrt(- 2*log(u(1:N/2))); % split into two groups x = [w.*cos(2*pi*u(N/2+1:N)); w.*sin(2*pi*u(N/2+1:N))]; x = reshape(x(1:n),sz); gpml-matlab-v3.6-2015-07-07/doc/usageCov.m000644 000765 000024 00000007063 12550665732 017606 0ustar00carlstaff000000 000000 % demonstrate usage of covariance functions % % See also covFunctions.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-07-13. % File automatically generated using noweb. clear all, close all n = 5; D = 3; x = randn(n,D); xs = randn(3,D); % create a data set % set up simple covariance functions cn = {'covNoise'}; sn = .1; hypn = log(sn); % one hyperparameter cc = {@covConst}; sf = 2; hypc = log(sf); % function handles OK ce = {@covEye}; hype = []; % identity cl = {@covLIN}; hypl = []; % linear is parameter-free cla = {'covLINard'}; L = rand(D,1); hypla = log(L); % linear (ARD) cli = {'covLINiso'}; l = rand(1); hypli = log(l); % linear iso clo = {@covLINone}; ell = .9; hyplo = log(ell); % linear with bias cp = {@covPoly,3}; c = 2; hypp = log([c;sf]); % third order poly cga = {@covSEard}; hypga = log([L;sf]); % Gaussian with ARD cgi = {'covSEiso'}; hypgi = log([ell;sf]); % isotropic Gaussian cgu = {'covSEisoU'}; hypgu = log(ell); % isotropic Gauss no scale cra = {'covRQard'}; al = 2; hypra = log([L;sf;al]); % ration. quad. cri = {@covRQiso}; hypri = log([ell;sf;al]); % isotropic cma = {@covMaternard,5}; hypma = log([ell;sf]); % Matern class d=5 cmi = {'covMaterniso',3}; hypmi = log([ell;sf]); % Matern class d=3 cnn = {'covNNone'}; hypnn = log([L;sf]); % neural network cpe = {'covPeriodic'}; p = 2; hyppe = log([ell;p;sf]); % periodic cpn = {'covPeriodicNoDC'}; p = 2; hyppe = log([ell;p;sf]); % w/o DC cpc = {'covCos'}; p = 2; hypcpc = log([p;sf]); % cosine cov cca = {'covPPard',3}; hypcc = hypgu;% compact support poly degree 3 cci = {'covPPiso',2}; hypcc = hypgi;% compact support poly degree 2 cgb = {@covGaboriso}; ell = 1; p = 1.2; hypgb=log([ell;p]); % Gabor Q = 2; w = ones(Q,1)/Q; m = rand(D,Q); v = rand(D,Q); csm = {@covSM,Q}; hypsm = log([w;m(:);v(:)]); % Spectral Mixture cvl = {@covSEvlen,{@meanLinear}}; hypvl = [1;2;1; 0]; % var lenscal s = 12; cds = {@covDiscrete,s}; % discrete covariance function L = randn(s); L = chol(L'*L); L(1:(s+1):end) = log(diag(L)); hypds = L(triu(true(s))); xd = randi([1,s],[n,1]); xsd = [1;3;6]; cfa = {@covSEfact,2}; hypfa = randn(D*2,1); % factor analysis % set up composite i.e. meta covariance functions csc = {'covScale',{cgu}}; hypsc = [log(3); hypgu]; % scale by 9 csu = {'covSum',{cn,cc,cl}}; hypsu = [hypn; hypc; hypl]; % sum cpr = {@covProd,{cc,cci}}; hyppr = [hypc; hypcc]; % product mask = [0,1,0]; % binary mask excluding all but the 2nd component cma = {'covMask',{mask,cgi{:}}}; hypma = hypgi; % isotropic periodic rational quadratic cpi = {'covPERiso',{@covRQiso}}; % periodic Matern with ARD cpa = {'covPERard',{@covMaternard,3}}; % additive based on SEiso using unary and pairwise interactions cad = {'covADD',{[1,2],'covSEiso'}}; % preference covariance with squared exponential base covariance cpr = {'covPref',{'covSEiso'}}; hyppr = [0;0]; xp = randn(n,2*D); xsp = randn(3,2*D); % 0) specify a covariance function % cov = cma; hyp = hypma; % cov = cci; hyp = hypcc; % cov = csm; hyp = hypsm; cov = cds; hyp = hypds; x = xd; xs = xsd; % cov = cfa; hyp = hypfa; % cov = cvl; hyp = hypvl; % cov = cpr; hyp = hyppr; x = xp; xs = xsp; % 1) query the number of parameters feval(cov{:}) % 2) evaluate the function on x feval(cov{:},hyp,x) % 3) evaluate the function on x and xs to get cross-terms kss = feval(cov{:},hyp,xs,'diag') Ks = feval(cov{:},hyp,x,xs) % 4) compute the derivatives w.r.t. to hyperparameter i i = 1; feval(cov{:},hyp,x,[],i) gpml-matlab-v3.6-2015-07-07/doc/usagePrior.m000644 000765 000024 00000006614 12550665732 020153 0ustar00carlstaff000000 000000 % demonstrate usage of prior distributions % % See also priorDistributions.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-07-13. % File automatically generated using noweb. clear all, close all % 1) specify some priors % a) univariate priors mu = 1.0; s2 = 0.01^2; nu = 3; pg = {@priorGauss,mu,s2}; % Gaussian prior pl = {'priorLaplace',mu,s2}; % Laplace prior pt = {@priorT,mu,s2,nu}; % Student's t prior p1 = {@priorSmoothBox1,0,3,15}; % smooth box constraints lin decay p2 = {@priorSmoothBox2,0,2,15}; % smooth box constraints qua decay pd = {'priorDelta'}; % fix value of prior exclude from optimisation pc = {@priorClamped}; % equivalent to above lam = 1.05; k = 2.5; pw = {@priorWeibull,lam,k}; % Weibull prior % b) meta priors pmx = {@priorMix,[0.5,0.5],{pg,pl}}; % mixture of two priors g = @exp; dg = @exp; ig = @log; ptr = {@priorTransform,g,dg,ig,pg}; % Gaussian in the exp domain % c) multivariate priors m = [1;2]; V = [2,1;1,2]; pG = {@priorGaussMulti,m,V}; % 2d Gaussian prior pD = {'priorDeltaMulti'}; % fix value of prior exclude from optim pC = {@priorClampedMulti}; % equivalent to above % 2) evaluation % pri = pt; hp = randn(1,3); % pri = pmx; hp = randn(1,3); % pri = ptr; hp = randn(1,3); pri = pG; hp = randn(2,3); % a) draw a sample from the prior feval(pri{:}) % b) evaluate prior and derivative if requires [lp,dlp] = feval(pri{:},hp) % 3) comprehensive example x = (0:0.1:10)'; y = 2*x+randn(size(x)); % generate training data mean = {@meanSum,{@meanConst,@meanLinear}}; % specify mean function cov = {@covSEiso}; lik = {@likGauss}; % specify covariance and lik hyp.cov = [log(1);log(1.2)]; hyp.lik = log(0.9); hyp.mean = [2;3]; par = {mean,cov,lik,x,y}; mfun = @minimize; % input for GP function % a) plain marginal likelihood optimisation (maximum likelihood) im = @infExact; % inference method hyp_plain = feval(mfun, hyp, @gp, -10, im, par{:}); % optimise % b) regularised optimisation (maximum a posteriori) with 1d priors prior.mean = {pg;pc}; % Gaussian prior for first, clamp second par prior.cov = {p1;[]}; % box prior for first, nothing for second par im = {@infPrior,@infExact,prior}; % inference method hyp_p1 = feval(mfun, hyp, @gp, -10, im, par{:}); % optimise % c) regularised optimisation (maximum a posteriori) with Nd priors prior = []; % clear the structure % multivariate Student's t prior on the first and second mean hyper prior.multi{1} = {@priorTMulti,[mu;mu],diag([s2,s2]),nu,... struct('mean',[1,2])}; % use hyper struct % Equivalent shortcut (same mu and s2 for all dimensions) prior.multi{1} = {@priorTMulti,mu,s2,nu,struct('mean',[1,2])}; % multivariate Gaussian prior jointly on 1st and 3rd hyper prior.multi{2} = {@priorGaussMulti,[mu;mu],diag([s2,s2]),... [1,3]}; % use unwrapped hyper vector % Equivalent shortcut (same mu and s2 for all dimensions) prior.multi{2} = {@priorGaussMulti,mu,s2,[1,3]}; im = {@infPrior,@infExact,prior}; % inference method hyp_pN = feval(mfun, hyp, @gp, -10, im, par{:}); % optimise [unwrap(hyp), unwrap(hyp_plain), unwrap(hyp_p1), unwrap(hyp_pN)] gpml-matlab-v3.6-2015-07-07/doc/README000644 000765 000024 00000040637 12546444650 016537 0ustar00carlstaff000000 000000 GAUSSIAN PROCESS REGRESSION AND CLASSIFICATION Toolbox version 3.6 for GNU Octave 3.2.x and Matlab 7.x Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-07-07. 0) HOW TO READ ============== If you want to get started without further delay, then read section 1) below and jump right to the examples in doc/index.html. 1) ABOUT THESE PROGRAMS ======================= This collection of matlab programs implements and demonstrates some of the algorithms described in a) the book by Rasmussen and Williams: "Gaussian Processes for Machine Learning", the MIT Press 2006, in b) the article by Nickisch and Rasmussen: "Approximations for Binary Gaussian Process Classification", JMLR 2008, in c) the article by Candela and Rasmussen: "A Unifying View of Sparse Approximate Gaussian Process Regression", JMLR 2005, in d) the paper by Murray, Adams and Mackay: "Elliptical slice sampling", AISTATS 2010, in e) the report by Neal: "Annealed Importance Sampling", UToronto 1998, in f) the paper by Naish-Guzman and Holden: "The Generalized FITC Approximation", NIPS, 2007, in g) the paper by Duvenaud, Nickisch and Rasmussen: "Additive Gaussian Processes", NIPS, 2011, in h) the paper by Wilson and Adams: "Gaussian Process Kernels for Pattern Discovery and Extrapolation", ICML, 2013, and in i) the paper Snelson, Rasmussen and Ghahramani: "Warped Gaussian Processes", NIPS, 2003. There are 7 code subdirectories: cov, doc, inf, lik, mean, prior and util. cov/ contains covariance functions cov*.m => see covFunctions.m doc/ contains an index.html file providing documentation. This information is also available from http://www.GaussianProcess.org/gpml/code. Usage of mean, cov, classification and regression is demonstrated in usage*.m. Further details can be found in the developer documentation manual.pdf. inf/ contains the inference algorithms inf*.m => see infMethods.m lik/ contains the likelihood functions lik*.m => see likFunctions.m mean/ contains the mean functions mean*.m => see meanFunctions.m prior/ contains the hyperparameter prior distributions prior*.m => see priorDistributions.m util/ contains optimisation routines, backward compatibility programs and small auxiliary scripts Before running the demos or any other part of the code, you should execute startup.m to add all necessary directories to your path. In Octave, you can also use the .octaverc file. 2) ABOUT MEX FILES ================== Some of the programs make use of the MEX facility in matlab for more efficient implementation. However, if you don't know about how to compile MEX files, you do not need to worry about this -- the code runs anyway. If you do compile the MEX files, this is automatically detected, and the program will run more efficiently. If you do not have a compiler installed on your system, you might want to consult [3]. Two components of the toolbox can be accelerated by MEX. First, there is an auxiliary functions solve_chol.m having a MEX equivalent. This can be compiled by either executing util/make.m from the Matlab/Octave command line. In addition to that, we provide a Makefile for Matlab in util/ that you can run by simply typing make in your shell after having supplied your Matlab path and your operating system. We recommend the make.m script because it works both under Matlab and Octave. Second, in order to use the L-BFGS minimiser for hyperparameter optimisation as an equivalent to minimize.m, you have to compile Peter Carbonetto's "Matlab interface for L-BFGS-B" [1] which was extended to also work for Octave by José Vallet. The challenge here is the Fortran 77 code. We provide a Makefile suitable for Linux 32/64 bit and Mac whenever you have 1) g77 or 2) gfortran properly set up. The latter is prefered on current Ubuntu systems. Under Ubuntu, you can achieve this by installing the packages fort77 and gfortran, respectively. Note that the code is GNU licensed. Compilation is done by editing util/lbfgsb/Makefile. In any case, you need to provide $MATLAB_HOME which can be found by the commands 'locate matlab' or 'find / -name "matlab"'. You can choose between two compilation modes: a) without mex utility by Matlab [default] provide $MEX_SUFFIX and $MATLAB_LIB, then type 'make nomex' b) using the mex utility by Matlab provide $MEX, then type 'make mex' c) compilation for Octave provide Octave variables and type 'make oct' In Ubuntu 10.04 LTS, the libg2c library needed for both 1)+a) and 1)+b) is not included per default. If 'ls /usr/lib/libg2c.*' does not list anything this is the case on your machine. You then whant to install the packages gcc-3.4-base and libg2c0 e.g. from http://packages.ubuntu.com/hardy/gcc-3.4-base and http://packages.ubuntu.com/hardy/libg2c0. After installation, you have to create a symbolic link by 'cd /usr/lib' and 'ln -s libg2c.so.0 libg2c.so'. There were reports of problems running the mex files on recent 64bit Ubuntu since the libgfortran.3.0.0 that comes with matlab ($(MATLAB_HOME)/sys/os/glnxa64/libgfortran.so.3.0.0) is not the same as the libgfortran.3.0.0 that comes with the ubuntu package libgfortran3 and that is used when linking. So running the mex file results in a version mismatch error. The issue can be solved by changing the name of to the matlab file and linking to the ubuntu file instead: in $(MATLAB_HOME)/sys/os/glnxa64 run 'sudo mv libgfortran.3.0.0 libgfortran.3.0.0_matlab' 'sudo ln -s /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0 libgfortran.3.0.0' Thanks to Giampiero Salvi for posting the workaround. [1] https://github.com/pcarbo/lbfgsb-matlab [2] http://www.mathworks.com/support/compilers/R2010a 3) CURRENT VERSION ================== The current version of the programs is 3.6. Previous versions of the code are available at http://www.gaussianprocess.org/gpml/code/oldcode.html. 4) DIFFERENCES TO PREVIOUS VERSIONS =================================== NEW in version 3.6, 2015-07-07 ------------------------------ - added a new inference function infGrid_Laplace allowing to use non-Gaussian likelihoods for large grids - fixed a bug due to Octave evaluating norm([]) to a tiny nonzero value, modified all lik/lik*.m functions reported by Philipp Richter - small bugfixes in covGrid and infGrid - bugfix in predictive variance of likNegBinom due to Seth Flaxman - bugfix in infFITC_Laplace as suggested by Wu Lin - bugfix in covPP{iso,ard} NEW in version 3.5, 2014-12-08 ------------------------------ - mechanism for specifying hyperparameter priors (together with Roman Garnett and José Vallet) - new inference method inf/infGrid allowing efficient inference for data defined on a Cartesian grid (together with Andrew Wilson) - new mean/cov functions for preference learning: meanPref/covPref - new mean/cov functions for non-vectorial data: meanDiscrete/covDiscrete - new piecewise constant nearest neighbor mean function: meanNN - new mean functions being predictions from GPs: meanGP and meanGPexact - new covariance function for standard additive noise: covEye - new covariance function for factor analysis: covSEfact - new covariance function with varying length scale : covSEvlen - make covScale more general to scaling with a function instead of a scalar - bugfix in covGabor* and covSM (due to Andrew Gordon Wilson) - bugfix in lik/likBeta.m (suggested by Dali Wei) - bugfix in solve_chol.c (due to Todd Small) - bugfix in FITC inference mode (due to Joris Mooij) where the wrong mode for post.L was chosen when using infFITC and post.L being a diagonal matrix - bugfix in infVB marginal likelihood for likLogistic with nonzero mean function (reported by James Lloyd) - removed the combination likErf/infVB as it yields a bad posterior approximation and lacks theoretical justification - Matlab and Octave compilation for L-BFGS-B v2.4 and the more recent L-BFGS-B v3.0 (contributed by José Vallet) - smaller bugfixes in gp.m (due to Joris Mooij and Ernst Kloppenburg) - bugfix in lik/likBeta.m (due to Dali Wei) - updated use of logphi in lik/likErf - bugfix in util/solve_chol.c where a typing issue occured on OS X (due to Todd Small) - bugfix due to Bjørn Sand Jensen noticing that cov_deriv_sq_dist.m was missing in the distribution - bugfix in infFITC_EP for ttau->inf (suggested by Ryan Turner) NEW in version 3.4, 2013-11-11 ------------------------------ - new likelihood function for warped GPs: likGaussWarp - new exponential likelihood function: likExp - new Weibull likelihood function: likWeibull - new Gumbel likelihood function for extremal value regression: likGumbel - new polynomial mean function: meanPoly - included derivatives w.r.t. inducing points xu in infFITC, infFITC_Laplace, infFITC_EP - allow for zero noise variance for infExact NEW in version 3.3, 2013-10-19 ------------------------------ - new variational inference algorithm based on direct KL minimisation: infKL - improved inf/infVB double loop scheme so that only very few likelihood properties are required; infVB is now internally a sequence of infLaplace runs - improved inf/infLaplace to be more generic so that optimisers other than scaled Newton can be used - improved inf/infEP so that the internal variables (mu,Sigma) now represent the current posterior approximation - added three new generalised linear model likelihoods: gamma, beta, inverse Gaussian - new covariance functions: spectral mixture covSM, covGaboriso and covGaborard - new meta covariance function to turn a stationary covariance into a periodic covariance function covPERard, covPERiso - new periodic covariance function with zero DC component and correct scaling covPeriodicNoDC New likelihood functions in lik/: - likGamma.m, likBeta.m, likInvGauss.m New inference functions in inf/: - infKL.m New covariance functions in cov/: - covSM.m, covGaboriso.m, covGaborard.m (contributed by Andrew Gordon Wilson) - covPPard.m, covMaternard.m, covLINiso.m - covCos.m, covPeriodicNoDC.m (contributed by James Robert Lloyd) - covPERiso.m, covPERard.m NEW in version 3.2, 2013-01-15 ------------------------------ - added inf/infFITC_Laplace.m and inf/infFITC_EP.m for large scale approximate inference - added inf/infMCMC.m to allow for Markov Chain Monte Carlo sampling and Annealed Importance Sampling; we currently support two samplers: Elliptical Slice Sampling (ESS) and Hybrid Monte Carlo (HMC) - improved accuracy of the digamma function lik/likT.m/dloggamma from three to ten digits precision (thanks to Edward Challis) - added gfortran support to util/lbfgsb/Makefile (thanks to Ernst Kloppenburg) - fixed a stupid bug in cov/covFITC.m (thanks to Krzysztof Chalupka) - fixed a severe stability bug in the derivatives of inf/infFITC.m (thanks to Andrew McHutchon) - changed scalars in util/solve_chol.c to long to provide 64bit compatibility (thanks to Ernst Kloppenburg) - added infLOO to allow pseudo-likelihood optimisation instead of the marginal likelihood - fixed a performance issue in util/sq_dist.m (thanks to Krzysztof Chalupka) - fixed a bug causing numerical problems in infFITC and covFITC whenever Kuu is close to singular (thanks to Joris M. Mooij) - fixed an issue in covNoise (thanks to Daniel Marthaler) - fixed implicit derivative w.r.t. likelihood in infLaplace* - added two more likelihood functions likMix.m for mixtures and likUni for classification noise - slightly changed the interface for the likelihood functions - added likPoisson for Poisson regression of count data NEW in version 3.1, 2010-10-10 ------------------------------ - following a suggestion by Ed Snelson we now support FITC regression - cov/covFITC.m and inf/infFITC.m have been added with Ed Snelson's help - the covariance interface was slightly changed to make that possible - changed parametrisation in cov/covPeriodic.m (thanks to Philipp Hennig) NEW in version 3.0, 2010-07-23 ------------------------------ A major code reorganisation effort did take place in the current release. First, classification and regression are now done by a single file gp.m which is completely generic in the likelihood. The previous regression program gpr.m corresponds to gp.m with Gaussian likelihood. Several other likelihoods for robust regresssion were added. Further, the code now supports mean functions with a similar specification mechanism as already used by the covariance functions. Previous implementations correspond to using meanZero.m. We merged the covariance functions covMatern3iso.m and covMatern5.iso into a single covMaterniso.m and added a covariance function for additive functions as well as the possibility to use only certain components of the data. Finally, we included covPPiso.m a piecewise polynomial covariance function with compact support. New likelihood functions in lik/: likGauss.m, likLaplace.m, likLogistic.m and likT.m New mean functions in mean/: meanConst.m, meanLinear.m, meanPow.m, meanProd.m, meanSum.m and meanZero.m New covariance functions in cov/: covADD.m, covMask.m, covPPiso.m The gprSRPP.m function which previously provided "Subset of Regressors" and the "Projected Process" approximation has now been removed. NEW in version 2.1, 2007-07-25 ------------------------------ covConst.m: fixed a bug which caused an error in the derivative of the log marginal likelihood for certain combinations of covariance functions and approximation methods. (Thanks to Antonio Eleuteri for reporting the problem) gauher.m: added the function "gauher.m" which was mistakenly missing from the previous release. This caused an error for certain combinations of approximation method and likelihood function. logistic.m: modified the approximation of moments calculation to use a mixture of cumulative Gaussian, rather than Gauss-Hermite quadrature, as the former turns out to be more accurate. NEW in version 2.0, 2007-06-25 ------------------------------ Some code restructuring has taken place for the classification code to make it more modular, to facilitate addition of new likelihood functions and approximations methods. Now, all classification is done using the binaryGP function, which (among other things) takes an approximation method and a likelihood function as an arguments. Thus, binaryGP replaces both binaryEPGP and binaryLapaceGP, although wrapper functions are still provided for backward compatibility. This gives added flexibility: now EP can also be used wth the logistic likelihood function (implemented using Gauss-Hermite quadrature). approxEP.m: New file, containing the Expectation Propagation approximation method, which was previously contained in binaryEPGP.m approxLA.m: New file, containing Laplaces approximation method, which was previously contained in binaryLaplace.m approximations.m: New file, help for the approximation methods. binaryEPGP.m: This file has been replaced by a wrapper (for backward compatibility) which calls the more general binaryGP function. binaryGP.m: New general function to do binary classification. binaryLaplaceGP.m: This file has been replaced by a wrapper (for backward compatibility) which calls the more general binaryGP function. covMatern3iso.m, covMatern5iso.m, covNNone.m, covRQard.m, covRQiso.m, cosSEard, covSEiso: now check more carefully, that persistent variables have the correct sizes, and some variable names have been modified. cumGauss.m: New file, containing code for the cumulative Gaussian likelihood function likelihoods.m: New file, help for likelihood functions logistic.m: New file, logistic likelihood NEW in version 1.3, 2006-09-08 ------------------------------ covRQard.m: bugfix: replaced x with x' and z with z' in line 36 covRQiso.m: bugfix: replaced x with x' and z with z' in line 28 minimize.m: correction: replaced "error()" with "error('')", and made a few cosmetic changes binaryEPGP.m: added the line "lml = -n*log(2);" in line 77. This change should be largely inconsequential, but occationally may save things when the covariance matrix is exceptionally badly conditioned. NEW in version 1.2, 2006-05-10 ------------------------------ added the "erfint" function to "binaryLaplaceGP.m". The erfint function was missing by mistake, preventing the use of the "logistic" likelihood. NEW in version 1.1, 2006-04-12 ------------------------------ added files: "covProd.m" and "covPeriodic.m" changes: "covSEiso.m" was changed slightly to avoid the use of persistent variables NEW in version 1.0, 2006-03-29 ------------------------------ initial version shipped with the book gpml-matlab-v3.6-2015-07-07/doc/checkmark.png000644 000765 000024 00000000306 11422240325 020263 0ustar00carlstaff000000 000000 PNG  IHDRa*PLTExxxlll```TTTHHH<<<$$$ tRNS@fJIDATc````d6 N`8,~e RS!Ke0O P`ʳ&] NIENDB`gpml-matlab-v3.6-2015-07-07/doc/demoRegression.m000644 000765 000024 00000011772 12076017514 021011 0ustar00carlstaff000000 000000 disp('See http://www.gaussianprocess.org/gpml/code/matlab/doc/ for details.') disp('Hit any key to continue...'); pause disp(' '); disp('clear all, close all') clear all, close all write_fig = 0; disp(' ') disp('meanfunc = {@meanSum, {@meanLinear, @meanConst}}; hyp.mean = [0.5; 1];') meanfunc = {@meanSum, {@meanLinear, @meanConst}}; hyp.mean = [0.5; 1]; disp('covfunc = {@covMaterniso, 3}; ell = 1/4; sf = 1; hyp.cov = log([ell; sf]);') covfunc = {@covMaterniso, 3}; ell = 1/4; sf = 1; hyp.cov = log([ell; sf]); disp('likfunc = @likGauss; sn = 0.1; hyp.lik = log(sn);') likfunc = @likGauss; sn = 0.1; hyp.lik = log(sn); disp(' ') disp('n = 20;') n = 20; disp('x = gpml_randn(0.3, n, 1);') x = gpml_randn(0.3, n, 1); disp('K = feval(covfunc{:}, hyp.cov, x);') K = feval(covfunc{:}, hyp.cov, x); disp('mu = feval(meanfunc{:}, hyp.mean, x);') mu = feval(meanfunc{:}, hyp.mean, x); disp('y = chol(K)''*gpml_randn(0.15, n, 1) + mu + exp(hyp.lik)*gpml_randn(0.2, n, 1);') y = chol(K)'*gpml_randn(0.15, n, 1) + mu + exp(hyp.lik)*gpml_randn(0.2, n, 1); figure(1) set(gca, 'FontSize', 24) disp(' '); disp('plot(x, y, ''+'')') plot(x, y, '+', 'MarkerSize', 12) axis([-1.9 1.9 -0.9 3.9]) grid on xlabel('input, x') ylabel('output, y') if write_fig, print -depsc f1.eps; end disp(' '); disp('Hit any key to continue...'); pause disp(' ') disp('nlml = gp(hyp, @infExact, meanfunc, covfunc, likfunc, x, y)') nlml = gp(hyp, @infExact, meanfunc, covfunc, likfunc, x, y) disp(' ') disp('z = linspace(-1.9, 1.9, 101)'';') z = linspace(-1.9, 1.9, 101)'; disp('[m s2] = gp(hyp, @infExact, meanfunc, covfunc, likfunc, x, y, z);') [m s2] = gp(hyp, @infExact, meanfunc, covfunc, likfunc, x, y, z); figure(2) set(gca, 'FontSize', 24) disp(' ') disp('f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)];') f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)]; disp('fill([z; flipdim(z,1)], f, [7 7 7]/8);') fill([z; flipdim(z,1)], f, [7 7 7]/8); disp('hold on; plot(z, m); plot(x, y, ''+'')') hold on; plot(z, m, 'LineWidth', 2); plot(x, y, '+', 'MarkerSize', 12) axis([-1.9 1.9 -0.9 3.9]) grid on xlabel('input, x') ylabel('output, y') if write_fig, print -depsc f2.eps; end disp(' '); disp('Hit any key to continue...'); pause disp(' ') disp('covfunc = @covSEiso; hyp2.cov = [0; 0]; hyp2.lik = log(0.1);') covfunc = @covSEiso; hyp2.cov = [0; 0]; hyp2.lik = log(0.1); disp('hyp2 = minimize(hyp2, @gp, -100, @infExact, [], covfunc, likfunc, x, y)') hyp2 = minimize(hyp2, @gp, -100, @infExact, [], covfunc, likfunc, x, y); disp(' ') disp('exp(hyp2.lik)') exp(hyp2.lik) disp('nlml2 = gp(hyp2, @infExact, [], covfunc, likfunc, x, y)') nlml2 = gp(hyp2, @infExact, [], covfunc, likfunc, x, y) disp('[m s2] = gp(hyp2, @infExact, [], covfunc, likfunc, x, y, z);') [m s2] = gp(hyp2, @infExact, [], covfunc, likfunc, x, y, z); disp(' ') figure(3) set(gca, 'FontSize', 24) f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)]; disp('fill([z; flipdim(z,1)], f, [7 7 7]/8)'); fill([z; flipdim(z,1)], f, [7 7 7]/8) disp('hold on; plot(z, m); plot(x, y, ''+'')'); hold on; plot(z, m, 'LineWidth', 2); plot(x, y, '+', 'MarkerSize', 12) grid on xlabel('input, x') ylabel('output, y') axis([-1.9 1.9 -0.9 3.9]) if write_fig, print -depsc f3.eps; end disp(' '); disp('Hit any key to continue...'); pause disp(' ') disp('hyp.cov = [0; 0]; hyp.mean = [0; 0]; hyp.lik = log(0.1);') hyp.cov = [0; 0]; hyp.mean = [0; 0]; hyp.lik = log(0.1); disp('hyp = minimize(hyp, @gp, -100, @infExact, meanfunc, covfunc, likfunc, x, y);') hyp = minimize(hyp, @gp, -100, @infExact, meanfunc, covfunc, likfunc, x, y); disp('[m s2] = gp(hyp, @infExact, meanfunc, covfunc, likfunc, x, y, z);') [m s2] = gp(hyp, @infExact, meanfunc, covfunc, likfunc, x, y, z); figure(4) set(gca, 'FontSize', 24) disp(' ') disp('f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)];') f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)]; disp('fill([z; flipdim(z,1)], f, [7 7 7]/8)') fill([z; flipdim(z,1)], f, [7 7 7]/8) disp('hold on; plot(z, m); plot(x, y, ''+'');') hold on; plot(z, m, 'LineWidth', 2); plot(x, y, '+', 'MarkerSize', 12) grid on xlabel('input, x') ylabel('output, y') axis([-1.9 1.9 -0.9 3.9]) if write_fig, print -depsc f4.eps; end disp(' '); disp('Hit any key to continue...'); pause disp('large scale regression using the FITC approximation') disp('nu = fix(n/2); u = linspace(-1.3,1.3,nu)'';') nu = fix(n/2); u = linspace(-1.3,1.3,nu)'; disp('covfuncF = {@covFITC, {covfunc}, u};') covfuncF = {@covFITC, {covfunc}, u}; disp('[mF s2F] = gp(hyp, @infFITC, meanfunc, covfuncF, likfunc, x, y, z);') [mF s2F] = gp(hyp, @infFITC, meanfunc, covfuncF, likfunc, x, y, z); figure(5) set(gca, 'FontSize', 24) disp(' ') disp('f = [mF+2*sqrt(s2F); flipdim(mF-2*sqrt(s2F),1)];') f = [mF+2*sqrt(s2F); flipdim(mF-2*sqrt(s2F),1)]; disp('fill([z; flipdim(z,1)], f, [7 7 7]/8)') fill([z; flipdim(z,1)], f, [7 7 7]/8) disp('hold on; plot(z, mF); plot(x, y, ''+'');') hold on; plot(z, mF, 'LineWidth', 2); plot(x, y, '+', 'MarkerSize', 12) disp('plot(u,1,''o'')') plot(u,1,'ko', 'MarkerSize', 12) grid on xlabel('input, x') ylabel('output, y') axis([-1.9 1.9 -0.9 3.9]) if write_fig, print -depsc f5.eps; end disp(' ')gpml-matlab-v3.6-2015-07-07/doc/usageClassification.m000644 000765 000024 00000005144 12227514045 021777 0ustar00carlstaff000000 000000 % demonstrate usage of classification % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2013-10-16. clear all, close all %% SAY WHICH CODE WE WISH TO EXERCISE id = [1,1]; % use Gauss/Exact id = [1,2; 2,2; 3,2]; % compare Laplace id = [1,3; 2,3; 3,3]; % study EP id = [1,5; 2,5]; % look into KL (takes quite a while) id = [1,4; 2,4; 3,4]; % deal with VB seed = 943; randn('seed',seed), rand('seed',seed) ntr = 50; nte = 1e4; % number of training and test points xtr = 10*sort(rand(ntr,1)); % sample dataset p = @(x) 1./(1+exp(-5*sin(x))); % "true" underlying probability ytr = 2*(p(xtr)>rand(ntr,1))-1; % draw labels +1/-1 i = randperm(ntr); nout = 3; % add outliers ytr(i(1:nout)) = -ytr(i(1:nout)); xte = linspace(0,10,1e4)'; % support, we test our function on cov = {@covSEiso}; sf = 1; ell = 0.7; % setup the GP hyp0.cov = log([ell;sf]); mean = {@meanZero}; % m(x) = 0 hyp0.mean = []; lik_list = {'likGauss','likErf','likLogistic'}; % allowable likelihoods inf_list = {'infExact','infLaplace','infEP','infVB','infKL'}; % inference algs Ncg = 50; % number of conjugate gradient steps sdscale = 0.5; % how many sd wide should the error bars become? col = {'k',[.8,0,0],[0,.5,0],'b',[0,.75,.75],[.7,0,.5]}; % colors ymu{1} = 2*p(xte)-1; ys2{1} = 0; for i=1:size(id,1) lik = lik_list(id(i,1)); % setup the likelihood if strcmp(lik,'likGauss') sn = .2; hyp0.lik = log(sn); else hyp0.lik = []; end inf = inf_list{id(i,2)}; fprintf('OPT: %s/%s\n',lik_list{id(i,1)},inf_list{id(i,2)}) hyp = minimize(hyp0,'gp', -Ncg, inf, mean, cov, lik, xtr, ytr); % opt hypers [ymu{i+1}, ys2{i+1}] = gp(hyp, inf, mean, cov, lik, xtr, ytr, xte); % predict [nlZ(i+1)] = gp(hyp, inf, mean, cov, lik, xtr, ytr); end figure, hold on for i=1:size(id,1)+1 plot(xte,ymu{i},'Color',col{i},'LineWidth',2) if i==1 leg = {'function'}; else leg{end+1} = sprintf('%s/%s -lZ=%1.2f',... lik_list{id(i-1,1)},inf_list{id(i-1,2)},nlZ(i)); end end for i=1:size(id,1)+1 ysd = sdscale*sqrt(ys2{i}); fill([xte;flipud(xte)],[ymu{i}+ysd;flipud(ymu{i}-ysd)],... col{i},'EdgeColor',col{i},'FaceAlpha',0.1,'EdgeAlpha',0.3); end for i=1:size(id,1)+1, plot(xte,ymu{i},'Color',col{i},'LineWidth',2), end plot(xtr,ytr,'k+'), plot(xtr,ytr,'ko'), legend(leg) gpml-matlab-v3.6-2015-07-07/doc/f9.png000644 000765 000024 00000053646 12441263151 016674 0ustar00carlstaff000000 000000 PNG  IHDR~͕ pHYs B(xtIME 5()"tEXtCreation Time06-Dec-2014 18:53:40ƴkb$tEXtSoftwareMATLAB, The Mathworks, Inc.R IDATxm8s\+\,\A  p+\A Wܮ @"9^t/(Q]f`9{&)X`a`I &)#Ƙ ;=lẁN )0I5Rp8g-]#\muAd~f[N*0d{w}%'>uЧt${!dApan=FGKNWQ?$Ft,@ ΀4Ѻ9I KRgX=V9}Lo@5kUHDdNh::Op4PI^\?)pOI>]gD7:b7{){$7GU_::0)̸XOVuX!"맕x,x)PI[IH `$ngASaxsC&iI` !?`)v+It 0˧q!LO$ ;n#s`do\>%/up}SdA!CtI#`e-(o M 0z,,M,m^7ы`IZTҏܿu`M$nuGUu7~. TҊ&Ws$ǒ6~3׽\ꕡ_]I2ͯhl9; |5t7l/(OAɯo> "_5O/xC }}N ș*\/Fܜ5YQ~W/q5W׹E-Xd6z!"=Jqߗ$}qhNgb{hIZgזۯJK e:Myt}KzVΧqDmޒ m,Kx/gcCԍ(ez-VIN$0:dۈ08|{yF0pJ~I)],*yow{lEBjGQWa?EJg\f\i(ΔH_Vg :)83D: ?0Y+M$Ε&Jr3xL~7}Y VM(BלmLD0$Z.1{p b͚,ol,wvL5!kV߯wpj_:1~6M8JmX9[ZgGvoZX'"PiL0l$3gl}~3_EPjW'ZP%s)O(?O=BժPb9-\O/L<{`΁D> ^B.Խ3g.β>Z$j/PGX-;ʢ:>A-iP| s2Z*yσOy3K,]nfԃ9I W&Gq?iÝf]-ߔ_3chIWCiz{F cީ[i:`.D"p}P%(E,e9R9 ]%K,^KCn,vov _/[nMs֠(oz0Ivq4GE~~gU( ?p%EЎrR+=uee[9>0K6)"f-tzvE:u[&_T~ \-u;C($+&9 l;|ȥUT\Pjċ_ImjX?BKQNm#eRnQ0L|جz%68/ܝU 7"Psq`d&Uscz}W~V:B2G?9US٧=& 9 0 _;))Xy| X$x/j|W)z R`$8x/x)*^d-+7ؾ75h}1rPK =ힽ:5ƌWY~e":ZF3cL)SocE}Şֶ`r_˓^B?  tk1(?Kv7nǣ:Fٓ>h_?Aw||F OI\Rܬٓʠ/޻) ܯμ=*xYNјW!-*:67Wng`)vů!^y2h柭?0XK0p?WPLg3X{Fn Q"ԄԖ̓.63)-*i>*dn2]{;4#xv\>/lyLIz9j)j e|z=Ò2\&"v0 /޻)lv|dQ KWd!63^{d&԰*i\Z}H5߳=Dl-cNޔC%ftkj7ȥr7ĤsZHi࣡ 7uK9t(EKڽJMcc*}86q0ujhjm6faړ|L(Xv!}w+EG%d[L9vl$].u$?x+7>>˹o({Ik'K'*QS!mV˥>7~PsEmJfֆk,= ĩKTK(Ba4+Ѿ?%<8(j7Ѣ˯;Tأs%h!k=bw~0 4s3ӛ.ؼ=~/YgﾴSc 3jd%4ltPv-͝H%]v9ƖL-N0Pepsfޔ:3Ǐ̢'ŭyi|j=7qI٦y0Eޤ/*m\x=d_wSPIqwRv1 e *Լ/OgP}ͯ..NHcLEZJfV' 袃jT.o.IƜ6f%R!aVgESNu}RN R+zJ,vJt[m/X1o^}z]~.sa?H.~6+W2;E|}y SOrPdP? X雹K,ǽOŝ!ho=Z9J6> Aio#t;)CSd[ -z`@%m%/c C(|.K.7w0_]bT%_q_U ^WJ.xE *|56IFnNv}i{٣ӛ,u--C[C;=E)擩!4φS3|&v}Oor;Lo*9S:N[$=mh~5ӫ*&w nn1ٝ{7ZRG˫3F+)1Fb?M2]vjKMg~3p'%ϑ+# n~ޛ_ "2 أ乼UG\frd3qe@\wah Mג?;tӛ { ;ЩE[lѮu|q6M"y*$Uю#]FB*Ӣ'C=k@Q"IN^"{ Ṽot-Lտ `YN¼7+wfj!9!+)1V*T JAOpKF^(} ;&${e~.94ӛxzgaSEf"GP-ܬk_zȫkj.qcKJZ-þN1\8WT6C ;5+ ֒?;$ݜ}+ g/%{Ϲ c\M2K @j1~/]yw1ٕ^ +<0D|tE&MYS0ULyQJɣC 96ˍߟUP:΋ I=ZTf9Rmӛ:B{LRL= MKbY0gA|\2'a Ҧ,]Fkbd߂#. ͭ?EܝԿtrk~WC T!Iiﻃx)?ᾙM3"Pw`Ӏ`aEUvCKt8*kߵP;,R՜r_7< 5(t$JZ)./!K38NǤ;4]^7=Nm+d: ]M`i0~6h`##ԺU|f-97"mDPdo ޘ>BN]K# ew1جjѬm{`aPC*W@(@~EyJVR3޴[-b>491L|Nȵ"Mdy5KW{Ա{v2 :y{ ;F oҘm6:NK:ݱg^AO_u}~ ?_/4?AKMG)Ot֎gͯgm|o+ֿ3hO{%;0޽1馒$cLRILT}M!w $Vj7o"Rŷu-}]o'9{H. wd \\q%*46+.~lPjm]Vo] "ߧj1}jXrTx,jrS8@|GTD{8qGM{tSI{D+)2ma_TpW2E]hS7= mJۮ6b5nM4LU\5(Fj$'K ټ8g^UX,v(gTI5eJsR>+o/Hؤi29E.G;֪DJ=b&95bz ~:\DzI/bښ{riPINkt\C/jh&=61DnN^R- [e&-Ôۍ*Stz;3jAq>3'nE.]M=Qͱ~<#wScF_Yb߄1 s&d G5&NΝ{Mp ϹhVlDXIFx܈# \Ɗ.nIJ"aOnj߮^[l̮!؛g*=^}of=4f*T-1о'kwG 3Fsȋ\O V-}3B3 1x9Y0(.[ 4J<̍|4B {s/y(xc`KwCVx27a/~L 2d|27\ A~.F':~$U¹dwù}NMl/_kPK_I5{((ۨ:Zn}Hy-SjIk׼7h"M.RI"8n;{!UyȓO_|-FtyZict|5O;1X\.>xݥ#DW'1nO5|5dHeO_n0Wd^h< ?O/lao u3{_U&m{U2¶[rݜנ6hCh_̶ JןqMv8UrF?"tSI`A0I)N s0IxI:)r7EN~:|Y~~_3T#,Zͥ?(Hj%}0w ߒRA}rʵ]͠k8&x|I'\4;8>q- 2o3SLohV|c~ K sxkGѬ. Xb `Ƽ]>qBgJPjӛ4iEsO.ZYTwէ͞42eudQXIDFG9G[=8 IDATɤ:ṠTqΊJIhLR=ޛ5Fqef(8rtSLojetu@GX`n)9)4ЎJ t&*Ǘ P. KL?5RXY%՟o[a_|&`s=>W 48M@9K̇J28;Y rp(w~.>;.m{suTQ놑meN*Rtk[p?+cI:9=u-s>TTSH2إLΣuG6iRks'e).[,p%F4s1^"?X cQŊ+?N沎bɊ&t9bq KL7p7bYLs>ZStr;%f\KRFty*~h_#NQrR9, ր< *QdY{TEʰΗ4G}IPKzeca_k.+ǗTCE# ͚g%pшYq(r$A3'_#f=Ke5Oa: KMޢr j87}ٺy6t-Q %l`\祫VQPѫ!PIg(N4 f8H8Y-T0Mo93rjzSv]]n@j!qʨ^WʜrF,$Y";ݔ*)h X1(ȱ},wa*l LJ *4TVZVSLU*:--3sK$*GKFA:2hO7)>)*%Ʃ\*BܳK?3MMqfmiD8`STx27KW4spc2{F@U-ϙ[t6Q T7 %!ZjMONŶL9..-9˯᳦et5v|/Xq? :gK1eXW .ɥ^^F%>|kS#YiRV;5?PIA3W= 3Fy j.gsI[4`plPrpGs+T2s̅ 1x/ȍ61"P#K;-MriEߑ-6~ dIܡ1? o~Q b ɜMvT[:qbA_lvDhn_R FCp{sn S4Ecם#Ӎ9w_Xpb bP@ǯY6h}ϛKZE"'0p?9 8;x6O8 ٺ=zc¦=jJ:9vvTq&y$ESQ mn)UyOl]֓9Σ&9ӛӛZgh`na'Mo1FDMA$S4Pn SN*&TR 73޴1KW v@%8dNRWT^ODzMVL6R>(9 "m*U8ii_AMglz 3h\ɥ+%ʧ%g /wJM6fێ;E,Z*fPdM ؑ=B4þ獼cŮ*lG:~#E)+ rklvG}弚˿ :8q/o þS}г hKZ*vܨŗ Tv`\xo;~/]pNlkP,TCBY̅?ҥ|tjm3+E͊(ٗ4[5vtCJqT%}4v6nD."S5/i)PIA+Y{6}+Bj|~-]pN4V]bZ}0vO&Z6B7MYK(*Ϩ$k _4&OS% (7ML؎^cuh6#$;EuYe>좃ʅȣ;mCM̀%Iڊ݃`Gʳ(|Iq߱Y /i2E[`_^-e>l/<2ڢNW2Pl~.]y7 gt-Q %{s`t|4~?x㓹Ś=k?\/O.󁎯?\ f+I+^?5W~jpcͭoo``I`;g5:ޘ'?y32^g?ʼWo`d{a͏SdɚZc S7O`5w#\žq=uo$pAw _`%ju\?!.attSgڠ_x265 EݾmtV!O &qA|bS=A}ݳa.\@Wx9 AZ _)R=NoRIǵޔn=]{H,+Nſ%Gy MyxV5(B̍]0/G-0I X?̜B7e˖Y׷8g2XTEU1/i4.u+AQш&鬔GM<څb?hQ]\Nhf(7g`tg?ٟ!?`MEI Z8kSTVڄa_uC:v%RKu6_R5{ >kf\|IRqpX`NtIi2O nQq뻘 oS0Oϕ 0/iV.-D% 嘸X7!=hOh_ n *`{tfY`. dQ`'XbK:+]\DDGiþ_K ϐw|DG2{?ʨK=6;{>G""cue^KqcGt7K~0w.@Gs>2|S-ͽ_r ¿˿0o悉n!ȥz+P+ cֿOFv6zLs ܺƒ׾\}-q0nA;g4Q';ͳbNqg[rEEp( E<5i6{c7!fR=0q7qmoˎ3j9Fhq-5@5L8&z(.1OoQaA.ʼX-*~+ޚYrgx:K7}v5"p` 5&IڣPIgE>[F$d둝>Y* #eG4N6R(LZF,S|W̭Văɍדk/A`)mO iq+R%feiz +3tgwfqtrP(L0&΀n/z\/]m{tr],JGOѱ-'BGqdT;Y6Xpz~EnޗZi>c5lvapHO1Uw%͟Z4%87=Fd28 }t)O #qʐv[8ʼ`ca   SdأrӛNLGItþܗ 7b9sርOјͰox=?p ZӅ50hֳޔ(:P kd!U漼EjNr=?U}ߒuVl~VQ LAG nӛp9ٌW7dMY7)[Z"$0[󘜚,m2vI:Ai>V *͠zz7̶ElunAsmyUKiSڈ5(ӛ/Z Ƃ`ۿ *<gJJqPQk41ZHs|Qff?f 0''X*g0p zLokҼ.]Kk|m:ہ.34 ^>6Q]s;>vԠqj|fTyvぎ+r{2xcϛ[hAr%_g̋_0m4!s݋# n $Q%}=*fq%>NJA]fSٛI{\0.huɠ9Yd:d.\0I'~#?a;ޙRͽø]{ȸ`j~"w~*wsld?Hm|ʼ?WAgw!?ٴ%Qj?*THk=u]u ?5ɉG(;^W`Q"5|5|=bbavWjF÷QPC *9V +/6@B~"gJ*54lz eMmܾr7AWn i4~o)|I{v@=Svt-Qn12T:ŵtΣs#? ɽҼQM>|IyģOL Dsa dݑnr@_qolq74|e;Zq4W ʩV&޻CWt1\}Q9:Z^:[̩+d5hz+ۇLx:1vɕQ E3ǭ&)h̀E;-Ax TҘ-A:>sE%%ԋ l6˥r\Z94h%0I[qTLRڔ.,QdVTEw֞bΨ9;E<9.kUC6%V1KWcأ+d(E|o0iZܼu._}H53x|V.9ki!=5ܕ7uEۏ-;KEӆgB(?5%\$alpzѽI7p؂mKK1k帴s4*"֒Jnj*)h k`*_i]&෨ɏ.K"O,MSvk5:h%')h Q,$}džSsgdR]ʼ$]6Ayi#gh}NIN=4ȗ4P0`n} TɁKʹ7;y%eS3V`ZN7խ-:^l`JcM65Ea n{HSe}RøuC +(vI/6˥×|6e!Sr)~T08GA9PIA+~*zX{p>`錾`pl.Cݛ4Mi]ׯ^&nhWIuّBꄮ wR7QZfNQ'5Kg`NN2Ouuc9I×)i0+ʜaH#[2˅ {6h`(}l}]9 {T=>1ߪsþٞ'?3&9>KEqUF ~_|aq4nc! Ll~<-]m{trzuסfL1R_qz-=MEԉ%cԔ5}8{? -xՇ9ޑb$+)5;4&}0{OEXFM x>j\ap,߫U 0IA+ww.޴x=^1&s=rL47j^灓h|_>(~\q%oGcpV?ף%  !  (UZ "2G31&&GNuJ:h Ja6\Q%F3|gU c<\(aszVBlcJGDU*iVx 06Fč{Pgr*5"HWՎb|uIIDATTŴ0ZF'GRc-*{wS% V?kï=zWR O~;Jy)9.n&-u)ui;%=DXd6pkO$^cω44t%6DR&l~%8kMKOorZ)B i|)21PQazM:YĕAN)Ol#XGa:S;&N\cVd$Sعaa˾uY LkFR;@@WR;! *ŭ*!VZ<$q҇=w6Vvu]iS[fVkbO?߮;_F ϼRbDP8*gмN/`΀G8^Chі\Pa킦)=Y,iR;zWIIU7&4l~.֠X[eT~FLQ*9u5.F0NUk T>M/L2B6$]jY|]RNhlGgؽkNԥ2?JnNCڱB Ķض'"\-wvo i=Gp]!G5G5X~Zmw<)Σr)!uJ\Kۏ{䣺Z):M ?ON '&Zdq(ykAۇhi#Ef(1k7{[m'TƓ1n{e0{0p?9@DoGK-dҋA1,*u+lNZqTsL VW14^&i675?Ãs;>G#ʢRR]Mad}幦P%{KK϶>5(w@F LV.^pR+%fb KW#ۣgi.Tͱ }?{cW l{ld3soJ >N*ן"[UnjͲ1`437vk}R^V+<&i+88tΣeأu>>lhDSzL?-Fc8Ujnh1s ڣd`nƼ/$taFAҼ +"ͳ1OgH[!s oJb%ȗ-|2`a8oH1]'tJrA.aGf\u+yQ%}Wg?lA7;4}S'#q/_F ,3'=7nPWIwcG? dƕ FE||m!(f|lPq?E٣Ƙ >#6RRxVԍyQq~kiO:tTy+e@yl2|,:7;}hqT2ƼK#"8#!@hQ/䔫qdywc"dKv^Ov( mT5@F<"]Lb:(3bGlh""'Bʮm.11)JQ1fuI5悙"SeI<*d3Du*UIi)ASazr ^jele"unRR/5bL w;+gcxvT<wSPI;`<Ymy^Z_:V4䧛b?u룕[ѢDyh{ƎOC*D3g_T :!:]M~*r[ؒ{)^O8/AUwO|.wd\@5(J ZDsض%E3A#XmK:-/QElpeQKRSEФd'gKb-7W{w0Ƽ+kx׷0ޡrϦ ^f+vCp",SiȪ\x#=ZwMa&8y9 m+ێ2>F8|ISϚ/Z~l~7`a]]-nb'<}kC.]ki@Ik ]PmGʢw];2+£ wS%Չ$P-f4{92r8(f5\K"ϼRɥVx +BfѼ콛J:9`+O \-eH"`o/3yf} Ma713V;':%)z|2: 67 [^A&)h`&;;p}e-°/7e]O׎3b>eO.)A`qHJg6 QzYL'OlӓrR3ߕC2b9dlL:%Ue 3]!{֑eרӛ} lSw{E ̶تOuFgK>*o,Mt* rRѴ=% =qUgs7Ƽoi8Q☊RYm4/4fO(n8F3BVxB%`zX3z+=F0K7͋6:♯4v<GvYqǔ.bj{>C*إ^t\^=>+a0^d7Y}tר*`;sSc0$}I26}yhTD?"A4hxQCѴB-#5.ŀGe37DQ L|mf`"ͻrc$s!2 DV !|Cw(_h8h\AtꬔVKF*Nf s!K8Z-+kDti^A*:en/;j<"4WȮKU(C"gg::F{I17KDMm{4."뚆=1#:bjUK礏>j))K$ՏMgQ lgqp7{_O3KD{x}g80A.]R(ZP ha\-G7b#(.C] w+hEn*l|`=<݌@3Ͼ]cvTi1Ƽ4 OQjLDS8E-u zgkt(ywi*bPI1;fTlw'Ujt z݅R.UU-Ј lssmJsIūt-6xŮݫ޴Jʽ9bjQu<h75-zр v&%35ۣ{\;Ւ /,]-y{$`zX3ؽ/G8jQħsɲکG){4*Tm\Nee|ut. dU>zʬzcԞN*4E ]Ibdő ;@{\|[Sxg9o r[`:zKc;s8w=ݛg>{⑉'&&-P!q܇ Wblk?hL|n]f̪GD_A-Pt.g"Iڎ߹iJTw mj8JcfF~HfFrNK,/=gWA%](םt~rRW7s n`n$&i_U'VMKU4.ZiJ,A'+Hm90' /d^ɼ"/yE&Lf`;/Qv܀f`^KACDo#mTy.#7ǨA'uށ+k$奾% bB )x.+|ޚ;1x/ǒSC!r^Ġ|'V4>Np=O^`+%&A0(98oT 7X>ÿ,jSd_!+ Agk_kޥ칯7`%칛"Lo?^n `YMףS@_LR0qͻn7u _c/y/Oz_8fnEQzW&Td7_;6 xv~_9/p>lg]zf> ,MmbQLҩ{ 6o_`/=> stream xڅn8ޯi!5+z-ld7h aZmdtpH[Jyjޤh% i×Ғ4Ϣ&J낥τ#SKm\@>'j0oPgwOKZ{Es'BlǮ49|$yzQH"I2Rqi  ͱ XpxҞ'B==-j> c$:9 8gz3Q,x_|fEfβy}·;iKCS8Vzv O^T3M#b,jzPu_}q8nw *+)ڐy7,-!A8?T3mBF!d{28π[a^oАP{zgsii*P-5x8tK8> stream xڭWnF}W[ WF۩6N-e6I}P-n탰ٙsfA R YJy$d0h)dbI8M")I:KP ) 3OIԼH #G-Ȥ2ҒǺ!+1X@ў,P "jJ:4#'$YAN k#k; _d#%!FX21vR`DHRH;){) <4ӈDF"XNi,ǁeVaOA`S| 'NBHgT-Hg||fMg}ahȋ1eawo_s˻@Nf{5?-nCaޕ͓"_aKβj / VU]> b STemdr Yo&}A4oA3s߳:̳ e?&x5Mi_u>{?$+S1M/][mrQBmoEMI8h 58PyKWV &O: C^ӝ^t^v\}mq[Ḙ"}^T =&Lw+m'E;Ausl:{l\͇;b:;_̛,w\yށ4{G,la=؄gQf\,/~l:yB/>]x]}Yp+]nK0aֻN}[=Imu7O/yu jGo })=6]p=V_V꼬ooR.u(WWO1V)ܒ-khY+:ǗMvT>U /:ޭ-w;sâ(u}ߍh%>z[\ǟ \s4x_x4FˁK?h1%j]nZ`|ƇaG`|zʿ7wm[qkTaPx\M'[ķgFzft!'~O'u[A&/'iq.qzvZB㧬)H?(,\O84^Oׯڡ U4A8A/,?cТٻ~^ ŗᱝ?H endstream endobj 248 0 obj << /Length 1194 /Filter /FlateDecode >> stream xMs6܊gWM7idfgv;=[q4N7~%`cHf$9ـGO9 9'~~|BOgz>LW4MJ-C-r<>i'^GPQ2y:Ea.y,̝(\JVK.m1}LonȺ]*1V}~nM?qP3F2I3+Yަs=YJ #;)g2+vEҼj/:~a竇 ]۷ aM—QG3Dk,oj9J2šYiٽE {!1b㟫8fwTױo^拆~њs K qlf3 j=(6>FRo]Q+ 7g%@GH6 <ռ$X'LIlc)}2;ݾYY[稝h.訋%̯L6յ|OG 8 -ZVoqv1zφ_ ʬv"‚l@"kWk"{iWPoDV2|E¯KS@2p*Ws54Ms #Fp+6\OWU-}-Bux|L ˫@*3ϩ*4`$VUrځJ-*^$2/d]ۄ:RvGEcLjwv4x͍nv;l c_ 9UPkl 4Ld k>V{;f5T.{Ja?ޥTA E: q@ MUd@y%kyᔬ͹4 ]UJxN9Ξo`lSܮt-8R[ /ulS7J4򮛚Yq-})>j_|YWNe'z ^OA ~d*ͻ ΅u Ũ]ۡݏf~ BhgfOӓ  endstream endobj 256 0 obj << /Length 178 /Filter /FlateDecode >> stream x= 0὿tyi& ~q*i-!Z 8pw4@ek@2PKh$c!\"kklϹ,G~tØKo\ZacO~PIE11TᷟN~$թI5 e;Y2/}ӳUȞMLm endstream endobj 267 0 obj << /Length 3310 /Filter /FlateDecode >> stream xڽZKW.mEArp\g|mNj9"΂V`zN~}TF=sB/RW|so׻/6+7BTrswh*e\e_ ymujoRO}7@R&}Uu{O-U{̡ޏu;J^ lSR^Ч E8J AS%s&ԏuoi*}׎݀5T8e0=6v(Jƹ,fefHzݒ Fnϒ[WzLoSo2Pf+R^N;;Hr>whW/nW ~?PԠ3Zcn{r"we|lgi pu;`J\GӒ=~z* . :h:b #5L]P$sg m7RaĠ֋V93;/WnB+=B gkNf|ß",s_ٔ@IVOVURVc[s;/-;Ilg:}}FUAv<ЋeAa: ĎWchɤT ֚7m(@s'V}XRQHigӃ iD$ 7/lc42;sC?Nf9DcOr,rtǪ( dž㪘 4B$qƊt bZZ_'>+>hL͓=.͂A7_T5c}U 0ËJz|v[7@=Ի9y?9J6n眢C;nSh}k3SxJ&a}  d|ĸ&]kA; s]Dm`@Xݚd6.Ou~ @IXDts&2v@S&1{aпX Q$6{V4$zT7m[+bP0 @;B=JXc>-pI-7MLP21$RT Ri+Fʠ'.B lY'Fʧzkjvkۺs"aw3[%ӭ0:nbu{o混w_S[ k .%s,q'Yu %YTJ(3Xʩ[5CGw.JE Sco@]+_wX0t'O!өs;0ZTAn-Bƶ^oChjb?hXB Vs`]f( LKNIXBӂ IA@Ǐ6{C`:3UlƝ{ CkA g2*1?jp}~y|RL,Id)Kbe{Վ@{ѐцءd)%*;GjzA2*GX M~EX̳f:b@RH"7qy㸮?Qљk)22r|h0C4QLٗtOu`n C3؎I/-ɓZ N=v<.b\N+ s6+[JAC@ñJLݙ;B<B:iErڅcEkLiC?PRtT p(Քyt }_=Sr5>zN8= ܎4,ȖD87 BŒYk,,RjlxRLxqS\rY# Voy!|YKdxͣ*+lպ{Aibso:q(?L1T2GK3rXe¼3a q˫8+AV4J4CLc}Zs BluӋCX/%f^{5uD=>w"!c@GmӁ\pxдv5=#ۢs:ٛ/@Ao''P⿌Bz" n1=bEt ]fPImO=U;!`!wdʉSI%~x0%.VK+e5/e5эQ Y;'F $ht]1JtmQ0t2åitj0ÛkZ}_iL2!!VS@RCGT 6n&-mB<+.+Q/HjL`ՊxC2\N֠hQϝ{_qd=,_{$XWTpDe1&zW5/V]I$Ku;~U}/ AfJl9&JG ,7_J>^!r%8SlՐ{6 n}wTܢb-㖘k&@87/ ~[.ωI^rBDSJz J FRdflϧ㧨$iR8/J"] Ϩ:P,yRyXkρgvAIyP'rj>KgW^hMNZ/a h VF^-z;qvC]w 1|0lm.`KX_R2Ə!bgeU CF P_!ɦm[[p XZL/é%)y2\A7boy`x\h߹.8Pc\Ƨ+~i`bAX\Ϳ6§J3_3]Y&+X'\ *@6N7+Xxh xx={@NC] P}uE$Q׷!ۤJ_ƳWֻ8K jI$DxL؟N\3LNCV,ޓ=ƧI,|;T?ܤ:2j="r;{uihDI}DrR0>ae }~qxG7i ̰* eŠ}_Ζs endstream endobj 278 0 obj << /Length 408 /Filter /FlateDecode >> stream xڝ=0.(hQF(Lc-98w~=:YvWxwf~ rvGC=2)J}ߺrP Eℼa&_ }%%XWӿ[[*ufEb2 ] tH48{¶Hp!W6IRG\T)F} K)S!;r-77n>,P58N"Le Pe'/מp.jBjJ⟇k n!<[ݪޚf,XW9si{?CkXbP"V1Xmv?g- endstream endobj 291 0 obj << /Length 3827 /Filter /FlateDecode >> stream x]I6W2q `&\F%Jhyx߇z'UM.+1 Yŝ[{qWW>N{ %RDTޯ}n8|?$8bQ̔- 2?.]fDv>dh 4NYqHi8q/T3r89Q^ц:ϴyw[fa6H,{Uzrϋm>vq6EQѰj_hR븎^5t}JӲOO.f8DjlH5eŜA_U}ꕩt?¾3ʯ$7}s0& Q4Tfn ^u\7p'?~hE K"P^MU~*0Јjlh/~K@;av}^<2i"*`~gP]tBMi)MF{ vŏ{%.pݷî]_j\_~+7 [c?\uD ds>MŞXtM۵j&QwM%eצd>d+K;m.q X1+*R- E,D혔8]W-3MommST /tUq )NI M{]uCR@y(Z@5E0^'4ʂX긪Y$cHO|WŞhl+'L+~ޣT\_ : 㒮zǩh]?Z$IHі^ v٬rX >T!\Wp%*1MAUDT} S?|j}}ma3{PpƜo!CPCeӅ QoGr'm((d{s'vedqj2F/q,Nl*$5ã բhJêR36Tە!6*[ЙZs(-+=)_Uj?7 [NVj8lrS;t?vx'>I;x\V|=g;9`"R2-yd;^O1foE/>CwJ(L!, 1@`FTG{-ԕCO\"*C(a0ƈ{R!sr##bg'rv zثJD졗:ҮD,\)S/e5V`kC R6dz]fMzenmv@S]*nXB*bw%"6ꍐ ~;!%py8a!1F)jqT## =/HlsX*.|i*tY!ԙ!thO4 ['JU:M1VxKEE#WE%h!ޝn?uJ? n^"Ds"B䓒#ߓz`RR&:!JD]!j+$}ET1=b5%S2^vUnze7"VSE\%Zh Ǟf_.?+z~3j >S>qB +cK< ęV#X*UP# ` 0ןwAK1Ɋ"WjgNJILڤosc\,߀̀R)wrB -%,Եͯ$!"<"e"N,߈XA%@,ɪ?Xr*wצϦ8}w,. I$HUr7[ReRנDqi]q*M&O siF"[P:$w&CwoBEwz#VwwBԮۘκ`(xguu I8GIc}ڳoqh*:XQr3C$NP`bT4[:?ʖ;.Dw2tvz R}l'H_0L#sENg("gż#l ;_R'c}8[54qA%I8]Z# "p慕DK]wz9 ]z1N,tlg#8Z=gvT<3ײɥzMe=5ˆi# )ܠ cIW/vq0BӺ oLQ}Cs*@ЁX|ǦBK \k(0 c܇YS.[#h21 ]0,-59iWz{;~k"*Z~.' FǢ3cڳ_a&:oyi`ly26ؿ!M8nl8suƺgo-:z# AxU4z;[ϑJ i[ uM|4w?Gi ]~< YzOdSf$D3D &/M5ŃCy"8)enH z =G8]x9Oe ~G8~qz)!0?vBaQ';JNP( 7Fq0G-M82 ޚ=SB7Ӝwtݒ9yJ}'#;*$z;OwG;] ~j endstream endobj 252 0 obj << /Type /ObjStm /N 100 /First 912 /Length 2438 /Filter /FlateDecode >> stream xZ]T}_aݮ*WٖJ $ aaGL=率/mxh[US֖J"@ɇ4ԃ{GNKӓTqtFw۩$14Hn&1*h7j6W;i|3*$kIV:baj2k/& dd00NK4vz:R5;ʪN`[!6R͓sZ$7Z{l$o]w'}prUR)qaSke$LNGOF ASLLE&e e>L{jUH͹Cr68zms4zr;?40U7.\ t:'^KNAM)>ap40Qt`i6tZhZTz_Ս'XR7d,4T , ֏  Zeg+eа>w9{=ݸ=a3Bj\+i7vGI'{F}?8yyųq,??<{BDؚbwgxeͶgmNy2ɔ'SLy2ɔ'SLy:锧SNy:锧SNy:gSMy6ٔgSMy6ٔgSMyuʫS^W/=}U.?||~Oͳ?XT&OwzL <=n 4Ҥѹ7jn-{]q#xi?&@E O iŲ t{ѳ+Kkʌs(l;S; dK"e6>7@%y`W¢s!d~nE=LruL0#(H$Wse{Id鄔Qqk3*yZA]m-z@W10}0?6<.YEtpN˹78#ՀϣT -҆ S0 fQ2ߒ-9A#QtY@"C>P<S;XqH; 67 :V4vl 3[fƬE~ÇS8 C.gLYq>#I/hQHsC?>ǖaB`66֋B'zfIO_ FNqwN w2wX,u_梢_={xwwKV<ޫ6yjS-#SSq37\dBM#0 ՉP]qb ab _wgg,f,f,f,f,6UdB7?MjC8RTŪvγ܁#r, VdG_E_SR>9;ru$7F,aŗ `[nT.ΗUu8 !3 rD_*RDa#vʛ:,TW@sW.8,(W|/D+_aY0Bn{:GWVO֌sg<'I)}RJŎ>C>/vy-)!0GШȮX OO?O~9y/}~JuMޠB~n6.8vu[/֥HxTiױ'd0u6(] JM1%LrdQ>83Xy:(*8z}vXP`Njl2]k]Q׍U_WEBN.J*uD,zc=J_9/֓t\f|ɭ0Y)SR.m\9C_s endstream endobj 380 0 obj << /Length 4212 /Filter /FlateDecode >> stream x][s~_$rgm6ɦiIܗ\CK1EjH5/pR EɐY3 M @t7G_TݏͫOR>a6 x~@G7omAȣ#J+Y5A# PGkxyuqii}qdZBt Q KH'qCZC)Y ϜG]h<4qRh@?ԆEnNaB&-!- * ib]hr3fZɡ}耷%T<zZ/k6=ݬ0Xcn&F w Xv8&VYT1l$q3M,x_!& 88ZMIK RO9isC9vh6Q,jie|+^݀7E f]Q7TB#z޴7U~+YAZ5"bKޮb**K3-}4'n./tBϤTq&b W2ռW30sC˛j]ʎr nƏn%ZMM s2+FhyeܾÁH“UK#[Z\@dF)"1g*ˁtܝvt5OrҭgMS~!#Bf[.b.Xn^WD3B z~4XeTyioJ<Ά"|2{!{id <uedF~c4PB idުH_CuLbuC}5b/6}CKǘ|тP?7lJkZf2yg`G `'kv+0R׮T7)$Uj<C/6.wƆÍ.oZcc^Ryĥ(hm$ٺPj TnʴQj?aSH'O*܎r1+)TpxHux!/fY/W]eH꤬^+0BJLȃ' oŻJnfN]a,IT,m]iL.{Q$h e2M'uzNNjbxWB: Ԙ%Ir5We>EvU2?]gu3Uh^:e*i;Xba!X ϒ2'7Χ򆍧$ꤪ]XToI2MyELU bhJ3 r$& d:L}A(#QRQܨ ;Zmhj4#汐*4̰GP⃀= ڇ/ZS\XH.V8n5$6efmzI-t>˲$@m\ޭI^M?G6˦qtDg|t{Kv2O&!>M䅤g.@-\>=aL5K qHy_̏IQɤ~KK&B.Är7e"8}nɎ< ;R9ّ l .A`(WWbr%`id#ouhږ 4<ۍ2oDJm?BB`QT$(ᾭh˥Ȟ+د 7{A[ V[ ΫJm*M{?޺LvwA#=׹)ޢz~t9oF짰-C0)&GK |I1cs{^|p(ۑi< )xL`< L uc;jR"ؚ-ñ!MS8#q:rvC9ZiV8u$"a6{Ս#.D3jьyρfB3/i`@Ƙ!($r H`\<1̠Mq@GU;쪷7nB[CT8KIS t!. :<ɖc"Y9>ge@N-:nseQ8'p 4źBt}}`k }N[S2඀ҷ~ [S&CR =1Ah(3ݩm9т8UʅhP`y_U&͊w ,Az}+ 5l%Kk^"G1}j <=Yd ˟ 5G@/أ[;4gfgCssl4󔆉~⤗;sxnN=<{M6 *@?lJ1۴-t`P"5$Ѷ-5VjۗdXw2^jtD(y<h#2>P;~TqlJ`lwW٩lIA\ICQ#uQnD+sNvy,۩g^:- :-R,_w羀cpnP }l\gL(?{:{ nڽެ_\\\]wWyRvB>BMcmUX ɡl,0M(l 66Tؘ뎢l3j칰ߜS8+`\luv3CH=>߹2%KCn=oD-8Qt I~\BWkBRM#+?<%G運=I7ՓiBx$ou.1v jYdA'EŲ#:qGU0UfÐgdFa|B95><7A_9Sye羿daf|3 3=eWz:Ya,96'ׅ ss ~7?T?g_sbbqfsf/A]AA!}Gx(˛W,= endstream endobj 350 0 obj << /Type /ObjStm /N 100 /First 873 /Length 1807 /Filter /FlateDecode >> stream xY[o\~_!\FFHԗ:&0furSÊh|gLJb$SkZ"{"ĵWJ V]\Ċu[pUZyr-\OM%A)yt@9yKj fM;-ɞ|+%`TT<4'X UbyC Sm F iڶ뫥j*)S5U /X=Un @RQ,o}l ztU3V+ɂ͓T- ޓhKsra{;0.&WU1S@QZM ;p&0aC(]`IV/}l1ε.F*I B{ -Y<@ d0LR·ւF0 Ox>_Ʉ.>ھ?z>_,6V[SѫY+¸ooQDxn(JG ~ gi&!zNOlk{R橹 +mcg,%PFA CfTmKq:+'pШwzb{Pux*%G;QbB\b݃|y(#6^7$vrt!{=yS䆒j/ƏzWJ Is82DE>9yr>+)hAbG2^JNNg%cis/O}ㅲ苳^w.69I'ϕo܇9G^<+'>+) #UQ^G?:bET(zO-5Rn(rj`E@'z?tCj=HɋOOg&B>cQ(F聫ߏf (qӇ9 4䚣/ͩ̚{ y\ܢI#U}}ty9o$ ^j;EŋT.HHqCrAJ*{K;N5xra;=x(X> stream xے۶_60&2L/D[HjM|{)B 7`!@ |Kw?k|W>I'ɫDȘ16ƈɫuDqp ]\c'~w^G)! ZPpoxIa^o^>G5#QOPfϘn*qյsHԏ0Tfǵ;;y[M囡2S{C60mUPPN\B@bALG0@bb^9| `o83(_BY@C9;ui C r~6'I03Q (1@2 3WɷK趴 hɎJ5|]9wL]t0J; &u:E+G +8 L/3x2i`d ۖFB nNX9$ V@}Z+\G A>et!嗯 8+F͚gJJO,IVVޭ4V@\-]>>Nk(f De dަ }$;2p2^s1lnqcYxCV~Cc*B ?4^[t```i`6B+Ah k u+ėP t;-쇾gJy0_]kw|nq ~q}zd4NC20=(6jP,Dv&'OtY ٱ3ϩZBY ;Ix'`'(ñJ40,QvekIKJ}vɊMZˬ ׫n6<_Xp5BXͬi5\fMlkPTwEҪGQc5~hY@/t51s=TH0"z))]b?";Zش0S*55?̬Y-b4Ce}YztUUn#z)zɕf~U5;:sȶ܇x 0Ct0FjnzM8 *"Hb=PGn,[wВ@9R~/|X @TmZ?]= X lڍj+~8dyC+Oiē r9;d~ѐlsbjm/a#EE.,z wDm~z-zV ̛MxFtBY+( kdbWؕFPLDX ?Z{rًjotcW=Ky?'G0K\M,~G6hA&`}E/g}@u-}C+bK~>݅Asf=npA JHIOI-.WvxVVcqqԷ|;M? nq ߇'= n[SOJ>Zm;t߿'[ckJE=f~xcY]?uchck 1 3- `j-`}hj}42Bw`j#O(yVdhtX=MVҤ G0 U d;nC1PC[H]`FTPnwd$oZ|!r't;j$EFj~Ŷbt^Y_Aĝ]bOp;;췞9DBD` *%{E)&5`SJNRinOߎ$b%)'&Jx t6Ff6Ps ǍuxC#pFEwS/|d1ߐ]$Pqh:P7pg~U{rc۸nut5%>!%!إE' G_6_:8kr iƥPn&&H̥33u爳枥zeU65L7r˵KBYemә7u&*m^۰yj4P:fC/1| %TOMUa+IN;9U;avws|,7^ux'K:JڑIn~d-ˁN /DpwE۹<GgCb.![1G]t=gmo'ĐksI,mibD9B'@5sɮ>OQ}Z"|| x3^;> stream x][6~ϯl4 C&}8[ǩSS[j<gFe݊&vh4&EI;\y@&2So|'/KuS$E7J,QHL޼4 WMp*ƶ!I\%&=rri"a7IqUOw䓕yPַwY:yU! clWsM].&G{ʴ* 5^鴭ϩLr"P6֫&ߚδLyee i{H{I 0r2Uۤ]HSΡ\u%'d^6i>k/#̎lO}e0a1.g0C+KCwHBzZ׭f:-DR7îa]SaDĢ{&"+$&&:)ְx Z[6R!uo+b(e/j렅dEO"{(<= ). }r wH;KɋɿNx'Ô0eLiVOf2=Q:y ԮAF5YW٢JV.bY $Oc``oc4?KcNuz e\3[&Pj(C~tAA t8^|g5Gbo-=EofLvulOUਜaMY5jOn3GY 0(ݒ9~be,K+G+C,Z}7INz.`{`}aV{T5"Ɇ%`V;.C)Bi{jUQ>>.v#aM,t"^8Nl8_MSOpf]}`, QpW`a'"o15%*"hF(3@%X ]Ñ@D|7'<Q Β*` F-C;31V !Hyo+!")by4Ȓ xW'lo7ى֙$ /X9#٘M6K]=oc.uϷz;J(u\ZGCI*cHBHw R77`7Կn4M7~Ğ0ރY}.,i}FP:bT&!8"g'&9fÃstNNky,{ݿGkb^vqmRC:Ibĩ;d~ct/0AaCǡ!ƕ(đS/$uB7A%snhrO_s#R(-'/LV8Bd?+ɴA>yfpU'iFKgziIW:O..'# ԮH7(ӼM!s!N2ml vb.$ǒGfbX1pk ‰]",#Ӥ-+_[~7=j ^yy  vExI _9yv朲lZ4Ä1&[xn&o8"DBSm9{}t `-@G[>6i$'|({L|ibuFi7$-m_sII(u(?jp :㚕*3MMvE@) s=\Bk,8*ϕp:+)+iӉ"sI[GeL;utHq -7t1y~&Y#3yr%q\Jb3ϦMg{>֯?GrԑJQB$P>/8'8.U^9o[h x0598ynFjcț9iovx Ʀw)N 4/P:H~ӕ$ xt"?41$C35٦eז})Q"#}@2Ȓ$ЇlfJP>;<.0EeERs;7SLLљȘ賵On+ endstream endobj 443 0 obj << /Type /ObjStm /N 100 /First 881 /Length 1539 /Filter /FlateDecode >> stream xYnG}߯ޮ[_$ "aG!e!.Eq>ƻ[k\##-3stuꪱH%)h( DĎ`K\Vǵ$渞dLy$uJj@TZGzCcDDځ5]gjN ְV7 1}Q-Jt>"svr \ZW[&9X%s eG,cmн4(l=`0#.LQjͽ|ǫ7tw`q~.:^͟`;duvfŭ_o߽z:O~b/G1E_-+PNm}um_f3>[.'htɡOo0[ MC{K&@=?V ru~^<}m{r~t/މIkH!"Ẍa&ncNO7F0+ǻޥQX;"MU۬`HVrlN׎ t p "yti܃QJ N/,W/?r(D*])R[n{$֮/ÚR(UE-F#-8z `h+-'V(P|uFQ o1p-% N+¹( dz-H[+d^.feB_ey]_E -x&A^([)LT[KBsp 5\ 8Y"nQ\ \DE΃"G9 ȰjX9goT\z@䛂f pRa)d hҞ af/ICXF$!/S{܂'eCzČTLE7OX1?}|[7_V׭ۺnۺߡr"rWPHqHNN!(w5H(wn&ޘtY7MږyO7^MӬ1y:LUC->wi[jb~L"CN{2"4b6zZ۴WشESF7pWpp[7p[kj56M5UE*l[xAA z+ĹMbXH}`'՚VZ /!jWZ*HlVd43x endstream endobj 566 0 obj << /Length 1412 /Filter /FlateDecode >> stream xY[o6~ϯ0eUEjחkuC ˃8N8t;HJز08erKXv`ON]gaaýBg623,TYvsrkȾʳ >X*!LQ~WRHkf&C@>BHV{H7tdd;SlD~jeiu^`V}>J"p;j[.>D>CNv]u֢]0yjc)?YPY_rOk~%4*.&-kLcfS0ʿ$YFU->‰;gb|Yr'h}`yQGf(<.6ŏu(I(c 25u}tB0دlKغSG8c*6~0}+q9O];pc g !z)FxuM p W]$B*9ZnuuS[x wN,F c{śs-ޝ v #['<2aYGy`$ ;82) ^M_ 9lsu`A" gR1Z"3 Wm|K4{Ln T+dG=v_]CKm.IJ+/*Kk:M!J%K$cȧ%I֜=A=F CCS& b2~|2|P4Osiƃ& N#@I韧 {/y~U˔CaՅ.}{\`|p.󟰆#áSjJDS]{BN[8B0C@Y6| M%a{+,6ܝ]/믐]b3FէS '" C+{x'Ae /YnMFUөn`z[u1,6ǰaed(HcwW<5*?9[lޏ'WdU,Gjs%cԁ\1?8hSշU .nc7x\8!(,XB~L 琗MI26ӘhR no)0/+{.0k>n|S薝W/*jġKs cX]ך{V1ΝyfQȘ휌;MǶg2nJ?%]*KIq-n\-jy:G/|YUo1EWI&+OCN6Ȉn="wiBy)^`"pHQ]7;\G}~W/Of5G-MQr|oupo$ endstream endobj 586 0 obj << /Length 4248 /Filter /FlateDecode >> stream xks{~t %wCƮk;j2d(0" d|H>P =YC pϽ}"\"g&B.xpv1HA*x 98~ )(x9l6NNToU1^v/B:`Q#ͪ)B)z8<n-lO2EQ_Ft~g{HF,232ļX,27ؘci:1cUBR/ONEci֡2+v8a8ee~[5y0).鰼gÉ_g(NX05~z`;Gllʗ |X.seZ;;Efu,SN* c,ɩ[><+Ӭ^>Bf Ý&'YݓikpXj,*8`4r:X.nfe^_Gi*3:\ gvxM;:HU;>[u0*ne>",0]yu-DȢfԦ/|upjG"o"[0 Ha愖|f<\wut'MwN/.q6;cv#Ä UămVVc"XQ!=y6 mU>O Ȱ 8[jU P'Fn: eZCOI|'k$l w̞ 3g=ihFA]P7˚ Tw=^iǓ8i84gL&5lcAB9٠周3T5xui"Y|^Ces]:IiL $`ud4+a ;nH$ *EPVƝFoYı"S"1;+PYgTƅh]@m>/-톴"iCu罒Ёttl)S喑^|%3\5"suX:5~_:4;]`nzI [d%TxHj% e{73pg,]3#9"2d<≉LylDK9t˪""C*:Dfqjjov;Kz15-w+f$" +sg[ /5="HæZ_GMН1eCSk[H޲[ow5.iȎQi?Ta;k|Κ=$wp x=VBQ"z5$=@$X0x;<8 iIUOqwϷ1[mkZN\$.ҶKɜ~ :G4oxj9A-BɔѷHY͕EETt\جuW.l  S =^2F6mzm7J? k#} 37CZzfAz9tUCI?F2P\MsqXEf/o;OLTb|Fq|cIY5ecUtQ*̩5іrKkQ#9-Z!, FOP!̀e| h3D`m~7Hț#ے T*at' `>l:Q\{?PhAR^rF !ak[^kWYyabhmΑYS6-HD(4%g'b-erGv"RGt|: #A^dʌG7KL Qw |#Ysպ$|5(4uZF{nѸ!_thםdQ! -e8wŌ)-٬o^e!},/GNg\ɣKVxžK| "&j9oZ*Z⬤5RVF^Kr9DF%}dgo™#7q2$a}^A%}J+[yzh}U_d.b}|#Xsg(Obw΄*Ǝ߿T1E}T[}LI*0kߩ%rtȼ1!p32uŃdVu@94Nv:VkQ-5+?.v|Ox?WVbw!; 9??LvY(`f4`$lb}qe3M9KzKyUDPUyQWΨ,Nx]B袩p^=Gyd3wE{\[ѹz ]e++G.oI5;pAt AveP:}e 9bĜ:GD{ʢ""I*2hP;ng?VeiݚRZMN&w8A4ku㘢/$γSiͥYXz[m)"qĉE@x!/D DtKPO9PsT"WEbLGo"5cK_x᫐ iC|f8$MmFe&i Z(7) Z Wė&w k?Fl~ ĎsgngDe.qdHaC\`X6(d*jt_|[> stream xڭXˎ7Wx,pM@"Vw/^>}x[(S}9yz㮮}Ҩ> NbHF9"{ +AiV$-TF 5㹕[NUz kګ$XUMTI^W W/LM_4W%̏GKqͳߖ3K;c72qu" qqN{3ɡ~ѽi7bR c#`p<֌*1Eژp2EmJ=8ڶ*G.^y|fIp^enP[-,YmRIsh*/sX(I^I^$/jz5Z.E¶[x$/kU“MI߻尨pPT Y+iIڞ8F55mVn,IWzJ)~s~EW$z~EǨ=T=~%z~$J2'UaIcI#,=z䭎EI#TבT=b W endstream endobj 645 0 obj << /Length 4207 /Filter /FlateDecode >> stream x]{ߟB(g[<\2m qҴ1Z nZĜ^tg;E;;! +ZcvvEQ4z(ҟJfdzGc1,(!|$8 #^OGo]hƣ*U]l'v!I( 'FX??&$b=8;:z3N8 _Y.< */i(6JQY^NҠ`+,+,XXn,m 2H6bOT`'B8q 5uHqK-QD{^pl. +,j+^D+-3,WXN!>DS?5QsGBֶ=Y&![#sΝ[ut3꺴K+t!K O}֍u&\TW 5++TW la#spSd/+*PmǑ'ވKD=3T,|)ˍV^kԙkFnL[{6df. 1JL3YyB ,Lp^ZRCU^϶F $-42nIrUm$35@QY\FÌјҐ7+iP/(WEU,'(t+UdX\WngvM1/gT}]Njy Yud{;^-$abcB4fAb3ƒ-/z$yRE.+亨VBݴC@QEZnUݨjE-bՕ uM~w&o PsO6RKBCwITw,$cEQLY>ٰ)MBU%oHoէkULo'jŠSr\^B5 vrjjHrEXD|v-ƶ裲$x7+ڑ`b-tWȈig)빮ȷU^]mKz_WBѕ.]U o1RkgU:BXhò$xyn[QخT]#r`EnXQ}[V|i ua/wp^}g#Dj$ew̗ʝ-`UZ6F.2$IW*Ibj=RHu%Rҕ6zbal 6&ڣo_*a%RYf>*0hc_CI&඙IcKMMc=F4kL7@[J  IC f]e'AJ*gc[t}g dW+&!FLۗQ豣`i\t4n'I˵kohh8Q% Kw ?{l(Y[zƛTC|{_OI0#*w9Tz0"`XRDŸc<§b<たoȡ$T}d1S`JuPKՠZ (opS-R?)4OoN+-'/:=Y] 9Ryo5P$7%5z"{igD/! ~5h:m5d3?omdAp\`u~U`S-:Zk?PJ֐aE%7`^!9H2r8Pn-P2zP=/7 .aZ_(|MKVX^jt,dySZ ;sJN}r$ϨOh\qHqVp0}UEDF`] B҃Qmn wF)+uRzp $-W#/j) egE#)Aeo|mؘ5ޮA`Ұ Hį:z{AQ2LN+kR9Tf –~[,ax;R_ OԒhh &!up`qi7+r U)$Dʰউ0FoXVK\ֶD#i/0忡$oRE=$rhT3*գY+wC.Ts},6 ?&t54fXRF-sA"ʒ] 0paZ W_ra$&SH#=>O -2XX`IO֎i%CVU^ۼ_7zW;~^+zځ=0ɉ0Ԧ%E JAS_ 9m˵)l̤}֗9Aҹ)ہ s47~TS-9DmVy,[=B9ޤƐ>Ύչak7 _&rAf:0ޙͿp掩EKޓ6pN0gsO_e 2 >k p]ݩxw }2 JmX<՜dgknJf>(fcSVLO32Jߡ(unI>VjF!K&)\:|PYR8:&dvd)1`$X7%ىP2b#rw:wd+|ų&pC3E5Kv Ӡ<(ʆ>ڼX8F#PU0d߽y]y<UDk7V_8:TsūTL)98j*d ,f6?Rl10u|mnxb]%\㈔6k{LXo@y:~هEGD#X8KbeJ(x'9CYFRΝ#etjoŅeB=tS_ឥU:huɢ3yة4{S&)9*”ڸv]߱@RkIGTfmM_臞X,.d҉q\@AA_ مT},B?0y-|Vv0Yh'Lxգ`L8vD#.w2]|etx{#Q&Û2 *-ָdԋ23]q?w\cqg͚6Dzq8g',o4D3+NDLPn`0,CG?d>a'D+ L?PdP,5R7y; hD% FD ɃF.wX#O>5RU<Ř7հL̛sj8e =-Ӡ^!@"{hϑ gcRϒJ#ux0y+qrǎ8y D F&MlMPgYSa=}~e'2Ќ%sRkVeߖ&)uڝe HC&Yiy ^?/o endstream endobj 695 0 obj << /Length 3770 /Filter /FlateDecode >> stream xioy.N& TH@(ri+$%Y(cfgyHEr< '1\Oz\ "Ӂ9cl ニuۋ׀W`n~-TLgӳ!lP ^Yy6eMek0ǃ/kRgKH+la-$ :f[ھ*q k-" Cs+AZCؗXokƑÉU½\3Z8WHE 3  /tH;< sٹ)IK+ X؀P g^ߚ?W=O<|e^9ZvDa%yiɟptv/(yv-j~8*T@wi?!v AמּlQѫA`?NI)rtaӮR[ V {;L0S 6ᦳ-EbT:Rn bp^XK {+ҢLȒjw}lc<\g襄xT^J A QE89O|0z6&xVʁ98U~qLtTٷf\vjia8H |.X9b+G½y9[&[Npd6E~Yv3sM孇nߏrd17bsc@08BG;w/wjR~ ?wz?XC;Jvrt|bF9ּVn96OEj9b~kxn^2(h( vjH =~nqJ5 u+TՊK LjmnC,\:/%*`*טV QZCo%6^! )a D/1?O#x_S\IцSɐrT ӑI&K %:BW@k*S/,#`?ͪF6H "wES!%7_p3:m1tiMPTD>i砖If \"Eٷq|PH냼. t pK_&. GMh45*2ůUlp"!5oGUT$" D.6M-G"8Ddm $ XlQ.k]lIRȚe.5:bZ$DC 5)Xv3+($Ld3Oz 80EhYѓO~=be9 n}x-LC0boO~[m'[B4b-I3 W}HZ))(dK,R@t*#FIm&!WZCEa qN٣Z,vֹw,,~JABzZ0e:YkN5蟒n߷T^ǡQHt%`559!H1pP!@0%kofxA7./7u '6cHv$-˕n 89d5?:)5GrLdMe?g3ύM{aƧ *Ze RcOb_e B e2$j&NuiH2ʍhh:&m2ޯ?Q`}B))*0_&(d{ݔY&1 F!<M R )2fnzqX2 8O(`;|20LkC yl͵sgM1.Sm_OmW\`;6P0jEM-c(xϕ|(u{ˈ3\xFgqqSROƶi..+eU9$ۖ[ UO!(Dm c0+5fF|e:UǤgyRqTNjڨu]T0pJdʨ!iאb[+Y jQBqO߇5W ?ۤ C }GT oM%/&x^uF5*j_8gou_f($rYgJd۹ Kp.>GK:LP뿾&جkNt2|n"{r5!#06GxVϗhhp5'tMSr0|/#f![X@:c>$RO?56hؾ^d8,'E(Euٲ(&Sp5|djVݬL+jS}\3%nK$T1q1ԟrSq[S,(vդ/ qYuu|4Iև\ S@fM$:}kYTyrK Dc'/`PE@ZQ?ÒP⾈s"2V%*9WaocWqO$B]]ƣˠyPzq4725?,Mʍs?mjh[ub+CVO5pQy=#؇CaĈcMp@=<37aw8ݮ[/~V,{&.u7܈JRf(4_2_W)vWewZEK#yZ7ߐ9Qs}=5rZ;/9.GrIa@ylb^7/ ;_~~C}W`-ba'25iM US`I/n}Sx/gEg6$l,Wap)ٮ,O ?0rMٌ8ͮ'F2}OkCʿI]+"Ӊ*ew )Kl!Uw4ZDt*~k9zdX2 `G h֐p.(;@'W7q| r)$R 2#fXrj aQ.Ƨ@:yХf(.琪ң>}:Vm̴cF'7Mqg0|1# &DJ9ߐ BJHS+!+PGnrjK:`F&'G:}64XHt'3ú-2p,~O]'il!~ܩ ؀hYfPs}=,ݓX9! #܎L )oބn yPwq'ԛЇ&$ogghHyz(Ef4e~% endstream endobj 718 0 obj << /Length 3978 /Filter /FlateDecode >> stream x\ms6_rU" M6q;i\0DE"A"IއD/X^?9;yDHcMFg#EG3F/7 dz:^iZ̵O|ugrUW\,<Ó#nG$lVvH2s&~$ ]9G&0cc4AGC]ć YaCO}BV?džK{좊Jp ޸""F7+vk )ccpڟƖPdG,mvK +0KdS ApKF7 eDLJC%hrdӺ  [Mj&6Mlk WQSDUt5_C/L(*6aͭ kA2Mir'Ô;9rm\q&r3nIL~ZEoa@,2to±[KyXd 6z:_rߦeUer1wDс5U"KspQEyi/ʩqV #Y*[LHݟPX:w>)ؖ3Bh_'3.*g+O [%E@.q]cI_Эk3$mR>{>PA%@CZ6;y|vd9Q*Hbt~%G`uol׹u1z~;Cjȏ%4FOulQZMA2(nأ "E*:lw-ІamB8$A7 6Ƽzӿ^6+e}{U;plhĥUm%NUACmclv+$( ;0D"vb#N(7 '0P(@PP=<(8(9:g 1D% u>1]W*Wi<#k(_gh~̟8/|MɁ,dz$sPxܤɊ=]7 h;S#=C5Cʑa/DQ7U60:[3{z:ҸΞ*_¬gAl КLd 4cjh E̖+׺b]^绸0i-hm7A+N/.j!R:ⲣJ &Y{px۔^٧]$N&ld ;\[] EIe#k[H2^ldۖ9@pUϳtyU8r h49\(|7;9H/3ךWT5:- Z=H˭UE2M8wKljI J6ei"IBmvZMs#'-%֎G:?ׅ:VWݺF *H'tl^5$ R5) bhJ`C&VH"skhRHyV-Ƥo;œ3uȱ! H^H~ajsȕ oh2z&Nڰ-|S2Gցܗ Lh_9r'! @"3X,qPj L-j醪{jFs:)1NBh Ɗg+يMXިxPzGO/p)xD7\,T += Cl3=L($:ƾY.Lö*cwP"f>c䬕N@S&WhH%iLk=?J.jtX(`){œp'§E<쵉 ZL:kAi;QuN՗ƫR^Ф{Y_3bC¨7ꍢVT 'sɜ`KQk{29zʄCX(I*Ӣڎ?5mߤJ^ǵ|[.鵿s|R|lܵv.bUU) }0i*s=OAiOH6spǧ"K+S9ˊe< TWΞVƻܧVӕe4ˎAj8y::x}oowR:\b<ĻrpW6as'DQ"uBu.> stream x\[6~оT,wu*[l.UqRGC"ee.3l\` A\/2[g~uoQ$뛙f3-xH]foeV՜I,.Ֆiޥ;|bA}<%ͳ۲\UW QrFtkbPFl6ܶ\+>hd4a,^a *?I|!]7YPDIGnh}38'GQp!݀ɮrm2Kë9*dWg&OvW9FYegmMjyOaLq<"c"#t_;FLy7H G31٘=\#Poc;$;tc}!d}ӎNmni38>Бm~x-v?jc&+a~3#2=q1҉zd1 s2t4`ejhqid?H!eT 6YYj|]vc+x'|UwJ" wnܽd&ad>V2mb _p\uʦXyԡP'wd<9QT Hv\TRd zsOW M TC >K䝬~$ e0|׵$=Qӕ~EO+s%X#_ >N!#{Rbu^A@ _J0'.sⷂ9۠})3p8ö2=t>+6.;@GtoB>'L,@mokpSn((7eVɀY(:_Ƀ|=bI٥nIlҪJ֮6XE{ E/8wxgHf<#D"b JmdԴCX qX؟Ff `+y2K]Ӆ;I\_NMxTvL͹f FaFW6Ӂd J6EnVe2oVI>U7MpƷ."`͊/Wo00!Wf3geDjS"B6"z`I{'ayvb?Ⱥ&C1QcW  7htݘH&3olDcr ;3QHNLmK⻤ NhBL&QJH|h.{FKmFCa~h9>"]~*7̀5L9l[nevF>+!#1߯MV=Podv%mC"KWܷm :@4hh>WgBu&+ I$h2Px+RxmKr0HN.Ԙac69\8m&Krk+4qs?!JtG^gu7x9qw;WnlnVSF N9N` , \+1}sYO4mLuόK1F.y՘ IM7 vPԿHyL}Irۛty˞jf: .dhZh:rN 4bFeL={p'DZil^G<:)}%[&`vvn~Nb0PZ#^s׵ܺH׍iv.1o?9s ?.Q\ rDiC?}u,IǠpᓅ jzdFrя?X]`CL=n}) =aˆqpGN4G_PUz2~nt%"Ӭ׋ 2 X'M[`m30OG Ox]g t*h@[e~K543nS xxBGfWbf}q{ Ms!>wOpW?1b|M cq pF&PXHPړz?цqxqL=+vo?C*No`T,7DGGTUi1|8GG8p_Y_qDy|VAwq1to@`j~gS7ۻi2}>?Og'(0%9$O$pq`(\p"[m$דJc&(f v /R^RAQyTiRնX}hs XlA24>p)r[;ym4> stream xcIe YjcEHM$ #22f`/)1HD"3AB-)i?x?EXO0LKIlAJɢPIaL)i)& ELzR9Y]q15䏣 0P]l:ۥ'TFi"=h]Lœa C*T&Bf,NHP{CW14(˳Ԏc2y[ų/=b.8t|z{d,Qfn(Lm}s[IfG)F1l-~`S,Q٢Ot-:T1LaTg'1uE1%SQnj eMXƾk2Y7A-KLO.)ɀ MqtobY@$M~5Rj_b"B/d0PBfPk\$<2 !zqB:EQg3pmj+p Hp IISAe<..UH;pFm!ofk]Z<D줁M7}Yl7>G]I`&S PAPADBI o+Ly@Vg\xMǽaZDOۖƕ3lU$`},a卺Ɲ,xO9Y_̺Վp 9A?àML WT+/`r"KrM|q,=#9- =YqR_Pntb-''BH5t1Oy4r֧h%a;pizwnTR=}[5ez >k#w5푸&G%Dc>~sߔp3[*, 5T-F,(8ǁ𧋃LO#Om@ Ϳ4BDiJQ!.!4d$ R28ȯ0_  `r>ʝɌ5 Xl\9Љi9:vyY.<]kyF֛bھҍrӰ}ywon9[Lmo[ ۢ)uqQkh Ȍ/s|e[t6ߌ}X`ۜ ܮf|fL !]eյщY`(bKpQwÄbm>#6xڶ~U7o)@1A2 .VE2qaМ| ݟl@>"B1CJm",|>Av>Z}l$+./f.!ȨEoG$ z(w5XIS[g $>>7{J1l>M j7v[<J#Enͦg}umpZD_fEDD=rjkC-<@Cv:pҰ |m}o)9Wa}zx}7mH0yȁÄTzEdDQj O -?S1B$L(k&o˵,>Sn\~dIvCvu+d/nxm(P*L5T`G֖0kYf18)`!S&v[;~h+UVa>Phd$2o~;\OFT8nG xf GdS=tY߹.AJ'MO]ڮC23TwШMKN߂8BG#h n RcB<蓒<Ji/d3K:s4x\a<5Acc_XA/6ʟyz2v bJ~=ĖV⍺ @!e3#",TRR|g.9 V9MIvǵL baICT ^$B(v@1y8DM,d<~*ϻ3Sǎ]ygؼy!I1.07iSGxjX{7~tpÅCt[UGgO(L$AfK>2Iɮ*)ESEr.yijWWgs3N:ċ 02QwX.WL-A&~=W `Y@5e>Hg,:،7jccjjrhy.gzx.8G2bF=:@?[4;ĽTq.cHCH)KC?Rk#Re@밚a҂y}*UU&E}M[t$ sA|"M`0\.<{>Rڙی_ nL />(dŲw -@-Vq砛ɑ]a c"Ȳ>6rY͘;5]-I W5) &1R} נF ɁA ve?b-^(b)[ KkSMa;¦j )ξy 8e',M!h'əѴ= [ LrD%[޿MP:L6'$!x.ug̩dlMXUeGuw׺6h;X<2>Klۺommo ۭ4* _oژ@ ؈F''G\60Oڶ r0ЬOB\A5zy@Ǔ'kWQQEJN٤ @[#n,p'hu>;dl,N6r]C4o:M$=%2;*wBToݳsO7hÁ( '%3gE$h"^<^$ɜC _YHt 6[UВ}mݡ#&L"c8#jl} mR*yK$OQ<ϠډZxFpՇIM =i¶װ[4GbeNBlق@D˄y\sqQ7Ŧ~?3w{ɋFLxw/!s絕?aps5Pω s/{gI]`r?E xC1HwK#_orG^Wu@Ė{@ϛ Iv:3]norDw'\|"N]oM;{$V=6bvWTEo] |?ֵqW K?揋k!V ONؤf(ʃ'REk:"90 WmjuiB͍=j2"wIRhC>?5gfֽtę:, M/b@>I^_ ,}N{'5GaO1<⦝c^ *j0 rEV4ߡM:X*~r5q%=L9I:^cUb7IqyŊ$Ʒ׶a ^K(R3ב@%٫eyb%{d0G~Onlt~Mgvʣ@1?Ogж(#ت?`}Nld6}Vug_UIh}D7f_xlՌ-yz̆϶n:L &mg+I]c0M1_{oҰ endstream endobj 637 0 obj << /Type /ObjStm /N 100 /First 881 /Length 2099 /Filter /FlateDecode >> stream xZ]o}ׯCr> @m\ƅlb]Z_InÍ֑9vjg8M1D{\hHȄ %1*R*]wg5O&'EφPO"c؎L 4!i' P F8@XS1H, JI1U&GL##*¸(F 5:QAVhX X3p@W__ ~51ya8D> < 3j 3 3S0tiqYu_H\ G#Ωp֏0acq`ˆ_я" !  O`oր%5 33`)^/^裀,;xŸTSXxX,:З)`DpH6$P F tUa(̠V80фY] fqx3^XD] FZ'LGrQ_^$]_1W`äOnev,Ql3?oM [ϯf?(Ȫ=kds.dA7_}h< w4A^ߨ_?# E79jW4_M\Q#th%\5ZLeO?ͱSJS>4Or/P=.gջW7T5GW|-޵~j:i~n^5?+;Qe'X%o SS 9pKȽ|d7}3>9yO~:?}ۉR{R6[py#ߩoNSC&T +` wok!-3,y'[e \J:]KkYs]d9 m|x9PqT'BJ(JցâH味Uyl꼰Z|3J*A@#$ukd=Gq\'Q|%ną;iV%+jW%KTETBrÕV\%nrVB%.MJ]x׹*W‚fC 9R,p3pVۀ*لDUIKZC%.2Hl#r`-? F8^J~?C󢟺9juJ}sZ]~4?~-l\/Ų]]O'^{8uZFnd8q%էf(Oε螮g<-<~ay l"mf$)tx>jɴ[ĽUy6Ϊrz655IU]?lS SkE>5)ǾR>͹WEQa&hs@s}]vًd-+;.˫fhsa|Ys`;\9eJ'Giם/|z麟v菔^>R벓kȩoɂ׵x gz`Zk?SyMvt8smϾC?f:A2K܍Jzk|卨m0 TiG_{#ס^dw՚&s[N<3Iۙr3с^>ކlݭ R}V9 ?7mJ6%<7mʐi8uy"9i.a_Mݼ]t{];0ǝPG+QD0BRG{| G endstream endobj 746 0 obj << /Length 1491 /Filter /FlateDecode >> stream xڝXKo6WV|IЦqI EP4=0+zF+-7CW+m 9q̬iIh O..2ad]x TfmRDRQϓgɊ\ȵAL͟* @8rufי4Cyjk Akq=ࡽC;}7}Л99^*j3Q!*q`D\Qł8p84Of\(QdxD i+_74|rY1g<]|mL>d?$$}wclW$W\umc_Lca&şz>^qI2ZX:t O,W!P,GeO%Ts %!/7O?tS=/X\b_$$ޗ>XBp9Y`}o{Bwz=u) C2vHm}0W\,˰cJx,zfu;W54`j  - MDZ,I iՃq.ˀs8e4`mt>o¯ݘAcY#2Qzԙ*5+yP6-uuۇ{h|]fa mŐCT?Dw%!"&JC1yjfH(tYv=ě6OZ;1=f֊q;~[=8 }vgܙLBʛ o$0L.^2 ,㙂jweڏ] xT5T>dsǫ{0 4zE=.bA>1Hl*(6o vۣ2)8U:6Z7ZT;g'gֵ\dy7_޹7v[,p5DY߷sLovXO{k ߘp8\b!rć({6BiCKp(3C$Ct3yo^>+W'bEpVVv]Wwk=Ef?tz־]9> stream x\Y۸~_G9$i< ^?tHtP-LwRY~ǨU[gSֽ_"+)G@,eR("j5xGyLxܡdH",E@ jj߄! \sR0KkOCcKK/UM6)]TlliYފuTAr,} KwG*b,s3*7L"%j~KHeMlX-?clm- N(b"Mcr\)آӰR *YVtG>.[&$1,K?| W,ޚc?q2?k'9U"IX9ȗ:E>:bTu oZMǺ`ΜF#boIhU/n6)LC9{r/]zHw+gvy٧]^ϟlG$+lẘ{qPDF؈W#Wۤ(2aHC,pH)4>`<B#k~[Ch>ݥl Tcn'7ݯbVJљINo_V5^eGuQ=WXdkU~wiN}ۼܸ`gxɶaj -N0kCG{wrWe=$j!S1"^@e0 ѡТ 㕮M-Q:)nҵ>:R5>PA!ఉtqVn r֝i3j7I#{oNr)##hk*!hUk K ?b~~;`UJ>z&GiZՌIeqzN@+%="t1QhO[Η!cr!-R P "&i" Ƌl6n&*2;yU?TnWrbWnD,‘(Yx#U5*/FJnF#h;n3Z/{Srx3ډ%὇ƹ`|M݉Gǻһ/}g^g,E|T a!{ Q՜%A5 HO{<]_ 7#~Xc"o_-b嬥2Q$RDv`}w8xx~ӖꩤK~ y<9 0yT\'+Y@1h}&fuh0{wc06\x&[#.I|tp#`;<N"zA[bz0Ul549fseeYԛ,Hu?!ḲSm5TЩ6g&DbQ:A\۾;h :va6?j*^ߔ2Ȭ ̞f2LQCVn!e&,&Ae &Rj4bH3-WB1\o…tYUwɧō!ӺWSgkMz$=>(<ܖzL]S6 ܆K=jƽ*(r"Vݽng?+x‡+5|ݯ teZ<捇Ȥ腽.>7Q4grv^6!݉˶T9w4E ;k<</L%0Dtd%\bX&׬׎".f&&H/k\UN^8MfymrۭxifcFq{_.5;qKVN *M=Q^LUf;ڃޘdr@%bbծzցjl̀ fRDbyR\V/lA"(Ē1{xGN4'ӉFzIch pjThjƜFT]1)5;gnRqZqxyc.n{a JZȇ!L5rPQ8׊M^c {=K,nN(އOy&AP !Fީ@J:*;j 9h΁S cMgKU?WO=aO‘i^rl^O<_{W*tcD,L[#&'IE5D#ʸXX\D; cBc'`hٗ($ces2GjΏD*M=Cb ^hB5 #bP +O,xI9Y}7w:u >HR;ݛ t,!j*0`^$#˯F\^ɏsD\=aj6{{|_h ѮnNב5W8H HL:!C9n2 eh@REIi Tu1-Cs]X2r (O@ ̩U i/]|j:ںد?곋'Gm .1*AdJ|S,Ҩ 3$܂KUacHku =ec~4)ɪt|&ȋUN\Baבּm>ݭ!|( 9LE1 D3w 6D{es;[ *yu&*ZNA AFa|@A̧v>/g&)+mae!N.l_ ]NAA.\{=y褓B_fӘԻ$<\NJUSj&:P(`Kw-Xä5{0}=>D×@N1Kl00~" `g~a~ug/Sȧh俙L۬s]&Eq|x)4y> stream x\[s۸~ϯKl%`g%mӺәn@Kč,zIJq}%*dX}hwl_[on^}$Nf:4x (b7C]o͗Xox8˖Y.* 2KKd 1fw̫2=*L4KvEW#B:]:/u15_v2:_[&yUjI 1F#D"F+;vu9<~1I֫2|N>x1f2Lzoсa%\qQb((IB(Q؁ e|&qk0҃L̰tmBDRwߵEHcqd4”A]x![lRfboo]firSkoo{=vQR)bՏfXEDHk;$i%r#:O/7;Q7?diFejQ~!nSjEP'tPĐ4ء=CY;k&9Sc;u`tW lt<&l4!jYݜy3O 킸x0~D $J]>{v1F FD1kLu!R'2].Sn_͔antm1/;gKEh !ȰDU;&@̇l֞uoŷqJdPiq>UJ/] Ef c ntW*?VRX9Xoc{ NG>]'\#\=wKg$[+ŔJqFT<Ԓ~u*%f]^^b.7RMnlC,OIgx42Gd.La\ݯάÈuJ8"釀vH陵C.'GYfqkcSw؅9DYDHc&H $1T$ͻG߀#$:mHZc2Nང`uvX v!uxd=\2-ιj,KKO$GD\Ay>Il3kqW|.=SWw E8wƶjg$vc>=MSB>۪׍$PΪ޻|i1)9-uuĸ ;͛E)[󢘘M.78 q&oƵ[Xuݸ%8̚&% JA{qI-]qЦ)>Am[1{EiG\G +  DOoQ"l3&#nl햵Q oc/eIlQ|lҶv": Oqcm^ 8M Ske+$T5u#.ՎqEmM)-q pSL^LiS: zKAoHejem58uz=k%_&mmnm5%93~ҖO*aZ,getsba&Y4 =neU+z`ǥź:''!׿|xynFbq[&Г AߨBL躅018Xkǡ6,_ByH8Ў3R|jSmr`/5ʭ#qXy,U? s=% Hm[؞@Z~ :h%chkhsh &}:b)U2=O%̒…Ox {_ bLe+WYXhfIjybf#ɴK;uKe+Mmu8CXv wn~t(ZyaSk0b5`H~yp眐/ s'hoeNQrx&B`,Յ5?,@Zpo8zلxp9z/;^1*VY/߯v멼 gvc6ZvK@o"8\.eSwM{.?RޮMZTs`mEtIyhg>W>}$Uց2xt__{ à`r $tNnYlSYWͽ7AqI^hOX*ҫSf@YI{|B+)9[C R{sq&n(_ :8s||̽^l_|VJ#?x=x'􃧚/xS:$i+P &^*nyGjɁ~9W-S﷙OC])? }FgAΠ E2 &~W{<~ {~,<ܗDGfOL=ǂ؍+/^H'FQp˃W38TU=q_#IXyܻ2 %UN5+aDR/ǒHJҽy?Ⱥ֑ endstream endobj 792 0 obj << /Length 3799 /Filter /FlateDecode >> stream x]ko8_aX:ظ")RIgf/tش‰Ԙ$N,/uI)tv(ȦR"ˤ}{NzqiZW^ӃɴsU`Si?I^~6V ŽYo^i.\#v Ÿ<{JўҜ̢DESJODAѻXmӘGMhDtnoCWQfHhG>~jE3=ż,̎RпoJtk^W757#sJGS?!=sAM=;굔39= jNDK_F7)<^gFt,U=&|E3H}&:#jӪYҵ,33vق&P?p{;'z̝#'AR ޡ6Q6C9AxK#!v9ހẸ(PV[38o4cNIh _FZFbPc8? F}h+NDϨ#ZNٴ8ɭ;URP+Fr%c-p|!;2|{yaSooG=5}0x{ wǍڇizEQ#xLi \Ҙl͍: rrы Snd./OKԘVy.zQwh3Ɍ(bު_Յ.xhg6`*S{wn BPͬ%PA,>9G?u(b51q rkkDkϋ%eEZ~|^Q>m2Ύ\:ؼ(ե=P,ǖ!9v!t9!a?N\lŚ=ayAnjS$4:5%wb姓ʾ@i#1ڷI FC|]Q5U8[SU`a$13)%gf2oH[|ʩ{Ǽ@V&nx ;A^;E,ܲ?3zk47~\3I:!r%h(itNLbmHL)?f.7˧ЍP!Yi~Z@ n7!ɼ9uP`061Il wCBH{ԏE͵={[o'W2oC4 )-Vp Y8<;ھ,m[ײ2pfz{F%cN!Ѻ ;v)$$б\ݢCTJ-cY[,6DTaZ \_[ 2 ZoPsNr8F7ZzAwFت,hKOQr=rȚ-J湣d.Q2t3;b< i̢٘2v"hr'(qSZ(Y ,O@J[Q@!(tCYy3fiwJ 8 daT&<";E1o[U˼Rf)(s[ ˂2e~P,^؎1G@ȱk&ԴHwvZmըRgDJ0c>ֆ̙Cŗ T%_Š׍p2K#PsRE8,3$T9Mh+<&m$>Ws3p؟6-fXcG[tڧ'0 ƓWJصOO7uAOI_ĊOǖ|pwk7<{\ϑ2(W 6r瘟5m24ynߛ[hs m7eVk4q}agձe_9zrf yƍ Қ*+l+;ZƋ3${അf6ŀ}ypO賆^:W@IrL/JKMsKhy9yIÍ &cڧCkz0Ҥ):t-12Y6h6&֮>Ӓc5:NB:N1W8pmmFt&V IXl!]Q%Mi1.ـA@KTJ)[=b|z7?5p++oF]2:t%j$ZPK jB%ݻY'Qw6[c ,g1J8 o em98te 0Zخ4R'z~Lm6-*0"o;޲--[9Ue%xp U=3#kjyX!̿NWu žTSnSs06PKdN;8޻H[[u̜M@L XU)QW?UrdUN}yu Qbj8ۡ^XPĀmr!7QӸ q,6` EH12U*l(w-^(-'6u]D S>]/|;7%Mu?J\)+ <ׯcT6{P(/VxD8`crUv,>e;ޟ1[j2o£[l> stream xXM7WKr,bI$x1|5`MK4eh r~XWd$..KIv7p +"Wq-Řu\L)Nj.r1` 8XS`!LB[  ruR `PӤL%e”q?%9:e蔅1h05 VAERIq+(\ĥ0'R*LKSK0&`WVx(E X& ,9 &ƫU\"+pA`VUsm 33f86 {S[rpc&uV0)Zu9E9؜[H@X8U' {%SHsїf+dYD`n&@'I\"A1)!1c`ASpqLYQ0šH fpqPWp =a T̈` ?.I`.+R3ȂČnjd7HRv% RJ +0mrv6~t+tݿ >-zrfw'HacX3h{vuYoxwnPzÍV8 -̇ޠ&#tx" oHZBq#Dp$ ^hO|gjC߹׮{s׽?վǃe?~a@t/z3{b1~%UFo0lw]lVzHE*k:\oI'Et?wt?xͱ,#%4>@yUBѷ;Pv.ݳݳn/7vvåϮlw]w>1~9g kВL% _lz7[?,[?vZwn rga1\N߭6ӫnq-(gz7EZlfrS}2k6Xȫd}!UĨ'C0!n~2onn+upo.ob?wW˿X-!<64h2g>RxZ菔XKGf+#I}2rx~S<W!}U;q1*8 9>nbΦG8M{*!FaQG2nǍ{ԪGv )'N1&O20?x pZQ'ᣢ,1hsأ]jԳ}]! ǿhAyWo[w=٤cx݈<$J*}Q%>WOpC_^At9=JvG(B6QAMXj٠R|)نk݄eo7! j34 NlDb|(F^$xF^l"ylXjâ*yCE FK&+vwϵe m9Gۢ6:ޘ?ic{|Ul^yMV; =h"wCf{JNA1iQLЂ%\O)(r7lrñ慒ț{#/,7FK/Y +6`_$_f6K H6$߲7xG ~fFʉWQ^Q#&z9f>"&P}$e endstream endobj 856 0 obj << /Length 6897 /Filter /FlateDecode >> stream x][~ׯ`LYb f:%u\nQ3Z3tMTRi8M4p.߹hv9CP_ Ww?}ѳL "f/fBfd4C^~+TYg?_q3%#KT>^P>'Ù/O48/_땽m8oRMy5w .2KEY*JbJ$oH Xzjwn/ENGxo MXͿnYM7p {}a?.n0[pA ͝OH7P 3KLNC3=߁ ^7OBУS<"+, `c4Jy8Z |ۤ>@.%$>>>#çT>v41#ݠ֣[~lNT*֣3Č9޽~'/-kSBNY^噽O%ڴ4w/EŲG;+ڙYP&b Cow\*rW R+xןK]znMc& c^f4 Y}z#0l$z7inL)fRE WVqw^,aJ>fc2ڋT(yOBflMfmվKᗑ[s/*Nt;y5F[&'c."$iTEog['w(>IZo o2ẕ"2bL|8lPq>4 .RF”cyΓMCJs`9?#Ejm{]\@!QI^Q%)v}7v}>LMOrN#OrlH^o*Q S"UM# QɏS_ۻ92\+K'~(T􏕏 o#go*CYPPu_u"^K]Χk"ѯ4׿:U )|Nc'%$nb*Se.'Ok{}g/,>Qh FgyH#I?:=gUi?V~_(Hhl(ϗt}~?T٘0[-cwՉɎ*<.&`r~M5PKSh@*l__wX=4w=3| Kr5][E{[et,y7 ;ka[gNQ)T,+ ΔhB(upbe$ #;2wG$nK(}:,?}"~Zג$%IϾ1'h(%n}yuȿQ91*CΘ6r\W!44)ie1'|s{Ym|߬߭Wm6gvfQfp'2f(a__Z]Po)I2M˷_VnyX/S@vvTBt箝YR?WCͳ&m5!3mL3t>eXI+Tg_! +.7V]~[oW0_WC~{u.w˛a+V!e┟%Ɉ SPd 3I2PqEfg7~y)TskT~f@}Y,ߠ9 V9oofɌQü׳WB 绖P~QL}ݞx0v?ŁŨ'$a8Z!~ZP.+1Iޭ.#H`Zn}L2lIu`3U*8zO-E}hBRgXVt[%4ST49Sj3qzIe)o~#sUK~ ,w\R˵mFTo L(qZ.LDf&ɘ$fV!akf $PA5L B[r'M4 <#@wܴIAlD> I(IbK6[ H6I oQ{~5űcSi5v<]ڔ~M4ꩆ7F$r=Lf@CXw@G0`#<+)!]XTkP vkW4>M%@(gEe1V,}aL 0x9.Aa.GŐԙ#,,Pniʊ.f#T[T8< -咬(p9-RiKJM K(Z¥UG3&A<^ÀaY3$ه1aYcdh?E]uMQ-6p'zPեTE)Lz 4X'iYv/(/8=bY0v pX9.T0m.K"Xpv@Jm4ɧ Lb6J<0%0>--<B!>YL)>}Fj2J^ aRie\9g:e 18Tiq uzxgYVJڤKl@`%\&A0?(&a ;ԠLg RӐ@g:R N"a#V&o]#d0BTUf$!3Ehr5-ywPMcKAuHac`*d%tʡܩwlxZz,<-RP O*xJ8 *EBҘ;d|<. z^Ȑϙ${쬻A Wي ?.&tt,#? 0vIkJ8Bpw!盫b\xƅW1.)) Pr [%ɈH9J媡h/枆QsjA}Oв'hPμ|kI:&$c-a1pUYaXjRƫcv&nuʦ,E6K9-R֍8"O׌7BHfXViQ@mW!7&Q-񃆾UB""2l,8+LIlo7>*_]0F[ {06p *9)`RFd_Ȼ dNƆ;.6% lt*)cn9;wc)*0R.`a0uOaQY^/V=vj;&,L `vNUiZ޸ʈ (`2R+2e|K4ϒ,YRfIi 4)R)RG!`^Lp3Ӵǹ+C ((_e'(v&w5zE4yT2C7/Gw@/QEJM#`#"l>^k8lyhEyс)=s"! SiRhO=aO)՝ƿ ~?\W`gj&iTљ5QYcۙ5zfAu!tе&%rLNnԋEF5j7En[iO0kH"JP PuRL6x#9m4SAYK*c¤ @2 CUV#烅V52UdX 1hPM=Q:)6xك*ö;^YDCMIjq gٺ:Rz}Kx1Lp"pdK=ɥ]'sG,;0 uș7"O ᚻJL=dK3(L |nP%-Xu FCdYu6~#ЖEe{`[9*?b_ףIr$c|qC<-sաeb sF!!҄Ol{)9_NhF4G|@YGcxH1[bS (,H)`rXsZ9`"0&IzΜ|nBiPf wVuc*{"nOUnj;%: 43F[>\nT>PvCmmfױȭ "=y5A/S)>!efI3iYƛ%*_3IՇG&:dRmy E⛤G-L#EavU~?Q¥rjlrv\n NR:s 7ǵV g4YzF?Y ,ưA,j-`߆ f bz,1M/Ql5S@Sg.&0 uVPX5C/hU="K$hSpA$B-\EAQA]auyݟڠK6^ 0ZQѸ cOV8 L(䥻W.GO,a̯r|%Dw_=AiD{j)Ө6L)Mco ޮ3 $ΰKsJrDt9E\Oc9y_wAM"lH8#(ws/[ֻ#rlrwY`.G߇" d&۳^➸ i3s:فp" G.dD;lTvd\Paʅ)Oۛ~}(Z׍9aP7A&-o Qdτ8?{FOou-Zpމ~UbWc-GU7!W3':j5՟HD>ۖ!jq VĂ3 1u觐 s ]O$MJy7X9l-f^nS"<+v\mXrKc!K4 "ZeSr4|ׂgDbQuGqƎUH%q'IU,l{y+{f3⡅T@Ù]P|s-7P endstream endobj 886 0 obj << /Length 3434 /Filter /FlateDecode >> stream x]6=/*MVI.hxږay#gHe[$ PD|h(]>HW!q9I:)^]^Gi> IDYT]b:MxQMmU?ǫ߾ZJHINR肊8WzYL4LoKkUMؽ^V3d19{ R"g]d4&9}Z+m,_6OR$*W٢K0rUeYi6 ,i5غv؂z=}qy^.Lj $s=_jq14z1uRvN $^w!1KCMйw/'~ NdgxNCvqnq 4i!|)׬-esaw/yo@CF nL T$Ψ&eI@Z((&7EfԂ:z}!8)A[A;vɣI Dݫ,itm{b<\ :;@ 3s!S^Y^R=;'%L]y`XbC5򗆥١`uTD'(!Q`N2fuj~CH9exYyOdFrz9Ou1ߘƂn+f3b6E;eysPr"DQ8%̚ǩpl_mm,Q,CZ?Ǡ&bXب6LY+y~k #W-nŴc + S!a<UŶ%3}i-TȵG?T\~ 2$<Ԏ`ZHw,9D$D8;@eAP4 shZfڱfU(k1?g~@G(%vƾF՘"g9gN4Xlj8Ґ <'1X*Sh c4-4׮8r卩 C2p EjKe2Hڵ] 1-"^U% ƨj%nhyQ)3S;bt(֍rs_о 5XSޏ0 m軍So}gʒ֪h}U%jLmg}4`ve i4S("\ 9ȗ'kfv$O\8SNrRU=; ;4l0#-s|G/AXkx@ǔ@M ,z I` 3P5S5p&vK(pOh GF3p ic93k=,ډN18k0N'bW·S0.șTL%5i6?BV U p.,i0\q d$i B:O^] ȽZKx>U V'qFRh7%39܉ 7{2ֿ]gPnYurq:\\c-uֱUcI+ hxO9uN <خ L]keCZjډ>qt <F"(FbCXM4{U¶ϐңh.+F+,5Hڷ$zcK؂L:bƒ@`#cTI/VlԄ$MRͦ;'5]z^峩"`L1{O sŇj~; m*PN,6HzkG&*z~#E~gYAF~K R9̸g4.DGĥ djrSL,z ul#1kgv4L21<<긷L4rЂ--&cMllWn\J 0!>gM<:ي vyy" )q-l3|.~s_"ؼ!%r;1̜P3],F,paV?JB`z_Uˌ mnhH]~̗]S0o&7ڑ$ EQ?5_Vmv{{g[9ƛVWPذS$S䠾4Ƚ-L?'S ; )@=GzNB\A{k(0(_L;_F6MdɇN1W?3nhSd'p'q乨w &MvRu8c I !t{u(`y崣6q7J#cPS==:xbRO|E]13l'ƛ4ZTe_M@d&}:J{'Zޑϖ呓̗DL! %PdzPe;&0,G@0%c/L7v a?LӔggYiȏ޽ endstream endobj 937 0 obj << /Length 4588 /Filter /FlateDecode >> stream xݝ[)ZM"t7а<ǵ޻sxH Z ysrz  )@MON@ngœeTp'^=ݟc9˂,,AE3 ٫<޼"X, IZ lSfJ:YΡ7gT"ٜ]-D<1Iq2jѼ˝D aLwPiDVP]B/w4ΑZpy;.u Rm{{ ~u|N,[fAdԫ87;U14UWBPRW {S^|f۔Mt?Ӝ8MOI}}܃k ]]DTiNU[/.wzrE"/J;m^:a)ə,phƢBy:b+u" . SnNSh?ヲ㏌ş% ?%U)PH8 ePmѳ,\1W8gTEc'꼒&ڵ^ 2 $, yyG8ynG&cҺyJ_KEj(K-mZW&>ͤTkot@ xb&z:)p1quaإ0'ۜzm柴l+۞P/Pںဴ{/#k< )?hx?B_x.5k={8e#{1~n_^׏kk }w?C igd4MB`uP>iW{S@Itwcq%QTfHMPDEZIq"w8)ߠL%ʹ\@>#;T,lrN>]?\O9Xh'oϭ9w#_! F/s,&xW^{Q vx7tՍv[;Ie]@ ;$i&J3ѭnv~fu&D .J4`ʵQҹ{t%wđ8 8gg'}y2sgLB 2A_-X6or]6`"?+m.z7;jeV>n]Q- UܩB`, 0c?\dF՝=^5M~}zY}J,uYue 7x,ZlUR;2l1_c3gBA,r'd\?C࿐sv~VY,B.:z=wy(;\i11G 0wً{'E ,DEj#Dt|/v4 23y XCT2ƬeX8H”FGqvWe"T4RV}oyE1DZ$,ow>PMy&H5M0j~hBMh4#-g_}!y)niǵCo:d(HS0H~u& oИ?՛@T a2\7N1?D y%ݔ`{QQLN::I ,W$, G>XHQ CzSoJghT̔: S &Hf3.%Dpq"* I@Ů\D]hR3"sB\dN#sJSȜP uȜR9<1GDip҆|*2N_ƌa+B莣{D4AaQ| QxxP;( !Qh)CH ͈érXD"*YtXE⵮Jp#'x?} sDR9לff1SH R am % tnA%)@K9ZsU]Uŭ#g'z˼O`n.iaNwj&źRr\Ow@-'h9%E-hI"EZRrGK_s=7z](ǿ4eoq8FmOrǬTewVN1Y9J#+ڎ$ʖJy+/J!+[)޲_z!i^WR3ig|^+vw/S@z z='ԁh@=IDCHO^4I<{n='zK?6CAO"=Uŷu=#pr.Yn}3A i;VuYc{B!2T-p8n23CHD1 b]|e}p--]9e%_Q(C]N;z4h/:cc~06p u (4vG%QҐ(R;*ew7(IfjK n.Ղ%ToW RWM((7ڽ0p&U(^t߲{69mokƲ "0|}ky`P7 n P!m}fZ鐯v~egJO0[ RHZS7ͽ[WfAY3ڼQOw_N>Pq"]oM}[4wlj%Ѝ^"r 7՝jR[ۇj ա0ś쀋ź]XnppX/7Arޅ-{ZCuۃs[;68ڵ<LOЇT+S+pWeQ5mC*ԣHl $:KյL{mUӔֽP`[o͚C;$q(g'|tƹ|lɻ|oVD*JߚG+q:ޫ)kcorS1/(7z~Qm\0M Xks2=ΕV]V ,XZ"ܦ8Mb'yt3N$E̫CmP۪D7.Q;uo^61bȥ:sj,vLl_yƆpYY=z=4꼪rRAV=Ӗl>kjMj_ e\z0D֘i~ؼ0V[0;Y>2]*_dUG3GqrlI8迅= endstream endobj 852 0 obj << /Type /ObjStm /N 100 /First 875 /Length 1464 /Filter /FlateDecode >> stream xX[o7}_gHZ A^"*V!/l R]F+ؠ2.fz)"0j[t se^Lf\["1OtsыǾيp 5dL'OHDKbBbibO$ğC@A#n`sL,!Y fyyjl4 nA! v{k+3芀3½V1ƳϴB76 [M t> '(p3&8ϠR:SY:Nb 0l5L`BG+o`70y@],`<1Y)`V 0 );f渊ubI0}6& |! Ɂ qp@S+g.VBGr` sZ`CF,`0`0O 2;N[ BxkLH"3EI {Mƫ~9Vʓ_*17o>[ݾ}WSXJ^ ŠVS!-{Maj$o:8Z̟TgAnOAY",# ZǧۗO7g帬?,͟g7Xxf;5cjdnݒ~ؼzrRi~cboQD낻srC+G./Ζߏ^YnO_mNjz~n}8ў0/q Q0h.AZMhiY?m ͋O=yx?xb;oѷJrt;^#F9赡 *H JmZR>N9"SX%9|-ۥ^DUzWܫXaMDNM^5"Tu&yWkI^2%9?C>n9ZQGHLvc\@nZpGx= r c4d:Œ>V>jUZ$*".G*bv\}_ENB + 7mQ),mRX+9SpG((-Hd⣉S2ޅ$I^ 9BN&>Ð0ćaO= sxY>ӞRS~)z._~@||s7|s؋a/3b|sd* endstream endobj 968 0 obj << /Length 3363 /Filter /FlateDecode >> stream xk۶MNgұ=i_qfODZ^%ug˙ )JJX! ⱻXh6JF%G^21BI&)]^ JℲt.Y-7c,dTd"+|4^8/_erNc1܌D $m41M8rӺ3,i\QT53*1mFN0Y\+QiSCq,hQ,LGs3bLEtJ-~P_ٷ0wcP>(fTSAU*uaz wQ}sDK93SS{kRB]\P)"(Z_$zzH?P,qݍH4z$M+W}-릯T`0?R-F R+mUȀt f%P%!qE [ A$q AALq(:0f?B[R W9RD0cojŜ!I &C:VD:X@&̈́s$}0Bެ3^ Qe\SP\U3g#Wg>X TzL؞s;CKO` J*xčp*Al ĸ2؉a=!XKq1L/A92HS%SוL4Y`@}AY@+,EaXo7Qx@$U2ZNyQ~RHp i E zL6W>Ω7c`lqCりkG[A%U:Bo : ,Wjc18RZ B VI^g &Gm9'/Y 3\8λZDhR}HeHicXB1VFla' v3Kw[C[_"X@iՀ1%<} Vo;Y|q1/G/}sF:oʹ=>A􈌾h,aV|LrU@9[ 88֫Aڇ mP)i0"+=&MRed 7;GȭHY:q|ʉ8)o %,~mq̼Pi+2kMNt2Jb{0|v90*xǘP;=Yֈ lXl̫z[䳛MO8$q֑lzMc:;d.1MęYCYZ;@+tG/ @i۠\&b_yFLd`0i;-C\wӫ}]|:";!7% p԰CNAVMHΠ [hvmӯ7]L܅|WJW tC.D*C*ZﺶT݄2.Pg"\c:AtOp| '8sfmƃ*ܑCv8O²l0BjpH青: ~OSoU_ HD/ȗ,}B<\q|$#ӠX ]P?,O=$ yϤ'ZVXwmV*L PUUBtUI%pj(:sQS}Æ5>>v9j2FDJuk88w! ۗ:0wL&r  ~ eKP֎jcpsAu酎p{."/#p˝Q7Vv_9D|٦DzoqJoӻA-A;"҆NߗZ*5xYMkޭa!̺] No#? F%-ŢrDPi}?Hل]]VF_ء'Yqk-=< 됹]җyrI=+,$@{yIBy,~]('3v딹$N$Ii8 Xp"{ 6yeC,㘈ZfA\c^S{ϸ}6ɤIwҿ* \ҮsB \qls;' ]*I??ʄ/ :CZ;5b@jeC!%𸯊ȠvAcO!,sw:ts'x\@"ե=}sp+=61!+&Q;أ4e86qVB_"{A Qnm_ln ϯ7+5esk34tOC*sFeHl=N\Ur8  (wtԃ=x"98:m?┆V;Zj 漮Uu%ٛnBu=NAXQԦtݒ[rQ%^8 iOIQ|Vrh\p-ftO}*ڍ*ԻsbAwJ'R,!ZLT&`!&u"{lߎ!Bcsye-#5}}L\'S(fW4(lnٮb=)&lրS)MӴqug|k6'Ѧ|53յIfK[-7tRUL|Bk K/%vKl+:RD/hcM1'4[-#Q;y`$ROk{q{i endstream endobj 1037 0 obj << /Length 3031 /Filter /FlateDecode >> stream x\ݏ6_ᗢ$VMr9 x(i/VF&qo8$Jmf 0p8f8ގŃ'4'9x3lOѯHUJ/Ɣl.Lg?oU}/}d4f*ZcT%1S'e^|u~_VbW7d!lbޕr97y)rZ>&0-s{dy}s)6bm*l' DXT@Ǔ|kFd@y*Enp\.7Wc@O(舋4F̩?ͼ_,Y[.Y-?ήq]=8Y+% 1W)&:c& +(Vbo("R&;j͕m=/nY4Ze3T=RFUh+ RÛP{7ZˁSVy\ÀMsVۑm3 =& wZ؋1ɛRa&:V%4'X-gH6P*bJ{~6LbJ !ev$``"KJF'hxeN81Sm!嬢eXla1rO4,e-5ۮyib P(y 81WJuOpP7qf>GGeU[\v&D-NߵGoA~!1*^os?hL!#G}GM#z E5#:,mz0@FXFCdGh+˒R RkJŇp4fFƌ1k.PAʮO!}xƁYbF^ <3iU% YCNOH̡4N/1!HcU-Ѕ۝ NOpaHp,ԳwH ζkl~!ݯ!GN-ḢA"/HgA"džD^C Dy0$C]z:@gc::D{\1w|@fp d4GrN;m}[9QKot<]_ =4#ݵYFSM7"ڢV˛> ])3SoiI|4tSw,c|\[=߷z?S=_@M˦J*9`u/EkjJ+~UzN!u=3niV_+Mp[%CwX^5pr<⼘q-f6r]![Sw(BHgul;wj Oj^/|ޒC_gA] .Sƺ=82j/ўt#6@ s˖bAfCP6_n[z4~zTx|PJ [VQ|BDlCDD&5_xG99I lBy=n-4X*Zh;`A )zpW`XT`ŷRCroR~weG3jJ J9=Kc}D#_g) Ϭ= ؖ,Js5=`b>H/Y-&Y*9ꕜBlH`KTuZ}ulLX?RSII$&%YEj?E":šMhҰdE@:=NܓzNz[k0bY^s hS5}Y`m3?ؓP0Wx'XZEahxfd>Fq\ξcC08v;AoJSX$Lvͨ/nѢd;P̘ʽfny(B| v#y~ Ho`m։]UI434H"8o1B/5P7ʣNǭcqъ7phK{7|&ehAZZGPm*> stream xY[oT9 ~_2DZT!qJK UYE|f.5ڑgűRI$P *W#UZ2):c 6& ;' iƒ .R[iXd(-UI $5!>B AB-w۰ L+}SaǤ- gib830Նy@)09pyFOL.#1~a \]V7W &p0r + P Rr'͙Y+T ПtXKpaJ6[{fO3Z)%ӌ&Rsk5|.аIi Bo [P^jRXC@%Fˀ&M` Z=&v.Iԓ Q6C}ҋ,I%C/ͰS>&{NޡExSn&^#@U}c by7)JzG&Hٛׯ7o~#b՚1^poJ%%͂fe ~B*ksz .Liycq],8sNl<~@"E7:UP@OW(-=Lgw郬{\-w ys|X9Qg\sM:lwl ),Qt;sfۑv~Bi雓٫|//^9rrzmpХGV䡾[u8[]G)=xr//x'ֳkOeϊ9LIZnB#+G ]$-"{u QUPF@惾D#BdBȞY9z$#@ڽ?>ܩL iNG̦PvB1\t/$ *#n1٭PH^z NRcOPH6u W(z&*eB{"FyZ12@ sṣfOE}T^B3Ax;^F A:qqtq .)x+2Qp#b]#&EDkqFm SYUbA^0at%=y cC,n1ނ`F1\(4>F^Bق >F`㪹\6IFދ.<(&ksU|9!񼌘y ܻ!Ꝗ+9 1s9nUq" wZsA\NKWHJ|TU ͦ: ~bo12My:c&lKp:4<;6! MxkwOթrxsrɧy7궗馊L׉gZt{:[wȴv Ƈk*7V~$H8*TzK2:ItPCYg:[UMA пxqwDeGrLC G5[<+FTp}q[Z*|ס~"t=꧟>` endstream endobj 1086 0 obj << /Length 3536 /Filter /FlateDecode >> stream x][s~P޴J܁vI3Mf:iL-6芲7ίW"-QwgLQ!pw8$|\^;'#9\^O$HFQr1yʐ|5QMz4%W3">x>Om-^r7*'8B:P&3"}7t}?rzVJMlQQ )̀vxňwB"$FEF~ uh& qk3!Ix.}H;gGmJrw:+Yxfk7mRQEk E!i}FJ$!V(9& %b֌  u)L)pJ1]g[wH&Ě|mu6 GPbE{Efp7*8^$xlu~mVN^vX"7aH¾)gWY!~1IK -4 mz4%]D+HT ҽWK>%DFNlkT8~znRR!Z`e`mh?ׯ-\bR苠T"hR\$Yn`߫5j]>:f2d:Kw O|Ĺa'+eQ<3,z4PJ#x2F 0PQSŘ?ksdХ;c f1(c4T>07|Kͱ&v.# %6[nXd!(^/}_-M~!UDh0(ōJ Š&GP818.deB*BTM <t5W}Lpkz.5$K~DU QRD()DBl%=RuT^~Kb)`y g|ҳL6sw[wt=[&1gR:Gsb m\dw __{Hr @BdJ׸sFj*sZb#ȗvvoH5e`|`.ś]YW=>  Fte!/̰F\k3ˋ] bĉ0I`ء+[F3tm{b &<6Z.C.n8xm;!+>ViZjR Z^&f72n&aylO?֘Y |tGM9(&ǁlAk8bc%0l{*X45^DFIS AB4E =  % !3;o`CGax{Ѻ<&?3rQ% k%sA:_Q4V@9lWƼ,:3vaͦ11]AHm?BϰH^ Dsy].fV'$ ||XZ{!ѢlSsG:qT pxC>7dBD2UNLP s9CTD!Uf7mJf79FzˆcU,k;%2a{dqL9GJ\"uw~ O5#H6WpOKͦEPJF`GyIR U"Ѯf&Y1jWAPXKK2"ua7n^n~ nګO n6vl QK7eG9Y)ݣD[QT$6\EH1/;~+@fl (@k}9"(_DOj]Hty;R&~F> ʜѽ@UBK-8U r(3r U %wmOvQ)s.7r_e-:1)*{ܹVY>OqMz˅4^O/θ˸ΦLmgyι {`6Z:fK<`u㩽rG3e7mAՇKKX٪G06TZ;[ҾI30J ?ukA+~ 3ȃ); (!乕+J_ȠU@`]S0BpASE6ؚaewNNΩ+]beÑt=@G]Ox( O/=&f/s,F$BX7h_(5[DpYGNZaZHQbB ^r{b7'8(J6 =M3]̮#ԔCz N:tz?9s-k9XItMQ5\e5 k;+uw8O435]l|G`Ξ:E~1+qmSQ[ΗHʭ9?˛͉{ "$24NH.yeހn}0# Ŏ?0~b`KG)p|f/қ=1ŦF̄F:0b6'N NWI!Zow~JB?2?zH; xDlZ <:[.0&wi0sDcԷDc endstream endobj 1092 0 obj << /Length 4042 /Filter /FlateDecode >> stream x\[o#~_hI4@m@\d{Ubˎ$勵9q8jdٻ%<]gyճ?Ec)g.gόJ5{u1q.}`77g7vzϯ 3c8꘧Ag T&vYVmzT7妪߼yΗwMuQu~ݽnKfeh()+sDi^X0CU4GM.E;Qa*_ fpaϾ~g D0cƚXg7~.350}h0ldnux˥Qie=z0β@K3K-IJs9=[D#9&[L9B&Ȓh6R534PT%)td4Hgy\?)STo/r *n;nH7d`zfbU{'=&Oem`f\W9Z! D#ؠn~@F$g23_q[z}??V6*/Nf&U@D$25CDte gJk@Sԫַj[}O^l%]j9V~^}ȹ!NGvxBtum\W3_mn߭n}Mߞ5ބS9,2s)(afof'z#U-\E2}UUؤR`ο6g=E3BnUYVV{Qmx 2!Uz q`vSkw^,p f 'z L#e$\HcQTXPJ=XRаM1CU6ĔRz~dna,}68"ikפz Ktv*@2f2@ -W@,=&@^pu|nb>-G25{ii h"{:eLtwn.H3WLdDhݖ\SRnywI$'^DkAsLABnphB"Q3g#*8[xbL(Q{h}i4GxiK\1̤@B<. [ b! ?f9&0"o+_@t?hp`?ǀE"neMl&N4(㋬ĸ#LBFcxHrvǙcҁF \WȐ݁?`w2Cf!Cj _d+ a"ŅQV9䒘"o%{3XIДqhK/8z\"AT\`e @ŧH<| Cw U>b5Oe#N.G<īAiwfKhMP>O IzXNY@;Fty70~W>L?j%gݢ^0tr"d}53b3$([}r)%fePktAkb"D{D@OÞ ДC(s}e><ph{,\.TPA,1U: 7w˫wզM3W]uzkh:jٌMnOoou3dáBZ8r[Nw'՛rW՟Xz0#DX=?szꃽnwy)N(~_UA_1HC#4q|Mg.E9X}h' /ƪ8Ppӱo n~qMk,ίƴz$e ? Ä"kHZtljx.iA(> GǦ r~.EȲs %8:&?ˆ=aA0Mbf j50.PYưc~@N Y,BFU Dan'}d&Q*CĝB2=E_{={͔xa<,Jċcf+O<ព!AEF;C-K?!'s 춛Eє) Vn>a=1tQtZLR@&"i>*&zj;t9g⇶=?ް TE?G4Cڽ IMF,W$7 j&%7&+q*mY,yV,$_-՛_!OTN姄Nejੜ <R>^hf ub TaOu>*q>"v>RhHԮ*by.31C&>KPH,oE8,OYpl2E9Ov+Qջ#E>FTǑCG 8ꦻqcܓK)܇3)uİPVͿ 5Lh.lwU ؒVE{u-\ܷ uo;o-7˛jWm;.Y;,ޟmw]we&%o9t6Mtq":R6Ԍ59(kTEgϸHXQM5)ػ@V Z"?N+laU]_4py}+lh]$fa=JE#LIlc^3F=nI?".F%BG _+8xߎ"pMKs o.ME%摩)+k]&k#.%>@M )V$*#Zl1_`G}Wcʁ+?xgBwTܰӺ[Z.PS?~ӧiAɺ_;k 4v[أUak̺ګ$/s*Fz^˦"9-XyjǣUE֢ *͋Đp-@;!Q|y330ѝQZbG}.UG yF,3-"m+VT] 0S}Jwq =yzT)0@#.3A8Lv:hjz]Y$ XP$'т{mg&?@_ynrUMj %S%PD,oB+pD=f-ߙFH<% d?6& bbN.+nqF%VN1lfOrɅKb~i%hihiDKDIOuZ?mw7n#_] !l RR=Id!'xam7,.V\-ݟVbأ~?p7D0"ߌ Uy!9;MBUjȎT0E*={3`K ?dxQ?`Bz?$CS `5I64?eLl[+c~'szi3>%2ek+[{Ք+ÜThMemwٗ ̂(3%Z a>R}BI~3p+E\jx7LӜr:';JZe}f(%B'WM?<'*ӵ41yf Q}7MVmbc"6ɏծ}+R"5O.2F˞h}$g.@֢h?2a endstream endobj 1098 0 obj << /Length 4938 /Filter /FlateDecode >> stream x]KsWlUL4W9VN$:%ʁ(DN53cwf `_h`۝}}!_>W(rΩT mvO_ǛWf'Jmg_97zWt e릡4u sep"6Ԏv ֤]y4rhz*2=2vo^|!җ:X<)=z/7xjMv16}CKݟ.PI1*'S5'jeYnȱdu[ꯋgAP5}Q- D$…q>]vRX cJ )R*%IHMIj;PGiW?~T(?x]|ݻϻWo|]?\PmTdKNNBiRE3{ [vWM-׷QW~9MB nR $!j'܋n}W(A] Cm| *P4"\kڛɮ '|ÕB)),vjP[:ݢf/ۑvSMTJZ$5i%"7Tt)lPeQzZ26Y%>!(QR wJ/n()EHCSKJDM_6 .(l -P;LЩEeeoQ/rjDг.e;U7yÎBҠQaKj12E*t+*wf-tkGÐԏ20lO)`]] ƭڤmxmD2qqeč ش>{ h:\R%z,Oy!A}puO̙ŀouzn%\"Hbdi7B(ϋ9#1?s*Y(˯"PYK+ 1rHƹ1p:jg<\E;o"˜=TL2-@dZLTOG&#|ę/> MMPOs4uӇQ7ZMrJM eFj_feFj_ S/hhH ќ*7#VT{aF]amvZSehMԽpbS:c1j9 W# w2+(h쯌+ g ~0-DA\ccon1s-Ҧ$iԵ1[~ -IJ} Zqk@AKnӗ$5P3fHHVѩҩD̦f-Q3rՈ O+PxJ Q@9'''FoІqjU%<帇s.V#Pr|'̻Xm%RpkwŨb!i"8|m 8WX[\P4Le27#OݹETGmkdo&ՊZ:͎IR09|7Rx^"!Jߚn\^K;zOZhSVKj6ϜL sR&hfΞ zΛ ＝hʠdXwܦu4Wϕ+7KLMHHe$۵Llzv2nN&lJ0mVoNα}ۉۉdsH=d\3:mQ9Rvsh\&M־YCDtq#VB)h;GieN RN Nk7 N`Y˕nagYfB.;8競'*gTnuQIt0LVf=7>& 7Y _7|]Q#7R:j8ѧWY/:˪!&}zpir Yrauyd-HÂ9_tLڮfBE4ϤLg 5=(44C9}ovJ: kh7GAF }osXbm  Tj Pxvp>ɴK1vuRU9I\FPuƾ#\SXeLtV 1",,pe8\yu"X,EJ=3ъQiTRYnB'R%)m*9eVl *(E YP=!U&N)TQvu^0wZo9LmUUh'~PsevXab3rU<$&5ՔjK7[Qو/3b|W ۾c6Lܜœ"@9&Oz(F E=o򀠸q9" 2xΓ/{BΟqŋ=\5dvӴgIZBY} t|Y9Wf4MP:wy0Gw#qK|e#ǕsS2 Ԙp rd0{3gխX B6@w2v6+ؒBw׹qI)TzK ȷDH:ũRBn=)N"DOtzN=wu;RFu!:Jmj(w.DA<ȕ%}m_|] f|,]?e p<@6V %`O{P%1H߃sAOik1ub7PŶ*>ɭHZʃH*gx@HJBFۨ]קǐIL[ (-H6WXuмڋS;״ciF(XZ/Ēw ׃=ްLmµ-޷Gj +kZcebq&RUpw/sWrK/};૾ ⫥o҄䥟:{M@07$k`" TC LhГib(t̩!U6.d)f⦯fT@-O깸nuWǪyCxvM&z~wϪ4ڃtZI(\hAwY=yJnT{l:զt51H<`U#ga0bKLxօ5{.OnhJagZ>&H(oK!j䛷\{v6k\rGI^i!+BX4k+W~XE=cR6p~PFv3 1qܤ򊑄2|yr&Q^茚+d,#|9*I>1;&)!bBBFN>DC$'W:1Z'^ P(xrF[(ϪLӗw]~I$F `N(!2`՞$>aMmqhd[^F hH.=cM$kSg[lr>^.Řau[PBޢxoZ{ ZgQu_26:e %YWXx&*d׺z;GǔϦP޷~PI7C@PBrۨf4ŠmNM_oѶv#} \kVSB"|;/%U(0ې*QT2RegIՂXNbT Hl눗Q*f$CaJPee<8N&-J%bUoWa]>P)tU}IMj"fTc3Aiݹ6V/?~ Zk5ǼE!(AF)F19KU"LAϯ#Ě X#Шz/#PTVqf4#O著a7ѣ9=r,9l% }r.OþτeLd7-xs<( |Zl6-Ex?l=e"& gD >Hм@`:JdjƁ@h żpON򮴖/vNI cjV4"o #B$)Y\S? endstream endobj 1102 0 obj << /Length 3499 /Filter /FlateDecode >> stream x\MWcha  Ba-mUl$>zg6ծ\ MVzbAy&g_Y?()jxWw/v߯߾yzջϯo/g_y9\(b0Îp~$ErZ:7a؝F-?TDq?yȳ?|30rPAX]Paxӳok n}Ӡ$g/{ZT\w\aD TV'q>'Dڛl2‘hLfᣆPN6AeM8ͥf{%à /BX %9av i ;X3CD~LIx0:}Qge.{AByy&`UBTQT%`Ÿ( E } DQQځ5^U4=ơYHЉk:EhC:,%Ϋɗ,v"EIaxQZ2%*? " ׹"vYU lH ¨aih( ]A :;- AO4>OM`kQ4ue{ȶ\a#Vb ŠErC+=N 5Kgl<^3I څVa"epƋ4%ʛbV(#B5ɂL6ؔCI$|mNB, Թ{*3*~Dby~.ς2nD(7OJpXveibY{*BefBJH$*yp,rr 3niFXI*b_ i$HUUD!C$ӄ>.*=i@'A\ -tl"4pu)q!@^΅ǙPi]*| QsIUnirP!vj%xb<: (鐘e2aH8ª9 ?-KdIkbvFl9cZZg8MUo)˸O9H_JE ^q_'E&c'/txx=؞v_a۾3.}$hr'_(bAZl&6ObYb#g.#GZ}\S̝Eyjd'7 Yh9/HĊr1 vPC%lvZ HDžHdQ(Tڎt%!!{eGV8eGRRM1b+8sl4AN:⨅šSA8+m,pM Fs,"H!MfZMυQxӆS#J ?3/w(9ﭗKxPNɀ3g]{ە̧IN|l~9v)0ZOfe0Pj,uUUS5\?\ k 4!)Coᗲ*u$F0Ŷh%&kR)UIZa,v"Ize$5L=Ɇ~CeOx==-ਵަ D#eO+{ yfOn~씮u߿+C#Co=-]- і0%NSqzmyIW.6-=Y i>SfC%ԑ7#eIw{RcLm<[7y/ezn %[WQNkQ}n0a~UGkn0G;[<4o0ATb+*& 4ﻛ2?j*Vye%s;w˟q}qк櫮^ʈÝ[y Mj6Kh]B…D3RvKR:ĿtYR/C`H9Ɩk N*#gV"PYۀ_V+d( B8U\bfDx죰c~&]ϦxoɆ%.;"&.%㒸L ̵y[^xI@htvvW8cL'}o:ɺEbv:i(e013/8ෑh2[WiTqIO9Hfe)}7͹xv~q~E]}5o#R:Se42%6}3v]]\ve.e6 0=/Y&ޣV*r}b90!zDLSQ/rlUpF B0W]"$*/Y4"wt-|TE4!B65ر*=lЎ_@G `j3r*EEC\ɻ$ !Ļ\=B^ٸS<Ś 'EUzKYXe!(5ªoPZ> =Xԫ+qdC8E7d6' &)̭a^/ľC_5boxCiݻe9[ȯ;&Nq_~: "]ˀ Bf;Y}dIZIM~V=t\+.;_SѴGlYo j+ZZVƻ*JTig%cbvvd3EW\yw{uxQW+=K)՞K"@ǫo]>؋.Kم0w?~.]E F)(YRkL-dz;a;F^4D1cmxT endstream endobj 1107 0 obj << /Length 3493 /Filter /FlateDecode >> stream x\KsW-tD~$nRTm%>D"e}>`fP\! Um1b8XeXe"ToՏW?]w&*P #~6"jp)SBt(Ws/Xec!B Z]]/W DKTX%ǟiďVHa}VeҢH)I qhJDgf';Ch5W0v%L+lq00u]xԆC6=T=dL($M[; IՃ 2Ijɉv0T09ITǎjv ˝q+X}saLVs!<.-_A44h%i6d,VZVf.b8Y,j.2e]m<{y22eTʨ 5՜NQƬ6*bcG9L;I\TF/.2h/8#1 6bK>&c?`ތMSƃ*̌1!{5H9ξ?-W^9>{D1v|n}.sooWB᳜.LJa[ܯ]g?V#Z?7 5rv}W/nWǧ]3z~r_w cMlz[<mo/oޑcyܹq vYtn5@@-{= 9tV3sO~Ȣ/4#xEo)x%藙5G0c0#ȶeK ̔]MYX3;IYK v _帇߹+3h)J!ClvDW \gr{[NaYLa]4ыT }{vʶPNk He%̕pP ֏ZbnתDjUpĪ2+ڿ# Sݏh<7TͩqԹ6"Qi} #cjX0r|$/%DzN%t$3I(¨/(C 3yN< 17u4 ;OQYk;b<眎w:[Py&x0 aLL`x0'u)řL[9>x26%;4󝟆O>}CvClB1QAaZ^[ZZ.FgdSg{G;jxE_  ˝#ɑF%fdsɩXvۈ)_1Kq283(3ܫ1Г azZ%_h=鐁rvij qe>b@Ֆ2C; D vP U'408 cgX%R퀥 N;"k""QVDM!4~ aH&xtnM3丙Ap̵f vPӣWX(с+r7sο 73ij&=>+# x7X#}\AeҌ6!g߼s("p ֌s86zȕ}2L'jJ65 [* ``%5<%.\`͘\"E~lKDI} \  endstream endobj 1112 0 obj << /Length 5880 /Filter /FlateDecode >> stream x]s9r/csǦ\\Ε|ݺĘ"$[50/`ؘFZ;OR h׍,.d)+킒K/-r|v}K˖Ǘ+onϛf{?A:,VLeBҢWyËv]:ˎׇ.U_Q4 GaDg ͏lO(5$jFf۠Z6XL1qLR]lF9%:]v?^a=11j_?6KNBi)FDɌXZY R`@N QF3 ֕ )ۡM2f .#%& y`WVИ!˓M6M6 ٤LۧNHjfh) ]*B/VM2l0T.$cĶBL!Mҿ!'D|?ʇR7TMjn6O^~HW(ފS|([HI3c:d>y;e_#2ufkFm+K  I'Nna>ku?Ϸ.[ awam7|}~q rN (Q L Zn5Q&}V՘M+9n/ݧ.GqEj▕ixR֟&e+ 8@8S Ic1C]3G Q۝탗7عO7Y~l탫Aę,g/"<U2;o|_iH:!X_o_^;ePp>% fW(gR(K eh=#kps):jK.{ή?Ͷdv W!p[Lr68뒌Ov[T~spKRW2t9+˚wgWeSӮ5m>UTWzY…zc""˞)=kwce#a`ww/>5GD!hcrO!v,ZAO]FSlUB||- IKJ:xWxR[\<(r=%|V'D7IWH KDKH0k51fAc\ٹ`eR崜9Vt^8Us,>}8:)xǬM mTS -w0|q\^k@e`_wZ/!m ;FUu]?f4)Jn ϵq ++K+rN}_5[04// y~@zֶ2ܚ:S|ԇ?.ƻ~L̋W?]iIE dISɖ9&+l=e ze#SdxAp4m|aD""`v#oTfe0DL|"$)Yg(Er)h30,cLTQZү.ObLR1 6VcLfEBvum EnrazS)k{ Ybw? #\"Bif9F!b򯉛X[ϕ[~vwE'{pJ.m6W.-֏ ܻVb,Uܑ <:ݥ?H{㺯^Uqվq$pJ_6վ Iicn)=AQIj[4Ne̟e oNms:4޷rs Ycc쨫`G=F BjiOOS$Ҥrsݜ=!TnjΣstF`IXs1.gcZh <(B7&pTLTeNjEܴdhhZ DGTU;?մy L.rٖjmպ~= 3ȬTu6+Ow't@6+tɂ ӝ%躧H?ʧv,1W}cX LjL%BJu'RWPhx$bRguiv֊  a$>] }G?T٦h/|%a7h/Zغsh8*DN8M4W8L o7*>R;dH`ucf{:k/j&Ej1ZZU4 "Ӯ n4EV`[ehW!2&4}8BB>U\4( 89)3z0u"#ㄯ+߶ݹ,؋\uJ>N `0fKq\0nT&~"u!7ᡆ]ڪ:Eu2L&faAyxP&2OyELx4RTN$ܬO?4rT_#5mD+xd|TWkɮp~#]3T\9$ܩ&Se2RṎ*@6bC<\-hNUk0Uva%ĥ3rL̝Q1QAAH qit&u3pVM;YqkU8D-\LfAf5oߺ;qa^چԟu˲zsULqk->i`#҃ڠ)F;˜»\E&ݡ_ٱQ[vW0m rAA@낿Ӵ~y,ȃAw7d³38W=%7|LiOOSői8ds,G5C:v5kɂYVN +p٩ټ[{#ݒrL[.]$w;nq~{~;\oo<䷧1!C&<=ǫgD MoCݻb\u{$VHl[R/uĚ6T7*aIj P1T"/tfю &hl8NSegm&R9Rpjdw$+1Dբ(F.}_!pi1փ@tcI4"ElaB2f 8sX/"싹?3$=.(p'*cpNPv't\OKʧ/ d$@C HJ"c%3J  ~)Mu $%k WAT^?,RAzauh˾9XP/`4&$`uȄgգgpa5KDP^X^߼@MfԹ0tul[uaiK`fvC`%߄;+MָՐy L|׻&eY꿺x ELnVp!fJQ-;EZ\,M~7=UX`*ymwP]5Q`];ceO:dU=P+b~9[6)C:fC?9CbSo>QZjD 9W D̰ * =+ e"fO;פHl {YxjD}E1Dpg@451FY(>Qېa~Hͯmw', E{2%^[ 1=Qvj8Еnug(Հbc+Ǟ8}A~/GizXE4@F_ [D:pX:AysF/3ysP_g%u_%џH5JA#4 m^phOrhv#_wek*“=<'7OV7b\%p~ ؂jQu}hߚZwnp67ChE]~P)g{ k3A3'=}w;@)/]^9s(yw^wk.%ٺ~5H<20`W)י|i.ĽԊ(D;+1^{wr{ty$I82 i_^C%jG$ͤd} Uw"=.Np3].R}3 H+Vcǝ_\eLs56.7Wznb ],"- M^㮞2kIR,iT('݀TlX#ܴj+k]c<1 Y4.IXi&,z4,zL0`Hit6!]tw_m&/$޷֠P(~aX6_t9U¡:|aH?f@@#ݓs h_-Eeb *,-L4wjĚߤlN]?ZI8Q Yh80$=EC4+8L4N$\ND3搦ee'PM9bSD?ֻ 2N09mXҷƝm*T qZ|+S*_f+?~wXYXVZpHTj1B<3++ YdZ*XZgXiyӵ=Qٺi.6[-8 &SkLikp-˖N!t42uyprKf$s@zEx>0l|Ƹj ֟+h[wn endstream endobj 1118 0 obj << /Length 2776 /Filter /FlateDecode >> stream x\KsWLnr9h]~u>VM̵̞Nj8e:=竟}d*qJDtQq>,tHK>$x &:o*iH \ D^žי &VdĕU#}Mhb(Dq9ju?-ǯz5D|Fa=i  P֢$ɋA3hd/r5 -) , >E!3{Nma,$jQ {Oo`x]_5'ea0>b?HCȊ .>\]xqH:~F7ǟ#J[21 o~10}19p6@̡d1{PX/LeX-ONqp-tp;^mc\V50jIS&> q}6.{ۆh4ԩԖT`Ts4@pcѣa2L n|0t1ų؄04Xύ d5yiRo49B+zI,I$'6D uLf SH)3|hm+6Q2dcj큊p&ծ\aV:ذͅUtpp % FO6Hx'$E@ZȅH 9 yLc ө@aJ8*ܲC)?c\9,9֋` ճ!u%_А ˌ܎n @^}i~t0mgNw ?ߝ"9Nk;V B.qGYX'5-nIڴ-gj9"/=rzȢW|g7qpdN[f}S=& . wf.a:_e*2w=:"<9ҵy{Bpa:~w*6#)O٩m_ĪضJ-@(9JFPD՜*oܫor<&+iz7(|iEz%Ԑ\ܑ1pT׀]CgMm?ԧK}ee>ݗYet_[b86_<_gRohUdZL<5eC\x5]?61B0猅(h *ח{o/9ʌjOi + K!88=Q7 xA12t6cpx6_鸽r.إ>ūbp}gorZg.~OkZ" \ ^UZ% RI_~Gؿ}#E& Idju*κxxZI@ {S> stream x]Ys~@]Tٮr\)'Vzȥ kfwfbAQ^8oxng?/ξA`Cfggs1{u5{=_g˫*=?lϛ,Xgv{uO qg ʐEo)gZul)oq]6=Eul& M!|~Qĉ7m*V|Qve0PAf YGTRDB"*z<؄+]DgO2MO/3"H7#ft自jweQFi@U1т1xH0~Cq.kz)&I O B) @) R!?OjF¶ؠim+e؍Uŋ1別lmOjȶ-s ~vg?/?ٟn x>\#|ow_#,je(&|r՛]V|VǬb{])ZnY$EAǎG!n֕vYl%RnPSx )]DY#!L" &i}?/m_c}Yew].qW*Fi\r}۳F f'~AL!L ǵ6=;|&A!l@qy{ ]ag魛zCeSEgX՞D SkCѥ4(S{T,7 (IBN`VW}]p5뷀E` QRD(s0[0:Y1)4f"fB-wtї›]OFANp4f /Wh dS:{:WfE(Kkai7,篤?(ؤoz; rTl8T`'p v9AμQk,gBuV o<~d'9{MKݠ/@"A>;~r&l%頃r$bc_w!;<tt0B1"G4]꒾xԊ#V ^5l#슂!aիaucc@41aK+0`EW^;_ߖUޭn`8&94bHMf@_-ضh{0 ̚ ڲqjLuh9wV jV]\/W]&xnWr엫M7E?wғ6u*OL_`g8+Є&yQOntF!\c=D;6δ4 o~uN/W׬E4]ljKpy լm xVdtGQhslkp)֔A3~$.3+@C,~vq5ol:/)'r=d,ވnĉC((DF6[nb-SOre=aiDVxa"b% im:h-IlE1mrc%Mpe{}o#}#%x4$(8ֺՠُ"ّ'SoЋT=FV#~Qb܄ kriurZn.>R:2B'.d_bo0*ͪ4> Vբ+hҞ~?v8r<"_ZߣE!jEHs6*UߑԦYEOg۴H7~Qspkj2z1>RN7xYbtf)AxZVw6X upeA/%iJ;Ԙ~bF8oǀ!ӃOO6|"G6llMDOY.1,p:'ҽI %PĠa%ȩfˏWS`WLon.NݦG;ꎭ`I6.(?1s_ P&wo! ZM&jh kgwףHoa5K̨@|5 EN72^XE@A=sQ;!]߸lt iÐLϘ``笋[^␤V}qHͺ*5@ 1V-Y#l1t P!}|E&JR@kB΅L({S|l*XYeSp.fFDhعD]MH[9T}M(,'$[ |cR;m2΍ҚLz?%,-WͷbG ћ\-TcgJCo]aUتIʾ,GӐ"\?_6/rvJ[ 4$r'2(x!EW$B! 5T}^ORN۞79 =讙adV29UxǐɉUa~oOzyb̓eb hxV,s3±M1ݮ܌ENِ(FlvZeanQZxAqy{XTR^\`vB!Gf}ހ9iZR&}T< #F$gY9w(pk@ > stream x\KsWHɼ*ǵT*U%Uqt$Xb,I۫fH!1w . \;$LA02ؐCh8CxP$Ηcxۧ֫%Uza}ޞ_] kp-\,D\jn7nuX)ϗ갪S8O+mO$pIxCB+%1$T$ "L$H2YQJ.E@vT''yޖ.{w>?VczsҘÎGa|Xo7)^- Uj n ~hApϏcSqD9ﲾV!OM(pXSecC3w 16I۰bٮR&$@yIt9 84|Xbz(O++CKm`$x%~ B,&݊mJ6`S}jfՅE  ׃x1kӦJC4XHX%j}gԗnܔ)f 0inGYU1Y2i%lPG=/˯rObA%_WlLH`'wmuݯR8ϩMh؄8pRJd݄X\qXaNzףԈWS|DNwG6G GB8VZ}'VXe,B딿yV4RWz{i˴ 8Lgo/{fw c VHIy<{[vQ|l\C߫|V p1N΀@1fMR 4Hsu654AJaO V5xVzլf&OEUL2X 7 vIARX>l2~>X`qB[UW7"F-3fQǦL F,#עծ=TF Y*"!H<_r\XQPaՇF 2M4Pr*jvx}ͭ;(l7IQCMĨVo)˛f+Wիi nW}[y㪺t߸ e" $w!GC:yQ*=<*揭QQawH Oc:'G@x9LAHAR ?fAS` +2F$[" A>I$?pAdd]Pڟ**A<*III8^:Zz$o d*Y<oq(Lqb%MALmL@6b("4an=q"s[ Mt3Ȯsj",Y:WxLC&S\<"9 J\"$=_}kٙK#haZW=,./҇JM6GS@#d9] YWbz HS)s 9@.H*@WL'`%:DED" 5 @35pQF^\x..0aP?Z̀;4'E{ B7G{^ΙCdPc1BޡNvt@Ajh5; S|aj*7TOa8\8Wx'pȲ;pe&\}|uA|).QD ʀGB EF Z$ dHaQmfY?"D+)5 aKX0.[ i;$$-VODmFe2IG2 VPkn>kY#8܅$ [U)Jsa[O7$R-D`9(grx5lץMFS5j9_ݝu'ʴoiϻrSiNm#yB[Dn-ImkЭwMq hBvb%;DvKr)՗R/JS--|9?\/foTf/fY Κ>,dgMQ_#NZ=td>˸ctJn vn bnFPh'Y*E{?=03Ga'HJS,mq( ?$̏q;mhRZϜ@w_oLcRH>r|>`>FL_wm~@ξk.yL  endstream endobj 1136 0 obj << /Length 3439 /Filter /FlateDecode >> stream xis673;Uvjqt3ݦvv6EcMt/ Enj HtvB\|}b"*b.$..#"%xLD]̣W)%L^SlجwѾ"JbC _')'1'ڽ}ʽlҿU$|WXPnA+ׅr(]Ll Nis^.=046ZFAϯ]7]3cŐy@j]t]D̔,҆J$!mmṭ,HLLR(V] :%sF?gmҮB/_W:f4kP atܤYr #ғlj0*u|v{Yf5]플hzDjw@gz6a:mML5mpk4ӫu#ܵ1Mک}ʊ:NrV/)KrVfş#Ƥͺ?2{ȉ)m.[b+iM֠bmmҪ A(W@JPQcI̴WILzamEr:KQLg L ]Z˟`ZĔ$o67/-`.s;?}HkS&b>,V*kUVp"έXi"_|籹b]twV:TYl*.L"Y>mD4oDNg&A I(uvoWRx乥rkҺ\Ck~&<]@C81\ Ez82%`e~0xfOI|+IW|-L4v ڜ5rxhg,2;,~7e$H % Lnh]Jj CUV/`˲@Q1RzȿyjPD`SX6EWu\B1 sv%H˼A8@9 S{fz޳q=Vp\ep4@70ْ$Z$X/љbK>1X~z{Tī" c>#qNQXɟdn'0y `a21FSJϩ"(cX<*$(tҷ8IU()%ğs|~g g,WOQ Xk|h]'sRbz +)ךa^5>G*4ݼ]Bs+f[*~kX>A% UM;<ր C1X9ENܥdrop3< KœSUFXm1EZSG졂Zvz4Ӂh6פ'xwVn)\nֈ7 M}#:Z2hYKYK_\OЛT GK6Q?q+CEH:]w*X,Zn@ԭA}2 ]ɗXAܵ(DkbUYv3BU Ս{АjE\Vveu̮YAsgX';$kiUjQO{ ؅[/EE $BɶA1E.cS2|ˁ,S5s2v8Ӳ͉ΜDu`:׶V$Uw)฻gXpxj 9B9/j:/k6| LV?p#d缾 *Gh[ƁTKvg?6lE3)X\6H.E] /}bޫ3N_[ػH*h׵[> {>rL@r8<;8ÞacohKU<4Gs:cu.(0 Qci;ӳL' X jx_5S)c .G8LaI?j*v^s7uz ū6~݌xEkdQfhScтeE 7?h`ryEzt@?4t)(ۻ 'WЖkT'a|yH!c hr彠At'S_wU"DԭE'M endstream endobj 1082 0 obj << /Type /ObjStm /N 100 /First 952 /Length 1578 /Filter /FlateDecode >> stream xڭYn[7}W}/g ubikqH ^{KY -E"!/r?R*HralXO*H&8AAW%-L UMULU"&Id#\%HFɤ}&G/fɕ.)>iMK^YП?.~;'t9{0 żWZYq†WA Q@>ur:K :`T)UF@$ux=ʬ#u9Nu^U ;RTהX{W.hr0 Nud@a*,JL 9<*+shҖƾ> KBK_tX**3-B}Y-_053NWM hj;8G%#<ޞcU&_zP*X 2 +Y`b(eVUVW-u$D&o ʁ1,(K 0" k%ׁY͢X,X-|L[l׉7)T6wrp0ޚ3,!|??/'6@_'/_n?c)6"Yl,m ۔{#Y̾X>0ͰJͩyI ?}CS1Y7},^fϖr/3;˼KB+KunEeyLևmqĜ\ n0"Z3n{\ɸ&eFD32#7'7'd-a+v9ˮX…:Gظ,g Z5@/:{/AwaZz~y25gfx oT3Ń noԫ 7i立oAHsLtq l6٢,M[m} ןו·͙2s]%E=Ép!Y-5DG@;1/ӹ̆U'c$=82 ߔم9i뫧sa"lt?_"t=C3^bc.VV#YK$`6w+~v|t @.X/v6wRҮKndʍ endstream endobj 1189 0 obj << /Length 4156 /Filter /FlateDecode >> stream x]m8_>0CŒ- 2 l7q5!ؖcىUna:8I*^,G%}/=' bb x /^/?"j.BÝz@R(ki&gj,KBb^=ϯK/v- ,RǠ2N\f)R_??wן]!*MX s|]aD`'=%qdSN 3d8* U$$H|U N*?GE͔;G+}}ׅsDzr*5d;Jz8O+c't6zJit:¶sP-k#g|@T9sNJy2uv\ۑkqYTUU^C'sdUh\Ci4IU?7'#"+{)2)+Q,d]aՊr0 q"Œ1j%G_||m4*Js]:Uylt%I,/TS_LabBUong\ݾ:ܔiM#Pݪ(oL2c0®,ҤΖndu:0l ňMUjb , mδ![OI]ⱴxID*Z5b%;V$VHG MZpJ=q:ՠK!YvZ zM_aL|n򵽷ZU |=U-NPH@96hPF>~'~v>ՁFlJ.ە5!b 35ЮPMZeSIF/n)o3kQ)lJdmƖVmU׹*Ul~IjI1EЇ :YP?W LJI^jO4=sf!1nZXa lk_5@96 "*j{_V"S(,XNbvx?. Fi*HcC=Ya+Xd4:2ޛ~ȊmZ`M1J٠w]v(^[QW7vţڀ, VbRVG/Ë5#&]jBzjx`=%M~gwX 1l֍욀e7X(hk[j Գ/"u@j C& f r(leNUjcd&Adt$`?̃+p%G*gbŤW8OC#`: /81m{~F&vֱ̮>ުW}˶oШF\ď8W/G6Yh0  ,;G&uoo?ﲭE4@sU!@ 94W& H41X86X40PskdUVF0*2dC4DvX7xYv5?>^"dxI/5k?eF&x[Щo|kNl{ g}^7E.S,P?0l0=i?eg^|?#x7ܺ)'LNfY~SmG\@m]@7 %TE+@Su$AIjCo<)é)Y٥t2ATQ*FԷÎ)~:3cU[ianMNT{7&0:G$GL':yI$@ f< DimpV2BQIlIR7_\l̙5Φ#׵+6wrv!S}jNO'| LQ %<) 2LzUfϕ|1W3P@5Cv`s<[s$y|0aDcLn*ێm+=*;Ht„9Kܵ~ i=C+lrh*MLUp>ۧ_Z?0kvgGGjwѐϕ01IS2d|o\zfW|#2{*mYXICUZbwwSq*H'?h Dbwt|S5. ;@*H5Uf_dz̳0MEpR'|UoFH&uUCV=y}::ExuP39aOڠ1T3eWg39;~?Š0p:վWe^/JH{Ɨ F~V"ӵ^f5CbW+"qy§bVaI'(׳a7n0 nn)K5lܝoFag݃+'lP6g,+?}g@rtxgwPϱm&6IЊf/Il\[2 9s6٤9?l^F@Z0Kq\/) e9*L608;{+ǜd=gO{y=xNUիs9cvwg:yS i SWLU~4Nyi.ls㵈[WwJru XVOKv{w<UF~e/`ZgowׅlFhl A5cݓc >0lsߙ~<[L# b_gTzT cC[Dű~wΜWfqWR)NZhn:3gJş?Qӧ:T]ΩQ p~g!y{gI}Wj/vw\  +.zS\tQ\UGS'W7%Q^TnPMVta"?3-iQ#;U> ZVl;iր%V9&TN+V{]<h)37V:K8{s]Re5I~feu~϶YK>@`#:=tɈN_gK0`,֗uX+;Ui*AFh#29;)TG:3u  X1 endstream endobj 1203 0 obj << /Length 3420 /Filter /FlateDecode >> stream x][۸~ϯKQM6w2AE mw&O;c'ɠ_"JmIgݮ#IxHQ/{/_>>y+yMOྒ'(>^.2tvN^O[; Ay_HoԠ.99%ЫpEZ):Uv#0 h0nDXv9 a0J2\̯@d|,=#+xbqZ~ rмA37BXet . X*j!=8^ :)Xov1ӆwJ]3R+ R~ ϻ5w>QtK Ts Vѽ \Q__[JU}ʕc l2u YV, j"~E MjgܵMM8{qeϡ9BUzѐṳعvjDu-'y'5=ץp7Y7C@DPhEgY~Y5_N%,hUOK3)c : H1;a]]$BU ]1N4~8iхۢkPA׹ُQ`>>Z 8ݚdP~,߽@T;\'i/l^ xnX$z@CWiAc WeO@'5B5H[M$K{ +=02R&rI1h fD襻$OCAElM#4R U|08EA:Q|-nkpk$}7{|0jN~jQ pp5;^B@uW.ݕԥm@cbqc@clPS5Bl bT|n6g6ժBe0`fJ6}l9<1`mEǬ*:с@t Dёg?Cg#. Į/]x ʹsh?$u\}*5|]H6ՙøZrC7v-0f ]gQZ#֮Lv^Y̊|+ceP@/8h_{ 4.De> H4%]GosB>:\a*`r}Ѻm[Cܠ9r0ZKmE,Jޝ^G(cArim`_(Mjck #@פh5c+@I#Y ˵k^qS(.IL4ȷ!RI-ARٗ~xS5e0Vjak5=;gxSpfKBwس4ۤ~\qл3X핾x~V8+M9mL-ir֨(RmT"D aʣTj{WNEW+^fv8Av8rO0 ܔva >j+@ Ph I6G~/|!\<(_~8K%Tt-=1@]nњ9vv;{Bwd ŠFΖU|$yX/v } զ5A3K@Ps"P,v%jm薁ƹ 2UTu-DIU5e~Q]Zt@'F"OlӪ!""KWk ԍϝ N}9ߢ{>U\WH7ɚބHԠ1m͉YtR돓Ω J G q+ qS5!̛G/6yPD)H JZyʵ4On^BhI"E"N+00溋tS0\ pi3"!k3f X70,G $6O0t),XtC}GL1۹6{讀,Hr\d#a#F!v PZlCקHX߁UH5C3ʶ:)%vX5IںzVU%ċKE!G s;,W1!;;]G8,vfՍ=~i>Ͼ&f̭XJk[?n]ѷJ5d SN}s]ZɧcSҎ+sDvƂ87vst -6 X8Kk@Pe4JGT!Ckg'dMo9>ܗ8tꉋ}'GΫT !i-?=2Ef.x)*+iZl ~u@-y?+6.j)UD*e5>]kbU Y-*{Z*wTnxc"-Q ɤ~>9d#ٹqRvͱrqJ$2e5ALMZ =Ke{ܟaX.K.v;ֺgɱ ("F{3c첐)QRo]s)v<⦔-+ǣc6ZfY%2JBvzybېZւCְAjY㢖'6o˩^K"!x\,j"6 vxyπOiq<=?P<4>tG;t3{l&'7'n7wgPv4Ksh[|[e-⮀!o~Ħh>lJbpLqFI"Q:qm\mI˗.x𥋛#vÕd8!{H$l脸MB *#[ qbd+%: NһrgȪHɣ˔,wI6أ6$Tkn!3SlQѠ$G ʝvB7+}o>Y>э{Z|QJf#]<m;~W$s18fy[mߌx$i$IJ!.ơpj$lyar$`$R!$9]doͲ endstream endobj 1183 0 obj << /Type /ObjStm /N 100 /First 967 /Length 1483 /Filter /FlateDecode >> stream xڵXn[G }Wc Y#@8),B$ Aӿ!-%!D%yIV*&ZyJ3ՆM-j"u %..IQD t^^i7`@ynf|~[#󏯮|bpsfusC0_0 eA[ЎLT<15c2xꞧ͋MB[oZ?Bo{ݫMW}p?u8W[qn&R`k9E`k9g sİ@15=EU`7X(8Db:x73e(d v& 11>vq=ck\Wgo+WkBnѫ};?Gg-vk5 þw|z%2ctr|NbA:9=J'G `M_Um|Q_A0F=vڢѠlRl Vn&,`(j G5d N톲}mr[q endstream endobj 1264 0 obj << /Length 4052 /Filter /FlateDecode >> stream x]Yo9~}X@Y{x9@& sĻ EGd;_lMUK-)n/t[Hv_E$:"OI ŒaYOI?QO)!|vGioٴͦ'O~脒CFHrIIxk&mb-h[ĤFQےkዯCJtj4quTJLi I JkM@|ҩNexuXwUVV]Ke*ZEھ_ζBj8\ڧA>`9(vaGe}JU,+2 =̆]﫷u&dyM#F Z׷u%)M<_ORI;Apq:lMk(ՀuQOc uXݛT)mk-OKyo6- nRAVTe E" RILDIV7J3շɵ&PV3yJb y*iRd[פ匎:񡎌 8_In2M()p%/h+V,e0c z1jeLǑ}X_ {p ,qݠTP0VA3| l#5=NSl yIV{IVFvΆٳaFhb0zv>֚,^XcJ7)1Dwf0;: U_^:ZB؊2ehJ/zj=a'S{A'Rl˼a?^&|_ϦK;>:< [#6<<SٵO|<)]/,MiT3aW;̙Qb)e@m]8 %гKX^zbGZu g%GѴw>_ZOkڲ~IfkYr0uQݖW֣!Amkч$/NH*,.\IBBDaQ)i+ɘ}UʫsSi*d[C5(<"JP:'\OzlYurk{NjKx)`X_(Ȧ"6'kܔTNJ&4a<6817eBLW\x<^?SpAdJdVn\|ldL| 9]E`A gm8w5u]$ ;U#kc*ķ]1rwj(? sT N/I"xfAIR4ɋ;G ڱcѫ?4pɇcU3s-uxϹ+G+{JK3WN]tϾY6,pX4CiΎk,ƾ0Fi%_E Tn:#I7EZHg]YHtҢ,hzCԍ:MkIv,,eAh#DF?R4jI# \F NV82sю;:QIɷr)(%;Iai(g?Ŧ# jYvd8EeV7yaB56 6CнZ1 (׈T 0*ag_ & +c aI<]g`>sglkh6%mQv IEgkHq$R\f r03Xm>3rZ;v\bYE8v&}?J>EX],_IÛ΁]y Pws^* Hހ(0uQ5f@= \ztj욾FƃXQ &+`+#C70 lgOtH{{S+sWߎt+8w%~Е) W/ض-%+)4FZI53 9"3Wh DQDՃF]Ukbw/ni B ok"Zd̓}#q@Gs. 82 (jz}ʗ9_?mO{6'TIQ,_ubKw[b-㏷bv%W5ayLXoܪ,yWGzK,t@l 3`k`bQg+Gގcn8yvVruJ>C-6h! p_WbW/HX<ꔹ?{:3$giܤsޕɴ ʮZhyM5p "@*#mk {M~l⹷s&s+^YvtjPrQمت=&޷Z33"<Ń8r^z4%c8W`x8wz32pS5wzt5(wI2^W6-UZ2<.r&+R Ҷ5@ЊFÅ5[3Ii@\[2*nvB{߀Y$x\*Y"ˍ*wr0͒VpgCf\(,URE[yZMjA<W +_U5+mZ;sy];ɻzF =\`|gr;ySjgiIJ-cv^xRXkzz̕/B٪he$  縓ڇ((@"t`_3S_r3thir4FZSդ%!ŌfنZ~~op2߈Y.ԕYh.dp"LQQ4 M\gva停 8SX]A#ö=k"{vF!k(DXn?6] S`g'➤Zfs =, #=PMsg-߅u`L]0yepSݱDƻ ^K+_ĕ"."Ƿ25 a}cGnSeR;LŤ-?HB{YnQ_X2N-U{CǵE)I 1@?Fu*H9Oa—'G endstream endobj 1312 0 obj << /Length 3602 /Filter /FlateDecode >> stream x][۸~ϯ0($mw P`eƍo=-+R<' 0^CI9߹Ny"٘S|zW/~x-u'*RqG^^հǯ>^(dvVH6Oz" so;.0Ew2Nl1#WہreV:.m٨>Ѹve6ݟ JեHȤJ\В/ ѯ-xZptAfIegI m>rD;,^xS3ꥲxy&(rw4-W?cNe{7N}7(8BM-W4V٠FJLИ ke6䁢|R;Vܾ_BRB[0,T~[GƜ΁ ~08QԽleR*f@nD#H:Ib!2ɢT/i%3F0u ;5P='5>sE2HI,HZ2/kJK1KYlî/췎tDBT`!ٙ~^*+,@K\\<+EXxse-qb[HʈM91QJs-i=<@c--8 %#֭Xo؈b?\zpHቻDNc86f뫞MVLs@2AP`FVb%6ll}PƱ3OSXH@)+ l%s^۩wBLb‚B)[CK26x myϔTP7LP0dԿf uEmUnYSiJкKmbMzp@O2,+W@kD57q%hhzbo&0vKjsf Qfn`n…etH4},JZ>cTQӖM)c\ @,طc#{>-&.uˊLsq)5s&jŒ(23Z ;4}-|eۙEXhBQV.m[aq3Pqzk̥s=sMe&vl2c4u@YCZVo5à (X엩@8ak݂iW^\t Ii dQ)yʭWvIamǟK P}5kA.sE r([{tEU ;B%ɭYBKj 08E=VIm5C\P}@›V<-7f;׏-k%#!DM#{>JU1*[Glsh'G%gbK~4od4 9 iIyg,Veq1CƙC6 Q*T#Y-F4gf?t|F>`H bGZa F(ı *W ̖󥳮7$o14^zLąOŽsÎʪ#v# ,'޺' qaE8G&yb~gqMY9tJĚ:kc'?SvNn UQx/O*k`YX{LcAÌ1 6  nPjprjUBEWHv|JL2O@YX6EY|,PBؓؼPQSxjY_+}bƦvڍY y{yOKn{i}IKGz'uST_BS=~]r1Q>xF!'G4ywxyS_%I'`r<(t.ucpb=}}c )5=nĭ) #\;${f&ʓdt|Ğԓ9,{ >{ kY>vȜě9N>u*l7eب(ؠzƜܽ>ag1'wNlRS~BD׌eeuPk8r>#NNu k7֙HP$q26VѥR鍎)u3dCNrY3fFgdҀ-˘Gq;ͼ8}b/|AegĨFVԤZ1E)}bg/UG0ʪ|4f=snΘ9!ģԓ&9Q) hߌ#RQZ@1|w4W&R͉sajڛ$Ja g'z `/+_:xʈQF;>gЇk?3%o{r9'!MI0nE! zifu/ΥȨA *@va endstream endobj 1259 0 obj << /Type /ObjStm /N 100 /First 970 /Length 1418 /Filter /FlateDecode >> stream xڭXn7 }߯cHrI$z1ˢz Aӿqi٠;9st(q(R ziu? k,VFeVZ VZ[tbQl,d@ }!ZF]H+B٬/-<\j@qzc7=.]ŴpCqzԘð:t C{ "U3V'\ "-l؀6{ "k2`Ia}axkƽ0kx؄nx& oLؘacZ*PPkxO>.˜ES\4B-Fk. A0DMll00^<0_l8h/!l#0F8rhjuL`uE5pX[Ѯ<ъJ[t%*0hsaa _ Ń ΁)lTUua1la1`c9:l~R?SJVcʃg޼9ܾdNȣ2Dۅ$+}mIVX1r\if!/jkzK*#PYeo4*JS/PdDʑSr5T5̃-x&S9r3xvQ8vi^Ǒ73]=47>>߿|('eq>۽(Wmm`iwv6o󗻷dGWݿ/'#)0c❳=N;wgmbnO߽X_ݟڝ/+|}v{$KLI'2a_+]Xw㞖7gu}^^'nG_ı)a&D 'TQ_:2 g~SFӭQ1K}Bt<=!jouS@vsdSBm$7VʂX26$iϚ iFR-fgd'cK,XUfM*4hά2VJUO]NRrd=,N8$( !E!4(g쑼\YIr2nvIhdYU endstream endobj 1376 0 obj << /Length 1845 /Filter /FlateDecode >> stream x\[o6~ϯP^$RZ1`[۰k!h|o]0俏Hvs(@P-^s9fYĢ{冖>FU뷽3IO#œeNe,:Db_XdQ£y_|("RIUi0x:x :Lli;4xzzI2tV;3,=d)K3+S f0P|)"sP d|8^a o3 b Jp *>+k=Th!'`Ȅ!<`ͨ\-,ooXS)Ü2 seݓC4iɬFE 5[y"8נ'{{=+D#c.@: Oas?4YɢW#35Nu ;Ln+`C!W/Y7K&@^ %s^T UQ2>4#oqWOՕ/S5sXS!${#U9ֳ^Ý3(/43J"Vo7%e7 WH+  !/!bXÓ):AÙ2`O7b?Dˍ\mO,YC߁r_ztFQa(pslQr 1pOCK#odנG}' 6WgDHKRdl$J#H]PgZB Un*eBmU6].AD^G֑%WɶsVf^e*|Rz,dv߫2u|:ϖ$*#"2k\lt Z)ivUT^I8ȼdgW#Ց $8T6 sHfмQ7!,W @)hU@ycUlω'=ƧCOwT[bC*+f41z&ߙ()*3\VQ]y?](0Yox~.qBy &5^UY)酻O/a=v1,rtڿ]9Kt~Ӻƍp%(,9"` 8GƮ_xuxKwHJR4Z6̊܃C3Ɨo$תll't|i1 #*ů +ubҘHRUO>4bpΒ|K#~cͳիZ4 endstream endobj 1406 0 obj << /Length 10169 /Filter /FlateDecode >> stream x]s7rs{&ʱt.'K\*nJ{^v<L3} h׍F#={{}(_GgBг/=S%)g?~y,Ղi˫MqYlUow7z=/ޞ׻_whW4Ӝvӳ jrY7_sӯ]Q}qY,ջ?m~w^-n6~Y<7G _#V9b{)`gIDHB߻w?cT$\˯ ~9$,n.20W~7+ 5WYwlshBDjr5m qX 2I;:iC 7Ǹjx NJ$WiD.4sY=be {FLJ+:)%i',W׆YE4mH.' [5!3}*udl&0`J[ϙӵ6fl,K~'x6qU2JmNy#"41iBf#*Y5)u"_W߀L.?6ǸZ 3Ǖx0S7><4cflC`^=gv{]kV[M͆UP1nUouJ$>En&K`D x}-E1!)*ia{TrX6p?, Y\!3;>Q-s\iWx>ZАar,Ǐ05nbP0d`<+eEdXXnbxv\Dl(@jJ24DZAfs۟09&=p3cƬ N6xFN`tzȉBb}dKr Y\93{嬴3fj|.7zD ש b6R#II F\Y|ɘZ^z^c YKQCsJ~Emni%ߤye[>ΖSJb<uWQx_-JG}УESCޑȱܕvi7hd;xC0Dt& 8.&cX4=Oc.;Gt&ՙ")P;*ֻTW1~=xy}].//6Mnc\o:XrߤnQ=MA$bNOǑӣMrκqf)u;@ĢS(PYijGÀƓ qδ̀MVA9,P U4niaR#:4+RaQ@x n2]%'6X51T9c8T$CI5jȶ"Ǽ :Yn‘N4<(T 󠈿$LSԢ7ήXeg]zio#;Ͳ4CvfNc-괶􋲒QOD|hBhQEy]`T}]F. #f䬄%^G.[27jxXpi 6hGk{>_Zג 鬥J{kpӊ/0] yR TX>a x‚!'AD-8I G.S*0Yy}K5ճyHid(oՉr>LD a8ypIB0>$#A/R6.& dD9nL><dc*0z.z1In J5v™޼+~փa3vQ̊TZc8UTq,Bmg:PN xrpTG{5k1٭#&PJs=> 4wC:"0.Ik*ڶttLLxw ]N}'3a98ZcNY'<>axԧf0CF83X>Fm'c*:9Q|< z7fK}7M8hյ!8}f(&,=nxRΐaTnCxSӷ=u:Sԉ)Y7G49E6åZ,o_:cru1xE+84?c>nV`#LFqP1 ͂6\˙JT$SbPD)f{b*+J)(n5+|fo<4oh Li"t#zlw0vVւNd0"f P'Dj1˃;2/t~ȷԐ!uwܴfJ"&8lDp}jb@Y`8}x9?IsK=_۟FY V4Re д<|1oeRtd߸v9;Oe|q6GRQ'Ļwh(O Zp5S-*$9'tvPR) ɽ5Q\`MS560Dc6vEۢfs. p4R^?B;i:iZ>:SWf"F0ֵU|V,)]KT8.~{ Ifhġ7_ďZ~4I\W+b: q 6mZВv Vxtbjޓ|ᠡyyeh?6`pF౷JQkʂQUvwYj1"ǖD#U+EҶux6}G4sV1)Rwo [̡$JWoNa0F6EKaKMnύA\fMt3v!hSQટZ,ܠ=n}iVCXشD:A.-@KZyҝIB11hs NbඩR}gGYҪ3}ͺwHQKzXauɯ@@r~w((!Hr32h&hN Q' _y.!XP\oI[Ʉei;M㽘jaь5+η[XOfKج6kE-<109oc0i4?bn3)E\˗XDŽtVnIW⼸] hve[M?Tn H,Ϙ:ǡ{jQ:8I?衦BV0qp+h1ΝZ%jdt\a nCӰH𮢮jsdowP/5?9 Sw"=z,F3pi<.L'Rejx|:h+Fg+ZaO0t@QiJilBL娏`-2Cb4g:MpԿk+]Th*Ө}gFÝz@@2CW$5k3ۖt8ܻ7GHq0: \An3f->UBZEK9eٔm)HI4FyR=LX.WZy1kra',keb:"k];1&iÃ㴸iZ|uYa뿠0w a+eA!yW@D+4O;7MѮhga$GS:8$O'U!yMH V'ΑbVSZH$f[gfO,욶Japєe!ʹ,dRQ> |Fe]GH2xȃɠN1)L׏&s8n VWBϺ>6*w7z=d]4cՌ<.QupQRlǡ~H ㇘, ЊSڔ;%yY7N߾+:" CGLlN0?f'2oZ*nS~Pb»Έ˪l2s;;y@wL?G$ }L Wr8}:.sNޘ<~9b&<r)$Vt˵/A+_r4ؖV[}McnsaJ1 Ӏ9T2A2ʇKƺtW<"퀾zVA.C7RzBbzxZ.*ɨ j mvOo^<::er޾ ȸ=8#鋮DS/A9&M_< HQHM ss\;*㭯)#Ef=txDK|:ዽCL{Pd>[բ/ i,d^TST9VA+LW6p-G. t<NČ^2OR>l ueFu1fUs{b#Zl0GBn^G8lhF- \;qa<:&j oMS@ʢ81F2@H3ӯr Gufq}/1H2d|IYc#&oyb'ަAS]a4hu.g?Dvg~[7?=KWZZbW >z~VgyN=^I7G4m^?oj~mf,J7N5 n /a]f^t/̾+aFf֝72ڬ/WZ+=~*-،Xn4*􁡆X/=ɘ8S21t:Fby-`rݶ^^xK[hzW "=;ab42(V$obwS\4fRМZ02//?hF_nXv!҃LmcVPe#`;,[XDcWk[kOհC?D}2 K;ӧ%׫GI.Wq}WABT}suyy^H}S|p ,x3hB8K*8fw &UehLta]צђE:7YelTykQ+ 4fɹ]ĚZ`o.ћvĹƊWZ-<\Ԯ O]C[FO+ ܷ^I;6PY$(fA,W20q.cU wxj.D#4`m ϑ M4F"#.I1W1d!z7$Ѕ$ #6J]҆O"Jt(ytNEazZ=Fµ܌HJϕvV_ƌփ1~b_Ob/O =w[5{8l,ƒk̓zOcݷFÁdw?.JoVlY)X}7 =7=1gss-a{AXknPg#FHV$I,4MD.M'CS6,CSy+1ħmBT2~ R5CS5;EӶ@a0N@oDž+ץT~el?7},ecB"L+Nv -<ݡT?S+kP?.5 'z 4az! U_Rڅ*E'r6TN)x8uF NS.2&?klLM&7o]Y>L:Aen:g0$dnAC|aT/makA9[#ETDFw9ঢp9U;'jx $JxWQUy7qWEoy |XK$ חzk2[G[?|gT\hvǪWMQ#)/PVy"i.[w8_]כe-Z^^mC[>ˏv[Rdx^{Yޮ:<wz0=Z *r_2"zT*) ߁Y[/.)!$&9V"a]{s .LΫ% XYVonuL^jme-N/J (={ y긲-Ng!J endstream endobj 1414 0 obj << /Length 4479 /Filter /FlateDecode >> stream xێ6=_"nH d7In'A=!dο/yx1u-d2-ṟ3b ^K{bA)?p<)ޏ"C$I*?bEst1Odzx;ͯe sOb]gGN(E"* 1SLSQ1Uj__3r>c4N*dž1GѥZ\|(-$C1>SN&2_|sreV*tg>b>]WڽPM+_Иt0D$mu_a>ZZs*A5at N$b,y=[r"L= Pv TK3XD DI{B!e=z' 7Kۻ  ee0 D Ț0SaK( R06d6= H5aJl:Ǝ3vh1RDE(efHZB;D^$ SlvX5'S |u{ZT/up%;Sk*gfekn(_c׽g1@7YCwӲh#@@!O[&(Ō]@I~BC{Q_2E-F˻;&MyY@Q J^zw+OU@RGQ!'S|g)8e 68qU+naw*j5 #jʽ쌚QS=/C(ra&h ͡S8uui;'rZ >}Mŀc4z׆3A !ǖW}thMdr@3v6]c@ۮ0r.M٥3_;Ҥ.NdxZ߬a,6p 0 GĘKz\^/Ѱۭ,az 9h`j_aMސ3'* ڟ_km_Q(Rʾf,zek \ %s{7"hHZg@kg4Y/T=$V-TW[$a$(} S>Lf'si|tcZLH*,TZLzā4S4jw MQiZKbѩ|4%W*R+ʂG@Y0k4g㪑Bcof"ʐ`ECW r&Qk4ͺt_VLH&@鹶?BOйAv~hX(RoyjVFfF!X<uI5 sf -KJ#0I:ȑ-"Ęt||*:;yQّڽOP[ ԤOML)={Z%E0⒊9 fY3ޖ /]CGֿR NQɩ$kP $lo,a<˔TH򄖁&ME!,.!7! D Rc~%F]zcP3ݪJ= CVY?*NHVsPx:HGaүd$@ =,u! NP]B~ R4ө9`⻄^C밡,b\f"<ǯ)>*̌z{c֡K=,:Tfվħ#2G³a >6;RjG84Sߔj)ˬQOKϓ+L{VUF:KmQ %!ZrJYi4'[P.^ +y Toh2_+{:m4&Fj?h٧"o.Gj=y?WMi{S7u.@jzpx(=s IO=y wJUaw1_ِi<_I͙Ǎ3c 6#{{IHzd.i\Ɓށ= Us;2U FX[z3cWuc\R.5jfW3V@Al?@=uXq^h3[|ezƪʆ)7`?{׮_IT fr`_ d&ۆ޽- ]<9GL)#lT`z%Gβ ${ywV=>"Y. <{"pj0 /cjU }jcQmkbuGڳ#mx{_HuԽHCaV X}0YZY L,gج!WƓF #~!mXV(lY6ro,QMݺY%ъ? > kJKi@`PYe˖/`Fu RS@u9Vr e U\ͦ\I{].`P)aGhbU?Y啕T8ZF G z>E}[op{( =MJgz5wbnZByTWH>CX8p`[ٿgMh+S?G u@ uMD;'"s,xKavnJ}W3_me9dŃܞ̌ .6mc"ڎ 4\)۝𑡍O׎%LZu~㥟DӤ%s`_r;h/M)(7V(6=ѕrܹ9Mi7KQgy7䦬ܙ ^Ylyw#e  _߲h{*9-i17A%qN&zzH2D[;lD.,GX]!AZ+s%^b PSh˕${> stream xY[\ ~_!<"ER,8 6 A,;]7GglAs$|(\JM9q)-1?.PKȔH6%%MN%ͶB&UPFOr4F^=*joHj<%%:%zKVSSЉ@a..ʉY ?\ څ #2GvLthw :lZS٧Wdo`X1#IV #K.4-b"튖tKR {)Ҋ *SNt42h6eAGW1):/b0 Zn*SfUPg %XEaά:US^):|+࡭Zxy~r?QXZ;2 xǘ2Iu Z}4)3ڔՄ~_29mA ór2Z֜LǔA)S3xkj*]}?S0P Cd˫{pQNh1UBĈ G%X9w*9Ȍ!O}1pW;ưM<mcXd%.,E*GYZ9S3ީiYZ6UGOL(f\@N5Fȓḷ 3w&(s#u؆I4X$QQfE$9*Ѭy(!hѤ؍,.fcwGt:Ow7kxbZzswUb}}{WS'_lM ';|Zl6[]k,;j'`^g e)e)2}9B񍟥q̏c{c~ṿ[&:ꚺ_q ALP~HiWmd"˃ayzY-_|򷿿vb֘rf]!}vÅnpiyVpdl Lk_ ڽΏGk|6K6=p͑{oi˾8 I QOءh8ַ+]߇:}9rXp{c;!!5^C yiwK6jzp]]\}|?߯ԊaLwɫP1U=U_ūзE{@XrT_eBT?jRH"NJރʨKN4jeEQfdQe>us }Z'> stream x]mo6__>ĪH/j9$z4jv$17ZjWjmY+ ˼'ob4qO?7?p9ITP1y?%I2 O& ?]C "J8_ K$f'"aӗb#DDWOwo&tOVH4C 4=o e28>F~"u O'u>GZ(J@-~KL/ R:fXuj1[)t>$ ٞXQ W[ݐD ('z:0iWBZ`Y%F4Ǧay%;]5wY"5gde״YCs_Z/%G x(+ }WDZ@Bɦ[:VvIqTa#8@Wb&nP}v2޷K"0E#\O9c 45}5uMgWY0%hN] N{ֿN x /pfhv=Аy|0^OfbI0carz|̐6Hs,̜p)f 9n0R3*\v`A}ŀ0hցvLtZR"`v-<%{f ~0$n9~e6E'ܙ}tIp pܒxj(i3F:7K 2O]&o E pJK:`Q5sު@1Mݹ$qD,DR *UjyZܬzUU֍l'q)8b.[X{ɼ#c#𕭒D  +ZIDEPS2Z׎%PQ&u^ vGytWDvjR $<몿nj NDc+툡v҈I)0T'1*?TEٚU]fŢhoLhں8l4%U^"7S(.j1.+XH*eN7fyDԞR8*:]YiaID%/B#A:>fuzV]eug]K?Y'M{HEg~fX"Йײ67\g)b  !(XMY[H>AQ FC70So~PǶ4Э(@m1fR;˷4AAS:#2Vo ^ky6O { YfT[x,M>3%metm}Vm,".<'D5sTNU1J%y(YfyQB<DF[3pT>qB;#n*3"kdLe;NϢq*Ek j,HQ6m͢ct_5ڡ$J*@^^b]{y=),la]̊;{Z]@sS?_uni,ō7-0_L~D*㘖?r6\J'!U'eD#c KA~I^ 1oRg SSV2T/?kMh0ڪZV_4F(1K| #a '(-2gZefW,W|eUt 3tQ[e3^JR<0Cb84Rh󠕽nZY^W|pbL,=Yo}*P  ^NΖG>ƓƄSƪKm:"R\L~;ܤk %p YWmfmZg%$K("Lh" 4`فd` Eyn~:V&(rEV׮C>Ǿkk<` x 1n"& A 5ݶ 1z>icC7|P|f ڃ&Zv^z4v[E{쎓~߀ϩvIb-{ߥ;2FU3>G|1$n+?K>EE Q5af 2a}Pgs(xF$'ݹO[DMqI?`[j`wRݸZ9ݠZy^Ǎwu= NӥE`]q*Zw@:6m|{ `Y~e9$ge%qaS-<uja{X{IM L}%6y~, 'l|W(_fI} Ğp=3.$,l{8 q ᲻͕4WΠAP3 ?-y> stream x]ݏ۸_aPTz)~H6 !ǖFlk{<;H P2IQ f6xӢ-|=sevh5׃H1dR &wVP `S]FaR64/TBVqC2 2𐻃witBfɦhM2GBmRH/~{EF˺г6'ޤnhK>}~AggWY7}?\"EDb)_U_Þ*36'!B" v^+ s*2#6dA!33Du.ASXJL+'+˒?=8R*@E ~@)qo[GJVbHXC{%wk=꿃6mBLDNqiZ5A+Paډ*Ŕ*McB}-^AW꫔DBY%L 4/7{.@/l2troV|aL{QƿdQƿ'$X1H8aH󥲯[.Xn#=ZVm)Y0JQr죈G;_&/A7Np! WO2l+ 9h9\GX·bz)ObyMȵ@Ni/|io_NK0$9ˏ=0kSxp7KS2e}.5cc^+3Me?diUrE Nrz{^,8uh3hz2ʕuO9^rF8#:2XxlmXmPDWD> 5! Aj!9#ciG3B@dN']R؊jhsf$"|Da폀|0VFpNO3R/0\}ŏPVXg jhP`k0M-.bK h*b5\5 f!"pSQyGL]@ QO7 ?bB%+t]U΢\PsAoQ]yt#%>%>\hNd <_## ?S42򸪋SpY 6-٩+u#N$ӎaNeiNQM]Q9zF~N$Ys"uD5H;93E8ga |Hͽ1[iINDF$1;^̦i@ho $΢҈lj ֘Uy |Eam,ৃ̤yKǺQDF 80B0F(] '@Nkqvb֎RvA0cVt.9e.aY-hpʂ / :mB2EhlS,v~ /8Rh(EtJ0bؼy44IޘXW;$.J,Aiw-sEVaDk17UIA'NTWn8be}iݙU)?A:ФGMe;q8~86ilՑ|q՝8[ 'XDl:Abm_F/G(sܑύcEC~|:W18OoMA|!~Cρ+iSG/Q0!h:Eaff%Y2Yg;bY-˟lȬ #HmZ;چ')ZfSXCM**.b?]=?0 endstream endobj 1461 0 obj << /Type /ObjStm /N 100 /First 970 /Length 1417 /Filter /FlateDecode >> stream xڭXn7 }߯cHF\@Zv5ˢC55aΙ#R(hR{(E#gu5809pmj Y $hA`0+"*¡6CrhdH "'Cj袞Cz8#^aT9eF{S% eKj^U4*ivŇEVt6~-^%em(85XհRE6`b=-H[FCaV3L6 1J5 1 C iQbablbnbt6 1偮<0<,CN4psB+d_-Lȝ$&!Tʰ0S d܁!T毅ܵK1WLxIʰP6J( Xxi `l7P%.4BN:#],`!CaQaQ;V1MHPt0ØLc$ca1,,dX k9`<'}z6F|dERҜyq{:VbC-ȕcgI cN s{=S#;ǁwiPL2\[$z$^%G$$g#ӈ`B~پ{qz:T?K԰ūa-$akđNY-URK(}cnījMdTIjn3.+ec,V>nbev:xN댋՚82bXbI^YJw5#A:8=Qzv{vpN Iؼ!ZzƖғ3ːL#R"6GwzE'\{LYj$v0F] D&gN9"Ζ$o]U~d("v|F)Juܺ8+}|^2_r&T{"g.CB*W; ~%vp㐪o8xzI3ᦗr\6Ʒh8tCzg' F [(}GZPu0rmriJ^KS\rҔ"fU4%ȥ)wM.MkriJ^KS\Ҕ&fU4%ɥ)yM.MkriJ^KS\Ҕ&54#ʥ)yM.MkriJ^.E endstream endobj 1563 0 obj << /Length 7472 /Filter /FlateDecode >> stream x]Y7~ׯ}r+lqcgsxb<1>~`w%l$GLP5U,Ld~H$՛+r/>WVJvJ+-xAz}wݵ.%\Y=??nxAKK%r` ~ s}['{=!l:vzgz+V>.~[N;cPL(\n\!5nJlayۗHr;F H^\Qd$^>B /'ISPw?;x+Rpk޻]Qp\}ß Œd.` o|Ӻ\iɡJ辭yر]L#/+$vDl_b,k:nwZjijNAgq %L v~a ]_;OVޭܿ_-7ޯoO6"fuARZ7\nqA߮RJ4y,TWW|e?ơf<b9ƇZf<";ԅ:dg}:PEBl'SUFs_EaAGsqG*@0|P\ K51 fp~BSܭ`R.)jZ:!bNSpUK`ڂ3VQrޫI BgQ+|ފl4H4ro h4qEڱ٭ƈp 2Z]Y/9 ^;@/1/x뻖O~Ҡ&N- Hmg)m tV׋Kht]B4cy'c9mxޙUWpMӦwfI|;5;(c-czA~9W0iAh+&Xr1Ҕ#<?&g (5-G꼛m3:e^Y-bFqLOBB(r&Iz]P †&feJ&C`l*V>JUjXо[PQ@WP\ZW15m[ mN?ZeVy:O:>d ˆ [f#;:cgu6~OU883#zFxYNҤ!$iEHDаt{CݴԾ*6KӤPW +X^(Ҡˁ1m" {5.x#/%s6R)6Ѐ5J,E<sbBӔ2#-qeaq&I #S)ƨ:00.+Nc `,+mL{-`ɡǀuC@1=yB k _dЌ8Hw5F!ՓkIhmɺ(dX21 9Tr0cDx/-ޫlpߥʮSgxPy fvz%juƃ6`\V’_&Y&z4HI6BFwn}}؈&c DhJJ.!.u)m9mCt~WblR0QuIg>+.7!6<B$"a"l632/`B0;j.'fJuf1!M6GTrrCz[ -f4'#kph"J0 QE ;!M֑=~T~V%ݰna?2C<7Đgg-IYerbat6C(FTm}Q`OqKW4BX{i:)uQث%hAeC|ۧۛ@lOP&t?R TZT˽K,@ZX$ۋf4c.k 0M'*;`FvM恣t\OqJn2B. +hmd˜fmEH HX7Ҩp熅V%z ./ Ho T$΀QP4keT$ CqiOM0=6n+Fnˊa#60 \f#ߔEhZFV^nx[r*LjHsĝP܉Hֹ Oԇ n1`Aϟ'Fsϱ? ٱ\\Se/*qňURk'Ywu[VQĀ1 r]0i]a7Z;:sO:=g ( Ech-d`C.0o|SJGVLW%teOm_uFc&sc-^H>h࠮oj}s* ڦm̞-WO8DEeC$JZQ?[75yȐtȸ)ㆎ-kQ˲1eid#I-DMΔ 7Cu}kPA[ҲzO[au{ih{,˓{^sAK,l),sS~͖lݒ-%eZҁ%M%v][#r!R)`}֫VDR Qb5?ԩmpE+ٞR"ccϲl >[ʸhcE̫^7Έo(׶Aiŭ8IR-=W|`0' qɸBڪ*U}`y=AAffQ8gv̇#3 d z^Mn6Aߒ f읟WΉ:j耼P2/ԙ͖y栧?фNDYhM[*/\Q5 3>$7K|Mf[~KX ʱNԣ6ϕRc3j jsϹOH32!7 D&3U7qZ*IJލp 1RVZ 9/\+_%Z3@ONRWEW1 Æhf;Gm}YԴ.84MTF&m&ΐ6LACe\ !9ɀyACGV}T^ҢA%tM:,=|zO 761T|֥^ 48.P%HoH(Fv!.N ݔ%6VHlRܖ;fe;蒬}+ jXTEx!3N,Q. I Ϩl!k12`4q"3Q+_  8D,X7>;n[5J}9PspyJ_>\Ցc09@$ vTjxIߨ 1Oa+KC[{U Mn1:s2w]b2WWn 'P}wCЦe8@4\i@J@d&P4 5gSTMQ&*)YeSZwSe(Tu!MM&d痑dmi mRYW鵛/(7 0C(??c@b4: /Z!B&6Sy D$`~jv戲q1G+Rqεi ,@ӹrb64G`(~͇+A;7z8popoJcjO\zt㘃} a1λ(bA.l0)ܖBD]Ɣ4I ԵY|2V yve4i/!ĈANa$v i` ˻sD(jyĤR=[f9g>{WA b듏y&L k@ꃾI(&\& @D =uc;z+pkHdZy-O 4g:ϬJq_lSZ9'FE<Xxߤ?q`3 N{ci/0x]_cٳE{6k{?-wtwqڮ:qru1&o~.ނgY=M`T\p3}dINq FjH(zI] mǫcF^ zuowCX>[-ow+d=0i $L$uy.7xPv~ln7sòxE*Tq:_pVU2r(v+c,(4 ys4Zz򟼋@?IԚgO3cv1,BD FwP~(+e}4''߈RեP`JBC%كlS$wu+V"V_ݝnyn}W*?moQ3tAf/:hg# HXNߧTX22CD%ݤEVaG7o) s>nNwׇC)8HᅉMy~ &~XHVЦBj$z_~2\O=-j7`zPÁJKn9XS^%Sz>Ey8_EsXn; P_ƥs|ݝGG!%]oN(U:B_V%%xaMc15Khq%-,yxq,'k .ĥ  BuX[M8i|rURt8%6z(?/jA B2|n??@Gz@$=Vk˱}{_vV<8᝚>,]CD}=9 OtaXnD4 bWjQȊ*'-p|dz0Lh'Փ(tCMtD⤍.*dpKnӤ endstream endobj 1570 0 obj << /Length 3697 /Filter /FlateDecode >> stream xێ6=_E5"")&P4mt[iزG%jLc8,; rD#wN& p N_Ƴ|kG7_tȦۑp 5p倸Rz)相8ӝV@tHl_ qƙ}aZ6Ou9ln,$'˝ l[qʄş؝2ũYe(޺Jz'n Ol{4l_T8 "H(.h%0F!::jQvT ~P N@  P4\E37a,u\[ `} t5i 1/NUptF-^AMW9D5V1 &jSrIB`ze4Yj&FAI?E(tYbQF`F`9(q['2) 6O{)jYslB9Mx4i14hOPnuښ_VPtO9l2*=({CDsD)exe&(? 1N±3*Tҧa2 #C1u(di\zƼdTN{a c/MUt(UJU._3af[YPJUGWgno GYgSɭJae8܂EDf$eH"@~!Ʀ^xg) cjGӐf(ٴ>@F3if/OW-Pn1ٓdÔƔh\֠(O! 44 7s;ddARnͅGWF"q0e0 -`UB:6 ShbWe ~ GxY;RO;gJ>b1(`{>mY+ -QEh˄wE a4_]F.<~m'nPN +ct"t@ ^1(B: G01F>IgK,EYGTM%J;lFa 6_/W-WĵK ՞T&w?#`eC 41Laj>ل=5AF˽PM=m(&cf(wFr"h%AX%<]L@! ATvBNcah-!42̶g.CʐcNwO#qBM3z=V1uv(5L"0֪ A %(ǓBjHA#_YK V_Ha3/D V*#m(܀J'ؕ;k{q쾫Y|h} Iv2/Iz}0M@.ʱ+ZiI q#//<ݯ!I'm KӴ0w0;M1aցBV$i=;g<%Y4չvk ;/yb;(]V< 9[$3hZ9h>sF.GR> stream x\n7}WK5PA@jEj,_GvuI䊺ZKci#uf!8-bY?߇"~}~V].N mRHѭ*^/MCUUE#g(QY;Lw h uʣ *^Y^e]^u(/n^fHҁˏ9u9B5q0̦FuMP՛ Lt/bs~am9iXT|9Ay3KASvje8ƧN]C=#?l#|ĂD0ֵ*y>̲/M [(^ZPU*Pא׍ EteVל1)c@rxڏ5$F@N /!7^Y+l,m ~ה 47=˴8S*!1SQ~|K]hi uhu?~6=x ,;* o?n|FǬt(L`ϭY(!4ND oB[זykqbdbF h2g'Bqy x&ނ"CqU\5fvNM5Q!γ駱ÁCf]~v}sc̹ZYK{Dz5\a=5Ԍib(.:+ requlC784jp}ܖ"}KN/ CM-ݠ]řC,Q6dB4fNi_Zk*5\D $ %?,<2V1YO)}yE!I1p?3xnB8I`,8ؼ6o(m>L> Unɸ]i7wzެ Dc(NcϮـ jbJ *e0.Kk!GAp- s_Tka8cpEI)j.AҺ!u ) Rvnߢ!$'"p[-^+^~ y yΟ F:(rt-)ǧpKSG1zTgL!!a|2syLLho<|O%OEY+QPy!% &J*iM"mQ+QN~i>7!r oKK56B?)>򉟋MziH!l@J4mio çKn5뒃>ոFߵjC: %8B_Y~pX}neەޮ+:'1uw"91%y$N?ǁ4g*n=$sNNt |9Q!܎#9}16SKG7/$q Ygadz60XRC5(57|Ʌ%3weQ7v41Pej{PN/=?9{f/"qlFŽm/مZ(ȁųt>.[pIU1j#y:|KIj+HۍCLIj$TktE܀H̀s7[%I'2jY)?gnGUvif:D^NNͲhb eѻ,zE譣ÿh٥캎5MP=H2iiiΙ^..ny4FR]*}T_*5ޗJͺegn2,ҖsƠEE]T0L+_ PvECM:v{ƾ^=|Mr|plErgzS7ܮrh;oBeVumeW3Zv1 endstream endobj 1560 0 obj << /Type /ObjStm /N 100 /First 970 /Length 1486 /Filter /FlateDecode >> stream xXn\7 Wh.+"%F<@N> /AO8hگcxIʦGsS8EHUYs\K*2JISR&)D/ZYVXTx(TsRq}J8O$kSRS+SٔXyJZ`>diit_`1.X=tB_6 ;otwE*< m4҆HqM)5OYêT՘Xq eOtX2heʠM7`C`j5Ȍ.EmDR揕(s"NNb;N )eyo/QZ%ݾuo[rP,O/6/iZ>xۏe‾v[V} Ii Z,ί݌#Zvqs0XR:v;M|C;;:?Ճ>xVb?Kr|ק[WV˳//7oWTWΖ7˃SD>s _7hٸ%QF %C ԽgiyyI(7_w?~r=dEo^SlZ|B33jlRʢ4VqI}}F*{kyG^ 7^c݁X Qj 3TGCx7|[!u(4jeʳ(s%ʜQ:\i`[[vDX(se=,0._^%,hd>1]D[&(bn&[4(13idh[q[+C-٘Z+bnvEN=fte.l0'Ew9K]XE q+)ѮX vŒI+M6X>5M{sȧFIKk|<~=.B(|Ǡ`VuXTV(1*k, PeĢWSh0VqQfQsQfֹcПd endstream endobj 1677 0 obj << /Length1 2242 /Length2 9464 /Length3 0 /Length 10653 /Filter /FlateDecode >> stream xڍTi$Kb莡Sb[iiTPK$yx߫[g}ϏVSSf VA@.Q!ËȨ ;gdA`P_PP($#U XԸO`P&,bk_,Vi'+ v`@ -,rrス>fxBvml Y2@4.LFO{\{# uqZ]tU`Ϊ:p w&@YYAPo`q4T^pjy {? Z}f qqA36CeaNN`( >9+~߽:\('gzP;XY/{?6[0 #+$(^Vv?v?5:Ü6e!6L_7wu ~+8l b ]!^x~~gva0?1JN^A_ $ =&_60ȟz7=j&{.u},t),N"wG?8˟ '_50@jsvwSe8~o$Mք?z?gk ?o'_~o%V0럃+ 1I Hk !60W̟g p[נ \b YoWZ&;M<?7eoaw/k_+W_Js& e< ކx_oA_?5 /E֯7"" 7J-ٯܮ 럷ѯ@_޿orx#k sI~^+w{?.{=Vӓ0+0aҏ<9% Y9}]/qSX2B]ϤS>ͭȳJUG'i5\]Yk:L=#]EAɩ+w\1]W gWeׇ/kZe*XF9cbLߏ3ZfNѡ9z=?=#y7/x7bgX׭ܘ f)߂q6w&gp f lM Wg]O}؃CE=w˪YS"G0{8}4 +,bt凿$erW5Yxjg! I 271,;$lB:qbOG2MF]˕Hw--WRQLhfze4sė[BDUDO)] [ |##YAyZ۞)2~$^iγ4kbG[bɸt݂,]*$eiڱ6O`+0DZ4!R,43b_?c`$e*Nuq^_QƵXٱpV|SB5F]oE#}C#Rh>ژ`y:~)3Ȝ竎pRd[z&H. *o:*9Me1ҍ}/L$_^ ϲvqOY+_ެGVدh$ 2i5n2層 R}q3%)YyU~KMZH1*MW{2d~ve>ݩmFD1.]ű\kB}rZ =JP2JT|ƒmxb囻%& 2l)-\Ąr*H:?R FLT.2\k ȕx 16ҽ%IH -@/1ȳO'8s/CˎqepXX"½U_&_wh:΋bl7}>6xی"EMMEmFOAc۪Jh$Z@h51a,ŗACLۜ,r<<6TD&٥s^1^mJ no=#/bcVj5o> D6vɘB;=@~M'Yh2wq:Z6[Jag2ObZT6]M|؋39HN- gG&Z6Zl'DK$qnvsFM!HC%gϛDmH vqsCXb:7O ?X(YX `F]]Cr#ctáyKqVE:Qp2B70>,Lꣅck5:ͯ0[(G2wQ_w*CSP6*I.ss"/=~+5)RhOt rjq!,P֔ѵYQE-x{88L_ãLpWȉoh=V@ٖq/hV`ش8&?o`%}؉R ҋFE2X:}-+!+,WJ վtD^Šh8{];Ky*'4[]Y†HqTY_^CP2 !I"hǑϯT4ϩ'W9W5<*ۤ CDfwS#Lj^o}>#=26Z5KY}oGb@Tvp|G,l\)W]Blz:/?`Ff0i>՝;4ވ_eUJ*Ū.#=VNwTlzW&N?`s%\":rxo*S=B+.MĬQ͐q +-W8hO }AJ.HdrrV]gṯ1dÐ?LwRԺS-N-+djCE9AJ)jP77$[?K]Jj&_& :pb,D>Z%)I Y&~C\a07hوM`M.'L#If*g޶صeoz銶?4ʖwh6;X,mDog,ph\ZsHހ6ӚㅛL Tdi fzkΛɧ46 5Kr!tzibk&ZGck"_)#.<[S6 ߄(M73B!oeҮS?x6J>3|htQ`|N=T͙0"2BݔLM]\}GUأTdo+>@:+]acڂ$CL].`Z\mNS.QHӐZ'1S^l2:ɮmL3 Bc&5gzk0eH>k0qrXI,FOzK/6qi;,vz)ɝ8WөƩ蔄($3CRy115-Y/+˹p:>W|L$9OKg13շݹi-]J`HAz;d-//- 5\'?F Zz{A}WcT䓥aEə7G .LFk;L.T}rhH%`ՈvcZ\v7h\6s_Ww\ݙF]" h6Ki9")dF=I>=Э;y<2i4 4MaVTcv\R/.Cf>1W< 89岙ai)5V*TϤұ ry xֶГ̗sYPyȔ.u2zPN\tP]jxtfz8eV4\ i W3k<#:"itZ/Ļ}~.;D5t5I iZ7P9>tNqu)rMcèl̑rWoh.m+T0+L"#pK<9i5,$7~AfQ}_9٪M.ZO{$4Ih|*kt[TEm(mmQ,یbp;#"FI3={wR cmmN|Qφ-FYo>=.*0Y@..-Tx̶*dI4&bVrxh!r 'Aף!ʯJa Tsϟ6^I9QRNOjʓ7h3"Eϥ& *bgue*1Hz}Pgْaj&_!Q7~ Q}2"|Alq)쉎IqhDQΨ)D CMՓU$R\g}WT qE5n:5J0w"E)ZMw HzeeKʡyN}i\=^ٿ$دTۂ5|.~'rm s)ӷG0BJǧdD1lM9KZ;dA'+7ȏǗ!dNyNW}#\ xFy#x'=<ю/C.橺hK|^O߄*,:h\(yen jDY/ $kw]^r:hl㖚'`Vؤ<3염-aÍŎfNH!kH-Ef 7r,&DpEL@2K5ήON9Jdf>ch'q @ JC<,ZTnRI'υ̩[7ȃVʱ]xqwO1p1)]pPRFv+ ~XiG<~1ЄfK9vH&I2Wƭk un W)s7qb}PsBswRg{:/Ĝ({$۶ۡ ?X |_+Y`p dJ^v>VlOO.jeH[~ ZQFvxi Hg˒XW2UPތsjލ1Ce'U7TY]֐6N(QHɪ}?xsKg"THҏK9du/#@? 8%۬p5_6 [ z?]Z|nҗ=:$D(XO.Wz%nd[>yȨݏE&o9>4W4}\oTU;H["=f SR3ev̳<-7UKI,t3gzЁA>p%z}tl64 FV&׽VsG2[3+Xbu%Y2j]kΝ}XAYdͷ^&ΰ 'S3@*za ->I2Uי\M hE1 ӭȇ72?@kfO)8fFJԃg 85Ωw0lMhc) ^oԱ'w5CYLHPVZxZfa B v[ 2,%_u(vLIjؔp1<ĩ`b(o ;]$U+[axkP;ʏ5о23q㗘MqS c׸Sۦ7(q(D6{cO;I>~51J%)$;A\U{KM&%A!!kZ"lc_pU=T?*p|NLa ɐq5aU7&Ojuv XE56>R #f}~tqd+KЯ _bᩘt儤OE1O71:< Qz3A!I(M%U a*cv'2JiT-wP1cly ɡI\H=D&igM0B;sMfIR[9m67u˳PwӃ<3=og)3wI+'tu .;dnmmt:/ݛhg$+Ngln)6N'ՑGIɇϧx|lvk$ؓ}>\ߪHAHS+0Tf3R36 blޟt|E1 L3fnQ&j#ƟRiz9{I3j.LB/ئ E)nGmD`|ĸ %b 9#pu$G]/ܩ+E[Mvk^unCϔ#ivc7BiMc @fKwģ*bfh6)OV /BYoNWo Ycw"LL0c>k@`Or\".d7~%b(Lʣ0`=cЮțfD~E~WLݸhE6=46=SмcX)ɗyʃv[-xD:QL9}{ Q%!,6{Ą/fN3d;9Ӆ/ElXb;M$yw<Dzq(ٗJFKm,I9)0ɦǩCE7Ov/`hoes*#+(?=[ ̓z9^ݟՀ$.m(ej➜Ad }f;6EFmbЍ6+'&pUaI908t!%,AMdr0/Sy~0aGCA/إ/fE2R|i&f4/6cӴu$  Z-s:Hn?J-*l*Oȝ U6ɫ7}C".ݍ"Hl\#i=V dlVP6&/YʆKJT\sXYr[~]J~bj'}W4`/mSk^-PVtQ[WB+X@Va?CW&n A%8?iG&hjt{mTqf|U p@Kͧ YF%s¾kF\VM{]ܱ AmF$ia~M: ԦPP&pD/jU;ޑeF?V>zvGkGg(G)[gz{m[X#&#!o\L㤸z$yI7oOVAnf ww0ޯ?_04q[\D)=?!CK p6%&Hé'0([gQ1UHj]bXmz_| )ywQu$Djya\Bs+j`@EZwt߅Qw+4z7궜!{CT"Q;Su뺨 4z&jihk *Uc; baԾ!ˎc$xV8{SM 2~s whpJ/j[uIaJ-2;mh[}(啘#"afE z-Ksc萼ݵP(GD$U spU0|U #m̝s(y@REknG.yVI$Tgϝ / @Əf#*[ ;ia!!a(B* )fiДYrKOķ+CQdN( m5a=_\Ʀh\{Mܴ߾h * ?]^$J REngurw5FI28]eC=Tn'Mf\UAٍrvtMMu.[ endstream endobj 1679 0 obj << /Length1 1914 /Length2 9581 /Length3 0 /Length 10812 /Filter /FlateDecode >> stream xڍxTk/R- )4 33tH4]- (!) ]-wskݻfg3kXt x6@(#/P20 ⰰ`A!PtZ#5)B Q ~~ ?A@dA_1RL2N 8Smy';ru+-p0 z:2` &}aPdD~­=;pv [ :##O'sdA!`祈:\RAD֣k ۏUC=ߺig7v}忴?z_ s)Ђ"g?E]-k4C#_8u_ѯy @](k o9d-Go 6Aa7ȓ0W9v@ @+d{ί 䳆8ظY:KoN_C `07Y?_ܠv6`(V!!$7څhA>(  f AC!D&\`o8_:Ɛ<@HȁAr* b #*\@w8 'qHy tM[ɚr""[#dz'HȚnT,2fm{!1=wwH?Wٺ!]{^@[dӛZyO͏+]Q׾ VvUïtgh8Gd/"PyozvPjO։ؒ|.+]Ux~*p_2 ^t0e`^E >#%v3Ks7T7t{Cun>!kiF_HU8 cTd Y( —e)TE.hV˷ 4CgszOi-72 o([k_1+ء\s]?&%W M1B[cV/3Sb(aOKHYHtѴV݇y/ A& ;-:z#̇td&Յ/9EunK@t.UɧrscqByD(iئ& I*EF^KH,SGX@E1 FnSGmLjG; Ǣ3˹ ?-IJڿ>o<5| J\,|:Kvk_'[ʶZKc!UaE86ť(#k͎,\+G0F#mmzV/:nqޅ1Xc Ѿ g,@o%*v{+[]_LtD>xC͟(hbD>Z)9 o!77U>ּ\l}PZUK"cpk]_~{N?C9 5n5y= P'@M"T}GjzO\`=i;| J:ZV4զ)#5kv;c{Qn=K;+zL~bc;OuceB;٥:2tPﮜ4=PQOu_\?`_d!8Bȏ*KX;J(.%:k#*6gb~E2v? 3^5zJZ28Dpp4-㞸MH@w%P:΃W1n_Aj4.Ix1>^sμ9S+i=E j@;N;٤L;W0<_crEc8^EBcS}e`80Er@P&;EOh|,X=sXz  ᝉݯ4 8 )zϜ+/cU|{e^36z?~*kF;d60N@#7߹b$i)"CQZM59nߺ.O$7s&Buow]LQfILtJLS!0X gEErǑΤ(Dh^<櫉E׶ Ώ˚t_'Z':IrG`s-':jph.7?=[^~zP?mؑLԗ9xtB TzgV|,"pBl/% F8l~(zU7Ϙ ^a?>xI }f'L>GϩBPHq0D4b7(-a .J) `0~*B?k-HMՍ1;=|m[K iy'?(TGۭ@o||,F$\Pyƒ@XHaQBFW N7 ]ԈMQ?v(PkO9]y=cП+rq[aYR܎ƺۗ݁^W?siŠ>絎>In.Kczmm]FB4eU=ŽÝ|۞_dٶtiY*1a<?8as-6L1p_:K:sxP.,Y^|X_VI`u3'6N;W {St)l G'֚յko&+r=,>~VWBWۋ##3KooIf r SMNpjQH*tuR :QvG>!RTٮ8Ы8.1~dY> 稛foY@K > ufJqeSHI)P7Z7OvUYыWu*F׺\]@QY K, Mf3x1:O| تPjl{K~ KM@RZ{T8UJRՔOeaZIuYTCz(D`\_j:h7YE)b,OcocKM&|g,0nu_ s!Yiњ}Pv ^é^y,e!ٸeZ7O$I<?'Lo&y?La"vT Il.aa$A3+ڥFuhIw o܂256h]x~y<0[uq{ίri5QCtXEKwUI G g/%9w%}6HI>~P/8/QQĈ_f볆d?dŸ~ |w |ua p0qpY=W%QQsetF꙲͡wI1U֌0 0Eak HYR-ߘyW屹D( ]_|ʇd롎<|3@S[uoE1ՖrbCdCϴmg%quI?׈t@/)-& 2LLY_2 hYi~tNLn}ۆC+PKwP8,~nҪ+3q8VWٖE$c; *} v~͹(uؤ-!]1yYQɛYE1M"|_0B7$ufbG&Υ,Zwn#o~ҚA`0}6Ȏڃ)j8i叇Pn:C;lQY(wNkU=I 1󺇱 zq:eIB7ݜ C!h+{?s-u0?p1] '{WU ʑy?7Qxytq7n^垠.t¯ZOYuT~h%,!<0BçQ٬j rV|".c_ ~t.hٖT^XҰu~?~ЌlR315֔c*,L#%S- wר9uP#A1wcɃ֧"cQ\ x<%G}L\{5$]S{3\.|#g9͠,56h9~C^y](cv7Y~08ssoEk#rKoA /Q~M'cI.ϷMÖoK6L2}?%N3*_b./}Lq̷moF߰~Ч&7X[F v|_,b(.}}cJG-ƴ+6!?2E[}ɔ<:q:3 K{>c=:TpKX3kd-g_H֥{O?S3s+umÐ}-B\mc >鿿^[H)莓 Lz7_ /yzFEZZre&R q8שOEABKL4/:l&:XpHZJAB jyAIzO^]'p㱃nI2àW{yH ]s8uGo<㚞iL7FB%_$zbOخ(>EW=̵FvUXs Ȩd]Rn u ֜E1FVU&R8r53V"zB< +a_4<VۼuG; "5F< FG6؊p+O(raO3dy[\a3 b^Ӷ^DҪnwzZHO"M 2%4M.(Ϫ箳ʯ^vDa4nKd竌pګZ^9a44+!˨7QuWFqkf]g pQ]O$vYIHLE8aK)Uo&tuPG&x^b`ž2 &PjrrLK넓fB,Uj&XW8 >ZrFhjd9f2[-SʩqoeSo7af6$OtrIi,͢wQ| x[5v<^ q5rћ(Z)Nəd轐 @inȿC?P ^;0::$.F i -VY3=RU\hV'YV$u_LPahcej6>ྖj|f\~XAN%pŜ ;MGT4^jr1*Y O8|0[y&G[ 6'ǹwBJbMgfx}_s3ܓj@W{ۢ}YS ѣ*6%Go݇}+ |d7 цb- o}"v6ղ+BQ8G{_HQ}6-"}sP8f \>"1nB{}S 1bDnOn6zSJBBggijb,fI!vb@G~!f7!7[45Ħ7~= KBJqVX{]g\o7| .,^|IgsL#(74_S9͍O<] ?zŹV%oi//' 4*&{ םiX"<~fv̋+T2k#i?!#mg,1#WΦNJdQdj9cE43㌗ήoW^6x<%#|jqI)KW9Ai2LM '? /c10 Gh4հ AIEng8LQ[N[ߴf;Q{}20lǴwH%/><7@(okH;TLo͵ :R;Ӓᖾrc:;Ǐa(ږU`h55iirh$>OT ˆ4CY>ڰ& R؋wҗb_ *zg}Pg)M2d?޺>*2ˢ>{]F6Gה^p,r V݆+Qm+@XDpKlTU"#+V /֛Gq0[m-yR׸ͥ#GܷB7~cxyX&"14R ߦ#sqh njA>cG(('HdP6F>5ͧ^.+T䅘pҥƝ5{ i&ӊ3T4N?6͵bk؃`f$An(&f|GB嵜ުl[2H~ :Jc6@Pd o^я@Zg^Ptctp9i (JN^@FH!f\qq'9.oaHuͽj,ZjVbk.e.AŚOsn^̛)mg=)Rr|G尨q!LG汵*ߒν$x||X ֤ycS443QZSܥHmgy C1z;DZ{64>K]oorXG)5 9Φ//nJhڹ [F}fjIYM]n !5$dطVV`Ww&1կ,򾥟̤ʶEBsd&a$>tׂK#GB|,ÍnXmj)(WN OOLZ!6P\!rN-03Kdf4SJYPON:e i|-83,]s Qq 㟽4D}3uW[_VCщ,t}AU78^2VL]] k9@vYK#޵8q)jB-ȥ"V:#fy`6/A] *0jV yRnw13Y^зhJ .yk3jr啫w*:Ʈḱ)gHt)N endstream endobj 1681 0 obj << /Length1 1597 /Length2 7313 /Length3 0 /Length 8401 /Filter /FlateDecode >> stream xڍtTk.(1(%C#H7҂0303t7R4 )!HJw7 *ouΚyz8tV8 '/ P7 qp@Q`"C $"B1% 4\A1IAqII j ha$"C+| (!!~w8@ A0eqB9`(Rv($rB2ܡ(; AA<9AtO0"p;G(C#\a}8@_] qpl+W"(w0 ;9`P-hh QA!@4: _e$T*v߹Ѭ!e Rqutmm? '_hҺЂeoW#jANmUGB٢'(/ "Uk( l2'AtH读% _6o- íLHT B @Dh: тxf2!t8ZD@3qqEm a$zPz;vam/6 KcbB# (VQ'WG]" bE:0]}kAoh+ sE e tF@ѫ; @7~@'Mq`‚@< `I8a}EH÷2yFwAq&(Ωo;M41g%ttF'}y yf.$e0;W^^BX ⯞2~fcƕh4#TMxF&䦩. PpAƖ̈jŊ2ac_FR R\ ԟ Ċ~Uh\5M훟|%|T$K-?:>C zXwSSń {lɾ_о1\U0: #/CzV Ip= 6V/N% L3#۸\n=־\lMs=e 4j:?YA+6lwaHNT&fं%>ӮQ:n)ٲ啚IDX/rj9kԪZy?Edj{/$ZLIȉ #CfPI&Bc_I/{a]2f>U-?E<0߽1BuTnvl:^қ<RkBL^P&R eyDc| ŪL͵Sr~#[ȽIA_[8~ dw Dv~A G+^9kǻLYb"4V ߜ߾[Ę"ۂp56"T.n/ʏ!\KA|gVUILY ч~c7O {e:bIhg|m:o#V!"yԱh]mI {ȐZ[ߑ7H{A93W^/da xgP\(z2n"3f#וp@#9>y/=qIZ| };kwv fivre_z>y0 D.|x^C!c|QaywALKx_o`&l7fJou"j=Zt9 k~y1Hpr*hsSif;H@ٟdJ'TNSiWI+Vk >f!WQݕ"R*DE:fEx)1e37~̩g9?jpgvo?W8#`ǻG8Y|Ta%1Ť!m&Y?%+hq*w`*(ɻ}au- $DyuF ]EʢK+iVc9>PYml%kMiXe'VnU~TwAr; F{ۈfTY!sdf"2[|$Q\3GfgszcK}2h/n?ϾE}eY ~;G<6i,!%͒Dy VlPߝFMxjoxy!аWu;.c̛k|n}DOţAJhx@LYr9[|2gI/R2bJFdn{Xu> 6 #|ߛ{gMt}7ƕ:gPAq_.QG$5`,V^^ѵ4/%R>'0- J: 0,AfH]2 AwWdSqID<̓lB"1SSzk[3FL5zYk#H2Siz(~a"zϋ 1?Z#Ӛ&8er(K~'*X`-MMy=xmCK/o@% :B}tϒ'j{O y3dUq<:Mɯ;5M=6st4fnȿ~o+F0FzdHFlmטta>Um# &6a Ά_8ˍҚGn"Ny\HD59|SE ?ٱUq1Y,vM'Uzƹ~mHyrfBQEzz#Oc/,=({IfMgw:#8_JgPyz匿$igڂ;\8@PRӹ+X%U-:Ёb;/%m^zEV4<ZՍڛ.=QoRUDpw?vZ3X 5Caz))M|g"b9Lj9TgiqE8}#v\Th`l+|>, ^QONe51OxL^QW@/I: rR1#hVX#MCC@tn4 BYP۶3%pnH&&5n\rl=4As7?G,d[|gUXw%\lor& 8aFאָ&ewh`Y>#k.<;0'#`Zt&t+M w:!9fz$+ryaB1eXㆩ|dKӖiOsʆLz[~^U{+^L-U/:VZR O׳Ay"^͖ ]1ǏZ狋╵1cV/D&Rôz%8h}L}唼b[T l4zluwZqXr-UVvN{#4LakZ;l ӳn?sqE8[aptzc&DVZqS{YraKQ^{s[|jde? n"O$Uh=e>s/I01s;4*^EZ.̀~LB$'[ZKޱtL!ş7E 5L _g}j+6%t-y#=z b<+0usOd0Ъ^ifL>=q`崤`maBf3qhᣑfQӓёasm~~To} )$1ϟ^-CS><4Qq^)_SRSڕ@pC}\)<Ƕ"kWƚlwIvU=guU,?vދ᭗zQc LFT0"c= \|^?7!>۞B{۫*[Eη1(viB5Tnn1L1Aր'2H(I-f V3){_M&&\dǎ314Iw)fX:8{̰r0&|WAI+ HnWKpU{ԺdϾ̞4cR 2~'cʩuBM4<iv= /P2o+%x&hCw UnJb&;n̽yQ`Zv"wq4֓ѥnXQ7PR,(qe #.r[ucz ({1G5L)6Ϲ|jޡ<#S'|Mw(W<&qhau#:"4L8?_Ӵ w 0.p~Y᠆rz JsPj+ m= Ev\*MFWWS&?70#2Ъ ;=*g&Ucm5Ya=>90hJIz򲐕kz;ΡީO;1߼$K$m?+(}YKK/gZklHANsYFP9hnzAݧlV [&5|b4^l2Iz,t/}/ /L3qMw5%coE${fY.FɽG/Joz/~]TVx8*Ur߫yw {A7V7s>\z2"ENR&-Dߎ>`EK^" %pl̟PPY8͸>#U*JIWQm{61[+ wgr[S6]8jay쒹*tFx|Wzݛ|=JbfS-ƸՌ48bU:&ϼƔz$hWl+NLN|n(d7I:ҵ;w$)I,`4+oafGüd1*s#Hc`rF<<s :x֒L[>+9 \:xv bk&Jf˥RɜCwF@"ٴ>GCP ڕxM[tߧX1)me|<ąÉǼbH=pݱܕ0gΝY. q T]wk77$R-T=hNf\ R+\k+9f5zu1%p̺z,vB+G,Ns6:.1\?rΌ<ُ{~wuƸh*א8jLWO|} qZ>GԢ~6Niz!8@P GFDEZC3WD[j7z([FsJSwBrJnuD ~%R Q(ߪ%\שcou!o"ع`5;32ù14T##)p4k1uqKAOt?t΃gQŨ\ gAB,$ w5jKV:r$0\V6 ۟U~i$- |J%Kw둻HM֫fLrsM9Iߎ]c5yŜnHOj RΦpa, %Ȕuo Մ>+g}XjdFFP먓.@)H:Kq ª𕰫+}m 86™Kև8 w4D9b",21d3*92d^MA2?Ucѫr|'<{q\DI*v}q"P3O(];y.} tIKnIH#752K,gb4ulIסqavGOlߕ b뺯5U%7=̰HS$*NB9w=Efͻ&/7(6Ű{TcI,qC4PpˎQO1 D`O))e}{꬟q0m$( n = nڻ̵绐NRQSt~bzY4;:Wifwvgd Jb˼\^},3H]6w[ S%1dQjf |]||u#ۓhk!W_۝kѸFd/ۈ:>~I6%_3RefC%%L]h]"C3]OS7 endstream endobj 1683 0 obj << /Length1 1648 /Length2 3429 /Length3 0 /Length 4396 /Filter /FlateDecode >> stream xڕT 8mNv%ed{fϞ%3L̘d z)eDdɒHY#dVַ;c뼞TPx#T337BU_.99g  sɹD[ D1ShVS`P(0؏DP8, @h.5;Z MZk_W ja>(` qĵ:3| ʃy.HYx.X.={x.RU|]u|iFk}Jw}EY g!ltussB:ue+"CeX0{q}^}'hHSg!hZ[<cC;|z'X, EʴMJqL(5ӟxTI/7I79!w{-MﺐWKJcxY w5-Kf1/=Vvo;S>yÝW 6^=uCQ:vb)pO02'>$u=fSrhu>~SD +k݋Wbt;5] lKK\>Q)Q/}I(EM(@ {"n5_sv$g1VbN+INɡe緣`t(mat]dwf֭x$7Ÿ^Z!P`ϸ#8G]GR:bUKA{k4% !j}yW{y/ {p7nڛZ_H_vUZ4lȏڙˋkΫ$kߡ NI\^5NNML#hڽ.P}&[)sZz469d X ۧӛjjbv~geF?T +Vģ)طrJ1mG[:h1oӼ}butU zoKHA݈M'U}Vuj";\ktSEJ YΛ]I8A>a C{4RowōC(+VG+9s1NI/ t%g,YU<'Խ:{j]д<~. QjǬ%vef @tVO,sQ:tYbQrϘ| Fv3Y3ާlxg|<@BHs)lB@G?0WHbJAQv9_R?,F'(id_Dw圝v# K%td]]۠[qem]ZYS!@,Wa hOL=0r&|f]3Duq"zFlYυǫfDsբ*oUvIǖl,zSӽ-"S1̩/?=~ԫܒIyŗ k:9Ubj{jJQ}HLK b<2 0Z$Uy%~" AݶG{L&{L/%CspM Mk۱\S&.Ci"cfj>W'tHrw鹇b}xh8Y@4;B/0-;H-r7 a'-kr/ eGrGө,Wy$ 'R R 9I{\<_i*4]0YxGx6H.0f~*(y^%sзW+ye8|9zB[q:bδ!7aoĵyexxP;\V&:'~jE6cNwmj5Re"վ^^L7](%@%z?2i|2-cF-_@t@£]z--]}?kAe.ՖN%Q<\"FVX=j1T_1< ;O]xcqq?J~(;[qK%G̭1"mB!Ym7N婄3lia}~0-ɴ_4Y,6V]s+PՆ:Д}ihQX}4Ҹ=&G♦ɢ).u7n$I 07Dڦhܵ4B6Ӏ0,oglR#{&ǎ_%-/||x-c ?=#RC9vp嵗!: Co9uqH픾i_e?^WYT$wyB j_%w:DBSuqDo!ܧcm~Kf Oٰpl<`"ܳOaik~O`uj2뎳:;v ~^WN{<5rHMl\N gQ[A+ﰓO |_ s>P(8o'a \J>:֘U.&h?dO~=?/p\Oʧ vA[!R> X5[G!t&' /4 ]^Rwi7g\m5gDʽ! &$ΗNyGemI4U}#;&G/fr˞Ptsfr/dM~ Q q)1ݺM$1]EkzCl'j I[j Qv[]eZC.tñ~FfC? z<|I]hC3Μ|=D ;|pIruZbt@6w+-8҅&GZq<_2_ZٙcBX[^Q\?33"~{ FW? yڣ{ie7]6ūDM}{!D+Dm6s"M5a]t]ْb+'=tΖ\2qr(W!UfoϊV_&r%^;]N{RUruIff;%b;Kc1B endstream endobj 1685 0 obj << /Length1 1391 /Length2 1741 /Length3 0 /Length 2602 /Filter /FlateDecode >> stream xڕT 8T&1+j"qk5rir4sp93g %עt5T&eɵD\Z=źJR"Mvyr{}=9fL5X'/g*P(tK 0ƅfqq $(bba8p "PJPQY"*Wׄ@`X8mD08"]Dv2M4M  S@02bQHpsd1`!I"+DqV( vY~3`'d `>&4ɓa;!ǃL \-*L\ (fc`#d $8GğTe03"p dW K1qԛ2/g;e7&Ń3 |I``,\={$W2 BgX qa qikr40BxrC1B†;B`6c"C0|1^Td7{窙2:!l39R4Հ% DP@b&./"A * LmA;_ؤq§ŝ5d6KEV!D `D$2_,KL3Ww3ޟ#3ILbsKř /=RU;winԲ_;qpChmC}{;#r{2mWCM72RYޱ]2aѫ[ kd V/yQԣ->?Xz0͋<F ~>A#HiYfD̑d*}zAlSQ+N,I_M ҷ*i]^kӃ?F9*)z)7QUI%p“zxPYQvN_RzIt¸YNY6{#bֶ[CaTSҤG w)UӨ6.z)\n16+IoMK_M CX{6Hy.CR)!}/y )]uu?O} zڥnŽЊѠR(Q*ALu֭oT樼/٦yvT- fvA"zKϵ!Frod1֫_ ]9]X,Gji\jփ|߅ѕyO/$,ŲulC|=[gi=)ϗ/)vT$ę~k {5NH7x6 z-F-:41I9Y{W'G7~]j .ԋ\ߔwQ_#,gsWlefW ̓1V3Zl3"{44 3@Z4{g+.px&/jYk9g])9Lq8+xwNIƌf7/ަů'׏_#^З 0yVΫl:}d'5F[һc+py&DSOwnM6ԗRr@eԁQLDUeB7˽Q!q/卉.tsu΁g#.k "Z-wmU;O(Isx^G w]^ܿ)uJؼ5 txԠ>+|m*.=PkN=U=#lB/P/ !> endstream endobj 1687 0 obj << /Length1 1317 /Length2 1506 /Length3 0 /Length 2330 /Filter /FlateDecode >> stream xڕS 8TNR M˄03g4DԺָǴRN39v4 3D6- *QPTQS*b+QU۞A}^>fv<Yfa ;[ÜF~)T*%8L~HpTY A`a之0"d 5A~A"5" c&#!PDb #0Ҕư2/  BlM`!NIW0oG$(<`!26.B>> " (p2C _|nS`cd1)_hЧrB(6 s"( PöXJ۴j~]uдK?'j(V:a(8sܣcќuUh>pgpN*AuVF^NWGZ(;Ce_vcﱶn RRu/2lycVzG5t_-8U:oӣS }rf&.MF*ġϪ)˲sNnM,7 rͺ XV}.k63!iZwƞg$W?XT>0!.izCEI3o/4-3늻׆;͙v~zgŖDݫ_ubT' 5']}UM CB#Z %1ޥDWCc4{1u֖wfVf|FxuA o4&3{1s7:YS՟v$Gk{CU#w\2MT_ly¨Is[?̪5zn0!`3Fkr}a'f}C/{եq7F%3ZzI~v ֊6M˗-Kȍ^/N~I9#fT=8~~V[v~fR?@54|A}:UWmsOl6CB8,ײǮgl.ɭe{NY52#~lү>'[V7\i<t /ȏ=?yw-Y'A\Nq{߮is#,)w;]+q)b7Oِͼvy<8d=eՅ%wFmfbsBu,)Z5U£1%Cpipn[.E7N~߼ǰd% [W?jV^*j8H}z׷w䯎PȊWt:8Cԁ(Ԓ +z_?kv[g}Fv6=t0H7ڤLt9/bjhu#8iYxz/}d `([{%ibj oZJOҧw=.q8/6Gg]מUbD=']qj>(Q%Cv7s[/ϊS&1ᇗUy̎@~GY}alernpNwRIۦ+{ѿ{nuHLvDu1ԇqxU[cS17G o3θN(w! Bv5\ϑl.!2楽9/I`M:FiwrT^;rӫ}:٨̱Q}2 Obrn\>RQRl}Jm07kj_AzDyٵ- w5! ;,gWY endstream endobj 1689 0 obj << /Length1 1425 /Length2 3010 /Length3 0 /Length 3897 /Filter /FlateDecode >> stream xڕS 8T JdVtP/ cɾDjw ("KP JKBED("BߡyΜ9rNj8ޒBjHu`bD:STt#C_z>E znpg&CiT GJo b:= f>ZM*pT\ MY]z@'9G*|Gl.8<} /~Hv%&VyCEՉp {[x9x>ݶ;E.Ļ 5a9J@=7)skǠBQ8Nj2o3DM<9}~8Z;f䑝߂>H Ѫ3B;7BEN,&fu2ME#eȧݣuGV;VsWYUdJqբYrONxy<Ӿ+͠챼lN{JqdWhx/Q[%ўٶ}βwCQd~=_v-;-c_+Q8V5I%ɵ/妭։ӫCL?9:j& H/ OыAJQ~Vdm6zsghbg{>ԔEkx0,.&1}UEN+oȦKlSz#ȕp)# 4C@nOs]vve|t2<&'G1z,{i a|`_Oדy KVEGK 5K3M#8_Ν׈7V^f}6IB,@am4rj}bZ8Êuc3 8QRAՎdQ1cQDs|P^ ڄVLB[Vxd6}vΨ̎l]uKNRt#6x2W8yQ۳B!/[iba,h\S1={PƧw=0\8ҳ>JdF}\S~E5;wgN*8^3\vlxx^wZyS}X/SpLTX SL_9iah|d(zF4@)}`}GFxw(qMn?/Җ0Ns=1ER|>L> nͭFgkdJFk)8gX(?7޷11&kiAL*}CG-daٽU UhoJg"icXrf%o+d릨^mvHSb>aIV;uWUnqOS-r٢Nrqr %\ud6cяs ,ݒ.fQЅ"O*)OwA 6_]-jkf{e.d 2"p_c}Ro~9HmKs,gZh68ɿR5֖^I1Gt25 ߚ.^❛G 'ˆ-葈W3$/EkYp<-~tWuh8!|vF@cѳfƣKdO,:<82mT,F‡fk}>ᇶʁF;_T9 1wC-yO{mm\=)860cKJV8tY0[ƱYzJumyۺE-,n-[1/p|1.\.=uMYWn7⹳kr<gSFo}D[| f|"r{gEݸ@$;15?Q0d波p+[ׄ\ }@Xt$U6Ƀ Ib;\]02{T9or}ؿ\O&l+K%lsyYk@oAG%ӵ^{z.nkpWrqw?CN1⸳/@p zY+tx!ޔ5oUSy0=ɣA*/c^M/+{No9jʿp$HW[Ree2A[V`xXq4nX}+RyWB 9)`Az32vw씙%zBGI"/%κ["yG>W8h!yFn@ d%C:|Hh|ir*bnl AujNku2Q::G¨PqNG󁻇F .}gWX@YiW+7Җ؄ zAZPY-Y6i{Ff.4a$ Sh ˌ):hIޗtf';.SEȗ*]tjQӫe endstream endobj 1691 0 obj << /Length1 1333 /Length2 2345 /Length3 0 /Length 3196 /Filter /FlateDecode >> stream xڕT 8T{/**Nv1[vd͚1sbB7ن-l7[ u/m7K nl^|!_R-a3:EP) @~^m K'bߏeY% f%G5D3e',C00Qe|b0a6p>thAlCNAChK)w7]Yi}qH:Th"ѡ0 CkkyQ?,Lv ˏ.KwX L236~Ihf<*7o97lkx)/vdp0 HDxͨ($ i."$r3i|=&K N/{LBPD>Fhg~]tng[DzƔUFg_[FoV_yaIu=+j-:farضzBRGG>^LYU>LH9Wոk<w;9R8uY~$VmZ]k64[sR|g.Aa{;8r50_v֩9WP'Y4*c-~o[WV9 Dba^7 X޷,*ޑOxE9yר8'a˟%PtHHs[1tjC9`|Ցqtܼ,{OP/Lh_)xUֱ*[E)<{n$;s3 ^|,7g)$mh׉lmojWsER&*r`ohq>XNzb]ًޑ7lC:GB%ڕ7旱>ILoltҽ_h5z݉j~}Jٛw+ Qy|OqfC}u_E\VNLN\=M[HGIg, D߽:=\BV,9JĤsjmwEmuo=~P%s8>X&h]^ɞ]*8hM {ӑ}n=Bp \H.v۔Isy(NP1g IJس{XڊOTˤ;t'y p ւh@̞nݸ4OĎ(<>StƼ]Vf񲤸7P)vZuh΂.e$x%)}ʭm!^b7yv֛Nn3 /3}V/i?;:?iVANף3رҊP;]ඹu ~J#I8)SK2E.:{ Z蹋N;LW|jӎ">dvѠUBA[f w9ݠP)#KasC';}SdMDaHO;ZBΨQ01V*ywk)@֔iXQz.-DXcYVQE!_Όx}5F~H+SoxY?Nf;DKkjDz OZhIrv`:g%J\Po4;Fmn`.汈SYOW/%'t é5Ogp8= 8Ug<}o!BP΍7Sw;l_jfuqZhǭ-[?@W z-'}F=gT 6Q> stream xڕuT .!ҡP3tJw H( 0t Ht4 "!- w18s"A?>".P{s!]MUo`4dDWp).ﰻ;ư iz1Cy撦)|+OdOJy &yNEQH.EډEFڜ,STiy{3]~b٩(GfFU!eb&Idr.1SѮ6N1hWN1X{/z,`|u41c Uu&KN:.d]>PS ԊPǵTX|.vT|?JC*/x"&k(əƇqʊqfi%ebDkL@<'$lGi >xެJޕ$XLB}*&~NQ6!N4]|՜VzԨ'"Ԫ>^ȍ?ݍ?.|fTln6ș4H,mlș=IzHkίj5T8I.3||o/T"|Bl3fw:{L|2vch3jW[o i]@7t?‚%Rߠl&]Жn-~ C$E]{rEc|J5*xԅ((͝s5!ˏ&%E*TtxR U/SL^9")/be;lwqBdϴ>xXm*8 -#lXdكUxP$s26ьY cua0"K/1:^!R̓np2ِӛ3%20^M֥ a8DLX7`kXa{Gڳ<?7M7=[Xwn6]+7Pi#қf]F[V{~RI*S%= ]<(wkvrU}S0OUI=YÒ@Tbjxv*ܳ%Z ^ap{:ia>:+8W}+Ћ(#S^L7cn D~s s( zS\-Nl krU%zjeolmn g(~*h'l0̃`0D[{a1S&'>mgFE<5Wvgr,Ha L͗3Ea ahpc]MqFO[y ]hRI*!B e W`qmuON{3o-Yz0:}r@So!tG'LNqf ]ooHJRi> MSi ]3m V3ondm03@_Q-]0rű@]"w_ըSncΚԂp\@1q+> ؠ}|5]3ȱc2m;yzefHwVW@,RuB)a( ڜ@SЎ$I÷bH@o?ލqUr49!I~<ʡĵ Dst.r>ɪ(|؃s 5yTZia.\.''nljAL@;"T|4Wx<|r \n'7JZFi$Q<4 >HV ^ ,Nf9a}z*gOx`17Y+?eRWc/;6orz<_KUEtGaT-_ɖve6M'MbC =.$$/8.|Sc9kɋ鞣XGgay@R0?AzxXxk4y[uOy  @Q8 PIt#Btounpy_.YoW[GZdy ox*Nݎ!dh I©u07{5!bTvj-|+7/NN$RxN] FQϗ*<& K}Δ:'l$,6- v :vmxS[0O=uZ<&TU""!AvvPfbHڙ)ՇͪfTb-=%35VfI cgӒ1bQFS$܅lA~tR''|>C FW䠞K1L1cJl:kUFf]2y0FAڳBnosg<&Pwk*b 藋o}f0"\VkF .Z r<d#nml5Fڊէ]q]DSN:TdTfCv[HFI6ANC7 bX(JsiX1Bǻ%t!tsK\Iʔcow悮^. LUu|:sa=2T5Bh䵮fE)Vo(H.>xI'MxS^!~dA&E-] dGkO|1e 9I2 ߞ@7}ٜ극y1 /mʷzxi8SVex_DRsA:ȉY."W *2&~$2b ,?^/eKyS: +G}~y9{SQn) NnLw3>嫐ro*cmy\4Tesxn0 ) r?rDE|n;MS/<+ZH WI㹋Z6+V>OF0P۸|'6 ȡӮf*U:#%G 4pX.?#}AxT H4TCDTNԆ"ce0,1^D,,L ڤpq Q?_Or*5 }pkK vEEҨ^YZ MLk^Kw/3R]˝n2txo JqP'"\ܢMe{{&Uz7 Ā0Vq1Yc>TP?ϟQZN_D1(bx5Q]Lk7SMHMD[VYWzv̆UnyZ-YCM;8.|xSL˛3,3 kb\؎{!Cj̧oVlna/|}iFi[쏡(/5ր/7dD^aE-o!=bJ R$#G%SYNo?I@/s2ܛ0vUN~k r+kܷ-90$6WS4|]+,|bn{k>F:o4֓df!K(9H+tGХ☪:4l;o1#qL") -3}%,/7w[&J5de7ZgB3Pi0ݯ6[{vY9-&6.&o)"xWѶ7(?pC2.QͻؙMNu~;#t.5n`/z&k:1䒃fG}}U<EO`IwJEQ)ZMq(G^>wY0U^?uXa_Dw|I!s].2޳ !2V]6萪2h8N+I(xB[ã_ϛ\#|DBHaݳ[$~Ec̜XׯĖ^.Rt^~׺#M&=ŗcl$vB <LssWj |Xը()\cKA*f|EgL4*9 TՑ>\K|S P}/l,FY)d}_hcͮ :/21g^!RZ!T/P?!!fÐ綕jb(%)zgeA-k{˫(o>~[~"nXHi%9E/^4!Sf^9瑬9"S@QjvYT-dEp'Z*o1Wc9ídl\KxA*{sa,Q5[tXf}2 h )&} :AX~Z|,Kwd$v{brUpO,d'h]V$qAx>㮟x?ɾˣ{{;eW7~ǐ,0a|]﷽y @`o6B:l}EMkxy&d{F.Py$GePۗm]Z3vdXñ06051n, x_yiBLr1(opeNt~ %8o 8,~,viZPbѮLI3NMQYs<^njqf AkwJU{/=-)B HI3Ix&Ne_DQ ѥHJxE+1 srWk"L-7*O+3l~L ,c4, 2Ldl;5r v($c^ e |L]+6uQ:'Z ~0^M<_)ˉRe4kP;=363jq=(႑Vk#mfCVSjޒ}~\hjenpyUVCJE"uDl'u07s1ݥ| +qC#lBC0!=l*Zd̙hX]ݗTMV+l+3K]Zu,qPmeXO.t?IvXc@"`CGqQ Sjחh<%g`FՓVbm(Xӄ|!"SsaCD @('4Q HZmm9dhXT B]ٺJe],~&F-k1VG^?hc,#fX[9BDlJǗ WgjgݜY#Pm:R] S5m!{>zQ!_ٙO#WhP[J}"T%r,s(۩7iM:+ӱ6~^vMw{ ASLk0r.5'HҠ4o"SUg y斍>"@-g*_G_O'*ABM.I[x;xɧ7FV A~qƦqДZps"y0f[ⳑTNB[OkݘKlXkehtIeHEݱ;iVxW1Arf_w2#DbmyK94_oQ KTVk.ܻIYQ^E1Tv ;(pq0fGN2MspЉF9ݕaAΏa0sn~'t4%PK}:!%ږҗВzJoאd9f+w-L֦C*+?+T |alQ4tMP$BMڕ׼$U[!zu`U[`ʀͽN2 n Ae H3ɧg OzAm"$oH"5N o}ūDBuqLnek!oƨuINMWS,&dM6~}'IA<j辚TiE̪U)jèqf^Dz=ٺAV>^h! eV6C]^] --B9_'eK^R'X{fQ"n0qeG7 `,>X5,ZMG޾7~ж)%N:0ތ<{i' k}:Wpc^q O.?כ'7ǺcF 3)`_w͉k]ܗM ]T:;skN/޲5ovkɢ P^xllox,@*_ ׮ak9O?\7,!)29JY¯8bJu4JXEBPeAh]Q*v!סܶi!4?9nvJzꌣe^Etm'%c; mmFS}?Ev]%G'_o=JpvZwdzqtۈ^W6 ?%]]gz5,ஏз9df!A]p%SAfk߭z579P ո9a"6"،:X#ɏXs㯘Q b!&֓@&͕엎ˮgåYuUj@Vy{J2&[9&!ri>th}4ƨ]"xQo6x+_5E j5i/V' sXELfBQyP,V?T[|EϱlN{$8;GD9{G~q# э#ϫ'[92-J,V xNrLj͇6n jFb$Rc|cÏ<%w7񨐧@ *4I׵9nW򄯿SqI\. Cώv:LIX\3ጏ;q|pHkjIt08{B~0uq8][҆fD4R>.ž=5eʹ, |RXr!E%[ ::pKAe6ɰ`9fo䏀. endstream endobj 1695 0 obj << /Length1 2021 /Length2 9441 /Length3 0 /Length 10615 /Filter /FlateDecode >> stream xڕwTj.)!)ʀ ]CtwK030  H J*----)9Zwݵ]~&z-]NiXEpruԅ<<|\'.GdA`P? d`)!v0(@ Eyx apQ3A =m r¬!`7.,G`fq p#="rv$Y9=@{m9˅г-ׅ"o !]5EHe yڿ_$ܧ9O!eIt}ah"#/6GdÎ@g 5k>xՓjSյL#ڧ'iF>O,>;#~c3q{[@n+7 =+ɚ:%Ҏ8p#k\|C)pNڬcnݚYJ3{Y6 \@9KuQHն3揷N ΢R [&b O'k7P'ߏd3/(1d^4lP JEFb:5uxL.,;FC'VnɁIc1E˜Wn3^k5!D azyפ~"}L/h -m y'%mvkV~C3_eec5Wb_dj#V!, jOKk&)!#x?rpgI[/=3 R"c|a TE?!Q7)N_g@y gLG9iSY1{,ةt*z#b.`dUvNIs=y*[.H/av~j\]N9}?:Е䉦oыyN.Kt"y>h\)2?#tЯnH24V< tİ]c]{vǶV5!sGp2"^9!#2m"x&NI3O>"?V?ѴS7繱5[lr,]D_IʾF O@*(sj%hKq20cƍyqqH E0:BˣS }@n ̍L@MY 5+X`X24`J5}O7e2ؐ?'j/¬xFv5jΗ2>j9* G8--/LA~ؗPXCX8 .گl&^#j'YHzXqPi>F4ٖ%唡!EޝuWQ.:v$VW_>7"h:$DЏ+!#꫋#n4PNB_Ą7i>Y3ƚN[N4e)|vWRQNwtߖXHz2T[|V0U ך#y|tr$LW= *w'w40ר}ts)z"u6,,rhR CramwYZPFGj5u;]KVW57 QgTJ b۽J`9`߫Zq!Z9<& lN0-mn^ ٽ͘yg5դ/TRvKRᇧSG}/A~?tʪe^oʄ#lT 0j!])~m+6,=NX&֐SI}%`PBEfpD5Q>g'# Ŗ%1ѐxO<xx 89[u+,J|x](|AF_-z8u)/yL:3m Fs6´8%+fʆyY~:H^rUa2qȩ1?.FUSj#f Qћ/׾oۅ5=_K7ʔ0Eb م{ȠZsrT'AY el_}EJ%A@%#D{Y 4V*k6 y)8X;.`!NAarX#)ܶ X)%]٧iLp碜[L$MyV2Km'/e5XG/}€xHVһJzMV< :5 ^>{TL(şwવ&XI-y Vr h|(ɨ;<T~D;hШ9ycmi%r1x?E.PBh߷R[h ~l9A}Z[@e=F'IὮhRqlU|O/ݖsz1>)0):wxޡQsu^lf3d1l'1*)fЄ$rAG! sCW46t>]QoP4j)W tM"O;gƗ=oxhb^ݻ2P>i^S2GY8Ԟc"L>B뱯Wkmz&$TFe{"zXBgbp&xdjuF@Az2r,)mcRXŐddw\,;mxM9&m5e8nn'"mߘ)qxM+]Z42udLsZ%yQúOyFΕx[ahry&ө-)!)Q# Ʃw^eH ,c>BLrb8_`ɗ{X޻)e4Hԝ'dK𮲃 7(Z0s(oyK pOq[8߸$EY(ثBe ?[ݡٹ ozIщ^N(޳"89 #yS5)/g=L `C1Ǿpnchٌ96]Lci2I5+%SW7+Q,2n+϶bLK,߽ec+)W= mc|U5H$UdAH3~yVƜv-gH vV %/:zl.Jװ}37}wqt=Ύ\b:!EcD_0.l3Ore:opɒk%ζ.̐K=%™Wv't)sN 3\림Sj&'y P<&=lk {"6ϱ0JkQ[   wGH=wߠV&&a No8ưҟʄtvsVXJ։Ao-[iMU<\6h*וO(OCްhhם[JkIiQ]zw-\llOp",q,Ǿy;'Ű$Q(!JFo!aP$Ú<;@bZӫ< ؚlS:^{-_ι[ N h|>#J},1m~vk[>Odk/;Yilا >hǍ֍ sE)P3ҿS CޫMuUImcuVtl>Z\ EI8 tB/1f *Qs3[äd9vL-٬o.Mf4n#=EJv{Q*Ѡ]VkÅ_`#&pF~#T[dpO,E'*"d'I">KܩKyFN{!+=&<1 O{@Ť6s:a>Qv1K1&0Ɣgև`Q|Q4}gN]ǢXǬCE2N>[ډQ$Cδ1ɦ ŕK!@,e@:>r.t+zZ] :BlbL?y7.\cU+ Dj d-#~&dH%MkǺzx"U7a{-]KSwyȇ<3 8[dGT◠`zdݛD(0S[|.0>HjI_K8owv<_b3ZhP`/wbe]eVU;W+w90HS+SBHJ bgrj6wi މˊ<Ch{V&~\(ix3Un#C?EC[S1ڠg/:?L|dklp,U/8[xR{Bww*D.Zr5No`(T \,s_Z ~s^3RbI\r3b3ruJo847nt~{7ܭ K8큌ĦDeRvA k=cbCD NdZ Rcw~ZJ,ɮ鏓b [ʥR|~k+;$~0tq:13|^MLҾ]nu#u!k<^߂ex40LOwUIy$v&)U翚{ ̸b=,^XLN~3X>ݵhc!3*UB+ߤgί^GNBqwݯp}kLC9=qYm:ե>f0sZ]ɏ TTTw}L]Eg5.ߠ36 ]>VLcv86lRmuj 8+n6|TR(!ӕv HeT[[GUdԽ@;7Y|B{`!ȍD~ RaCE.NTlxrQ l>rKUÒ!>%w Z$u9ۧF5i: \Vsc=~%>8qR? >sQk9O*|S 0]*= v'ZgV\W2[SJ]ϐ4 STxgV3amse7I3Cӳ|MvxȀJ./蓫ܙ,r#x)"ڡpOGNKxc02B[ @p>mI% X0_T2VP]8/彂jҩk7ḤPEMN$[Aʞ^jChLC[ _<{Ɗ'Wܱ 1᦯F {0_etFG/R++8ϥ.)2y^P9TL@י7ΰ|UQzwGGD6w:2V9?`$ӷWo;Q< rHh8i9akgmlp }j5|cb(I /,_)ͪ6+-_JK_egYbV1ӑ7%90CD3!:P[?-Ö'q閖޻I`&g5H61 ?mT^(^šҰAmSx=FQx+j?T4U Bڄq>?|f#i>wE#2v%S~JݜA&z~0K/ 9-KQvړ,v9ɠY\9hl['-&<;|V1l>( ob^;ZܡVL P}&k,c B+(Rŋ4Y,4?9>hR;\O$~qz_5~nC)N:DlLPhP\Rf{DƷ๸0e"x,uj /[6EDjj~g'0Cb[io?kIrJȿcMp&>AEnXS]!9v>l|Vp{ vkw&w@3i1MΏ ';SThw!*pIsIsv߾ fh;+::;>6vLW}lCeG No)Jm۩渑&_T̬ˊIvl+s*7ќ[_ M/!S>X*Qw]ms2- e;3lJ<-nt%G9Ut짞D68ҤR8:yZ=H.Ğ%C+91;a0R(A!_4)HΩt*.ܑO#zKt)Ge#|OrNts 6 (_hYc>ep]|_d\tq (qAd~^Gtimy[N'_ϰo:䍷A}DžSQpy[Ks`߲ct5~񇠰`"p;q}OOu·Ug Z8ˮLh*>3}hsYO. ks.Vu+_1.ZgU/vV?s>ަTAtXk}=)9֖2T:a- f `}K2*~I H3%>]?8fm;RNtg/,('ۙ*`EErX]{}ɳԞ vУ詈YI;*s4ٔҙ/n0z Glh|kJ YOݦ,C}p7CVLh 0ùYBm=r~!b &xBa/fqbWlw+ܒ /Ow%bK(HBHٟX:U8ʎ.|1uuA0ڼ 퐩nzm; ޗ\gu]VMWE#voUgY[[g*Uo H1!:cT?.{_/o˖%G\H,Dz'T FvʆDm | W#4'YoZU}|]&Ss{0kBρBխ-Jk03l\OKX eQ } =ڦЙI Ԉ ކ{b>ԋ}| J9%]Ellki^Djj2S|8M-Ry$$+.Be* Ko( bsT'#?c){ԣ^#| ۄkb;.oID'F:g!σ9qVFM6!]c.IttORAs]썙O\ҘU=!tFqPG3%0SgT_,V@]-2}Ufٛ30t2~\Zlʎ$vģ#Dׅ%a}.4 ݁ vVD0[Ȥ:>f;/n:/FQ> Gڞ5Y~;+4F6%B~wKkᢊHL:KMB]$V{I*U0Bl{k^O5&-RQ*"VnI_-_dGb8iSo)IcEz9 i)u֠ endstream endobj 1697 0 obj << /Length1 1415 /Length2 2754 /Length3 0 /Length 3636 /Filter /FlateDecode >> stream xڕS 8WT~҂2Sm, 13fƾR=DQ)KE Y*o%*~o>}|IQKk9m< P 9{h"ۓ[VJg`^P1k7o{ۓ.;tDp9‰ӭWRaht7he_^\)[5ۄ)QgQRMilȫFHNkpG9WŘ]}>Gf92R|޹^ry.05cʳ>#[T+ ];+ '|.'S'ݱ^pwa`)1x-73ʱw~OhfpkI4:uAkF Ԕ!SKbY6 F2o%Dmݤ{À̼UbHayd{Zx~!vI@WH^ 6%N&\WU(1j<~Oyϕ^[_\!1>ѽ@'EMZ)Y [gusp2^gC7>Yb-2{ym_orKQinxlIs_|y Kn^utw}3d=mv6b6 w@ݣ 3,CbXIfGsۂc=2 '+OHfBL* ,tZAd }hHmd0]^^rmcR v;kӚ;C/B5!1QfDžGuXq1Ni,:%88vdՃl2=ekJ?ˢYX+^v괔D[Ɲ9RCj*mn`Lx4 Tܨż9\6 0"[GtR]u~~vPz29wiUr90[\ ĺdk:_ʁ' ڇ<<68e1Z#Ǧy⾟C|啜B`PkQ05mQ;xk($k9>hEHhԥX= Lg6[#J 95 剃MFR^<%k5p";sw WCRd}ݚSd_C{KEHq=-R5aC?@ ԿUyL8cDv{Q k?U:x,bڏ/=@d4/o_[T_.9FxzQL}[[!qDzT[.c"t1 CVӌtH:Otg*8&7W8UsU-3UsanZK ڽv|` 3l=)(` ߜLSE+*>>w̋r9xؒzY<@yolCNUD4qύξR}}tfuKCSͿ\!( endstream endobj 1699 0 obj << /Length1 1264 /Length2 1538 /Length3 0 /Length 2337 /Filter /FlateDecode >> stream xڕS 8T'&)ILyxL3H63IQ!JOJp)"'IJwJ.9};߷_kZ{M|,8|,b(nAhVM21Ep<&2C8Ja'|NN<0,@ lFbR6a)ƒPBXLd xGL1Iq6 xd@gEQ q\¦R,! B `i$-OH kI2BD6B(H !Bx0*#2"P>,ρ;X!FoFT xX*++HP$\οD 8:'E$R#Re;|GL,Q\FX+="d Nn|B`9\PѩFe@| E>Jzb0054x!Ŀ#OY}[H6S' Q( BK|/ A(,jb8$x,Ť$%Ӷ IRYLfiIUv!(- e*M-E`5Hڟ͌0Y @Yi0H^TJ(\UbU_mqa9#=z%?x8/~RVV\ /V%^hf@_6?6ShK nٕ.Ee5^R= δw_YVY,]ꉎ@ف%Z -yuv3=˧u)?H=f* d*x$ϴŊ%-f kT]7!rދM:S , &5Ȇl5knp,nN@Fиw{zJiިÈGꇸ ] 0ܣ|͛oj7Ϫ쩬[.~}!,lٝ-Skڼ]ZzLmNwt mr߶.SNxkvga:=zR-eaIcgWzh;KeHi{fBeylp1*نjߺ4Iyxɬ5/Ϲ[vpn nWLSX߶f,-X)#i?ZQ \G}9Zb7PX99ZՙigajDB~0Yw[;l@MRBAFlsЇ}^q2iX詔B,Z9p['>Į2(9f^HvJEAVz]txC150q{~Z0Co󅻦4/}}~cV}%kt?hK&֩/9:($R?[̷k=xE#^b)9|‹ z+|ͬ;ם>wAdiZ?B&**}jbVw %fIۗtg:h w%UOv5pdLKgUu̶JoӪ OO'hl|[iF疧WDG|Mg29J.V-XB2~ Gݭu例Q[BtW][>Ք& :NV:Z/8f.b?t7Y47v}ْ} Zo: nL'fQ}yOԻ7 m󤹵Jd?>TDV,dǺS hkjQ455첩ʸY!BfB3P-*}:4<%-ξ"VwVՈ}w:즐+uPvuy[~vziUjidVaM)-u\ l/i5DC+fߦ{pC Xi7 ~jߑɈye6F^)/ɻd$׭Ac\=~v8{LN!SKI;7CyeafحKLF H|@D.ǿ9Ì-[}Nwi O1EX[~Vgv u?j9[3ZIM3'M{`bKcQT+68}> stream xڍtT6! 0 --C %tH -"?sk}ߚyk_{;1ʃ8 +hk @(HfE9A CH(& EŔ@_D8 J IA Po"! PC->:A)]P;{$_Now3Z =FnCr9#;YNeЇ !wd64>6=n! [ j!o]``p{;@ME ;`^Phj(;#7'vr:yPvs ` k_ `[P9}P օlRor+*l_["(" ! /ED>xV!Gݺ\P~[8H6_oDTo+AWۻ@." w l=ۤ>^ebC8 q 8f6~볈t$MZCɧYT8}tsڀޖ~*Qv;$ۉo3>v}يޭΖ&N[@qѯYly4ln[oZTGoQE` [YZ=<.cOҙӳi܉&DnX"uY CAd+=Sw> {i4>E+o<[D ʽtw(Hoqnd0:,bLYi\m,zSpIn9G(^WQ=f.Xh}{֙D\фNO{,(.(pbhx%sCݞ|n'rb5yKAg_di9Λv$Ā$zWʫ́Qߢ'X'TYi%^cyRZ;Z-?0y¥~.I;6wOϯ ۿ|TA7I?|C;tFOm8}5: .06SMٴ}H ŏAIO{h#t[0hR$ UtuV, c4he `0 g0 Ӻ;'ӻE7ߩ;h]VY}n ,/Y TC>@DjEmɌEP s=LN㥙gDwSQu0XEU['Z2wRD%L1&Zsy<55,`}=05Vq" Fp&|8{L]Fq?Ք^)QKߴ;Da|xah@.ۆ)? upO<1Kk]Gj;bÎ7"q\Jq| o/VLTaߦ]7>)|j83W5#mTz;n";@Ƽ^4=ns/dv#6`tX%t/oFMjP.X@f]Ec;-ɳtw|>@[ўW}2"ANFxkqE#󚋔; x Jhf>f௓zhNuep_™sPת"i&i56JYߧUcK`s)+MIAIw(h΢lȽ%}L򲴱{.|~UlS3ozae?bi0Z1%2D"$G\՞|]P>kbQnI}ˁB|!UoST*h K؟tL).Q^+ȯ'ެ.YFG;Zܘ?~ 5{[=jig czơb<=C)?!3=Nf4>Hcl ̉s.KeO҆{2tV2B+UEcZh_e1rpW7--m$+^^׳. g21\>ƺp/5KoQyy՝K;ml[Ei&=n_H{^pr|I Sδ&ñ7q6Zw>f KOXϕ>z/ME"h>xB.kt^omx0"1K&6}ggϲnpb'<=:3QZ!jH+CyZqh5-K+Ptud&Ǩ)@|pu豤SX9Sn! L;6<{%C #kZZ]*Nߨq\g}݋zf41P,wG< Lo1 ]bILkKlhplW9 D;t^:MGpLE"D%P-vgIC<=%WKkm1d_}oN);K:iCpس+] +)X<(G9v63HNlʆl|:{2YtwV*RT~Thd.n]p_m}x4rDǂoTh1 G'+wVr03U:2} kML:uĘum|!j$Y(| yfb# =j3@k~#}OPߢ-I?[=̋֙G27%,`_Q*n1%e?Zs+s^>)]%HAL|AŖsz@#q–+xJ1l[cꧏ4grQO i› -'^΃µO?AYgҧ s a5%# HuÙ By{ 1VYNiɸȝ%;_ NBY݇5 UZ6$r5lՃ\ F"3J^kUNm3dn\+t$1'Ej;I!~rK5Nwu(h"'n4`%|.#ie Vca37ZN7d¼E8>7ZG-޾ *H ו͐T(b3xVE9/fP (,? *hVU"-ar(ZF|N_AÆ]+ q $"nA[JƉ^4L\ҟzЫml-fgWB±BW#IGUߕ@C*8O =¬/^a}Pg:4TLf|;B d#O]b!jnz2UitgK/U]e,ΉH}h$ +jEwB{bW]Mɘ3VS1\b>{S7;T걉!'llRMIVUhz1n097o8_ jywm'es5͖ |k{r1?)]c!R~B憈wk#)'XMBd6M"~ϚEYμ`k! ď(Gb\ݯK0 yՏƳn^+[KƳ}'$,dN؝+g$I-qdҕȯ8E8a=)͏zQ:wFٔpͅ8_lKLEv˖[0~Lv%] k:s<[S_ fHɳ!Q_0s,U3^τQsr(/V^B.E8%7~եό;_*Wp$bf ]',iGHG5 Y<'Y^Z V@u,vAڭ.vH5%)IMOfL[?^ð*~DYKIvT++e:Wb;ZrJwd+FY`uC yTPR]p*00Mp5,pn,OdVvIB3"3 ~g ǑMȕQI)UJd#))3 '6ԉhMֲijOйNк&;veo~TP%x܋>/<99z5=>Т} {+P#KcD3|)Wڪ&tJxNt߮2K9*;΅ !`EOx q3ڋ]Vvwp Ÿ Q)%6#Z)οz]m$l2ViNܷ~ky,НTfHG5Tj8MnOÁrms}j- Qiȍ<:*7Iwwe ̙Y-;{B]hߍ"S c1*SVC!Z}S袑>rkǐ!2lFx2)eF${{@KV%({xG-%տ=%2՟) 8T`kNOqNԓ >?H3My^`חl6:GA\~ >1?tqgcߵP׾-@PL+5i߳6Wui5W (c}]Q@Qjbv"ihHi:T檹v8GǨ?\Y;4dQ%>yU-1>.AZQ/-qU0y_#5S.g06]~Lvn DIh tO4BzK9lm}0tcHO"(:#^2Ʉuov|ȕFL0jیO){GU$ڙ'ZȳA-m0Z)J7DjwnzUQ'եӜxO<ԉV:uޝDZ}CލeoP\z @Jʔii|RqTA|`4!(߻cDzy]&ڐ+8>F2LyV֍]aovAG޳d[6L/xu WbW8^WΒl;L`u:Q\nm&yYq&/s8F LYa_dh,.lϔšHOG"1!>:w51}&M!(ͱoGNtzTx{|-W }t),6 y"` ct{i<;ma$x`8ڥeH+bF`צ8zwfLj@eu'4Q1F)scnQlΛ|0NGu[\ncMvpw$ߥG\*޹6|6 \c|KW36O:: ),vM5:Ytrji 2\˪5BJүcöSYfLVq M~Z|@cK1!v{fܨ]/ ׶nVv>vBKiy [b*WqGh"+h[6X 5BYo +γW+}z zE8uC#8ڦ["1Nyab'>6Z?få7VsC+*0D93ZlkdIJ򛵣$9'W){'ϥ00' >eDo,>tޘx7=N&H}5%"T@C3) wl-~i-%K'ƭ]knh/^̘zOıa!#" zmY#o/2i=5{2qK܊Ip;%޳4ӌ(IJoca MmX*?_9khc]$K +ajM%u8:"-^8tҶ=8s%S*G?\s|{ߎ0fMBr/i0fFZyV)ȻN{$=,*hR sŖ^KtoGd*"ʄd|$Eh4C@ SLb{a;[GrJ^qK1p(>kg#>kܔr'+Mئʍb̓r?@t endstream endobj 1703 0 obj << /Length1 1390 /Length2 3482 /Length3 0 /Length 4391 /Filter /FlateDecode >> stream xڍS 8חWJ^k̂=Nc([1 ØcldIeɞ$Vd-DY [w]wu=sss92@]2&•sMep DDĂ@#!"g@'LRC bht6M$C/"JJp8 +C$Sm7@C2 h)~T M uıBQN8R X 0\@wX 0'c *.4E b=dChz8`el ,\?d<CH$Oz R퀹1p~Pįt?H;,N$gO )]c(͗&`Hm"Ic1"ƉN)jz?R 'ԓ@>fNh TK㺑>$@yQ`I/@'n94 E(*b]`XQ'bL!Bxz` oQ;D4 t& ? C߯]a820}#K3S˩IH@ZNSY9Y@~3 CYǿb Hx2pG9SO \ \drA@‘p,}'SvS]A~;EM, #x׀ək|O]/ChX2ٱtC’qk"0T*kA`>@0(L/Z'S!O"tM;F R0o$п,3ؙGX/*q;jYoA2V9Ե,n!.9GˑotzϚh|Y5?eL}7+>QJ|u,x)=ܪ=df>q%>}RKioTDK>,U}r0vEGPP` H/,o0sYk>NH'2G#Rɇ޽SKk0rD4ؼ.Z,\Uo\˰F_ǔp!kֆŦ¦|j}nbrCl/.GcxȡPZNɩE.y1[cRmٓQHvT)TWE穀kDuP ŏ1sc߃ Œ/z nA~+5íKt> IΜ yґF>ظ~6/oGnv:l-!ӣ9{P8)C #j Yhsx-lz|EVУȩw>JM+$uמ͠B6F1$_IM*BFֻ$|+K *IyjQVWG9ђfh+:Sbg9`a3[|S|Խ3LHk*/qqX)i@2[]Bݞ37Qg,x70SXJ^5<{^e6, BݻQq6ذXr%]%(4T2y܆*o5"+(l`^fA_2˕z+ɢU쎠+ǫl֙޾ Ss̯ū--c>)oBJ*ƥh|Ȕw ?;tGvmx})iLG.u[9 I2kv* >p (:do*y|(@J'|Hɢ_'ftKP, e6XgK}"'!e~ `AhiO]QGjeٜLv3&)i͍l^+{ǒړ$Q̻[05swfƦqFx/Ɉ= ĚeξMar=}^< v 瓛Ⳑ69ov-|=^PR;ړ\=Lg`tk/bNQ7Ta h'ݻ}C+袞JԨk⛥]*E d;On\~5|}DϮh4W> S*QߥJ^nf! u}z1KZX;;Hra8Fa^5,?^l &WRW;)$In-JFVy}N}&Uٕ݋nR -Wu4'gϽ m}>r_13Vg!c}(GʤTE{XSDFg$Qk*8vY{ƾV:#Ь7G=~6I~7Qi]ܬOy`#r=;ڤy{j'wmO-f+}a4*w<{(\}:~(x]Zv-~|/H'_M&UWS٤$V {梕b$EkP4x̅zMs(RúUzbEfz}ҥL,UBQPq磋3᫮.1׎v9O|EF2/:@YU2MK>Fzsq3M}74qu L  *a9>'FU_h#(lܧ h%oQQ0+*籁U25΃âZ+N  p\O+n, ధO&PI;9c/`a/F2zdYfUJq`̜‚Pˆ5 Q{% 䰞JD\(jn(7HFV7ޖf(gVxT^7QIdvE4و۱g^5=+vǧJ|psEe>5L0vjx!cvO'd&^)t(0ƙTtMbm_C-"TvX=1+i)G1RT;h8'aATGڌ-4neǙmUL {Ú$6tK>M?!& Xva߼[B3:㸅_I *GƗ8Hr)'¢bS^e6Iz:slH"wʃ3 -JRTboW]:Ck‰S'_S#|H?&Er_5#bfR:e3ơfN-ך6B N1<ˉ|iYox~yՉoYX̊&[$ki3}#Sz7:ՐȲ#:}FӤycW] ֑oOI]H kKx4A u^s6ԙui봏 uZV}̀apkÝV)#},EbvOImS=qsJ:enY뚽|JUF4S[3+bSV@ى+5Kva%fw|-qvХP8 %34*jWK)gKD?=2e4wwrW} ͿgapM) ůO2= iBiUW]nzRq 9-Edv/Ob:?0CD+Z?Lg%,DX*46L[ƨ) [K~.pԪVđy/``.sZy}Hkh:&[nCK/^xp) CVxYfFu`5 g'j\泱Ŵ8Coc> stream xڬvP]%,xp݃!C {~SOnխs:{mYGJL+hlgh"fgLHP Y>, Ž&@g ;[ @ t029X9j.N@fNf + ; ,)@`laf lF&pc h0s;YؚM`x؛LF&Nv;JaDW8=Ml?>NhJ&N&&t}h #A[r3d$fnb`ld0Z;ͮjkl,|-hNFV#SZWbJʚ/?-2:VZ:齿=- J_#$dE ebPbl>iQz[31q71]Y3@y (PLm[k i(oL`gjM.uVCPy7,{疲 ѤSYFV1aÍ}Iʏ՝62`&}+`ؑ`j*WHoE=yW8Sm(H>tfSwb*v_V ƆJٍ!_c c6F=Ϧ~ 9q,,#L7^P>#`w;;1Q rkѕ}g:I `0^0z.+3$:/E&3f>#9OQݹI2V\RrK]l[P'<7V75=Gp,=|Ǝ+Wp$#tEj2J~GBZsv12ucPUW_Y/)l-M1#*] )Jͫ w, joH^Bu5<'DLZ㪼.Bz;M7GaQQhncNDNz @[xIt7Õ= "bـ(6֧lZOpGTly[c;Iٖ枥픻/S- g{H(UfWsC6l6Nx6s# ?Nd]lou\jҺﾢEl%p{:2;tMR1589_mPNi+AE&(;,eYW81A(yf aSM-B=1jxZ+[b%#>Ggǯjt2YWS&SEdŽfR I}Qwn _yܺΖs׊v 2Ӧ4iB<YFom'tjdL#]Y}.j_}=ÒTL8d* ]#s5 T3()W.BJdᣝ] AH>YlYr6~4FT\8J<>dWyɒޒlxׁq5+7uP8*?'d'(Kzѕ OH )(1R +: ˿xJUygaN1a*˅{!qT, F5+2HFJ#U-i" &MbnM}'OH#7_ԿN7~pQ˭+eX-[JP UW ?)(y| #W]~ Lx[g\3NK0D760ߵ!6[:{f@] Exrֿ\mRobIB (}sD hzEϮ"4}^ Kv7FSA"jT`UR?-,w6 ν:6-] AЮO[Jc T3^ء7HOţWFt~#ՓMHCSjIlCDU\WYſLf%+{i3f[ М[!oIs*Vn`$o!8Xsv+ǧ𰈒ߝpGHF~zw^LyUӺ%"Ŵ\kQ`Н! @$V~3YSFd+sIvD8tn tv_,y|[sb@<ʔʋ)pGKO*FaPfKXDq!j8!XN_qU{tY-6/l:]|ߺdBͿ*KqάJRK"3k;?'k3^kc][Fʄi ^;cs i`[l[-&<Xaiԡ/g~nY7cu6m㨜ZnjА`CE]nR-ֺ=i'B$a"lqmKoa>qmBAV^IAwMi o/IJYYTË_ q&U_0zp[o֫M4_#XKєٛ ] 72Cd}*4uUÁ9/u? Rq!RL*&iEQW8Y_sp8j7loBeX8/%jX0nlDLFPɊpꠠ4?-`' - 7!PZ>$͊QKሔP7<ȋWkA^@06t󦽠 v3 䏄Sm2l+ 'àCbHs|ؐmT_c&q |7)_vT:^!tJ*Q̜`fI> 1.+ w ٱ#89 Vt/˽͆qHUs ɗQz긇K=Z& GLꄪb $_~HgqK]hŌ" /I hP=BI=ŽZĹ:q?{m ڪy\*6AQa ȗ-]5k:!P21$LI/ن5D U)zea M!TCjP8,8 *Slt}qqJ BK賋&vnZWޟCOp n[ÿ zXc23Ǹӳ:ٟ`QrDcLagI3KԟfN}f˛Q"FC,EEu<7Zoje,}=$Z $voCSSSQ0lgF|ø O;= CҖ'Alf#h7)zᫀV0Vzԑ %{0{ = o8BhUO'6gsE\OMN>ii>AH_7BMrmRzȈ,SXN-(ד 4vv{EK45ѯUwjzx]hNuTcwAO&6dcSc3ry_4kϹuSa*ݗ !O:RzD(H"D@tdQV'F𛪴X#0$" ɾ7ӂ2?'} ;{5 kr M* ȃUɲ2?,f $#?~:q=^Itg䰯Ù Կ(}èn^Bށ~rj9-mk \Uʜy,%[W ynM./|(7p3V3,Ok0MjҶ6Hx-/zyw8Ʃ`E-غ05gKs DicFe49azL`GgXT4Xùc@s 0yg~v1l gizGLur.? ߥ.2gӯȬn>!dU 2"8Ź2%IFN7dw=é%4Q3rKVճ] g01 "x\\KU\'{)uZ_=O\Ԟu4HQ۾k=K`؟^w/[OQ 'xRݬ&U5wáQ !Qvّ3xƗKu\:X O ZYVU‰B^5T>.F-(VTmAcm'lW~%2{L u3\`9߾Ovͬz4TPF0i8aUYTDuC7gmjqw0WOr HnJ>Y = # 2BK 4v έ}Tp?#h9BkF+O-(c-gg_ @~:#L'!zCgͮ:9gfb_Ҿ#eq~3A_8 2y10SPYM?[EKNX3@ T:B"NoP9 f]iK2/cj]pw7N$X;UYYZ-Ƶa||a%cly-g DӷoxWesNb~Z5H̕y I<2K?_;62q[`v?\r/WeܔEZԦ8ŖD0Tvs~*(3~i\fkv.htM컏3:g8a# XL2?KImZr7o#~͵FjYR .2qwh?Gs&8Fmkiѷ{t1GX![`".0dpӸ&UX\.|z(A*&\Ci>W19C&M6$g*BxiVսAJ7-WV*=?\:( _dlkne.4,2*\a=b;_&$RCӱ+$p$/$E'd! s"ORٲxT@+XefC,ZZe*]ik lGxC\WG_A/O ,6Zs2mL~CeS9!9+;#~_S\WzJrN JS-.c}_Hd{[c#%"K} 'MS Os,{}Ai}o7XI[FM"za٪9? kW\Inc@V Zoh:-Fv8FL(KY{)`>M`}3 Gۆv\y0q,?" s] rZAh72^#)z/;yLpBM`(N@lo^L)y_nmݮ'qRtq4#bxly0it| o%EFrHe;0VMiBtxq ? ⿧kX?:?<„-Vm$,ddTh6љJՠh,Յ7P+(7vgoT!kQ K̲ҧs+B},G sKM "ƓXLl(sEybdKO#bYrɲΌXW Yi B.WGxoąѣ]-G}\ad3@_68J䃩%Ph(XNdTW//w#VX}TެX\+ ?`[ Qz6 P90_U" ǔ+d,:}F{/!#%!yو~+-~1#V~=6狽K־cfG0g)6_+m:eI),|gN+ jW3E75Sp fT#;ٹ=lfW\$71Ib A,Lo# l[";` K~f0V $ &x).MoGtv/]+kH!nupÔ.:ĝ$Yz;锑X[e--b_397N~/x`5tG RY[ȔthYNUK.>ZuaLzXH }F{;:$Gg0kmd|ޗ9pT=8SK#TfJ4Qcw=$-3~]f҇0|Apb%iڦOĊl$n%4Lw`PW36`Kbf5O@IWNyT{u Ga40PD^Scd(uxWDdkNTEb xb AV3,jT__v4~qiEgMig$~ׁacQ9 7Ǜ .!~*dNڭt6Km8 ! U7'ew+1 \VɆ~{N.G*D;n+6Qqä7e ]o4[)x]~:5?+5&9Vi:UQ(tam1* ROr:Ƭ]34z\ʑ}T]tYI bd pķ6d6iYtF &&L_֖ܘJS>*+HЙ=?gc](B麲 u\=/mc߷yy)\³8ORM_]yʔ y mMVz8NBݓ! 6lHTOhd9F=XN'a¹|%q5^ߌK/_ؾiLK\uv3x)ƾk!k.<_ _{̭E[ s?hdI^ztmB3,?N@TZi ;cw[Cc+S2>asɅ1q JvYt\!0Rk6)5v=[?^o[%&5ie/`Xv*~}*5h0:( \BYq!ݧ9:mƿ ֽGۥť9")1?&1T64G/{XAcA?;` $!k7]"2yl WP)^Fv܄4\Qw}qd[*c>wjz}̽2}O'\wԄi''v5"xs#pAZY$sj:ǝ[#@ߑnnt9֖y!͐ aDNY)ۉXT9|$Zŀs^AMh$9`Fp>O`3+[n˹;$q8v5+nлˈ<#r3{JWz$*! /o225sEsf@LQi؍-0qlq^Vx6](/}|ku?dݓFEQԈ| ߮x#F,7$,Gf/fF :vʔ )K8>]T ۶ n1HCT?C3|tvHRUNvם>`na]b[6rm]RIrtoIN!&w͵nlr@MrXI` bV,[W8XuT fu_sFR2iw3$sJ&.~?4 A&>w<Ɍ6fW+R'*ߔ&Dul!5L HcD-)R] M{J 6U·LF;('zG )KܸfDE.\.H?Z4gn40JTJO\"LkϪgS&ʯ&qam^ѹhB,0Y>f+p*U'k.zu ]bd/wp7"X3lL]J7H(/@ꂹ.;a'0؀AlH ӗ= YWHr=ˑo1p3k::'!"!\~?{ӐViUhFp?0K*i SBja2e#SM%=Yc<$3x֧AL#~XE:y5Ly|I*TJ}paYu,um 'EU5Jfc^b:(7 akLzDB8*axوw̫4,MlRMzZ(5 (l~9:Ѭ2K80YB;.l%+qsm(R`2h{&@uv=v&z !"G_jmɅPZq>]V m UPf.S :4ɐǮλwHxy{^4.! #chP)L2ZDY]ǽ9\֞#a^]/&dl9 K%(j(cԌ k9PífGwDq7ymJm6 o 'i/}VɠU+>'͈ϰ3K'0"4VNs+Su|"GJ _qɪչo٫t!J|ӪzBb٤ @MX>Qs8k?4-uo.!F0{7nY\Z}MhF]   xYtRޱ[!_Y\J[yRݿb*W]ȋ` ]6wwW[iُ΄@Ӿx0bZ 8J-o9c,+hP{V92W}¢IC9&t e`ԶlݑBwU&cQaM00PVdd3 ?vp<&^OZ*.gj B$ݧg˦I{6<{hQs0h#'A3]'ӕvҳ`MH(  Q7S+d$ qTcRWe= YM754:FH#lOŒc&!K75!&+t)+ H xPrьqEN14 ~P3E!OJ,Jz1gt8 Oؔut3 C%|%g͞n5k#1a[3^`bel_iI &S)Tmi@_{ R*v4Pq/sǖ͞ x9_d|7!4HiVKtZf]P=k{ @+VJ]3rK'j̦)7mJky׮Vpe)^@C߁}x$0MycSWd Ǫe2"CJ] %ώ;Z *Dc!- AO l06! %|WLݓ3 )La~[EE eA!/%\:KV午QSNW/q:1+C9eUى Tu||7@hܔVh0`Y9Nut<p~\FxʒMsP.$T.V# i"H$".J>ik@6JV4ׇyZ{Xoe 'SPx/A=eaWn(C>EK5vxEm;)0q(q7K!$u;]&Y+bh8d p&y Iqۏ_ހRVauh˕kwCks 52Dȉ5绦N pس'ѻ2{yZ7 K_Y=ϿF<vmFj")ŴEJm S<6|s<hΈ\`:ǜM('9HNװ+#Fi6xfQ\nEyYmZȩp[ָ?W fy_hӜIhe.[Ғ06RR$ g ̌1?>JsFg`,Mڏ'j%4ɻU^$UYB PCSo*}1]z2Ay,tzSÉN$F# U4sin't'ʩ+ΡAd0< N !$VVJ^QI3H&—tԺM8c|N V6`480[{=¨ Ѻ$3EƮЗC۽~~VҢH)pmY'`tĽH,ꪙQ<`F;hj_;[iSBH%g-)My4x ƍQI[_K}E(O'M7{VZ|NiAƜ0u$2CFwYھϴ"䨮(jr>f1Y3ޛm&=y2ZҐoeH ]Ww0N⟣T JIE:'ZqḔMy 5i5QRCLyC5`8SMi'16sN4i ;'UQe,>`i'>TRLFs@KSB.O) =ұP[}Ϯ҆ "Lx f'Fm^HD* 1,Π|ۀGr_?q/'sc\4^KʛnIpslo \.?ӄ%Uh ?!ᇾS_r8 umkˌ :q{`.}Aٳ;<|%BZV@6CQ&HMdU {~lwyMd#XqfB鵚UE:Un9YYE474y{DQLeƭ[Mx&; Uh1|}.9@q /WځDDffTDyP ХpC^(8``'g2G7G#OQ#f6rTj@bSkh[CYҒ+ɼĚj 4LSm?:zǭb:|?FXqTKct7,$\x _?עtbEUYMo؇>"=-~=2qRDD?x-\J@\3Prsk\ʡʲs͗|H9$)S;- eKcFqEF0islO(p$.A Wj"v5Zo| ωU&/,ڽ{08oE*Ž0rvV/v'-P# UE`#勎q^'?AZ 4d8|lu$[cT)$^.{2.}Eg]\G\[mܔ[-!>s2!1FRV12AkNgc.ϣQ@HC/݇dOܠcS1 9CTZ`P;jfk~Q[\ÑL5I=U0|Q buVGWv=uIQ}̔2/aˮr;ш_nd5ŝ( u` >$ػw[S ;:FO6 }O0*Q?S}jpHT{9\QNak (ǘ_N8VtN4@uT^Lƒ Ooz7_| Yjr@wYwHc:\M8KBS3g埇ԀMٿN|Q9%5c"E_Q]VMѕUaZSo~vH_D6,0t:`v^6gR@eI(90f#9?;砥IQ 5Kv`207Hg OzRna%ОRL%wsdq[ + b2brLE|b\9䈸뮂,rF>D rޚa?܊"EMJiNmmu6`9D|\#8sKBPO֤u48WgSk\Z3g:ERÞm(4x1[Pu vv=h1_;:#m:rHj L ݫ 7oB?*rj^ ^!&EVfqH#\50cqvv<>EtW:k~XڣV 1o[A@\OQuA'9Dl ``quOd mƢ8sɺ]f = 4_+J/Iܣ+ &&=-z_(>S;ѷW%T _(p"kD")&?v3B;^^Dά]z6fF2o jdVjng,ISQNz6|W}vR p֘]j%^"?Yz#I"E(4p `Uf(UMGJԭVCxb˜[%M|a0R>:|r_nf̪0c|&s g>9 Z<*K^Fb̄S$RݸnCvZ*\ħH|1iZ5QBPo"#^ކ=0H䈚XvK=nx4 yK^af?)Y,pˡ!,Ng (k# ^Ӽ dX<(AOt+Efq4@:Ƨµ99Q= TdFRhы.cRĎuJ+ ظ)HqoY՝IמWq(u%-zkٴkwnŶSeYN%քVi$]"ɁO*Q-8`HcG+(_r4 6a+1 VwN8YB_NM`y 1E>9+_)DM54cBr-f+q#ڧm>]"~.kT>Kjq#@k9?=DFsM ֳw_p艘kmX~(t_ О X8iߣ=[ DYWLL,qCw8lwe7Y$T8A'#p@gljb1%3@K7t f7Cz[~h$ @z`ʱH(OJ=WU =S٪@p\] EZQ`K--A_T_OFrA/j!)-ZJPR\5ڶWAL82 v&`.V=HSz5f9tXht.ndDS'Hw'"Unp|ehA"COA dȟ7yR4t qɈ(س.qMR.zecv1r)ySPG;gzަ= !rB\igTPӌ8xK 2J?TƟ{91(VkXD -,^; INu[,qcrI=YSW O<'xK~~&+jӊJA͞[YId.9@c50,%g4>[;`d _la4i'u™`F;XSEO]dMاؒ(hl*~1|2k1.L69e},ߤ#"JJ|;7!en'Q`fnKzh9\\t6Ge۳N𜌸t1tVL.MK4T3ȟIQ^ZxSPFEIxMQoњN[uPVc[9WE?ck؟IBv~m~lӅw}A?ƪ,Dޏc8"UQPGN b f\5r20nr)wXy˛@|?fng}NP$gpwlNCjkc!geP$㇗N T䎴9 q<%_{?<^L㵟ez8#8P`Z}))BjCe|A0W{gz7<yͭ}yx^'4b#T\ia Ǿӭ;5L,aϢUL'LcKWudfr(jINqVVCɄ[9<Q]10.tTޡt]wiJ}ݞX*:)(3d {,\pf3kDlEa)̻7U9UKxӊ0x#7 9T羷`)hS6_ɠJ$ڝz QՎfkm#2ZfNq^)~BzUfXT3ظ\kuF)ܹMͯa\|GA>UP kTFnq] 8kYdKSQjH5(Zȕe\y:bisDs|A40#I֭\+r܄^rSDJS})OG0P}0-,saXW-@k -^1RGoF]M-N -n9;|Xqr%aj={> I wRekRvX@W~8D Rza╵"j8t9> a0#(UZ"o^BJ D%Hî3"q:֮"7ŷJ~{`{0$7ul#۟uQvo=@ [ \SŴOOFDc漳_^}N66G!`Wn{ Q5~$&qjҔ__(bꩳ?|E񜤦`5٘z?JJneEjC&X B^~CV>b T~?z ~e_MDkaL@T,w0a`6: "`jUdo( Ȝ8 gIZ-u~oZ5\Y/p0(+j`Yh!DM jdԮ!W=)򯏿T*ƒZ%8I}u XcaZ'ΞϷEv2Ud،a.VKpSW?繺G;:q)qTS/Ĺ C̙ɺ@2&;\61f +mqlbzwxO$*l7qF jv9 *-8u,Px<M!fNߧxOV4}@ŤaMR#NۥJhgp.5s'XYU#V*Ic / [fy [S&?Whb;cخw^LK՞3{8™eAz xA'":8Ҵ \^KG8/Wk"c}a! 1U|PN9+&)6Y}Vwqc4/n98颯 R8$HׅVy!5{b#Zw捱dgx _ ҬwK삔Spww -طW8޴ըܘO^ 3ˡzגECࡵzp,w憌Ie_>ɍ&ǯpɋ!aNxYr\N1壒enb"s]Q`=Ff@Ek:Á7ޓu:>oz@*֏׺l"fjҘQأ5#?QzLBKWS'Sl(:u@⳺xo 9|!]^`SZ~J6w,Im LEfĨV(Ө)ղ}Lͪǹ7fO_Y4Ǣ%*A=*jTFrϵWۡxg XsNF^80PgtȔ}t,I,fu?A^r)_<2$[YL4@MlG |!Bb;6Uɗ؁+`6 ?::4i:EM*I#-}Ic }u󌪆{DCP!P<;e-@/i+kΧ '? * )2$Yj%Gd QxL!ղ巖<4L`h9U#RS~Dc3 "@ ujw{p(kwZ4~`(zte抺YIrVFd*zKF,E?א;c @PPxRy> stream xڬtݶ&\]mTlTlsܱm۶mI%XcL<{m2by%:Ac[C#=#7 #㧉L0p41pp͉d DLDLܬlDL\\2N>,l,D,l\LdDf"csSsG++7"1@ nl`h r5qt1LlDv@[c'#sS"G3_nv#і։܁ hnji4;)98qDS8c>??p;эa$? :b:eFdcHd099ml`geף'̧/^6& sbhfw`W; h!2Dw6@Ĵ`lhKʙ~|HP<0|o'Cscs9J; ''{S 3gޟn_ >35쩿<`m9gF45ek%mLlɿNv :|.g;5& %'O'vUB!#g?MD>go%?k0Uzpf<9YYX,?d10oVnxouPcM&071,8?LbcysG#3"+?*6_!o`}_>+ediW>fˠ -*%,BJ%Rr̚ o8~_ZhJ)L{ucbg#cfOf\^iY}ǿ%׿M?`Wm%P_|ܬ'eg(zڬh@5qV"45V:{U-c_3TZ" $~>H2|dϵCBiL`4PxаNNOLQcs u՟J'eUK!x{ {Ű0=? ϒ+u}܅ T)y<<^dRo=q Ӹߤ[T4x6íIڧFwDz^+a~rO e aUݘAX3d ;IW^[dN>;837ȗϐ"CDo6\}08G]Lw{[Wh8BjΑ pZ`_ cʹ,ޝ"cw\$KD XXH$!h;"%캫ZTAS>.W܆DIiG~mj#J,N AP$ZGICP>9s ϱKXsP򽊒Sg Ϯ bܑՅ&{=7%ϼj,ߖx]rmrJY=8 %`&K|kB[Z~?ٽaZܮKz !ҟC{r}.#f1<_q@j(U|Zqqh=tPR!,: -v: 3'[xM)X9a548#ւ}sW@x铘E'rb uesgE06k͙; \iф0– $ ƒ,~b(0#Jŝ; 5}wb MMwbSk<3AhY_ .9إ(n^^yW9&ؙX[Jy~J!cb]Bla=!.CޙlugAT~_ҘUFp=N81]8u]yWd`%j4TS w+Z3PV] m%;㏀coGcՎ5g[Hg !< Dt^pipk!%"ht.N'?][J-fi\\wi0l}d'bk$6DR$ f*[0G&C`30xfd:M6?A toh)|Zď873g)YJ{a Ne%q]$zZnU%l{Ka#՗;1_g65{ֿQC?VuAc%iO \,R.GܿhgxE+~ IF?w_OGtߞ-K0Er^ CZ$@": ,"o!wT6 B|:sKZ,#uwOV>h҈(r=EDr6_W|ŤA|QiB|w ܲXeV=l+WiD56 ꋛ$*ie_UXpYi3B jn:-f+f!,[kB'rH9h bghJF.@tDeO^%Bh(a]e ,5zUGj-N0B(05Ut֒U=@+KL36!!0cM-ٻǁUdB0޾Eu"cUfֳ|]ӋwsHh^uϓe{ Aг&ldo+xTN=-H3#a']؂vuZ{>Cwpo|i8 񒸐yt*7aDi7+"jӰC<[9 ;h:rs ҝ${ssoBo=:OxWNQj*|"q;r2lIwGhqR.{SKü Ҹ)?/#fDx &q!]ߘșp.x.#9R&u8WKKADGUH沽J~,JW eyOKO)a[n31&B"fĝZtCfa}չ>6= "SM2O6UYt='I4~{Q{wS=oՄhK_Z 4_\OTxH0U(돌C)5hUɷ^\ 30 n\KȹP4]b_#ǝP9'^HF,-qD:2k3Ԏ m$3vak%9bxg4=Q:Z> XS_}wa+hoNPe"vv6ii%@({TpgD⚼w`hՎP0tlBX//Sw ;' yThNͽgR=S^EPmn9oϛ`y)=& KK;|gToB;ERRtg 2,_Cckiܾ (N5)VLS1J/iiNi\#H_" QScm+My# Ip=knƣ דr[f :BaN]aXl^.l,Bf o/4X Wr;K Y񋙜KqMGoqjhBqxğ)ұmQ?3O_x% D{cW'LZXW+!9^YIsls0'Ǖ02%aX*!NH5™W#=H@7RWی[5"Yyϋ[nCRrea!⋵{yvdH/G#95 #@w%_bqxE 2^KpAF2*=a;,`9iҰ홬`w 3qos'{75k?S _v&eD.aO}Y4%a4z9T顅*ld!Vᔤ{ޮ5K?\ӇSKJPbSA&vگ!Q) v<}/l Ƌ09?s2(#\T[ Qb5N;xxVNM NX KU1Q#ڶBiN__ &mJ?fB6 xCU YL'^r&> e_ u9R %Fz*ݮY),J8,݈95[Ygsdr*SؔIJ%K=ۏ/_ cxU5+הUjQ )ab;IX7^PԠ0+٧cIb/5)ujdϘ9cAt~f?>rs -|s$Ѥ{- L`ކ;y_;liћ77s5pcڗ U#oUckAv"cD"zK!#5HL!s&S J[% Nb^ ӱWdyKh, ,ׁJ{e45wRym}.T;-t~ `$<w#j&ĥ_ PphߠnؒO'|7uI+NC 4"lN Le+mAl٣˹ s&-pʗ֚bAԂRyw7Vʶ8M ?O_|ݗ'.oa+γq^qj0:K@G^dc._ K:8ٯM!2 MeԸ*x 369RCl0}w<=+e˹@Gp$+~gџ?jN'Gb[%&MR~t=NP|.>J&tռ0O#9_2$f^oS1un^pk+؛Ku73oăRe*`hTĖj rnGyiD`!!pRi{ExEV$IJ )O ƍ+Apo!$IH~DTǧ5'5S[1 M:da_;{_9SÁ8 LWal<;!OiUZ%#A,kx#]P wNXh5 1orλ?;k%$X|DU3Avn]t&utzkvNzR(5h~n(P<@Y-4~tsx . w e\!#03Aɹv헢jT$F3:9YWsnR: ;WX܋է|^vAP!UaB\!,\Gbo& ro5ar@.Lg$)煉aOzyi6+ \w~/ Z)v:R%|l y2yryFMP*=? Cbl]71?b1|~nz"!@i'WOd5xMRRW/:b')A:\ @ƌ:o(JŹM [MůL3 gɬ_a왯\ud^"g>!>.+S3A+|JÑAE=H1^0`@g&=&3lTėSBӣߦ:|D5KRTjk ^6KtFB/kmfXu$;\~0U;5CG{c ƳF?zfAT,,*ggcrL8bwž/װnV᡼noَ5|ɊB'b cFBſ^^JA,bԖ1q=QaE d^YY/)p`]99)Dx]  [>ʧ2"`CPVݥ;_@%֮_ЛAr;EձA/#J 7q*t##qC]m,!`H{ET{-o|BPx&"پߧICqufӳot[bx@ x {==UMnQ,a5ߜf}I!혤tjxHQJ`+imlq^Rr*1a:>y}jO2);2;Ӫx"BEc&U+z%l=ayR. 1Dy*)/.t;XT/*G[ޑ,:,'QY.V1[E{љ*:7n4~^u%4"ʽ+=Y􄂮t3<$lgi AM@*38b@@hia'=}<A…PhYTzMsHJe;-xއgt#x2n -4u/$Stm=~kg?Go?wc;EokEtmƮ.>_cb?= c{%p\0x0ur4uJ'\:5Deh׾/z#XUS0WGR;,&t(Zg3 G#}Ii~$Gc:S5֥~KGPjxAr-z'Br=rD$@uvH)ᖙ|jCqcRݏ>\!\O6ՃLU'^a͋I62=2imٳW/d#Oe lf3hB.nBfGoA}|#Bf \2?Z*眫 bcq?XB!QePk#ht~nB rn=j4`#ڧNU- y p][QnQ3KRjͩZ<.5n\ε SG}ϻ`śW>^y O)I.-[ɟg0ب<9R zJ~GF?loɘvjP3]c9i ~n02; >ê SSQ4P|M H#Y}"Ttb1.hʿ]K:~g5vq϶<.I7-H%?>ب"ǔ\>H}ŋ'YzXG2(ZmE 5=iyN33D^}J5&?B_R<]5lt)Xu1\f:)d>Qs5Xzj$f7)rz3`gg!j VIMcK,d456` (uJbV??;P͗/u2LR[v5w2-aٲ%̸Ɲlȟ2x`_aJ׺)<@#ݿWA)Y8z&#b;b!F(i֝UE u }1b(/)dI_5𞟫ƐWrƫ+Wd Ҡaq.h7Mѯ͕"PENOvԏJ0MuO&  aui.a#wXՑn]csԮ6\vݭO'u*67}S} VpTrc*Ug&0b~)ģQ ֊ϨQc>KKK/#?pV|6ӥW>3v4t.3;e36c{\D/q;gDDZ]ҿ |ٵgjSU MO-ByHF5S+BS\LJa+4zsv'IgQMgrL#&tm%&лAL;xہ=yx]%,l.0Juah]P gⲃ;xوEP9r;7 R)c\/9ݥs ̋dEl t&+eeGBfAT;'=Jc;,$R-ݙZYHx|F0c[%bޛ(}Az?`yv]Od.)Onpq]aM#O4FW-5 >ླ0\KLc 2QXZ3;DfJ=-1a@} '7RpXeW -#^߼ZȢ;{u5M&xq(ײSz`p?qurSgV)f>+O(ƽ@Cc5;{\ Z] M_`X =W7SlFYICj/W ?G2-Jt`vi`',c a2:O/֗a.VژÎ{s^ާ`및ԮdlT' )F'$"hv]CN M~{}?CBn"dy_{ѐE45)0T2i'V!)m(zohЩFޣ Ywz‡p-JޤVv8nl/z~:{Ez~o~Ҭa*byyIB:5@OdZ:B{;!Þ ?Ig07u%"5nƓ8rE%eCYdϧF-wnKe_oIXl+KK͗Kouͧ~ QlBzT}= -v6b%gΩEofTKCiȖ 9Y2Ngw:.bbqpeo$$o{WE$Sv 'Z+SEt׈7`.):rA.~H> Yƶ" U̓7 esA`8ʥ,/Qt#_&\Qy~"qwѩi/Wɵk1ޚjJZ{;X sihL XC9iZ3rT; נ/V[܇Iw%mA'~ߌcSdn Qʍqrbt,Z"Զ+ &WoWgyG)^|³ -g82,<L_XKFC:nerac"MB5TCpT}gw7xٝA=t)0US#7=8wOW=6 >D ЧҊ+QǪf_6_Gz;h{z6@^8u4dNHn< L%h l4gs l%CT:ӑdJcrW&XmZ2'# pD0p/EQu0˩YzAGy~iO&6sU (L{r-Q~!I`Jk1FU)SB]~;<{b ?I L$ܪTzZǰCUay{ m1T81IT{+~ Fp|Sj?bs1fi!ZiB5Ϡ<㵘< W_e0;x)P=qv17US$&"f-=f/̡0i9#nvym9wSNFt ~_,UBmzZŒ4oWGN∠%Jp']nH54 17[v`$fkQ}QSux;ji}{NVhGtQo+*3!kQ\|\X$Į׏N 2CK <ٿ> U(_3tSe51ɮ{8l;,xGF!Yfyn;w G'0sac2n+.fd 6I߁qewz}hMHzLNbƞ\jCP3 f}I17kCm0ƥBm\ڻ9J2NЄ. mW{BnZj`cnŊ_Q"&sfZ=趐Yg(on;EnJݟR)'N =PC~|OxGꃫ|(ROTpWUv93t:59湚PoOV T:/.Zj.2opwᝨo5N':SB>'Y ).nxm +Lbzx +bqaMs ;4jw**K|m֚IȊo1Pр9 ڒ +|vA1Яp).T/``Uwdl4q/+%<(-g T3'h9*P%O-YDDnAt)~)苓Xxh6 Ʃm&:ov0ar3~+%}+bOx8]ЍY-`dI-]"4[2s I?"T8dC+Z[:ǙŬE%)?:Ȏf*}ۈ=v!uk jn-+MO~dĆ}xj>8R7K@nH/0w/-I|b i\iC*:/ňtCoy^m C2HE@#5#%6CRVb[MdxkLQ͸$VY%CB7 ܪ(0HKP#S%6" h-X:XB6Qq5+zx|vbk!Nux0\Sk`7N/W faA+z4Ϗ 􈝼Řf&4ٸ4Ѹ3 ѝ"ĸ5K|bjSgM0/"CdIxS?6ꛏnNvaXs =$jشV:25Xn0Kf6o}KTIA@<->X =OGq٨QgW8}}Z ɧE32#PU@8xt|Rp]#vcCX`޴E fٿ U!ͬFs@8=70Bf䦚_$ߍ85FWpQȺ #Ⴂisj rPt'8dʇSr2LcM'}. KxczӝC< ;-dD#*}'X*푋@kS#ɷo Tradgd~gx#У_8n[Y!A37\/Auz~q>qw ڡ7nJ9Ά0s)oa7͂{[!뚧(};pݸrM8R|:il qѐ~Ib:sbDu !N& wUmɝP~d:h'62>5jfCb^ tu5lET:_, j7ìYnPe%gaeǬ٤W6h/ϣ7c+mfq;ݮ[6l KZ%A fKvL7:Z-&(6\ބnuҝ$:Oqe6NJgsKd%+Q-D!@ʇ5Q3눀%(+,|;Gf%krY)E,UKM7܃ =Ъz?q)wH7c(瑣3LvFҢІE} `Upq6y>L/#kvz eÙзMہuf\g"#{CwtfrA+-[^)}rWis6 f0 v$Gd'3w_\t%{ቆA˺w9:*T Z#5 '^N .2_{ o<5ZXnZYN(zȦd6n>-B}OvJ~#-[:h /poJks] Q;'Y19xSMވƢ-zɧU|޲1VlЫm00ɖo$Iq >O+V3^nW F&㊦ybE@\<"aڊ,Nu̫qQ;]I=؏m ̎z[C:/ 7zFL2rf#RaP`d0Ծ㰡iRMVsScX뚎c&z4i2 ,^t٬H˛q*4@j6ɱ{Hr[OZѢ\¨}1+.@| pB ,c5FGX[r0(6*F[OPGc,(>n,ijЗVEIp+ e/ eC\yJ=g'@ TAZleBB֝@m`I0,j4D"3:eW_ {6dɥ}8Yߠ%ްū4Vk_2zb7xKK6 k 'P @d=(]X>^{;:?PVѤfvBx +xGبpc#A<GZS|ǚM%#1Eu8u\ssoK=Kn&˝wg ʃ SӤgzQ=<Վ멋}T<,apU Si"oFtrg UT/3jfl rR:KgsSYT |0血D'?.o4s 0wYѽk!a;ũ\Q-8;]"GSj\K^] lliAz\je55) `bu {Cdcmdn "w\( v8Lr邡S@%&πI{ | M& igs^ H(Y&*|Γtm#zO6  M'Za907ț[#Wb*A ޙn,)DK 87\c~-+4 NTfYiiEq쬎U2~=EJ6]@&MT<} =c)w7wy[W.#%EJo!7w #|603 3dKmsגR}mi׆ ò¼uw.h8$0oVk wP/݌Y)yW"WWěC 7$3Ca6cӯ\k<o-&#D˺i߂ ²^R$j5Lci8_`2%O%"e|`"^ZXjv镉 a0c }0+Y#qŠDtMw~$zV!12au#+CД#du |a/ȏ"xU.?hD!c=мx %τ̯BS]\|[$lx>[.)Aɕk=0hXV4=PVvpǖY]ߑԑcH8YLu֝y'8rmv4m_=\jx? GAiA4 k稱pNR!e} zA1{ɚ'B6*s?`~ԦIgP.anAyH˭Y)Eiۇi_ 5|=Dk 8 #GnUDpTZ#GͺӲA8t[_A熉2x?oR츚0bIӚECKxjJ#FE-έWdRwMx/ksmZE霷PPvcw˜9K8C˳(k!K4҂1)~49[ ke %< GEOX,.k%'Vm?@ u$$|s6h UI1CIi`u!ӕ׍NZz@n&T}S_~}2=3/;dCy~xN-`ko6H*7 D|RM^49^OȊyz> p'+,lC<.LD2 H\ "}v]UBkbz©"OTS_ӥF=!J4{\T Vl&-TuFEOϘEpQpJGڑ#\FʣFi5֨.-E\I@CТI(iaz2Eȅ2rEAbqt#U[&V&þyY̼E4;\I"| ]v&$pdw1PwGe]nB- J\Ƹa \}hfpS;45`*A$tie&S'D8bߠJmRt9oiuʁdߛ Ey@=L7κk @J% dTO^Io`#hɌEԾ? Hd,FA>;O; k(DKwg eo_:b%p[f MNaoHeK2ԧf&2 4&\:Xkwši0}j0? H 2X*44 qtΌbՔߞV\:uK͢TW i׵"ppG=# {ڛ| Ʌް\št|".q ٕdS!lU"h:c[:g<oCgd{V?iSAeiatݺRJݭ,[Pie -RԵU!N.N>a8ٽY˓$nruϏ Uw$D[.~;jtQZ{.)=%aQr r8^Ѿ!xF~T[z"텀r~îʏ/>G&ѪșOp_"Pò9iSDjZb) Sh/m&V7Q} 4|x HڭBG1OVS93:<$aY,HGߍp(a-h\`4/W`v13JtjxT"O )"bPb,Ea5q@@Ж)Haܡ( iϫDm MYuaqA\Nݹ ő¸}9 85q⚪ *eVsEo[as}ije/d{B[!sW<Qrls)MﾨMynm:Jk]|WNvӶw3 LY]n:ĄtcvED &Œ j+rU8TJ)|;Oh4n7H(#fYjhuWgΑ2>싗t %܉_a"Z^ BeUzn)AY-gԋoX[ޯ0Vm`#k֩P~n@ׁKW@WZڸA,'?luo^2{gl@Oe:O/i[Ubr2Lq31o h,CfPPf\{B-R@&e:V%LImȞU/6L_Eg1 `!pjv .n|JC ᑺL(D$V Y,|j|`).ɨ<܉PeoL//( @ƸZ/(=rL6]-z'4{P|L+_Bwض:qWL6\FG% ?Ԧ 4Yf3]wޔ/3~HCB9gK Hmנ9Zh81@8]EvbQ.WيٱvFIW6q^J13 S_1"2:JfC2=Zē0p4RcҎI~LYvIP ˹|5-8'Ϣ넄-depo˜?^6~Y}/jdyuM2aJŗqPMW=335ñLgLBAg_eɁƉ._:0" 4I+<*9K-QTެ,[j@%{U-PzмVa`*vJ2dѲq*F .'& %hwO}hYM8x:kXb:-!E~ 4DY_C/X\g 0LՃwi!/s;zIVq/~Ků\ B]vMa񆉨\ kQQd Y(Mbjp)_섔)bl#$-I ߏӹ۾V8ICKlW/߬/H2@(wǡLq\SbM\7sVA6NJ:cq9 W%&T6' #{F[w#E"d(~fJKPkF8 e &%O#;5VQr@˿ݤSTeaL˓t9]=T: Ͽ}nXV[ \WShb~#\Iuj-}$X] Sr<@W]0VN[Q\6ص>.=.*&pZ6$JJԘ3 -v zS[^уMMJO<*4[Yk~;ĺv32 NBsCĮUD+ ' VnIU!J12ΝljN+¸´HiFR"0![ S`Pe}/+?6dY_|HSzL^{P7:<ϵ1ͤVœC3AOg,w-ɟ"t9G8i%T(LJQ B`f\sk~*}Y3Ч]1J_s&>?. sԱ9xLYH;CѢ#:/!7= -V]]B\ Bk)g^e}O0I&Bzwg`!Em]]O $`+>7 Fw܇ނq);10CǁZz؀~ M;'t "wQQ\SXT c ߙ`Ea(X|5SŽȷar?^Ñ[õAr@ i1JU|d.@ }zx3.'` 2 ;.ۯgbl8́{d玲0DlZ2sqa26͹+T#ʫ{@%S\s!0Na4;l48wޏD{{&; HjEI{O" 2#/l{ACY. n\H>rvVX& "*[#U9I,R24t[k8;'fsfd#@>q 7~<0ѪYЪd:ӣB3L ^nœwnwr־]7%r0Yiʰb9]~Wʳp`=,6w2jTzIʣr,6l=D;:8뻭[~67dP)<<D?*_;eI1Da;:7R +L6!|WݣXL:N;W")UZǣ{os|<7!]Ц@ Pn+^d 1JwqC8opvC9RHݣ 6էpsP4\$voi3οV 1N6YLO{ЩTB WBTy{e<@VQ8mm!iU,D:Z0?yT*_Z\Ax,|XLh$ Q|1Es@ '/YG3NijZ3gFR!M`ر]*"h?ok(^6OWAǔS/,)3fr~pQE (?-1+X&}LD4F(\-f@cbh 9z}L1@.gW9x{Pqۊ 5FLCFI_ط-Hs(gnYK}&rsX肎^} xÕNKKE=;( 2EhX *-.!\(; jD) tɴ ?[ep 8xxCݣM+S|0*K`>)e]3{ xGpcR[+jQƓf`jYBoKbѣBR``ʇ1P Y>FC; 빐MEKO"ѯqv2c@2!}0k Lq<QJ< 3sN1ݫ~fnp,K?RPh9:#X+>Me xankDGY$NkƩmx\ȱC/BŗqBq;)Le\⹀.R2~%]^^nt4'ĚJhzM ;$iV[٤9D]r}^{Cpzk\+^Aı#FP_Ўk|}ȑ=/XD孧Skٰ+r(qoUN;}mwihish" ϗ@kh1he?':IVpqۓ|RLY''Ÿ i'nn){UiG#btens_kt??Zƺ-YRWw}ع'9"CS Jy<^OfiFY=/SJ0a5r; զʘO76 S\&!3~d Btx:Xx}7q҈^C$} Y.^ԕPcHVRJ8B1 FЦ7όґ0A폽A& YQ=4" qq+4i[B =J&u ' CV2Yf~0,~VYF@do/)i͌ y$|CE.kU+ղ3Yyfi=Qsw[^˂/ur  s'Z}΄!4쀗gݠ R܍9ຒi^RKY.6Add)m㿉.qXկ|eԿĔ['v{%#85P2T Rm)xI w*wa-X8J\DIK)įOMB?# GIJ~1~1Xf,>mM^JA^aAIg/Y`hpd*nuŅ*z ^0nF: MrT/^{p%uZ ATMшstګ5Ec7 񝈪@o&}@j/zbI8!שS[j4  I7y;S9\_zy?V5_$w]#%2%X–Ż"٫-S_52vG#WZ2Nܖ:a GeBA':Z[v I?[5uo; K*9ޅ650yK)ηᘒ.F2(:0wpo:C-O%LkR}n:Za0& UD RO}Psum%܎:L2xq8dpSi`N^jd{EӚ1\i{FOkKY\i^AG/ ID\ɘ׼<3OgoW1E(qIN5+ zڟ=&bMzsVoH:S_.zAk{>=zIE{Vyj} ԡ~Q6شMI},i\KWݙo䯃>k`䳬ϕQȯ?b_{&QT  dgȓ]GyOl99%k+:%[3P҅C+DY$$2{sߪH83UA״ebԧ3<(?H׮N835V_J?*8 %fsMK@/kE7ƺa9Hv(8!/P%og[:_:d0.X4U ?9PAZo-;cM'K,Ʉȵ(ړ-D]ar*+Dp<_v,!4&]}b~L:RQ zu,İFN nU,un<03۴=?]>^ZFJ&!He{s?Yǥs_'4\Bʛail'WYڨ>-m.XAFeuAx] SKd U}Urw5_BjfsPdaaza"ck/@6ƛ)L.|EO%)lޓ`5G~uLۃ Oe 5t9?Wg\b8G5GA=Mɱxa[_MjɠST9규@{6`6XޞU"mUϜQ,ˏ ϲJEG;X&csoF߸9X\_MX9]wZ{L 6 t`׌mTpȭ .U ;lOPt=8u-q`@slFTX/ZJ`%ivY1"}qC]4(];EB!%FQh"UW{;b]-.PZ4⟥չ |~5xwb nO3QpgҎև|)ZSkIpsNj=7Bvk Zc$` GX=#Y I|jjtcq[N[sR[6tu"*I_8?? Cf[7HpB~3{$l9G,jԺj CTɕ%&Ƙ yyZ` ЃD,A:^,FB%ϓ旍X87zEJ/$*3Dut`ړք  pat#&4:%sGRcݞNR/x@?$`cQWa6/!;߉^!)ҟ xLg<[`Z_,?]'i'?S"X`"r@N%_a+-Sa ئs 4A+.tkCP{7"_0 ]36WRG\P7/IfН,Θ2 :9|m? PmmŜzq[ǼKgC>}2X \)̒Ӄ_bjs2ӧZ(UӼ ?.vv)^@JofJ7TxUq1jOjZn_a(`)@:ZS')!dGۺKRwAuaU!J5')Oc!%ws%QӭxwJ}Ct׾%(/[I8dzuG0k*^'/?.4|#Gs= I5%z c>b`S;ˮS*YHwg\yCMvZBYW!λڰ+T*GNNQ gB}Ke[0XѢ%./QrW{#ڏh׮A ΨfzS# NA'*$FԌb4SBvQUh_}:-׫0@SWk>'uV[/}pa(.?"Ez{S$RD_oAT׏3mw.;ݯW{|:,؀,&]YI$9QRX}boe[ànM2JLeb endstream endobj 1709 0 obj << /Length1 2426 /Length2 26282 /Length3 0 /Length 27282 /Filter /FlateDecode >> stream xڬteݺ&vvTͪضc6b';6+Q?wsocǜ|YkQ(( :330v@k #ӧ Blag+t69Zdf&3'7; SQM h``cg-,>q=&Fv& hhmp3uv:Lv.Ffgs0v. 'ʈowPpr6qbZ[qDS2q2qt51f[\˸a;925IJp7~s?"118oldo kicglajYgΎ@['SGϭߎ;89-Ml@OtW;_O'_c33d,>+haOBPBM?Q=ϩ#?~zgfS4ilbtPs$fO 5k&ۙ_%[O p)1kk92Qhm,=qoUP7t) ښ}VB '1 wc g#s)rU[cǿBW+LLrY,#+ۿR#˟Ìb"Rʴf_mJ3s@G}uL=-L=3lٹ,>ig`7̟Kec͘ 3O %oەk9/)~x\25Z)Ǽb1AxLg?ҭSYzf)1IíCq|N}+` ȑ`b"ے@HoM=iW8Um$H59?wC6޹RyR<_﯉>n㍝>Q1f%&S>t.]oP 1;'bȗ*9@\V- ꮢ0zcb+G^]i3qGN)E4vF<30vom˩meorćPg֙!ا< =?$ϸE{q-#ifz k(dйloF6DԦereE@O}e&$7(ʦL&1¼mzV1"R-uH^?Ĥ-*܎:/@ήh©奺M>4[Sp^R+&؅^.Se熳XĎyPBsOL@ liaikxјYzK]Dڿ7Bl[čA/L~Oc3z09hԫ؆B`w"yٛ@Wӑµ]-t B,}F9reIr9sM!L6Lpmq?y ]a#ҡS^ң{bQ')(P)U`jgƣ.(&n]i\۾Bb@F(EG荃1M22fIRqՏ%=I@[/,=~Bt(? )McMͧmb\{NuH@ծ m~^ޮH3PrrDi4lA<=SM#J=hѯ8ޑ(Т5 țԬm/ GږʣxǖϦ^Tst7^҄ʰ )ܭD3:Q FCb2RAgNe#!m4}wt!jDM^p`1T*ਤ9(LYoI-d<&쮁NdW7Q|Ӣ4^95b\q'՚0,wd4i^1p=W+1Ӯ(ӉEa1߇oX@ {~oܟM9qvб<uɞ+cBMW(`QLC 8 5$Y>O;O.Mfuڳqtj5d[  Z ~WpPF?+M&vptՂ)b:ܧJq5؄F1#׊!&!)}m}fˤ߃b 9)Lj>l6C#J{rp"tMs'(dy*µcȣdmbXAMqT}b(`lq cד6WHG,BޏLBxaW ӫP4XIB_{F†z:fvJW䴊=ԃT0{L%PƏݯ}QV_Lr@{t1T# Bd& .N.) c)`]cHrm,4WAi^#KJy\6 O $7U^|I )}(b.S{= c񽳿R4Up#_BT_s%1܅.@+'[aXՍgD-)r^NͪE"6b6.ܸZuVlS#6H's1LM`Gl!j'!D_?sB$.ygmWW·\)|0m@ 6sF#g7Wo ̒J<)Ho TxڴNHm=Y^78 4?FCm1'Zx `{ [3Sj)OG:JS>%.ph!,4㥞`~f8ؿ=9\Ko<*B h vݬc y"]<`&CAa"?S[mJʴ]07)B =p`k\DR!~V%0ΎQ^3J _Z ТeGeH5 k!6R]06ͣUF uf=CUNA|C~YPrjQX[hr$ɱəLFKHOPexVmw]OUW,QwT܈TJKEXYEk$ NL`{m|VƸۡ0*ie6p7)%$UOԣ*Kq@NXyQh.n+J%Gw0Llf9SEWQsmǒD"cP%mIX_ufEe.=ٚYۨ[pxXCZfn@X0 _UlR{:$iXM'W=&~=W!zO7J\^}^囀 E7~"JA4%V)ق~N7Wʃݦqgè-2ef_9ĭTxCr?hYeM/q!l_o&d^=Yqc$L(=q'+e06"rbE$&ywҪ:"nHaQ|'_ '[scc6oN10ȁ'Tv " uTbC/MXޥ)]#'ew#i_?&A+ i#|:wlZ4 AxŵيׇLU?,S)7KQn,ꮆy[N>l#2cEǓn$Pب$CuRk|ןG`.U8csl:0pJ6V#f;'JYC `D`.7ׄ_&| +;y{)f;,Yp26!<+'՗l;gGKתf61C T^,NǝZut*Z7r0,KIw!R..= "UxIt&U .Q~62 PBn2aL'Ia1+zٜ<;nbJ\= 6^`g,GV4MbC[".O#?DHwC{o!O/3'azyDnl.·A-4d,;}J0崞*{fXc!h^h_erZ($cbXG>QLԡJ_xhYrE/Zb+ DDNu. .Mp|`iQtpZEM !eo3o*Zx*fIWše9?~)[ K^v"ȶɄWݏ NnxHsn> sZ>kqS,@^nZdԃKv= WL?|okĢ=j= XL ߱ W+ [s /X1?ԫ0ъĞSl'P,FL\VPJ w7EP. \HzWе.$F+K-",45oq5lEpO k>JP6׍{;6rrgISJ;zsi}p誤jԲ>eOYq<ʄ&nii`FV&5I7K q{Gti,s>(1t\8oWlN#u`n! xdUk[0dMFR ~]@?)F`͢Jbzˊh 53gK!7l!AG~v|eټ /JXq~$&_X6jN̰sEt3Yh_QIdR8lwu~\)Gd u zgL Ȅsx؉:0G) a Tkz9׫q`2lCۻ,x7X~LGb$| |Sv­9*]MjLI54OrY`FOf`ag\1p,RaF~5']nϟ99wJlyBW ͰK:`7N`rN׭FcXe-cQdtX'l@E5RN W</0_dj[+,4$!vS;JSH9.~P;32`ʴf>'>\0,4QÔLrA RՎ hOKJHIlƝS"+|яhH#aҟiϜw.p7yq(=gpi _/_ yTC7m.$VvX\aǦR+')S!v&P럻yC#}Lڔq>OVu>blhh$ Xa\!]CN!ӮIi䱐(L`_VMrT5-FF㬿1;b;o~[@Jh3=8O5ŪO) @{l)h':Θ}8nR$v4WeX!G &"Y\([#]U.[MﺺW9DgD$Gd4V/{8rwp51oH ៴t,b[EAS2n=O&ҭSZ5>6ʖ C ZӼז"{YN mbO)XCcrjiWNoao' WDM#WDsszAW[YFKl>-_1@B'FWqMb1K}]^(+¨21%)h6CuF4M>yl0FΠHku]N oy,cdm؆$gf×%2 SfY= w(l4okBnަS|!aEh]({`mV"YO}4f^;iSNsO`}]:Z:yrc/S;Y }4k㒙o6XF H_R93qI}lNᔠsVbٶyf;DD3cg >v~KU \ozL0Qiժ~KN "nz}h!&eAP<-ĂЋH{w'Y3j]gnN.(b(6 ԭ`C{|-c%K`;y:ZM"`f'J9k:ݿ 뢸=a(Qr&Y vJR ѳ{ iIfCS!H*ޕļo;G{1 Wn^UfP{.bRΆUJ5^nZ5)nVo[W14h ,SbY?Fy @ 5eS i{S۵)uIbKUȩHRit| NKߢK= ߶Pn ]|cJ֙HO+@ \i*9Ќ=Թ +ȠX)]B3Q2ဳޟ7ce<5A)\i7Dw֓=DJw9iѱxJ5튌wX_}n\xs`'{<ҐV#1w6ĎǿZ€cZx2Ej&6;q5lss8fe>5EEbP0ZjcP6"1 Jd w!<ȢӍӡV O>%eHjϺG5j-Q,$[(4d\jZO=uS܈EWqB_@{yrHyiaȯ-R={ˀ'.lL*}3$B}4O ֏$^֪NBԆ:M+&)/]ׄD+S)sm9Pʷ߂ǫ.90(ԗfDT"H9{,W{jɈvBIMY],l1Ts: WF4PP!뢈G{/]zR Pᨘ'n &\oB6Jlv,E1 Q8~ j}C+qO;k3VK]P$.J|;#N?6o =0' EAvt`6Cu,8g\t\᪒7xifN&F_u'i;v[a: Մ.K%5dqo, OHWo#t37r2vmFode/~MVV^M"fo!GNPpx$KW. ?܈Rx) :E̟ƟWWq"a.aDJW_"SS$90ޤ -v.e\ҝȃfTW~ӯ,Z5}KET1'eЄM}0A3Bìū"p@D%4S Z$9cEW/F_d Vn %)æ:5ǩIvozr6sx~OI jA6seS,`ʡИ% T*si3An:0 >xTf'Nv(<=jlƘQVWt5af%$1`ZFwG8P x9lrY k̈HLJ:xY5kJL=~Z/oTìLV͘8؁$Ƃv J]vE7+GxXNJQ%fXn%m|dn$9ɬl #bUNJRaF"i2/Zm0oin1Ltb3Y^O8J,ZC2mW+M S.R'0^KcdDS+/+?ԗʵfdmC|᡼dCtɉ4(Wo[٘ 6U WO rwr~_N(`K ƙ9';to N؃Ǿkk/m(rGR[ފT#4'fΉ͎siť *88"[DY /bO-HIOs0I5׫Yjnj솝,ĥo2Sҳ`{ FOq-LMTd h@"jl/Pm8N`y"a9 ߌQ6+(a&Or\fʈNGT\S䢤cbӓ)QE>$-Cg.znC;@|_롕 撍/GÈ|/qRڈnynʶub!6)XvtI_b[t\?~Sp? -A Ïc~I+]67:)zEdTbMLdO+'ǁSwm`۽-Si%z>jgdZ_6867+@.,a>aؔ'ZBBm-pha*jː=$ڬZU(cVj(5d(JqXΫ܍ǰS;9gK0C9h|Gö#>%]vF F?ֶkS> )%YԵUa,xwLYُXȻ%Q:wU{BkSK U|T4vJHOj}ߞl_nW{4<ԡk wSw{<{}d] e `#:Zf?(? wm{fCl}n_0POm{##B>r!a JfT4'9;ÂnDwJwy i*FA62PPPv9)[ ]‰ݶ n `J_8%װB,cNJS]_12z'{ey^>Ѝ%@;nrsEVeSQ0 l eDi w=K87$Z![e-­ž{(z;c&%œAkJמ$*Zg,_8=EF5^[q3\g?oa /Vty0'&yrfBVK Rp&!t5Q4;ᗯPQ-l^gZCGַʑ FfR+Mk"o㑵2n Ds@ )o;|)U25#Ld'ㅗ Ů'_$vnjJ^3@1wv LhѶX|#h2S׺ǔN!9V2N%UR4w.**}[D@hao8J1PhխJs2s'X NmMQb%cy\JPJ]~ {&tY*N{+D x%>UM(vSoxn|" &n2FQ|q={{UJG _֒ |]#KH|z (x?a"d*8w_}} QL+p3ӛ1 /Qm Zh?j. d׾ VgnrCڼP^*ǣ!,Tϣ b>8@W6 ?97VurN'wKfEn˸?ejjD :~nx3bۮw3t+sЃ>b/fOH$< .Q"WZ QQYcwAʘaT|2~+;be5P qA֋CPgjuAIVD7/N'}k,Ot3'dc&iXͳH-SὬ~V/U+dbyaV&g}&vbRr\h"K WoEm6Bφ?,Gϸgat) ]2 E=CV=VoȄ?^$Dzlsu ÚS7@kuMu^e"{؅<ńYxv?~kvEp,k@ꀃ?[te5^4zK}PՆ˕+y㫤d{k6I+>Bv_9sI281&R;O"5H$Vޤ6^cd)={!z6y.Cq{ Y+ 8k p~y2^1jhnjJ- Dꂈdgd9vldɣC;V0 9<zJ՞N ux 9r:YFO4j5&>Ԉ0iɌw{~)xe ~˿\ڠ2OTh%2Tns{EIltN hb|]Ϥ2ޟM˧9l^)dH+y[|?WvkzxƮBIl(ޖ 4Ԣa *Q.G WqM`.qUSܦGÕA,Ɲm!^;k_6'˽jM:9'Pv`9{]Ct2iWKeT*`X `ׇGk]ZєV6BC'ռr 9OC:Ӂ|-e[ Ѐ$_Cgeszp3i-6`IɮEE i4"7Apv}c×oDE9\D6r&1 m[xRp@{b>1]䷅z\lk(k^5#w6ѯ+q2lZ Vеsc c|G:e۞2hO3Hm/1l4i 3[&fDu6=Ȱ;Cg+XuC=xjZ⏋ljPrQiO1#i)&`.L*{wv2!:e}f^E17af:PR\jRbz4N** yT&rS[Yw!!n%?>MT7yCa 1V洢=\ OKt0Dɕ%.]C(B yGÂ0BYn\d(Pէ l.GV-+[%`'}j|hх={"=C?bjh#}6%cςM,.8`&fa/^Lu$e%1Pa:̴` I)C s*n>u 7 GoQ>,eVD{;9(VK6Qn'{$y%"͖2],IM~W#<7"y+ /7+q 5!UdFnaT!ЭbHɪ_!2HbDG҈\8N-f/')ֺ̺'?`7=2SJ.wj '@ly2h ?M@hPO|bEY^]E88ttbWn5θU %/_|:e]-~rl>Fyy&l4\ Qi{4m% 9*/q!\!lּ̲I)2zLmgY@evF|70!CxQF']ȨanSVeSJcz϶qwU: @4~鬖{D\%k0]A[U׃. W º sW3f,0]-t[{h2| AϫH97|3%CrROpؑ}Y!:o6`|!MS )]%\ub콝2qd,mLIz9K1?gpх+l֯zp!1GH uեקQ_&sSP2qy4d{aTuπٝtIz}pj.2,-㈉;7F6`3z_Pnvxdš07CǸ([@$ۮphxDɫЕ,@އ_YxFg 05M)U;7:&ӝdC; fL(M (h &*ֺ\&ŷ-甥 6 #;6Z>UGl:M_k[1aB4mjjlE-WŽ 1&(A7IsC:}lB*`B zbW'5KӍ&v^%l%Yft" e!D`7j ^xd $\JdL-$472tă`c"##ߕ_TJPмd 6b,oP_{8- gV9O<1FF! LP_S\/;ߏLIIbFm"AvAR{C†ՠ pv1,0?gg$엫RM0n]zޢgc%;y =mTy!U!#.Pn@Ƽ4^]FLk $.x.ERrwv>.hّ+ЫUA4k<{/d>Bzᙎby=v僭≣TΧk"K]T0 =EvE+$/{FQϓK}zfd3R%ޞb]ڌe |3Y|PMMeM֋p~sλ;ҽ2(Us Yצ.XK_DMo/OaF!86y=6=FPW{n_Y)mڌqZF]TTo`\zB)mٙIQAB=cZ}9' y;v t8qG N~R F}EݗgG%6B d%5P񸗯>^[dvEU&UIƆ~9h r/Å ;(7pTA-cs 3De챕Kom)75pcg2%ܷzԗ9KH80<ݏ.[?%W6pՖԠRkAo1m*'጖:2{ r ,lg(d3awYa`VKtR23(r풍4"3qEArS:wd~ {ZǺ3{?] 6Y.S UG┨Ъ$M<1M&k@jjQ}7^uKJ"I83榴Bgq[e=u|ʑsk {P=дӣ)=i`Z$q?!Pȃp|6 %6iry~8j2DU'pj tbjH-i4@K/dM>dm3[ kpwjouX3DIw]90H{PQgyTs+fh9UeF725>T廒ZPA;}4kk1 TiyEn)j/ :ws f6fy@i8ʠGŸV]Y>``RmcQOv&.6T$ggkahLKi}4EsY7\(Q$ ƘZㇱDG'}V}ưt). ABgfc&\!> Eŧoԗٵ ]]2mAYp}L n&D1ڭxMNO{vךHv6]lF{l8‹>`xQ|rj>1BmC. Q}Q] ?AѪA$1(9?3f,ӖlX ^a$/) N]P̌fO4W.leaB'*ZaDDg6TYQ_ӤEqs|NҗF%Iy:U]5Bek>#$ ߘfvzs)yeݞEC E_ Ro=SG>YrS6yu4u}D8UveKNyHRmփۂ8‹W >tOK9 ǁ %Lvhh>S0k8jÌ#zHmTa#T3th2h(wݔ H)@Jq}7aU-<'LltM;1h-/bsރyr {o.7eӐwgxuNUb *k'fG]_َϻGs;+((`t!ۚV>Uy& 06I&#/bHJ9槲]&>=[%Hg mRɱLtW/,WVļNOԏyZ ͔8ZlRgphb"$Z5-Upc+TՍ):qO/c1pL_~ ”Zxk**Hֈ}8g>H`䣧gy0+/o$wSavN; B/SxЋJgDR$,Yd0ޘ_GyWyc3N4)~E墬|tޖhx^"K51e[X4YS}j=t9lbԘmSӃ3ɱ<;D5zCqF/oZp˺TC4&9Za˓Ay&7,_%AڼA3${'K]UY I>4Y`-Sė:Ʊ؃XiClu"n3}P+3UW#|ϥP1CBWMVו/ǵsMac4wzeva^5mt`rXcUx Y+[nTe{ﹷj#燘84ط :GTH:t4?`&kCᣩpI(DJxZ_h0aK!#f)$,y"Kᅦt'N2 }VSȈكfCҦMz!K6*#s ;@k" V_ |(G3E eLF08R-&>]Yi^'6kE{}T?Gb:Ƭjq/<3 ywXl9Na(Mgy vǮ0 |t0 HʸWk6]pxWj2{~U9~%\J>V PQ:-E}ߞ K4MBv4xl~b&gI:(vyMw8t̯I'8eGNn~S`F)t" ~ q|ӌЃ:~KHFI1ؾiOLqQNJ?Yx4EyT}.o#liyэ;D?+($xܐHoM#1t4L!&%ǿ1]+H*Efw`¬0-G]FWOeΪ;t)L_ )UtERBX'Y S&Ljȴ:X;g /<5QltpoGBF7ɋs<7x=rauDrF[Y ~AϚbA sLŘፌbGVۥ o޼CJ5J擂H2u"e"OD ȤLu ʍ @$l&,x̘|i|h(R08DKqHCQH#egpEJde ^VM$.h \ScHS@KKihnU++XBs'k( pTJ<yhcÝDeK<\tinq?DP]wYdVFaVpX)lmWq𰔆TJt`bˊLɄA2z6{1/iK< qm-5FEA_JAi-v֚A碪,>JA3/($-( !Cs"NsrXz]=^Y][rammD {ԎU(6p%,: #BduWb 2$UׄdUIAS 5%NQ NYBE䘅o wSG6w͋ːSIkwBړ(7{^CPvvQ9_v-h;ΦmP?R$Ega}yW\PcSNί'֚4LS{gɛ+ hV/=GU$=E oJ9zYOEk ,0CD7 䂢qb +Aׁ4Ї;ᄐZUutWP`@_KnV%[B;e'Blo 5!*7Y/,αApRaHEO+]HҟDK¡o.DBV[mwCJvD[nyb,w'\Wi%3|=x'Iͦ)tf|pYkPǔ3oH M] Oz{?FLa`~?dfvJ˃ ا1ɏwXHu8U!-3w8sT7::"-혱%aVr5nQl;]qIY+Mn/A]ja:fMh4lشmڮV]BYT+`LDHh#Y"Y>چx/6띭YʝpjCX{'hi1iM\:S-?fҋ\D)iF?;Qvt@4v)z>(͢(0GiB %M{ iZv_i5}v mɩ$l1T $*-K9FXo^~Xf~oI&bT"}-,;T:y^unϯ:}] wvԙo`)HSdZ. XHƻN ;ǜF1l" y})%F92[Ru6/mU(H'5g]u .3~FQ%XZW uRQO%q#|0EdWPFv}۰^ Rs7N ַ܈{I4֩mŐjZ SQ|yEh 96nse]dzLзwc0R ԩ$P&(>Kmu(òVwˎcV~Y,{V]ڂ` to=+hЪ-&֮)>X8 'ۙMrpg{uqNRJ@P恽Vѭty?Ej\\{dى^7lt/-lK%;x5㩘^; }0JAwu1+a&-zfN?ZT6zr>#`)| S4*cNj76mCnL?(3Hr(6 ӽ؛\Z1Mf#A "O)7QS!n'oC۩TF![ o"91T t+)e5@V1zۇG(3]QaO9 Pc{R3$5&E}yN IT34hXTxIƁ4޿Oذ*ډfN& ݝ_'-1ᒂk 1fn$//jJas\̝i\$"vo c"'˝NT}+r)?:ySURI՞k6 j2"sXz3e싨/5PݧphsF`{D 7RĽL `S+__*ub^|SUl.<ؕ"AS .O3G$e` %vk#ۈ̜XTǫa5xu & l"zr{T!شv0/wjX/ >D +>_V:wܢ]f#DRɤ:WD4n:V/r& [zlOZ"g)aR(NJ:N/.e:{A2Z,X^5 ֫R#Hq3(~^0(Ȱ jC;k)&f|:(L'!6!Nw|t_3(((ܫ{2HDpk=`ocv#-&6"Dow^pI|YҢ[cީgC{ >e3Q `fj̈)\{vNӲf{ZM֭ [ﭒ\YVi+kN5"p6pZhj&K eCRW\R7 a)qwl%1G= ,|n_RI],mx75q*F;= ri2?˂$l߽M3x%27/?#M[e;U[mz!zߨJAurOV6`/3h{mZ0Crچ> stream xlctnݲ-v$+m۶msŶm۶mضr߽9~綯?ѪW5Yc۹x:221pTT t0N.v".\uS C wt4pPSP14tػY[xܜ]\Ylj p0Yژ4%rqS;S'C1@ٔ`f03&gs3ur-@VTEPL^N *L" 03HřhScUGp$`bi025ovfM\ (`bjO)B kobdNX;7"$] !Cct05Qto'WM7&Yn'쪉eWUW&[N&c[?XW5tqh3E=i8l Ffv3kdjo71̟U{c` ?ѢJpYUUF൙[mda k|w8#޽]ɇb8ShpUJsłp05l[c9ۈ{? T,YReH;KOv┺(ZLk-B oݒb[V8i4\[ kk*,0vK`ibclU~o >\3Co <ӏ.BDoqAf/IWFm?TvF͹ͪ;F̒H- ysnH8n xOrnNPWX'oaWEG&-8~8c )Cl]ځm؝BK%U0=1Y' ,ݚκDpv8 [2g6y &\5_Ү#K\SWFxWV?%26K4<+UxԋK{!O,>8c:ON B_*X3Xe+(U Wp̓qbbۿ޺[e0{qorU-h/a3Cd=[ ˎ gӏֽ%O`g`'[wIu}'SH*LvΖ{p6_6R0#"Y=Ҩ|7ۂa~- ^ EAiZrm᝸)T'/fȻQe=x+ ScV=ُK&`|@2eJq*N8i]dop#P-r$;iBJ`U1d?wԯ?3BH4b4;U2ˬL9'l\G '$j+QB`7yX,ٱ{񰁔ݮǒدhi-T#p8+]rr|:ՕhbP? _K`?WB*N:m5 g ,3Q~=C۸48Hv5 e!\^R*"opê(bУ}im|2~q)]'n2E9[ \ >H!q<:0ӱ$N+(`yk&i6Auh>sӥݽ)!_[Z쪙uD&t{m26@"6ylLprHQ4Ñ.LA3-_SxnỎN5@"%?V @ h[Nܯ0镴"9[ve6hn3܂%$01}ݛf*kq? ~Rk7"G2t?K DfU{  Lqoq-M?[ 8ّx>9py0h|qؖl}"ǗZv?'Ap^-]upޫ|k ߝ+h%m[,$76[?ztk->B6Vā>Lٙ(701Csnx84SKv ulW0;=J@lXx&hۣ+`郵1qzoGI]/R}Y ߪb=D83umf3ʹkMNT0L7D,E_bF[%X~ 1\ɇ xR+ Kin~~ieG5݉Wr"EŠ{.N7l\b|`zGr^1fD?%gmIq_dF J==υHBQ|l}gVxc~k(őx[jmUU]s݊^JK z]\vRA&+Aѐ-9lX{ /}TVko tTO y{&z3ڜ~h3z>(s5BH;K44HRμ.V׎pH4'4~xisLl |V M8jM`~NK*S 1L-Sy_A{ƹ-O Ix>crZeQ~M7ޞHO*\WO<Y3SS/uV5T;9NB)04JZ\xP>W".`8Ci'4.5+_e"M:6RUtÚ\}Z =W8iϹ"5J6䱝֩9;U%͈w qyya޴H't!r"bm*hb}Y$&p-֣83F8B{G[$ wA?aMmJiK1dQGTRrPźbܿqIiPRbUpS1KI3i/|=#{bll>GXi+BD $%3)Pu x*r,avlɭ*Eղ|wxn7"޳r U`ܽ_?]-Qcfo O ~I@|R}|)}O'^*9z "}icE!)l&5ˬ.x↳(v-oXgO͔5]r4O~}TƿсƪZ1q67EH^dKI:`C f('R4#ΊJ/oשֹiyhfcd>:\eywo&*u7قVNȠhNiI߿K\4)Gn-mBh*$azЇ s EZ<|}gg"Xa9:'C~z{X?X ̕y.4X_<$/ #e©Y܆RYL$!{;/sG;;|w#â #W&+/HYJ.'ou0UD(Ku(0WbQ1Wߠ a 7nV[ vv\6,M%*(*xe$ < %,զ.H$p}Ĕ9Hj)n[Dvfu>o7SB>`z`ѯEVp5m_Kʣ¶(Aj2F(H~L 7g-VQy}宱-mOlͪtxwB@@Zk%Iž<= 1*ߞ68xU&ϹIKlP1Te#oS/ HӀ{nI $_둠ufM mG~}*I6/*"4mdzn"Чu&70ĞKJ}-R !{{f]&woXM+u__+TQ?I8,qCˑ.]KJ؂=wlS*"+;JYEb@:5y'2zbI]GUo;p,VOzׯ#Fi뎋q@_Lx]iAOkvDi,+lOYj.%mrYej0WdMvQwr7"%U4%R{;X'G#z0VDTzJֶZDCgV퇥{mmy髡L#7iRHD;Ξl*e0`}f^,\o㿿c_DA5®;9R~("ı6w8;G h_yٝ#2 R%b{7^mѸw[(4ۚĕU%|acV#>{(z){w9(ަjQc_̚aPgW%ށ{\A;8'squ:Ejw gr`D] dN.Bgy6:͹ iZV ~9d<Ҏ\"&Ec{,$v38G"J UuE  !d1nXF:2\khL\ nb"E@->|,À/_s$Ϣ3>"[5aG 0]JT7 墪* XvQ]%Y> /mv)ε bH).:R@aq,٥#fkKo~-5CuT6*[a@,I-jH呒Hhl>=F!z}044XJӪ?5mR)3%b­ϓ6&#ثVc6& \LM 6qAþXU$rtÅ5z",'8/kˮ@*][7?h?۷z֏6s{]6´O=H1NZWTaW3^qP%HȲt#-3dY̠s qAuq C~,6qjR2hL1+?*%sWҠΖ@t_M. tliexNh1]!(߷<|uu"H ,4>snOM!n4RD Gn"&{ߛXeOvr3N˲ŅjF֜'sDŽVP<;V"?o -IҒ1QV$' ;E5XYP 6``#)^^ sChՏ*dhm$|x#T/@҇-\/UHxO[NB5-rN.%NzJ}A-r3׌jIͿg̃J|;*:A v'ÊiP^OXJwnluDyScc[& b3)1̯1,lBa/޹Cp1L?̛H|d\{rt< N0{,n^I z-. h=TDKX p UxekQNeS`: qC "Pkb\l+g{xH<\Ue'L֋Y!g>ԅ^HKH g}-mh֡Oj`a^[VbŇKWIOvy)|L ;/PhҖT^R2Z7'~Xrz7 %_PJQ$~aWvaTheQȶO$tT. )-HrTEI(&LOJ/,,xګ';=C I4S%)C}{YVX'%?ygQkkQBFcϰV0z#P.|?eоd0'`10NMCL߻̳YO(7ޝ$:gDr<|bWZrkwud<;!JMf!7GP!o&2pHK" f$J|W+贏nD~n$Q8q*TksGl\l}k,( M) yl[CX(cRs{PSWy4˙W[YSu~FS%e9NOu"ZAa#"sLIc:m7`P=0O/~%SXLErIt=!I8&*O6G>H6ф:~DN:tl MW).N듼)/ dE2M:r‘t^E CofXHu<` ;ŘA۹7Q"w6obdR^[q%#Xthe qro5ЍD΂1CRu6R$O-HBY+#ٵYrpOzR(5ӈo3M=+l٘官6yiD-Q&Z2qY9 {`JZ3޶gNN_X VhewtW[>Zun'z&qXK7N˼WpE!Wc?kfu-4S? Z=SК;9Hb}';S5,=ݹ<8Cwd򁐄hrxDn-2~6N| 5̫|k'ؿi%SG}s6-/)b-&8U0)肢M6녣.ߝXnḺa3[2E@tơfwLBDJe76d;C◔tﯦi=fY$&tP^82(7-ljNP H#u7g>{AN [ 5*67?y>!}6s#CDް7  `BWB*y͚._SBMuVz9A3g/TFX/O,h|RI=[ z:k❵# -żRfG{բof>UO˱k_q^&Kyp(zsJrH(338AVDׁCSGO2!&6jrZz!.=n^ וmy\X\ֿ_F_q߆nRgy 喖R.%lF#duBA%yg9ė%@EX,NQW'כ3xgZ_niʥ03"FIXMX PPFsmb]^p|8=k% ABzswB<+]H^Hxdޟh:}a0r{PH?[u$D'(,riBL}لwsׂhw/߲`Mwz|8&w e!A_" VDSPZԵ wdjwucUBfBi#h`)-既G*Vl-Y=.tZd0crXHS AM_&8V<<D7Y$yG!peV&/ >Zt?hREVm=v;XߚmTC(s$Ns 7s8*?.գH46EQ?;&Zq<#أnW&8+`&;G| ̩V8P2F;n=-0ďaGJ.72&[`[RnKUp4:DvԌjyX\| jgTω'%T ,ܗqfFJZzmM W'a:@wSf@ i|9H|oT}^h5wHADēdm'g4X|#4=:si8Z^i=u)eIr%N ٢fa63BMV4gPRXMur0 r='4F0\lAR8Q0v0q9qn>9ct!hA˅Tʶ%#QItAuXx_gKfryq:%reLWnX_垷juQ!1 )Vz0/Ub ֆb0rU%)M,!qUr'ghh?ݧZD+,?8L"D7eq+'S?9 0 C]Q"{ss LnxW=>FLf%շ V #/)}RrR^ћ-r TQ,)1"c*kBtIo<ȁEKjtäx5s%ˈAnjX]!xR%ۭfmZ99<:S[EA DhVQnnwnslK4V4Ut=byQyme6RRtoz|M7?MHϮ=b+xdF0o!):v?eSi Fb*"`# zS鉘ý(k!bt t/0wkߙnʈg~BY<%#̫nr_W缋DžClK&F#xL|R .U*ljӹ!, *EwU2kA,lq-}0Pnռ YK d+&*iK3 ~_>7d\;KF' M"ˮn|9@P-Ļ2TV9^E1%"URMҿ0ťSK:S#I5j/Ζ@+V> |_k CC}`CZMrQ=fɊ# TVسefja{nl -lD#y;HwyԠ1k۩>Qg0gB^wQAR;72y||$f!DR딟a4e h0ZOGa@טɹQ@v̺=9L"ӫߟX9-zl"ʊK6VUWR z?)BJi]2Ez O Yp$LU,l NЬ G mTMߨ5KLעw^3V36J7=U, K[og[8 ,ʮu."i80߶[N1I.RANmbh}ū 2M|.CFpI!j؆%N*ґ`%0 s/ؠoqw@_L25%~ؕѤW j4]qu^_(iw;}y95_hqxȈ;a`v;dG 5`XkS&~qCpSXϸqo$ROV[&X,) iP5~&5y"ڄ8 D>pE ;CzifjrW^N{I6z{tlI^b?I?*;1c6"6'9( Y"^::qFDtnS[[7NRY ǰt̰&@' & #\՗!L+_ =jw⇧Ehd.LȨ=`l@XU{8Ne9LEb>e|ձ>YWd6ĂN.2CK8`iHV4?~f{KK=v?Cb%d*JChQbwj糸.C}ԋ_yN[o=XтxS 7nCxE%f# IHM.}\ S{. SdGGJ('q0℘-%&/U[U/Q5{ L^]^DjjqbT/iד}?5ڔ2V,ItTo & ݎ}dOnF?Ρ?mJ6h{#ݧ&EQ"+A :a$ɤ9>8T{܂lPgˡA*=3W`e_ wt92C1WZi|': :PF+Vd;JT F4޻-?Wp* ;"D\\L,hX xENpv6:]PC7:XNT}>Z{qvH(فd/'}#X"l?"~15hhӜQ|K?%}zFmϹ̇?W ZPwn`lJpѡ%\*jZB BH/"X~GS>]gI”^K@e~"Ada8".t+#[`zs/ajZ} W)d3 wyyڼZ@8" 8a3*ѽN,A +Cr/}Ӫ!] 8VB:M{K%E4B.|")r2<;](/ .fF@H!X$hmQċdb]ˠb'g9Zp;l(DEQ,zu"?J0Y ujPv=Tzltz̷! cF?(FNM4KN]Rn;sܠ⑔UfyA+B9s$KͯɃ@6[@␝(ͥ:*\wcXz8|UXS^=3MKt0y\ IAPq {5~[gE'"u}^;L?PT@g%Гgxd<ʒ"gDY ><6o([GB`^<d= |Vpc6 4@8%[Q! =HB be<Ӱ74wN 7z̺% e1 ~CA' FŠׯ.1mU|λpcx o6¥%˸dyB}2ӢNM 0 O 3rmq_Ss(:Jttd팉Yt|3J@r&)F̋x\xR-ŜNB$r,jTLNZW$h?Yy\=1J}TdpMxw %@KElphٖP *qAM% _Vn.&,f~/;QN_@e]x"q}1QR .MޒC_Rw0VH# y$P8"wa؆O)w6Dz=i}%+uМ %y9< \/[ǎ%o@?X=XKM sѤ=Ш#"zY|cm9bx${ kD0^_O1HȹZ=1*Snz~Y`rLq1#H;l ]@P!V T6HM+'!B)vh9wp."Ze!2i'3 KgoOUJ-- *2ćgݓkYY4^QT+ GҺj(nwOp]NHrN%|./kh/-b㿸Tuz<ۭ'ݑmhe~cfK*ȧ+5~$Rm2/?ţI^e8834zgd}&`;hm$~vhDu{u($ %_&'B6 dq#2cI u>Ǜ'm&j%Yo2r7S:XwC[BLX=oPi:2|enm毚SmB!vD, j1]A6P$8!9&vVa!q>0*)2y;)±n2tpb$ɿqLmFV̔*ZYvsz,:A!4:O~꙳/LfLOxs\Nc&N,KM ScmEeu*$,{xHB} x2b&|@&i9}Y}.9Wh8ͭ- \,ǯ*ha/Dި]qS, |+ /v[7V+tvx*94WVD(;8h\N f6X#"vFH9R`κ{ (8*qX47 ЃD ĻA~(؀B]a1+~J_O-Ai;9]j :H3Wݍ]mV7ի-,cJqz-Hzny7(6 2k8uV$O G #dT~y*]~^NXtH?ctNGΆ/z{,N)ŭG yJlM,&<O4bqjco]Lk<6|jGWti oz&X V q)џڶkBtE.9@G43wlpqwgFSo悹D{IGAVaDnkFȺ!%vϷ{R C8- !y;:m;`.=H e,=?utuТh\( S>ڼ`߽8:XUKd|:oue?XC s8]<~X[.5PWL@E%I_33ak'U/LH_ZݡQ᪳] ΑXt[\JWhpZfê0iQP[󱟟ϫYb1nv:6pvUdЯ`3b S~֛N;6H0!H YbD 8Պ!D >={JPET+>^Z<9~fk7M1͡亷[1jGֻjt1@ub"v.ѿ^؅)NYRğ䛧G&W?vIXD)2Zl8ƯυTȴiN^pCX_ֺtH&&eGUTw||ZSG# tb͠cdw$z ϲ:@V@a w nb%@ӷt`ܣΰ'şwH/BҿGNSTbt+CnPm[ل,/G!h2G?6C찔bHLrQC4+92VUcb!gԍ <RQ'v~c݌yRb&X oD"!arnUQ+Pe,heASR"~$[pIvԆiHmg'0.ԫ|-)ecAdT^tvۋ83 d-&:*$~Zn4rHc-ydW u/=s8uN2fyƷJX!?6_w)M"bIɷOI֮֠^pxgF g&švj F ci&gf.9};ޭeQhwUyeL&(Ǽ]5n$,)R?3@̿ gc<%~ S 1"=I!GO' P~n-<)(7s!4yMgrCeQ4e$DXiu`1t2TjtX2=6~n_=v+^ANun4FꕧaB yXЖ, ‹qO[+Hp3c8|.qsg@C mF0_%p!&(?TƢl>TP0ry C-f)4^+ QKvӐgyB­ʚW@~*d ݆.YʅJpŽxhj`NfKpueֳG:VmΩxOsUqjϒh#);BO}1eɎOkR5=P3*LF R`mO1 ;+$e7Xv_2=K;¶kUTm4| AThp'OȏAzsfDgQhr0V YV "WiQ4ih{A^Վ<sȄ1uSzhN(g-pUWnjo|糩ቐl"Zt3u%1cNQU?%[qXUj<G'[h4xIȿH9+)ZBUm%^zKih%:LM"IsEWX+ lKG+5TCJ"bC?xF*٘'/_tuo-֠vAg{#VY(Cx A#n+8L ͕xhI$4/I1}bȇa)I5r-lA(]oEa;k$:2YyO2i 䫖Y8P7VWB>vԹУ3.1RU?+o 3}AS Lcs:{P'v+!9r XsnݩjBd[F(PÎ  kD B#ЀjfY3T1z"ߗRRKp2w;oj]ӖqZ_k@4nUF-vEǵkfE:f̲TD*[G'9A;niv`{3ŝG eb9Q;3?_mQ!Q=t(-V(IgVvQ,nj 4rvuR%;zᢝ)f TV|?t#|Thq^WٸZOcc3N.~ oB}wћy$EnP!_TnUpb)\??[a~T p9i"8D7&ö9rH_dKߟrP)QM2Cgg'G{TC7ZЗc{>'5Htǩ!"'59gJUHNLz#*&eN!r0Hmr+v\ٞDB&u*nVP9+xFQ%P4OR[mD_]z=ohtiq>KǶ_p7UVnnRc8}ވl:ٵ\.:TJ<]L_Tw]2pb j>X_ (J%rfD~#L7|vk;訇6zTgw-2S J'dJ:*4ٵ?kR։k=W2R'BHS?Z'-_l\&;qjw'`W* {F-ےVWw <60Ў:F9> i31R!=u>AR]{QdD-R%yvO"7X Zy#Pwt :3lr1F4_;Vh4fڂ{ ]ge3Wz0񙘚aIJS-LĐ^#,PnYe8ys52Ł|gÉMIJ~ f#F[L*̶&JYF$%ptQ\3r-IQ rKha7OX^aMu@w;դ5cޱrgDdRHpVzbI&޽4b9tK)jJ| =R>FB`24ɾP` =-E0&ự9mDuz< .3ia6-b_n*M?\D5 }Kv1ɥ+^BU? [?A<Θ {n;"&S 4AGw?w MtO]f.`"D#[XÆ_-=(u֑)*<6`&/EV(B.n?UVV$-4v>-ci\Nŀdd6J8b04_]Um'J]_[ˑ86Cmg9nN[DE˜5E׽*|-Gn7I .Nc x3IٚG>^Tޛ׿S aH眍&N0 Z#Ю`(-1O'p~V_nKz(*4UڧUjٜn~ Y<8INt.L$8\UT@oumeMk\Z]0C[~S;NV‘Aߙ"$ϞY;/5%DX#!p u0H8V,-gsaB~q#D0w"}up76ywc$a<`Qs:r"}+JF|'kWNba\Ƒś7'/(aUqt1ՐҿA<(2tB֝c@v֟!4sL%X Ԧ~٘DĜEP{b3);̤&FDUX_Ihrhta1 xe.[rLcqbˢZڰ0pskB>fʂ#w%YGv:#% _ތNκ.o? p _e)jF|$Mhzb R54bdO%FAkyjgQvE\Apz{Onր5d{J8J,!~+#`%quS6Cj*)x{,Cu KG ;Y)w^u9ATꆅS?ߝDo7 SP 'NݎYF=i4b`YC`P FA/ 4P aR⧂ ƺ6Rj! XUCڥQQ6}dw4t3wDZ ?.H Z5L=yYǵ:kg5h.z<{d\4op/xjrYLʬH,QD1D^ "2XqLG/@H03}xwe>#\{:Kud64 z<̤RhŃd#P{Tux`2@+ɹ%߂^k!.<&5\3e2.Y9 jkNk9Չ'ÄO;c+WU1i7iջ:,% eavs>S=ԏ;)mwh 63;WZ&cPqAF6Hƍ¹,G,<)Ng\&c <ćsc1$QiJ܈c |lx[I / 距n_VT(ctv>`]ݣևW9 ɴc&Fxx^vT/9K90`g p  7G .a/Y%Zޏ/oQi[\e5ٝA^m:Ej 3N}x+ZFiJ9o9E҉:lA)Z3Ik`heV!;@Ko c dw6sBJ[ 5yK\0xzھ Ҙp$ֵJ jcbW?ch:1. IIDRfa8rrCt(=6's"pf ]au-W~ OhĤAdd6!૊ JBxf"aqRLoKoD I/%nGGo)->Tq)?HB&k`rTf{2AMžNP&3+BWVD.\ul:r0:e"ž,^n*97AKaPI\ZEO̙Yg J^Rm7:u<.eJa{o/vwȪo>Vn\+FqpB5xw^|.kB& TMu UobP#}m2Ԥ$a`C`]m$}IpgWx$M 2'P1GQk@!Y~3:I6 uψ:\!"ud)㜽+p̳l&ʕp%_6zoY܊AcY45h@sԝa'H{7[ѾMK&rnu#%y[IV"%[L{1ۜkFRWf利 2'kC''$+Yw2i3oCim* X,NEL (G6L s *2;7Uem7toY(;q?GlF\?)2Mgwt"&@Z8L  5?!5)y; N$ڝj~"O`k7$wxI_><~b!jt:ewya!&.w.r}.-cDuyP)TкQs$1/L"xSÒ/$Er@ޖf_5 M쨔5MW#1ee@m7ZB5tQAt0E>{;\y!U^Ŷ_)56RP˲(">nV2 AjJ(kUp9;r1ߗwpa%q8JRyS yB_npY ^`ad!l29zx=4B/uLXiERر)&ԓOB ,֙ .= <j|Ȗ@W<{ ܮڵD6fE<6Ϡy7(xvSʡa 8o/z$j[^]3e6Z]{G~ ׸Xt lQ@'jghaqEG6!to|)7=CvϭXJ%5.nՠ,T_-\Xnku@83(!R !J2i.{&[">ev=#q U%lB*$];_I3Uء3fAV|K>[v䉄3KzQ shB jvOP p`R@ʳb]j^j1P;c sl!jhsx"kYoq=dl u*tNW@ćgzT0ˣ^N}Qr"ڻ^]X ]"f(vi@U `@e~2f!b*jYۙök: 7jUa). =aks϶NaܯΧyMl?°Igңp6b+dЩ7HVC>8fmZO8??rI&j?; Ad!i8_ \fB#XevUaM~.̞B@dV(ޡ 55Pl/*O#Px̛+9F.;CD#q(؊=rL˲zE9fP]~+ &!)B1!ܚrl`_F ؎h~SsLZpqME??m!bǼHsRsܛ"Р-OYdZ5ORzF`](_qVL+*=K3 < ~q\U_@O\@Z0{HȊxXCu%J `mG(k؅ JVIRHu|]E;J 2UɊE!x i[K~K9g&9)ѳtNEB}q# rZy5.a`7 D~a/|) 㐉@.B f踰韹 2Yʶ|Alޢ`nHL $ۙp_NƘ(B\NG`QXhs_Ax;̊$n?Ly>aFC(1)[77`mu ܸ6P )sXBIth>\7p‰2!d[:$bVhA ؤ/$gTDNNqM}Kh _Puk% ? OL^glmAAFZ%G*N ׏suMcefϦxm֍xԟ]bBN)(h) r !'_FJԪup{XjC+\eɫ `qb[/}Y)x^^i6Tg߈~r,"]iA *ZɷzayXȼ Vi `XyؾtYx ?s;V"M"=EH("2ƒon^K2RmJe@[ŤYBc ?dfS{{i)s!W-_:%noD&Y;|P]W9B[=Oيա( 6R ,ģWܟrWkJ/Ó^H遜ucj 7%X IOP&GbȣJt0%zOUg⣸ ڶ3̸"˷adVUX dֵq/!pQ^uBe?/ +Q@J,56-tqm1Rp%YmR4P\Zy8 [UiOgŇ4N{Zxp6{~8w挕9j,,=FݣP|gCJ>%nmŻM"W!5b8\Vucqio^CD d jDCzyUYwtiH҅}pIࡸTAD(_<ug N' m WEPzZm_tiV[j9q+"z9V;TrV빺=][,? Hv+*?#I[P*u hثr.ет&7֏;M젮{H|-\5Tңm$gjB%*tP0GwnҔ/7+gW H&/2Z\zn3-!55X- àdL)xn^hY/1\7hO6I.`4D*L z1{t'^P3LR21LdYV_urڟB.*|g05<~oVJ1{|HJħ%\\şt"}~|QFXc| ܽ6j ;_30eATE?N [8?Ekx'o)ubή5|S6{Pg?m8Qtr%1gP[JrS+R+P;dFmVO/l2-55qYq N;g+ 8²س{#)W@4xNKeeiK$ɖ@PzqNNAdJs%}Nm餖ADܥKy.8]xlVRy1NG*To_`=7$Ut [`C:#+(Pʞ98 ,V0[w]^֏=,_$Ty}DrVcδ3+ 5 |lBwɿ`Sk#۹SJ, b[ocT]md3LRa6̎ovav"viS&%ڋg?q*$ ^|;v3H01)F G% vAE\rK։"[hySHq>n%dp;}W/&+֍M%\ǷGoP8?GqM|B6Q,Ԡ[ꄪ\>-Gj (f 3pnY "MB E r20\`8r(G;8 (a>*OW1vHwa䧆M6ѧuI Nk e`CQd<3*]lQ"ɴRjy>Q?[q8x#XEQ2k?)p~䩻?,3>NT?/RrbԑPe.- Q ܂$M{ߋl.'\[YBBdB~?%AEҴQxRwאE"y*i`C)e.=Y{>qB;fxZv/> gbޥ<+OTeV̤_<.pׁ/]}ݎ0-60'3?b!Lc)OϊE gc u0cWކI$OW 6|h험?ĺ,ዑi=^ #}nn^;q G|~nG+8M8R߸VI' kj,'bO{&g+rzIaq-5DQ4Eؼv㈶ZX6ӭWK֬nEM <*P0 *΍$*aqK*a@(߁6iw]FF)| u44hk+.8:U,t&R%|?VdtF e<̇v[eā$]QmԦSdc ZT+YDvku.U/?$mCc囥9>,709 HY9S]2#.7)hn뮀H/y)hxt7ir dR9+R|p˙qP%r I8pR-,X! )V|:; 7 υu|=8YavgRT%[zx0Pˁ4"#%p }`Կ7R㞎;P-<%5ChfwFA;Ë|D^w("c%.O+%'7E%y^a%u%Ո3 (^Al%3!Pjybz:OPm+FmѰ$*bIO%:% cQɤM#@AT,B*1냗 @Lj;S@w=jqwd9Xz25LLMX03jVԋq˃g;ygiA#_bI3Z_- [s}>~a&KG 4*%V3Lx-QoThsb骒WU Ɔ:W/ݔQ K0 ntA&T^zqP!/juۢ@ b%J &څ'T`z(4'XY @FKXWFSqsDo,.RGd(-6ye]5I' IW.WEt-7 {-Bjw .H HzlMd9^ ܦ+]_J&v# YZDk~;zz3s8 o`xj]9;ymA!ՁNq3$ˊ$ϜAL+9־Pg@wW6:Fol!Yi!i`UX;t/O?,Wa@♋n$<mP!>4=uqa"+lsK2ƅK|߶"3 .@D7 mwLJF|V;e#9.ZkTZ SA~Q--Vlp=g0J"lĸisr C7˾V3Шbr/2JȨ [A)E01@Th'і.Й?Ne?T8s| [L:BV>y䣔"R4D\a*R rlKO/*-)Dz|epeFq)D&Rtv8ACbGzZ܏]3Y`(G5g‡9 x!-3!Qmߤ2,o ֛!4-\D -&"3Gz2}qI;CMu/caќEM8m%R֧2~͉Gܥ hg K][PW.7ʃ?:% 43/3V/2_@4v lc'p5C9pM4y{B b 8LUqZx/J?aZ\] 7\W0?yc"%\u{@G &t;|+AMP6-7Ҧ hCիʚ;E+̓G"8 YMjZ1B/=P\U.!-/7qf R 2!n)'tBLPx&?t&_['aM6S#c_C_Os>b SF8vecV?o:ÿխ[3::xhvFçPmpGlYQsYW^~zlǁA} n]_4N;XV^nDCd"_\_ e*di*b~ay^n-KEoJ/}#*pOCj诪8˒ς։)][!+K %8sAfʫCd\zg?sPIG̰ aZU5.p-%;n/Taє-$6[##Y9Bv"M'WmlQ02& >Fun 6\AD:%,.=յ aFK0ٔ>sydL\lWNsޚX;!Jg rnjzζםhXK拓b~6'YnUx0*Gz IU ~YC+`=N'⑰/}lXŇ|U: ';?DLƦ;h3ِ80iZdJȰnLps3B5F}*;bZ~Ֆ _p|U9a5 60b(U  v,+s[Hzh=\2]b5 c"@ZrFQ ާD2G[S`2}eWD&3k0Zn3tUk,=[H뵰Kp˺]U͌2{%)>zuu3h]7EXBjpm9ǭ=>W)'kw Y;̾^!Z ( ?_gSc*X"،HS$bɜnX4N:B Qu}ϲaP Fg-mT%/> MBU, {dE@K |R>g0eF/X;ӗ; ZfP5o{50cB{3{V;qЊ&S[tuZa2 |:Kc}w ξa֓v")sj7f@JF;8AS$ֱ%ՏII[>e XR#1DHG~W>kJdԄ] !,`|ϧ7Կ?oVqjk.ֱKW R8^W1CIw Lx^0]=6LP W2ȉ:!m3g9de 8TXO"ἘH[@V+AS=gr˙zuvhLvc轄: N.:l*XB@vL]kXZ).;]|&gO0xl愷"kl>4 W\̃4F0H$s9(Rt dx$ո@_S1\ SF.l S @w|%[xf܇;ߒc݁דU-563,izc\."3L0J`n!(=oa bQ=/9=ü|?e:Q_K`*ԒlwN6I,2쾑E:"~A/hJ5R"elY6W .'3[:w#w=0R ө񼿄]bi֛ͥ26n-Za 3a V3?x#QA <÷+W>rlԷBbCJ 5} H6thlp>tev816 f8+H_@;Jf]˵n?q;M1tlf&hyGK#1~TQYн ĵ߱9_ėh4$\n+ ;g=4*ц\3~?P09f7ҮICvxCxvX~7V}]B߲ͮu@?˒"|ǸQUIpEoWS2p ѮwA Ղ.SH%k0EsS5{@xրZUoH2s~ME8㨶iW$ }!is_V% @H,6a֞SZʸsFU,NgΔUvc"IjqP+ex6ު.碠F=4 X0ᄛw+cRNIF߃"2aXoYA8մ|wۭR9h!b[a,>d`.T՚`Iݫ'f̎&zn4,̑RUގX^a+:VCpi5[xF60ѮkqG@[ ȭU7~ޔAOQ)&&&TLqV<<5&L3OeD 0B#_Xn20,bKظĢzJ.KV$—ŀ4f^W(:FNҡk$;}Q /$h[ve.X1ϻV\ٻŸkLZp$sS9'O[GHMWXOJ^atW1#zq:rr>4y1]ЬR_ n\WO_bp5:v8<㚅D5 u_h4KkA ؃ szOgdX@޽k> cYɰE8<ȘNm#KP؎,]5}-6F9CqJݺ- y18q_k ?yPՖa 4.ljY?;gr_ڧgаYWmY#Lܧz4@D4[/筏Q"毜oޮG+ngO2冞D!Ȥ4Ӡܸ\W odVyC\O֊BadI ЛR"è͞ʿ)vUyue^;[0OvGVd:FHHRLc^?LhxCwK$6WS}Dึpt@a;St2:6S 7?x ~>Ǧ v`'Ʀz~e.SSiDmpս:T?n{ 0@/ _J3IrQ.s)@r"K33@E7PP +ERW0g[m0>ؙ>B0:ަ;s1?Mԑ/C+vMe[%#zn6՜gW-56x0ao >ķK(loYpOm@ʵOԩĤC3[hm~^/*L,/d`@3K1J"Y֥(T1kRm8ҝօjdi+R{&21ˮN?!ȓ:}qUZ!X%[3ِڠy (㚂 ƙ$A:LXuo{gH8"$ff,ܪF2e ~3=iuQ"?zE)fYl?уp,F3'>k(Jlog_XIIcI?s(Ѱɷə?ߡjM!AGewvOL}2 eIND`2 p9SR4S Uv@e jVxӣ0~U+(Çb{64ɣot1rr?@D'l)*8ʗg^S[Oo6T_LO%..Q>m[|zB^";P|2ρ'juuB2; }"~/-yPбeͅ#$!_ڂX L>h<BEvmj ,FbZs/I;:^}oԴ"h$6ꅐhp,<8:a^ì%Y&K(XJ2CAҞUDzWb{Yqt:菝Z(kBP]{2v U;g#y5 щ͎+_W.OF}"X($6Ʃマ3-D>c$Fuɞ lY{jQ8< jF1+2wF[hLE1nБ{ w{M_ʡH3>ﯮ3jB[B3g.EEd~?Z S$)^?비TkYϔvl"~CJzir/y:sg?aY hSe%X"MV;kER ne`,2g WO<}R/mP*d4y-aMEKm̤?ȷww(Ǯާ ! Dj<$@^ ]tS$T4|[{XѳZbq\ SP0 7@5cGU1 ܣ2/)ӣ]o.޷U&0jL&ZB c# *hgDӡNK Vq6ƭ$󃨇mNԌ+<*JnX[Ix-#e#S-G&²BZEkzm:с0jHrR|+ ƜkcTA5ԯ,n8o cT #:|w3Grߕ2b|A)Ow e WY4>>RTK`)q( 9Fn$i/]5}|`ujhq&`9Z-#2ӺWƛ} w@EQ®1iYϢt"͈r$Lh/Qq<~%Џ^IWDvFz:.8 my-}bb̜saښ(BQ,R{S J`1|wt6OStB~{ެg?赡X߱ c 'LNyGNa%L 1JeyLފ]y'酳{"(!7KQ%qZեUoYbzg̏S1[ 'sf-m|4 ʓBh $GjOH*Im'ηEp6 &"KB6*UKAE^1:̫M~PsN+ZKz:u<(4jH׿uP|aH+78Zؓc#xܣ G/[##n"5ڡUwnx[x pD3ѩq{`*l'hek-ȸ* [LedēgG_Be)gzDpmٲY`q(I~T"W!cĦ#Q>LH BI7l9\ @R$6ߓHK ?nL(U 30FԜPVN*TLV 焅!btBŢ7=E"noͼj YxPϚ>B;yVNSk3%4G> stream xlsp.v~m۶ضmm۶{=kfj2"q{;UO3&:&zFn*#LNŌaf P1s01`"NV.J+6FVVNu{{7+K++??T.fs+3RB^ afgddPt52Z9Q6LLə̜\fdo SWW0LLsq_\W1_:yGabZ,`ś=?jSW1|8jc#odkupu1sٛ9Ž?q5uR.F!dg}QY9[y*Z÷f93S+WL7vO(^ATS^ATf1;{S+; ?|9fE#d`0@_)$%,lM cag018Y} ˿I'; lA%Hs9A60_ݳ< ^T? s $bN)@Nݣޜt2fOwrtn0f߈{ MvjZ;{9g5hx]<, rV9xU}Gj2[ݏdLG_Ķ6YGa'hJYm+] ;IRõ c<\p0j/)C6Xg`H=tIsfJW'ژEq 1tDžmT\UE(Y}$^%cmuQ&^htEK.%Od 8$70tЙ`y)^qK)Cd$9"ͤ(EK7M;<E"<~wOߺf m ՗ P{@\J~m5EيE3f-' YYJ]uͶ D4uθq6FX:S:cP4Qթ^twfn$@`H3]5Ƨj<.b%lfS41Pdv"m&d'.l4j=Ϻd9-8Qﯟn{%:a|"9TYrƍ5" kp{ ^o)76jҙJ.s-72SvD]7b%$A߿iNxmEaAPπ1dQl& wCr EƈlP/C_% f$m2<|K{U jG_2Rz;Ǟ}zun6˳PVtZ^~X `5D֨)FmlͧVHmrm(/)sDqI ԠѨP =U`[wb?~ɞsx5}>΁(N 6C_:3.)j‚ ~T0P1ԓu61ݿoc_ArBSôYR㮛&_g3 1H~ޅe-OH|v*aKusٓ 8v4C_@)HHX(;]UC< 7hfK]ioQj2 DKi1re ,ьU? k_ÉGYc\9]=MX*1פa`P4g1AK}5 Rܓd."i'/nHo`r ,SA X̦d[c tlf$}f{ɋ)7ԛ8ǕXpBdkfvnr_%Ck$yʍ%ttCQ#6|^w='~h*R@pHL3 %62$gFMw;N:xiCf8r=Y5_# m Wwylj"}?ާW2ZSƃyRiHkg'0)ԯalE^,u˃GD!oOIa@jŇb!}/KM1֏^Nw1R0tT 5s]Xt=L0Z^|QÞ$فb ?ïBI}0Ob_0۹)Hq-\Bze\Æ%?Et@'~'Hx1a."@/*sEv'0_@fHbT(=Nk`/sUGIG)eS^f%9W^N׫_TS@"GH/UA|s^L=lgH+R{* (IQF.p +;|}&L@kزr<‘̵Pb UzFDr,ylM7XE(x,% ^NSZK +cVR|u"%Q¿£I~h@A]Fv﷼8rNwט4MK;=O;,>ڝpb;1. dS ]^.sx@ƫT IUNTF!VLˀ*UlxS 4JGcGe[KN@ }6.GK*7^ p)gFlt}TL 9ѱѿ4Ph$yxL5p5!KkN('8H-w/1?\[>}P%Z/mH-vI=lHbiU 9N6XX^#! l,|*4,$ڬ,mOb,&vP[ɽu4$u29}?8sd"vqy̓É+MVȯ{R/)C i|~'rICW6OzӃv]:ؐ{:]ﯠؖ9̅ 2`á5R3jKF5ϻ'3GNNL V8:ۍ|4/U7ە*`pezIOc }MYe;(wLmϧw~$ Q=:nlCKK7VZlTf&ޏYWFX֭R66\ ` PY h-q/tH+%k]/@GjLSZxop3@i!=\YO3HUXT:ŌL}W-!'#dTZ K8naSk,߮[c\Z( Cijys/JW"w}d  ,Q>6?Ss23}]β[m}ۍr [O3s]is7"]^eSa>O} 4`Й *ify[;IsHPgVηDna1kZ#)*+0W{hRsTwd֪֊5{ueNdD  dR)#!JxFU;"Ks)`ǟ(Ew49I?]ai-פFCFj&nA%}37jMy0Anu"Xs/xdJ7RV/XixOզ?m)l]J&tب-ɯF;hee\F%2ެ@,-Y輫䲯:LgS*ujn!7hR/M $M9AY^>fQn W43TL~ө2 ARV a7#X>9m3<!TU3VeHčp7f G9$w=׃~B4IcRD4YQPwU'Xvt1vfCQpǣnFҁ l;~G|g#Ȱyn΍_Y?9 #F:`FaZ偵 =h^57#7x7я7E HZ$DY c(y݄ȴRF-WBE A WN]SNxg~kL T~  [$/ W { :~ cQ֒ɏٻ1QDa D o3}U^f %ptdONK&U# jr KO05l$H@Naែ턅y%EwXy5j)|ltbV-,^tegݰze˘a90lBEQe<m^̫(G[vvKwf [u*2W)8Ffx6J{[&?3f!}i BeI"Nczۏ?bdk)N\"\(5QE^EI1ձQ}J&bEOHmd9ǚy51v (9z9rH!503oaLx< qPN2*L`l-}BɆpKmaV$QTِI s~a1</KЁJoqnZw Uo4T2UĴc5wE# ay"Gg6Z9s$0pqʲbӍ} 39N.yuD4ͽQOdWQ̤W:XWR`yC:Ha4`:YcdeR-?S/{v!Dt?ǁD?`H}w]D|D?I1ɢ_@ʡ"tݯ*$[2L%Oo$ 1^j/&)b=dG[^v; W)Õb"!0JWWv'{9ؖ#&/X~ ѻhL1 0h4 C ~q'G7ɹϛv,ʼn"Ds:kpodP|qA U-a0a!cB[XUʨø=ص{ߴ5DGk`8/?I@NCUYMh\zdM!E)B RM?x iujt o' 61hOS \|C= }|逝*4viMZ,$6INvr>7`kE [8Q&* c"B5H燡1Q<#b.kA@tG 4#ū]_g)gqiDeܓ̓ݏfIcx ʇ]iMӼӹ_~!B$wP>,Xrt-xHr]HS9J6~AKZs1%&ۥ# nRI-fI;z0 M+#Dd<Mx7#Mތpsx?&q޸;/ QjKчKs=IzҠ$~i*U ta^[RJ8$qC q92B EݻGkmynIPdܲP u"Ajw1d~ ?4J!J4)P5 "Ć30pl@Xwo7d騯q޶`YiS|j^pz^,^~`zꍦHmj"䪵ϼ͉NxZ׍g='0a*|7جCIo0Q8oNVٻq D']m ~p|;p} g~k..TeӬb"Wԧ^<ě\*ZO`jT'z* 6 Q;Xx+F+6 5 _vKʈm838|23(R t.+8d%e"gWlu2l`hj`@;xII7͍=5[8Ϧ&VDŞ>1L Xfb}h\!Lݩ,ȲmĔ^Nhō}fF:Bgv _lS^9ym LaǧJT{6ܧIf&G [gR"}u8p+ g1vb~PQ5zVN xjXS&Z^g:k}-۔f"\awb^+`whM3 +rIC @ ɲ˿T[@oTwDGppxi}e6~*sSې$̔nƍ/Zqǀ~R*gpK\JU0Ia!y$q6 >$|z/u4aᔘGM cI];N+:`6 #,EM\uv9YWPu!J{k)эD5?HgİWrBc ;x }i){p\/5"*ue(tH,} iy(T+fH˗su|l(< 9u_0(y]\vPUw bŀk / .*{׃"9n[-lKG84OLvy}[Yv-)t>MI #qUk&|w j)`瀀i}Z` Jk4فfϬ.#8&.q(EQ R6I'C9_`-a 5pل ̭E? %7> RryTl~8]%i&ˁ_5#ڢ-e Bq: Ԓp~Wa nm@dE|f`Y/~8.2:?O sHQ--;OCJ~Pw~􉑥𥛶oXb JFtl#yDy@t!EH3prv&E5ݰNzK"'+p+33hW^6ָ #CK= /4, 9,Lj:[0ᙢKGϜ#/6)ɴ֬ +*l9{|㬞tq4fs_/qWR΂4$F=%/Zʄשa"nȘʭVPݜW*9e55җ=# E';Qjx | Co*'\lI8l_#EZd =S\rjeZ;.˷l{ Zlm'A ANi=R֫0q(?L̈ ,lBL-7B<1!{ OpB45= ֙жpo1J`0X Yfshԅ|Γwj\{4&-:xײFST^yۃ쿛6R <%WkXU &6U YmJ/;=djy섭X2&}P5Ո5ZolRC tHˈO`9P_4Nf3f:^w\Oţ$3RDLSH5<2 ѻe< p 8N7综-jokzzXb&wےieձ7;7x|6Z{E?a}^al` Gp, Bƺ7&3g+0D7t 2@)cz[ݺF#;x`v++w|hUI : |% ߂g%n |F F8VE"ҦcGf f;u L7%o;/8km+csWs.cjV볃QSY4&ݞ9"JN%]` z SC-dUMRm#L&VLˆer'_)BA?3sh}ڗʽf`7PF|&;3fIB(fcn6U|[| Uͺ(ߑgۚTp)MkψY8}qa0"MPz`5Zјty[V ~r7謦;@{ϙ]S 6~ɥa5* YA>Zo*Ypb@EErc|1,r:*u'@܉>rۜ|U*5I:>"&S~):?`49dGq'l(MؾWAk8OP[r CL2i.7"CH?۝dv,t9©NXr5utl#S+\ϱq/ax.yOJA0ùL f48tS?FH !.*0ⴾȷDh7#B/vbHH."]ܧ&5wϿlgYNB"s}Z |cCiO CkrHJ}1:iU߄⊲$-3S' XJ9cnӞ1{ւ$En"]XD/i耬h%aarr݄Δ3su/?:}`J6i_jZzxdѨ{eRt£$uz+Hv\spe6SP&mt2iZnǩ'/P8Cva:^!DvK5o7~-C<Wl!ח6#M7!*o(]Eow2M);{jz1"q̠f`ie_T}YkKP{)ExU XčVy3rjf. #3XdL/'X{H\Nrl' HL3ȱl6cTX*s&*i_(ݜ"2(r|!* tyLȦ ?Yt.t;V~6VBƭ&  5fY ih#Ǩ ׾&j&tK$p'+^E=!Cc mP{hN}#O <n݇!].k3a]t:rۂą6͑5kVa=ˉx5 `sKQl}32 C|QAGhYy5Vreۦn@RjwT]4s]ڲW^/N鼐4q<s +hc)V1qoA%Mfxг|ƞ^;ҩk"nmn!<{հ_~G#'9kH8\rDoRD5\[?IȜ7ʶ+nNRbՓ4`=;CawaiTb,OiSnۀ5'YKa^셋Dc/aVlkvIY7]R~F?eBC5T<ކLd Q׹}xZ0V.\}UR.y y5_@ssty>qP`9^.["UZgmgW_렴O)FGvEw%x0RjC\Vy*QP2yi-L5ŕy<1ָXfZ$|nO2fCűkqkG#N(i}HFbK~YA~*9}wueevi4rM"\qr^_ '*ϕ:NYf*ZxRcφ.iDq_1By2뢺Zpzdj ݧ=Yn¹p -v̦<Ӏ_cNzy%H3d|&7/[S7Q"q;m6X:t$$뼧~0. J٤ϖ`Uac"\C#՝ex'W{Zvy1fd uh݊)қA%R0'g~(_ο,l^ܜ'&T8-Dd7!jB p cew +dsHgY-r?]'{SITNz q)^L /ZE|2bY5N|Ɔ+%^6P<4c+.ȝ>O"ZKYB,s;`;Gjd+NPOt`mlcD:ccD$M;mLB wZ縌$<{%P ŷz[!Įc-@BĢy$,@ω 8+2KnQOÅLO*^ w6|d[8P_7E~.;vֽιnO@7TMb:oHӈبˏ$$kj@I HۚK[{z\;( kk7;Aw&SM+UH\o\u, 2F'OB ]Vx|.)k.3 i`0 5`]/y?d+D"GSOUiVo'5<3fgnjyS$>l;Vbmcu 8q4~Bl`[#]*hzP{ZF=lb=\\m1m־',>pІֱrm=-edZv*4=h9}'l`)LS'i2=t~lM4wBS |TLK_3%w&$ Fmht)M'l!ƻZF߯ooG_+YgT (Hɲl'c4='$yt@05^` ez@2"^Ɨw"eu\Ðb-ns;"zX[p؏T?\G'j mi^!whpį:OS=oMS6$X ď5 QILlyS< yA5|2v6z6]U3e^<ܑ?V'$mGA)q@_Q4,#t1 5Y4xPNB6W |Z[ %wp>{4SSBIIP%ԙ @Ê|߰` O^뙛Oz2z9P/ny;sKUݝA̭WK7.|)u:"J׽C94:G:6kHre @*z/:Z'~[[V"B%{VVSyt;/ɖZNYJcES.inAM5lC!Dn OLbieLE_sDEKX !]R!ZOK@'4<^@{UhhD1Yz0XjaB9+ۦCP9Oxv+bI󉡐 ZxiSg B!yv"ɛuR[=&S(4ENJ0w8)z2PʂdvQ*+uO_?G$I0AǕY855JԉD0i%%!|8G2W}Ee 53n5O\9 6k a3uUBq~H/26 :oPsGg {iQ,7j#_6ZJ9]7 5쪃37 MYջGAJb_eU {vRnfmEg`#SPf2ͻ ~AmgIM`7sch~s2'ܧ y~ǣw1i$CM hvZ 4A!‚f^ŽWm+2ɾﰺb #\kɎ{r|/{ɀ$C h\*Ӓp!ol~m ' Åj8ѡ9 4K gHQ)>Z <t[9)#KJ|/ĸI<OGsHPH鞏ZDQWp}"8Tm"Y;=x59uq(6I+];ZԮ&vN懶^!ȑ8w2~z/G6~Iד9|6d@*vi[sSo?Za^Eo/flt0_|&1τb9g0UKw pp"CZT_9oa{:\t*i"-%OpC^}Э{&:Rz\2A/V㟹 mIac,]IE!넲W~I΀%s*2σ$@SZ{}5hXAPp2iKB `sˀ}  @ Hŝy` x"cDhvLPj(7鿗Uݷ(tiSP3)ӂEZzm<;D$p}.ZVѿexĊDSA>(%6o¦hVMG; xk-aĢOuiI! 9al/<4/qWfnf"6ڶ=>Ɛ=ůacz!_ijl`I'XL8B)fn5(,Aߵ#Z Fߑ$/gqP_]hi /)ⰉoA;laK^ Ҫ\ n!B_uxƳ%"5Z U~bfdط(qݰj?uH\B Py{k#(M! fIE()SjǥCC {?98M}g`v'` 7݂4[n NM2澙'&@b %iV=tT͎h,,uR -w+=030SU)Ïϯ{FWR(d/S6%c,5cI@%c=s']UM$A If?(1 _R#OM9z1=wT@٬ٓ@cd[˫YBXnrEih0 D~-$f1OV< ;MoӲ5@mWKka-MHXL4!ţ}||:%-lRLgCxؠ~NE \? yP+1Z(Tͨg1 "s^"8UƳ! H>̤s}c{ML ρ-ۇ>]?^*U+i%*Gs݂-<`/'!gNKZUVfaLۭe%;ZM~{v,Y' 1:<9 P fa,,*OV`k,`2J@6VҚupۈM'í{EUL严v/{t;^0glޭ VnNU*i&px{yN =)O4(SP?aΤڧWn<&r;ٝ ~)A5QDc\q׽GP @.ہԙzG7{ 9"ѥʉn6(?5HCRHyocD'Nz+vCD7 Fzoz0['HS}NC4'tVETe([S@5Z*O:$() r:lׂ2U|Uia΍e^IˡI)GLڽ"o 0|؛ .բ]|Oײ_}`q42H[㌏ zc3 N@?{qc#+Y9>b. = tvZ^z_?#9kDVS{O$ dt`Z=Hp`Fڵr5 *T"RҲR(#rp/8LJ &A(A@ V"[C j'qb6(eJ}Ks6j&SɕC/?ZWYRrLB4F ]A ^INp~W%-k|?q(([ʊM AKUG4 Xepe}^rCt+2'KYq|Eͯ3" qݔ.3e)5Ug^ߘ Eo(j>2钽`3*cYiҘtS d *GoW t8mh|F@[tqN#'Oh6T׈qʆi e,=Shn@A#]n(rT $f҆4(;[rA'e]I7Tk0\Q*o,Y}`7`=K.z7!X50paE*ۀ?]+P7*kcW Eu1 <K1Py#iIՎţ%7Szo6~PsćbwOSч>TCqglhF+-s"%UׇW5X;>B2'7dB2>&pdT~lfw+ԏ%Ph ƇKNR1>N[8FwZ%2‡eY" NK|ڽ>zεR5Hbi^ cYT%ⵅmρ:1οٞr*C㫙:.]״x*` YKWS3-}ܾRET >?\g uI$Zoh~a ;tl5w^w.,,e= Kq ם4n؇쳏"Ż QwV?KCN*7^6۟lamd=_=DglQҞYtG?,( $$i #Z*FAb?O1+UBG6>_PϽT2"p 0|%R!XY0f!KUW聼:s񲶤WC3 Թ̕pd#[JPr>#c՝6:* cj1͖:`")1cZQj@8H(F8#[V5Eu|ڥ07I.NJf/yp+oO,1(U\ -QRpچ0Rw4,)>k[q>17-N3@k,s93g/?kGҀ/ȓ<$֦_m)Qq})e s&-BIXGdD!´M)J|?bb 9'̉{؜w?/Ud?V f՚MM,ް~ŇU*n7ʘb/G1Z G?߻P_7)N*~Ws'ƁlprCN3jkÎ0%@݊/wSTȫΞd'{ 4ܵ$ME/֯G6LMVTfZ E7TnC@f7w. FӧPW/K5] +OjM FIG{G@98t`BK}-X49h VT(Phlfu%_]''i;EOK2ØX g u"6<. 52w СnV G.'+,;$HlXVS<9]-(JR4$]t!# 17%U{dJ,WzNzK}DxMzYR=|t|m Lgk_ԋK5X&Gaa[p][ҽQzQ7AB2͒D21t㸷+0DQ)0lGЭnb#/a¹u"8žy>?XFh6X $1%_Nƥo&{׬=޺ <s5ⲳHAXFTf*K<*0w.. p {rX@c'Q4F ?8򦲘?1)(\"G|w@"AYhͪ .,'~޼ wKőS+Wo.^[פZ Pw >G:vռ5) ,8 -%c[>榨{"-$fe"=9n-ȼI2K_;K HFPԌp(C$Z;_ϛLm]voډS}]L&BFƥ24j8wCS HǁVo $;[M9a`~K\-jP` :q(mJRS{{Wu]u ѹ/1Yy$14}udUHWdW_/u5Tͺ|QoNwH5 h~0꠶S }x0MЊQQ]VFW;`ޖ!8ON4i!8aI =z ѻtk`X择wӳDN$dL/ȶ]=@Zfiٕ,lZ,!u4۸ f,H]DY׃2RF<3m5]CõmN{Sq,͌):f:,i¥-0UyO[ۻSvE8J\).:SFr`L0%Al9suA yɊt%wNRF}%vj]Gڳ5"nц|UQAvC.jјL24wjkQ T+7T Sn\Ĥ9im.sB*ԌԒ!.A"Y0o3Tj pyFZo̚hX%6$l|%܊x^~5T&l`3\ i?`R% کe}dž!q#hXRnWr#[B:#s[4{w7>Tp1`6WL·ؠ.]q+W$`wW/Las=7R0آz<3LF; #9ؑh\ߔ΁ f{mTvc5 ak+2 8-E2)0*V̉fMeG}-Ex{B؋I'j$+r+<HB>K茇ZyXR_#f#5<RHˡ2[BX=zs/_ZGquɫA0fx]Z `ged{FT]- ś܄-#&d8e='JV\x6@r( ;y#Z'Az> }ɨe:nJdߗ9}#v0EW5=Sj:uxj;p&^/^Yy'qxǸеatN/]TH;:~@9v%uu#|Pqa}Id D)VL%WhI^ql(Κ!a8H 0%uДiH'A|J#|fXa}Mi4=tupᓘr:2oo4U>j+zYuBŖr]d"["S4cnpґA?m|Z98-hAn` v"hgNDZQ[zk$Hp9@cژ`MhpԦkz5}2J y} :&\UfEj(3 ?pnN4?*4bP芄vxmi7Yј_b&:tKĕ}jl*V`:2SvG?tĭHyulR+b%M *FR c: nQ'J6UzWBQ(W_,݇7TxhUr"ω;Dj lb2["( `8TCW.Eմ<䳯&[/3e ՕBmukԸLe`蛳Ն#FXfLp~vLiE>hS^0L[$ҴVeip|X p gE?54bq]hjv+& _OR˫DVGutzunj4 ȊmORêq9I6{ 0bj5 U3Fu2T=Ox;wIE֋~J}Zx]])P4YoBTqq7WYB6a5f4{?Xⱉ@'NW]q Ϭt[zFzOL׭d't B>2O M{W#-\‰]@vsX'Uꙭ%: .K4N׶ I%jʖ~9Z66|;l8(1.)DbM\͟nMt|rVl,_?^E%E .Teb\f BCFYC\ɦɃ~wY%7JП1fPp;O`:45'C[P3p:C|iH^ Uw'w5ZK}fxoiF޸Ca{m` ؟ }޹<4sIEgyGY/?\_!ִT.{4>?FGHWbW 1D7eE&-._=C۩ >5m)zRi#Ue6twSp.mѠd$FrΚ|f o }Ļt90ԀaJ&©Dkת\ '.6\r!h4aqBE D#TbV7;AqkB!ibO2H?4d,WQw"njo%>05s\mlj!?99t^ 2~o.d'cBj WJQ59oW=|zr# Ɖ #^`v2fr@rvh= cuc tq;?Gl)y"9wĕ$4qwᕆz?VQv;>(in0 3{0FWzɼhl̄Fa^6Do6!=BмWE%]YUK;F\M|42();1:&٣ВeG'Kwux^"z #:Aܠ8Ϻ3]>\盛r[b]昍RDE0qSTfzrč\p &7z? Q 6Nr R?2rhh yj uq~m;rtt(VWiv 5T w(.ǡ&&sH7pAu$P%A*;cxAFNa4q{ŷtT϶Xm GW$ج!up0Du\C|A8[ޤ1q[ϋ}wrc59Fiy~phyHo:hRCiv`Jt`p0W)t7RmK=tء_O{6O7Q7Q$+^s3DMвQX HvdiK?%6 }q&@w}$ݻ{3`;]W;BIy\f7Rijg. P @4VaxYv㜴 fG5ӬƷV{Dyl/( 4SP{쓈ܖdqZsoD<\5yPc'u"ƞ(Xl8'_mۭ5O,MߣsSQl\93Q?6_Dsnm^ıZb c 6{[VZ>)q+ǗaW _<0֣`[eue&T*oxgP a?5l{PӠ+( |OA4@7 @OF\0&+ syɪsrJմ\r^wjpX}=#D0m x(y$n\, zCysrZ-ݫ> Fl4ቧ(ݺ;d~9 7xcg@CY" V=Cɼ 4EP_[ߤҚ@*s+8DLVfz xںCW_d `Υab(лױ)b7 ~Yd ($ڌG\],gs]:*2bW)JŃS~<a;( )TaZQEVLu(S҆uzxHǯчcw\ |UiyR;KâfOHخq!"a6@bo vyQNV}!~z#q}^;vʷ>Rgkv9H~#0%rt&j_\ܖ35hOS^IΣlE?BtON R?"&ߊ/@-ّƟ!憿=N5 –GJ~<=N$bf96t5[ cRSG V-絲"]Te^/XqJkϪ`y`JIRc`L=h ֻjAC-`qթ+l+ YfpJUFFxT!j=ֆaVS,*1o@Bh7i}Q#, {EQk֥BA 3KаWɢwdŽQQ=1zQ.[GuTᴋƎݱ w٫8oI$~r:.Viz皋eN $d'fat7P~4֙Mqz탧,%eM*u#x&nxCxcfzN$ԗF'iɱ'm#k "! \< ]:tQ1)9;u?'NDuBjwlW(~(gBF%qCd0~*K!z1ntjP?SWڎ&sWZp|v A'KC*P<tu0@ ,\ ZO&f=㲸* /ok 3O"Uwhwm νPgW/6JBn# ?i X]HM*_(3Oͣp o_C+Mu9mS|5猦1ikW',xLgZL~<3 ܉cZ&vQ#~>kv̵ؚQ}E+ng&ym2![l̈́$T۪%_c<HZZK1'cVk|f &Af&),+TB?L:GhVNOGn{ 2k|Xd:?`E w%@m [1iS qSԆת,6}-j 1#WXa#KxFDB"Bgp;?3d0%x^rIZK7!wPjJ\$֍aݿQqF[# %(ʧQU$29l :}Rpۧ?S^=5q^HiUp^lckZ bZL VY.]pɨicM~.+oF>D{[_2JG.3ݷr_ b`i?NyϷ_^Q/)ur '(®Z3u"ڣk=uKhb .MNE4AL)0EηOa d 8#ƫ,-JKټxXܦU8wnhM_ [. & '`!%@(got\Qqz$a}p +Iƕp/(Uc8Q)aK \p3Mߟs7^*+q0ܴ%6X=: oTst5>mk]R1 <=;(Z 4;؋V d0 JǠb];w^峺kWn%TED#Fg?)SzbKNw5e2ّuF*Sq3S Pe.+aG>?, zZK* vе*$D 0)' ZjؽK҃wzmY# 2|HbmࠜzQ.r{;v-C bAn`D qn//~ؖr |&re!ƣa9Z75&ܔ^2Y  C_tvbY)D b]a)專g. "iGM& \է3'Uqy=y,-Nu\ɜF),E2$"?x=9KVSBRR 54z_~W7K*HI0IoCtj&V{AZCUU:uI5oZY2 mɊ Z$=aֲ(nɄձO"Ҫx!WI0$#%#h]UEk1EԢܫ GX20Ťēb@ ;c(9> 8535+P )s2&~V)<ޚ &TJaC^M:Kybτvi%4׻0q2 egd{ cٝ]cAtcQMSQfd*<{܀*"$]-}G?v̡j2dKڝ̔yVf d)dx{GH]Tה>=Uݫڼ=ǐ Ԇ,))Y1[H[ɩfeͲ7}.CC}MF4k0);>cw^CsJ0]L~O|h,N{NK4Im*'_ap(k1QU;r,xDy3p*^>s篫0dV/Tt躶K%Y~B3lCg""@8Z8)OVɔO Wkr/ÊUp*CK8;=azʖ+SGseph0^Y3.BstfDRdҰ g*e٦ʆ6vr]3v8 iZjo1ҭ^?=l pNht4?:Fv* M50R9dF_J'`{9 4d'=l!ӹxC%|LO):?"hMG g:/R%x]z?:hTbg\Uuve|]4 Ď>$ #`ݝ:xExF>5dZͷ>HwfׂuKe[,Ȱsk [$Ϝ?zTQFkc`\0Nj'X 0l^+HYwí )td3 1E>'uy.ڧMhiՙs?t,{1ͼR> zx}su-)"U(([i\?v *ygU<z}DE# `SLD"ntlaAKJ}EJ/f {%;l"w!i1|JQ֔_5 P!c3":0O2C6䤼 Y>-u6Dm\[1]QQi&-UR"q]&~-[pI}ޮJA Q'RKWS砎T1:(8)VeK"./CV}{b1o{ `#Rih?ּJ~;G_P ^02±Zq`)H-;yx*5͌sr,,¬9I亰 4~bkEe&%coR=HTJ35f:u2G{2 5ȢngTBqY>] 1_o&@X}ktL6Z* ظۑVdbҊ 4hCq} L/0A } jXGC)[~RWb,Ɂ@}1^| ylmUlTم|}eƠ?˘%-+Z[HX?ױO@Y4 U9 =Q˳z?+Am,B9)rU%*.wƻJ1:-\]v%}<^[} $18:;F-Ŷ|-\3m6*>У0AMfheRrisXHzي2BKiXQdښ1ڇ:f2z hgY <#^+ϕ5"O::9k!r7wgbiU+zeW$l1Zh,m'R[o ǽI]v&<l|R oKw$>_m|{lVWX\3/Sr #\SiI1H%oG"ȅYwM/ʄ]S<Jt:¨\o;||I^ɽb#%}"Bq/+X[ǁJF+ ^YtL"4Ն nfSqiC{r? ,g`CvmONٷG9T 1 zPA37 >2ҵ"X[p z YyG_k;"WM.~ASPGR!..$ N[kv>8</t ;ʷG<\""(\w}m\C2(9ٖi5PmmSH*Rt\qj)I1H[ks6y- lo]cN?x/ofLKɩummap](~[z*nde*0-Z8zNg#3Se7e"٭6? !mEq_(W34𤂃`jG ʚ ŻShd4zp)̨@H(ò#g4aWD\ y(h( Y{ 6(bxYmLc㚇l7{TJ\gЧuo7My F+Z,8(EgRY Ɇ)R&J-"'H4Ay8о^dTAeY rێY7jfd/ʜ%L`{Yq}g> f;3?89gUYKY<J&/Y47ٕ&ʋ,J>UMrY>H"4,1xE jIM|N@.嗭 VhMcv6V!y}PfE̋_~v?dPG,V {+i<5`^79Q_B?ϟ U0FT!64u%p+S%uJ빏 s3(Jص ,`ȏNA+0tUam،JBaZB5e`1"lգm[hۉ^Y"JVBkj ',Ľ17(bys䖑ppq۹0o 9gpAəo<(^Į>n)Xrq3.)lmAe٥HUl5D@5DM93orJȺ30-qU} ut$! x iT `EY vynYQEm,2IZc(DHyNfQ+. mw%Lg6I'9?tWz?d5KϺsɤD4b8yf榐m|XV3ޝ 5!N[?uVPW@ /"o_Co& J =  O̎G;eGFJj](F*$Z&˭62.bm51qut tMdi#'ek޾Rzr(42NT4f?pHo2!ߋW?}u]*VEQK1" GQBJ;!eF&zֺ}9S-;^OzWvUDu$~JzCa"%-I0Ȗ6UweTq|Fk5OWզ'- B6i:-iI endstream endobj 1715 0 obj << /Length1 725 /Length2 15995 /Length3 0 /Length 16538 /Filter /FlateDecode >> stream xmstf-tl?aǶm~b6:ձm۶͎{Ϲ|Y5W{MF$fgadec(00aȄv"@.: 02avf c5kC  G9UɅхE@ 04%rq-bdma 0:)v;c;[D/@G:dEUT*C[?m;;Ou4Xֆc'o`ba 0Ynv&.?hH 0bm-ghPػ8v&@G[hh򿩆67C1mߐ;D]Ձ>YͿS5;SʿWQUO+,jklgbakPvGOCG +ZvΎmiYBBv^Zf6#38:m%?f݁ưkv͡eԳˌ+S-MpTp~^H5OLF.Vo)օ]CaAuN__ Td9r(Eg6(mDڎ[]xO]4kRpm/yAYaVa;I% sKK69d ,eYIAEI[& R5F>7ٵ&@,>0f5~s7oL'QMa%= n`^d+.Y+-sh-"ʬſ~LxēDX^93;k)+x\oUT4K&2dfj|.>Fٍ["fݏ.Vج{btà˪N$:_zc/euoFD`ϫEV!e3Uy^x5^2YLuDS[v^' l"_e_r2"XQ-nvG.6KB8#{䠃)Eo)Y"9&C[3qVL*,lΰLJ4.hO:He;7c,:0LbmiNe% qҳWQYaT+f >TsU0f~z}w/E*VDVҴvE/], 3;/,SHL}ӣ;(4b֓0JJY^AOxZ%:Re7֢_IPH t?ߦ2.zsb3E3FB طj#.\+U@D%T> vl6-"PySlJ6{&*8`-/?7A_:~ >TJb+!.!*_8SA-W2n)^uש…"@liwW,y ' s, R\+㱵'yb]*wqQD)㉚m/Iof{o0 GDHTΘChd х.{T8*sordR@V:QR["|Yp`kIܒ,#;@`)H<'3 ŗVN[FtQ>9U$od ]tdm2*fGg4Wuޜ5dizNA  qdsXS*t3QfrǨfi{A<N@ q3$\,Z0b@X`Y^@&usR(zBO3O.$6aĘm!uB^T(O\'6g[:Ah>hgy~g=,k H DIH(qJf$f yu؛t٬D1Ѳ_}vkzZe R0H%hL?>Ifo! _\vޘN%mNπjDMݲP6P-9_~=\q"^l.ʧOLX&!?N| /r~`:;DC4n6B| kXߨfyǫ L*x Sh䅂s'ɭ~KFdh0bgVyq*lؐՋDCETZ<0S|_=5NYY]s9"jSr&W=8>Gk"CQsIyxr * 2M55j~rc,7bҚZ t#q/Bc <,}˦LY$itnA(2+$)!Ccu o TQn!lo5cK f #}d?b_p)H` TH ꃯq72-t/5G:b:L hTs&,x@xtR^n}*r)tIEAQC30T:}p.R6o6{F~>L"ړFT ,;'Jө1"3ı[UfPΫ`+6fVG_^Q-ƧoQJGg dѪ*Q_k1Ϭݜ(޵ivfA,74Kn9 vNaYVZKUoD_kh]ķ}## ;[cq+!,lH+)D%縙eSp&1 p T]ensu~;J1&,STM?'sY'-MũKL=(t.+O?4zZ(5_6o=H 1+?(D_~v(POxo;ۺlm N~jpcIzeI۩0#Mn5r49=W[mV"q6_txuWaBk_0kfWN|/:9Ǵ 0vZ3_8Y#$du=Z[bM^C"id-1M~3Z`0,*7֜7w&]܉YpVr4.ב X-Y_3LR/Š|Be)V/b(QD+ldv2oh, %F0ޠl|X/{[)hz&>حؐI>{xՐ%Bմ'׽ZC"C N#% Ԗ='&ŸQ,OUS@RfxM[e&֝Eξ!@&wb|Oxk @*'8k"Q@i'QdK%1,ן*LDhvgd hc+6܅&>Gsm?\z3qCN(4)Q^ w](Xƙ6 H Cbz&Dqsު#kЭGS\b%B"aڠNUZe2'KQb=T K /3]~1:nt.eHH:f)%}9,?M!->}*6MOB{(O7ԐZ%9П1[yI{r,XT=&5MvJ~#Ƃ8@3n0:F4hudbB}Sk aa}H1[&mcN_`*Yut5?hac5E ۚ,L _Gzd>u5j~. ǸA˜!U䔵 .)Dq馳QJX~dzJqMݸ|1?+XtcOTHݏL ifʤnhIthTLb[!BTApf`!PIW5P/dlV4Zhty)&uRO>LVڷJMA,@$Jk.?N9ݤv,*OS;UPu.%?ɕE5yh!3br5!-g S8)KO#֚zK#?QၨBs:uu0-έNq΄`ctɻ ^FZ4Tt14P.HVU=K0u&"2+uzY#9PrhC /q;ean^j+yK lVV5G {Ea-HU Q" OL!f߮Jb\78X@t}б(:OubsroF @>2Va8)䢖 MnS Sބm.幜} ro,5Q㹣~:eګn zuu2|ǒn```Fl"exasJf(6)8 4.i,U0۫/Y~ѸZf}UX4Yj[saut% a3v96OݒYIywfW.|gZ^#2]tQ`q6+w9巀o%$)tbXV!Mԡ_k`oz3Oɼe-aym@*@f9lmqWН@PödjgߴޟVNxt6_gGgvqkOeGOMφAp< ,Q,a{;{̠NxAV;>#ӜBLy/e8r}3MU+#U;;d/*ޞI =JR>ē_L.mָ%uM ֐LbĽ'wɳjH@A3>ZbJFsi!7GQn?1?͵>_[g>9اdөt.)kDNB6.dxVEkt(<@5dYDO鬲h`[٩SQ!}Vql(|n/6FP!H1Rчd$HcG>r04h[^ ,%lXJar;" mRD 0 [CWAiKidD8s2#okJ[Zg'\?-\=Hr&\/~Kbj;j9c™7cn]r^l}"tj"_HT'OM!*;T7էAԽ=Ǜy;0u'<2c]א'n|rܚ׺Ia}HTǡ'?urC-?3e_@hX9QNc?#Igp rVJwlG5JߘBiBMbqXe sj5Rtoxm #c+Z+wy8,:]~m 31,X?|hPr, ԬK|雵j|ߖF'?n/5aQldWPkY&̜):G'PTIIM'ŬcW۱Zn ]*@Ԫeҥ(bsqʛO*ݯGBc+3h~P 1`mLmA "J\".l;dG |^WpړHBpuTHIS*ob Ci-c&sO1l>^t] +~rdώScq]bH$Z.V 7z'>1[;ؚZz&%vXXF O^a8^=L8Wm 8T/~zB'MwH67(~ u\bqK_ !ڻeTUŧŁE׉WW@Ĉ:h](A!V y7u'3b_[vE{bO~X*O;[9POkû'2r~_5M+K=`0$DfV,wG xRJGA2} B0J=젎s+0窑/hR.: [ kň1Nƙ79Tï@(HαJeNJVea@߿>":c6si#\d!luPU~ydcDƽJaLorRb8dN3ث?{޽5cH.кJRc>Qe DHf۔"g4c-mhkasӃ{&Kd.V&;Xg۵j/6hqXN e'>`L%WA1 '?ŢmrF-4Wd xuv -DžsvZ<,g3jb׉hb:2m-WY̾͘EY~:(sj14w9q*(su摔ۅoxNVw"9Na)HpP‡UREu8u?$zNЇh7ߐ$Q&m4x*T$0 ӨzlEx]t@u?> eD-\99bbxݦM>@F%dfЙ/!s3"V587x4pD4¿ AH>g2lOU̖6}J3 bPd}63 PvD/>H-:,=g+D8iM5]py{}b_nk=K.\\' T%z6Gz~[1hl5mo)m1ӧZBas6>zZPbOBTfXx7*|/"}$'F,*k{WК5_vb"b:n(Q8x܍c4i8,Ðzu ?og2meYAaƯu ~]h7 n/F4FG/=AT7֖&oB-Ζz9p I!=rw_*ݍCM.L|`zD5NdV&sq0s%;;,4o$tRgVFiQCl#֮~T>^ES`*N̎I~lNT^=Jb|T|aTPUz83ĥQz!_טjIqzd*:-r:zO|JX Lb)z+yV;:G1WnJ,WH]C~N Ӣ?.#f Vn"Q"v^}\v/}$OOZ|-ro&l)_^ /ŜOV yKfj=p{˙=o{o!dk1S}* ԁAvVU_-\XGЀY-@cB~9B1ԛ4 L?D2v͍˚P;DMpN]B:Vr ?יÌ s z+[CKRSjJ`0JCsO5j&8 a0z~;O`;v|`FeJffMw,[Y~:gߎ  !>G/JVo2({v>(5:x*}.^ ۿYc+S?t߯ǝ"%jg@=l)8cdϼɲug`$~jjI^Ϥs9)ù=1f2/w"pƘ(G4PXG_BX pLB< )BmoUʗ-z<0泞yӉiARϸݚvY0%SXmw{L-L8a0m*3 9eYhol^ޏ/"*MX3½A̟ԓTKHݫ#jJe"Gqo],ON!O畦ñ ɡp-DExXm P8 FFځDiJQ ;Qfοx Ey͓0)r]ؿsmlxYe#F .``Z Ϟ|22rߏڙ1G'pF(71Ip%3Sxmw[G!jX "=烙噟v%PV22\d-gA[/AWmq_ڽ-azf]k>4%p1=UQ/aY񐤌zݛ1Mb(7^@ny1J`]N) 'q:#l)< ,R|n6| U&(Oaŭ2"nŗ}ޑk{eJ5 FcQ]c>chv(Nwt:5hquA_K+?Dj˦`[,fW&\w+Ԡx*2t.֦8R u.QQ|a%BcHF9֖?5emʨk4zWj'\RJ41d[T1 e;8|\ѳ͓#GS B~$g*{QOZ^i.QܬoUqBMGU ,_4u[c;%xBf)(|zn=jV ⴝ\n4sU"rܽ)Gº*}u~73;Q$EXIl\S=dS@**(?4wfH@fVRUNxra$6=VAA0>Ep-?ꗞ,,,0z$rX`}$.zX{r GCV&M6R CI|b.Rx5GSJi;aI?H̷~͠K"JRS5l&d<3YR4Jӛc|}læu]rdwRJԢsر=~co'2LleWbEoMxF(0!/(uM ̌Nљ58h2.b@hUHm&BmɷCm.щs FA >|aVvOF9—! ^[7%mՂb3nYj@}*9 iF?UUUTA< |t@z8>uSDt$涋109 DY_Eݜ@&"1@4 kd2Ӓ 0~/K,W৆#ۮ]*|˘= h۶79vd|ch.4>):G綵| -$H8KIRrt M0e]]Ϣ Y+Z! e4M&CU"ScWy 5'h4T xE? "ާ{(+C kZl#ߌG}5,wJ$vf{_,oDBCAP-\{.3)O룽E. `[HP)>9CbA IPyRZ-LϷmK-E;gK_kje}cH{j؆3?f>mzz ;4 5Lۏ)침]R iOܗ9OZM}b*k|Roˤ0x963PQ`|PXEXbˆNORIOsW#zNV5Y[xoS%S0,d;xn*Bå:1WOl4Iԣk# Y.2PxU[ﺓ(e67"It} {zfRd kO ;v=ᬸo UcF6ld ;anԹ:rdΤO5=zl)V~"i,UXp??=b/ aǸP0ؖmZhe a~\7 #` uH=ZFlkQO/)<>Z2I[s#y' ][I\(}Ha!$o-LIb =W g6e ^)dH` p`%9y}eN+IiZj+D+Wc8~>,CUnZ9Jͮ` hÿz%gM1)nȕN.σNWv5NX}atTWx[Ny3#Wy(g7Yl_+X}^5$BE+{^4K^B #̶֜.K=S;/4J7 OyUCe@VcWCC Ff3 v L:OҎ`"[iR ?N4r:w [KFbYz126m'{P $4Gha mLZUĕKғrfbUӪ2YFjiZ,{o04Ǯ^'NlvȎDPyZ+Q,) H+OBW*m&I; қ5ǓTVØja㦛!3=@+H}$9\ʭY[z#MW❀VFtR" ?r;zf4rd{'ښyP}:FrVshȎ &H" b҈ͮS\HNPFwi"vt>1MQT|iXYe qkDj@~iPj5jU^N&H46Y0ʈ}'.}AK>)FSfVHK{tӁPEpw!RX8<tn.uAJ:]w#ȵ&Pʝ%0M> stream xmreTj5!0t -)0 080tH4 H -- ])-r}}׷ַ?:p{z M AP`nj D99UP0FBeP'  |NW @xp{=`WWA  ]g P50h4P0u ]8{" &xAaOwcSeu}ScUaSU y`C!H_ EpUD?>pAG _/ߴ<=N^<l~%OzPt5m~Cז<_O:Svx}#$j޵&GU)lꈩWV^eަV ?߂:Xwe~E7yST?@1 "ӛ34rG hB:+ډDA˯919= 'nl (qG'^#4IzTdze:AE76z~rZgq%UW]@Uf *Љ[&4%ݓ3]j g)qkQ?>rGXME9R>[R(3NtF~WPHaEk9c/v]J *2C[, aYWP-n.j$mZʺbBqmSwτM=,us?c䵶mTi@HcYr;2:Ơ69Aj%|+:ܲPݘÓEeH:OP$u4L6}NqhOiWkw2aH)X<^?pSjat,h~Li2oUV hA#JVu| E'F"^/p< x`v.":pɏhH+7$?FY4F*֤fO.(UfuWrMNw)NO ƒÃJΤk3bNMNu%$vzp?)ʋ-Uɔ(F m޻x] iagopDW,cψ?e L0uT}q/p}nRs뭸OI&] v7,ܲaK暗/LU0iD[RNdSpr\\LQļ I9by:K4Ӹ eLP-zX}*@ J %2mx0gxrW(l+QZ3ﲲf74+Ԧ-~#=Z{ukv$WbnbX[{Œ9|?QsnWd>J-.`RZ 0{BsU iVO\Z Dopկ)NvagϖnfAr l-׆PRQ;7)=Ev&BX5I2:~%`\7ǮޏJ륓wO $IʩFҋgKR$+<9AެCP1d+2 -_?tTO/aX?A^E+oߛg5X.?Cgf{"b>-{5s4_i.>CgV7UϮ |P T:佰؈@?mu]&_(r o%O.Q}[ƪBӌ"04֟j'`UI_(yAYLMqyf%LeYux[f*#|n `d$N4ymjn#N#_) D*hFLqEsZ tlx\9TGjٷe^&Fb(dLe*Ľ ш_+Q}D=7kFJʧ1Yxsbe>?&:z7(![>1rRA[º+xMԧEv7ٴT.{= IPW> ,5jG_NO_p$il_yJ~6VPc W<<%qc ;zqu Dƚ?qz!I| >Ɍ{gA0?9@e2C5?WyK.4vMeqwM"+ t8e 8,Nj#Np4YmȄסۙ~9n>^O޼?`:3O8o&9zhg<08vD=”*vOCZcpW*~ хN~C-W5! Y-7M4ǷK F/ f5EknezWNmQ`/ڟZYNuJfݻTmd,8?O' G s렐z^/(Ʌ 'Keݠ0zW]pNᑘ0G^= s> \[ajš\H/|joXcG&g|PA.{<9@ZZH| L^"=ܖtOۘD PV]=9v^|WQ2jYNʵظ6i]SrzpoIϬcZv̕83^t)t@k;E# +T,cJᗾΰIKĘlN̠wWD̕55e  [w%`x_^Nr>=5B9?q5]cJ<<4.VlM՟qK5UK8 2Z%]ۓ{O;<[s;!nj=66:A .*=K}[;hC/j6C˜3J,n-2oԿa8a}7.Exߑ6]:4Iɷt*= ?.65q_)T&IԼ_gb`eb}w$i8%WJS#/j#e07Mg`>.O|]#j%tF.,%:~PNyvHFƥ42 ntXI%jƘk 9ISʒ64J'Y|4"RWHY|?QqɌ$:G86mDhWJ-MQw,B| ?܋sN @;j0/ﮥpuP()&LZuhȊ|.JhILͺg#jk#5a[h92njE%=*W^yxQYS/2iaTqs ޻YA=mNZY{RK+^lݰ5=Fh;M=ql6XB8ruaqeoz| endstream endobj 1653 0 obj << /Type /ObjStm /N 100 /First 1008 /Length 5719 /Filter /FlateDecode >> stream x\Ys9~谪p!YvWRE%)Rmϯ(UEҤ왇 A _) BQܖPܡ GRr)[(=P{|+s*C:`# #Ua)wBT֢FM4j4B9IPʠJjVbUDjPJ8S(f -YWhmFv imM E4A[Yp T*5]8[q.>[J牍Hp8ӆKUZR>tPMX(@HDi°$$XrX[ u!7iyW(+B{UPi$U$BrpزVMӮHwE8AL`8"k!lCP$thI 74AZ1& 0D I\x=xC0B;Fk@{7T'i@.BMzZ%hV~)I5xK!@$1'9$I6YIFU%KRi+MN`a'1KX-Nm8Ш'ą ?, D3Ԩ&4L ->{i'&Uo߽/:2$'+đ!w0Y;#G`,q`g`j oG Gwe{ȓ\\YqD`s*sGڃM'矋򑉦ƳР^紐(x?W8\xVm4)‰< Eކ#ДўE;sOXúRI0U# o'UH)ei)iX*Z6_aDP!ڙגvhp; sC`׆6e.*cF+K-Q&,n<x9Ԧ=Hx>u-y#EEmZ\z/xlwDo$/w ZU@\(rRO)(4:y'9ve#;&)-3!3$ o0J1$.$nʌH;,C=cdVX! Ŧd:VOt,>,eŰ)8=d>${'b"&@ $H 헙j2/z|H憐+þe[z]5'Z ϳc٧z)̰IoqT4N;VHF-@5!#R ۑgY"N`k!r $Ԑ2)"%D` q N+*6GW0:)8~\S'dǁvk- (O-+a.>cC^/dL2@+tecg"n)* CXEփu([rsdkt8<oT#lX' o!#)t0T^1Dͨ_6ҏC,e+ &Tl X*sԯۊ̫kcs% 00Wʼ_)eplގ;;6Kd2N[gY2+y)Bӛg=l ogb4>\!i&j˸D\u[لoa,Hɮz=-,wYê4:# {)mC?yN؊'ZU18QHK> i>(9v|hђÛMaTpA v}!*9t% RU`&lK=N-L(LKuHӜ[Ǚ;뒏 S1 \>@D:CVקbyTgq'Zo+aY#Z- xNbX뢒h r v +Bĵw,X>q r+.9 oL:4B!)dvs$'8+A UJ,tS`#.ކpE yIB.O81"NZơ4|rRZg!Em9؆T[1lܝ`H"N8Qjwo|sl8`k .)Hyeݻ*0m^|ʐT;D`>`|>΍b&<'n'⭪dN-sT-)('[ 9wh5<ⓔ ;*v?lZqo`G[ӅKWq`::c oN\YVFLG  4{׆Y V|:Ϻ -_ <2bg5Sfΐ6BcHh_rFo>iyU\b4qʞmhh0d!-O4tL|*ujao(;{ZssvśpMjyL7wߑ^&]0iMRO%_6A G [%dcT_ƨ3-Z20/ V&W& Φ_꿖q9ͦ_^/؈SPKQ]*3WGḾ.Rf9^n_˚鲾]|Tk9<דa]&z6tR niy3,\12m=M/!zV_sJtAh9?/h/'pN?<u6TR)o@:/Jv:@FO7YRK#` Rԍ8۲ {xzO6cΉkov@히U=dy hy:s\]Fh5/GMnk8_.Yju<\uqGI%8>xvFJzl<)<B8kEUfrfKlmŧr8ˈO8mHqC,m! C!0l.p{Pz|~K6嫊}$*soU_n[1[ 2 TO7ltE+Ǯe9Yףv>*gr1Xz1 'r~ a!j A'˅bnpO~խN?1kgzR>(OOʧE;q|S-ߗr0^S.X^%\S~*?Q\g\8 a1`LǏQQ3b\V~.._vq=?~-3IR.mr]*qG4t_E*\7O> m5Le%ڊӄ;ކ"ÔǭSX~>g?5snt4gxvryN\S*dB[>9*!yj |a ~:\L'O-їx_t)zr"ݑRna.97/ )-N'~,?>rFSIyۃ$WӛdM.w\֭Ιj|K^r[< FnH|!i>1I1r4 FQ/C TZy_oI!1m<6*N> stream xڅO )t^?I0$ 0$M0]# ?|nF僧GKc%-KhK?RR\}YgYꒋ-Mjli)/ˢ%[,=ϴ.Gk:/GuZNS=4dͶHܧ1%[\ k^ :dsw!Rz(Ukhpz$q=nVKd)]|B.Wq/!11qL *bZ "9GT5/kHO!_uI 3SmwV O ˰O] X%*աRX7 c%p>b-|HLV)cx4'Mx6I8y`9QEyab0̱$nHݢ )'i!A2f鈙E2b!ӱFf2`l1Xᆵ灪9ftqC$wqdK] bKe{ݏ??=aU~/>}ykxo?ށʻm-4*|͛$߼Tf2?~>?vȈd{ɭt]l?X厡˱C)cܡd+XΗ0Vbcc߈Y=&!^ g)r GD qO$~ DZ='F. 0$h;ƹʈUmOHXVmfj O%@@wW8[f>_$u{x-">;~ \CIzhهK\$ts^KyWy,}ǝn_N \xc?{|,D x>0fse0>fKc< jV.w"RI+Po.bʣ}X۱]&/q]vdf(~wo?ïo>,](hJj\{ ]xo?}Heۘ|ǛyZUsjc&Oߞ?}˿'{'Cx1C*;,شĨ ^½DMLM rmL6'0qG0fBP$Mzvv&Z#H5a:ŷ7vj#UjB~m}o&B*dR$O҃:u}=MAPB& glbN-{T6.mpLm&6Fl+cm3}K6׳Lo(S&Bšt,2L>ѧ+3(eJ {Dϔ)%5!Y@PʔRL)ap*R*(e*\DoRSB)UP*RB)UP*t1Tko& ܩJRjT(J.0<+kT PRJ)5PRJCC^ lm%~(NE(5JI@QJ2zfJI@Ib@I5 4֨NC Z ޥR B 5kia]j+[>BMީ ܹr3;7e_ޏQ\)Q%XDooTpFWJiTpFWJiTpv/[ t-ԕQJc'G7JiTpF7}`>_DߨF) nҨF) nҨN(ֶ@oTpK TߨNkP4M6' &j(hki|/Dg:X#: 526QU0T=Rnc lNm\ƈ7Dv^o/=;4"?@~]lL)%먨jLK@Ynyj4qP(%c5J EJCAUeNYX$*iJ$g9ꄂjg=8~}ǖh[IzlRBD)ZɂjI}*P12zl2,TnL: 2wE[C=L)[P-SJS)%TTt1Z0o&2 e A) V ] hԉ>[D:j+R $T(%B ůJ]L (UJ xwPRʽ@&|%~}*ARJ*ARJ*] anCLo+DSTF))(5JIAAŠsk}TG4F!) 5 IQH HBR@pdI~6]~FB ``$TQ7(IͦG4Nt2@t u8a+urON0v>)%%~dtP%^N*h>=9()FA W0m4mtLm)uvo)uw6Rg'6:5g'6Jg'6JK;mӷm8 I 4*[DRϔRϔRϔZ5ϔJg'6JZ-om)L)aq*MgJ gJ ]]<+CSRBSRBSRBRBW.t^Z-omhJͩ>4^(%4^(%tj^)%t^iFZѓz3g H 5(^'d=M%F -7hhIDC据gh>$o&BBKB!%uШPH88%zJ6W7чԅRƩGpa'g'7'I3~Mȍ?)^avae;ٰWزl+lN6vY 4~Mx#ٛjN6)8ٸҨ ;v&ov0r>PrJilR'Ɇel'kohR' ]tlrJ6agD[D"FǶap eJB}W'%4y?] KlZ~cd4زnGZ2 endstream endobj 1787 0 obj << /Type /ObjStm /N 100 /First 1049 /Length 4160 /Filter /FlateDecode >> stream xڅ-_ChrԚX8bGu!4tm=#b-"G/LF^3c{C.OgS(=G09 ͫ h-MgR .gE#-}VRNe@αu-9^mjH=oS;Dl %w/M^_.goZ6KeZ .go*9Bgo9|'fo:78|c nٻE;=zigoЛ| a6-$[Df9c7jiR|ι#]kCҿlߣcm؎ 6{Rp9{^m6?Kc`^i**.s1T a) y|5f$e&Uj`' C)9` i0=۷~X{wN9M6l6a6tWhw譢__n~jCH+i`2f093 ޽Q^^F(MSmP9oL^ \ v=iĀv7 F{*wj+ja6Кbs^Dr ^7u8ݔ5^ӌ: >8 n.p V oMK%hJ{f̆0a}I'>(_2B߂v(o,oκ\@CI Z:3̹1K$ Z &P/*t,aH2KK"mu8Ē(\˚X0wq߽Y$eX$("EI,ʰHbQcQl3qK,ʰkbQE2,XAYxVg]Xab &cX4â؄1tQnRWe,tKg,tKg,GϺ6Yf7YIIeIIM*-*ӛ+yq[,`Zr(+$JoQ""m [=(4 $mR[f`sjqNig{ËL֨kQ(T3=iC*%n"+(/ Kπv \Aq)pť(.hťQ8Y4Fy)x5KQ^ ^J* o2^Q``3AJRlvK/ix `.LAA Jl9f/6.jE^&O ;?Xx`^"3-(ѕw:VϗJ*_ ?_+rgyO+^>_+k˔vu®j?V?(E}+_H\"UՂ}-}-<Hl=h +58_&zVA.=0m&P``B%A)*s FXR%`J)XR``JH2C_&CTl kfX0% /*(`F53 %qdmO5CҨ`Nus S`NU _^NyurʫS^t k|Iޕp5+(\Aq :ucO5/k53؛nF4DF%_<ڦ1 l66VcSWj爹F7ln;lFm6g6xRl5LuE%^S>$S l6F[",b[1׈7f^3 (\(0ftp5% /*QS`{)0 WsDP9b o2)0u9՜CjN WqQ9^u``<`p `N.Y4ȄF^ktApF WktpvyKDBePZPZPZN- JU^&^^^VVt͢l~yFyIÛL8S^zmP^zmP^([mP^Plq 'yAAAlAlAWVlۓv`"T%*Cŏ 0[|V2a&vӺ΃DK[i W 5ؔ:lǤ?[ o2nMyi 兴ۅBб7րH])0ݮnWS yBVs^k{ËL8!vvR`H])0ͮt o=}ƪ]EQ^Ⱥ(/dn;ef7B \֥DFq!vtQ\H(.dt k֬ o2)/$nw I;%qxIËJ$nw I;U8Rm2_{DPY5⟨mDi,-O 3",OgI>;!៨ϲbD}'l_ψvEAy?('x :uVqV)/iw ^ZAP^+ A? 兠,ϯ|?#߯1? ŅB̏Bq!GT_ψ Q(/(~T A?*I٨JrQ/ixQCԏJ!4#GR3"~=#+GC(0D kQ%!t l\vFįgD|E:eCJ=ͲPS.W=){;#k-K쵅ypnY(- [zJa-f[ݲ0 eaIݲ0;ΈͶ0 QQ`-nYݲ0: [_m6)0lS`:%8zV'}x$՝Z813Ȇ[ b $<9O#IO #3J-T>x!{^o'7{ˉRb9p@%(u~raɱfEV)F&5zlv Pް6Lqf-m0a/beYq{h@=c΄aARCb 6`1tHglo㲹pZe{ɖ >iOe=$(-K.M#Gjg7ٛM`$LSRSp kD}S>>$ˊ=>#ΊPJuv${Y^ծo)SZ}dYzo", DBҍd>DJkx?wo]J: endstream endobj 1959 0 obj << /Author()/Title()/Subject()/Creator(LaTeX with hyperref package)/Producer(pdfTeX-1.40.15)/Keywords() /CreationDate (D:20150713084224+01'00') /ModDate (D:20150713084224+01'00') /Trapped /False /PTEX.Fullbanner (This is pdfTeX, Version 3.14159265-2.6-1.40.15 (TeX Live 2014/MacPorts 2014_9) kpathsea version 6.2.0) >> endobj 1888 0 obj << /Type /ObjStm /N 71 /First 709 /Length 2648 /Filter /FlateDecode >> stream xڕZn}W1 `Yܪ̃e+`[rFx\R;"PU$b}=sqȗڐ]Q86y(I[N?;_bGQm0IkGLE[ޑ-rVpy QȐ\CBmDqA|>-vq(mfq+潋7>i+4KD_r).*. 5g쓢ejf*piP|uLҸQr,5dj"P vk:ĕu0ImFStL4! ACMlmeǾ!1R'qdh;κl4u)(9ɘJ 5F8Ime$kH.DS\[ZWCK:0ՠ_IuW<b5P9W.nӇ_O ck][xH=˓.[j/G?!V괵hyD=(b1zJW9^ ZP4}M}d+Ypu`u.'ԧ$:z=D3HЬX1-OPuYzPUu`ӽLºBt'BDV5i1ܹ|q~_tIOݩrdwy;*$Y!o^]v|Ӫ8Èc@z4=n?g;wKBM WG&Ye×G@"׿4vo.w׹a{y?NksCwϲ6*tGK">AdNoᬌ=lz9"?{urfڜPf}ݶb'o/|~Oj3l˓r1)v=yqj,]||B c̊hWe: rH-5]S-5lJ2ҧWCzTRٸƜZf2áSym\:rm) EYui(Exr;/6NvSS\4.z9OL wnn=o96Ku}{zΝuQss>7u/7 h4Ղ߿~}_o篊tj6<F@pB꼶'ϞGA%J dF`)SG3.y^01*U\'JW`kb&J`*! bvu]bY`*UF3pԏyHm R2]~H!eW1 H* U\[B]`Ul7 1W1ppdLn;3# 44{u/:$ Gihy6<B޿-emD4w.B<TA,YfӃ'N\+ (X"KO :Q^:AA 2ZPTr"CU$ 9f12$4:&G#&HA9$!c?&/􄂾1[ń 3#XARʩ[C[:$ǜ@ PTN2/+ =RPTrVPl/V& 33HAQʙ!P2!1C!\ևP:)Ff! 9RT2el,Ԙ =e)sdy~,0!,f! )fA~ 5e__dv,`FD#Ɉ1h)PM~֋) VݾM4>˛ t?^\PaDBJ endstream endobj 1960 0 obj << /Type /XRef /Index [0 1961] /Size 1961 /W [1 3 1] /Root 1958 0 R /Info 1959 0 R /ID [<5638764D8EC74D20F8576AED029B54F5> <5638764D8EC74D20F8576AED029B54F5>] /Length 4398 /Filter /FlateDecode >> stream x%Y%Y{\}zq0jY=.!> GԮ-: GQ9n}`T ǫ=Uk4QD-bq>R{6zΫRH]jq&^˳bqW";ΩEoMj>gk;zOޅ{WYʳjo>'jέkO!֨ Kli0X}  *;Lv 81입О^z腦^z^{wVNy;zg^ꝫwvΞx;gz#HL9Ί~8= <ώf&F3BKCv¡4XQO I~?X,XHsk,fԍOeb_H }|"mٛ m՘^~ $Ƕܳ&^Ĭ W/\p W/RmkCI#0>"X K`),V*X a-6Kx76;`'c> p(pN)p1؜3bGp2\p M!1܅{pCx TSO%>TSO%>TSO%>TSOJ^*y䥒J^*y䥒JJ8DJT*Q>f!vZ8LSIN%9TSIN%9TBS M%4TBS M%4TBS M%4{_9Cg^Kx w)՜|" ~sQxJ\JrEq_xvof2 ~ ^F@hnfP)klϛx(44qa&fEJ~5>!M,X +`%Ր;`]bl_J'QIU; `:?x5N@L,LJi)}KQIs`:kBJ?x!fL{͘%^05Wǖ~x$]<dzj+4\Q{Sgopx߂ЂЂЂЂЂЂЂЂЂЂЂЂЂЂbHbAmAmAmAmAmAmAm[a\\\\\N-IԂ`OZp^L4sp.5_:[u. }1z6wB C4vǠ)|9 ^x S %%%%%%%%%%%%%%%%%%%%%%4or^r^^5iDRJ(ERʭ ėėėėėėėėėėėėėėėėėėėėėėLW^K~x4g(b7K˰Owyb"2Lė]O|I|I|I|I|I|I|I|I|I|I|Viezhj60F0saVhtF#݇Q? Nze\aTů dx%UCƫ!Ar@!F*~U"#Q&qQDa0{@BuF#݇Q7HSx'㝌w2x'㝌w2x'㝌w2:g^'㝌w2x'㝌w2x'ޝxw݉w'ޝxw݉w'ٝdwIvgp n#############################################################?vVoR!L$́L\|X a,%rX+a zal-v;a=~8#0 GpN 3p>sp.E \MSqv8N:9 endstream endobj startxref 495959 %%EOF gpml-matlab-v3.6-2015-07-07/doc/f7.gif000644 000765 000024 00000074517 12076017514 016657 0ustar00carlstaff000000 000000 GIF89a )))85,,583338882821,0 w??DDDWWWIKUfff`eovvwPjl[\[..8448Xwu.OOqqDDVV^^wwwwffai7^xj7?)EQVFfwrmqy|pXDULeiv{hC22DDUUJJeehhbgTg679(/99V7Cv5Rvv~~mmDPIUVYDLU`YdUpfpitwwryciFFGWi?_4w|qDGTYJeizP)v~mQ\Jakq{`^I̓ڰꎤ孕ѮƘōٺ͙ͬ͟Ӕѓ̪ڰ܉⫱ˈ˱!, H*\ȰÇ#JHŋ3jȱǏ CIɓ(S\ɲ˗0cʜI͛8sɳϟ@ JѣH*]ʴӧPJJիXjʵׯ`>}'سhӎEڷpPܻ֭xR[ B+ N{;<xR)Q7 A!@ˇ %hY Q >Ȝe{` '@ ztԣ[^]zۿw]<Q ??;x׿{D0t }P{,@vހWt u@\x&PA Px{3!޵X @ PGY@48FPBPg%NQHExT FQ;~Qm U-":@6<<@i%A=1!SPQ%0‹FzE rHX6TkSA\lDֲj?򄏘"9O "FddQP `P[%șEIX%X[EOcN#YGǐ8!8$ Ɠ'T k _r{  \WTE$> BrD>1aD|+YLԁ3J$ST s,^&_Lגz!3! ɇTH &@88Z E%ȅKZH+eB c!<\Jdُ&kI=b~eF" SOHH"$59QD4p$D(GR$T.)'FҎ=r&P\JQo3P[ (Ђ.3&D /0Nm %т =8&#I4X@ *?t(LX{@5aiB^r!BKd*_{XK*tG 9IВ"|B͇p!ԩDfD#`n6⇒ I8bԑ"ՠZ\H:veS?޹^{ئC F<+;G4pGF|]<2&k5H JR.Sr ]Cr֏Vťy .ڐ{\5!=g8v];e\D)Gnِj q[І&)mM* _H5'=HREҞ#ьB$Bn ihLΦ07"~$nMxّ3C'.HU]c a"0B^484oBڪ\D)iȕ(Nxk.xV+E4C"1\V&N~x+iQn`A!~#lH ߻Ӫ5P(~|\~dǓ"3b !Ë }&`H+]XΕ&frL鄧kyi@t!nA1惸#A0MY5x7!j15@ָoEbiLdi"+>gbd,ȵumk]#Tp"UW!3F3zMnHiSs/}YCZ-<3 N1|CAq{P$p+66#HNLB !'dZ{$ G4dE(~ A(fX YO*߹7OޑgI#r&4%搲A{$x:BQxF}w G#Kt[!0sr<됧59B+HSːrt 3A0?}tW 6Dz" YF]L˟=i|?컁Ǘ9c%QZC=/H8F+]gW AE#H~πk[ ڕr烴 +pRWeh E  | wp7'n@~V|^Gv%61<5B^pdMu}At`tЀ[‚'7 Xk#P _M pP{@tC]Ig~Qȧ|ߦM1Bp(te1z 1~i jC!| Hx灀{YAT!5WĀq D @t>h{DH!|@挩QHS%k9D1s7MQIWhwlz @u֓VWCx|mU 0@!)#qcggsHK)ZP^n=BiHP{ H@D(1I D [T "YJArnVWaFЅzVc]&π~ A?x 2 ߰C |{ GB)Y) Cwg~RepMф1٢)Z2,!w2J&̩P0aAhXt FgiI- I/!bOq HD6wǖ C  QZ@@Htxƶ[ʥYU\.rf΀hϐH1yI@|*H@ )! Χ앩V!1v:*@ ;;p0*:˰H}l!PAa-J󊘓vqMKR` X޲TD*I|N 8:`  < HPA E(p4.z:IAXyZIGFC n9;`;A`[ ڲ0˲25[OVqNWA n0pu}@17p  6` G{G KЏaGRQ31k{%7Z^iVMu Kۺ {DPFa6WpS~ik/ >R?#TU18aI0@F  6 A6Pq;ꉮ?i^D`{۹p6p˼Tp 9cA@ѽV;6:jWP˿-ۿ2,pv0 /P`N~yOx:q([+ 8`?;v0U(Kwh<[<D ^ `;1N{l[<'«h9TA a]py I ̺>P : 6`J,Fă3d{dZŬ,Q cL])D4v'[Г}z;~  Px<Fw[M+I]L\c ΐА~ʴ &`))&P ( =EwKW~-Ȱ MEHX\1 fe?C 0Л@% q>0 qQT| @ w /a{P;WqkOCK'`AQ Q)P !mi9^b aD J] P(V=}RehSGi 4ˣGY<0a njpP +h)~`qanAR$FNnjȺ˽`;- v}L )`IڋrWʿ=({Sp`,S$f_7It 0;!l%  <0=C q Ѩ /7 п?A~Rq`Q!+)}~ o<Pp׀ Qzz042yP=8N= E}{fM!lQ3@Z7`m& A@<3)^mN)a@C.p qJ:p]QNRXY}) +ڐt wۻ  i]qlO սk9=p G/.@tz A1p\ Dиj  |P Q &X!! "/ A3c!^ J% AAmܚg`Im,EݿԣR9KAa^t !;`^@+U> '! !?B>&LcP aL~K2/4n6`[ܝP  |=CN?v6|J>at_*Q[ {Cͻ ZQ!TR1B}|yw _A~p$B >x >vD߶ (B6]f@$J-{̆ fSΝ@0P'p)TR01UjC)2Cð ~H`! lSbp7C!j #g"{nbM ~(ϜB+@P|YRRV/A~z j`=W]d)h KU?{A|HB94i&:nK ƟfwC C3 Ak?m>| T)AJ`"Wb Ew*=ZkI.C㎧! ky3R c G!Rq]l#F?3Ck F0r .~*3Q^PA1ެ;/ pG0p;(;@0=}~s|cg8,~ }seᄕt@HPmZX (zEbc`pҪGtuZUA>vs(e j!R?{ {C<l@ D@3 y<*tqɋ45B̫aDO*")WÅ= {P€sB @`c5X {'L2ԄUB$9(}P6>Ms{~0#7(^|CDhĩ`•hh }CphKhӁF"n-9Ar <M;Fums$Dt` vwFl YDtKċe %C4$:XHH:`" wpG#y~dž0HjBwG_@f8} j% OIсy$U K(zA:yl|0PH _ɹ}Ka bx`{d X\p;@B.tLПʢP}@qWhfpN:]`H[J]KK )('~(2hg`[~X? }p?"0 mA!K3'F,\ׄM | MLpIP~ AT ?HPPPPXLh@{PH\6`}O$[$֔MMɀB̹(lc>XX} D`D%#"(ӁE2#0| Wg~ @{0]hr`Ѩ =8Ed؄|ːO@J-M4}SL(Ŗ/h8!8pz*e@,=1S6mESE8U9STE\bxHpnyLpRS !} }Fgρi08S;E?TT-U0URU]U:}Օ)<ʆ6x)ȔI`ĖUUW`FT@ǁkeHCҁ5lwQ=Us%W4Utuݕլ|0zᴐs5ZC9Bf `q@܏v؃M22$ nI-WqYrZ =:u 1XD\0҆p|hШ ځ#  PV؇|H>6H|/(P[MI [R-WR%[7R0A'dە`B[X#zeG8408E0ˁ!I04oX/ Y%ٻLѵ}EߔDՀǮ$퇆ۡ@a M" Tpn VAC؃O=X^p`vvЍ V}Ke:!@^@;mGp6Mp$Q' bh6p0"~a͛pf%n`{J>sF^qK@r0]pkZ@iceEEpJx||6Ńn\pEf=fbr04;=]g܇3PȇC8͇FY`HXR5#HƇ؇nubޏ~>5/P_"1xˆvond`h`諞d07ɂ UhF^tnBE8>sh7pROr+nzPᾎdR*чehF>8iF/PHgMڨz表jjIdS 71`KvF=np)ƹ\R y;҇EP݊2`u`Zp 𺇧rͥ^Yp#fEh-WfPcƷVb m 툠}Cp\-0dz'K`mW~n jV  Zp{Y<@/_ZEAzּ+~̙mOh?8q`^@mi"nl p0w@vb؃\JE]x )J:nxz[q47zl_p?4-O $fU l&jj8dŸf u([kg6(xul$>c'cordef %o82(_Xqi2togPb|Ou>GM&gON֫Nh'h́y0wHkw (~G,nr|L?=dzWu8ADE pPUpY]b''|M}yyOJ#8 0hͨlQ$a%du~phq`@#8]yj}@C-L'&ҘQ0D'x'˘~x hnѸıY*|n"vio󹢕H{\n;v`uY(lC „mjicˆRL"0`7rG\@VL@ Kh1KK8 9ԏBJ0(R$mT(-(:B3a;@!<(j[EqIQ#*T(6#Ȩ=Hίm1HܓY ǒ$H07sy߃w!@@dESǴ5R^o,&Eb}u}H}<(;*XH EQ B~!ׇXB^dT Ĝ շʱ֑cOX2I͝o%>MBLO%Fj R -R{]w,ArW@bOBі8H!&+ (,QV5b;xP2{v:fяaE%I|!laLf~ p`?M@-T 儛9N =i!{puP3\+@C+4IpPd(BPhSA@!ŕN2ϔHO8U@~\%5GR-OYQ9d:$;pG6wJգI*P<>%3tDzAĝyzfBoE◙]KT'O+GOBiJ?@26@\+ˊΥDeY@ #X PNц sNADRYzY -%ҝjB Fk{w #? HeԊp; 'BAfPQHsBeMװI;l-MIdqt$2uxϵ>*b +ހDe0eIF=N:{ {İ=9=oGOMJ=g+y`fIIXm0 AH<ճ l vЁ v#w'KCL1 |xAGf8p!Gw=}4~WԤRD 01# "8Ez8Ȅ M$,p}H,"Cg!p_B3)&_h هf=yi7UFL.n+EQ9 p` zPdVn!x92G ^8Br%:j(1CC`r-,/-i2t&I͑yT &Ik16x9G(#L aRRiuVe;H9˃h#aSYG L Վb`q(F1:E:^Ew#J>E1gA wV=@:bB!s_EP ~pppd-:3"C˃TJWOB\t5EɫD +܄Y5;0"A&ȜALavn==tl@o |Uf#JI9uBV?2s',毲# p\FQ{V抭BefٍpЃDtAz{RcU$#0 j*F[?t@ H3q z&-lHG6d-\4pTZ!&ӏX a[v@OhWxA Rƪ`q3miI׻"Š ycg9g~<;D}|cqYy*Є Lh"%`CQE*~ R0gĈ[D2!-IPb֔#^h _ZD)*iΛ7|6!*XV C(ՊH0L@Vr Yv?nH g?~%uƝr?XU7k wo򶴽.sJ9˫q7)~>xb?dW JO>@.ɨh|HdA,D!#}wd@ksyQ E8iW76m$ȩS*PRv.cWF{ G> qݜ Hڝņc?g1<5-L_Atg1O-nQ}d^^), W 8X)}Y1^ !AC0dB,;4AAՐuVy I zIƣ9_~ !aOT5 `~ VR8iuhQ&@aC @HD $?(H#DÒd\` ̂\-G@b@ !AL=Tm|S7A8"@& 0%5\&rbF%\,>ʵA3bSb㯠 f/СHA+^S-8->Ə$yW? ؄E@^C:<Aă `SE[&C'@@u HB_[G!X=8|>n)Ov5-0dR;_GTA0#0E*@ A\C p#E@@JRJ,&,&H4h&2L`R%QEQRJ2JQ@%/Dxx1>UD-ģG8#UD!-DP$5AxC\&';=D&B穌G~SM[5H,)A}"D҉edn}Sg{zh|(}[CAV-9B@#D"-mZjJ7XG[5(B@z Ąer_&eo50A\%yĊ(yF`}.h!1!?DDD-D/Ι;BEFr5&&llJ('fB6NCqBUB (%n&ESWq/IP4 vhARwIݠ8 耮m)"gGo?l]CI B:S?h?+BHV.kf{8qP-H:04xq<@|񫌱^1#Ex^#\ A )DjGx TY\4C+/AC D-Aeya{4d.B(2,rP2I/B-_ϐAX @,%ĕF\t0 `&ܺB Ax9,eSe2>خm.jѤ)urdqPNATd]] ,Up(C Ct% 6*_$@Gש@ 鉀 L4r:H%IV&gE'MMi5kL+G3P?Pr)w%p%@tSUmP5C@¿!B OA 'AuIu%qrzqbi>uphJ$dN:x-kAhv u?ɫP<@zHM4jKA/duC@l։>6X *s0Hs{'9 -, ''C,Æ>̱!GLIbGDame;3I @QA u7|GÀ TciPĀ:BxLS'EsFP;A:W;ā1ACI-.ñS9WaACaA||ȃ<{|ϋ<}?X=<%HcF0X zQ@QXg(R3zSzBS'Bk+)Dk m!>wC8H &*(~whīS؛ƭoDV( 0, XGo*oؤrq"$MbB^Ë' 3p? l/_G|,#EOSʾ˾>T9@]J&TaN  1fԸcG|̖kӯ 4JS M $C觃TH $FB'oz4L#0h\plVg~%m ;n]{iAXZh EFY$v 1b'Ѝ>FW*7N1LId&)~~Ə~ M6ph1FW`Ӳ+W[flWܜk֚Gv #}ĴHL'PydvȧulLȟ'i''*(IլF5؃:ZZmU.[ɔs|\M#:(FSi3[yC РQ5 }y|4I'S5SBu,%Z'7u0lluW5Yj$V Yقu:HG'yBa1ڶtÅL$Z4+y`S\z2"}@֞g~rE!)p }[!3X`Vf% Wɦ׭m*I?IPdX |•P!2h7Å@~Hh<E:$~~ACZư[R^jYub,a%]!u̙bFd(GL֫p<p2)n}"jkt#Lm.Ch$C[PV_תtէ:#' 3})bt8[6ƿwCW&KԚ: :J<z֓#Mm,Wֶm{D1(|4hmbVBU=0= ¸fn%D40SC+4WFJ+_7!㗲>T-S-a[+]Ko{րJVcd9_nc=Aj[9GMUB06F1ڠL ;eߞ* ]CtLL#`p0(O2m#aL $b(2Яf6B.l!*σ* e]܁A!OLTG!N#T!0Y$b@ka #Ta0Laol2/2 4tP̳k<`f#"CI㈔۞00 ]os#a$%<#J b8D#$՘ d-H! %γ#\q@%cnA d  % )QN/! $]C!(@ 3b!#(oO F 0A4KpTV('VHDDD!w1"!Ax>bT #=0JNNh# 0d=#a6%ʢOt{KQJXhG>Mb*_#[ @?X@7R.4.J" @A'C@#POL#PC N#EQpME1JJcF?ZnVi@bДT! m87"r.8rJY"t2&T+?bvh4t!rR#ڑ +DʯM\(R#V!2A<lTt4- ##9N+2`U2@YT!IuZR2B"O#BB91Bx|:U@S#sU[H6 5a `XX{bYIEA+f&7"l#v6BA9O;1 T  S V_ie` a;A85jbC U5! . !c6. @ -MڂAS=>BL ,#%582#9DMM'/!" *&h4>A (ru !Pm_zpmi f-]Q5"Z2lUludc,`Z226ʄ6eΕTR`g"-SpA2N P"*RWbiGx6~a%C{{` kXWc{8<mb4B 0DԄ-3 %j6z4Mt7BHtz≌#sug%,BbIlQ8vl<$`,>%)0t#Fo!#HA4N#;O8E&OHab,Q/Ƙ{Em=}ɶVl،AiW#XJ6 V!D_'ށF`HF#HD2~`ohK Z/!?);W#gom65L؞S͸]C%?v_AYj6~.#3]SBk5r`0/5csG¸JX2bsji#'b[;*&SYu'+Xb7R#X*ox*+[L;@7#;S!6eqv)9Sֈ5b.aaR'^ ԡH1NY+ֵ\Ӌt=}# wMQEJ8;V2Əxboutv٠*Zl Lt*]yk v!$A]2uԗ"3b[!T1"y2"VX#(27:(Z(7e v<u!g~ x>2dt37Da#|F#I{!:!VUU[!E&dx|*S`smhAmFn}U zJ gMB\MG,uBbBu2EEA"OIrABqdfP.xBG~hIqB ( Q$F+cDG^FsEIEB'C^A}WAqZAߐ*RdP.`2(Ej ;BE=!-MQmJ5%IAd ^!d>3ݨYAx9U*EL*ŲRBFgTVJ9) ^ϕ@I(>xYQ7Ty sQ'Mx{бM)e(§p@=:i&R2 MGSAk9=. ;/Z;!OA˛ZբuCcAǭ:4|e b^h'>RXr?;RD# R7MqQU42!V#@BGt0`eFahAuD3VᲺ#F^ڗA7_?2\"ggTd||KG6)QDLLr`PDVt$ G%U)HJDF~GvXsv:qAnEM 5E!D~Y} FրLf*ԙv$*YȄ ABBHEN,h,er'ɯUA3* @ga)Rq9@4PB͡h?PkbD< J6F@B AHH: ]$ 8?rӶ 3"O'(5QkhҌJ.dFW)U,$;5)8z?2~ 6#ZCD'xͼ80i>v*x.K,P@mf"] JHuwEBy^O AC,., Zn{׊՜bFzG._A;@y"Dzp"vG}]7ǢYgh)Q4M߫Yhna|3|XP Q W1x@)Ta$PC7R96+R}Plq fm'A KT_p|2:(s'a 1ENu54gaevx.Ni~0xXF$3Ȇ87h @s=ȃ@xp` `q%pn&GNR4eTxPL0Ximȉga-!998txs O@ A @a}( FC-#PKg+?@XdE;o<3Z8@ 0 p D^!=Fbش}a('Ǒr~ vE2/&cpV70 c HfwU0*(@/9-2-R `2i\@Y;YFf?! %UDB7FEYC@a! ?V9 g ILpɕNٔJtyv9zyrٗxo ɗFIY~i)u uy9` ' Psx|qЙ<q1 ȍG@Dˠ)D_4!6)0JnFW>9 nXr.haGifi_ɝ "Yy\쨚Ɋp@ (y7yO[~Fa% hFy^)l+#r?cB|hfxՃI!ڝwXx9` py1*7gZ(D%Gz%` v19buAU:`٩Iɑ胿|0$FXA>6MB!MqWܠTkZbZuf: %*)oƏG}*F ?sb $5/zU :e MJ=B}CBN1 :PFu~픛[@ Q;lډly_YJA3:ɑAڭ! 6kSilqoњV!8f ʝ ;aAFa$`NC@YZPe2Hiʲzj ӧМgbAA "h` [D0NU@: QۉRf:zpG?ob{QAA:J 8Ѩt YY |бUK;ٯ;zCиA 1 t;0*tQ q Ԯr9G!0-g#"z@_ hޠ8P#GnRK Ѵq  Ae 4;+ k9BA@˭J` 7`:H[H5ozgT+[w]F1?B DkU6P;ř%|; /rNf3xХ5u=|@B\ͰzAk[, A]us34Dphu/r < I X.K-FQA`&gJ+(!k%Dn Dž E%!O!T=[z(IlzJҨFAÐliuYͿ ,gA׷3Lˆ JL(h0aOli`<ο|\`@N|`J"t ʲ; cV ) mpr+\"Gţ>p 6r6e{h\^O[Q+0-s}C-0HP D@EK`G~* ?h@!҉X -]QF1 1!D}D р֐li22,~{PM_%F@F gmjr+o_:,a$ lG9-(L H".^UB/QA ?~(zap!@QAo3,uj{΍Ywݟ^\; hA.UGQ@>r3' -~3؍! 1b# ϾRPQHJ-rnb@DnN000^ ? NN5{RA,k'.^y.A pg1p+ @ ~f͜W}^h1B_ n kFw#}P@nBq!kE_ ۀ 1 %f ;>\;ñ☆ ka㭅_앏. .WB ^Wa 5 P=,+? `.o0_ 0DohYg/%/rEpg /OVAoކZ>OWA >kDPCÅ֪+]ѣg>tvhdcNF$J1eΤY'lRW`BhD-:gOԠ;En=H a3'8'Η\re6ZhΥifAPEe)ݕQCĕߣG|g#H+Z LY(6^y'gP|y&P'\akJG a gle-#h0[,V)atfZg9@_^vB /{D{f, 0:_=(w&Hy9oQ ged˽˸24&Nt^kpjJ(g*@ilsK0cfyƾgqG@t(0&UʂqJb ;,t2Y2q. J<9=iü̚)HKD"ljC}J4 #"ԋL.*p"y`=Qb}TzFeO1)&A2G(h)Qg!حPGR1TvYfJ 2Px!VEYe.Tv)׹ֻ( ^AàI=:Y~ ,82bGwm !G꣝eu w)w.Z)<uY͂I*4}jvHid%m$b)6BÖz̡BIh宻k#"Gahy&ܥ'~ܚ>rh 'xb )hŗ)ͦ&Ӿm~ߖ.n)dNhF`]fFD Fֶ&ù_#'f<0"|G 7 Ԇh5+~EwTIƦ7ez+WCrs#^|OFH H1eZ6.n1<(Dq^wn)4j&z/h5|B*l'nL>Prs?j.&0CB$P{ !C#dˌ;!/׸}*H5b0DRx&>b'E=?yJW)/yÓ|1Q&d (A= 4&C1GFH+?1jt t35lCR>`RdPEd&JN,fM)MawJUE^J@2uMyDZvSEB8lU'G2&Sz%PҜ&iNMDt{kc3Ls~UfײFӝnIk 4R0WjZCG PeNz,in beNnMζccV.E+50V!duV6t.8>;|:VQj#r׃ղ~]e/{vlg:|XWJ_XAA ^vo|3_oOЁXGp[Ađ epX^:0M;=̋c#sAv _15[U1] Lt}- gd9˶:Dc\ j&U~цHrQL@stR;LFbdaVQgÿUMfڽLh2 h4L\BȲFSG4;j'jjü#aCӍQ;U50"64cb֩4Wlh?$Åټdw!ua2ho73Q39mO#xlyr&dv |w j-8PDaX&F^odz\i @7[5Q3q}CJ3ޏ\h(Yo<;(UM1G ͛nL+M0@uip0@^9cBX9cw֛Kgs7׆$];%(!žw6xRg~J0^7ۙ{_Wbdx@A@oDŤgxJ| {M#NHSp)@@`oW?ٟ_G?_?? +P?C@?@K@C9d9CKC=ÑK;>,DCL3>DDDWWWSMMfffkilwwvXccZYY55666Vvr-EEUUSSvvffSNEEUUMMvvww}}ffff-b|q6?)DJQNVEfw~qmPqxzpsaDJTZSfvypcZ..DDUUJJffde66737Vv9JwwDPIUVYCKVafphrwwqxchWGV3èe?pvV0w|lCKSYIfkS)mvwtQ[MbroWfʓڳƾƙɎٺͭ蚙Ӊͫڋ㫱щDZ!, H*\ȰÇ#JHŋ3jȱǏ CIɓ(S\ɲ˗0cʜI͛8sɳϟ@ JѣH*]ʴӧPJJիXjʵׯ`ÊNhӪmNkʝ3YtݫrKo LX#?.̸1WjP#HUŎCݻ?XHW^ ϒp\zx y:֫ONһ[;޷[OA߷' u xӱg{7`vQ7z|P~ȟxww j`yX!7`d68(`PT@TP @z5Gvax]F. A`^Vg@P=OXBPs &Wu`Ep8Y9kHL)H t@OAdR&!hJi6'&w*DZꪬw<8` joX\"AvH?HJG҄??(`SPo@s%|F[@K` ;.{>N$ a6LO Jڈ@?d&m7Щ@I?d_US1,P8+#6NxO #J B<{ne:xӋH.=Y#PC4< E+@t[A%|R<$@t?7P0a 1@D3 ~EaіOG #xʣt@.xbP'%,?7KRč?D鷿zDgfK/|"4PRxZ=<$0nTptuCaHq#Q(FHB3w$E'DEF8=t A2AC%h@CȌ ^vЍz4F@)A,#lx@z 5*P@t+K`3h ?,"tG,$G&0phaL*>XC 2:ۻ}|E }ŷXar),&x# \bt:H 'm^a|vf|͗W|uftޔ GVP&dE-`{og' K0 9 @Zh 1qaQɗV %hpp % 0B vAvo" 0pvp Zs  m0[bQ ͷ|PPQ`ZS rV"0{[:? M0 2tͰX?8fWWqj$x8?b@ P@) D$3]={J4  i |.>x|؄0J73'xh+dhp ZW(@@ꈎχB(`&iPNk6jlDD9`p?G fӰL^ l |K71i Z0?&"dF0 $0Kb @qA(N GPE }px[Ѵxi;1!}WyIc:&CqrgPP O hF? >:x@m k`bÇv uC9J|!5}9C38 H@^@ dpU! ' 0 v|pSH嘎xyGIz!)G``& A RIEȴ Йq2!CsgFu ( CFעh~[uw7p :CPPg xj^JtAa戠< rEc`刐"Yl4pWwgD e1 Ѡp t>? 0_[a Q2hn i \] OQ x Rg ppڣȦnN[Cjz |*(cx=dX(p16`I@H4X W ҶXpP 8ѠGZQ%x\f7R  i@f*aFlnbԠ4VY"(`<BV+Kc0TJ7"  Tx$mƪ hbvt[H<*g;P PV ӕ@(9K@ :Y@bW Z@+Ѿ 3Hs&I˵+?(`<O zdoZ# @doc) DO1X}%G4E? o 0]Ѫ,/^\!3!͈?+GN&A a{L\b14 (u[Fi=ю;Q} lA `rt 7I0=)j񫻗m}Jp@g\plԡ]aVa`pPS'Lֺ۽mY˄K L-*VJT15ol10, \r&_2۾=]2IpY P@ ~-)ȹ}(xhkqJ5!M]tjX k-"U9x:/I0o P! k@m6qPm[ ׍J+P@K ~e܍Z|4:q>fng~iΦj=Q!q(!h^h\d ;̥VE?Mi 6\h梺kN-=Ѱ npd*=Σb0v]r Cb=8\ `3͛柾n>@T~s=}(qg(CsߴCQT(Yl^&hm.;q-GmM}]&;ⵞ1Pf}8Mh0 ^~߬ >N81:ƩiK +pJ/hƉ B@9iQه@i_.BN} rS|]S*o{c]<\7e1f_q߱ z=~OPkp1X=(^>)_bQt+Aܳw:`3A ԰vd*pg{3Z_D ! bĝ QL2%Z멳:SsaDf[{lz/.-^ĘQF-"h%Mܰ!]!L5męSN=5r"69:ۊAU -:VU\4bBSH /3AT\E]Y*OkR@^2 FXb ئQ9b״򸊆M9vlꦱ)U*~0khD>lUr.\99iV%+ENgW*i-ϟ*$Hp')Ad1d×̙H;iNZp9  Тw*İ %n >RǛ%|''!QEa a$}ya8.2 ALjAL #2J)+., Pӟ37$`R@PBEP׎WfY'.pYI⧔+E={Hwa9R8")j[=R0]H^WUk7fYC~LA! od1"oYŸ٣Gr k(djt֎k'-'5 q#t̜Zig~P ?I$ŷ(\ uݫ!n:m' q*G.LBG"Y"Z(pm}"FbǍ74\quoH iC]_O< s Ĉ2l"MMPOO4}GH0*2U\TC֥BqD\ e!ql2 G B@X )446[|&D$D[UD;5 " Qυ-\ Cxiьn8>"DPa L4`E&$b6X_fFΥB22 EbxFN2R7< A@z@DBg49X7G$B`р#4$VÜEј_L$ٓwp, wu#EbYvy:FJZ9 *25B>hDrs !'/.\&O&\"2 Mfӡ2>Ԩ;B.$J]&ÔL}̩Kp8Z {C]8ц^ "sL1R̜0oU&c,<ÒiQai2HNSLUHVpu| W)hj^0;֋Դ8M#Yj$t@jT(3Qپ>!X1ra3v" %~Ԓav?0r<%ݩhF>x2m$@%#`ؐ[@GVfQs Ll@YH>AeƇ6 XCO9} `b>yy`';kQ1i],c1i|c:\?2 yHϐaS?#Ї^lx Ѐ- q9{kwL H^j3sk.:ǃ# +Vu<~m9A o6IΊ2 Vdx xbÚ,*jAl:>gD"vo܊ ՙȖQ e@3Q]lo y@ykJ `Q5n͑k\hL͐p\T?)TuW@ ]3 fsׅCQ S4݈mԈVh'! I`2wA_ u,JJ׵1=G^ʅ&ܨ H!nm3^bΛTS~zݓExpVؚ XV*)DŦCѻ?Y?[p#wXRԩ}(D'5A`T:B]Tfqgoz] l,R&HG _zEĊU Mɛ;x"`36R0ar50qЫGS[? ~C=NjAX1!fC_K6r89h3@P T?##۫qY*)#T/ '8"4"8[Ox-,ۋB)D@C-ix/k9*(`C*⪫($`JXL0)/d;+B@ C5P BA5錋@Cy,XAdQY?ZdWXĐ{%Ar(ZDPG#Р8􇰪eC! n<3 HL?dYɹp$񇵨+!*&ċPL%!f7A9=0h5ȅ30j4k3H3ĄԝwȂ`?Zl:bGlW8C{Hp~t*P.4$>4țT ʖq|08n6= )0D4pLKxB"kH YA¿[#XkFLHÈb,>* 39\+ ȰD(CQ/++!b@Ƞ]*MxĪHwɊrz8x LMOL Lq9=o*8aWNN2{L>+ LLV4Ono8<~1G飌)(BT z܇?HiK ݝt5O}PD QF ?ɊJ咖hNjȆX(!H٠(eJ((̈́8Ȅ 1(CB j,OuPF'j4B}06(Y8AȌohgp%#t ҝS-.sQ%SR5X#>]bZ`ˈM@J7|gpX |5`"5Q|lռa[3 dE97yPO蓬0Qh"0lLgwTeŋ?v  f%"YHG#%OV)NPh:%4hI)}%VWyxאW{; 򣉅dI>a1 Ą*T~@>e<= էbYاX̦&d| } q,ʪԛa Y9θ: YsX؋Z- ˨X XZڐXWwiD 5ZrXÍȆ!!R@`[mѨ})܏(Vey%ĭԽ]a G5hV <ڈUx|9"LIͳ,@EeE^feWKS>H9Bːȹl;zXouA c*| ސ~W݉9ͦLU 4^(+3?<8~? [uҳXNxhx']A;Mx}aE`^ h61 7Pˢ` ,8G#s^'87<6h66Ή V%,cAЀ-BY/@u.נd=Sc.`Ze%H:;%H7ȓ#xXΧTB|;XxAz{,h:Mfca`cZkbd 17C?0D8U,u!˲Y9%-8f3(`՟m` !ʘg51J\ J,м,e˳@G--d[- 6ב6a 9&U~< q#AIRM=FSE48,N#>be6iޑVi`i5i >^n'$1#=B Ji 09_^֝Kjk{E|d0-|h)*5Ýkepk%@֨ kkjl&&Etd~ #yzVh3hC!tHm8mи6pmnn}iV)Q@8ސpCU~@ )+ʝn%6Eo(o8o輅ledT=p*ߌ]E?ba.^HFpn8bg;iVz%ۦJ-W qK cxRR+R)}ȱ5PE~^gqj9pﲸTÎ0NV0CHLk*Y1C dad#ĈwMljzIm( 'up4p h(Ng)Qo؅%ˌ =ψq@pa@^h3b'b uX(i0Po2sQF@ݱ]ʘz;2V2dtl0SPde]FV=^gaMl6/!xoO˺xQx׈`xBj=}H-,v.o%.6ΈHp/^8b6ktN gGw8KxzWB«geq/myGBp+Pʦ0 sv5`* 1z :ԚNb` 3@̤f؏ T'!n @ ,)4ȇ\ܺ8{ppi NG ؉3.iN 8H3nFk@,a@ ;pͳF1:B?l4Oab,DK@岽wQGr㓮!#I3lS8٠  ?~D@avz|@{`Lڶy ցC'14YPdضWFnR\d0bːw.EyDTb ߔ˨!+I*'D*=)8E.ʀB>1F50 z0^Hjp̥*S8>̇rIyǬB[Q1+G$anb=4<*r KIY:΁(Q 0x]ﹽ촧*J@&!vTGMȺԄ&Әn@!Os#! S8ւ\ / 6ex N"$J=1SBhD؄#ֱZHq@| l 0D1L DV լ%A1\r3$&lwqQI0E82 $:V$'<zyӒD4~0?eTdRƥ% ,AQ; il#PE?ZZ: N.'c|Ё$CD$"A %04U;.P:{$Pj",|& j\@zF=Xe nP|ӰS!FgAbSI&Mf?n"Z2&sX3&S\ӢYY`AׁNo8=G\!NC@BiL(OF[oP==7'EQ"нvin]8! "";fu#pM@`MP@`JH=h !DJ rT'}' zo7@8fI> I2Gn ? rye/}@opKӠ@vm_{3y~ ={dVnJ 9C 0ڛĈ,L@ y [xB6- Z~?%x@_J^ tE܌E^=G٠ZdC@{5Ë`ݰD_Yh t h&h&Bux@xł'^ 6E E1ćO}9DP2` X39胮i<`X_D`xyqxBy-O A)C'!p]KDFE2/,D ?̝!x-_=(^HB$4=( &@7L& hN*L2"@.hh =s2&<$DFDFd AcDEВ!@/X,#DCZD ,ZA4]1p礄XD̯+/ SJ)DC8 V,<;\L$D~ȁ@Uъ`#$Y(i7| D<=E AB(meB+! (Jl"S1֥Aҽ:l?)(P.C DC8"D L;J@Do ^?d?lV/bvoRd*C 2 >@C$; ̳N2jlԭ%sKT+&\+d+A9dAMhI>(%uc\((Z9+OCX##(r[2S!~KuئAEܴ)D*PƭEp%n< 'Dh4F f]ٙ5Cd8MŰtAq?-22HPtKFqE4c˳ ?h(.54𰛅 83ؙ,G-@6 *4J D78CC ,rqˁ|fA"b;#B^ b d8k8* _yz{7jόqw>H,HPjB:ne64AP@ y6HVX)/5Q8t]-9?TCv86JTCAD4 Ŀ݃/=( (E @[ YFD.ޔKt o4.A ͑rD$:9[ۊr@CI ظ1?B' d}7Y!AD9-CH$蠂f0y?~K|}z0qVå; pz;_Ya_)'E3B#>P/tk{= a5-0R]cD't6;e~YӃAzHt L?BM$0 #oph[W~c~=~YN :#&^_jkCDȧEe 9h2 J3PP)๳<2D@'@5D@-@40l,g;/rrcG}FdI'Ezr8b`QJ(qF0>hQG*8ZOS;Myӧf5!V_˥E5|-nTF Do9ttN%DdV$NPtcl\;+Z$QQ#GL))ɏb:+뚵HB]w&:TV`r`mɅQk6C#x7Nw2^$@ N`FZP‰v<&.G-LÉz1 #7 3Cր3#8Râ[EtBh#hXFGR"Nab!Z@ <1iLG(YA0e ~DABjѧ?:6*C5Q{q$,8M9(ĢFģ%A UϒjQ9 A dfR`z>@9A:mP$lHϓ:ZngzHAbN(aG?'K'r'0|Fz a=H,D =4rKQx`0j|ɱT%hy`lPPL0(a0dՓT8a=@82&A  ~@GYfߑ7-I<j[wL"5*uMQwş}; YC4`(e (FS5?@hokJH8LZej) Hg PwNW˓wirVACv9~rZYaZ#~ ʣZfIè0cd2`c7а0̩ 1Xi%5԰Ɲ}(.oP I7G?+]18|$10AkP7z޵@=%)HrW#G1z>BpfH0?xc k NFWԗKPb0?A`^7,=4TJD2e`Oy0%;rdC1)cb|KiA0o,p!4$6ŘV\8#5S$DCD&$F|C4INuȓhDp|P=vHB)Єzi>1LL[d; b*+pEmI<щ50Zy!1C75Cݔ%˕Ilz 9 tbK~4ZkDhّ峖HD@]#i@gj292pE~7 GE@^)=e (D@؁y聂l#$lώhCmz ֫cL*F6 H({G 0,!EMbS(&H)E=cG6hG$T# 0/OFTV#؏TL+A ؘɆ1PLaJ@1( bE\9tV:"[%hZԨ"Q`9v8S|lJ~ȣa"G f)]M \\y$E5*6P Mf |mPFpyl3>'FmD$ ^\>ӑ1Quv@,bxeЄ8AR;"VmD^4%o X Tm ZB8S4cV|_.xq/D5`7I9a#Xr %B XFHR<=}; -GUüKA!Rd-p BdD 2>2 덏%'l&g#e=I¡򋾌`d_P.BPK $L*)y\g(xAE E$#q+LiwjAz~"1aߙ< `drEDVi{/ڀlE91=8X\(IԢ~0]"\%EbGB1C}JY(x6IBO A؄?jYeeB"햕'BObk8=\]'3mn7|j^aV#R!8wĎ°42M6G:А,oyO`a{D> 䫛>˵ |ݣ h :aS!  `VYWHCF"&@D4|#ZXMn%&(T~톯~@ݒo+8G'R&~ G`# K>N?rl4#r'ހ;"LĄIP 'l$+@"ī|@d`AP+m1&)$ $<ѭN$"0LPsR #z @iNg~$ !I~+,䁧ЏUF@ #j d.dLa" #taɪ(14$# 'M2%0f>̠p/XA#UOC $!@2Wv@PDd%;ɶQqF#$+ r ?$ER8ؠpF!.๾a.(\7l:Z0 jA`r>.2Md =32 5 2$1B-U3%Y3: ,1`RaF rr(lPL,Ni"R40JHa*3 5'xPgJB%&vj#PNR->AE>V6Ev$/`/iax^#ܣr^:=!l?`Ќ`*E䶠A A((lFMj>$UR-Y$##@ۊOl A@ةT <TZe0_wp,s=A3!!j%tt8+X#GMS$5-*H#" $TBOe !/kDRw!9y s1OFJ)g Ѐ\fOsU/EFu963KU5qPE)H5HUH`@`>D]dA@fG ~A-b;qbJ"xX fQM.B Y=zn+3R4zjZ?BZͪZ= UJR@8¸ABU@6azRLb(B_?=:`0:_LA*Ϯ.EhOHftab5#xG?Z_#L9PV+Xh/aJ`ZZ~ ,x(I*tJDA!& =Y4gH$$7ycԓ(5$k-=lk!A'NPokCb_#TmM`в # j{OYâ#c3j(:s8.C`tp~$8`$ #^7o4(L=B|a N!.aA(SuE\y=qt+zK5k<t%Dtb,8BҀ`ܬ[`b jco< qRSA*%Yp#Cqz+6M %$)8':W8%5OV;G #  @ؗpITL,^oӦwRp3WW3*"Yz9&v"%67PD8k~+r . TbT`'~a9dX$a \yȕ31JdA[7<x?djّkc8'S\;gy `|"`av[#P(MY1ȕbYmpäVoAPYNUO)q#,8Y%ي5X YC#%tS]İdUC_Q!BoEat2X`aiـYiF7 #8$c7<#XQ_θVa(TvU9waIzbao@v5sȚjD::{cW( aAh1.@ tmNBdK4#7!/.!Ȉ!ғO*w];yïqդzs" jꡔ1v$ l# *ᇃزɴUɊa3e6(iRRmQ bAb0wfK1EQa\a Ltsg.6[ő[:¯8YCtj*TB鐔Wnw2C6 LI `ͤ{ck_)e! >,") `v =(MUjz"2?2 il%1!ZmK @ݠ i4!\K;B.@C.H(#Hs$#>h#AV"ٍ݆9wf G]%OV無x]!O vj R̐'no*W|\!s*ig@!fG fRRB0Xaa~! 0"Rj O(RjgxT~MΙË+a}oATal9o;Bp7@a A~QB]DGwb1b!R-]XDu$z!@3ߑv~Ocis^~$|ҕ!v7UC'r0A ^!.TAA$m4Ÿ <0… :|Dz*ZxF=@1B+?B$ 4}J`& @$'HvG,WSI)nxE֭\ q0d˚= RC3r6[hkK0ehx $N#Eʱ Qn f,PEU?-1D#vO\ʬzPlٴkTЀo-ʵH׶iɥnMh)"sJ/S R&[-!=G=&y]mr;fa;AHB Lo%hQ \EЅ:(A+:lM (O'" ?qNBD $ t=KR#:!SB#Vj4)9 #%(iL` ^`C?|1RlC6*hdP> e^m\ꑇ_yR0 2NQa |rBN L@.!O (A,$@roA)A 2E:*r*_NA҄̃h5RLвdH<00 M8Bh AvAY "K\v ɝ #YIÕ!\y {<]ʈ;b!SdҐQKEK\$/I 3dpq<8@R `![92DPgB B@Q8٢y A:ِ|Scw!߁E "4FE@=I,`C-Dt:A|`Fq@:dyGJ `\};UQXBVh=ܚVU5N`#Q{*W+bXp,e/+YfֱMd7kYbֳ%-g?kЦv%"U6d/P2Ԗ|o B 8 yy.DH zLp B(JYYx]sR,oy%[68P@j­};i@~ BfE~ (>.ln+[w~zJ5F,Ko,X_#Օ@amH:̇Q2jE {ƭ:Cx. '.ܐ aP/x3``t :$xމtxf0iҕ"/}@4nOp(AeH8Oꬳ/ Ib^}G':`8PXB I#h9l XR!op :׼&x ?8Pz !o:a50ʰ$P= &C9]'=mlr*O5ЊAC?3LX%b]{N[ir2O Y#jV/ D5 H`@[f@#pPnpF 0 4ako50b-8QJ0O,T/MyJ4&'u`X0F$D>Qtn;g)Ḝu !8) D1`-挷~=g ir;7Ļ5ا//y:|e.Hdk:X L~|,*0iwsL~f=bK8 |hlתꇏwKYQ[T $߂o0]B~"yJFpLf3g4!nt" oS%h18[$]H&h ,Ep0`pFuq7GsYB(B'ͧG*w 0pw崅$w !|×Ih%X;>D' :}I ;h7eMAdPh6t0p# rAV#Tx)eh؇a(ń "+ 1 ; lo4(khGGc+ zT ͆] aHfB}CxhH|g: 7xR kXPh똂Xjp@}=& hUs&h툁Y yٰ Ph Ph@y)AoA4'`x's7#MQJhj,FBCu<* YdC=2̀ !"e+ [`qQ|>(t[̯,W`˹-1tQP0XkY4"T" |Yrδ Wyc@8YJ[JHH:,Ξg[R:`Ljo <*/ j, 9 JW }`' -1m3 4*nLa( D]YH]K;<%,=kp\HlEnEh <,M+˵>\(hSv@0K~\,\CW'9 jj@ yAlW,=ڢ Ӹ<6V; JP}ٝ aڦڹ]ڧ Б`BHp!?q C\Mڠ?͇#P=}k MGmEmE-p]VڏnzEK@դ۷=׽wpzjWJҬ;@>)*`}X#?m0pmE<B>BrۢFѬfk(@ mpH@xIPZ@W~7m[i4?nQv p怜B~],LZA7쇠jܵ鱳鱓g p)N%iDebe> :]?"fo&`îclPnbA-ru5nNj%bLT~jfYg \]P. !b-UgSiz~+pŐ)OTh OTv$@ p8<=0&U0@@@9>WUoZoYRS`_Obdfjihs/uwo@xv_~|7_Y5oM42_>p}/J+ROG ou__*fl/>Ԡfn.6F>fS3\/4oVׯ͕/FD>+vODU \D ]OB2tm ? .dC%B\oG!E`I)ZpQK)'RKK1uHp?3 xY襋9S,]B:)U$&[~BeBӅH &}{_0.,qݴ5;;(\w9w\ i*jĦfA@ʴaΔ[XS{QS-{re>E,|;yBB}޻zMz5TNz'K PpA8:>jp: QүÔ0!H Q$.!:ҺE)F:hFIDoI\2( Uh# P!!=2G'ݔ. T0AQфƶM#(>E!R ;gpml-matlab-v3.6-2015-07-07/doc/f4.gif000644 000765 000024 00000031525 12076017514 016644 0ustar00carlstaff000000 000000 GIF89a """***---...///000111333555888:::;;;<<<>>>???@@@AAADDDHHHIIIJJJMMMNNNOOOPPPQQQRRRSSSUUU[[[\\\]]]^^^```aaabbbcccdddfffggnhhhiiijjjkkklllmmmnnnpppssstttuuuvvvwwwxxxzzz~~~tteell --!!%% ""((//22114477==>>ccBBFFIIKKRRUU\\jjaacc``jjkkppppzz񃃃파읝롡뫫鹹!,A H*\ȰÇ#JHŋ3jȱǏ CIɓ(S\ɲ˗0cʜI͛8sɳϟ@ JѣH*]ʴӧPJJիXjʵW/HKٳhӪ]˶[ʝK] D˷߿fŚOÈ+^̸ǐ#7ec5_g}S^"E,/LSho? b.N8cNJi CW{:Cw~Ν5"Q81wn=u_\ ov#ki!`twg`|vE^F e<ߎWoK܊rM,krwqϾLx03ng,U#"l' k wHXO{E(\'Uṃ@V'p`qpMٜ<]p5<%1V7Y< R14q\ HA%͹dps@qwn{Hc 5 DϑX& uJX=t+ϘEJn}cg/D_?*0 CgN K#(II2@ %bF~Ͳcq.A$D&06q-<gV3g(1OyBK3九I"Ĉ'z/T:JtT+we`tz[ʄI#[%V6% UN+4p B-ښo&"W5H%rnJ|Mpcܭ)jJ٪ȩT:K d"cmh"i&=*$`5PQVy:{- ^@(lemMo0lU|[FvR _nC&5_ȎAc8K,zLSBZM\7tzD. \`EItH9D6ҦuM(qQE1 ȭt11&#Pڜ]ƫ* 4O !!$Fn6KTs;$=)cp@H*Mx  %(ۜv~;$yd&qF,ܑlwdM4t:pI&~JX3ԺǦ31/PдNq!1JvE Y=ۊ:QJsfo)vP'z4U֝Tq741I l>?>c71{wmc[! ͑kG&9iANBwBnm|o737}!|u95U 9D!w6qo:q5l'tǂ$bzqSO6qp" 1hģvۑ.!m13k8|*xAӄTH#Eb+!r`傋&Va?L ˜a &;0o/==LU9Wq^1!MbG))qrnAk"w lCaj`Q.fZU Uhk'+t !A3wU? V&bXʣ"!Lá烤GevQsX_Q jlӑ.qNzPSzea7# *'s0a!zs D#!/ fDէ wvQdfQH媿 Qqw73,~izb=H=U{&ecɚ*G)TFb; YէdgkI/ [Ikv\ =%Y0sPf2+A.vЬ팀jC.q?=n|QiM=uщܐ9k>A 4W"y]jR֑bfgχqKq7*<3nw{=4{E)R;]/ɺyyBՇ-ՊpǐaJ1 ?Q۷\0f<ЊŽ Լ:ι#3Oq9{6}f3Cń$3$lL5߽/<8F A2YR|#.V +؅mՓWEpQsǍ Aw}*ͽ0rѼU ƌ؇1'y߹Kaw (pܐ!0,~`Gaw atnE?Л ',e6M7]'*Dt~$7RoNmd_qCM; ?x2n,eo<ڝȊ 8p1a&QQ﷍szm݊!AP/sOPtmRab`!K!t%\7Q0bs4O>~/q$SꎁaK?>{\@ZmAUQLп 6a<mqEualv8o`Ѵ(_~ Pv7`#!^|}k)lh)}|qN9#C&ƕtar'&0S(}Vi!zț!`'POQLwf1Z%yN=ﶯoB O\"ZQL'CEݹDVpI@ DPB I (pL/#S"5~r MDRJXjA'P,:}TPE4pS"f"è4>2RG!QIX2M@sZ ]\uV^}QQ(R^\*re-̐)foSoOΥj6h j>BbjFcƄ/Cm =Grj??x+EEM;G!`ZqVޑ ]qտi&BJ?>r66$P tj N(<Њ4L!r/B 7롁+*).aj@B;ˌ&lB Cdǡ295b+8&1VF^M8CB,D. " Ŀ*B74̏"lʃҮOA=$J;X-<|;:Ǻ-JOl+CNGN84ЏZ,N$4Kˊ1R;]IuWa[ D(`mPjֱ(OU 6۠PVh".!< V1h6(S^kA$p> t_6X "r'Z`m)01w\ ߃L2Aa]@"XEرNbPҌ5ďL^c*ԡH:N=GD1@ޝ?a[*J~=zǶ`ZN.) :[JQ% x*AXB rĄB;@q6 ޹;o.2#)b`k}c:"enjέu"0*K ȏ(t!BQ >.oʓOP_.߾GOR [,?oN$ Ч!ܽNHu0@F*#L<0}栓 /!H[It 0> .!7^3 UsƐER~d;M`rLĕӐ'A$;Q %|y=,!D38p _JqE|` I(q sF6YnCn B<1+z0ɹpq*"t'sl"CDrC΁\ d3Y>$/ <#iI&B͉2$HX9hO\?$ !&كz”&(Ak4Љ9p~3 <שNy g;y3d;)za|4}CiІ&ed/c2?)ёZtH!zRT —J!L\S>U}=4$cB %S)ط` r$ I'Zq3LFy۩ W$+ *BX1P*H^Xd*^BYOd !J d QJ(e ~HՖFA} >bvwq4*B׆կA. oXW gNk(ҙ!؇ed .4SRի&$[-r\0Iu˩+3!~u=p+)n';F7! PY,ɄeEN'cMUlBїB>8I34<%SEV0L6=nƁLb>ֳW:%rRA2 ́VoʵF !2@r%?zX+9yÆiX432kM  A,ALPY6H&Zd1fu'zpllH;. Ѡf$2iI_X ʏLr\z6v,LNFF#ȋ ̨'Aٺ m :V.oHtJv[)& ?\2Qsݐ1#9'.@z ʲWOX=?F\!B-/3^ȩ#skZ Gm"Dk1$GZ-Ϝ]Lb*=#hw7V`XPtg ?-6 u4gO*74*}Y&^uW9'`*H.< JdE2{Ud:94ͳ̑\:9C u[d\ ~^#> g]_@l({9>CĊ;:? TZs*(5  4k 81#IA!2` 082<<d %.O.ɻ S K&\!N;d=Xr@%830r58F`sr$<2=48RC˘c#?̘qcrD c5  uBM,PGPL0 2|XDŹP ON{6E9DxJ׹ƂE9Ө 5[ 8|*P/ a /kY M{n@x%*kFЙB{|m<t8T9YD+ dGT.` '[H|T} M0g 4P9 Qŕ۱e%[Ʊh|#gGFC"CHEtXQJػ$/0ā0G xID="H (Nh4h+ʐ GܓĐ>N ZlP$HTEGS:U:)!96(Ӛ{8֒1V[IJ?ǰ<-\% @ 9`ԆϽpW(K&h-7:`{NVN5eK;$֩RaPk)8t]gkAWZN3<Čݞ]mU̘ EٝSA_}X5e޴OtGY٭NkO="@YXe%x,zIN8%Pԁ\<8}9W^KϮX)4Xۛ yN=x/Yΐ֣}ٍݙ̝Q d J+[hY}MM {u[ m;gE6@H40S\U՝ `MK ٚJXH!,3pE dYYW\t͈GH4O/˲(45¿Tq֭ԀA_ތqݺh%^PWΩ88 d{y7-VĥhAeP`ȁ#8vݚ`a_]ˠ7j:p0Y#>;ߧ3=^[){ik;8`a$6_[,K%T:(Ҳ<؁MS[kiS( Z=]9)1֋|Es֕^ЉdLx _[`pUt<бж=8. J dPMR P}Jϩ?l%YAIZ ,N>)1\)ϲhf/>TM qOP @X\~ЃЃHI(g}m[v>:;( ;reVaT44`+.в!n `g"><#F^!a\IP&.´U+Ѻ&Z.~xV  G |GrjCC0jBA#dh)+^M7"0hkR)huϨF)fZ)Rƞl6)~lͦNhmֈ@6D MDxEmG/Xk+ظbg` kgLf(Eo ,1 =G4gnQmAIpB汱ffP(^dЮ`F%ֻ"ｰeee2m!1&hA(ӭ $3{Kv( ?oYniF7 sЙᆸ6F`#W _ 7;0s?^˨ >c&^ gL__-qO6V ~#L0;gs`_ ]'@9Pi}qu.2ӱeF۝-8 :G7oߵDvP ?Iu H|!vwpǭ&2g[/A xPsX5f" pضtawa%vS泖j#V7'O.߈ҾIzP0X[7}ߋ" ةqHߘ{z wgt׈GFcxzx7(㋦'|7|S1ȇɇuHHg୮7/z`Gr&?$ ];oΠ{)H}(10πS.~}ťC]/ ?~d4=IߗϖgY){0~T@,h „ P'8"ƌ7r#Ȑ"G,iB (K3Q"^^Ҭi#=](d3B-j(ҤGU*ؒ&"E ݬZP=8qG9̤lQhעT-Fp >ȉ"hC܄dDЂ31䋏#˸D4Ep僙(.m4-~R#<Vm6ܺOSf"u7ʗ 3͛aQw%lk?˽Jo]G$ uo=IlƼj_ZOE7}UA *n$XU &aHrȐgGpMhS&8$7ه"G/ˆPZЅt't$3#(R)M~|$M:WdP!6$Y>9R0tY4wZ.d $IZ]BHuڹ`HI䝁 ^GקDs (jo}Vj@>ZTaNbr2)j)QV G*du(5z+*H+N8%hq.edN!-zmĮDkfhB&m.2 oޢnzл[ hFa$\ ;0K+lqC\O$k1(g1-31:L'r) sLt1\PKM4KA_œ 5a=6e}6i6m6q=7u}7y  b{ۉ@wV"80fz¸iVR/נ hyY9끮A_i v>{A;X"&Y |MRg}?rd}0o+BϞ~Լ5"z R}xu1\#3 q*g&3G8Á-!s>!(!F@=?<;.>@=@B?15DFCFHE;;IKH0AOM0KMKNNFNOMDA;GPROQPSURLHUVTCLWXVINQMZ\YKR]_\YS\ac_a^SXac`ceb^YefdfheX^il^ikhd__dlnkjnqnpmiekfdjrtqtvsolvxuipxzwsq{|y|~{vtnw~}~xxn{섅|~톈뉋쏑ꔖ헙陛뜞蜡럡磥槩骬歯豰嬰泵䷹滶⹻ν!Created with GIMP, H*\ȰÇ#JHŋ3jȱǏ CIɓ(S\ɲ˗0cʜI͛8sɳϟ@ JѣH*]ʴӧPJJիX,ׯ`ÊKٳhўzu*۷p}jݻxU+} LÈ+^̘08o5gE˘3k̹Ϟ[ xӨSNHװcc>12* Aͻ NqߙEs=أ pԲkνO{I˧/tbV%[>={F$@H2~ 6}4C @!DoLsC:A8(bxTx4]bC"C $E'ՌLS.(ӊ3i$F!N,A'TV Q@ : %R"?i eㅖ\QhG5u ?  ω64#9}29H㝐1/0Rs*p'59$nx>4+::j70B('jH6^`HV+hw҈ .[QJ^ݫ4&ނU^BO:EE,Q`߼KM5tp3p rh5)܈-]#O48ZXdƤ41Ś 0Ì-H1KBm#4,9-PÇ BSݦ4أ\v\5|*׉F3hـjF)swq4w{ gBQ ^aw Ҥ8҈w{P7:ӘI(CxsdzV ,:;`s5/BpxZ99Ǔ~M46c^'>[5.g4pi$ ~$~+ȾǺ5ֿ>@U ; hL9\#.U"P6,xA!(QTFQ =p%P s N!UfTPԈ^6H#JFD<1+8T6A%cjUαj&]Blڌꆦ1H0a sU08/FE f8DidKU`$5DZ#kh0 X)CZ%xL /IL;H2$ t#AL* %%6#F?VJ8,0e.FeMqΐn \^z;?ΩC<?!t>=ϩ0PxΌP uDX0F-# '@y5riH@,G}#!iKHqژ7GҍT>$Gi|@/̱Azss3B#c,S X"wX\H$TT#? / PGf$r_*t̘_1ئP #8P:.7^L٥"xct}P^Mh9-uXVdZ&2aY7I `6!.p" ($qQyLppsUجVI `#21i`c=^Փἒ0!`p0G1ġt}F4*Z5{-y+&}ApvuVKS밙2tڿ D<③b ^mںҡ4t@fq QDn_"Cra'DU! 9PHhʨ(M #GLލL@5ƒ1W9P13 D!*D dbh>H+!8^i b ixFAB1lb V:B!1!@x=_ D@b;i@0;{>bm; dv@h{i/"niIrBUpHLJY 1c G`$|i?.r0 8B؁px=rr$q| @Ks\=Hc.)r 2t=Hat|Ugd+T4fڄaM <8o%eƆ34q@P ޷u<>ą[yx @vKR :p`.$aZH?d509wYX=e[Ld*ٞzDX4L (c͞/Cl8CP%;UԘ}򕏒8_^*@ 疮QT5KM%Ґ b:(A>Q}7IP\ h6}d 9WHĀ=s @ (x P@]H#ҰP=!w$(w#*HXExV' q A&I0 J'$0&6"2W6(& 0o G K(]` = :a D! ߐŔ |)~K#4p IpCqŰ#Ҁx8@-؇.|'pAq2c 2 w*t1T=#`QP,H:Ap uhvOn?a*`! aBgȋi=q1ڠ\%`w8' x ~t҅5!P"" -_ U6De1 6$ Xx pn)HMm:F -*_y&yj1 :/P yG]ׄ @qGu2g sy;K1a R9 Rst(gU Q0 r " *?)[&] )ИՐb9=f:7blYB)(@>P9YiU1 l"5Ydt ‰ ȗ G2R 2հG a?3q0Cv::2 *t >R 3'P 1 " " fym S(B 8%G䞄>, h0 t ܒؠXzq ܑ ͢+ r 4p!=ubb&ę4ܹ=]FE0(Y 4z(`" X`i3q>u( **й#؜4? ^H Wv$Be  o8 2I _W3r hB s L`T'r-{@5H x'a*pF*y) @2zT0 H^ƫ$!@ P ,4 P2QVK( Ƣ _y K`ڰUzt p*@(0uL&O1 PLQ Xq,~pc y.MCjtNddQ* v }g*hF2 q<`FLЬ Q;+*E*OPAa o אh+g0{Je) T6ި0@j/ũ + 2!l_=@foT C^J<0)H8 ΀*P R{!&ް hEא."p q !Q ~Ѝ>@ɑln x@2[DEz;+ Tٔ[E+jm) ݡY*${h i4 ހ ͒,2Ut*P{ h&{ --Y';"P@ph&"C{&s@ $rV bVL1 Ā*>9 a'V/PL @1 py|DIpW@4YaPkl@ ( 3` \` J~k)Lc0AݰF*")' p ,G/B W~*pX)'1_8;`p~ Q͕Ta0 Icexj`R! \  R X0L*߸/:P@0@PP*@QAĐK'Ēہ0b *TFe)4q(MJ*(M/2gyي 9LJI9'*I rK,] U'@DQh= -0@n%&-O`w a"&pj l%$M@Y1 ;@a "?PpP )P߀;G@ q@9;ͧHkp^9Pj!<֪*\$7ف `ִ$X#gE2圀P p uu 1 gu2X3cB3 esMu*@kv{'@8\d 9݀ 000Ӏ=~U+hw^ Τp^=ه^* ˣ㑉1 2JP!]g <TxX 74bA&{9^y0`bB>" 5t )E3 @!ax@@'—!]φ<5~^hd! '8}&0*:"|0`፨N j `d`i: XV^el.<'"y}$2`b@!R̀90"])TN - Pvm\2Ur +Ґ)P@?p>ѐPn1l JӠ# n'䀕 DX/zfR ǸǸF)' &ǘmþ ]I VG?U0,hDQ o0/oP0q H?/U ܓT * 1Sґ /'P7p Y Tޑo?P7u1ӯP7Y2{ȀJ DP 6tD V8Qŋ9vLH1F"?fD5ԤU05 cYMԦ-Pˆ>x$yr EŚUVPQEAؤ'I"Vm-jpZuś^}")1A '%P_NsQӚI4\4Ehҥ{NOfвNVùm߮nu=xNន<kjRČƜy-lc)6LuZ8O60  ءW;|]V^kVsp䚞h$+BAh8%C ,H%" C+ƭ>D_> k z:k lpǦaGXȮh <4h)`MϭZL13MkʚTk)%$`qF.&z2Ѭ)34РRJ@L5trSO9:~y. lzd"F*-+9J|bZJE= r8#MH{-gm/O=}gZk Ub@PԉJZbRӪ ovD%6$ tYޠ!`&BkFlj%iZkZx`fWRx" D _ba{ @Yx 3N:ѕjt ahBR/4؀CL`NV&sC\@:KH@GKgʑv/L?$u5ceħVtwI8K#M Ad% R.#V66 P&`{~>yBN#ԎEF!N#~h [:ֵ~@\YG7?]6pDT' BHi܃VKp4׹%K_KpL.c&a: 0Nŗ2bsqW LyEҵ.Y._d-F#b dH:(ᒻ7bI[5ZW&.ſA€y2 <89D*nU s#& !ڗ֖)Zҍ"T3ķC:W* d>܊>dX;ńx7YgX",$ލld!&Xh1hX8F$=,*֘w[BFP/iC28qckK8XWUȎE8Ec?D9oаU4kxY 4>7"(TX :_0(9"˔h j꺤=-?crܸr(FzǔxY&Lru4t=Ag |zlHyqM0}v1sI"%s< t0QFr* 4z?#y:*k0 ~>ssYRkځ^}MҠk@#9Nsh9KH/h߸sh%Rp Usl׹G`0膡[: 06;+{"ʑ5"C[a#aӡZOoSlH  h=t+56؁U )&QN H 2?&L?C{P]Hø#– HEpijȂw+GhP@Hp1';Hl+5H(ƀ J@>3xizOp ](g\(F]j {{Rcbl(+hcZ,-v@Z-ظxJƙH{(B*qQ 8rrzV8CwCȀȋȌ,CGGh7a R8szs mk#):Ǚl[(ᮚ hs+PW) q`JqNhJieKkQOh"C0Ml p`# Ik ˺ i =#sʴ'L@jPKBf:(qp!-x>ɨu(rOl/FR/-E^&#b+`ҁ+Ԃk6,6G06J}EjL 0qI9ĠS@#VgͽJ|Hi X*l)ΐ N! p0]4i@rM&,#H(&H5s{TP?cqQ42O8Hia ؃:vG[- ZS^0 i83)"Cᬕ1rr0|!T0: B Ȃ@jQ:jقQ,X4'ɸN iclKPZh( eL3xSpK\b'e-1GomMHn1SpKjP|cPi];sRPj( U[u.7 it Vt>5EڸW}XK HgMzXΪ" vؾD &7<2^%+PWFkSEm`OT|-b9זh`MKlsӇ`DCGCGC7`@e[XO%iP׿7xsFj sK ]k 9}7|= vb?PI^Y\X\:܄Xi7^yDF)d'ވ77/8NpTe*Ȁx(\PUhvhENH870hӨr E_Uxo޴D~{9j]wڽ8FhN4X-HZr fXMzv (M f$J0̓6 }8֞`X)܊eҋx~ۋ{10ℹؑ uD@Ip^J %״zɄd~)L3h%84PR“0u\ͽ8Y`t$ #+:ЃFJȇD 2.0je-8@-$0XOqSH8䴥+pT gE\ ,C6\x&Ǹ+o19m*#ɨ p9[PS ]7dVp<.fSX`ُU }xR5^1lȁ$vXNhx=p J @zƋ{(LЂ`<``V$h0hix% v.Df"DDhGD0jP r(..h̀a,Vpvl l?(-ekh 8،kl=dXjႸ1m*bXЃl`nXw>dn~ЋeKboYwA.Յ9.K4h`w uNŦF&5]b8@ƾfPl9Ї$F\E 8PjHOOqZ " o H~~e/)n,r`VsU•l xn0jF `ljsƋ=O{ 2\Xa` 7h#X8FBqj8IN>6ph3n / E9<@Pl" י\^$>gp)DB,ƵW p })%Ӎzà 'AKgCp%8>0`ݻrvw8 (@0bx *WA}&305jAj7$v$ʅPtJhxh njN]5a'қЮI7Kiu) G+X$ Ӂ( Y gj)ӴzROCvf*`< +g5(df&@~H\ %0a+,@3t؍J&M4v(w|& $G{abϧY(m܅ta5i"OuUV7^g"ndĎ~jGJ;78z?5#2$)Z љJ]_zӜQM)H[1nTBc!:ar4Ȑ䄲Q$X|?[^_f2, XsOLi"liJ{:*@(N[-Y+ )}8(DȲ!5U8&^WRF^yϜNS5X2_ޅ'2[{u~ז[8cuq !Ǒ҄-KR,MbxJ@Ӑ=?">*8\sn47 $#W5*Hx@Mn4PLpfM<<+;HKO^sD< TYлRArULqtD&B+\f7C28S'?Jя!^D#oa8 30b3`t O؋8Dp e)d8N!xz%\@ Pg]`>B*Kn@k @&P" pİ4M7Сn&*:1-&Z%S[/ d~DZ +%vH0J=P!e :,bì`jc#T;+āo4RɐAv "jD|A LihcDWAR_$Xd{f`PNR#-5ЏTI݋4^)BxEF2K\jb}%ũJ̵s`;,%i\łz Xb``$1 4 (5Pf8i!V/&_/m ~)s_p A~DCRe!`qt o@:%l4q-lT#@q^q>r.>@O"8"KKј!eMvaTO 6" s`A>F``'cN).H:pD۪4XPVzi# 2;R^3&!!K0m|!zڴ68O@Ҙgk,({f8zd^DuzgePPf m!ym# =& bAM?Ptbd:c/!O:<7{ cpĜ 0d2agM#N4 )퓇ޛU0, yh ChQxe T>׭  5I>_(ݞt _q2MxdU: H8y߀ $  5F"ְL%f\L6pN NC7pM@I7 |堡AŞRG%^"-v4|B]9'^?< "J: zB L@ OCxyE&$,Ң%J;0\9c9:LipM )`4cP]#d6-/Ԛ-,p[BjM89 6<`) =f>!$68iB)$[fFPC52͞}B.d=&ma5EBcedj9\LA7h 9|&hJhiEU'La>9-[dN&5D7p.>,>T'Aojo2Fpuyyޥa<UE4$efO`ǽSE8pa/EWcXr]xx_5|&(&'LQN@>h'|B!&|(&4B(&T:4$ HI!2$4xB|=h]$(sT)(|f6(&pH%_g5#,#0#4Nqi%E1 XR ۭ\{28`(\480N 'i@7䘩2> ~4[)14A*TH7|F 8V4i4)ipMH3$~&|X̂C#%X#%g*F~Й9-M^\]ሩB^5d@;a$J)*]bꛕG5jHkH3D$ iqBr_l+C '-5`C.-߾_>;ZY*% >^DX.<@L(! ܃[ťȉ-D>A&5~lib,rX96+WR^^\"CdzQS.4$k@&E6 >m4\gD3xk%1CC@n5(HxA5CFFA#v]%k$6 ՝Fr-H"PО|ø@6x5`5qP-W.,傰ah0hjHxnJ8%d%Fz64dDK7-B9|~^mz@8W(BetBPBoiЀ4(ÿb oRpaf cf8Ec 4`o|\h]Bp`:@ː0]DB6(q\)hX*]%(Rn ?C.v É4P`4ht'@*"E>AT_}M4^ԙo0u$D zNeI~M"|Bʃ<-4_$pt09Ha1|'D$:i.M!^9/ q30_4 F3is=EM/(Q XgDC耋]x@˃ 00Cw{I1K\CHG\&CVk 'ژI VM74躷14k:z]MusZED2,!N0@]ȣPK";+H[j$ká',_5NC{~5 fw 0vzJT:g0^$q4D UN\6?Tĉ'<.h2`.C:h(KC1pBT+hpöLB+$?.?::B@C<H|dB{\*xB^?w_ kW/;@X4j&T!i8bE1fԸcljy9dIϜTeG\ZxZC )0G4i7_f( %dF:jUWfRHLmJ[7l`т$қh:Օn]wUůpD6ͺoNq$-/|oe˗1g7"p:Pi(kMQOٷ-m3fπiIikt,cYqӮ5WtfիĞhB֔Av ~}Yvl5%Ai(f CˠeF!P *o)il!K,T**lKʙUiaܺ$F_-ǁ !^t` O4L ӈ5b$$3 #≻S0DgLi~eh)8x0TYw9+ AVyb5Zh6x}i2vdy}wn-3eǑ뫾 hqioQbLi9m}U Vip4̐ǫJ_F'3턑ZɼX[6z=i,]Wة!Xdmʦl_hDDaOCasːX塽F74s$o`7TO3N #þw}E޲- `8nL W>@ Cu"'s!_4xƒ, ]~W6Tm:d/m[:#.d'Fq?9a‘7A h  =}l,HE.t#CB%1IM6'AJQf4)Qt+aKYΒ-qK]/LaM)N,eBFM NwAڤfjM"H D&t1{BS=i q"If橁^ @($Q z1oR]4'1Rd@BS)IeRaH Ftf4M]S.$EZB0UB"EQ+T-AjF +@K*ԮV`hU/zl~L"M̸V&QM c@=@%Z^&B? M5Z,TV}Q6(4?d@ 'գ =X[ Vmq[G[V "bqV^e@sN /˅ue]nZwr8L h@FЁ"ȀЁ 8O"~(@6;@B-%80G.$=R6qZH8j-(+D19xD D_x$IHH9gWఇ EP"pG8T,vG\,lr`"eű$D><NFDDȡ +D6c0%Q$@_5"liTaQdg)wv" n=F?sI)rh4E<}cp)6SZ"g5j8˹9C6'#4I &%A0G  DܙՌmhAx7_,!H$M P;3X4z |s"D*:3Xa?`c I R5(8ΠWAo?.c *, O]UQԱ^SE2=uA9rs/ˆw& ;gpml-matlab-v3.6-2015-07-07/doc/f1.gif000644 000765 000024 00000011576 12076017514 016645 0ustar00carlstaff000000 000000 GIF89a? """***---111333555888AAADDDFFFKKKSSSUUUYYY___```dddfffhhhwww~~~88HHXXllhhppttwwڊ磣!,pH,Ȥrl:ШtJZجvzxL.zn|N~  uIT..SFDNOtBIKsK.+JrzS"pl<X-ŋ&:a!Ə ilj8@ĸH&%ݜ IKbiO9nA^ػO< w@~\߁ڇ FXZ@xah Gb'fa(>R"'"+^43*"`;;&7VcD$CGNPѤOa;W% :3:_~xΔQTo hOe鄜MI_ʄ&K U[ө6yXA٦})X ì Q*g=jYfpvn0:Ĭ 4(}%j),X m0-ET0DZüBK < SpOw\1{|l ,r*ղ 4P>+*PV-mIn0+yڍ`}pR@uм邰cc )V x'zAJƊ4`iD#nސ{14mGNGTn9N;zU{x5X cyn_$-Ͷ.e !Ժ'b7gz`>_h;QףN/}M>/ SADPF)Q AuUoLGxoB<*ئj D?kaאr(|ab(üaXvcnF 8%n~yh2ZN]q9D71FeBbB4F]d#E9JWcH )N #eb.!e0@ Є")baRj`P l^±KH C U`L Ί/e Ph\ `+_(NnI8m$Y%rh5 -^j *ЇV#lN\fQ@.mJWI?*`A[KPv;;lg Dk*o%B{ WP`#bHq)`%$K[%,oYׯAq&S$V:7|{|)y0>Έ!abƈP{I.Z承a C[WW ߌq<+ T1C @[e ,bsOB*v^LJ_,$-=BrD|8A _!B7 =8{vS сߪ)^!QMupI/B׎fbLfC4#R V jWr|l$ˮP{Y0BܐhCh>k'1gJ/TmѶ¹mI'9lwi$ۺY63o2X?܊u$~784hYl>{~x0g7&'xw`p8m;,Ѱ\ƶ~AGH\NDYGIι4 np]AOur1RF6CuK?[ʰt^1Q?Ws,W k.9~Xjoj`Og{j֫(t-sIGNÿ+\A@orV .3[ׯ=k* M  6t8*_Wxq+J]֯Uku\~/ߺb1PD`q@!7~eu88Oz}FB OQfEkzZyħ3wIV0,mVvRq z[ #1}O@a'GP:ǃFqH+ HχmgG}(f}IFLQ8C| td8fVw!-y%넂2xivx?HE.`|zQh<ЇFĈE`wE8kHJh${XJfZCaqBj`P0z g<68hH(E8@!Do@eʨ#twg@͈8hMD؃Ƙb@騆@n*;k:qpv8@8۸l}~0V j {6Xd=#э~YzDmYB<}'2KGx3-^>0  < dqd%IsJv-[U9Seyg{bm]F1(VY ҭ7PM/}B8\T-}DLm5]s ]ce=϶Q)^mSʢ`=zP&=4%,$'E \T[,*PO@EMF}_sHEچȬȱ =gSۜpͺ} ^bۍͺ9EA;gpml-matlab-v3.6-2015-07-07/doc/f2.gif000644 000765 000024 00000035352 12076017514 016644 0ustar00carlstaff000000 000000 GIF89a !!!"""***---///000111333555888;;;<<<>>>???@@@AAADDDIIIKKKLLLMMMNNNOOOPPPQQQRRRSSSUUUYYYZZZ[[[\\\]]]^^^___```aaabbbcccdddeeefffhhhiiijjjkkklllmmmooohhxpppssstttuuuvvvwwwxxxzzz~~~kkQQUU !!'' ""//2211447744>>??BBFFHHIIMMRRUU\\aacc``jjkkppvvzz}}Á파헗읝뙙뫫鹹!,E H*\ȰÇ#JHŋ3jȱǏ CIɓ(S\ɲ˗0cʜI͛8sɳϟ@ JѣH*]ʴӧPJJիXjʵW3LKٳhӪ]˶[ "Kݻo2`0V.޿ AnŸǐ#K0$G9cׄA>SLH$CiXLא;!AH3gϦ 4oC>qɞ 2CW{:Cg:ν{ٙmlu_h^9/ ɜ6 =P 1D|%(YfG~3'G *cnR}9_$H҅aH7ϙ(e⁊8r ReB2Q&mk$!TVP&-"H\ڏgd Zr׉lrmv(RY3bf ɗ1B X(Fg"Ӕh$RJnlv)fa#H(Q0A:֩ff 뮿@ ~+oF,FD!dlw $~ǯ&m}jٖkU(spۭ|,zba+ULݹDljj$ܐ{lnV'4 '(>kBǫ/՛yIܑqkR,9w YXF7Kvn{v#~Ý>%XZ4y*ʹm  cT#z#ƙ}/?Y*]5νqr!seO0 c4 R yoBz 5;A d@1`W'⃠kG"8‚q"̷d`V!_J D1| AP|ā(їsB.2Dstx)ð8Ch ` MT 7 8df?"~^Ip<̍ ɠJnCi36$U(G .BV.QٛA!)C$ Ich-Buͤ&dUąa%I/ǝ89d`DžcLL遖| \'RHϕi ́2Cܬ[E931%g{dA`B+ (P (Pt<7 1A1*`+'S hXM(. SSD.2uXfCzd@BV1M%N,.#L$U"LX(# Qhid7#$؁aX$"CM2NƲAJE9Z("$))eWʮĩ0@!6Y(]2{DT{঵ %`f;PH޵`w;^񮕼5(k7]/z^ͽo~_w kdw5p_‡zse@–n zN! kUJ(\N撩}P*9Rw!O&%ȏcDt$Wr2|2I-$ʢ2-PzWb}uCR1g,9 ?fv%ik *fdeK n&zd"k (gD/0 @ fMG!,$▮!uilat2lCAzj mSײ]S281?dA$mudwUqrI*uMr|0tMV;#kAٞag_"A-l`R 9tE:N;q"O ]{M~&Y6rQ\UC7vDDsW\SψAo 'l^[aH@-oy[3:L%0k>GIxkؘM]&g`;>u!Sv^<{DO]o)"-N ,] !sm_~MnؾZ.OAa߷INyOt/d[AQz ޹ЅM7Z9d'.ĉMHYy2 -Glq{v0\>}5D }`RuQ n0HS!YU|A`h 2XwOzZ1pS0Y]z' HHV #'l79 )|#x|f1D !w33}sWFE0xQ%@:I~űBXnlx)rg0s:mqWLYqQ-qaH:}Ew炓>na XIU3ExQֱR/AH'}rC@rjzAsHyZ~wuw1qKaF,HS-NG$vmׇln8JfCz`*qmIօp6Y7Bt9 F'Q1'|88m !+ уĸ/ 4&5B ZD Y o5h'R|Qp@S8}-H#LOdhaa 2>I8OHw4 s!,WtwS`B]A,rZ s` %,!x+qAq~@W~a82:UO9٩QtD80V2KS j0%&)PB&ѳq[4Oǁ ixP L 92?VᵍQ tdBYMK0SwRJ* >ƫKb Њ>(,4'lYIAd`"b [֥ٺj, 0e0hֻb#~6Yj뼮1KM,K vUK}F c"˶fx1kM:(*9¾0K|emyS"Vvfy{a6"$ȢJ?6%, ZҎǡb5fU0ĊSV~&řRŝň!o. \Kr r؋k-T`Cq,Ǚiz;3hZ%CdA1d+#aPk*kSQlK  EX)(CiRU0Zt)l'@L C,9 'qZ%Ubc q #9g(l*YZ:Dh0kk'ṭͫ(#1X8#&a.|KOa m-_M?WhY}ֿʴbӐn5M'~֭asD¢|bW ar=R8 DiYP1"Ϗ-B2"qi߄ٷ948$-+Y ,nm;)"dv}0IOŴ A}#-$֠Ő1aɁB6IG,*|^m\&,LL7 3M9kT R$-Y; ,Ps W W 2 p0Y0eld: EZD~ϡԭ1 "d+OϩmN[]Xq qẠ--ZfQf AZ; Il+=AK?f]cdH 3-aV3#S*Lۋ`i>IOމP:-RSؘ1򞀤 U}}uD0oԁr#nm) ˍAQ=?b7bP"09R1xhxHL3.QNѮ"= =3m L\ A$#|.vM3*;M@-W|S<1]n]j G؄J \2 d)L鐄Aa+- 23 c}-CYFcez N5'90GڒqGD#k4u/A*41SOdIpvz\:eMsT0zH#>MQ&톝6B 4/$&s-C :"  ne6Cf~s4H(-O\6MQ @ DPaCAD=QD'T@F `IqF%M0J-]S̕38 JI4Yd' $1ȡ'>U) JM^'O~8X% Z`StBm!s 7葨`RFZ^xpǕ-_~,x $h)dqb&Ғ]~"쁔 eGF vBL*DG2TkPHPRr62h⌠Ö=1 )$@Ǐl!7+9/I43), 6Q0JsP2;/PԋB)d~j#LH02?$N@LkC{h!H|p؈$,5\ $*|oD$ zRŊ@Բ-=J*j 5Zh) d@6$+)tI(+8>tKG- @+N61[)HD ŃE%&hDSuŵ9RCh1;|$<=ܬT=%]u"YCV%sPlB(Vex:@H\tZjMq A`P nkFÍI;dTi-.0Ҳ VD?{ٽ%?p8,JQ' 譍Pcxc )e͍@9d#$5-4А@+jEnZSK^I#0>us&#z%SAT83dOzdcAVNhJ??N$s ^2_yRb*QC3JpC0 2H87)%ȶZNJO3@ZIP FC)ؠSF."Xӌ娞d{!LX#bBB\hm=D I"H7Dqi~!<pfb s  `$vVƒi\#GGӍ[ZB2Ad'''29,, L",aQC9yC3SUCS+R9LA;ZuzAE$(K"5e~#KrȻ}%A 2=v]W94 $w'I'+q|6ӶpTӒJL&y >ڤI *\ByvDc% ˀ&dStU$0 ZBR L0p l3e. I/3h9$uRRshdI$Ҧ0?,^.V<Ip*ȓP:q$d(HTB6dq$mS.ID9QVӒ`V׃8y ɺ) Y|bSe-x +P`q4Q,^Ϣp/1;؁neRX,"b!/B[|xJX7&%iM[ɢWqD2 qA!hAR H-/} U#c~$]dLGN pA@r5H˕o6-);&p3U;lЦ{2X0IiOaHhkXD@$$NΆ c@R&ky&CPoXHe" գIbM- z ܈"pCqAe g8H[K$@iX$D2M5"Ie`*Si3a8CWR#ǁ4W7U&8:l[HLRv,YS\ML`i0,^{ `=qY  rϗ$цx%I@i!< g`ƛo}JMD˧d &58D( q0'&UqCbݢ8~),*.yBYso2rLZ.:]<'+$sCҦXY$}eR:EiKM&[H V]n CvVY7ʋ"X>)E rLu &M_鳗};O·byZݐ*pO^=m?y<3^I&42sY#tⷳxBӝ&)>2m&܁4C g]*O;[V;p;+1(p[ #s r KJ?Q5- c72 K 68rc7389+Zg:YÏ8 +H3-@*2 6 Lkp* + ;  8d )$XzA/< AA4C;!Zy4-D; *yz"+lc7 P QPB4DyC(A@P\z# lꔫHB83@>DAPMȸ#zʠCN0X>>{]|)EГ N4 CʵG)AH =TF7؋ op|9G  ,ĈO" @`#P4P,|2rNFc `HEU<  =#F3S[Is|{60,A GPA $ Ès:#ȕDK7ěD%~!@bԫ%"II]ˮHLXǀңKĘڵ)$k Nh͔ !l6Nh?pM (M=QJX: \ ̀JƇK; d >H ,@-! t J" "Ő8y+ قHpNk ˕وχh ߑtMeLM}JDIJ5c.4ϐNuMG9Y"АpKˆU~NNG)iH`4]s=jU@ eS2 v^`+{V19iq\nhH$hpxGGf(l(LXBzwzvu@r@Jpt'WsoQ<7Un+pG2g⫐r x1DDnkwm]4FM(#% ,*CJ9[raXppwdZ>g?xb/7[NO"wXf\,J`zGIo&rz`Z?=ǃĕS!|f(wϴJ_M0km>zCuM"ZM =` u q‰뚯XO# 'ʐ4 ,z]7`O9xn(A|׫tϕ#J7՝B5X[  aDoy7L˭\'PN ?Ar|tM2;Wo-@AD,h „ 2la&Rh"ƌ7rh "GB@UhW!DHI(yz,@/M$3ȄWm'#=Y4ᛕh.4!ׄL/aJhj:s\:MOɧ1>44H]QTEBQBL)MHͪQHv4|HPD] *!Rn*1G%Fn' 9|hG(yjs@g1;J#)grخxH { (U1 A7P9l5 YX6+f$UA.A.кsCS!I /t/#]!(AFԠPq-fOrE/cQ"h\K¿ixEbNoYt$ޢdaA垰ěqf9,U$ hI; '"*ں̄6_įїAhcjM̰yp Np+A UFE*_??gH*-?ora|`Ӂ{_iԓ`)`1`w A``[JRMhViM ĝ Z >} .D%Nm'Y!B|[ ։K9ҷ`O^!@IQaXa/E_>1 _",_A N1 " a!!*!&" #b 2!."R%Zb&n&*b$!&&b$j$"""*rb)z"&^*R+"b%")"$,6"'2b,b/0"A#2&2.#363>#4F4N#5V5^#6f6n#7v7~#88cZ9njji;#eB 멆1?#Ņ$!A(UIBAaTB` СAEDN]d!&dFdN&eVe^&f6af&gRfgvhZhi YijNH(D#(B ?TRܤ&qDkV#؁hD%%P@xaz wZ%AZeZX#wN'dOjIw'J穄B sngYV}XDJ@m(u (Jc*@w"h}'~c̀ GZ hUpAxB愒GhqN'h{(I0 (2(>h H HhtO''ezFza'hBǨ $vhth>mJ(At6P AEH2 8JtD!\kNZȢ KjGF*:DjV^Ǫf*Dp֪)i *+RH;gpml-matlab-v3.6-2015-07-07/doc/usageSampling.m000644 000765 000024 00000005114 12076017514 020614 0ustar00carlstaff000000 000000 clear all, close all n1=10; n2=5; % number of data points from each class n1/n2 sc=1; % change the scale of the initial problem % 1) draw data from two Gaussians S1 = eye(2); S2 = [1 0.95; 0.95 1]; m1 = [0.75; 0]; m2 = [-0.75; 0]; randn('seed',17); rand('seed',17) % coordinates x1 = chol(S1)'*randn(2,n1)+repmat(m1,1,n1); x2 = chol(S2)'*randn(2,n2)+repmat(m2,1,n2); x1(:,5) = x1(:,2); x1(:,1) = x1(:,4); x2(:,1) = x2(:,2); % rank deficiency in K xtr = [x1 x2]'; % class labels ytr = [repmat(-1,[1,n1]) repmat(1,[1,n2])]'; tt = (-5:.3:5)*sc; [t1 t2] = meshgrid(tt,tt); x1 = x1*sc; x2=x2*sc; xtr=xtr*sc; m1=m1*sc; m2=m2*sc; t = [t1(:) t2(:)]; z1 = exp(-sum((t-repmat(m1',length(t),1))*inv(S1).*(t-repmat(m1',length(t),1)),2)/2); z2 = exp(-sum((t-repmat(m2',length(t),1))*inv(S2).*(t-repmat(m2',length(t),1)),2)/2); xte = t; clear t xtr(end,:)=[2.9,2.3]; ytr(end)=-1; % modif % 2) set GP parameters cov = {@covSum,{@covSEiso,@covNoise}}; hyp.cov = [0; 2; -Inf]; % ell,sf,sn lik = {'likLogistic'}; hyp.lik = []; % likLogistic or likErf mn = {'meanZero'}; hyp.mean = []; inf = 'infEP'; % 3) predict using approximated inference [nlZ,dnlZ,post] = gp(hyp, inf, mn, cov, lik, xtr, ytr); [ymu,ys2,fmu,fs2,junk,post] = gp(hyp,inf,mn,cov,lik,xtr,post,xte); % 4) prediction using MCMC sampling % set MCMC parameters, see some more details in inf/infMCMC.m % We have two samplers implemented, namely % hmc - Hybrid Monte Carlo, and % ess - Elliptical Slice Sampling. par.sampler = 'hmc'; par.Nsample = 20; % par.sampler = 'ess'; % par.Nais = 5; par.Nsample = 200; par.Nskip = 100; tic [posts,nlZs,dnlZs] = infMCMC(hyp,mn,cov,lik,xtr,ytr,par); toc [ymus,ys2s,fmus,fs2s,junk,posts] = gp(hyp,@infMCMC,mn,cov,lik,xtr,posts,xte); % 5a) echo results fprintf('nlZ-EP=%f, nlZ-AIS=%f\n', nlZ, nlZs) fprintf('acceptance rate (MCMC) = %1.2f%%\n',100*posts.acceptance_rate_MCMC) for r=1:length(posts.acceptance_rate_AIS) fprintf('acceptance rate (AIS) = %1.2f%%\n',100*posts.acceptance_rate_AIS(r)) end % 5b) print results figure subplot(221) plot([-12,12],[-12,12],'r'), hold on, plot(fmus,fmu,'k.'), title('\mu_f') xlabel('MCMC'), ylabel(inf) subplot(222) plot([0,10],[0,10],'r'), hold on, plot(sqrt(fs2s),sqrt(fs2),'k.') title('\sigma_f') xlabel('MCMC'), ylabel(inf) subplot(223) plot([-1,1],[-1,1],'r'), hold on, plot(ymus,ymu,'k.'), title('\mu_y') xlabel('MCMC'), ylabel(inf) subplot(224) plot([0,1],[0,1],'r'), hold on, plot(sqrt(ys2s),sqrt(ys2),'k.') title('\sigma_y') xlabel('MCMC'), ylabel(inf)gpml-matlab-v3.6-2015-07-07/doc/demoClassification.m000644 000765 000024 00000011117 12076017514 021615 0ustar00carlstaff000000 000000 disp('See http://www.gaussianprocess.org/gpml/code/matlab/doc/ for details.') disp('Hit any key to continue...'); pause disp(' '); disp('clear all, close all') clear all, close all write_fig = 0; disp(' ') disp('n1 = 80; n2 = 40; % number of data points from each class') n1 = 80; n2 = 40; disp('S1 = eye(2); S2 = [1 0.95; 0.95 1]; % the two covariance matrices') S1 = eye(2); S2 = [1 0.95; 0.95 1]; disp('m1 = [0.75; 0]; m2 = [-0.75; 0]; % the two means') m1 = [0.75; 0]; m2 = [-0.75; 0]; disp(' ') disp('x1 = bsxfun(@plus, chol(S1)''*gpml_randn(0.2, 2, n1), m1);') x1 = bsxfun(@plus, chol(S1)'*gpml_randn(0.2, 2, n1), m1); disp('x2 = bsxfun(@plus, chol(S2)''*gpml_randn(0.3, 2, n2), m2);') x2 = bsxfun(@plus, chol(S2)'*gpml_randn(0.3, 2, n2), m2); disp(' ') disp('x = [x1 x2]''; y = [-ones(1,n1) ones(1,n2)]'';') x = [x1 x2]'; y = [-ones(1,n1) ones(1,n2)]'; figure(6) disp('plot(x1(1,:), x1(2,:), ''b+''); hold on;'); plot(x1(1,:), x1(2,:), 'b+', 'MarkerSize', 12); hold on disp('plot(x2(1,:), x2(2,:), ''r+'');'); plot(x2(1,:), x2(2,:), 'r+', 'MarkerSize', 12); disp(' ') disp('[t1 t2] = meshgrid(-4:0.1:4,-4:0.1:4);') [t1 t2] = meshgrid(-4:0.1:4,-4:0.1:4); disp('t = [t1(:) t2(:)]; n = length(t); % these are the test inputs') t = [t1(:) t2(:)]; n = length(t); % these are the test inputs disp('tmm = bsxfun(@minus, t, m1'');') tmm = bsxfun(@minus, t, m1'); disp('p1 = n1*exp(-sum(tmm*inv(S1).*tmm/2,2))/sqrt(det(S1));') p1 = n1*exp(-sum(tmm*inv(S1).*tmm/2,2))/sqrt(det(S1)); disp('tmm = bsxfun(@minus, t, m2'');') tmm = bsxfun(@minus, t, m2'); disp('p2 = n2*exp(-sum(tmm*inv(S2).*tmm/2,2))/sqrt(det(S2));') p2 = n2*exp(-sum(tmm*inv(S2).*tmm/2,2))/sqrt(det(S2)); set(gca, 'FontSize', 24) disp('contour(t1, t2, reshape(p2./(p1+p2), size(t1)), [0.1:0.1:0.9])') contour(t1, t2, reshape(p2./(p1+p2), size(t1)), [0.1:0.1:0.9]) [c h] = contour(t1, t2, reshape(p2./(p1+p2), size(t1)), [0.5 0.5]); set(h, 'LineWidth', 2) colorbar grid axis([-4 4 -4 4]) if write_fig, print -depsc f6.eps; end disp(' '); disp('Hit any key to continue...'); pause disp(' ') disp('meanfunc = @meanConst; hyp.mean = 0;') meanfunc = @meanConst; hyp.mean = 0; disp('covfunc = @covSEard; hyp.cov = log([1 1 1]);') covfunc = @covSEard; hyp.cov = log([1 1 1]); disp('likfunc = @likErf;') likfunc = @likErf; disp(' ') disp('hyp = minimize(hyp, @gp, -40, @infEP, meanfunc, covfunc, likfunc, x, y);') hyp = minimize(hyp, @gp, -40, @infEP, meanfunc, covfunc, likfunc, x, y); disp('[a b c d lp] = gp(hyp, @infEP, meanfunc, covfunc, likfunc, x, y, t, ones(n, 1));') [a b c d lp] = gp(hyp, @infEP, meanfunc, covfunc, likfunc, x, y, t, ones(n,1)); disp(' ') figure(7) set(gca, 'FontSize', 24) disp('plot(x1(1,:), x1(2,:), ''b+''); hold on') plot(x1(1,:), x1(2,:), 'b+', 'MarkerSize', 12); hold on disp('plot(x2(1,:), x2(2,:), ''r+'')') plot(x2(1,:), x2(2,:), 'r+', 'MarkerSize', 12) disp('contour(t1, t2, reshape(exp(lp), size(t1)), 0.1:0.1:0.9)') contour(t1, t2, reshape(exp(lp), size(t1)), 0.1:0.1:0.9) [c h] = contour(t1, t2, reshape(exp(lp), size(t1)), [0.5 0.5]); set(h, 'LineWidth', 2) colorbar grid axis([-4 4 -4 4]) if write_fig, print -depsc f7.eps; end disp(' '); disp('Hit any key to continue...'); pause disp('large scale classification using the FITC approximation') disp('[u1,u2] = meshgrid(linspace(-2,2,5)); u = [u1(:),u2(:)];') [u1,u2] = meshgrid(linspace(-2,2,5)); u = [u1(:),u2(:)]; clear u1; clear u2 disp('nu = size(u,1);') nu = size(u,1); disp('covfuncF = {@covFITC, {covfunc}, u};') covfuncF = {@covFITC, {covfunc}, u}; disp('inffunc = @infFITC_Laplace;') inffunc = @infFITC_EP; % one could also use @infFITC_Laplace disp('hyp = minimize(hyp, @gp, -40, inffunc, meanfunc, covfuncF, likfunc, x, y);') hyp = minimize(hyp, @gp, -40, inffunc, meanfunc, covfuncF, likfunc, x, y); disp('[a b c d lp] = gp(hyp, inffunc, meanfunc, covfuncF, likfunc, x, y, t, ones(n,1));') [a b c d lp] = gp(hyp, inffunc, meanfunc, covfuncF, likfunc, x, y, t, ones(n,1)); disp(' ') figure(8) set(gca, 'FontSize', 24) disp('plot(x1(1,:), x1(2,:), ''b+''); hold on') plot(x1(1,:), x1(2,:), 'b+', 'MarkerSize', 12); hold on disp('plot(x2(1,:), x2(2,:), ''r+'')') plot(x2(1,:), x2(2,:), 'r+', 'MarkerSize', 12) disp('contour(t1, t2, reshape(exp(lp), size(t1)), 0.1:0.1:0.9)') contour(t1, t2, reshape(exp(lp), size(t1)), 0.1:0.1:0.9) [c h] = contour(t1, t2, reshape(exp(lp), size(t1)), [0.5 0.5]); set(h, 'LineWidth', 2) disp('plot(u(:,1),u(:,2),''ko'', ''MarkerSize'', 12)') plot(u(:,1),u(:,2),'ko', 'MarkerSize', 12) colorbar grid axis([-4 4 -4 4]) if write_fig, print -depsc f8.eps; end disp(' ')gpml-matlab-v3.6-2015-07-07/doc/f3.gif000644 000765 000024 00000033052 12076017514 016640 0ustar00carlstaff000000 000000 GIF89a !!!"""&&&(((***---...///000111222333555888:::<<<>>>???@@@AAADDDKKKMMMNNNOOOPPPQQQRRRSSSUUUYYY[[[\\\]]]^^^___```aaabbbcccdddfffggnhhhiiijjjkkklllmmmnnnooopppqqqrrrssstttuuuwwwxxxzzz}}}~~~^^WW99 >> !!%%'' //2211447766==BBFFHHMMOORRUUVV\\aacc``jjkkppppttvv񀀀Ӂ파읝롡뫫鹹!,k H*\ȰÇ#JHŋ3jȱǏ CIɓ(S\ɲ˗0cʜI͛8sɳϟ@ JѣH*]ʴӧPJJիXjʵW'PKٳhӪ]˶[&Kݻt(߿ˊ-+ԪW+^̸ǐ#Kf< ǫ&|aW>>SDI/L^ͺkȞ qYB@.(dwDJ}?D-ȓ+oȫ6D#oN}!z 3mla蔆47DH>AϿql$a&zrB)bC .Pr\H`!z3 (-8(I(׆H*䡁hipX<Ht6BCLKL6$yg"YKGfib`JsRXuh|cKW&;ۚxJ oJlhbcLaenH}V%HL!)hFebFVY#XZ-JF"lhWvU덷V&E[&VG&ud'Iv.{f;':fsB&DT+ &fãڶ޺a@ʝ嚩X^{$RrB`"']P1͸FTw$&Bv bw "E=AAqbH@raXu0W;mb'P.͜"۪f"֑nacuKRi3h+(z(c46Ke#vAxlSkc+ y'"uMҍ-t{V=8^7v ă 銧Rvcw*~Q,*D{J{ĸAcrEbzRԘT%N*ĊvʊG ή!D@NQZ!UCMZdMFG.pkiRc慞Ft "J3TbɘWRp&bnyMbIt $HJƲ1+L4 |GGItMY}T N2t%D)n"dVI|/1Otc LgZ57~9Lk1{q3Y'9NyΓ9?1:T:"/ayP*NL;*cR#BQg1Iu<"Fe,d&Ǭt83qg H-IibN J$>@p*yiίO̩bӓ"kYChdV}X4Ҩeje_@^xA!QŜB5 `  puHA7xA6dYOdjJۛ֨ UzJ MTfZiblXRDk@ՠ+<ʢ&i4Éxw"lAJPɔE߁wpf\ p[R%aOvp! kp3b "NUa?5m8 bU"g9^F@UercJyF2M{gT*F} pɫA3UQr1ucjI"l cŐ_1K#k[T$n0d"= a~Wù䡹$2&깾s.1Mv2IJX;o ȞRQY@!\5^:1@-e[B/R-4 +N[FuA&˄M)ZN 9Ff]H3 gDǼ۠ MH&$nm?En&]MtLX%ce#ܺ,$Tj'CWdc|yt:e|Dvh[#FI }<#tL4E<:D]&(W%#xR'B4HU$)X_&V|5h!#զ8 C.3CX2zE|@)1;KdT>F~Io^"rdXyݓUu!2<5=Eb19NJ5c'#м+3#{ w-r$}ʗ"Ο+BihW#B g b([`(/!Pvnv"W&HrW|dC{t'* q\$5eT2\n"yxDz4@aq/?X6:w DaD|8gT}}ڷǷ%?4SJ3`%gq,q}g{8pTW}B}\9.Y2gNa@QLY؄QA`*=6A񀿔p[({㑁j]H0>dD'|s&pt0Z$]r&H7IX+m+asFcQq(H!665:ACF4~9wLW{P!{0эq4M;#8Dk.7w2Ň/QoFl!w#498Wx5yLXΈ^ *)^q%7 ;0N 9d 2 2. HzOs uHCExKY _Ï%uomxW*KД ,B/D7Q玉<H"%xx2!F F' M8i G(hC?%zOP>/hjG3ԧw q2 2$Oi1AvHqbAN Uɜ,Y!x{3 e)9,qMcihI!etgI"z9 C<)KI9*< :x~WGhSS*QS2xH*zT,щ6'QoI|\K  (G#ኃ)~Zz?7irafJq,cT5z)]aViNo*p)=*tyV1?,"4 1p'ERxg9T*3^?ĚPGu^Dv a>t1|AB5L.q}CíԅxVwd")jRm,288<Q!! Pe)كjrQMH%!Z6ۄ/j5lljkYFtB~m HޙLf D 5Fpð;Ts:Y#4 VXVmh0CJF^x -])whYM}|~Kwp֖Vݓg?#BuZ"cCx @z}@e,M㓣 >Jr:p^$phNK[,8> *A$DWǻHV͋55%0=BC7b[bb0bb6k_nzv@ _cg'qE<bTqpcB< ALhM$i:~Q!Įik0| 5dH"1/Vۋ}ས Ŧaqu<I&5xgSU"?}ƪ b裱}PɀT k(z  ?>~l0l$><@w?\ܧIjuᴖQ,:$>ߴzxItZN|!ZGsEZ&գ8 A $Khq#i+k$EL  cuB̤w1Mc]`@ FV 5(: A1&HÓa]qargAwQ4!aز7MFFb*ȽJu V͙ͯͩ`M>RG 9][}sNag ^9Ӊq˝d8aK]1lW؆M[;|cx'pBL&ƞ$C ћMƝ;L׎18Y+۝޽QTaD!Ğӯ -MP;*HI3)m-MAh8B$--\W+nF⨴2˚LqPeZYU qՊ CdwyสiԜN֨&Q˪ڕBl8׼&E-M>ϱ3v 6쯑惘}G1=qz _J-wX&JLKT'#q銁!$:0g)ɡn5b~3 1)%-U $펤!N 1k;T^p{XgxJ.2P&/P O눱U4>^JDE <-&&&ލ1=K OL [ i$)/qwGUO!AEpcmgwzzF O2.lncrLL, j8+  d-s6=`1ONy % |v'^nB )qWwYQK@ $HA@$B[zQF=~H%2RJ-k KEt)HڀTPEAhP=8+*,y p*R@iVXe X NO0K,qU^EYmWaa_u:-_lNk9KhFF+R+><2Y^Pn޽MR'PcSp?/Ԇ"Y]QJ{;PI5^%̍HnĠTp7y <^?W֩'O$@x1Rԃ0¬@h^yd"Hc0ĠP03+bA1Oč&{MBw$L+}$:Cmd#S2c +2˖jl2x3L㈸B4z1Te#R/erH44W685/5?.5@-EVIVs1ΒfC 1>uW42k-US>rŴTW}6pVSU6nx_9pԍHy}UNb !&wT1#7u7 XO=BU#'l&,3DS);H0&dUfFϴt9Ervf2}/K=:lJk4nިh'7#hx(i\-:;p䨌T [ǤNLV-Rƻ)!_PQ`ҠیO<ч=9rrՆC81ߛxh ϭjQO\ I# r;o`knA߷:0a'/ГDmbŴr[q*!_HZXG°j  HĢ %J6h:AWx,o?`pr _Q ;ؒqeDk`Z-t#JR+ R8]g~Yіm(%җ*J(ZdD( 2Df.)е' *`tLzm'$VQlHO0!媇2ʹ!Zw;BqB܎Ȓ a w$U&9cA2)pt#r!DRfBw@{I-a""( ]B)i)u[UGSdL"AODyGeG$RL{fJ\`"@X(U̓H,VD1'#(<)Fihf!49g! -u45bЁGuxK3>ONmZI8e0.kD*|P%H@B\;7"ρBH's*Vl \lڥ"h { eH =H m@0;&znlf/ ъx=i7+rM-lW٦u| D[٢vekZXŬN( *ԏ+:>/du-;t,(x$ tFv6V]sP bɾH,hn_< dyA׽Ɛ]A4`-HDi;`F[^<1 (_!IuRX ua D0Mxj)E1 )QE⋟8\ h DҬXBc^AˆʄH0a)at0;9^@C&Rʰ A e``. t&v:h|37#n>)yquݜ@# ĺ)(-C! FFrckԌlBxMܡe+NЇnي nN4pH PX ];nÇ}jLƉwFHDrD _J+"u!B숭.NaY#U[N+Dmn xI du? ^AQZj_Ʃ+PXڞa.8[)X<M%Id`P  %`y\Nʺ*EnN{!bݎzԫŠ@"(\,4Ң }t}[*#'i{dT7p R \@MqFw0Nvzzh ]lJWK(/r$R@Z#4]_s)%ħ7~QND!>^0{H̦cC6hF2{)?3{-s#E}^}&BY,8?h1 p>aU [R<7o@K,@[ - TU|[ Zs = @q/ 89ŠP; eb6$1M;"̕lP >:>.ڑ 6$H0W8i+d>3C" 0q;𪱠N2@{ =C&Y*Rxåı:{rJވ) LD, ; 1A*d?/:9YL 99cġp>3 0 9% Z -z9-̡?L1lO$2 :ؿkCP2GZ %c^ TLR ǮW+ $xbD HmWpy,X8Wl +d 0d1KXcI0\Z䍿tư@Z)A 1C \J n\QD 60X2"6ȏUpωŵ GJy s&z M0sQ̒AP NȏxR̓گɘހ@ΩXA܍PK%au)hH qNGjѹ b@G=UK#)3HR@JU`4 SviƒFRjdl+!KMOx#SQӈGr><T&Yw)ɳB䯌8EtK /}bح-b׎xb&FLXJ64f$[  <D]bCA.gd`eFե 8R0{l~cNf0e AQ;e批d&:HZa~\ؠ(DHN`&nr孻a] \hM \!N@3lgGfL,02,di3GS׌j^ ImUЀ RxlZ AuCkX'%OXa8V­*& :433/8h¯U6\@NX5 iW5M,Vb\ܾXfr1瘨ֈ-1X)`.D쥄`~ {952hm$AލUx6 hދ% XX֌a.֕;ꮶRh(!UpoB䡈 l–FXeo crY!@yHY8b)Bselzc2halq0= > Q$T8 [!s$Z Sx_8U8z4!R(xt!0PgMHgBn]KT5g ;B`d d!H8umGI0ti݈8D qrsH'*H[N.[$xL=[[ kO 82.-;7Gy?yO".RDyS  hN04}2F| 51P>@:o dWl EW('@k7ZȘq+,ZJi؎'r<Nh2R]OnJQR/Sw 0sfWA[93wv|{nXhϗ8*;q ,Jwhs{5K NӑR0s va'&rd2w}7$ W@ԉp@nڗhW,h „ &Ĉ'Rhb- 7rE^Ô 貖>*g|Ec˜:%jU;-j4J2-PG5Jdә F!kSP-Z]tdײeaظr %ջC_ֵ}Aޡu|, Jq:},l9+ Ȓ)ewp*t§=QfڵXƈ4זعwE_v5OΧ>:3nxnnlADJ)V=| (A^է^\`Yn)" rk)tu_h'h&Xr`PGJV)!&GI!=lgv…'kÁr`B'#bËA5ZAEB^B4gg%MLda4($j\ 'D(mYd } 9G#u6ki))% Ș؆UL{vAwNG\yVyQuUJZ6)u:\u Z埝hũWq\{)*Vd_(s6e"rYGۂeH ܳYf|b.!\},|]V!^V0%KRa"^J{a]q%o-Mi)])`& \|zRuR&|3WG#] "lY]bJBt\[O4qm>K\W {=7k(=,'_d TJ}Z=}wBho\@-Ax/❳3%=O  hIuGR)9X~WJ~^z; |V#?=y.PRέ!QR=^Po\k84ʽM4.c?&t~ @ p1޷7-&JP., +6 !`mjt.i!.&0a, sOQ߲r鰈r o:Kg)6I|Es?)f*Fu`EHNt&6G.Yρ4!nH *bQOtq)>@!#9٢gW"ᎊDEGD0.$*CQN*RY4~0F?R%0{C ƌo#K3J@&rJ2,y2Eh)3x '?=)I@v\^bBEG mM RG`P41fN\32l ԥ2N}*T*թRU*VխrMW*ֱfU)"D'\*׹ҵv+^׽~+`+=,bXSHHFM ],Rl`Xd+ ft8 ִjs[D"/ֶ|P}A.׸ԫn%Օ("v{Zyŋ*м !{;Ͻ}|}g_8n".uKSOwkׂ͝s%Yx"cblx IF>2%3N~2,)S,C2;wel9^>q,uyffri[IֻrA[ O-.AO%}45V"yΡU4~_nQngAsz(IDZ/0N,RiRGz(Ep&ʭӶ X,nY`W"@~2|eZ 6![w&PR+@Em^c%X~#B[ "s]l(ބ7 MorNZK"NX9":9~_^{ҏo#e%\/B4AT<KH@;gpml-matlab-v3.6-2015-07-07/lik/likSech2.m000644 000765 000024 00000020502 12546444606 017501 0ustar00carlstaff000000 000000 function [varargout] = likSech2(hyp, y, mu, s2, inf, i) % likSech2 - sech-square likelihood function for regression. Often, the sech- % square distribution is also referred to as the logistic distribution not to be % confused with the logistic function for classification. The expression for the % likelihood is % likSech2(t) = Z / cosh(tau*(y-t))^2 where % tau = pi/(2*sqrt(3)*sn) and Z = tau/2 % and y is the mean and sn^2 is the variance. % % hyp = [ log(sn) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. The moments % \int f^k likSech2(y,f) N(f|mu,var) df are calculated via a Gaussian % scale mixture approximation. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-09-02. % % See also LIKFUNCTIONS.M, LIKLOGISTIC.M. if nargin<3, varargout = {'1'}; return; end % report number of hyperparameters sn = exp(hyp); tau = pi/(2*sqrt(3)*sn); lZ = log(pi) - log(sn) - log(4*sqrt(3)); if nargin<5 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>3&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability evaluation lp = lZ - 2*logcosh(tau*(y-mu)); s2 = 0; else % prediction lp = likSech2(hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 ymu = mu; % first y moment if nargout>2 ys2 = s2 + sn.^2; % second y moment end end varargout = {lp,ymu,ys2}; else % inference mode switch inf case 'infLaplace' r = y-mu; [g,dg,d2g,d3g] = logcosh(tau.*r); % precompute derivatives if nargin<6 % no derivative mode dlp = {}; d2lp = {}; d3lp = {}; lp = lZ - 2*g; if nargout>1 % first derivatives dlp = 2*tau.*dg; if nargout>2 % 2nd derivative of log likelihood d2lp = -2*tau.^2.*d2g; if nargout>3 % 3rd derivative of log likelihood d3lp = 2*tau.^3.*d3g; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode lp_dhyp = 2*tau.*r.*dg - 1; % derivative w.r.t. sn dlp_dhyp = -2*tau.*(dg+tau.*r.*d2g); d2lp_dhyp = 2*tau.^2.*(2*d2g + tau.*r.*d3g); varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' n = max([length(y),length(mu),length(s2),length(sn)]); on = ones(n,1); y = y.*on; mu = mu.*on; s2 = s2.*on; sn = sn.*on; % vectors only fac = 1e3; % factor between the widths of the two distributions ... % ... from when one considered a delta peak, we use 3 orders of magnitude idlik = fac*sn1) and nu*sn^2/(nu-2) is the variance (for nu>2). % % The hyperparameters are: % % hyp = [ log(nu-1) % log(sn) ] % % Note that the parametrisation guarantees nu>1, thus the mean always exists. % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-09-02. % % See also LIKFUNCTIONS.M. if nargin<3, varargout = {'2'}; return; end % report number of hyperparameters numin = 1; % minimum value of nu nu = exp(hyp(1))+numin; sn2 = exp(2*hyp(2)); % extract hyperparameters lZ = gammaln(nu/2+1/2) - gammaln(nu/2) - log(nu*pi*sn2)/2; if nargin<5 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>3&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability evaluation lp = lZ - (nu+1)*log( 1+(y-mu).^2./(nu.*sn2) )/2; s2 = 0; else % prediction lp = likT(hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 ymu = mu; % first y moment; for nu<=1 this is the mode if nargout>2 if nu<=2 ys2 = Inf(size(mu)); % variance does not always exist else ys2 = (s2 + nu*sn2/(nu-2)).*ones(size(mu)); % second y moment end end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' r = y-mu; r2 = r.*r; if nargin<6 % no derivative mode dlp = {}; d2lp = {}; d3lp = {}; lp = lZ - (nu+1)*log( 1+r2./(nu.*sn2) )/2; if nargout>1 a = r2+nu*sn2; dlp = (nu+1)*r./a; % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = (nu+1)*(r2-nu*sn2)./a.^2; if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = (nu+1)*2*r.*(r2-3*nu*sn2)./a.^3; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode a = r2+nu*sn2; a2 = a.*a; a3 = a2.*a; if i==1 % derivative w.r.t. nu lp_dhyp = nu*( psi(nu/2+1/2)-psi(nu/2) )/2 - 1/2 ... -nu*log(1+r2/(nu*sn2))/2 +(nu/2+1/2)*r2./(nu*sn2+r2); lp_dhyp = (1-numin/nu)*lp_dhyp; % correct for lower bound on nu dlp_dhyp = nu*r.*( a - sn2*(nu+1) )./a2; dlp_dhyp = (1-numin/nu)*dlp_dhyp; % correct for lower bound on nu d2lp_dhyp = nu*( r2.*(r2-3*sn2*(1+nu)) + nu*sn2^2 )./a3; d2lp_dhyp = (1-numin/nu)*d2lp_dhyp; % correct for lower bound on nu else % derivative w.r.t. sn lp_dhyp = (nu+1)*r2./a - 1; dlp_dhyp = -(nu+1)*2*nu*sn2*r./a2; d2lp_dhyp = (nu+1)*2*nu*sn2*(a-4*r2)./a3; end varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargout>1 error('infEP not supported since likT is not log-concave') end n = max([length(y),length(mu),length(s2)]); on = ones(n,1); y = y(:).*on; mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only % since we are not aware of an analytical expression of the integral, % we use Gaussian-Hermite quadrature N = 20; [t,w] = gauher(N); oN = ones(1,N); lZ = likT(hyp, y*oN, sig*t'+mu*oN, []); lZ = log_expA_x(lZ,w); % log( exp(lZ)*w ) varargout = {lZ}; case 'infVB' % variational lower site bound % t(s) \propto (1+(s-y)^2/(nu*s2))^(-nu/2+1/2) % the bound has the form: (b+z/ga)*f - f.^2/(2*ga) - h(ga)/2 n = numel(s2); b = zeros(n,1); y = y.*ones(n,1); z = y; varargout = {b,z}; end end % computes y = log( exp(A)*x ) in a numerically safe way by subtracting the % maximal value in each row to avoid cancelation after taking the exp function y = log_expA_x(A,x) N = size(A,2); maxA = max(A,[],2); % number of columns, max over columns y = log(exp(A-maxA*ones(1,N))*x) + maxA; % exp(A) = exp(A-max(A))*exp(max(A))gpml-matlab-v3.6-2015-07-07/lik/likInvGauss.m000644 000765 000024 00000011107 12546444606 020275 0ustar00carlstaff000000 000000 function [varargout] = likInvGauss(link, hyp, y, mu, s2, inf, i) % likInvGauss - Inverse Gaussian likelihood function for strictly positive data % y. The expression for the likelihood is % likInvGauss(f) = sqrt(lam/(2*pi*y^3))*exp(-lam*(mu-y)^2/(2*mu^2*y)) with % mean=mu and variance=mu^3/lam where mu = g(f) is the Inverse Gaussian % intensity, f is a Gaussian process, y is the strictly positive data. % Hence, we have -- with log(Zy) = -(log(lam)-log(2*pi*y^3))/2 % llik(f) = log(likInvGauss(f)) = -lam*(y-mu)^2/(2*mu^2*y) - log(Zy). % The larger one chooses lam, the stronger the likelihood resembles a Gaussian % since skewness = 3*sqrt(mu/lam) and kurtosis = 15*mu/lam. % % We provide two inverse link functions 'exp' and 'logistic': % g(f) = exp(f) and g(f) = log(1+exp(f))). % The link functions are located at util/glm_invlink_*.m. % % Note that for neither link function the likelihood lik(f) is log concave. % % The hyperparameters are: % % hyp = [ log(lam) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % See also LIKFUNCTIONS.M. % % Copyright (c) by Hannes Nickisch, 2013-10-16. if nargin<4, varargout = {'1'}; return; end % report number of hyperparameters lam = exp(hyp); if nargin<6 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>3&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability lg = g(mu,link); elg = exp(lg); lZy = -(log(lam)-log(2*pi*y.^3))/2; % normalisation constant lp = -lam*(y./elg-1).^2 ./(2*y) - lZy; else lp = likInvGauss(link, hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 % compute y moments by quadrature n = max([length(y),length(mu),length(s2)]); on = ones(n,1); N = 20; [t,w] = gauher(N); oN = ones(1,N); lw = ones(n,1)*log(w'); mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only lg = g(sig*t'+mu*oN,link); ymu = exp(logsumexp2(lg+lw)); % first moment using Gaussian-Hermite quad if nargout>2 elg = exp(lg); yv = elg.^3/lam; % second y moment from inverse Gaussian distribution ys2 = (yv+(elg-ymu*oN).^2)*w; end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' [lg,dlg,d2lg,d3lg] = g(mu,link); elg = exp(lg); if nargin<7 % no derivative mode lZy = -(log(lam)-log(2*pi*y.^3))/2; % normalisation constant lp = -lam*(y./elg-1).^2 ./(2*y) - lZy; dlp = {}; d2lp = {}; d3lp = {}; % return arguments if nargout>1 dlp = lam*(y./elg-1).*dlg./elg; % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = lam*( (y./elg-1).*(d2lg-dlg.^2) - y.*dlg.^2./elg )./elg; if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = lam*( (y./elg-1) .* (4*dlg.^3-6*dlg.*d2lg+d3lg) ... + (3*dlg.^3 - 3*dlg.*d2lg) )./ elg; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode lp_dhyp = 1/2-lam*(y./elg-1).^2 ./(2*y); % deriv. of log lik w.r.t. lam dlp_dhyp = lam*(y./elg-1).*dlg./elg; % first derivative d2lp_dhyp = lam*( (y./elg-1).*(d2lg-dlg.^2) - y.*dlg.^2./elg )./elg; % 2nd varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargin<7 % no derivative mode % Since we are not aware of an analytical expression of the integral, % we use quadrature. varargout = cell(1,nargout); [varargout{:}] = lik_epquad({@likInvGauss,link},hyp,y,mu,s2); else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' error('infVB not supported') end end % compute the log intensity using the inverse link function function varargout = g(f,link) varargout = cell(nargout, 1); % allocate the right number of output arguments if strcmp(link,'exp') [varargout{:}] = glm_invlink_exp(f); else [varargout{:}] = glm_invlink_logistic(f); endgpml-matlab-v3.6-2015-07-07/lik/likMix.m000644 000765 000024 00000020074 12240130514 017253 0ustar00carlstaff000000 000000 function [varargout] = likMix(lik, hyp, varargin) % likMix - Mixture of likelihoods for regression/classification. % The expression for the likelihood is % likMix(t) = sum_i=1..m w_i * lik_i(t), % where lik_i are the m individual likelihood functions combined by a weighted % sum with weights wi. % % The hyperparameters are: % % hyp = [ log(w_1) % log(w_2) % ... % log(w_m) % hyp_1 % hyp_2 % ... % hyp_m ] % % Note that the mixture weights are normalised using the softmax function, so % that 1 = sum w_i. Here, hyp_i are the hyperparameters for the individual % likelihoods. % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Hannes Nickisch and Rowan McAllister, 2013-10-23. % % See also LIKFUNCTIONS.M. m = numel(lik); % number of mixture components if m==0, error('We require at least one mixture component.'), end for i = 1:m % iterate over mixture components f = lik(i); if iscell(f{:}), f = f{:}; end % expand cell array if necessary j(i) = cellstr(feval(f{:})); % collect number hypers end s = @(x) logsumexp2(x); % define a shortcut to util/logsumexp2.m if nargin<4 % report number of parameters L = char(j(1)); for i=2:length(lik), L = [L, '+', char(j(i))]; end varargout = {['(',num2str(m),'+',L,')']}; return end mu = varargin{2}; n = length(mu); % number of query points if numel(varargin)>3 inf = varargin{4}; % varargin = {y, mu, s2, inf, i} else inf = ''; end v = []; % v vector indicates to which likelihood parameters belong nhyp = m; % number of hyperparameters; the first m are the mixture weights for i = 1:m ni = eval(char(j(i))); nhyp = nhyp + ni; v = [v, repmat(i,1,ni)]; end if nhyp>length(hyp), error('not enough hyperparameters'), end if strcmp(inf,'infVB'), error('infVB not supported'); end na = nargout; if nargout>1 && strcmp(inf,'infLaplace') && nargin>6, na = max(na,3); end argout = zeros(n,m,na); % collect output arguments from individual lik fun for i=1:m % iteration over mixture components f = lik(i); if iscell(f{:}), f = f{:}; end % expand cell array if necessary id = find(v==i)+m; % hyperparameter indices out = cell(size(argout,3),1); nn = min(nargin,6)-2; % do not ask for derivs [out{:}] = feval(f{:}, hyp(id), varargin{1:nn}); % ask individual likelihoods for j=1:size(argout,3), argout(:,i,j) = out{j}; end end lw = reshape(hyp(1:m),1,[]); lwn = lw-s(lw); % w = exp(lwn), sum(w)=1, weights one_lwn = ones(n,1)*lwn; % the same as above only that the weights are repeated varargout = cell(nargout,1); % allocate memory for output arguments if nargin<6 % prediction mode if inf is not present % lp and ymu outputs are of the weighted sum of integral structure if nargout>0, varargout{1} = s(one_lwn + argout(:,:,1)); end if nargout>1, varargout{2} = sum(exp(one_lwn).*argout(:,:,2), 2); end if nargout>2 % ys2 output computed using the parallel axis theorem argout3corr = argout(:,:,3) + (argout(:,:,2)-varargout{2}*ones(1,m)).^2; varargout{3} = sum(exp(one_lwn).*argout3corr, 2); end else if nargin>6, i = varargin{5}; end % make arguments available switch inf case {'infEP','infLaplace'} % the two cases are structurally similar if nargin<7 % no derivative mode if nargout>0 % lf={lZ,lp} for inf={'infEP','infLaplace'} lfi = argout(:,:,1); % partition functions of individual likelihoods lf = s(lfi + one_lwn); % weighted sum in the exp domain varargout{1} = lf; if nargout>1 % dlf dlfi = argout(:,:,2); % derivatives of individual terms % Using f*dlf=df, f=exp(lf), f=sum_i wi*fi, df=sum_i wi*dfi we obtain % dlf = sum_i exp(lfi-lf+lwi)*dlfi = sum_i ai*dlfi. a = exp(lfi - lf*ones(1,m) + one_lwn); dlf = sum(a.*dlfi,2); varargout{2} = dlf; if nargout>2 % d2lf d2lfi = argout(:,:,3); % 2nd derivatives of individual terms % Using d2lf=d(df/f)=(d2f*f-df^2)/f^2 <=> d2f=f*(d2lf+dlf^2) and % d2f = sum_i wi*d2fi, we get d2lf = sum_i ai*(d2lfi+dlfi^2)-dlf^2. d2lf = sum(a.*(d2lfi+dlfi.*dlfi),2) - dlf.*dlf; varargout{3} = d2lf; if nargout>3 && strcmp(inf,'infLaplace') % d3lf % Using d3f = sum_i wi*d3fi, d3lf=d((d2f*f-df^2)/f^2), we obtain % d3f = f*(d3lf+dlf*(3*d2lf+dlf^2)) <=> % d3lf = d3f/f - dlf*(3*d2lf+dlf^2). d3lfi = argout(:,:,4); % 3rd derivatives of individual terms d3lf = sum(a.*(d3lfi+dlfi.*(3*d2lfi+dlfi.*dlfi)),2); d3lf = d3lf - dlf.*(3*d2lf+dlf.*dlf); varargout{4} = d3lf; end end end end else % d(lf)/dhyp and d(d2lf)/dhyp if nargout>0 lfi = argout(:,:,1); % partition functions of individual likelihoods lf = s(lfi + one_lwn); % weighted sum in the exp domain a = exp(lfi - lf*ones(1,m) + one_lwn); if i<=m % the first m hyperparameters are mixture weights lf_dhyp = a(:,i) - exp(lwn(i)); varargout{1} = lf_dhyp; if nargout>1 && strcmp(inf,'infLaplace') % d(dlf)/dhyp dlfi = argout(:,:,2); dlwn = zeros(size(lw)); dlwn(i) = 1; dlwn = dlwn - exp(lw(i)-s(lw)); dla = ones(n,1)*dlwn - lf_dhyp*ones(1,m); % d log(v) / dhyp da = a.*dla; dlf_dhyp = sum(da.*dlfi,2); varargout{2} = dlf_dhyp; if nargout>2 % d(d2lf)/dhyp d2lfi = argout(:,:,3); dlf = sum(a.*dlfi,2); d2lf_dhyp = sum(da.*(d2lfi+dlfi.*dlfi),2) - 2*dlf.*dlf_dhyp; varargout{3} = d2lf_dhyp; end end else j = v(i-m); % index of mixture component under consideration id = find(v==j)+m; % hyper parameter indices i = i-min(id)+1; % index of likelihood parameter of mixture j f = lik(j); if iscell(f{:}), f = f{:}; end % expand cell if necessary if nargout==1 lfi_dhyp = feval(f{:}, hyp(id), varargin{1:end-1}, i); elseif nargout==2 [lfi_dhyp,dlfi_dhyp] = feval(f{:}, hyp(id), varargin{1:end-1}, i); else [lfi_dhyp,dlfi_dhyp,d2lfi_dhyp] = ... feval(f{:}, hyp(id), varargin{1:end-1}, i); end lf_dhyp = exp(lwn(j)+lfi(:,j)-lf) .* lfi_dhyp(:); % d(lf)/dhyp varargout{1} = lf_dhyp; if nargout>1 % d(dlf)/dhyp dlfi = argout(:,:,2); d2lfi = argout(:,:,3); dlf = sum(a.*dlfi,2); dla = -lf_dhyp*ones(1,m); dla(:,j) = dla(:,j)+lfi_dhyp(:); da = a.*dla; dlf_dhyp = sum(da.*dlfi,2) + a(:,j).*dlfi_dhyp(:); varargout{2} = dlf_dhyp; if nargout>2 % d(d2lf)/dhyp p = d2lfi+dlfi.*dlfi; dpj = 2*dlfi(:,j).*dlfi_dhyp(:) + d2lfi_dhyp(:); d2lf_dhyp = a(:,j).*dpj + sum(da.*p,2) - 2*dlf.*dlf_dhyp; varargout{3} = d2lf_dhyp; end end end end end case 'infVB' error('infVB not supported') end endgpml-matlab-v3.6-2015-07-07/lik/likWeibull.m000644 000765 000024 00000010704 12546444606 020143 0ustar00carlstaff000000 000000 function [varargout] = likWeibull(link, hyp, y, mu, s2, inf, i) % likWeibull - Weibull likelihood function for strictly positive data y. The % expression for the likelihood is % likWeibull(f) = g1*ka/mu * (g1*y/mu)^(ka-1) * exp(-(g1*y/mu)^ka) with % gj = gamma(1+j/ka), mean=mu and variance=mu^2*(g2/g1^2-1) where mu = g(f) is % the Weibull intensity, f is a Gaussian process, y is the positive data. % Hence, we have llik(f) = log(likWeibull(f)) = % log(g1*ka/mu) + (ka-1)*log(g1*y/mu) - (g1*y/mu)^ka. % % We provide two inverse link functions 'exp' and 'logistic': % g(f) = exp(f) and g(f) = log(1+exp(f))). % The link functions are located at util/glm_invlink_*.m. % % Note that for neither link function the likelihood lik(f) is log concave. % % The hyperparameters are: % % hyp = [ log(ka) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % See also LIKFUNCTIONS.M. % % Copyright (c) by Hannes Nickisch, 2013-10-30. if nargin<4, varargout = {'1'}; return; end % report number of hyperparameters ka = exp(hyp); lg1 = gammaln(1+1/ka); g1 = exp(lg1); dlg1 = -psi(1+1/ka)/ka; if nargin<6 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>4&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability lg = g(mu,link); lp = lg1 + log(ka) + (ka-1)*(lg1+log(y)) - ka*lg - exp(ka*(lg1+log(y)-lg)); else lp = likWeibull(link, hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 % compute y moments by quadrature n = max([length(y),length(mu),length(s2)]); on = ones(n,1); N = 20; [t,w] = gauher(N); oN = ones(1,N); lw = ones(n,1)*log(w'); mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only lg = g(sig*t'+mu*oN,link); ymu = exp(logsumexp2(lg+lw)); % first moment using Gaussian-Hermite quad if nargout>2 elg = exp(lg); g2 = gamma(1+2/ka); yv = elg.^2*(g2/g1^2-1); % second y moment from Weibull distribution ys2 = (yv+(elg-ymu*oN).^2)*w; end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' [lg,dlg,d2lg,d3lg] = g(mu,link); elg = exp(-ka*lg); if nargin<7 % no derivative mode lp = lg1 + log(ka) + (ka-1)*(lg1+log(y)) -ka*lg - exp(ka*(lg1+log(y)-lg)); dlp = {}; d2lp = {}; d3lp = {}; % return arguments if nargout>1 dlp = -ka*dlg + ka*(g1*y).^ka .* elg.*dlg; % dlp, deriv of log lik if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = -ka*d2lg + ka*(g1*y).^ka .* ( -ka*elg.*dlg.^2 + elg.*d2lg ); if nargout>3 % d3lp, 3rd derivative of log likelihood a = ka^2*dlg.^3 -3*ka*dlg.*d2lg + d3lg; d3lp = - ka*d3lg + ka*(g1*y).^ka .* a .*elg; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode v = ka*(lg1+log(y)-lg); ev = exp(v); % derivative of log lik w.r.t. ka w = v+ka*dlg1; dw = -ka*dlg; d2w = -ka*d2lg; lp_dhyp = 1 + w - ev.*w; dlp_dhyp = dw.*(1-ev.*(1+w)); % first derivative d2lp_dhyp = d2w.*(1-ev.*(1+w)) - dw.^2.*(ev.*(2+w)); % and also second varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargin<7 % no derivative mode % Since we are not aware of an analytical expression of the integral, % we use quadrature. varargout = cell(1,nargout); [varargout{:}] = lik_epquad({@likWeibull,link},hyp,y,mu,s2); else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' error('infVB not supported') end end % compute the log intensity using the inverse link function function varargout = g(f,link) varargout = cell(nargout, 1); % allocate the right number of output arguments if strcmp(link,'exp') [varargout{:}] = glm_invlink_exp(f); else [varargout{:}] = glm_invlink_logistic(f); endgpml-matlab-v3.6-2015-07-07/lik/likBeta.m000644 000765 000024 00000011336 12546444606 017415 0ustar00carlstaff000000 000000 function [varargout] = likBeta(link, hyp, y, mu, s2, inf, i) % likBeta - Beta likelihood function for interval data y from [0,1]. % The expression for the likelihood is % likBeta(f) = 1/Z * y^(mu*phi-1) * (1-y)^((1-mu)*phi-1) with % mean=mu and variance=mu*(1-mu)/(1+phi) where mu = g(f) is the Beta intensity, % f is a Gaussian process, y is the interval data and % Z = Gamma(phi)/Gamma(phi*mu)/Gamma(phi*(1-mu)). % Hence, we have % llik(f) = log(likBeta(f)) = -lam*(y-mu)^2/(2*mu^2*y) - log(Zy). % % We provide two inverse link functions 'logit' and 'expexp': % g(f) = 1/(1+exp(-f)) and g(f) = exp(-exp(-f))). % The link functions are located at util/glm_invlink_*.m. % % Note that for neither link function the likelihood lik(f) is log concave. % % The hyperparameters are: % % hyp = [ log(phi) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % See also LIKFUNCTIONS.M. % % Copyright (c) by Hannes Nickisch, 2014-03-04. if nargin<4, varargout = {'1'}; return; end % report number of hyperparameters phi = exp(hyp); if nargin<6 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>4&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability lg = g(mu,link); elg = exp(lg); v = phi*elg; w = phi-v; a0 = gammaln(w)-gammaln(phi); lp = (v-1).*log(y) + (w-1).*log(1-y) - gammaln(v) - a0; else lp = likBeta(link, hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 % compute y moments by quadrature n = max([length(y),length(mu),length(s2)]); on = ones(n,1); N = 20; [t,w] = gauher(N); oN = ones(1,N); lw = ones(n,1)*log(w'); mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only lg = g(sig*t'+mu*oN,link); ymu = exp(logsumexp2(lg+lw)); % first moment using Gaussian-Hermite quad if nargout>2 elg = exp(lg); yv = elg.*(1-elg)/(1+phi); % second y moment from Beta distribution ys2 = (yv+(elg-ymu*oN).^2)*w; end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' [lg,dlg,d2lg,d3lg] = g(mu,link); elg = exp(lg); v = phi*elg; w = phi-v; if nargin<7 % no derivative mode a0 = gammaln(phi-v)-gammaln(phi); lp = (v-1).*log(y) + (w-1).*log(1-y) - gammaln(v) - a0; dlp = {}; d2lp = {}; d3lp = {}; % return arguments if nargout>1 % dlp, derivative of log likelihood a1 = v.*(log(y)-log(1-y) + psi(0,w)-psi(0,v)); dlp = dlg.*a1; if nargout>2 % d2lp, 2nd derivative of log likelihood a2 = v.^2.*(psi(1,w)+psi(1,v)); z = dlg.^2+d2lg; d2lp = z.*a1 - dlg.^2.*a2; if nargout>3 % d3lp, 3rd derivative of log likelihood a3 = v.^3.*(psi(2,w)-psi(2,v)); d3lp = (dlg.*z+2*dlg.*d2lg+d3lg).*a1 - 3*dlg.*z.*a2 + dlg.^3.*a3; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode % deriv. of log lik w.r.t. phi lp_dhyp = v.*log(y)+w.*log(1-y)-v.*psi(0,v)-w.*psi(0,w)+phi*psi(0,phi); a1 = v.*(log(y)-log(1-y) + psi(0,w)-psi(0,v)); da1 = a1 + v.*(w.*psi(1,w)-v.*psi(1,v)); dlp_dhyp = dlg.*da1; % first derivative a2 = v.^2.*(psi(1,w)+psi(1,v)); z = dlg.^2+d2lg; da2 = v.^2.*(w.*psi(2,w)+v.*psi(2,v)) + 2*a2; d2lp_dhyp = z.*da1 - dlg.^2.*da2; % second derivative varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargin<7 % no derivative mode % Since we are not aware of an analytical expression of the integral, % we use quadrature. varargout = cell(1,nargout); [varargout{:}] = lik_epquad({@likBeta,link},hyp,y,mu,s2); else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' error('infVB not supported') end end % compute the log intensity using the inverse link function function varargout = g(f,link) varargout = cell(nargout, 1); % allocate the right number of output arguments if strcmp(link,'expexp') [varargout{:}] = glm_invlink_expexp(f); else [varargout{:}] = glm_invlink_logit(f); end gpml-matlab-v3.6-2015-07-07/lik/likUni.m000644 000765 000024 00000005066 12546444606 017300 0ustar00carlstaff000000 000000 function [varargout] = likUni(hyp, y, mu, s2, inf, i) % likUni - Uniform likelihood function for classification. The expression for % the likelihood is % likUni(t) = 1/2. % % There are no hyperparameters: % % hyp = [ ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Hannes Nickisch, 2013-09-02. % % See also LIKFUNCTIONS.M. if nargin<3, varargout = {'0'}; return; end % report number of hyperparameters if nargin<5 % prediction mode if inf is not present lp = -log(2)*ones(size(mu)); ymu = {}; ys2 = {}; if nargout>1 p = exp(lp); ymu = 2*p-1; % first y moment if nargout>2 ys2 = 4*p.*(1-p); % second y moment end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' if nargin<6 % no derivative mode lp = -log(2)*ones(size(mu)); dlp = {}; d2lp = {}; d3lp = {}; if nargout>1 dlp = zeros(size(mu)); % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = zeros(size(mu)); if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = zeros(size(mu)); end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode varargout = {[],[],[]}; % derivative w.r.t. hypers end case 'infEP' if nargin<6 % no derivative mode lZ = -log(2)*ones(size(mu)); % log part function dlZ = {}; d2lZ = {}; if nargout>1 dlZ = zeros(size(mu)); % 1st derivative w.r.t. mean if nargout>2 d2lZ = zeros(size(mu)); % 2nd derivative w.r.t. mean end end varargout = {lZ,dlZ,d2lZ}; else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' % variational lower site bound % t(s) = 1/2 % the bound has the form: (b+z/ga)*f - f.^2/(2*ga) - h(ga)/2 n = numel(s2); b = zeros(n,1); y = y.*ones(n,1); z = y; varargout = {b,z}; end endgpml-matlab-v3.6-2015-07-07/lik/likPoisson.m000644 000765 000024 00000010122 12546444606 020164 0ustar00carlstaff000000 000000 function [varargout] = likPoisson(link, hyp, y, mu, s2, inf, i) % likPoisson - Poisson likelihood function for count data y. The expression for % the likelihood is % likPoisson(f) = mu^y * exp(-mu) / y! with mean=variance=mu % where mu = g(f) is the Poisson intensity, f is a % Gaussian process, y is the non-negative integer count data and % y! = gamma(y+1) its factorial. Hence, we have -- with Zy = gamma(y+1) = y! -- % llik(f) = log(likPoisson(f)) = log(g(f))*y - g(f) - log(Zy). % The larger the intensity mu, the stronger the likelihood resembles a Gaussian % since skewness = 1/sqrt(mu) and kurtosis = 1/mu. % % We provide two inverse link functions 'exp' and 'logistic': % For g(f) = exp(f), we have lik(f) = exp(f*y-exp(f)) / Zy. % For g(f) = log(1+exp(f))), we have lik(f) = log^y(1+exp(f)))(1+exp(f)) / Zy. % The link functions are located at util/glm_invlink_*.m. % % Note that for both intensities g(f) the likelihood lik(f) is log concave. % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % See also LIKFUNCTIONS.M. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-10-16. if nargin<4, varargout = {'0'}; return; end % report number of hyperparameters if nargin<6 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>4&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability lg = g(mu,link); lp = lg.*y - exp(lg) - gammaln(y+1); else lp = likPoisson(link, hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 % compute y moments by quadrature n = max([length(y),length(mu),length(s2)]); on = ones(n,1); N = 20; [t,w] = gauher(N); oN = ones(1,N); lw = ones(n,1)*log(w'); mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only lg = g(sig*t'+mu*oN,link); ymu = exp(logsumexp2(lg+lw)); % first moment using Gaussian-Hermite quad if nargout>2 elg = exp(lg); yv = elg; % second y moment from Poisson distribution ys2 = (yv+(elg-ymu*oN).^2)*w; end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' if nargin<7 % no derivative mode [lg,dlg,d2lg,d3lg] = g(mu,link); elg = exp(lg); lp = lg.*y - elg - gammaln(y+1); dlp = {}; d2lp = {}; d3lp = {}; % return arguments if nargout>1 dlp = dlg.*(y-elg); % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = d2lg.*(y-elg) - dlg.*dlg.*elg; if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = d3lg.*(y-elg) - dlg.*(dlg.*dlg+3*d2lg).*elg; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode varargout = {[],[],[]}; % derivative w.r.t. hypers end case 'infEP' if nargin<7 % no derivative mode % Since we are not aware of an analytical expression of the integral, % hence we use quadrature. varargout = cell(1,nargout); [varargout{:}] = lik_epquad({@likPoisson,link},hyp,y,mu,s2); else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' error('infVB not supported') end end % compute the log intensity using the inverse link function function varargout = g(f,link) varargout = cell(nargout, 1); % allocate the right number of output arguments if strcmp(link,'exp') [varargout{:}] = glm_invlink_exp(f); else [varargout{:}] = glm_invlink_logistic(f); endgpml-matlab-v3.6-2015-07-07/lik/likLaplace.m000644 000765 000024 00000015412 12546444606 020102 0ustar00carlstaff000000 000000 function [varargout] = likLaplace(hyp, y, mu, s2, inf, i) % likLaplace - Laplacian likelihood function for regression. % The expression for the likelihood is % likLaplace(t) = exp(-|t-y|/b)/(2*b) with b = sn/sqrt(2), % where y is the mean and sn^2 is the variance. % % The hyperparameters are: % % hyp = [ log(sn) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-10-16. % % See also LIKFUNCTIONS.M. if nargin<3, varargout = {'1'}; return; end % report number of hyperparameters sn = exp(hyp); b = sn/sqrt(2); if nargin<5 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>3&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability evaluation lp = -abs(y-mu)./b -log(2*b); s2 = 0; else % prediction lp = likLaplace(hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 ymu = mu; % first y moment if nargout>2 ys2 = s2 + sn.^2; % second y moment end end varargout = {lp,ymu,ys2}; else % inference mode switch inf case 'infLaplace' if nargin<6 % no derivative mode if numel(y)==0, y=0; end ymmu = y-mu; dlp = {}; d2lp = {}; d3lp = {}; lp = -abs(ymmu)/b -log(2*b); if nargout>1 dlp = sign(ymmu)/b; % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = zeros(size(ymmu)); if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = zeros(size(ymmu)); end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode lp_dhyp = abs(y-mu)/b - 1; % derivative of log likelihood w.r.t. hypers dlp_dhyp = sign(mu-y)/b; % first derivative, d2lp_dhyp = zeros(size(mu)); % and also of the second mu derivative varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' n = max([numel(y),numel(mu),numel(s2),numel(sn)]); on = ones(n,1); y = y(:).*on; mu = mu(:).*on; s2 = s2(:).*on; sn = sn(:).*on; % vectors only fac = 1e3; % factor between the widths of the two distributions ... % ... from when one considered a delta peak, we use 3 orders of magnitude idlik = fac*sn1 lqp = -zp.^2/2 - log(2*pi)/2 - logphi(-zp); % log( N(z)/Phi(z) ) lqm = -zm.^2/2 - log(2*pi)/2 - logphi( zm); dap = -exp(lqp-log(s2(id))/2) + sqrt(2)./sn(id); dam = exp(lqm-log(s2(id))/2) - sqrt(2)./sn(id); % ( exp(ap).*dap + exp(am).*dam )./( exp(ap) + exp(am) ) dlZ(id) = expABz_expAx([ap,am],[1;1],[dap,dam],[1;1]); if nargout>2 a = sqrt(8)./sn(id)./sqrt(s2(id)); bp = 2./sn(id).^2 - (a - zp./s2(id)).*exp(lqp); bm = 2./sn(id).^2 - (a + zm./s2(id)).*exp(lqm); % d2lZ(id) = ( exp(ap).*bp + exp(am).*bm )./( exp(ap) + exp(am) )... % - dlZ(id).^2; d2lZ(id) = expABz_expAx([ap,am],[1;1],[bp,bm],[1;1]) - dlZ(id).^2; end end end varargout = {lZ,dlZ,d2lZ}; else % derivative mode dlZhyp = zeros(n,1); if any(idlik) dlZhyp(idlik) = 0; end if any(idgau) dlZhyp(idgau) = ... likLaplace(log(sn(idgau)), mu(idgau), y(idgau), 'infLaplace', 1); end if any(id) % substitution to obtain unit variance, zero mean Laplacian tmu = (mu(id)-y(id))./sn(id); tvar = s2(id)./sn(id).^2; zp = (tvar+tmu/sqrt(2))./sqrt(tvar); vp = tvar+sqrt(2)*tmu; zm = (tvar-tmu/sqrt(2))./sqrt(tvar); vm = tvar-sqrt(2)*tmu; dzp = (-s2(id)./sn(id)+tmu.*sn(id)/sqrt(2)) ./ sqrt(s2(id)); dvp = -2*tvar - sqrt(2)*tmu; dzm = (-s2(id)./sn(id)-tmu.*sn(id)/sqrt(2)) ./ sqrt(s2(id)); dvm = -2*tvar + sqrt(2)*tmu; lezp = logerfc(zp); % ap = exp(vp).*ezp lezm = logerfc(zm); % am = exp(vm).*ezm vmax = max([vp+lezp,vm+lezm],[],2); % subtract max to avoid numerical pb ep = exp(vp+lezp-vmax); em = exp(vm+lezm-vmax); dap = ep.*(dvp - 2/sqrt(pi)*exp(-zp.^2-lezp).*dzp); dam = em.*(dvm - 2/sqrt(pi)*exp(-zm.^2-lezm).*dzm); dlZhyp(id) = (dap+dam)./(ep+em) - 1; end varargout = {dlZhyp}; % deriv. wrt hypers end case 'infVB' % variational lower site bound % t(s) = exp(-sqrt(2)|y-s|/sn) / sqrt(2*sn²) % the bound has the form: (b+z/ga)*f - f.^2/(2*ga) - h(ga)/2 n = numel(s2); b = zeros(n,1); y = y.*ones(n,1); z = y; varargout = {b,z}; end end % logerfc(z) = log(1-erf(z)) function lc = logerfc(z) lc = logphi(-z*sqrt(2)) + log(2); function y = expABz_expAx(A,x,B,z) N = size(A,2); maxA = max(A,[],2); % number of columns, max over columns A = A-maxA*ones(1,N); % subtract maximum value y = ( (exp(A).*B)*z ) ./ ( exp(A)*x );gpml-matlab-v3.6-2015-07-07/lik/likGaussWarp.m000644 000765 000024 00000021625 12550665721 020456 0ustar00carlstaff000000 000000 function [varargout] = likGaussWarp(warp, hyp, y, mu, varargin) % likGaussWarp - Warped Gaussian likelihood for regression. % The expression for the likelihood is % likGaussWarp( y | t ) = likGauss( g(y) | t ) * g'(y), % where likGauss is the Gaussian likelihood and g is the warping function. % % The hyperparameters are: % % hyp = [ theta_1 % theta_2 % .. % theta_ng % log(sn) ] % % Here, sn is the standard deviation of the underlying Gaussian and theta_i for % i=1..ng are the ng hyperparameters of the warping function g. % % At the moment, likGaussWarp offers 3 different warping functions: % id yields g(y) = y => likGaussWarp = likGauss % poly e.g. 'poly1' yields g(y) = y => likGaussWarp = likGauss % 'poly3' yields g(y) = y + c1*sy*ay^2 + c2*sy*ay^3 % where sy = sign(y), ay = abs(y), cj = exp(theta_j) % tanh e.g. 'tanh0' yields g(y) = y => likGaussWarp = likGauss % 'tanh2' yields g(y) = y + a1*tanh(b1*(y+c1)) + a2*tanh(b2*(y+c2)) % where aj = exp(theta_j), bj = exp(theta_j+m), bj = theta_j+2*m % % The code is based on the exposition in the paper Warped Gaussian Processes, % NIPS, 2003 by Edward Snelson, Carl Rasmussen and Zoubin Ghahramani. % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Hannes Nickisch, 2013-10-24. % % See also LIKFUNCTIONS.M. lik = {@likGauss}; % in principle any likelihood function can be warped but only % for homoscedastic likelihoods, in particular Gaussian has feasible integrals if numel(warp)==0, warp = 'id'; end % set default warping function ng = g(warp); % number of hyperparameters for the warping function nhyp = ['(',num2str(ng),'+',feval(lik{:}),')']; % number of hyperparameters if nargin<4, varargout = {nhyp}; return, end % report number of parameters nhyp = eval(nhyp); if nhyp>length(hyp), error('not enough hyperparameters'), end [gy,lgpy] = g(warp,y,hyp(1:ng)); % evaluate warping function i = 0; if nargin>6, i = varargin{3}; varargin{3} = varargin{3}-ng; end varargout = cell(nargout,1); % allocate memory for output arguments if i==0 || ng4, s2 = varargin{1}; end % s2==0 ? if nargin>4&&numel(s2)>0&&norm(s2)>eps>0, s2zero = 0; end if s2zero % log probability lp = likGaussWarp(warp, hyp, y, mu, [], 'infLaplace'); s2 = 0*mu; else lp = likGaussWarp(warp, hyp, y, mu, s2, 'infEP'); % prediction end if nargout>0, varargout{1} = lp; end % works for any lik % the predictive moments are very hard to compute for lik not being likGauss if nargout>1 ymu = mu; % first g(y) moment sn2 = exp(2*hyp(ng+1)); % Gaussian noise variance ys2 = s2 + sn2; % second g(y) moment % ymuM = ig(warp,ymu,hyp(1:ng)); % median % yupp = ig(warp,ymu+2*sqrt(ys2),hyp(1:ng)); % 95% confidence interval % ylow = ig(warp,ymu-2*sqrt(ys2),hyp(1:ng)); % ys2C = (yupp-ylow).^2/16; N = 20; [t,w] = gauher(N); oN = ones(1,N); % Gaussian-Hermite quadrature Z = sqrt(ys2(:))*t'+ymu(:)*oN; Y = ig(warp,Z,hyp(1:ng)); ymu = Y*w; ys2 = (Y-ymu*oN).^2*w; % first and second y moment varargout{2} = reshape(ymu,size(mu)); if nargout>2 varargout{3} = ys2; end end else inf = varargin{2}; % obtain input variables switch inf case {'infLaplace','infEP'} % they have the same structure if nargin<7 % no derivative mode if nargout>0, varargout{1} = varargout{1} + lgpy; end else % derivative mode if i<=ng % derivatives w.r.t. warping function parameters n = max([numel(y),numel(mu)]); for j=2:nargout, varargout{j} = zeros(n,1); end [dgy,dlgpy] = g(warp,y,hyp(1:ng),i); % warping function derivative out = cell(nargout+1,1); % allocate memory [out{:}] = likGaussWarp(warp, hyp, y, mu, varargin{1:2}); % query lik % works only for homoscedastic likelihoods where y and mu can be swapped if nargout>0, varargout{1} = dlgpy - out{2}.*dgy; end % apply chain rule if nargout>1, varargout{2} = - out{3}.*dgy; end if nargout>2, varargout{3} = - out{4}.*dgy; end end end case 'infVB' % output does not depend on mu and following parameters end end % marshalling of parameters and available warping functions function varargout = g(warp,varargin) varargout = cell(nargout, 1); % allocate the right number of output arguments if strcmp(warp,'id') % indentity warping likGaussWarp = likGauss if nargin<2 if nargout>0, varargout{1} = 0; end elseif nargin<4 if nargout>0, varargout{1} = varargin{1}; end if nargout>1, varargout{2} = 0*varargin{1}; end end elseif numel(strfind(warp,'poly'))>0 m = str2double(warp(5:end)); if nargin<2 && nargout>0, varargout{1} = m-1; return, end [varargout{:}] = g_poly(varargin{:}); elseif numel(strfind(warp,'tanh'))>0 m = str2double(warp(5:end)); if nargin<2 && nargout>0, varargout{1} = 3*m; return, end [varargout{:}] = g_tanh(varargin{:}); end % invert g(y) = z <=> ig(z) = y via bisection search + Newton iterations function [y,n,d] = ig(warp,z,hyp) y = z; gy = g(warp,z,hyp)-z; dz = max(abs(z(:))); % lower bound search ylow while any(0gy(:)), y(0>gy) = y(0>gy)+dz; gy = g(warp,y,hyp)-z; end, yupp = y; for n=1:12 % bisection search ylow<=y<=yupp d = max(abs(gy(:))); if d0) = y(gy>0); end for n=1:12 % Newton iterations [gy,lgpy] = g(warp,y,hyp); gpy = exp(lgpy); y = y - (gy-z)./gpy; y(yyupp) = yupp(y>yupp); % keep brackets d = max( abs(gy(:)-z(:)) ); if dsqrt(eps), fprintf('Not converged: res=%1.4e\n',d), end % poly warping function g(y) and log of the derivative log(g'(y))>0 % or derivatives of the latter w.r.t. ith hyperparameter function [gy,lgpy] = g_poly(y,hyp,i) m = numel(hyp)+1; c = exp(hyp); if nargin==2 % function values gy = y; gpy = 1; ay = abs(y); for j=2:m gy = gy + c(j-1)*ay.^j; gpy = gpy + c(j-1)*j*ay.^(j-1); end gy = sign(y).*gy; lgpy = log(gpy); else % derivatives gpy = 1; ay = abs(y); for j=2:m gpy = gpy + c(j-1)*j*ay.^(j-1); end gy = c(i)*ay.^j; lgpy = c(i)*j*ay.^(j-1)./gpy; end % tanh warping function g(y) and log of the derivative log(g'(y))>0 % or derivatives of the latter w.r.t. ith hyperparameter function [gy,lgpy] = g_tanh(y,hyp,i) m = numel(hyp)/3; a = exp(hyp(1:m)); b = exp(hyp(m+(1:m))); c = hyp(2*m+(1:m)); if nargin==2 % function values gy = y; gpy = 1; for j=1:m ai = a(j); bi = b(j); ci = c(j); ti = tanh(bi*(y+ci)); dti = 1-ti.^2; gy = gy + ai *ti; gpy = gpy + ai*bi*dti; end lgpy = log(gpy); else % derivatives gpy = 1; for j=1:m ai = a(j); bi = b(j); ci = c(j); ti = tanh(bi*(y+ci)); dti = 1-ti.^2; gpy = gpy + ai*bi*dti; end if i<=m j = i; ai = a(j); bi = b(j); ci = c(j); ti = tanh(bi*(y+ci)); dti = 1-ti.^2; gy = ai*ti; lgpy = ai*bi*dti./gpy; elseif i<=2*m j = i-m; ai = a(j); bi = b(j); ci = c(j); ti = tanh(bi*(y+ci)); dti = 1-ti.^2; gy = ai*bi*dti.*(y+ci); d2ti = -2*ti.*dti; lgpy = ai*bi*(dti+bi*d2ti.*(y+ci))./gpy; else j = i-2*m; ai = a(j); bi = b(j); ci = c(j); ti = tanh(bi*(y+ci)); dti = 1-ti.^2; gy = ai*bi*dti; d2ti = -2*ti.*dti; lgpy = ai*bi^2*d2ti./gpy; end endgpml-matlab-v3.6-2015-07-07/lik/likGumbel.m000644 000765 000024 00000007610 12546444606 017755 0ustar00carlstaff000000 000000 function [varargout] = likGumbel(sign, hyp, y, mu, s2, inf, i) % likGumbel - Gumbel likelihood function for extremal value regression. % The expression for the likelihood is % likGumbel(t) = exp(-z-exp(-z))/be, z = ga+s*(y-t)/be, be = sn*sqrt(6)/pi % where s={+1,-1} is a sign switching between left and right skewed, ga is the % Euler-Mascheroni constant, y is the mean, sn^2 is the variance. % The skewness and kurtosis of likGumbel are 1.14*s and 2.4, respectively. % % The hyperparameters are: % % hyp = [ log(sn) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Hannes Nickisch, 2013-11-01. % % See also LIKFUNCTIONS.M. if nargin<4, varargout = {'1'}; return; end % report number of hyperparameters if sign=='-', s = -1; else s = 1; end % extract sign of skewness sn2 = exp(2*hyp); % extract hyperparameters ga = 0.5772156649; % Euler-Mascheroni constant be = sqrt(6*sn2)/pi; lZ = -log(be); if nargin<6 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>4&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability evaluation lp = likGumbel(sign, hyp, y, mu, [], 'infLaplace'); s2 = 0; else % prediction lp = likGumbel(sign, hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 ymu = mu; % first y moment if nargout>2 ys2 = s2 + sn2; % second y moment end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' z = ga+s*(y-mu)/be; emz = exp(-z); if nargin<7 % no derivative mode dlp = {}; d2lp = {}; d3lp = {}; lp = lZ -z -emz; if nargout>1 dz = -s/be; % dz/dmu dlp = dz*(emz-1); % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = -dz^2*emz; if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = dz^3*emz; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative w.r.t. log(sn) dz = -s/be; % dz/dmu dzs = -s*(y-mu)/be; % dz/dlog(sn) lp_dhyp = dzs.*(emz-1) -1; dlp_dhyp = dz*(1-emz.*(1+dzs)); d2lp_dhyp = dz^2*emz.*(2+dzs); varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargout>1 error('infEP not supported since likT is not log-concave') end n = max([length(y),length(mu),length(s2)]); on = ones(n,1); y = y(:).*on; mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only % since we are not aware of an analytical expression of the integral, % we use Gaussian-Hermite quadrature N = 20; [t,w] = gauher(N); oN = ones(1,N); lZ = likGumbel(sign, hyp, y*oN, sig*t'+mu*oN, []); lZ = log_expA_x(lZ,w); % log( exp(lZ)*w ) varargout = {lZ}; case 'infVB' error('infVB not supported') end end % computes y = log( exp(A)*x ) in a numerically safe way by subtracting the % maximal value in each row to avoid cancelation after taking the exp function y = log_expA_x(A,x) N = size(A,2); maxA = max(A,[],2); % number of columns, max over columns y = log(exp(A-maxA*ones(1,N))*x) + maxA; % exp(A) = exp(A-max(A))*exp(max(A))gpml-matlab-v3.6-2015-07-07/lik/likNegBinom.m000644 000765 000024 00000011145 12546444606 020236 0ustar00carlstaff000000 000000 function [varargout] = likNegBinom(link, hyp, y, mu, s2, inf, i) % likNegBinom - Negative binomial likelihood function for count data y. % The expression for the likelihood is % likNegBinom(f) = 1/Z * mu^y / (r+mu)^(r+y), Z = r^r*G(y+r)/(G(y+1)*G(r)) % with G(t)=gamma(t)=(t-1)!, mean=mu and variance=mu*(mu+r)/r, where r is the % number of failures parameters, mu = g(f) is the negative binomial intensity, % f is a Gaussian process and y is the non-negative integer count data. % Hence, we have -- with log(Z)=r*log(r)+L(y+r)-L(y+1)-L(r), L(t)=gammaln(t) -- % llik(f) = log(likNegBinom(f)) = y*log(mu)-(r+y)*log(r+mu)-log(Z). % % We provide two inverse link functions 'exp' and 'logistic': % g(f) = exp(f) and g(f) = log(1+exp(f))). % The link functions are located at util/glm_invlink_*.m. % % Note that for neither link function the likelihood lik(f) is log concave. % % The hyperparameters are: % % hyp = [ log(r) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % See also LIKFUNCTIONS.M. % % Copyright (c) by Hannes Nickisch, 2015-01-21. if nargin<4, varargout = {'1'}; return; end % report number of hyperparameters lr = hyp; r = exp(lr); if nargin<6 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>4&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability lZ = gammaln(y+1) + gammaln(r) - gammaln(y+r) - r*lr; lg = g(mu,link); mx = max(lg,lr); lgr = log(exp(lg-mx)+exp(lr-mx))+mx; % log(exp(lg)+r) lp = y.*lg - (y+r).*lgr - lZ; else lp = likNegBinom(link, hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 % compute y moments by quadrature n = max([length(y),length(mu),length(s2)]); on = ones(n,1); N = 20; [t,w] = gauher(N); oN = ones(1,N); lw = ones(n,1)*log(w'); mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only lg = g(sig*t'+mu*oN,link); ymu = exp(logsumexp2(lg+lw)); % first moment using Gaussian-Hermite quad if nargout>2 elg = exp(lg); yv = elg.*(elg/r+1); % second y moment from negative binomial distribution ys2 = (yv+(elg-ymu*oN).^2)*w; end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' [lg,dlg,d2lg,d3lg] = g(mu,link); elg = exp(lg); lZ = gammaln(y+1) + gammaln(r) - gammaln(y+r) - r*lr; mx = max(lg,lr); lgr = log(exp(lg-mx)+exp(lr-mx))+mx; % log(exp(lg)+r) a = 1./(1+r./elg); da = a.*(1-a).*dlg; % auxiliary quantities if nargin<7 % no derivative mode lp = y.*lg - (y+r).*lgr - lZ; dlp = {}; d2lp = {}; d3lp = {}; % return arguments if nargout>1 dlp = y.*dlg - (y+r).*a.*dlg; % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = y.*d2lg - (y+r).*(a.*d2lg + da.*dlg); if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = y.*d3lg - (y+r).*(a.*d3lg + da.*(3*d2lg +(1-2*a).*dlg.*dlg)); end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode b = (y+r)./(elg+r); lp_dhyp = r*(1+log(r)-lgr-b-psi(r)+psi(y+r)); dlp_dhyp = r*dlg.*a.*(b-1); % first derivative d2lp_dhyp = r*((d2lg.*a+dlg.*da).*(b-1)-(dlg.*a).^2.*b); % and also second varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargin<7 % no derivative mode % Since we are not aware of an analytical expression of the integral, % hence we use quadrature. varargout = cell(1,nargout); [varargout{:}] = lik_epquad({@likNegBinom,link},hyp,y,mu,s2); else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' error('infVB not supported') end end % compute the log intensity using the inverse link function function varargout = g(f,link) varargout = cell(nargout, 1); % allocate the right number of output arguments if strcmp(link,'exp') [varargout{:}] = glm_invlink_exp(f); else [varargout{:}] = glm_invlink_logistic(f); endgpml-matlab-v3.6-2015-07-07/lik/likGauss.m000644 000765 000024 00000006631 12550665732 017626 0ustar00carlstaff000000 000000 function [varargout] = likGauss(hyp, y, mu, s2, inf, i) % likGauss - Gaussian likelihood function for regression. The expression for the % likelihood is % likGauss(t) = exp(-(t-y)^2/2*sn^2) / sqrt(2*pi*sn^2), % where y is the mean and sn is the standard deviation. % % The hyperparameters are: % % hyp = [ log(sn) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-07-13. % File automatically generated using noweb. % % See also LIKFUNCTIONS.M. if nargin<3, varargout = {'1'}; return; end % report number of hyperparameters sn2 = exp(2*hyp); if nargin<5 % prediction mode if inf is not present if isempty(y), y = zeros(size(mu)); end s2zero = 1; if nargin>3&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability lp = -(y-mu).^2./sn2/2-log(2*pi*sn2)/2; s2 = 0; else lp = likGauss(hyp, y, mu, s2, 'infEP'); % prediction end ymu = {}; ys2 = {}; if nargout>1 ymu = mu; % first y moment if nargout>2 ys2 = s2 + sn2; % second y moment end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' if nargin<6 % no derivative mode if isempty(y), y=0; end ymmu = y-mu; dlp = {}; d2lp = {}; d3lp = {}; lp = -ymmu.^2/(2*sn2) - log(2*pi*sn2)/2; if nargout>1 dlp = ymmu/sn2; % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = -ones(size(ymmu))/sn2; if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = zeros(size(ymmu)); end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode lp_dhyp = (y-mu).^2/sn2 - 1; % derivative of log likelihood w.r.t. hypers dlp_dhyp = 2*(mu-y)/sn2; % first derivative, d2lp_dhyp = 2*ones(size(mu))/sn2; % and also of the second mu derivative varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargin<6 % no derivative mode lZ = -(y-mu).^2./(sn2+s2)/2 - log(2*pi*(sn2+s2))/2; % log part function dlZ = {}; d2lZ = {}; if nargout>1 dlZ = (y-mu)./(sn2+s2); % 1st derivative w.r.t. mean if nargout>2 d2lZ = -1./(sn2+s2); % 2nd derivative w.r.t. mean end end varargout = {lZ,dlZ,d2lZ}; else % derivative mode dlZhyp = ((y-mu).^2./(sn2+s2)-1) ./ (1+s2./sn2); % deriv. w.r.t. hyp.lik varargout = {dlZhyp}; end case 'infVB' % variational lower site bound % t(s) = exp(-(y-s)^2/2sn2)/sqrt(2*pi*sn2) % the bound has the form: (b+z/ga)*f - f.^2/(2*ga) - h(ga)/2 n = numel(s2); b = zeros(n,1); y = y.*ones(n,1); z = y; varargout = {b,z}; end end gpml-matlab-v3.6-2015-07-07/lik/likGamma.m000644 000765 000024 00000010735 12546444607 017567 0ustar00carlstaff000000 000000 function [varargout] = likGamma(link, hyp, y, mu, s2, inf, i) % likGamma - Gamma likelihood function for strictly positive data y. The % expression for the likelihood is % likGamma(f) = al^al*y^(al-1)/gamma(al) * exp(-y*al/mu) / mu^al with % mean=mu and variance=mu^2/al where mu = g(f) is the Gamma intensity, f is a % Gaussian process, y is the strictly positive data. Hence, we have -- with % log(Zy) = log(gamma(al)) - al*log(al) + (1-al)*log(y) % llik(f) = log(likGamma(f)) = -al*( log(g(f)) + y/g(f) ) - log(Zy). % The larger one chooses al, the stronger the likelihood resembles a Gaussian % since skewness = 2/sqrt(al) and kurtosis = 6/al. % % We provide two inverse link functions 'exp' and 'logistic': % g(f) = exp(f) and g(f) = log(1+exp(f))). % The link functions are located at util/glm_invlink_*.m. % % Note that for neither link function the likelihood lik(f) is log concave. % % The hyperparameters are: % % hyp = [ log(al) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % See also LIKFUNCTIONS.M. % % Copyright (c) by Hannes Nickisch, 2013-10-16. if nargin<4, varargout = {'1'}; return; end % report number of hyperparameters al = exp(hyp); if nargin<6 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>4&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability lg = g(mu,link); lZy = gammaln(al) - al*log(al) + (1-al)*log(y); % normalisation constant lp = -al*(lg+y./exp(lg)) - lZy; else lp = likGamma(link, hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 % compute y moments by quadrature n = max([length(y),length(mu),length(s2)]); on = ones(n,1); N = 20; [t,w] = gauher(N); oN = ones(1,N); lw = ones(n,1)*log(w'); mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only lg = g(sig*t'+mu*oN,link); ymu = exp(logsumexp2(lg+lw)); % first moment using Gaussian-Hermite quad if nargout>2 elg = exp(lg); yv = elg.^2/al; % second y moment from Gamma distribution ys2 = (yv+(elg-ymu*oN).^2)*w; end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' [lg,dlg,d2lg,d3lg] = g(mu,link); elg = exp(lg); if nargin<7 % no derivative mode lZy = gammaln(al) - al*log(al) + (1-al)*log(y); % normalisation constant lp = -al*(lg+y./elg) - lZy; dlp = {}; d2lp = {}; d3lp = {}; % return arguments if nargout>1 dlp = -al*dlg.*(1-y./elg); % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = -al*d2lg.*(1-y./elg) - al*dlg.*dlg.*y./elg; if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = -al*d3lg.*(1-y./elg) + al*dlg.*(dlg.*dlg-3*d2lg).*y./elg; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode dlZy = al*psi(0,al) - al*(log(al) + 1 + log(y)); lp_dhyp = -al*(lg+y./elg) - dlZy; % derivative of log likelihood w.r.t. al dlp_dhyp = -al*dlg.*(1-y./elg); % first derivative d2lp_dhyp = -al*d2lg.*(1-y./elg) - al*dlg.*dlg.*y./elg; % and also second varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargin<7 % no derivative mode % Since we are not aware of an analytical expression of the integral, % we use quadrature. varargout = cell(1,nargout); [varargout{:}] = lik_epquad({@likGamma,link},hyp,y,mu,s2); else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' error('infVB not supported') end end % compute the log intensity using the inverse link function function varargout = g(f,link) varargout = cell(nargout, 1); % allocate the right number of output arguments if strcmp(link,'exp') [varargout{:}] = glm_invlink_exp(f); else [varargout{:}] = glm_invlink_logistic(f); endgpml-matlab-v3.6-2015-07-07/lik/likErf.m000644 000765 000024 00000005637 12546444607 017266 0ustar00carlstaff000000 000000 function [varargout] = likErf(hyp, y, mu, s2, inf, i) % likErf - Error function or cumulative Gaussian likelihood function for binary % classification or probit regression. The expression for the likelihood is % likErf(t) = (1+erf(t/sqrt(2)))/2 = normcdf(t). % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2014-03-19. % % See also LIKFUNCTIONS.M. if nargin<3, varargout = {'0'}; return; end % report number of hyperparameters if nargin>1, y = sign(y); y(y==0) = 1; else y = 1; end % allow only +/- 1 values if numel(y)==0, y = 1; end if nargin<5 % prediction mode if inf is not present y = y.*ones(size(mu)); % make y a vector s2zero = 1; if nargin>3&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability evaluation lp = logphi(y.*mu); else % prediction lp = likErf(hyp, y, mu, s2, 'infEP'); end p = exp(lp); ymu = {}; ys2 = {}; if nargout>1 ymu = 2*p-1; % first y moment if nargout>2 ys2 = 4*p.*(1-p); % second y moment end end varargout = {lp,ymu,ys2}; else % inference mode switch inf case 'infLaplace' if nargin<6 % no derivative mode f = mu; yf = y.*f; % product latents and labels varargout = cell(nargout,1); [varargout{:}] = logphi(yf); % query logphi if nargout>1 varargout{2} = y.*varargout{2}; if nargout>3, varargout{4} = y.*varargout{4}; end end else % derivative mode varargout = {[],[],[]}; % derivative w.r.t. hypers end case 'infEP' if nargin<6 % no derivative mode z = mu./sqrt(1+s2); dlZ = {}; d2lZ = {}; if numel(y)>0, z = z.*y; end if nargout<=1, lZ = logphi(z); % log part function else [lZ,n_p] = logphi(z); end if nargout>1 if numel(y)==0, y=1; end dlZ = y.*n_p./sqrt(1+s2); % 1st derivative wrt mean if nargout>2, d2lZ = -n_p.*(z+n_p)./(1+s2); end % 2nd derivative end varargout = {lZ,dlZ,d2lZ}; else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' error('infVB not supported') end endgpml-matlab-v3.6-2015-07-07/lik/likLogistic.m000644 000765 000024 00000013771 12546444607 020325 0ustar00carlstaff000000 000000 function [varargout] = likLogistic(hyp, y, mu, s2, inf, i) % likLogistic - logistic function for binary classification or logit regression. % The expression for the likelihood is % likLogistic(t) = 1./(1+exp(-t)). % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. The moments % \int f^k likLogistic(y,f) N(f|mu,var) df are calculated via a cumulative % Gaussian scale mixture approximation. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-09-02. % % See also LIKFUNCTIONS.M. if nargin<3, varargout = {'0'}; return; end % report number of hyperparameters if nargin>1, y = sign(y); y(y==0) = 1; else y = 1; end % allow only +/- 1 values if numel(y)==0, y = 1; end if nargin<5 % prediction mode if inf is not present y = y.*ones(size(mu)); % make y a vector s2zero = 1; if nargin>3&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability evaluation yf = y.*mu; % product latents and labels lp = yf; ok = -351 p = exp(lp); ymu = 2*p-1; % first y moment if nargout>2 ys2 = 4*p.*(1-p); % second y moment end end varargout = {lp,ymu,ys2}; else % inference mode switch inf case 'infLaplace' if nargin<6 % no derivative mode f = mu; yf = y.*f; s = -yf; % product latents and labels dlp = {}; d2lp = {}; d3lp = {}; % return arguments ps = max(0,s); lp = -(ps+log(exp(-ps)+exp(s-ps))); % lp = -(log(1+exp(s))) if nargout>1 % first derivatives s = min(0,f); p = exp(s)./(exp(s)+exp(s-f)); % p = 1./(1+exp(-f)) dlp = (y+1)/2-p; % derivative of log likelihood if nargout>2 % 2nd derivative of log likelihood d2lp = -exp(2*s-f)./(exp(s)+exp(s-f)).^2; if nargout>3 % 3rd derivative of log likelihood d3lp = 2*d2lp.*(0.5-p); end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode varargout = {[],[],[]}; % derivative w.r.t. hypers end case 'infEP' if nargin<6 % no derivative mode y = y.*ones(size(mu)); % make y a vector % likLogistic(t) \approx 1/2 + \sum_{i=1}^5 (c_i/2) erf(lam_i/sqrt(2)t) lam = sqrt(2)*[0.44 0.41 0.40 0.39 0.36]; % approx coeffs lam_i and c_i c = [1.146480988574439e+02; -1.508871030070582e+03; 2.676085036831241e+03; -1.356294962039222e+03; 7.543285642111850e+01 ]; [lZc,dlZc,d2lZc] = likErf([], y*ones(1,5), mu*lam, s2*(lam.^2), inf); lZ = log_expA_x(lZc,c); % A=lZc, B=dlZc, d=c.*lam', lZ=log(exp(A)*c) dlZ = expABz_expAx(lZc, c, dlZc, c.*lam'); % ((exp(A).*B)*d)./(exp(A)*c) % d2lZ = ((exp(A).*Z)*e)./(exp(A)*c) - dlZ.^2 where e = c.*(lam.^2)' d2lZ = expABz_expAx(lZc, c, dlZc.^2+d2lZc, c.*(lam.^2)') - dlZ.^2; % The scale mixture approximation does not capture the correct asymptotic % behavior; we have linear decay instead of quadratic decay as suggested % by the scale mixture approximation. By observing that for large values % of -f*y ln(p(y|f)) for likLogistic is linear in f with slope y, we are % able to analytically integrate the tail region. val = abs(mu)-196/200*s2-4; % empirically determined bound at val==0 lam = 1./(1+exp(-10*val)); % interpolation weights lZtail = min(s2/2-abs(mu),-0.1); % apply the same to p(y|f) = 1 - p(-y|f) dlZtail = -sign(mu); d2lZtail = zeros(size(mu)); id = y.*mu>0; lZtail(id) = log(1-exp(lZtail(id))); % label and mean agree dlZtail(id) = 0; lZ = (1-lam).* lZ + lam.* lZtail; % interpolate between scale .. dlZ = (1-lam).* dlZ + lam.* dlZtail; % .. mixture and .. d2lZ = (1-lam).*d2lZ + lam.*d2lZtail; % .. tail approximation varargout = {lZ,dlZ,d2lZ}; else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' % variational lower site bound % using -log(1+exp(-s)) = s/2 -log( 2*cosh(s/2) ); % the bound has the form: (b+z/ga)*f - f.^2/(2*ga) - h(ga)/2 n = numel(s2); b = (y/2).*ones(n,1); z = zeros(size(b)); varargout = {b,z}; end end % computes y = log( exp(A)*x ) in a numerically safe way by subtracting the % maximal value in each row to avoid cancelation after taking the exp function y = log_expA_x(A,x) N = size(A,2); maxA = max(A,[],2); % number of columns, max over columns y = log(exp(A-maxA*ones(1,N))*x) + maxA; % exp(A) = exp(A-max(A))*exp(max(A)) % computes y = ( (exp(A).*B)*z ) ./ ( exp(A)*x ) in a numerically safe way % The function is not general in the sense that it yields correct values for % all types of inputs. We assume that the values are close together. function y = expABz_expAx(A,x,B,z) N = size(A,2); maxA = max(A,[],2); % number of columns, max over columns A = A-maxA*ones(1,N); % subtract maximum value y = ( (exp(A).*B)*z ) ./ ( exp(A)*x ); gpml-matlab-v3.6-2015-07-07/lik/likExp.m000644 000765 000024 00000002600 12240130514 017245 0ustar00carlstaff000000 000000 function [varargout] = likExp(link, hyp, varargin) % likExp - Exponential likelihood function for strictly positive data y. The % expression for the likelihood is % likExp(f) = exp(-y/mu) / mu with % mean=mu and variance=mu^2 where mu = g(f) is the intensity, f is a % Gaussian process, y is the strictly positive data. Hence, we have % llik(f) = log(likExp(f)) = -y/g(f) - log(g(f)). % % Internally, the function is in fact a wrapper around likGamma with hyper % parameter al=1. % % We provide two inverse link functions 'exp' and 'logistic': % g(f) = exp(f) and g(f) = log(1+exp(f))). % The link functions are located at util/glm_invlink_*.m. % Note that for the 'exp' intensity the likelihood lik(f) is log concave. % % The hyperparameters are: % % hyp = [ ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % See also LIKFUNCTIONS.M, LIKGAMMA.M. % % Copyright (c) by Hannes Nickisch, 2013-10-29. if nargin<4, varargout = {'0'}; return; end % report number of hyperparameters varargout = cell(nargout, 1); % allocate the right number of output arguments for j=1:nargout, varargout{j} = []; end % derivative output if nargin<=6, [varargout{:}] = likGamma(link,0,varargin{:}); end % log(al) = 0gpml-matlab-v3.6-2015-07-07/prior/priorLogNormal.m000644 000765 000024 00000001666 12441263151 021355 0ustar00carlstaff000000 000000 function [lp,dlp]=priorLogNormal(mu,s2,x) % Univariate Log-normal hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = exp(-(log(x)-mu)^2/(2*s2)) / (sqrt(2*pi*s2)*x) % % where mu(1x1) is the log scale parameter, s2(1x1) is the shape parameter % and x(1xN) contains query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-08. % % See also PRIORDISTRIBUTIONS.M. if nargin<2, error('mu and s2 parameters need to be provided'), end if ~(isscalar(mu)&&isscalar(s2)) error('mu and s2 parameters need to be scalars'), end if nargin<3, lp = exp(sqrt(s2)*randn+mu); return, end % return a sample lx = log(x); lp = -(lx-mu).^2/(2*s2) - log(2*pi*s2)/2 - lx; dlp = -(lx-mu+s2)./(x*s2); lp(x<0) = -inf; dlp(x<0) = 0;gpml-matlab-v3.6-2015-07-07/prior/priorSmoothBox1.m000644 000765 000024 00000003353 12546444647 021502 0ustar00carlstaff000000 000000 function [lp,dlp] = priorSmoothBox1(a,b,eta,x) % Univariate smoothed box prior distribution with linear decay in the log domain % and infinite support over the whole real axis. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = 1/w*sigmoid(eta*(x-a))*(1-sigmoid(eta*(x-b))), % where w = abs(b-a) and sigmoid(z) = 1/(1+exp(-z)) % % a(1x1) is the lower bound parameter, b(1x1) is the upper bound parameter, % eta(1x1)>0 is the slope parameter and x(1xN) contains query hyperparameters % for prior evaluation. Larger values of eta make the distribution more % box-like. % % The distribution p(x) has mean and variance given by: % mu = (a+b)/2 and s2 = w^2/(1-exp(-2*g))*(1+(pi/g)^2)/12, g = eta*w/2. % % /------------\ % / \ % -------- | | --------> x % a b % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Jose Vallet and Hannes Nickisch, 2015-03-27. % % See also PRIORDISTRIBUTIONS.M. if nargin<3, error('a, b and eta parameters need to be provided'), end if b<=a, error('b must be greater than a.'), end if ~(isscalar(a)&&isscalar(b)&&isscalar(eta)) error('a, b and eta parameters need to be scalar values') end if nargin<4 % inverse sampling u = exp((b-a)*eta*rand()); lp = log((u-1)/(exp(-eta*a)-u*exp(-eta*b)))/eta; return end [lpa,dlpa] = logr(eta*(x-a)); [lpb,dlpb] = logr(-eta*(x-b)); lp = lpa + lpb - log(b-a) + log(1-exp((a-b)*eta)); dlp = eta*(dlpa - dlpb); % r(z) = 1/(1+exp(-z)), log(r(z)) = -log(1+exp(-z)) function [lr,dlr] = logr(z) lr = z; ok = -351 % both mu/s2 scalar => inflate D = size(x,1); mu = mu*ones(D,1); s = s*ones(D,1); lds = D*lds; end if ~(ndims(x)==2 && size(x,1)==D), error('x needs to be (Dxn)'), end oN = ones(1,size(x,2)); if size(s,2)==D xm = x-mu*oN; xs = s\xm; else xm = x-mu*oN; xs = xm./(s*oN); end dlp = -sqrt(2)*sign(xs); lp = -sqrt(2)*sum(abs(xs),1) - D*log(2)/2 - lds; if size(s,2)==D, dlp = s'\dlp; else dlp = dlp./(s*oN); endgpml-matlab-v3.6-2015-07-07/prior/priorLaplace.m000644 000765 000024 00000001476 12441263151 021023 0ustar00carlstaff000000 000000 function [lp,dlp] = priorLaplace(mu,s2,x) % Univariate Laplacian hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = exp(-abs(x-mu)/b)/(2*b), where b = sqrt(s2/2), % % mu(1x1) is the mean parameter, s2(1x1) is the variance parameter and % x(1xN) contains query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-09. % % See also PRIORDISTRIBUTIONS.M. if nargin<3 % return a sample u = rand-1/2; lp = mu - sqrt(s2/2)*sign(u)*log(1-2*abs(u)); return end b = sqrt(s2/2); lp = -abs(x-mu)/b - log(2*b); dlp = -sign(x-mu)/b;gpml-matlab-v3.6-2015-07-07/prior/priorWeibull.m000644 000765 000024 00000001654 12441263151 021063 0ustar00carlstaff000000 000000 function [lp,dlp] = priorWeibull(lam,k,x) % Univariate Weibull hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = (k/lam) * (x/lam)^(k-1) * exp(-(x/lam)^k) % % where lam(1x1) is the scale parameter, k(1x1) is the scale parameter % and x(1xN) contains query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-08. % % See also PRIORDISTRIBUTIONS.M. if nargin<2, error('lam and k parameters need to be provided'), end if ~(isscalar(lam)&&isscalar(k)) error('lam and k parameters need to be scalars'), end if nargin<3,lp = lam*(-log(rand))^(1/k); return, end % return a sample lp = log(k/lam) + (k-1)*log(x/lam) - (x/lam).^k; dlp = (k-1)./x - (k/lam)*(x/lam).^(k-1); lp(x<0) = -inf; dlp(x<0) = 0;gpml-matlab-v3.6-2015-07-07/prior/priorSmoothBox2.m000644 000765 000024 00000003762 12546444650 021501 0ustar00carlstaff000000 000000 function [lp,dlp] = priorSmoothBox2(a,b,eta,x) % Univariate smoothed box prior distribution with quadratic decay in the log % domain and infinite support over the whole real axis. % The plateau pdf is built by cutting a Gaussian into two parts and % inserting a uniform distribution from a to b. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % / G(x|a,s^2), x<=a, % p(x) = 1/(w*(1/eta+1)) * | 1 , a0 is the slope parameter and x(1xN) contains query hyperparameters % for prior evaluation. Larger values of eta make the distribution more % box-like. % % The distribution p(x) has mean and variance given by: % mu = (a+b)/2 and s2 = (w/2)^2*(eta^3/3+eta^2+4*eta/pi+2/pi)/(eta^3+eta^2). % % /------------\ % / \ % -------- | | --------> x % a b % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Jose Vallet and Hannes Nickisch, 2015-03-27. % % See also PRIORDISTRIBUTIONS.M. if nargin<3, error('a, b and eta parameters need to be provided'), end if b<=a, error('b must be greater than a.'), end if ~(isscalar(a)&&isscalar(b)&&isscalar(eta)) error('a, b and eta parameters need to be scalar values') end w = abs(b-a); sab = w/(eta*sqrt(2*pi)); % box width and boundary slope if nargin<4 % return a sample if randb; lp(i) = lp(i) - (x(i)-b).^2/(2*sab^2); dlp(i) = (b-x(i))/sab^2;gpml-matlab-v3.6-2015-07-07/prior/priorDeltaMulti.m000644 000765 000024 00000001032 12441263151 021512 0ustar00carlstaff000000 000000 function [lp,dlp] = priorDeltaMulti(x) % Dummy hyperparameter prior distribution to fix the value of a hyperparameter. % The function is not intended to be evaluated but exists merely to make % the user aware of the possibility to use it. The function is equivalent % to priorClampedMulti. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-12-06. % % See also PRIORDISTRIBUTIONS.M. error('The function is not intended to be called directly.')gpml-matlab-v3.6-2015-07-07/prior/priorInvGauss.m000644 000765 000024 00000002174 12441263151 021215 0ustar00carlstaff000000 000000 function [lp,dlp] = priorInvGauss(mu,lam,x) % Univariate Inverse Gaussian hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = exp(-lam*(x-mu)^2/(2*mu^2*x)) / sqrt(2*pi*x^3/lam) % % where mu(1x1) is the mean parameter, lam(1x1) is the scale parameter % and x(1xN) contains query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-08. % % See also PRIORDISTRIBUTIONS.M. if nargin<2, error('mu and lam parameters need to be provided'), end if ~(isscalar(mu)&&isscalar(lam)) error('mu and lam parameters need to be scalars'),end if nargin<3 % return a sample n = randn; y = n*n; r = mu + mu*(mu*y-sqrt(4*mu*lam*y+mu^2*y^2))/(2*lam); z = rand; if z<=mu/(mu+r) lp = r; else lp = mu^2/r; end return end lp = -lam*(x-mu).^2./(2*mu^2*x) - log(2*pi*x.^3/lam)/2; q = (x-mu)./x; dlp = -lam*q.*(2-q)/(2*mu^2) - 3./(2*x); lp(x<0) = -inf; dlp(x<0) = 0;gpml-matlab-v3.6-2015-07-07/prior/priorDelta.m000644 000765 000024 00000001020 12441263151 020474 0ustar00carlstaff000000 000000 function [lp,dlp] = priorDelta(x) % Dummy hyperparameter prior distribution to fix the value of a hyperparameter. % The function is not intended to be evaluated but exists merely to make % the user aware of the possibility to use it. The function is equivalent % to priorClamped. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-12-06. % % See also PRIORDISTRIBUTIONS.M. error('The function is not intended to be called directly.')gpml-matlab-v3.6-2015-07-07/prior/priorGaussMulti.m000644 000765 000024 00000003466 12441263151 021560 0ustar00carlstaff000000 000000 function [lp,dlp] = priorGaussMulti(mu,s2,x) % Multivariate Gaussian hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = exp(-r2/2) / sqrt(det(2*pi*s2)) where r2(x) = (x-mu)'*inv(s2)*(x-mu), % % mu(Dx1) is the mean parameter, s2(Dx1) or s2(DxD) is the variance parameter % and x(DxN) contains query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-12. % % See also PRIORDISTRIBUTIONS.M, PRIORGAUSS.M. if nargin<2, error('mu and s2 parameters need to be provided'), end if ndims(mu)~=2 || size(mu,2)~=1, error('mu needs to be (Dx1)'), end D = size(mu,1); s2_ok = ndims(s2)==2 && all(size(s2)==[D,1] | size(s2)==[D,D]); if ~s2_ok, error('s2 needs to be (DxD) or (Dx1)'), end if size(s2,2)==D % full multivariate case s = chol(s2)'; lds = sum(log(diag(s))); % lds = log(det(s2))/2 else % diagonal covariance s = sqrt(s2); lds = sum(log(s)); end if nargin<3 % return a random sample lp = randn(D,1); % unit sample if size(s,2)==D, lp = s*lp+mu; else lp = s.*lp+mu; end % affine transformation return end if D==1 && size(x,1)>1 % both mu/s2 scalar => inflate D = size(x,1); mu = mu*ones(D,1); s = s*ones(D,1); lds = D*lds; end if ~(ndims(x)==2 && size(x,1)==D), error('x needs to be (Dxn)'), end oN = ones(1,size(x,2)); if size(s,2)==D xs = s\(x-mu*oN); dlp = -s'\xs; else xs = (x-mu*oN)./(s*oN); dlp = -xs./(s*oN); end lp = -sum(xs.^2,1)/2 - D*log(2*pi)/2 - lds;gpml-matlab-v3.6-2015-07-07/prior/priorGamma.m000644 000765 000024 00000002515 12441263151 020477 0ustar00carlstaff000000 000000 function [lp,dlp] = priorGamma(k,t,x) % Univariate Gamma hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = exp(x/t)/gamma(k)*x^(k-1)/t^k % % where k(1x1) is the shape parameter, t(1x1) is the scale parameter % and x(1xN) contains query hyperparameters for prior evaluation. % % Sampling is done using the algorithm from p. 53 of the paper Generating gamma % variates by a modified rejection technique by J.H. Ahrens and U. Dieter, 1982, % ACM, 25, 47–54. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-08. % % See also PRIORDISTRIBUTIONS.M. if nargin<2, error('k and t parameters need to be provided'), end if ~(isscalar(k)&&isscalar(t)) error('k and t parameters need to be scalars'), end if nargin<3 m = 1; d = k-floor(k); % fractional part v0 = exp(1)/(exp(1)+d); while true v = rand(3,1); if v(1)<=v0 r = v(2)^(1/d); s = v(3)*r^(d-1); else r = 1-log(v(2)); s = v(3)*exp(-r); end if s<=r^(d-1)*exp(-r), break, end m = m+1; end u = rand(floor(k),1); lp = t*(r-sum(log(u))); return end lx = log(x); lp = -gammaln(k) - k*log(t) + (k-1)*lx - x/t; dlp = (k-1)./x - 1/t; lp(x<0) = -inf; dlp(x<0) = 0;gpml-matlab-v3.6-2015-07-07/prior/priorGauss.m000644 000765 000024 00000001565 12441263151 020543 0ustar00carlstaff000000 000000 function [lp,dlp] = priorGauss(mu,s2,x) % Univariate Gaussian hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = exp(-(x-mu)^2/(2*s2)) / sqrt(2*pi*s2), where % % mu(1x1) is the mean parameter, s2(1x1) is the variance parameter and % x(1xN) contains query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-09. % % See also PRIORDISTRIBUTIONS.M. if nargin<2, error('mu and s2 parameters need to be provided'), end if ~(isscalar(mu)&&isscalar(s2)) error('mu and s2 parameters need to be scalars') end if nargin<3, lp = sqrt(s2)*randn+mu; return, end % return a sample lp = -(x-mu).^2/(2*s2) - log(2*pi*s2)/2; dlp = -(x-mu)/s2;gpml-matlab-v3.6-2015-07-07/prior/priorClamped.m000644 000765 000024 00000001020 12441263151 021010 0ustar00carlstaff000000 000000 function [lp,dlp] = priorClamped(x) % Dummy hyperparameter prior distribution to fix the value of a hyperparameter. % The function is not intended to be evaluated but exists merely to make % the user aware of the possibility to use it. The function is equivalent % to priorDelta. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-12-06. % % See also PRIORDISTRIBUTIONS.M. error('The function is not intended to be called directly.')gpml-matlab-v3.6-2015-07-07/prior/priorMix.m000644 000765 000024 00000002362 12441263151 020212 0ustar00carlstaff000000 000000 function [lp,dlp] = priorMix(w,priors,x) % Mixture of Univariate or Multivariate hyperparameter prior distributions. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = sum_i w_i p_i(x), % % w(bx1) is the mixture weight vector (nonnegative, normalised to add up to 1), % priors(bx1) is a cell array containing the priors to be mixed and x(1xN) % contains query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-08. % % See also PRIORDISTRIBUTIONS.M. if nargin<2, error('specification of w and priors required'), end n = numel(w); w = w(:); if abs(sum(w)-1)>1e-9, error('w needs to sum up to 1'), end if any(w<0), error('w needs to be nonnegative'), end for i=1:n, if ~iscell(priors{i}), priors{i} = {priors{i}}; end, end % sample if nargin<3, i = 1+nnz(rand>cumsum(w)); lp = feval(priors{i}{:}); return, end lpi = zeros(numel(x),n); dlpi = zeros(numel(x),n); for i=1:n, [lpi(:,i),dlpi(:,i)] = feval(priors{i}{:},x(:)); end mx = max(lpi,[],2); lp = log(exp(lpi-mx*ones(1,n))*w) + mx; pi = exp(lpi); dlp = ((pi.*dlpi)*w)./(pi*w);gpml-matlab-v3.6-2015-07-07/prior/priorTransform.m000644 000765 000024 00000002256 12441263151 021432 0ustar00carlstaff000000 000000 function [lp,dlp] = priorTransform(g,dg,ig,prior,x) % Transformed univariate hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % prior(y), where y = g(x), % % g is the transformation function (monotonically increasing and invertable), % dg is its derivative, ig is its inverse, prior is the distribution to be % transformed and x(1xN) contains query hyperparameters for prior evaluation. % Note that p(x) = prior(g(x)) is not necessarily normalised w.r.t. x. % % For example, to put a nonnegative gamma prior on the lenthscale % parameter ell of covSEiso, one can use % {@priorTransform,@exp,@exp,@log,{@priorGamma,k,t}} % since ell is represented in the log domain. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-08. % % See also PRIORDISTRIBUTIONS.M. if nargin<4, error('g, dg, ig and prior parameters need to be provided'), end if nargin<5, lp = ig(feval(prior{:})); return, end % apply inverse sampling if ~iscell(prior), prior = {prior}; end [lp,dlp] = feval(prior{:},g(x)); dlp = dlp.*dg(x);gpml-matlab-v3.6-2015-07-07/prior/priorT.m000644 000765 000024 00000002170 12441263151 017655 0ustar00carlstaff000000 000000 function [lp,dlp] = priorT(mu,s2,nu,x) % Univariate Student's t hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = ( 1 + (x-mu)^2/(s2*(nu-2)) )^(-(nu+1)/2) / Z, where % Z = gamma(nu/2) * sqrt(pi*(nu-2)*s2) / gamma((nu+1)/2), % % mu(1x1) is the mean parameter, s2(1x1) is the variance parameter, % nu(1x1) > 2 is the degrees of freedom parameter and x(1xN) contains query % hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-10-16. % % See also PRIORDISTRIBUTIONS.M. if nargin<3, error('mu, s2 and nu parameters need to be provided'), end if ~(isscalar(mu)&&isscalar(s2)&&isscalar(nu)) error('mu, s2 and nu parameters need to be scalars') end if nargin<4, lp = mu + sqrt(s2*(nu-2)/priorGamma(nu/2,2))*randn; return, end lZ = gammaln((nu+1)/2)-gammaln(nu/2)-log(pi*(nu-2)*s2)/2; lp = -(nu+1)/2*log(1+(x-mu).^2/(s2*(nu-2))) + lZ; dlp = (nu+1)*(mu-x)./(mu^2-2*mu*x+s2*(nu-2)+x.^2);gpml-matlab-v3.6-2015-07-07/prior/priorTMulti.m000644 000765 000024 00000004270 12441263151 020673 0ustar00carlstaff000000 000000 function [lp,dlp] = priorTMulti(mu,s2,nu,x) % Multivariate Laplace hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = ( 1 + r2/(nu-2) )^(-(nu+D)/2) / Z, where % Z = gamma(nu/2) * sqrt(det(pi*(nu-2)*s2)) / gamma((nu+D)/2), and % r2(x) = (x-mu)'*inv(s2)*(x-mu), % % further mu(Dx1) is the mean parameter, s2(Dx1) or s2(DxD) is the variance % parameter, nu(1x1) > 2 is the degrees of freedom parameter and x(DxN) contains % query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Hannes Nickisch, 2014-10-16. % % See also PRIORDISTRIBUTIONS.M, PRIORLAPLACE.M. if nargin<3, error('mu, s2 and nu parameters need to be provided'), end if ndims(mu)~=2 || size(mu,2)~=1, error('mu needs to be (Dx1)'), end D = size(mu,1); s2_ok = ndims(s2)==2 && all(size(s2)==[D,1] | size(s2)==[D,D]); if ~s2_ok, error('s2 needs to be (DxD) or (Dx1)'), end if size(s2,2)==D % full multivariate case s = chol(s2)'; lds = sum(log(diag(s))); % lds = log(det(s2))/2 else % diagonal covariance s = sqrt(s2); lds = sum(log(s)); end if ~isscalar(nu), error('nu parameter needs to be a scalar'), end if nargin<4 % return a random sample lp = randn(D,1); for d=1:D, lp(d) = sqrt((nu-2)/priorGamma(nu/2,2)) * lp(d); end % unit sample if size(s,2)==D, lp = s*lp+mu; else lp = s.*lp+mu; end % affine transformation return end if D==1 && size(x,1)>1 % both mu/s2 scalar => inflate D = size(x,1); mu = mu*ones(D,1); s = s*ones(D,1); lds = D*lds; end if ~(ndims(x)==2 && size(x,1)==D), error('x needs to be (Dxn)'), end oN = ones(1,size(x,2)); oD = ones(D,1); if size(s,2)==D xm = x-mu*oN; xs = s\xm; else xm = x-mu*oN; xs = xm./(s*oN); end lZ = gammaln((nu+D)/2) - gammaln(nu/2) - D*log(pi*(nu-2))/2 - lds; r2 = sum(xs.^2,1); lp = -(nu+D)/2*log(1+r2/(nu-2)) + lZ; dlp = -(nu+D)/(nu-2)*xs./(oD*(1+r2/(nu-2))); if size(s,2)==D, dlp = s'\dlp; else dlp = dlp./(s*oN); endgpml-matlab-v3.6-2015-07-07/prior/priorClampedMulti.m000644 000765 000024 00000001032 12441263151 022026 0ustar00carlstaff000000 000000 function [lp,dlp] = priorClampedMulti(x) % Dummy hyperparameter prior distribution to fix the value of a hyperparameter. % The function is not intended to be evaluated but exists merely to make % the user aware of the possibility to use it. The function is equivalent % to priorDeltaMulti. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-12-06. % % See also PRIORDISTRIBUTIONS.M. error('The function is not intended to be called directly.')gpml-matlab-v3.6-2015-07-07/inf/infEP.m000644 000765 000024 00000013640 12504606112 017022 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infEP(hyp, mean, cov, lik, x, y) % Expectation Propagation approximation to the posterior Gaussian Process. % The function takes a specified covariance function (see covFunctions.m) and % likelihood function (see likFunctions.m), and is designed to be used with % gp.m. See also infMethods.m. In the EP algorithm, the sites are % updated in random order, for better performance when cases are ordered % according to the targets. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2013-09-13. % % See also INFMETHODS.M. persistent last_ttau last_tnu % keep tilde parameters between calls tol = 1e-4; max_sweep = 10; min_sweep = 2; % tolerance to stop EP iterations inf = 'infEP'; n = size(x,1); if isnumeric(cov), K = cov; % use provided covariance matrix else K = feval(cov{:}, hyp.cov, x); end % evaluate the covariance matrix if isnumeric(mean), m = mean; % use provided mean vector else m = feval(mean{:}, hyp.mean, x); end % evaluate the mean vector % A note on naming: variables are given short but descriptive names in % accordance with Rasmussen & Williams "GPs for Machine Learning" (2006): mu % and s2 are mean and variance, nu and tau are natural parameters. A leading t % means tilde, a subscript _ni means "not i" (for cavity parameters), or _n % for a vector of cavity parameters. N(f|mu,Sigma) is the posterior. % marginal likelihood for ttau = tnu = zeros(n,1); equals n*log(2) for likCum* nlZ0 = -sum(feval(lik{:}, hyp.lik, y, m, diag(K), inf)); if any(size(last_ttau) ~= [n 1]) % find starting point for tilde parameters ttau = zeros(n,1); tnu = zeros(n,1); % init to zero if no better guess Sigma = K; % initialize Sigma and mu, the parameters of .. mu = m; nlZ = nlZ0; % .. the Gaussian posterior approximation else ttau = last_ttau; tnu = last_tnu; % try the tilde values from previous call [Sigma,mu,L,alpha,nlZ] = epComputeParams(K, y, ttau, tnu, lik, hyp, m, inf); if nlZ > nlZ0 % if zero is better .. ttau = zeros(n,1); tnu = zeros(n,1); % .. then init with zero instead Sigma = K; % initialize Sigma and mu, the parameters of .. mu = m; nlZ = nlZ0; % .. the Gaussian posterior approximation end end nlZ_old = Inf; sweep = 0; % converged, max. sweeps or min. sweeps? while (abs(nlZ-nlZ_old) > tol && sweep < max_sweep) || sweep tol error('maximum number of sweeps exceeded in function infEP') end last_ttau = ttau; last_tnu = tnu; % remember for next call post.alpha = alpha; post.sW = sqrt(ttau); post.L = L; % return posterior params if nargout>2 % do we want derivatives? dnlZ = hyp; % allocate space for derivatives tau_n = 1./diag(Sigma)-ttau; % compute the log marginal likelihood nu_n = mu./diag(Sigma)-tnu; % vectors of cavity parameters sW = sqrt(ttau); F = alpha*alpha'-repmat(sW,1,n).*solve_chol(L,diag(sW)); % covariance hypers for i=1:length(hyp.cov) dK = feval(cov{:}, hyp.cov, x, [], i); dnlZ.cov(i) = -sum(sum(F.*dK))/2; end for i = 1:numel(hyp.lik) % likelihood hypers dlik = feval(lik{:}, hyp.lik, y, nu_n./tau_n, 1./tau_n, inf, i); dnlZ.lik(i) = -sum(dlik); end [junk,dlZ] = feval(lik{:}, hyp.lik, y, nu_n./tau_n, 1./tau_n, inf);% mean hyps for i = 1:numel(hyp.mean) dm = feval(mean{:}, hyp.mean, x, i); dnlZ.mean(i) = -dlZ'*dm; end end % function to compute the parameters of the Gaussian approximation, Sigma and % mu, and the negative log marginal likelihood, nlZ, from the current site % parameters, ttau and tnu. Also returns L (useful for predictions). function [Sigma,mu,L,alpha,nlZ] = epComputeParams(K,y,ttau,tnu,lik,hyp,m,inf) n = length(y); % number of training cases sW = sqrt(ttau); % compute Sigma and mu L = chol(eye(n)+sW*sW'.*K); % L'*L=B=eye(n)+sW*K*sW V = L'\(repmat(sW,1,n).*K); Sigma = K - V'*V; alpha = tnu-sW.*solve_chol(L,sW.*(K*tnu+m)); mu = K*alpha+m; v = diag(Sigma); tau_n = 1./diag(Sigma)-ttau; % compute the log marginal likelihood nu_n = mu./diag(Sigma)-tnu; % vectors of cavity parameters lZ = feval(lik{:}, hyp.lik, y, nu_n./tau_n, 1./tau_n, inf); p = tnu-m.*ttau; q = nu_n-m.*tau_n; % auxiliary vectors nlZ = sum(log(diag(L))) - sum(lZ) - p'*Sigma*p/2 + (v'*p.^2)/2 ... - q'*((ttau./tau_n.*q-2*p).*v)/2 - sum(log(1+ttau./tau_n))/2; gpml-matlab-v3.6-2015-07-07/inf/infFITC.m000644 000765 000024 00000013077 12504606112 017247 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infFITC(hyp, mean, cov, lik, x, y) % FITC approximation to the posterior Gaussian process. The function is % equivalent to infExact with the covariance function: % Kt = Q + G; G = diag(g); g = diag(K-Q); Q = Ku'*inv(Quu)*Ku; % where Ku and Kuu are covariances w.r.t. to inducing inputs xu, snu2 = sn2/1e6 % is the noise of the inducing inputs and Quu = Kuu + snu2*eye(nu). % We fixed the standard deviation of the inducing inputs snu to be a one per mil % of the measurement noise's standard deviation sn. % The implementation exploits the Woodbury matrix identity % inv(Kt) = inv(G) - inv(G)*V'*inv(eye(nu)+V*inv(G)*V')*V*inv(G) % in order to be applicable to large datasets. The computational complexity % is O(n nu^2) where n is the number of data points x and nu the number of % inducing inputs in xu. % % The function takes a specified covariance function (see covFunctions.m) and % likelihood function (see likFunctions.m), and is designed to be used with % gp.m and in conjunction with covFITC and likGauss. % % The inducing points can be specified through 1) the 2nd covFITC parameter or % by 2) providing a hyp.xu hyperparameters. Note that 2) has priority over 1). % In case 2) is provided and derivatives dnlZ are requested, there will also be % a dnlZ.xu field allowing to optimise w.r.t. to the inducing points xu. However % the derivatives dnlZ.xu can only be computed for one of the following eight % covariance functions: cov{Matern|PP|RQ|SE}{iso|ard}. % % Copyright (c) by Ed Snelson, Carl Edward Rasmussen % and Hannes Nickisch, 2013-10-28. % % See also INFMETHODS.M, COVFITC.M. if iscell(lik), likstr = lik{1}; else likstr = lik; end if ~ischar(likstr), likstr = func2str(likstr); end if ~strcmp(likstr,'likGauss') % NOTE: no explicit call to likGauss error('Inference with inFITC only possible with Gaussian likelihood.'); end cov1 = cov{1}; if isa(cov1, 'function_handle'), cov1 = func2str(cov1); end if ~strcmp(cov1,'covFITC'); error('Only covFITC supported.'), end % check cov if isfield(hyp,'xu'), cov{3} = hyp.xu; end % hyp.xu is provided, replace cov{3} [diagK,Kuu,Ku] = feval(cov{:}, hyp.cov, x); % evaluate covariance matrix m = feval(mean{:}, hyp.mean, x); % evaluate mean vector [n, D] = size(x); nu = size(Kuu,1); cov{3} = reshape(cov{3},[],D); sn2 = exp(2*hyp.lik); % noise variance of likGauss snu2 = 1e-6*sn2; % hard coded inducing inputs noise Luu = chol(Kuu+snu2*eye(nu)); % Kuu + snu2*I = Luu'*Luu V = Luu'\Ku; % V = inv(Luu')*Ku => V'*V = Q g_sn2 = diagK + sn2 - sum(V.*V,1)'; % g + sn2 = diag(K) + sn2 - diag(Q) Lu = chol(eye(nu) + (V./repmat(g_sn2',nu,1))*V'); % Lu'*Lu=I+V*diag(1/g_sn2)*V' r = (y-m)./sqrt(g_sn2); be = Lu'\(V*(r./sqrt(g_sn2))); iKuu = solve_chol(Luu,eye(nu)); % inv(Kuu + snu2*I) = iKuu post.alpha = Luu\(Lu\be); % return the posterior parameters post.L = solve_chol(Lu*Luu,eye(nu)) - iKuu; % Sigma-inv(Kuu) post.sW = ones(n,1)/sqrt(sn2); % unused for FITC prediction with gp.m if nargout>1 % do we want the marginal likelihood nlZ = sum(log(diag(Lu))) + (sum(log(g_sn2)) + n*log(2*pi) + r'*r - be'*be)/2; if nargout>2 % do we want derivatives? dnlZ = hyp; % allocate space for derivatives al = r./sqrt(g_sn2) - (V'*(Lu\be))./g_sn2; % al = (Kt+sn2*eye(n))\(y-m) B = iKuu*Ku; w = B*al; W = Lu'\(V./repmat(g_sn2',nu,1)); for i = 1:numel(hyp.cov) [ddiagKi,dKuui,dKui] = feval(cov{:}, hyp.cov, x, [], i); % eval cov deriv R = 2*dKui-dKuui*B; v = ddiagKi - sum(R.*B,1)'; % diag part of cov deriv dnlZ.cov(i) = (ddiagKi'*(1./g_sn2)+w'*(dKuui*w-2*(dKui*al))-al'*(v.*al)... - sum(W.*W,1)*v - sum(sum((R*W').*(B*W'))) )/2; end diag_dK = 1./g_sn2 - sum(W.*W,1)' - al.*al; % diag(dnlZ/dK) dnlZ.lik = sn2*sum(diag_dK); % since snu2 is a fixed fraction of sn2, there is a covariance-like term in BWt = B*W'; % the derivative as well dKuui = 2*snu2; R = -dKuui*B; v = -sum(R.*B,1)'; % diag part of cov deriv dnlZ.lik = dnlZ.lik + (w'*dKuui*w -al'*(v.*al)... - sum(W.*W,1)*v - sum(sum((R*W').*BWt)) )/2; for i = 1:numel(hyp.mean) dnlZ.mean(i) = -feval(mean{:}, hyp.mean, x, i)'*al; end if isfield(hyp,'xu') % derivatives w.r.t. inducing points xu xu = cov{3}; cov = cov{2}; % get the non FITC part of the covariance function Kpu = cov_deriv_sq_dist(cov,hyp.cov,xu,x); % d K(xu,x ) / d D^2 Kpuu = cov_deriv_sq_dist(cov,hyp.cov,xu); % d K(xu,xu) / d D^2 if iscell(cov), covstr = cov{1}; else covstr = cov; end if ~ischar(covstr), covstr = func2str(covstr); end if numel(strfind(covstr,'iso'))>0 % characteristic length scale e = 2*exp(-2*hyp.cov(1)); else e = 2*exp(-2*hyp.cov(1:D)); end v = diag_dK-1./g_sn2; % BdK = B * ( diag(diag(dnlZ/dK)) - dnlZ/dK ) BdK = B.*repmat(v',nu,1) + BWt*W + (B*al)*al'; A = Kpu.*BdK; C = Kpuu.*(BdK*B'); C = diag(sum(C,2)-sum(A,2)) - C; dnlZ.xu = A*x*diag(e) + C*xu*diag(e); % bring in data and inducing points end end endgpml-matlab-v3.6-2015-07-07/inf/infVB.m000644 000765 000024 00000014052 12504606112 017023 0ustar00carlstaff000000 000000 function [post, nlZ, dnlZ] = infVB(hyp, mean, cov, lik, x, y, opt) % Variational approximation to the posterior Gaussian process. % The function takes a specified covariance function (see covFunctions.m) and % likelihood function (see likFunctions.m), and is designed to be used with % gp.m. See also infMethods.m. % % Minimisation of an upper bound on the negative marginal likelihood using a % sequence of infLaplace calls where the smoothed likelihood % likVB(f) = lik(..,g,..) * exp(b*(f-g)), g = sign(f-z)*sqrt((f-z)^2+v)+z, where % v .. marginal variance = (positive) smoothing width, and % lik .. lik function such that p(y|f)=lik(..,f,..). % % The problem is convex whenever the likelihood is log-concave. At the end, the % optimal width W is obtained analytically. % % Copyright (c) by Hannes Nickisch 2014-03-20. % % See also INFMETHODS.M. n = size(x,1); if isnumeric(cov), K = cov; % use provided covariance matrix else K = feval(cov{:}, hyp.cov, x); end % evaluate the covariance matrix if isnumeric(mean), m = mean; % use provided mean vector else m = feval(mean{:}, hyp.mean, x); end % evaluate the mean vector if iscell(lik), likstr = lik{1}; else likstr = lik; end if ~ischar(likstr), likstr = func2str(likstr); end if nargin<=6, opt = []; end % make opt variable available if ~isfield(opt,'postL'), opt.postL = false; end % not compute L in infLaplace if isfield(opt,'out_nmax'), out_nmax = opt.out_nmax; % maximal no of outer loops else out_nmax = 15; end % default value if isfield(opt,'out_tol'), out_tol = opt.out_tol; % outer loop convergence else out_tol = 1e-5; end % default value sW = ones(n,1); % init with some reasonable value opt.postL = false; % avoid computation of L inside infLaplace for i=1:out_nmax U = chol(eye(n)+sW*sW'.*K)'\(repmat(sW,1,n).*K); v = diag(K) - sum(U.*U,1)'; % v = diag( inv(inv(K)+diag(sW.*sW)) ) post = infLaplace(hyp, m, K, {@likVB,v,lik}, x, y, opt); % post.sW is very different from the optimal sW for non Gaussian likelihoods sW_old = sW; f = m+K*post.alpha; % posterior mean [lp,dlp,d2lp,sW,b,z] = feval(@likVB,v,lik,hyp.lik,y,f); if max(abs(sW-sW_old))2 % do we want derivatives? iKtil = repmat(sW,1,n).*solve_chol(post.L,diag(sW));% sW*B^-1*sW=inv(K+inv(W)) dnlZ = hyp; % allocate space for derivatives for j=1:length(hyp.cov) % covariance hypers dK = feval(cov{:}, hyp.cov, x, [], j); if j==1, w = iKtil*(c.*ga); end dnlZ.cov(j) = sum(sum(iKtil.*dK))/2 - (w'*dK*w)/2; end if ~strcmp(likstr,'likGauss') % likelihood hypers for j=1:length(hyp.lik) sign_fmz = 2*(f-z>=0)-1; % strict sign mapping; sign(0) = 1 g = sign_fmz.*sqrt((f-z).^2 + v) + z; dhhyp = -2*feval(lik{:},hyp.lik,y,g,[],'infLaplace',j); dnlZ.lik(j) = sum(dhhyp)/2; end else % special treatment for the Gaussian case dnlZ.lik = sum(sum( (post.L'\eye(n)).^2 )) - exp(2*hyp.lik)*(alpha'*alpha); end for j=1:length(hyp.mean) % mean hypers dm = feval(mean{:}, hyp.mean, x, j); dnlZ.mean(j) = -alpha'*dm; end end % Smoothed likelihood function; instead of p(y|f)=lik(..,f,..) compute % likVB(f) = lik(..,g,..)*exp(b*(f-g)), g = sign(f-z)*sqrt((f-z)^2+v)+z, where % v .. marginal variance = (positive) smoothing width, and % lik .. lik function such that feval(lik{:},varargin{:}) yields a result. % The smoothing results from a lower bound on the likelihood: % p(y|f) \ge exp( (b+z/ga)*f - f.^2/(2*ga) - h(ga)/2 ) function [varargout] = likVB(v, lik, varargin) [b,z] = feval(lik{:},varargin{1:2},[],zeros(size(v)),'infVB'); f = varargin{3}; % obtain location of evaluation sign_fmz = 2*(f-z>=0)-1; % strict sign mapping; sign(0) = 1 g = sign_fmz.*sqrt((f-z).^2 + v) + z; varargin{3} = g; id = v==0 | abs(f./sqrt(v+eps))>1e10; % correct asymptotics of f -> +/-Inf varargout = cell(nargout,1); % allocate output, eval lik(..,g,..) [varargout{1:min(nargout,3)}] = feval(lik{:},varargin{1:3},[],'infLaplace'); if nargout>0 lp = varargout{1}; varargout{1} = lp + b.*(f-g); varargout{1}(id) = lp(id); % correct asymptotics of f -> +/-Inf if nargout>1 % first derivative dg_df = (abs(f-z)+eps)./(abs(g-z)+eps); % stabilised dg/df for v=0, f=0 dlp = varargout{2}; varargout{2} = dlp.*dg_df + b.*(1-dg_df); varargout{2}(id) = dlp(id); % correct asymptotics of f -> +/-Inf if nargout>2 % second derivative d2lp = varargout{3}; g_e = g-z + sign_fmz*eps; v_g3 = v./(g_e.*g_e.*g_e); % stabilised v./g.^3 to cover v=0 and f=0 varargout{3} = (dlp-b).*v_g3 + d2lp.*dg_df.*dg_df; varargout{3}(id) = d2lp(id); % correct asymptotics of f -> +/-Inf if nargout>3 W = abs( (b-dlp)./(g-z+sign_fmz/1.5e8) ); % optimal sW value varargout{4} = sqrt(W); if nargout>4 varargout{5} = b; if nargout>5 varargout{6} = z; end end end end end end gpml-matlab-v3.6-2015-07-07/inf/infKL.m000644 000765 000024 00000024061 12504606112 017023 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infKL(hyp, mean, cov, lik, x, y) % Approximation to the posterior Gaussian Process by minimization of the % KL-divergence. The function is structurally very similar to infEP; the % only difference being the local divergence measure minimised. % In infEP, one minimises KL(p,q) whereas in infKL one minimises KL(q,p) % locally be iterating over the sites. For log-concave likelihoods, the % latter minimisation constitutes a 2d joint convex problem, so convergence % is guaranteed. % The function takes a specified covariance function (see covFunctions.m) and % likelihood function (see likFunctions.m), and is designed to be used with % gp.m. See also infMethods.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2013-09-13. % % See also INFMETHODS.M. out = false; persistent last_ttau last_tnu % keep tilde parameters between calls tol = 1e-3; max_sweep = 15; min_sweep = 2; % tolerance to stop KL iterations n = size(x,1); K = feval(cov{:}, hyp.cov, x); % evaluate the covariance matrix m = feval(mean{:}, hyp.mean, x); % evaluate the mean vector % A note on naming: variables are given short but descriptive names in % accordance with Rasmussen & Williams "GPs for Machine Learning" (2006): mu % and s2 are mean and variance, nu and tau are natural parameters. A leading t % means tilde, a subscript _ni means "not i" (for cavity parameters), or _n % for a vector of cavity parameters. N(f|mu,Sigma) represents the posterior. % marginal likelihood for ttau = tnu = zeros(n,1); equals n*log(2) for likCum* nlZ0 = -sum(likKL(diag(K),lik,hyp.lik,y,m)); if any(size(last_ttau) ~= [n 1]) % find starting point for tilde parameters ttau = zeros(n,1); % initialize to zero if we have no better guess tnu = zeros(n,1); Sigma = K; % initialize Sigma and mu, the parameters of .. mu = m; % .. the Gaussian posterior approximation nlZ = nlZ0; else ttau = last_ttau; % try the tilde values from previous call tnu = last_tnu; [Sigma,mu,L,alpha,nlZ] = klComputeParams(K,y,ttau,tnu,lik,hyp,m); if nlZ > nlZ0 % if zero is better .. ttau = zeros(n,1); % .. then initialize with zero instead tnu = zeros(n,1); Sigma = K; % initialize Sigma and mu, the parameters of .. mu = m; % .. the Gaussian posterior approximation nlZ = nlZ0; end end nlZ_old = Inf; sweep = 0; % converged, max. sweeps or min. sweeps? while (nlZ_old-nlZ > tol && sweep < max_sweep) || sweep tol error('maximum number of sweeps exceeded in function infKL') end last_ttau = ttau; last_tnu = tnu; % remember for next call post.alpha = alpha; post.sW = sqrt(ttau); post.L = L; % return posterior params if nargout>2 % do we want derivatives? v = diag(Sigma); [lp,df,d2f,dv] = likKL(v,lik,hyp.lik,y,mu); dnlZ = hyp; % allocate space for derivatives for j=1:length(hyp.cov) % covariance hypers dK = feval(cov{:},hyp.cov,x,[],j); AdK = A*dK; z = diag(AdK) + sum(A.*AdK,2) - sum(A'.*AdK,1)'; dnlZ.cov(j) = alpha'*dK*(alpha/2-df) - z'*dv; end for j=1:length(hyp.lik) % likelihood hypers lp_dhyp = likKL(v,lik,hyp.lik,y,K*post.alpha+m,[],[],j); dnlZ.lik(j) = -sum(lp_dhyp); end for j=1:length(hyp.mean) % mean hypers dm = feval(mean{:}, hyp.mean, x, j); dnlZ.mean(j) = -alpha'*dm; end end % function to compute the parameters of the Gaussian approximation, Sigma and % mu, from the current site parameters, ttau and tnu. Also returns L and upper % bound on negative marginal likelihood. function [Sigma,mu,L,alpha,nlZ,A] = klComputeParams(K,y,ttau,tnu,lik,hyp,m) n = length(tnu); % number of training cases sW = sqrt(ttau); % compute Sigma and mu L = chol(eye(n)+sW*sW'.*K); % L'*L=B=eye(n)+sW*K*sW V = L'\(repmat(sW,1,n).*K); Sigma = K - V'*V; alpha = tnu-sW.*solve_chol(L,sW.*(K*tnu+m)); mu = K*alpha+m; v = diag(Sigma); A = (eye(n)+K*diag(ttau))\eye(n); % A = Sigma*inv(K) lp = likKL(v,lik,hyp.lik,y,mu); % evaluate likelihood nlZ = -sum(lp) - (logdet(A)-alpha'*(mu-m)-trace(A)+n)/2; % upper bound on -lZ % We compute the Gaussian Q(f)=N(f|m,s^2) minimising the KL divergence % KL(Q||P) where P is the product of the cavity distribution q_n(f) and the % likelihood p(y|f) such that P(f) = 1/Z * q_n(f)*p(y|f). % The cavity distribution q_n(f) is an unnormalised Gaussian with precision % parameter tau_n and location parameter nu_n, hence the cavity can be written % as q_n(f) = exp(nu_n*f-tau_n/2*f^2). % The minimisation is convex iff. the likelihood p(y|f) is log-concave. The % optimisation is performed using Newton descent with backtracking line search. function [m,s,kl] = klmin(lik, hyp, y, nu_n, tau_n) ep = 1e-9; % tiny Hessian ridge for stability gthresh = 1e-8; % gradient convergence threshold lik_str = lik{1}; if ~ischar(lik_str), lik_str = func2str(lik_str); end if strcmp(lik_str,'likGauss') % likGauss can be done analytically sn2 = exp(2*hyp); s = 1/sqrt(1/sn2+tau_n); m = s^2*(nu_n+y/sn2); % init variables to minimum else s = 1/sqrt(tau_n); m = nu_n/tau_n; % init variables to cavity distribution end ll = likKL(s^2,lik,hyp,y,m); % evaluate likelihood kl = (s^2+m^2)*tau_n/2 - log(s) - nu_n*m - ll; % init the KL div up to a const for i=1:20 [ll,dm,d2m,dv,d2v,dmdv] = likKL(s^2,lik,hyp,y,m); % evaluate likelihood klold = kl; mold = m; sold = s; % remember last value kl = (s^2+m^2)*tau_n/2 - log(s) - nu_n*m - ll; % KL-divergence up to const dm = tau_n*m-nu_n-dm; d2m = tau_n-d2m; % compute kl derivatives ds = s*tau_n-1/s-2*s*dv; d2s = tau_n+1/s^2-2*dv-(2*s)^2*d2v; dmds=-2*s*dmdv; detH = ((d2m+ep)*(d2s+ep)-dmds^2); % Hessian determinant m = m-(dm*(d2s+ep)-ds*dmds)/detH; s = s-(ds*(d2m+ep)-dm*dmds)/detH; % Newton for j=1:10 % backtracking line search if klold>kl, break, end % we did descend so no need to step back m = (m+mold)/2; s = (s+sold)/2; kl = (s^2+m^2)*tau_n/2 - log(s) - nu_n*m - likKL(s^2,lik,hyp,y,m); end d = abs(dm)+abs(dv); % overall gradient if j==10, m = mold; s = sold; d = 0; end if d0 using the LU decomposition of A function y = logdet(A) [L,U] = lu(A); u = diag(U); if prod(sign(u))~=det(L), error('det(A)<=0'), end y = sum(log(abs(u))); % Gaussian smoothed likelihood function; instead of p(y|f)=lik(..,f,..) compute % log likKL(f) = int log lik(..,t,..) N(f|t,v) dt, where % v .. marginal variance = (positive) smoothing width, and % lik .. lik function such that feval(lik{:},varargin{:}) yields a result. % All return values are separately integrated using Gaussian-Hermite quadrature. function [ll,df,d2f,dv,d2v,dfdv] = likKL(v, lik, varargin) f = varargin{3}; % obtain location of evaluation sv = sqrt(v); % smoothing width lik_str = lik{1}; if ~ischar(lik_str), lik_str = func2str(lik_str); end if strcmp(lik_str,'likLaplace') % likLaplace can be done analytically b = exp(varargin{1})/sqrt(2); y = varargin{2}; mu = (f-y)/b; z = (f-y)./sv; Nz = exp(-z.^2/2)/sqrt(2*pi); Cz = (1+erf(z/sqrt(2)))/2; ll = (1-2*Cz).*mu - 2/b*sv.*Nz - log(2*b); df = (1-2*Cz)/b; d2f = -2*Nz./(b*(sv+eps)); dv = d2f/2; d2v = (z.*z-1)./(v+eps).*d2f/4; dfdv = -z.*d2f./(2*sv+eps); else N = 20; % number of quadrature points [t,w] = gauher(N); % location and weights for Gaussian-Hermite quadrature ll = 0; df = 0; d2f = 0; dv = 0; d2v = 0; dfdv = 0; % init return arguments for i=1:N % use Gaussian quadrature varargin{3} = f + sv*t(i); % coordinate transform of the quadrature points [lp,dlp,d2lp]=feval(lik{:},varargin{1:3},[],'infLaplace',varargin{6:end}); ll = ll + w(i)*lp; % value of the integral df = df + w(i)*dlp; % derivative wrt. mean d2f = d2f + w(i)*d2lp; % 2nd derivative wrt. mean ai = t(i)./(2*sv+eps); dvi = dlp.*ai; dv = dv+w(i)*dvi; % deriv wrt. var d2v = d2v + w(i)*(d2lp.*(t(i)^2/2)-dvi)./(v+eps)/2; % 2nd deriv wrt. var dfdv = dfdv + w(i)*(ai.*d2lp); % mixed second derivatives end endgpml-matlab-v3.6-2015-07-07/inf/infLaplace.m000644 000765 000024 00000017414 12504606112 020062 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infLaplace(hyp, mean, cov, lik, x, y, opt) % Laplace approximation to the posterior Gaussian process. % The function takes a specified covariance function (see covFunctions.m) and % likelihood function (see likFunctions.m), and is designed to be used with % gp.m. See also infMethods.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2013-05-02. % % See also INFMETHODS.M. persistent last_alpha % copy of the last alpha if any(isnan(last_alpha)), last_alpha = zeros(size(last_alpha)); end % prevent if nargin<=6, opt = []; end % make opt variable available if isfield(opt,'postL'), postL = opt.postL; % recompute matrix L for post else postL = true; end % default value inf = 'infLaplace'; n = size(x,1); if isnumeric(cov), K = cov; % use provided covariance matrix else K = feval(cov{:}, hyp.cov, x); end % evaluate the covariance matrix if isnumeric(mean), m = mean; % use provided mean vector else m = feval(mean{:}, hyp.mean, x); end % evaluate the mean vector likfun = @(f) feval(lik{:},hyp.lik,y,f,[],inf); % log likelihood function if any(size(last_alpha)~=[n,1]) % find a good starting point for alpha and f alpha = zeros(n,1); % start at mean if sizes do not match else alpha = last_alpha; % try last one if Psi(alpha,m,K,likfun) > -sum(likfun(m)) % default f==m better => use it alpha = zeros(n,1); end end % switch between optimisation methods alpha = irls(alpha, m,K,likfun, opt); % run optimisation f = K*alpha+m; % compute latent function values last_alpha = alpha; % remember for next call [lp,dlp,d2lp,d3lp] = likfun(f); W = -d2lp; isWneg = any(W<0); post.alpha = alpha; % return the posterior parameters post.sW = sqrt(abs(W)).*sign(W); % preserve sign in case of negative % diagnose optimality err = @(x,y) norm(x-y)/max([norm(x),norm(y),1]); % we need to have alpha = dlp % dev = err(alpha,dlp); if dev>1e-4, warning('Not at optimum %1.2e.',dev), end if postL || nargout>1 if isWneg % switch between Cholesky and LU decomposition mode % For post.L = -inv(K+diag(1./W)), we us the non-default parametrisation. [ldA, iA, post.L] = logdetA(K,W); % A=eye(n)+K*W is as safe as symmetric B nlZ = alpha'*(f-m)/2 - sum(lp) + ldA/2; else sW = post.sW; post.L = chol(eye(n)+sW*sW'.*K); % recompute nlZ = alpha'*(f-m)/2 + sum(log(diag(post.L))-lp); % ..(f-m)/2 -lp +ln|B|/2 end end if nargout>2 % do we want derivatives? dnlZ = hyp; % allocate space for derivatives if isWneg % switch between Cholesky and LU decomposition mode Z = -post.L; % inv(K+inv(W)) g = sum(iA.*K,2)/2; % deriv. of ln|B| wrt W; g = diag(inv(inv(K)+diag(W)))/2 else Z = repmat(sW,1,n).*solve_chol(post.L,diag(sW)); %sW*inv(B)*sW=inv(K+inv(W)) C = post.L'\(repmat(sW,1,n).*K); % deriv. of ln|B| wrt W g = (diag(K)-sum(C.^2,1)')/2; % g = diag(inv(inv(K)+W))/2 end dfhat = g.*d3lp; % deriv. of nlZ wrt. fhat: dfhat=diag(inv(inv(K)+W)).*d3lp/2 for i=1:length(hyp.cov) % covariance hypers dK = feval(cov{:}, hyp.cov, x, [], i); dnlZ.cov(i) = sum(sum(Z.*dK))/2 - alpha'*dK*alpha/2; % explicit part b = dK*dlp; % b-K*(Z*b) = inv(eye(n)+K*diag(W))*b dnlZ.cov(i) = dnlZ.cov(i) - dfhat'*( b-K*(Z*b) ); % implicit part end for i=1:length(hyp.lik) % likelihood hypers [lp_dhyp,dlp_dhyp,d2lp_dhyp] = feval(lik{:},hyp.lik,y,f,[],inf,i); dnlZ.lik(i) = -g'*d2lp_dhyp - sum(lp_dhyp); % explicit part b = K*dlp_dhyp; % b-K*(Z*b) = inv(eye(n)+K*diag(W))*b dnlZ.lik(i) = dnlZ.lik(i) - dfhat'*( b-K*(Z*b) ); % implicit part end for i=1:length(hyp.mean) % mean hypers dm = feval(mean{:}, hyp.mean, x, i); dnlZ.mean(i) = -alpha'*dm; % explicit part dnlZ.mean(i) = dnlZ.mean(i) - dfhat'*(dm-K*(Z*dm)); % implicit part end end % Evaluate criterion Psi(alpha) = alpha'*K*alpha + likfun(f), where % f = K*alpha+m, and likfun(f) = feval(lik{:},hyp.lik,y, f, [],inf). function [psi,dpsi,f,alpha,dlp,W] = Psi(alpha,m,K,likfun) f = K*alpha+m; [lp,dlp,d2lp] = likfun(f); W = -d2lp; psi = alpha'*(f-m)/2 - sum(lp); if nargout>1, dpsi = K*(alpha-dlp); end % Run IRLS Newton algorithm to optimise Psi(alpha). function alpha = irls(alpha, m,K,likfun, opt) if isfield(opt,'irls_maxit'), maxit = opt.irls_maxit; % max no of Newton steps else maxit = 20; end % default value if isfield(opt,'irls_Wmin'), Wmin = opt.irls_Wmin; % min likelihood curvature else Wmin = 0.0; end % default value if isfield(opt,'irls_tol'), tol = opt.irls_tol; % stop Newton iterations else tol = 1e-6; end % default value smin_line = 0; smax_line = 2; % min/max line search steps size range nmax_line = 10; % maximum number of line search steps thr_line = 1e-4; % line search threshold Psi_line = @(s,alpha,dalpha) Psi(alpha+s*dalpha, m,K,likfun); % line search pars_line = {smin_line,smax_line,nmax_line,thr_line}; % line seach parameters search_line = @(alpha,dalpha) brentmin(pars_line{:},Psi_line,5,alpha,dalpha); f = K*alpha+m; [lp,dlp,d2lp] = likfun(f); W = -d2lp; n = size(K,1); Psi_new = Psi(alpha,m,K,likfun); Psi_old = Inf; % make sure while loop starts by the largest old objective val it = 0; % this happens for the Student's t likelihood while Psi_old - Psi_new > tol && it det(A) = det(P)*prod(diag(U)) ldA = sum(log(abs(u))); end if nargout>1, iA = U\(L\P); end % return the inverse if required if nargout>2, mwiA = -repmat(w,1,n).*iA; endgpml-matlab-v3.6-2015-07-07/inf/infFITC_Laplace.m000644 000765 000024 00000026153 12504606112 020667 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infFITC_Laplace(hyp, mean, cov, lik, x, y) % FITC-Laplace approximation to the posterior Gaussian process. The function is % equivalent to infLaplace with the covariance function: % Kt = Q + G; G = diag(g); g = diag(K-Q); Q = Ku'*inv(Kuu + snu2*eye(nu))*Ku; % where Ku and Kuu are covariances w.r.t. to inducing inputs xu and % snu2 = sn2/1e6 is the noise of the inducing inputs. We fixed the standard % deviation of the inducing inputs snu to be a one per mil of the measurement % noise's standard deviation sn. In case of a likelihood without noise % parameter sn2, we simply use snu2 = 1e-6. % % The implementation exploits the Woodbury matrix identity % inv(Kt) = inv(G) - inv(G)*Ku'*inv(Kuu+Ku*inv(G)*Ku')*Ku*inv(G) % in order to be applicable to large datasets. The computational complexity % is O(n nu^2) where n is the number of data points x and nu the number of % inducing inputs in xu. % The posterior N(f|h,Sigma) is given by h = m+mu with mu = nn + P'*gg and % Sigma = inv(inv(K)+diag(W)) = diag(d) + P'*R0'*R'*R*R0*P. % % The function takes a specified covariance function (see covFunctions.m) and % likelihood function (see likFunctions.m), and is designed to be used with % gp.m and in conjunction with covFITC. % % The inducing points can be specified through 1) the 2nd covFITC parameter or % by 2) providing a hyp.xu hyperparameters. Note that 2) has priority over 1). % In case 2) is provided and derivatives dnlZ are requested, there will also be % a dnlZ.xu field allowing to optimise w.r.t. to the inducing points xu. However % the derivatives dnlZ.xu can only be computed for one of the following eight % covariance functions: cov{Matern|PP|RQ|SE}{iso|ard}. % % Copyright (c) by Hannes Nickisch, 2015-02-09. % % See also INFMETHODS.M, COVFITC.M. persistent last_alpha % copy of the last alpha if any(isnan(last_alpha)), last_alpha = zeros(size(last_alpha)); end % prevent tol = 1e-6; % tolerance for when to stop the Newton iterations smax = 2; Nline = 10; thr = 1e-4; % line search parameters maxit = 20; % max number of Newton steps in f inf = 'infLaplace'; cov1 = cov{1}; if isa(cov1, 'function_handle'), cov1 = func2str(cov1); end if ~strcmp(cov1,'covFITC'); error('Only covFITC supported.'), end % check cov if isfield(hyp,'xu'), cov{3} = hyp.xu; end % hyp.xu is provided, replace cov{3} [diagK,Kuu,Ku] = feval(cov{:}, hyp.cov, x); % evaluate covariance matrix if ~isempty(hyp.lik) % hard coded inducing inputs noise sn2 = exp(2*hyp.lik(end)); snu2 = 1e-6*sn2; % similar to infFITC else snu2 = 1e-6; end [n, D] = size(x); nu = size(Kuu,1); m = feval(mean{:}, hyp.mean, x); % evaluate the mean vector rot180 = @(A) rot90(rot90(A)); % little helper functions chol_inv = @(A) rot180(chol(rot180(A))')\eye(nu); % chol(inv(A)) R0 = chol_inv(Kuu+snu2*eye(nu)); % initial R, used for refresh O(nu^3) V = R0*Ku; d0 = diagK-sum(V.*V,1)'; % initial d, needed for refresh O(n*nu^2) Psi_old = Inf; % make sure while loop starts by the largest old objective val if any(size(last_alpha)~=[n,1]) % find a good starting point for alpha and f alpha = zeros(n,1); f = mvmK(alpha,V,d0)+m; % start at mean if sizes not match [lp,dlp,d2lp] = feval(lik{:},hyp.lik,y,f,[],inf); W=-d2lp; Psi_new = -sum(lp); else alpha = last_alpha; f = mvmK(alpha,V,d0)+m; % try last one [lp,dlp,d2lp] = feval(lik{:},hyp.lik,y,f,[],inf); W=-d2lp; Psi_new = alpha'*(f-m)/2 - sum(lp); % objective for last alpha Psi_def = -feval(lik{:},hyp.lik,y,m,[],inf); % objective for default init f==m if Psi_def < Psi_new % if default is better, we use it alpha = zeros(n,1); f = mvmK(alpha,V,d0)+m; [lp,dlp,d2lp] = feval(lik{:},hyp.lik,y,f,[],inf); W=-d2lp; Psi_new=-sum(lp); end end isWneg = any(W<0); % flag indicating whether we found negative values of W it = 0; % this happens for the Student's t likelihood while Psi_old - Psi_new > tol && it2 % do we want derivatives? dnlZ = hyp; % allocate space for derivatives [d,P,R] = fitcRefresh(d0,Ku,R0,V,W); % g = diag(inv(inv(K)+W))/2 g = d/2 + sum(((R*R0)*P).^2,1)'/2; t = W./(1+W.*d0); dfhat = g.*d3lp; % deriv. of nlZ wrt. fhat: dfhat=diag(inv(inv(K)+W)).*d3lp/2 for i=1:length(hyp.cov) % covariance hypers [ddiagK,dKuu,dKu] = feval(cov{:}, hyp.cov, x, [], i); % eval cov derivatives dA = 2*dKu'-R0tV'*dKuu; % dQ = dA*R0tV w = sum(dA.*R0tV',2); v = ddiagK-w; % w = diag(dQ); v = diag(dK)-diag(dQ); dnlZ.cov(i) = ddiagK'*t - sum(RVdd.*RVdd,1)*v; % explicit part dnlZ.cov(i) = dnlZ.cov(i)-sum(sum((RVdd*dA).*(RVdd*R0tV'))); % explicit part dnlZ.cov(i) = dnlZ.cov(i)/2-alpha'*(dA*(R0tV*alpha)+v.*alpha)/2; % explicit b = dA*(R0tV*dlp) + v.*dlp; % b-K*(Z*b) = inv(eye(n)+K*diag(W))*b KZb = mvmK(mvmZ(b,RVdd,t),V,d0); dnlZ.cov(i) = dnlZ.cov(i) - dfhat'*( b-KZb ); % implicit part end for i=1:length(hyp.lik) % likelihood hypers [lp_dhyp,dlp_dhyp,d2lp_dhyp] = feval(lik{:},hyp.lik,y,f,[],inf,i); dnlZ.lik(i) = -g'*d2lp_dhyp - sum(lp_dhyp); % explicit part b = mvmK(dlp_dhyp,V,d0); % implicit part dnlZ.lik(i) = dnlZ.lik(i) - dfhat'*(b-mvmK(mvmZ(b,RVdd,t),V,d0)); if i==numel(hyp.lik) % since snu2 is a fixed fraction of sn2, there is a covariance-like term % in the derivative as well snu = sqrt(snu2); T = chol_inv(Kuu + snu2*eye(nu)); T = T'*(T*(snu*Ku)); st2 = sum(T.*T,1)'; z = alpha'*(T'*(T*alpha)-st2.*alpha) - sum(RVdd.*RVdd,1)*st2; z = z + sum(sum( (RVdd*T').^2 )); b = (st2.*dlp-T'*(T*dlp))*2; KZb = mvmK(mvmZ(b,RVdd,t),V,d0); z = z - dfhat'*( b-KZb ); dnlZ.lik(i) = dnlZ.lik(i) + z; end end for i=1:length(hyp.mean) % mean hypers dm = feval(mean{:}, hyp.mean, x, i); dnlZ.mean(i) = -alpha'*dm; % explicit part Zdm = mvmZ(dm,RVdd,t); dnlZ.mean(i) = dnlZ.mean(i) - dfhat'*(dm-mvmK(Zdm,V,d0)); % implicit part end if isfield(hyp,'xu') % derivatives w.r.t. inducing points xu xu = cov{3}; cov = cov{2}; % get the non FITC part of the covariance function Kpu = cov_deriv_sq_dist(cov,hyp.cov,xu,x); % d K(xu,x ) / d D^2 Kpuu = cov_deriv_sq_dist(cov,hyp.cov,xu); % d K(xu,xu) / d D^2 if iscell(cov), covstr = cov{1}; else covstr = cov; end if ~ischar(covstr), covstr = func2str(covstr); end if numel(strfind(covstr,'iso'))>0 % characteristic length scale e = 2*exp(-2*hyp.cov(1)); else e = 2*exp(-2*hyp.cov(1:D)); end q = dfhat - mvmZ(mvmK(dfhat,V,d0),RVdd,t); % implicit part B = (R0'*R0)*Ku; diag_dK = alpha.*alpha + sum(RVdd.*RVdd,1)' - t + 2*dlp.*q; v = diag_dK+t; % BdK = B * ( dnlZ/dK - diag(diag(dnlZ/dK)) ) BdK = (B*alpha)*alpha' - B.*repmat(v',nu,1) + (B*dlp)*q' + (B*q)*dlp'; BdK = BdK + (B*RVdd')*RVdd; A = Kpu.*BdK; C = Kpuu.*(BdK*B'); C = diag(sum(C,2)-sum(A,2)) - C; dnlZ.xu = A*x*diag(e) + C*xu*diag(e); % bring in data and inducing points end end % matrix vector multiplication with Z=inv(K+inv(W)) function Zx = mvmZ(x,RVdd,t) Zx = t.*x - RVdd'*(RVdd*x); % matrix vector multiplication with approximate covariance matrix function Kal = mvmK(al,V,d0) Kal = V'*(V*al) + d0.*al; % criterion Psi at alpha + s*dalpha for line search function [Psi,alpha,f,dlp,W] = Psi_line(s,dalpha,alpha,hyp,V,d0,m,lik,y,inf) alpha = alpha + s*dalpha; f = mvmK(alpha,V,d0)+m; [lp,dlp,d2lp] = feval(lik{:},hyp.lik,y,f,[],inf); W = -d2lp; Psi = alpha'*(f-m)/2 - sum(lp); % refresh the representation of the posterior from initial and site parameters % to prevent possible loss of numerical precision after many epfitcUpdates % effort is O(n*nu^2) provided that nu1 % do we want the marginal likelihood? nlZ = (y-m)'*alpha/2 + sum(log(diag(L))) + n*log(2*pi*sl)/2; % -log marg lik if nargout>2 % do we want derivatives? dnlZ = hyp; % allocate space for derivatives Q = solve_chol(L,eye(n))/sl - alpha*alpha'; % precompute for convenience for i = 1:numel(hyp.cov) dnlZ.cov(i) = sum(sum(Q.*feval(cov{:}, hyp.cov, x, [], i)))/2; end dnlZ.lik = sn2*trace(Q); for i = 1:numel(hyp.mean) dnlZ.mean(i) = -feval(mean{:}, hyp.mean, x, i)'*alpha; end end end gpml-matlab-v3.6-2015-07-07/inf/infGrid.m000644 000765 000024 00000017155 12504606112 017410 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infGrid(hyp, mean, cov, lik, x, y, opt) % Inference for a GP with Gaussian likelihood and covGrid covariance. % The (Kronecker) covariance matrix used is given by: % K = kron( kron(...,K{2}), K{1} ) = K_p x .. x K_2 x K_1. % % Compute a parametrization of the posterior, the negative log marginal % likelihood and its derivatives w.r.t. the hyperparameters. % The result is exact for complete grids, otherwise results are approximate. % See also "help infMethods". % % The function takes a specified covariance function (see covFunctions.m) and % likelihood function (see likFunctions.m), and is designed to be used with % gp.m and in conjunction with covFITC and likGauss. % % Copyright (c) by Hannes Nickisch and Andrew Wilson, 2014-12-03. % % See also INFMETHODS.M, COVGRID.M. if iscell(lik), likstr = lik{1}; else likstr = lik; end if ~ischar(likstr), likstr = func2str(likstr); end if ~strcmp(likstr,'likGauss') % NOTE: no explicit call to likGauss error('Inference with infGrid only possible with Gaussian likelihood.'); end cov1 = cov{1}; if isa(cov1, 'function_handle'), cov1 = func2str(cov1); end if ~strcmp(cov1,'covGrid'); error('Only covGrid supported.'), end % check cov xg = cov{3}; p = numel(xg); % underlying grid N = 1; D = 0; for i=1:p, N = N*size(xg{i},1); D = D+size(xg{i},2); end % dims [K,M,xe] = feval(cov{:}, hyp.cov, x); % evaluate covariance mat constituents xe = M*xe; n = size(xe,1); m = feval(mean{:}, hyp.mean, xe); % evaluate mean vector if nargin<=6, opt = []; end % make opt variable available if isfield(opt,'cg_tol'), cgtol = opt.cg_tol; % stop conjugate gradients else cgtol = 1e-5; end if isfield(opt,'cg_maxit'), cgmit = opt.cg_maxit; % number of cg iterations else cgmit = min(n,20); end for j=1:numel(K) if iscell(K{j}) && strcmp(K{j}{1},'toep'), K{j} = toeplitz(K{j}{2}); end end sn2 = exp(2*hyp.lik); % noise variance of likGauss V = cell(size(K)); E = cell(size(K)); % eigenvalue decomposition for j=1:numel(K) if iscell(K{j}) && strcmp(K{j}{1},'toep') [V{j},E{j}] = eigr(toeplitz(K{j}{2})); else [V{j},E{j}] = eigr(K{j}); end end e = 1; for i=1:numel(E), e = kron(diag(E{i}),e); end % eigenvalue diagonal if numel(m)==N s = 1./(e+sn2); ord = 1:N; % V*diag(s)*V' = inv(K+sn2*eye(N)) else [eord,ord] = sort(e,'descend'); % sort long vector of eigenvalues if n>N, eord = [eord;zeros(n-N,1)]; ord = [ord;(N+1:n)']; end % special case s = 1./((n/N)*eord(1:n) + sn2); % approx using top n eigenvalues end if numel(m)==N % decide for Kronecker magic or conjugate gradients L = @(k) kronmvm(V,repmat(-s,1,size(k,2)).*kronmvm(V,k,1)); % mvm callback else % go for conjugate gradients mvm = @(t) mvmK(t,K,sn2,M); % single parameter mvm L = @(k) -solveMVM(k,mvm,cgtol,cgmit); end alpha = -L(y-m); post.alpha = alpha; % return the posterior parameters post.sW = ones(n,1)/sqrt(sn2); % sqrt of noise precision post.L = L; % function to compute inv(K+sn2*eye(N))*Ks if nargout>1 % do we want the marginal likelihood? lda = -sum(log(s)); % exact kron % exact: lda = 2*sum(log(diag(chol(M*kronmvm(K,eye(N))*M'+sn2*eye(n))))); nlZ = (y-m)'*alpha/2 + n*log(2*pi)/2 + lda/2; if nargout>2 % do we want derivatives? dnlZ = hyp; % allocate space for derivatives, define Q=inv(M*K*M'+sn2*I) Mtal = M'*alpha; % blow up alpha vector from n to N for i = 1:numel(hyp.cov) dK = feval(cov{:}, hyp.cov, x, [], i); for j=1:numel(dK) % expand Toeplitz if iscell(dK{j}) && strcmp(dK{j}{1},'toep') dK{j} = toeplitz(dK{j}{2}); end end P = cell(size(dK)); % auxiliary Kronecker matrix P = V'*dK*V for j = 1:numel(dK), P{j} = sum((V{j}'*dK{j}).*V{j}',2); end p = 1; for j=1:numel(P), p = kron(P{j},p); end % p = diag(P) if n>N, p = [p;zeros(n-N,1)]; end % special case p = (n/N)*p(ord(1:n)); % approximate if incomplete grid observation dnlZ.cov(i) = (p'*s - Mtal'*kronmvm(dK,Mtal))/2; end dnlZ.lik = sn2*(sum(s) - alpha'*alpha); % sum(s) = trace(Q) for i = 1:numel(hyp.mean) dnlZ.mean(i) = -feval(mean{:}, hyp.mean, xe, i)'*alpha; end end end % Solve x=A*b with symmetric A(n,n), b(n,m), x(n,m) using conjugate gradients. % The method is along the lines of PCG but suited for matrix inputs b. function [x,flag,relres,iter,r] = conjgrad(A,b,tol,maxit) if nargin<3, tol = 1e-10; end if nargin<4, maxit = min(size(b,1),20); end x0 = zeros(size(b)); x = x0; if isnumeric(A), r = b-A*x; else r = b-A(x); end, r2 = sum(r.*r,1); r2new = r2; nb = sqrt(sum(b.*b,1)); flag = 0; iter = 1; relres = sqrt(r2)./nb; todo = relres>=tol; if ~any(todo), flag = 1; return, end on = ones(size(b,1),1); r = r(:,todo); d = r; for iter = 2:maxit if isnumeric(A), z = A*d; else z = A(d); end a = r2(todo)./sum(d.*z,1); a = on*a; x(:,todo) = x(:,todo) + a.*d; r = r - a.*z; r2new(todo) = sum(r.*r,1); relres = sqrt(r2new)./nb; cnv = relres(todo)=tol; d = d(:,~cnv); r = r(:,~cnv); % get rid of converged if ~any(todo), flag = 1; return, end b = r2new./r2; % Fletcher-Reeves d = r + (on*b(todo)).*d; r2 = r2new; end % solve q = mvm(p) via conjugate gradients function q = solveMVM(p,mvm,varargin) [q,flag,relres,iter] = conjgrad(mvm,p,varargin{:}); % like pcg if ~flag,error('Not converged after %d iterations, r=%1.2e\n',iter,relres),end % mvm so that q = M*K*M'*p + sn2*p using the Kronecker representation function q = mvmK(p,K,sn2,M) q = M*kronmvm(K,M'*p) + sn2*p; % Perform a matrix vector multiplication b = A*x with a matrix A being a % Kronecker product given by A = kron( kron(...,As{2}), As{1} ). function b = kronmvm(As,x,transp) if nargin>2 && ~isempty(transp) && transp % transposition by transposing parts for i=1:numel(As), As{i} = As{i}'; end end m = zeros(numel(As),1); n = zeros(numel(As),1); % extract sizes for i=1:numel(n) if iscell(As{i}) && strcmp(As{i}{1},'toep') m(i) = size(As{i}{2},1); n(i) = size(As{i}{2},1); else [m(i),n(i)] = size(As{i}); end end d = size(x,2); b = x; for i=1:numel(n) a = reshape(b,[prod(m(1:i-1)), n(i), prod(n(i+1:end))*d]); % prepare input tmp = reshape(permute(a,[1,3,2]),[],n(i))*As{i}'; b = permute(reshape(tmp,[size(a,1),size(a,3),m(i)]),[1,3,2]); end b = reshape(b,prod(m),d); % bring result in correct shape % Real eigenvalues and eigenvectors up to the rank of a real symmetric matrix. % Decompose A into V*D*V' with orthonormal matrix V and diagonal matrix D. % Diagonal entries of D obave the rank r of the matrix A as returned by % the call rank(A,tol) are zero. function [V,D] = eigr(A,tol) [V,D] = eig((A+A')/2); n = size(A,1); % decomposition of strictly symmetric A d = max(real(diag(D)),0); [d,ord] = sort(d,'descend'); % tidy up and sort if nargin<2, tol = size(A,1)*eps(max(d)); end, r = sum(d>tol); % get rank(A) d(r+1:n) = 0; D = diag(d); % set junk eigenvalues to strict zeros V(:,1:r) = real(V(:,ord(1:r))); V(:,r+1:n) = null(V(:,1:r)'); % ortho completion gpml-matlab-v3.6-2015-07-07/inf/infFITC_EP.m000644 000765 000024 00000027525 12441263152 017641 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infFITC_EP(hyp, mean, cov, lik, x, y) % FITC-EP approximation to the posterior Gaussian process. The function is % equivalent to infEP with the covariance function: % Kt = Q + G; G = diag(g); g = diag(K-Q); Q = Ku'*inv(Kuu + snu2*eye(nu))*Ku; % where Ku and Kuu are covariances w.r.t. to inducing inputs xu and % snu2 = sn2/1e6 is the noise of the inducing inputs. We fixed the standard % deviation of the inducing inputs snu to be a one per mil of the measurement % noise's standard deviation sn. In case of a likelihood without noise % parameter sn2, we simply use snu2 = 1e-6. % For details, see The Generalized FITC Approximation, Andrew Naish-Guzman and % Sean Holden, NIPS, 2007. % % The implementation exploits the Woodbury matrix identity % inv(Kt) = inv(G) - inv(G)*Ku'*inv(Kuu+Ku*inv(G)*Ku')*Ku*inv(G) % in order to be applicable to large datasets. The computational complexity % is O(n nu^2) where n is the number of data points x and nu the number of % inducing inputs in xu. % The posterior N(f|h,Sigma) is given by h = m+mu with mu = nn + P'*gg and % Sigma = inv(inv(K)+diag(W)) = diag(d) + P'*R0'*R'*R*R0*P. Here, we use the % site parameters: b,w=$b,\pi$=tnu,ttau, P=$P'$, nn=$\nu$, gg=$\gamma$ % % The function takes a specified covariance function (see covFunctions.m) and % likelihood function (see likFunctions.m), and is designed to be used with % gp.m and in conjunction with covFITC. % % The inducing points can be specified through 1) the 2nd covFITC parameter or % by 2) providing a hyp.xu hyperparameters. Note that 2) has priority over 1). % In case 2) is provided and derivatives dnlZ are requested, there will also be % a dnlZ.xu field allowing to optimise w.r.t. to the inducing points xu. However % the derivatives dnlZ.xu can only be computed for one of the following eight % covariance functions: cov{Matern|PP|RQ|SE}{iso|ard}. % % Copyright (c) by Hannes Nickisch, 2013-10-29. % % See also INFMETHODS.M, COVFITC.M. persistent last_ttau last_tnu % keep tilde parameters between calls tol = 1e-4; max_sweep = 20; min_sweep = 2; % tolerance to stop EP iterations inf = 'infEP'; cov1 = cov{1}; if isa(cov1, 'function_handle'), cov1 = func2str(cov1); end if ~strcmp(cov1,'covFITC'); error('Only covFITC supported.'), end % check cov if isfield(hyp,'xu'), cov{3} = hyp.xu; end % hyp.xu is provided, replace cov{3} [diagK,Kuu,Ku] = feval(cov{:}, hyp.cov, x); % evaluate covariance matrix if ~isempty(hyp.lik) % hard coded inducing inputs noise sn2 = exp(2*hyp.lik(end)); snu2 = 1e-6*sn2; % similar to infFITC else snu2 = 1e-6; end [n, D] = size(x); nu = size(Kuu,1); m = feval(mean{:}, hyp.mean, x); % evaluate the mean vector rot180 = @(A) rot90(rot90(A)); % little helper functions chol_inv = @(A) rot180(chol(rot180(A))')\eye(nu); % chol(inv(A)) R0 = chol_inv(Kuu+snu2*eye(nu)); % initial R, used for refresh O(nu^3) V = R0*Ku; d0 = diagK-sum(V.*V,1)'; % initial d, needed for refresh O(n*nu^2) % A note on naming: variables are given short but descriptive names in % accordance with Rasmussen & Williams "GPs for Machine Learning" (2006): mu % and s2 are mean and variance, nu and tau are natural parameters. A leading t % means tilde, a subscript _ni means "not i" (for cavity parameters), or _n % for a vector of cavity parameters. % marginal likelihood for ttau = tnu = zeros(n,1); equals n*log(2) for likCum* nlZ0 = -sum(feval(lik{:}, hyp.lik, y, m, diagK, inf)); if any(size(last_ttau) ~= [n 1]) % find starting point for tilde parameters ttau = zeros(n,1); % initialize to zero if we have no better guess tnu = zeros(n,1); [d,P,R,nn,gg] = epfitcRefresh(d0,Ku,R0,V, ttau,tnu); % compute initial repres. nlZ = nlZ0; else ttau = last_ttau; % try the tilde values from previous call tnu = last_tnu; [d,P,R,nn,gg] = epfitcRefresh(d0,Ku,R0,V, ttau,tnu); % compute initial repres. nlZ = epfitcZ(d,P,R,nn,gg,ttau,tnu,d0,R0,Ku,y,lik,hyp,m,inf); if nlZ > nlZ0 % if zero is better .. ttau = zeros(n,1); % .. then initialize with zero instead tnu = zeros(n,1); [d,P,R,nn,gg] = epfitcRefresh(d0,Ku,R0,V, ttau,tnu); % initial repres. nlZ = nlZ0; end end nlZ_old = Inf; sweep = 0; % converged, max. sweeps or min. sweeps? while (abs(nlZ-nlZ_old) > tol && sweep < max_sweep) || sweep2 % do we want derivatives? dnlZ = hyp; % allocate space for derivatives RVdd = RV.*repmat(dd',nu,1); for i=1:length(hyp.cov) [ddiagK,dKuu,dKu] = feval(cov{:}, hyp.cov, x, [], i); % eval cov derivatives dA = 2*dKu'-R0tV'*dKuu; % dQ = dA*R0tV w = sum(dA.*R0tV',2); v = ddiagK-w; % w = diag(dQ); v = diag(dK)-diag(dQ); z = dd'*(v+w) - sum(RVdd.*RVdd,1)*v - sum(sum( (RVdd*dA)'.*(R0tV*RVdd') )); dnlZ.cov(i) = (z - alpha'*(alpha.*v) - (alpha'*dA)*(R0tV*alpha))/2; end for i = 1:numel(hyp.lik) % likelihood hypers dlik = feval(lik{:}, hyp.lik, y, nu_n./tau_n, 1./tau_n, inf, i); dnlZ.lik(i) = -sum(dlik); if i==numel(hyp.lik) % since snu2 is a fixed fraction of sn2, there is a covariance-like term % in the derivative as well v = sum(R0tV.*R0tV,1)'; z = sum(sum( (RVdd*R0tV').^2 )) - sum(RVdd.*RVdd,1)*v; z = z + post.alpha'*post.alpha - alpha'*(v.*alpha); dnlZ.lik(i) = dnlZ.lik(i) + snu2*z; end end [junk,dlZ] = feval(lik{:}, hyp.lik, y, nu_n./tau_n, 1./tau_n, inf);% mean hyps for i = 1:numel(hyp.mean) dm = feval(mean{:}, hyp.mean, x, i); dnlZ.mean(i) = -dlZ'*dm; end if isfield(hyp,'xu') % derivatives w.r.t. inducing points xu xu = cov{3}; cov = cov{2}; % get the non FITC part of the covariance function Kpu = cov_deriv_sq_dist(cov,hyp.cov,xu,x); % d K(xu,x ) / d D^2 Kpuu = cov_deriv_sq_dist(cov,hyp.cov,xu); % d K(xu,xu) / d D^2 if iscell(cov), covstr = cov{1}; else covstr = cov; end if ~ischar(covstr), covstr = func2str(covstr); end if numel(strfind(covstr,'iso'))>0 % characteristic length scale e = 2*exp(-2*hyp.cov(1)); else e = 2*exp(-2*hyp.cov(1:D)); end B = (R0'*R0)*Ku; W = ttau; t = W./(1+W.*d0); diag_dK = alpha.*alpha + sum(RVdd.*RVdd,1)' - t; v = diag_dK+t; % BdK = B * ( dnlZ/dK - diag(diag(dnlZ/dK)) ) BdK = (B*alpha)*alpha' - B.*repmat(v',nu,1); BdK = BdK + (B*RVdd')*RVdd; A = Kpu.*BdK; C = Kpuu.*(BdK*B'); C = diag(sum(C,2)-sum(A,2)) - C; dnlZ.xu = A*x*diag(e) + C*xu*diag(e); % bring in data and inducing points end end % refresh the representation of the posterior from initial and site parameters % to prevent possible loss of numerical precision after many epfitcUpdates % effort is O(n*nu^2) provided that nu0 R = cholupdate(R,sqrt( r)*R'*v,'-'); else R = cholupdate(R,sqrt(-r)*R'*v,'+'); end gg = gg + ((dbi-dwi*(hi-m(i)))/t)*(R0'*(R'*(R*(R0*Pi)))); % O(nu^2) w(i) = wi; b(i) = bi; % update site parameters O(1) Pi = Pi/t; % O(nu) gpml-matlab-v3.6-2015-07-07/inf/infGrid_Laplace.m000644 000765 000024 00000032473 12504606112 021031 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infGrid_Laplace(hyp, mean, cov, lik, x, y, opt) % Laplace approximation to the posterior Gaussian process with covGrid % covariance and (possibly) non-Gaussian likelihood. % The (Kronecker) covariance matrix used is given by: % K = kron( kron(...,K{2}), K{1} ) = K_p x .. x K_2 x K_1. % % The function takes a specified covariance function (see covFunctions.m) and % likelihood function (see likFunctions.m), and is designed to be used with % gp.m and in conjunction with covGrid. See also infMethods.m. % % Copyright (c) by Hannes Nickisch 2015-03-05. % % See also INFMETHODS.M, COVGRID.M. persistent last_alpha % copy of the last alpha if any(isnan(last_alpha)), last_alpha = zeros(size(last_alpha)); end % prevent inf = 'infLaplace'; cov1 = cov{1}; if isa(cov1, 'function_handle'), cov1 = func2str(cov1); end if ~strcmp(cov1,'covGrid'); error('Only covGrid supported.'), end % check cov xg = cov{3}; p = numel(xg); % underlying grid N = 1; D = 0; for i=1:p, N = N*size(xg{i},1); D = D+size(xg{i},2); end % dims [K,M,xe] = feval(cov{:}, hyp.cov, x); % evaluate covariance mat constituents for j=1:numel(K) % expand Toeplitz if iscell(K{j}) && strcmp(K{j}{1},'toep'), K{j} = toeplitz(K{j}{2}); end end xe = M*xe; n = size(xe,1); m = feval(mean{:}, hyp.mean, xe); % evaluate mean vector likfun = @(f) feval(lik{:},hyp.lik,y,f,[],inf); % log likelihood function if nargin<=6, opt = []; end % make opt variable available if isfield(opt,'cg_tol'), cgtol = opt.cg_tol; % stop conjugate gradients else cgtol = 1e-5; end if isfield(opt,'cg_maxit'), cgmit = opt.cg_maxit; % number of cg iterations else cgmit = min(n,20); end if isfield(opt,'nlZ_exact'), exact = opt.nlZ_exact; % use dense matrix .. else exact = 0; end % .. computations instead of LCG if any(size(last_alpha)~=[n,1]) % find a good starting point for alpha and f alpha = zeros(n,1); % start at mean if sizes do not match else alpha = last_alpha; % try last one if Psi(alpha,m,K,M,likfun) > -sum(likfun(m)) % default f==m better => use it alpha = zeros(n,1); end end % switch between optimisation methods alpha = irls(alpha, m,K,M,likfun, opt); % run optimisation alpha = irls(alpha, m,K,M,likfun, opt); % run again f = M*kronmvm(K,M'*alpha) + m; % compute latent function values last_alpha = alpha; % remember for next call [lp,dlp,d2lp,d3lp] = likfun(f); W = -d2lp; % evaluate likelihood post.alpha = alpha; % return the posterior parameters sW = sqrt(max(W,0)); post.sW = sW; % remove sign in case of negative mvm = @(t) mvmsWKsW(t,K,sW,M); % symmetric single parameter mvm rep = @(sW,k) repmat(sW,1,size(k,2)); post.L = @(k) -rep(sW,k).*solveMVM(rep(sW,k).*k,mvm,cgtol,cgmit); % diagnose optimality err = @(x,y) norm(x-y)/max([norm(x),norm(y),1]); % we need to have alpha = dlp % dev = err(alpha,dlp); if dev>1e-4, warning('Not at optimum %1.2e.',dev), end if nargout>1 if exact % exact marginal likelihood computation Ki = kronmvm(K,eye(N)); Ki = M*Ki*M'; Li = chol(eye(n)+sW*sW'.*Ki); nlZ = alpha'*(f-m)/2 - sum(lp) + sum(log(diag(Li))); else % Fiedler's 1971 upper log determinant bound on A = I+K*W a = logdet_I_KW(K,W); nlZ = alpha'*(f-m)/2 - sum(lp) + a/2; % upper bound on nlZ end end if nargout>2 % do we want derivatives? dnlZ = hyp; % allocate space for derivatives mvm = @(t) mvmsWKsW(t,K,sW,M); % single parameter mvm mvmZ = @(k) sW.*solveMVM(k.*sW,mvm,cgtol,cgmit); % Z = inv(K+inv(W)) if exact % deriv. of ln|B| wrt W; g = diag(inv(inv(K)+diag(W)))/2 A = Ki*diag(W)+eye(numel(m)); Ci = Li'\(repmat(sW,1,numel(m)).*Ki); g = (diag(Ki)-sum(Ci.^2,1)')/2; dfhat = g.*d3lp; % d nlZ / d fhat: diag(inv(inv(K)+W)).*d3lp/2 else g = zeros(n,1); end % no implicit derivative if ~exact for i=1:length(hyp.cov) % covariance hypers dK = feval(cov{:}, hyp.cov, x, [], i); for j=1:numel(dK) % expand Toeplitz if iscell(dK{j})&&strcmp(dK{j}{1},'toep'), dK{j} = toeplitz(dK{j}{2}); end end if exact dKi = kronmvm(dK,eye(N)); dKi = M*dKi*M'; dla = trace( A'\(diag(W)*dKi) ); else h = 1e-4; % finite difference approximation hyp_cov_h = hyp.cov; hyp_cov_h(i) = hyp_cov_h(i)+h; K_h = feval(cov{:}, hyp_cov_h, x); for j=1:numel(K_h) % expand Toeplitz if iscell(K_h{j})&&strcmp(K_h{j}{1},'toep') K_h{j} = toeplitz(K_h{j}{2}); end end dla = (logdet_I_KW(K_h,W)-a)/h; % finite difference approximation end dnlZ.cov(i) = dla/2 - alpha'*M*kronmvm(dK,M'*alpha)/2; % explicit part if exact b = M*kronmvm(dK,M'*dlp); % inv(eye(n)+K*diag(W))*b dnlZ.cov(i) = dnlZ.cov(i)-dfhat'*(b-M*kronmvm(K,M'*mvmZ(b))); % implicit end end for i=1:length(hyp.lik) % likelihood hypers [lp_dhyp,dlp_dhyp,d2lp_dhyp] = feval(lik{:},hyp.lik,y,f,[],inf,i); dnlZ.lik(i) = -g'*d2lp_dhyp - sum(lp_dhyp); % explicit part if exact b = M*kronmvm(K,M'*(dlp_dhyp)); % inv(eye(n)+K*diag(W))*b dnlZ.lik(i) = dnlZ.lik(i)-dfhat'*(b-M*kronmvm(K,M'*mvmZ(b))); % implicit end end for i=1:length(hyp.mean) % mean hypers dm = feval(mean{:}, hyp.mean, xe, i); dnlZ.mean(i) = -alpha'*dm; % explicit part if exact dnlZ.mean(i) = dnlZ.mean(i)-dfhat'*(dm-M*kronmvm(K,M'*mvmZ(dm))); % implic end end end % Upper determinant bound on log |I+K*diag(W)| using Fiedler's 1971 inequality. % K = kron( kron(...,K{2}), K{1} ), W = diag(w) both symmetric psd. % The bound is exact for W = w*ones(N,1). % % Given nxn spd matrices C and D with ordered nx1 eigenvalues c, d % then exp(lb)=prod(c+d) <= det(C+D) <= prod(c+flipud(d))=exp(ub). function [ub,lb] = logdet_I_KW(K,W) e = 1; for i=1:numel(K), e = kron(eigr(K{i}),e); end % eigenvalue diagonal n = numel(W); N = numel(e); % dimensions a = n/N*sort(e,'descend'); % approximate spectrum of K if n>N, a = [a;zeros(n-N,1)]; else a = a(1:n); end ub = sum(log( 1 + a.*sort(W,'descend') )); % See also Prob.III.6.14 in .. lb = sum(log( 1 + a.*sort(W, 'ascend') )); % .. Matrix Analysis, Bhatia 1997. % Evaluate criterion Psi(alpha) = alpha'*K*alpha + likfun(f), where % f = K*alpha+m, and likfun(f) = feval(lik{:},hyp.lik,y, f, [],inf). function [psi,dpsi,f,alpha,dlp,W] = Psi(alpha,m,K,M,likfun) f = M*kronmvm(K,M'*alpha) + m; [lp,dlp,d2lp] = likfun(f); W = -d2lp; psi = alpha'*(f-m)/2 - sum(lp); if nargout>1, dpsi = M*kronmvm(K,M'*(alpha-dlp)); end % Run IRLS Newton algorithm to optimise Psi(alpha). function alpha = irls(alpha, m,K,M,likfun, opt) n = numel(alpha); if isfield(opt,'irls_maxit'), maxit = opt.irls_maxit; % max no of Newton steps else maxit = 20; end % default value if isfield(opt,'irls_Wmin'), Wmin = opt.irls_Wmin; % min likelihood curvature else Wmin = 0.0; end % default value if isfield(opt,'irls_tol'), tol = opt.irls_tol; % stop Newton iterations else tol = 1e-6; end % default value if isfield(opt,'cg_tol'), cgtol = opt.cg_tol; % stop conjugate gradients else cgtol = 1e-5; end if isfield(opt,'cg_maxit'), cgmit = opt.cg_maxit; % number of cg iterations else cgmit = 2*n; end smin_line = 0; smax_line = 2; % min/max line search steps size range nmax_line = 10; % maximum number of line search steps thr_line = 1e-4; % line search threshold Psi_line = @(s,alpha,dalpha) Psi(alpha+s*dalpha, m,K,M,likfun); % line search pars_line = {smin_line,smax_line,nmax_line,thr_line}; % line seach parameters search_line = @(alpha,dalpha) brentmin(pars_line{:},Psi_line,5,alpha,dalpha); f = M*kronmvm(K,M'*alpha) + m; [lp,dlp,d2lp] = likfun(f); W = -d2lp; Psi_new = Psi(alpha,m,K,M,likfun); Psi_old = Inf; % make sure while loop starts by the largest old objective val it = 0; % this happens for the Student's t likelihood while Psi_old - Psi_new > tol && ittol, error('Not converged, r=%1.2e\n',maxres),end % symmetric mvm so that q = diag(sW)*M*K*M'*diag(sW)*p + p % using Kronecker representation function q = mvmsWKsW(p,K,sW,M) q = repmat(sW,1,size(p,2)).*p; q = M*kronmvm(K,M'*q); q = repmat(sW,1,size(p,2)).*q + p; %------------------------------------------------------------------------------% % Solve x=A*b with symmetric A(n,n), b(n,m), x(n,m) using conjugate gradients. % The method is along the lines of PCG but suited for matrix inputs b. function [x,flag,relres,iter,r] = conjgrad(A,b,tol,maxit) if nargin<3, tol = 1e-10; end if nargin<4, maxit = min(size(b,1),20); end x0 = zeros(size(b)); x = x0; if isnumeric(A), r = b-A*x; else r = b-A(x); end, r2 = sum(r.*r,1); r2new = r2; nb = sqrt(sum(b.*b,1)); flag = 0; iter = 1; relres = sqrt(r2)./nb; todo = relres>=tol; if ~any(todo), flag = 1; return, end on = ones(size(b,1),1); r = r(:,todo); d = r; for iter = 2:maxit if isnumeric(A), z = A*d; else z = A(d); end a = r2(todo)./sum(d.*z,1); a = on*a; x(:,todo) = x(:,todo) + a.*d; r = r - a.*z; r2new(todo) = sum(r.*r,1); relres = sqrt(r2new)./nb; cnv = relres(todo)=tol; d = d(:,~cnv); r = r(:,~cnv); % get rid of converged if ~any(todo), flag = 1; return, end b = r2new./r2; % Fletcher-Reeves d = r + (on*b(todo)).*d; r2 = r2new; end % Perform a matrix vector multiplication b = A*x with a matrix A being a % Kronecker product given by A = kron( kron(...,As{2}), As{1} ). function b = kronmvm(As,x,transp) if nargin>2 && ~isempty(transp) && transp % transposition by transposing parts for i=1:numel(As), As{i} = As{i}'; end end m = zeros(numel(As),1); n = zeros(numel(As),1); % extract sizes for i=1:numel(n) if iscell(As{i}) && strcmp(As{i}{1},'toep') m(i) = size(As{i}{2},1); n(i) = size(As{i}{2},1); else [m(i),n(i)] = size(As{i}); end end d = size(x,2); b = x; for i=1:numel(n) a = reshape(b,[prod(m(1:i-1)), n(i), prod(n(i+1:end))*d]); % prepare input if iscell(As{i}) && strcmp(As{i}{1},'toep') % apply along one dimension b = zeros([prod(m(1:i-1)), m(i), prod(n(i+1:end))*d]); % prepare output for j=1:size(b,3), b(:,:,j) = toepmvmsym(As{i}{2},a(:,:,j)',true)'; end else tmp = reshape(permute(a,[1,3,2]),[],n(i))*As{i}'; b = permute(reshape(tmp,[size(a,1),size(a,3),m(i)]),[1,3,2]); end end b = reshape(b,prod(m),d); % bring result in correct shape % Real eigenvalues and eigenvectors up to the rank of a real symmetric matrix. % Decompose A into V*D*V' with orthonormal matrix V and diagonal matrix D. % Diagonal entries of D obave the rank r of the matrix A as returned by % the call rank(A,tol) are zero. function [V,D] = eigr(A,tol) [V,D] = eig((A+A')/2); n = size(A,1); % decomposition of strictly symmetric A d = max(real(diag(D)),0); [d,ord] = sort(d,'descend'); % tidy up and sort if nargin<2, tol = size(A,1)*eps(max(d)); end, r = sum(d>tol); % get rank(A) d(r+1:n) = 0; % set junk eigenvalues to strict zeros if nargout==1, V = d; return, end D = diag(d); % inflate to respect interface V(:,1:r) = real(V(:,ord(1:r))); V(:,r+1:n) = null(V(:,1:r)'); % ortho completiongpml-matlab-v3.6-2015-07-07/inf/infMCMC.m000644 000765 000024 00000024661 12076017514 017250 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infMCMC(hyp, mean, cov, lik, x, y, par) % Markov Chain Monte Carlo (MCMC) sampling from posterior and % Annealed Importance Sampling (AIS) for marginal likelihood estimation. % % The algorithms are not to be used as a black box, since the acceptance rate % of the samplers need to be carefully monitored. Also, there are no derivatives % of the marginal likelihood available. % % There are additional parameters: % - par.sampler switch between the samplers 'hmc' or 'ess' % - par.Nsample num of samples % - par.Nskip num of steps out of which one sample kept % - par.Nburnin num of burn in samples (corresponds to Nskip*Nburning steps) % - par.Nais num of AIS runs to remove finite temperature bias % Default values are 'sampler=hmc', Nsample=200, Nskip=40, Nburnin=10, Nais=3. % % The Hybrid Monte Carlo Sampler (HMC) is implemented as described in the % technical report: Probabilistic Inference using MCMC Methods by Radford Neal, % CRG-TR-93-1, 1993. % % Instead of sampling from f ~ 1/Zf * N(f|m,K) P(y|f), we use a % parametrisation in terms of alpha = inv(K)*(f-m) and sample from % alpha ~ P(a) = 1/Za * N(a|0,inv(K)) P(y|K*a+m) to increase numerical % stability since log P(a) = -(a'*K*a)/2 + log P(y|K*a+m) + C and its % gradient can be computed safely. % % The leapfrog stepsize as a time discretisation stepsize comes with a % tradeoff: % - too small: frequently accept, slow exploration, accurate dynamics % - too large: seldomly reject, fast exploration, inaccurate dynamics % In order to balance between the two extremes, we adaptively adjust the % stepsize in order to keep the acceptance rate close to a target acceptance % rate. Taken from http://deeplearning.net/tutorial/hmc.html. % % The code issues a warning in case the overall acceptance rate did deviate % strongly from the target. This can indicate problems with the sampler. % % The Elliptical Slice Sampler (ESS) is a straight implementation that is % inspired by the code shipped with the paper: Elliptical slice sampling by % Iain Murray, Ryan Prescott Adams and David J.C. MacKay, AISTATS 2010. % % Annealed Importance Sampling (AIS) to determine the marginal likelihood is % described in the technical report: AIS, Radford Neal, 1998. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2012-11-07. % % See also INFMETHODS.M. if nargin<7, par = []; end % analyse parameter structure if isfield(par,'sampler'), alg=par.sampler; else alg = 'hmc'; end if isfield(par,'Nsample'), N =par.Nsample; else N = 200; end if isfield(par,'Nskip'), Ns =par.Nskip; else Ns = 40; end if isfield(par,'Nburnin'), Nb =par.Nburnin; else Nb = 10; end if isfield(par,'Nais'), R =par.Nais; else R = 3; end K = feval(cov{:}, hyp.cov, x); % evaluate the covariance matrix m = feval(mean{:}, hyp.mean, x); % evaluate the mean vector n = size(K,1); [cK,fail] = chol(K); % try an ordinary Cholesky decomposition if fail, sr2 = 1e-8*sum(diag(K))/n; cK = chol(K+sr2*eye(n)); end % regularise T = (N+Nb)*Ns; % overall number of steps [alpha,Na] = sample(K,cK,m,y,lik,hyp.lik, N,Nb,Ns, alg); % sample w/o annealing post.alpha = alpha; al = sum(alpha,2)/N; post.L = -(cK\(cK'\eye(n)) + al*al' - alpha*alpha'/N); % inv(K) - cov(alpha) post.sW = []; post.acceptance_rate_MCMC = Na/T; % additional output parameter if nargout>1 % annealed importance sampling % discrete time t from 1 to T and temperature tau from tau(1)=0 to tau(T)=1 taus = [zeros(1,Nb),linspace(0,1,N)].^4; % annealing schedule, effort at start % the fourth power idea is taken from the Kuss&Rasmussen paper, 2005 JMLR % Z(t) := \int N(f|m,K) lik(f)^taus(t) df hence Z(1) = 1 and Z(T) = Z % Z = Z(T)/Z(1) = prod_t Z(t)/Z(t-1); % ln Z(t)/Z(t-1) = ( tau(t)-tau(t-1) ) * loglik(f_t) lZ = zeros(R,1); dtaus = diff(taus(Nb+(1:N))); % we have: sum(dtaus)==1 for r=1:R [A,Na] = sample(K,cK,m,y,lik,hyp.lik, N,Nb,Ns, alg, taus); % AIS for t=2:N % evaluate the likelihood sample-wise lp = feval(lik{:},hyp.lik,y,K*A(:,t)+m,[],'infLaplace'); lZ(r) = lZ(r)+dtaus(t-1)*sum(lp); end post.acceptance_rate_AIS(r) = Na/T; % additional output parameters end nlZ = log(R)-logsumexp(lZ); % remove finite temperature bias, softmax average if nargout>2 % marginal likelihood derivatives are not computed dnlZ = struct('cov',0*hyp.cov, 'mean',0*hyp.mean, 'lik',0*hyp.lik); end end %% choose between HMC and ESS depending on the alg string function [alpha,Na] = sample(K,cK,m,y,lik,hyp, N,Nb,Ns, alg, varargin) if strcmpi(alg,'hmc') [alpha,Na] = sample_hmc(K, m,y,lik,hyp, N,Nb,Ns, varargin{:}); else [alpha,Na] = sample_ess(cK,m,y,lik,hyp, N,Nb,Ns, varargin{:}); end %% sample using elliptical slices function [alpha,T] = sample_ess(cK,m,y,lik,hyp, N,Nb,Ns, taus) if nargin>=9, tau = taus(1); else tau = 1; end % default is no annealing T = (N+Nb)*Ns; % overall number of steps F = zeros(size(m,1),N); for t=1:T if nargin>=9, tau = taus(1+floor((t-1)/Ns)); end % parameter from schedule if t==1, f=0*m; l=sum(feval(lik{:},hyp,y,f+m)); end % init sample f & lik l r = cK'*randn(size(m)); % random sample from N(0,K) [f,l] = sample_ess_step(f,l,r,m,y,lik,hyp,tau); if mod(t,Ns)==0 % keep one state out of Ns if t/Ns>Nb, F(:,t/Ns-Nb) = f; end % wait for Nb burn-in steps end end alpha = cK\(cK'\F); %% elliptical slice sampling: one step function [f,l] = sample_ess_step(f,l,r,m,y,lik,hyp,tau) if nargin<7, tau=1; end, if tau>1, tau=1; end, if tau<0, tau=0; end h = log(rand) + tau*l; % acceptance threshold a = rand*2*pi; amin = a-2*pi; amax = a; % bracket whole ellipse k = 0; % emergency break while true % slice sampling loop; f for proposed angle diff; check if on slice fp = f*cos(a) + r*sin(a); % move on ellipsis defined by r l = sum(feval(lik{:},hyp,y,fp+m)); if tau*l>h || k>20, break, end % exit if new point is on slice or exhausted if a>0, amax=a; elseif a<0, amin=a; end % shrink slice to rejected point a = rand*(amax-amin) + amin; k = k+1; % propose new angle difference; break end f = fp; % accept %% sample using Hamiltonian dynamics as proposal algorithm function [alpha,Na] = sample_hmc(K,m,y,lik,hyp, N,Nb,Ns, taus) % use adaptive stepsize rule to enforce a specific acceptance rate epmin = 1e-6; % minimum leapfrog stepsize epmax = 9e-1; % maximum leapfrog stepsize ep = 1e-2; % initial leapfrog stepsize acc_t = 0.9; % target acceptance rate acc = 0; % current acceptance rate epinc = 1.02; % increase factor of stepsize if acceptance rate is below target epdec = 0.98; % decrease factor of stepsize if acceptance rate is above target lam = 0.01; % exponential moving average computation of the acceptance rate % 2/(3*lam) steps half height; lam/nstep = 0.02/33, 0.01/66, 0.005/133 l = 20; % number of leapfrog steps to perform for one step n = size(K,1); T = (N+Nb)*Ns; % overall number of steps alpha = zeros(n,N); % sample points al = zeros(n,1); % current position if nargin>=9, tau = taus(1); else tau = 1; end % default is no annealing [gold,eold] = E(al,K,m,y,lik,hyp,tau); % initial energy, gradient Na = 0; % number of accepted points for t=1:T if nargin>=9, tau = taus(1+floor((t-1)/Ns)); end % parameter from schedule p = randn(n,1); % random initial momentum q = al; g = gold; Hold = (p'*p)/2 + eold; % H = Ekin + Epot => Hamiltonian for ll=1:l % leapfrog discretization steps, Euler like p = p - (ep/2)*g; % half step in momentum p q = q + ep*p; % full step in position q g = E(q,K,m,y,lik,hyp,tau); % compute new gradient g p = p - (ep/2)*g; % half step in momentum p end [g,e] = E(q,K,m,y,lik,hyp,tau); H = (p'*p)/2 + e; % recompute Hamiltonian acc = (1-lam)*acc; % decay current acceptance rate if log(rand) < Hold-H % accept with p = min(1,exp(Hold-H)) al = q; % keep new state, gold = g; % gradient eold = e; % and potential energy acc = acc + lam; % increase rate due to acceptance Na = Na+1; % increase number accepted steps end if acc>acc_t % too large acceptance rate => increase leapfrog stepsize ep = epinc*ep; else % too small acceptance rate => decrease leapfrog stepsize ep = epdec*ep; end if epepmax, ep = epmax; end if mod(t,Ns)==0 % keep one state out of Ns if t/Ns>Nb, alpha(:,t/Ns-Nb) = al; end % wait for Nb burn-in steps end end if Na/T dE/dal = K*( al-dloglik(f) ) if nargin<7, tau=1; end, if tau>1, tau=1; end, if tau<0, tau=0; end Kal = K*al; [lp,dlp] = feval(lik{:},hyp,y,Kal+m,[],'infLaplace'); g = Kal-K*(tau*dlp); if nargout>1, e = (al'*Kal)/2 - tau*sum(lp); end %% y = logsumexp(x) = log(sum(exp(x(:))) avoiding overflow function y=logsumexp(x) mx = max(x(:)); y = log(sum(exp(x(:)-mx)))+mx; gpml-matlab-v3.6-2015-07-07/inf/infLOO.m000644 000765 000024 00000010234 12076017514 017151 0ustar00carlstaff000000 000000 function [post nloo dnloo loo] = infLOO(hyp, mean, cov, lik, x, y) % Leave-One-Out - Perform Least-Squares GP predictions in the style of the % Pseudo Likelihood in §5.4.2 p. 117 and the Probabilistic Least-squares % Classifier of §6.5 p. 146 of the GPML book. % For Gaussian likelihood, the prediction is the same as with infExact, for % lik{Erf,Logistic} the result corresponds to probabilistic least-squares % classification. In any case, the objective returned is not the marginal % likelihood but the predictive log probability. % The standard deviation of the noise either comes from the sn parameter of the % likelihood function lik{Gauss,Laplace,Sech2} or from the covariance function % by means of using covSum and covNoise i.e. for lik{Uni,Erf,Logistic}. % We then compute the negative leave-one-out predictive probability and its % derivatives w.r.t. the hyperparameters. See also "help infMethods". % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2012-11-07 % % See also INFMETHODS.M. [n, D] = size(x); K = feval(cov{:}, hyp.cov, x); % evaluate covariance matrix m = feval(mean{:}, hyp.mean, x); % evaluate mean vector if numel(hyp.lik)==0 cov1 = cov{1}; if isa(cov1, 'function_handle'), cov1 = func2str(cov1); end if ~strcmp(cov1,'covSum'); error('Only covSum supported.'), end % check cov cov2 = cov{2}; npar = 0; for i=1:length(cov2) cov2i = cov2{i}; if numel(cov2i)==1 if isa(cov2i, 'function_handle'), cov2i = func2str(cov2i); end if strcmp(cov2i,'covNoise'), isn = npar+1; sn2 = exp(2*hyp.cov(isn)); end end npar = npar + eval(feval(cov2i)); end if ~exist('sn2','var'), error('We need covNoise in the covSum.'), end else % Gauss, Laplace, Sech2 sn2 = exp(2*hyp.lik(end)); isn = 0; K = K + sn2*eye(n); end L = chol(K)/sqrt(sn2); % Cholesky factor of covariance with noise alpha = solve_chol(L,y-m)/sn2; post.alpha = alpha; % return the posterior parameters post.sW = ones(n,1)/sqrt(sn2); % sqrt of noise precision vector post.L = L; % L = chol(eye(n)+sW*sW'.*K) if nargout>1 % do we want the marginal likelihood? sdeff = 1./post.sW; % effective Gaussian width mneff = K*alpha + m; % alpha = inv(K)*(mneff-m), effective Gaussian center % all loo marginal means and variances at once V = post.L'\diag(1./sdeff); r = sum(V.*V,1)'; % r=diag( inv(K) ) loo.fmu = mneff - alpha./r; % GPML book, eq. 5.12 loo.fs2 = 1./r - sn2; % GPML book, eq. 5.12 [loo.lp, loo.ymu, loo.ys2] = feval(lik{:},hyp.lik,y,loo.fmu,loo.fs2); nloo = -sum(loo.lp); % negative leave-one-out predictive probability if nargout>2 % do we want derivatives? [lZ,dlZmn,d2lZmn] = feval(lik{:},hyp.lik,y,loo.fmu,loo.fs2,'infEP'); dlZva = (d2lZmn+dlZmn.*dlZmn)/2; % derivative w.r.t. variance dnloo = hyp; iK = solve_chol(L,eye(n))/sn2; for j=1:numel(hyp.cov) dKj = feval(cov{:}, hyp.cov, x, [], j); Zj = solve_chol(L,dKj)/sn2; % GPML book, eq. 5.13 dva = sum(Zj.*iK,2)./(r.*r); % GPML book, eq. 5.13 dmn = (Zj*alpha)./r - alpha.*dva; % GPML book, eq. 5.13 if j==isn, dva = dva - 2*sn2; end % additional part for covNoise dnloo.cov(j) = -dlZmn'*dmn -dlZva'*dva; end for j=1:numel(hyp.lik) dnloo.lik(j) = -sum( feval(lik{:},hyp.lik,y,loo.fmu,loo.fs2,'infEP',j) ); if j==numel(hyp.lik) dva = 2*sn2*sum(iK.*iK,2)./(r.*r); % GPML book, eq. 5.13 dmn = 2*solve_chol(L,alpha)./r - alpha.*dva; % GPML book, eq. 5.13 dva = dva - 2*sn2; dnloo.lik(j) = dnloo.lik(j) -dlZmn'*dmn -dlZva'*dva; end end for j=1:numel(hyp.mean) dmj = feval(mean{:}, hyp.mean, x, j); dmn = solve_chol(L,dmj)./(sn2*r); dnloo.mean(j) = -dlZmn'*dmn; end end end gpml-matlab-v3.6-2015-07-07/inf/infPrior.m000644 000765 000024 00000013227 12441263152 017615 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infPrior(inf, prior, hyp, varargin) % Inference with hyperparameter prior. Any inference method can be used. % The prior specification is done by means of the prior parameter that has % the same structure as hyp. Examples: % 1) >> prior.cov{2} = {@priorLaplace,mu,s2}; % put a Laplace prior on the second covariance function hyperparameter % i.e. hyp.cov(2) ~ Laplace(mu,s2) % 2) >> prior.mean{3} = {@priorDelta} or % >> prior.mean{3} = @priorDelta or % >> prior.mean{3} = 'priorClamped' % fix the third mean function hyperparameter to its current value and % exclude it from optimisation % i.e. hyp.mean(3) ~ delta(hyp.mean(3)) % 3) >> prior.xu = repmat({{@priorGauss,0,1}},D*nu,1) % put an individual standard normal prior on every inducing input coordinate % for FITC inference % i.e. hyp.xu(i) ~ N(0,1) for all i = 1..D*nu % 4) >> prior.multi{1} = {@priorGaussMulti,mu,s2, struct('xu',1:D*nu)} % put a joint Gaussian prior with mean vector mu and covariance matrix % s2 on the inducing input coordinates in FITC inference % i.e. hyp.xu ~ N(mu,s2) % 5) >> prior.multi{1} = {@priorTMulti,mu,s2,nu, off+(1:D)} % put a joint Student's t prior with mean vector mu, covariance matrix s2 % and nu degrees of freedom on the weights of the mean function e.g. % hyp.mean ~ T(mu,s2,nu) for meanLinear % Here, we do not have a struct as in 4) but the index in unwrap(hyp). % 6) >> p = {@priorWeibull,lam,k}; g = @exp; dg = @exp; ig = @log; % >> prior.cov{2} = {@priorTransform,g,dg,ig,p} % put a univariate Weibull distribution on the exponentiated second % covariance function hyperparameter % i.e. exp(hyp.cov(2)) ~ Weibull(lam,k) % % Return a parametrization of the posterior, the negative log marginal % likelihood and its derivatives w.r.t. the hyperparameters. % % Note that at most 1 prior is allowed per hyperparameter. We consider the % specification of more than 1 prior for a hyperparameter as an error. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Hannes Nickisch and Roman Garnett, 2014-12-04. % % See also INFMETHODS.M, USAGEPRIOR.M, PRIORDISTRIBUTIONS.M. [post nlZ dnlZ] = inf(hyp, varargin{:}); % perform inference if ~isempty(prior) % add hyperprior contributions to nlZ and dnlZ nam = fieldnames(prior); num = zeros(numel(unwrap(hyp)),1); % number of priors per hyperparameter for i=1:numel(nam) % iterate over kinds of hyperparameters cov/lik/mean/xu ni = nam{i}; % name of the ith field if strcmp(ni,'multi') % first catch multivariate priors p = prior.(ni); for j=1:numel(p) % iterate over individual muti-hyperparameter components pj = p{j}; if ~isempty(pj) % only proceed if a nonempty prior is specified idx = pj{end}; pj = pj(1:end-1); % grab index from cell end pjstr = pj{1}; if ~ischar(pjstr), pjstr = func2str(pjstr); end if numel(pjstr)<5 || ~strcmp(pjstr(end-4:end),'Multi') error('multivariate hyperpriors are called Multi') end if isstruct(idx) % massage idx into a vector idxs = rewrap(hyp,zeros(size(num))); % structured index for nj = fieldnames(idx), idxs.(nj{1})( idx.(nj{1}) ) = 1; end idx = unwrap(idxs)>0; % linearise structured index else idxz = zeros(size(num)); idxz(idx) = 1; idx = idxz>0; % binary index end if sum(idx)<=1, error('multivariate priors need >1 hyperparam'), end num(idx) = num(idx)+1; % inc prior counter hypu = unwrap(hyp); dnlZ = unwrap(dnlZ); if strncmp(pjstr,'priorClamped',12) || ... % clamp derivative to zero strncmp(pjstr,'priorDelta',10), dnlZ(idx) = 0; else [lp,dlp] = feval(pj{:}, hypu(idx)); % evaluate prior distribution nlZ = nlZ-lp; dnlZ(idx) = dnlZ(idx)-dlp(:); % update nlZ and dnlZ end dnlZ = rewrap(hyp,dnlZ); else error('multivariate priors should be non empty') end end continue % jump to univariate priors end if ~isfield(hyp,ni), error(['unknown prior field ',ni,' in hyp']), end p = prior.(ni); if numel(p)~=numel(hyp.(ni)), error(['bad hyp/prior field length ',ni]), end for j=1:numel(p) % iterate over individual hyperparameter components pj = p{j}; if ~iscell(pj) && ~isempty(pj), pj = {pj}; end % enforce cell if ~isempty(pj) % only proceed if a nonempty prior is specified num = rewrap(hyp,num); num.(ni)(j) = num.(ni)(j)+1; % inc prior counter num = unwrap(num); pj1str = pj{1}; if ~ischar(pj1str), pj1str = func2str(pj1str); end if strncmp(pj1str,'priorClamped',12) || strncmp(pj1str,'priorDelta',10) dnlZ.(ni)(j) = 0; % clamp derivative to zero else [lp,dlp] = feval(pj{:}, hyp.(ni)(j)); % evaluate prior distribution nlZ = nlZ-lp; dnlZ.(ni)(j) = dnlZ.(ni)(j)-dlp; % update nlZ and dnlZ end end end end if any(num>1) % check for hypers with more than a single prior num = rewrap(hyp,num); nam = fieldnames(num); s = ''; for i=1:numel(nam) idx = find(num.(nam{i})>1); for j=1:numel(idx) s = [s,sprintf('hyp.%s(%d) ',nam{i},idx(j))]; end end error(['More than 1 prior specified for ',s(1:end-1),'.']) end endgpml-matlab-v3.6-2015-07-07/util/binaryEPGP.m000644 000765 000024 00000005753 11422240325 020166 0ustar00carlstaff000000 000000 function varargout = binaryEPGP(hyper, covfunc, x, y, xstar) % binaryEPGP - The Expectation Propagation approximation for binary Gaussian % process classification. Two modes are possible: training or testing: if no % test cases are supplied, then the approximate negative log marginal % likelihood and its partial derivatives wrt the hyperparameters is computed; % this mode is used to fit the hyperparameters. If test cases are given, then % the test set predictive probabilities are returned. The program is flexible % in allowing a multitude of covariance functions. % % usage: [nlZ, dnlZ ] = binaryEPGP(hyper, covfunc, x, y); % or: [p, mu, s2, nlZ] = binaryEPGP(hyper, covfunc, x, y, xstar); % % where: % % hyper is a (column) vector of hyperparameters % covfunc is the name of the covariance function (see below) % lik is the name of the likelihood function (see below) % x is a n by D matrix of training inputs % y is a (column) vector (of size n) of binary +1/-1 targets % xstar is a nn by D matrix of test inputs % nlZ is the returned value of the negative log marginal likelihood % dnlZ is a (column) vector of partial derivatives of the negative % log marginal likelihood wrt each log hyperparameter % p is a (column) vector (of length nn) of predictive probabilities % mu is a (column) vector (of length nn) of predictive latent means % s2 is a (column) vector (of length nn) of predictive latent variances % % The length of the vector of hyperparameters depends on the covariance % function, as specified by the "covfunc" input to the function, specifying the % name of a covariance function. A number of different covariance function are % implemented, and it is not difficult to add new ones. See "help covFunctions" % for the details % % The function can conveniently be used with the "minimize" function to train % a Gaussian process, eg: % % [hyper, fX, i] = minimize(hyper, 'binaryEPGP', length, 'covSEiso', % 'logistic', x, y); % % Copyright (c) 2004, 2005, 2006, 2007 Carl Edward Rasmussen, 2010-07-21. if nargin<4 || nargin>5 disp('Usage: [nlZ, dnlZ ] = binaryEPGP(hyper, covfunc, x, y);') disp(' or: [p, mu, s2, nlZ] = binaryEPGP(hyper, covfunc, x, y, xstar);') return end % Note, this function is just a wrapper provided for backward compatibility, % the functionality is now provided by the more general gp function. lik = @likErf; inf = @infEP; mean = @meanZero; varargout = cell(nargout, 1); % allocate the right number of output arguments hyp.cov = hyper; if nargin==4 [varargout{:}] = gp(hyp,inf,mean,covfunc,lik,x,y); if nargout>1, varargout{2} = varargout{2}.cov; end else [ymu,ys2,fmu,fs2] = gp(hyp,inf,mean,covfunc,lik,x,y,xstar); if nargout>0, varargout{1} = (1+ymu)/2; end if nargout>1, varargout{2} = fmu; end if nargout>2, varargout{3} = fs2; end if nargout>3, varargout{4} = gp(hyp,inf,mean,covfunc,lik,x,y); end % nlZ end gpml-matlab-v3.6-2015-07-07/util/binaryLaplaceGP.m000644 000765 000024 00000007030 11422240325 021211 0ustar00carlstaff000000 000000 function varargout = binaryLaplaceGP(hyper, covfunc, lik, x, y, xstar) % binaryLaplaceGP - Laplace's approximation for binary Gaussian process % classification. Two modes are possible: training or testing: if no test % cases are supplied, then the approximate negative log marginal likelihood % and its partial derivatives wrt the hyperparameters is computed; this mode is % used to fit the hyperparameters. If test cases are given, then the test set % predictive probabilities are returned. The program is flexible in allowing % several different likelihood functions and a multitude of covariance % functions. % % usage: [nlZ, dnlZ ] = binaryLaplaceGP(hyper, covfunc, lik, x, y); % or: [p, mu, s2, nlZ] = binaryLaplaceGP(hyper, covfunc, lik, x, y, xstar); % % where: % % hyper is a (column) vector of hyperparameters % covfunc is the name of the covariance function (see below) % lik is the name of the likelihood function (see below) % x is a n by D matrix of training inputs % y is a (column) vector (of size n) of binary +1/-1 targets % xstar is a nn by D matrix of test inputs % nlZ is the returned value of the negative log marginal likelihood % dnlZ is a (column) vector of partial derivatives of the negative % log marginal likelihood wrt each log hyperparameter % p is a (column) vector (of length nn) of predictive probabilities % mu is a (column) vector (of length nn) of predictive latent means % s2 is a (column) vector (of length nn) of predictive latent variances % % The length of the vector of log hyperparameters depends on the covariance % function, as specified by the "covfunc" input to the function, specifying the % name of a covariance function. A number of different covariance function are % implemented, and it is not difficult to add new ones. See "help covFunctions" % for the details. % % The shape of the likelihood function is given by the "lik" input to the % function, specifying the name of the likelihood function. The two implemented % likelihood functions are: % % logistic the logistic function: 1/(1+exp(-x)) % cumGauss the cumulative Gaussian (error function) % % The function can conveniently be used with the "minimize" function to train % a Gaussian process, eg: % % [hyper, fX, i] = minimize(hyper, 'binaryLaplaceGP', length, 'covSEiso', % 'logistic', x, y); % % Copyright (c) 2004, 2005, 2006, 2007 Carl Edward Rasmussen, 2010-07-21. if nargin<5 || nargin>6 disp('Usage: [nlZ, dnlZ ] = binaryLaplaceGP(hyper, covfunc, lik, x, y);') disp(' or: [p, mu, s2, nlZ] = binaryLaplaceGP(hyper, covfunc, lik, x, y, xstar);') return end % Note, this function is just a wrapper provided for backward compatibility, % the functionality is now provided by the more general gp function. if strcmp(lik,'logistic') lik = @likLogistic; elseif strcmpi(lik,'cumgauss') lik = @likErf; else error('Allowable likelihoods: logistic and cumGauss.'); end inf = @infLaplace; mean = @meanZero; varargout = cell(nargout, 1); % allocate the right number of output arguments hyp.cov = hyper; if nargin==5 [varargout{:}] = gp(hyp,inf,mean,covfunc,lik,x,y); if nargout>1, varargout{2} = varargout{2}.cov; end else [ymu,ys2,fmu,fs2] = gp(hyp,inf,mean,covfunc,lik,x,y,xstar); if nargout>0, varargout{1} = (1+ymu)/2; end if nargout>1, varargout{2} = fmu; end if nargout>2, varargout{3} = fs2; end if nargout>3, varargout{4} = gp(hyp,inf,mean,covfunc,lik,x,y); end % nlZ end gpml-matlab-v3.6-2015-07-07/util/logsumexp2.m000644 000765 000024 00000000706 12230176225 020331 0ustar00carlstaff000000 000000 % Compute y = log( sum(exp(x),2) ), the softmax in a numerically safe way by % subtracting the row maximum to avoid cancelation after taking the exp % the sum is done along the rows. % % Copyright (c) by Hannes Nickisch, 2013-10-16. function [y,x] = logsumexp2(logx) N = size(logx,2); max_logx = max(logx,[],2); % we have all values in the log domain, and want to calculate a sum x = exp(logx-max_logx*ones(1,N)); y = log(sum(x,2)) + max_logx;gpml-matlab-v3.6-2015-07-07/util/solve_chol.c000644 000765 000024 00000004371 12301637047 020354 0ustar00carlstaff000000 000000 /* solve_chol - solve a linear system A*X = B using the Cholesky factorization * of A (square, symmetric and positive definite) using LAPACK/DPOTRS. * * Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2014-02-13. * * Modifications from 2014-02-09 as suggested by Todd Small: * - types of q,m,n changed from long to integer, q error check had wrong sign */ #include "mex.h" #include #ifdef MEX_INFORMATION_VERSION /* now we are compiling for Matlab */ #if defined(_WIN64) #define integer long long #else #define integer long #endif #define doublereal double #else /* now we are compiling for Octave */ #ifdef __APPLE__ #include typedef __CLPK_integer integer; typedef __CLPK_doublereal doublereal; #else typedef int integer; typedef double doublereal; #endif #endif #if !defined(_WIN32) || !defined(MEX_INFORMATION_VERSION) /* not Win32/Matlab */ #define dpotrs dpotrs_ #endif extern integer dpotrs_(char *, integer *, integer *, doublereal *, integer *, doublereal *, integer *, integer *); void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) { /* Matlab call: X = solve_chol(A, B) */ doublereal *C; integer n, m, q; if (nrhs != 2 || nlhs > 1) /* check input */ mexErrMsgTxt("Usage: X = solve_chol(A, B)"); n = mxGetM(prhs[0]); /* number of rows in inputs A and B */ if (n != mxGetN(prhs[0])) mexErrMsgTxt("First argument matrix must be square."); if (n != mxGetM(prhs[1])) mexErrMsgTxt("Both argument matrices must have the same number of rows."); m = mxGetN(prhs[1]); /* number of colums in second input B */ plhs[0] = mxCreateDoubleMatrix(n, m, mxREAL); /* space for output X */ C = mxGetPr(plhs[0]); if (n == 0) return; /* if argument was empty matrix, do no more */ memcpy( C, mxGetPr(prhs[1]), m*n*mxGetElementSize(plhs[0]) ); /* copy data */ dpotrs("U", &n, &m, mxGetPr(prhs[0]), &n, C, &n, &q); /* solve system */ if (q < 0) mexErrMsgTxt("Error: illegal input to solve_chol"); } gpml-matlab-v3.6-2015-07-07/util/solve_chol.m000644 000765 000024 00000001742 12240130516 020354 0ustar00carlstaff000000 000000 % solve_chol - solve linear equations from the Cholesky factorization. % Solve A*X = B for X, where A is square, symmetric, positive definite. The % input to the function is R the Cholesky decomposition of A and the matrix B. % Example: X = solve_chol(chol(A),B); % % NOTE: The program code is written in the C language for efficiency and is % contained in the file solve_chol.c, and should be compiled using matlabs mex % facility. However, this file also contains a (less efficient) matlab % implementation, supplied only as a help to people unfamiliar with mex. If % the C code has been properly compiled and is available, it automatically % takes precendence over the matlab code in this file. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2010-09-18. function X = solve_chol(L, B) if nargin ~= 2 || nargout > 1 error('Wrong number of arguments.'); end if size(L,1) ~= size(L,2) || size(L,1) ~= size(B,1) error('Wrong sizes of matrix arguments.'); end X = L\(L'\B);gpml-matlab-v3.6-2015-07-07/util/lbfgsb.m000644 000765 000024 00000005563 12240130516 017463 0ustar00carlstaff000000 000000 % LBFGSB Call the nonlinear bound-constrained solver that uses % limited-memory BFGS quasi-Newton updates. % % The basic function call is % % LBFGSB(x0,lb,ub,objfunc,gradfunc) % % The first input argument x0 is either a matrix or a cell array of % matrices. It declares the starting point for the solver. % % The second and third input arguments lb and ub must be of the exact same % structure as x0. They declare the lower and upper bounds on the % variables, respectively. Set an entry to Inf to declare no bound. % % The next two input arguments must be the names of MATLAB routines % (M-files): % % objfunc Calculates the objective function at the current % point. The routine must accept as many inputs as cell % entry in x0 (or one input if x0 is a matrix). The % output must be a scalar representing the objective % evaluated at the current point. % % gradfunc Computes the gradient of the objective at the current % point. The input is the same as objfunc, but it must % return as many outputs as there are inputs, and each % of the outputs have the same matrix structure as its % corresponding input. % % Optionally, one may choose to pass additional auxiliary data to the % MATLAB callback routines listed above through the function call % LBFGSB(...,auxdata). If auxdata is the empty matrix, no extra % information is passed. It is important to observe that the auxiliary % data MAY NOT CHANGE through the course of the L-BFGS optimization! The % auxiliary data keep the same values as they possessed in the initial % call. If you need variables that change over time, you may want to % consider global variables (type HELP GLOBAL). % % LBFGSB(...,auxdata,iterfunc) specifies an additional callback routine % which is called once per algorithm iteration. The callback routine must % take the form ITERFUNC(T,F,X1,X2,...,AUXDATA). T is the current % iteration of the algorithm. F is the current value of the objective. The % inputs X1, X2,... are exactly the same as the ones passed to objfunc and % gradfunc. Finally, extra information may be passed through the input % AUXDATA. No outputs are expected from iterfunc. If iterfunc is the empty % string, no routine is called. % % These inputs may be followed by parameter/value pairs to modify the % default algorithm options. The algorithm option 'maxiter' sets the % maximum number of iterations. The rest of the options 'm', 'factr' and % 'pgtol' are described in the L-BFGS-B documentation. Note that in this % implementation we divide factr by the machine precision, so that the % convergence condition is satisfied when it is less than factr, not less % than factr times the machine precision. gpml-matlab-v3.6-2015-07-07/util/lbfgsb/000755 000765 000024 00000000000 12550665747 017323 5ustar00carlstaff000000 000000 gpml-matlab-v3.6-2015-07-07/util/minimize_lbfgsb_gradfun.m000644 000765 000024 00000004526 11422240325 023071 0ustar00carlstaff000000 000000 function G = minimize_lbfgsb_gradfun(X,varargin) % extract input arguments varargin = varargin{1}; strctX = varargin{2}; f = varargin{1}; % global variables serve as communication interface between calls global minimize_lbfgsb_iteration_number global minimize_lbfgsb_objective global minimize_lbfgsb_gradient global minimize_lbfgsb_X if norm(unwrap(X)-unwrap(minimize_lbfgsb_X))>1e-10 [y,G] = feval(f,rewrap(strctX,X),varargin{3:end}); else y = minimize_lbfgsb_objective(minimize_lbfgsb_iteration_number); G = minimize_lbfgsb_gradient; end % memorise gradient and position minimize_lbfgsb_gradient = G; minimize_lbfgsb_X = X; G = unwrap(G); % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse rewrap.m. function v = unwrap(s) v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s)));% alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; unwrap(s{i})]; end end % other types are ignored % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also unwrap.m. function [s v] = rewrap(s, v) if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed gpml-matlab-v3.6-2015-07-07/util/glm_invlink_exp.m000644 000765 000024 00000000673 12230176225 021414 0ustar00carlstaff000000 000000 % Compute the log intensity for the inverse link function g(f) = exp(f). % % The function is used in GLM likelihoods such as likPoisson, likGamma, likBeta % and likInvGauss. % % Copyright (c) by Hannes Nickisch, 2013-10-16. function [lg,dlg,d2lg,d3lg] = glm_invlink_exp(f) lg = f; if nargout>1 dlg = ones(size(f)); if nargout>2 d2lg = zeros(size(f)); if nargout>2 d3lg = zeros(size(f)); end end endgpml-matlab-v3.6-2015-07-07/util/minimize_lbfgsb.m000644 000765 000024 00000010574 11422240325 021363 0ustar00carlstaff000000 000000 function [X, fX, i] = minimize_lbfgsb(X, f, length, varargin) % Minimize a differentiable multivariate function using quasi Newton. % % Usage: [X, fX, i] = minimize_lbfgsb(X, f, length, P1, P2, P3, ... ) % % X initial guess; may be of any type, including struct and cell array % f the name or pointer to the function to be minimized. The function % f must return two arguments, the value of the function, and it's % partial derivatives wrt the elements of X. The partial derivative % must have the same type as X. % length length of the run; if it is positive, it gives the maximum number of % line searches, if negative its absolute gives the maximum allowed % number of function evaluations. Optionally, length can have a second % component, which will indicate the reduction in function value to be % expected in the first line-search (defaults to 1.0). % P1, P2 ... parameters are passed to the function f. % % X the returned solution % fX vector of function values indicating progress made % i number of iterations (line searches or function evaluations, % depending on the sign of "length") used at termination. % % The function returns when either its length is up, or if no further progress % can be made (ie, we are at a (local) minimum, or so close that due to % numerical problems, we cannot get any closer). NOTE: If the function % terminates within a few iterations, it could be an indication that the % function values and derivatives are not consistent (ie, there may be a bug in % the implementation of your "f" function). % % Copyright (C) 2010 by Hannes Nickisch, 2010-02-05 % global variables serve as communication interface between calls global minimize_lbfgsb_iteration_number global minimize_lbfgsb_objective global minimize_lbfgsb_gradient global minimize_lbfgsb_X % init global variables minimize_lbfgsb_iteration_number = 0; minimize_lbfgsb_objective = Inf; minimize_lbfgsb_gradient = 0*unwrap(X); minimize_lbfgsb_X = X; X0 = X; lb = -Inf*ones(size(unwrap(X0))); ub = Inf*ones(size(unwrap(X0))); maxiter = abs(length); % max number of iterations % no callback routine used so far % m is the number of saved vectors used to estimate the Hessian % factr is the precision 1e-12 X = lbfgsb( unwrap(X0), lb, ub, 'minimize_lbfgsb_objfun', ... 'minimize_lbfgsb_gradfun', ... {f,X0,varargin{:}}, [], ... 'maxiter',maxiter, 'm',4, 'factr',1e-12, 'pgtol',1e-5); i = minimize_lbfgsb_iteration_number; fX = minimize_lbfgsb_objective; X = rewrap(X0,X); % clear global variables clear minimize_lbfgsb_iteration_number clear minimize_lbfgsb_objective clear minimize_lbfgsb_gradient clear minimize_lbfgsb_X % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse rewrap.m. function v = unwrap(s) v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s)));% alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; unwrap(s{i})]; end end % other types are ignored % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also unwrap.m. function [s v] = rewrap(s, v) if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed gpml-matlab-v3.6-2015-07-07/util/gpr.m000644 000765 000024 00000004667 11447142675 017041 0ustar00carlstaff000000 000000 function varargout = gpr(hyper, cov, x, y, xs) % gpr - Gaussian process regression, with a named covariance function. Two % modes are possible: training and prediction: if no test data are given, the % function returns minus the log likelihood and its partial derivatives with % respect to the hyperparameters; this mode is used to fit the hyperparameters. % If test data are given, then (marginal) Gaussian predictions are computed, % whose mean and variance are returned. Note that in cases where the covariance % function has noise contributions, the variance returned in s2 is for noisy % test targets; if you want the variance of the noise-free latent function, you % must substract the noise variance. % % usage: [nlZ dnlZ] = gpr(hyp, cov, x, y) % or: [mu s2] = gpr(hyp, cov, x, y, xs) % % where: % % hyp is a (column) vector of log hyperparameters % cov is the covariance function % x is a n by D matrix of training inputs % y is a (column) vector (of size n) of targets % xs is a ns by D matrix of test inputs % nlZ is the returned value of the negative log marginal likelihood % dnlZ is a (column) vector of partial derivatives of the negative % log marginal likelihood wrt each log hyperparameter % mu is a (column) vector (of size nn) of prediced means % s2 is a (column) vector (of size nn) of predicted variances % % For more help on covariance functions, see "help covFunctions". % % Copyright (c) 2010 Carl Edward Rasmussen and Hannes Nickisch 2010-06-18. err = 'we need cov = {''covSum'', {cov1, ..,''covNoise'', .., covP} }; to map to gp.m'; if ~strcmp(cov{1},'covSum'), error(err), end id = 0; nhyp = zeros(length(cov{2}),1); for i=1:length(cov{2}) if strcmp(cov{2}{i},'covNoise'), id = i; break, end nhyp(i) = eval(feval(cov{2}{i})); end if id==0, error(err), else nhyp = sum(nhyp(1:id-1)); end cov{2} = cov{2}([1:id-1,id+1:end]); hyp.lik = hyper(nhyp+1); hyp.cov = hyper([1:nhyp,nhyp+2:end]); % Note, this function is just a wrapper provided for backward compatibility, % the functionality is now provided by the more general gp function. lik = @likGauss; inf = @infExact; mean = @meanZero; varargout = cell(nargout, 1); % allocate the right number of output arguments if nargin==4 [varargout{:}] = gp(hyp,inf,mean,cov,lik,x,y); if nargout>1, varargout{2} = [varargout{2}.lik; varargout{2}.cov]; end else [varargout{:}] = gp(hyp,inf,mean,cov,lik,x,y,xs); end gpml-matlab-v3.6-2015-07-07/util/glm_invlink_logit.m000644 000765 000024 00000001422 12230176225 021727 0ustar00carlstaff000000 000000 % Compute the log intensity for the inverse link function g(f) = 1/(1+exp(-f)). % % The function is used in GLM likelihoods such as likPoisson, likGamma, likBeta % and likInvGauss. % % Copyright (c) by Hannes Nickisch, 2013-10-16. function varargout = glm_invlink_logit(f) varargout = cell(nargout, 1); % allocate the right number of output arguments [varargout{:}] = glm_invlink_logistic(f); if nargout>0 elg = exp(varargout{1}); varargout{1} = f - elg; if nargout>1 dlg = varargout{2}; varargout{2} = 1 - elg.*dlg; if nargout>2 d2lg = varargout{3}; varargout{3} = -elg.*(dlg.^2+d2lg); if nargout>3 d3lg = varargout{4}; varargout{4} = -elg.*(dlg.^3+3*d2lg.*dlg+d3lg); end end end endgpml-matlab-v3.6-2015-07-07/util/glm_invlink_expexp.m000644 000765 000024 00000000653 12230176225 022127 0ustar00carlstaff000000 000000 % Compute the log intensity for the inverse link function g(f) = exp(-exp(-f)). % % The function is used in GLM likelihoods such as likPoisson, likGamma, likBeta % and likInvGauss. % % Copyright (c) by Hannes Nickisch, 2013-10-16. function [lg,dlg,d2lg,d3lg] = glm_invlink_expexp(f) lg = -exp(-f); if nargout>1 dlg = -lg; if nargout>2 d2lg = lg; if nargout>2 d3lg = -lg; end end endgpml-matlab-v3.6-2015-07-07/util/sq_dist.m000644 000765 000024 00000003657 12240130516 017674 0ustar00carlstaff000000 000000 % sq_dist - a function to compute a matrix of all pairwise squared distances % between two sets of vectors, stored in the columns of the two matrices, a % (of size D by n) and b (of size D by m). If only a single argument is given % or the second matrix is empty, the missing matrix is taken to be identical % to the first. % % Usage: C = sq_dist(a, b) % or: C = sq_dist(a) or equiv.: C = sq_dist(a, []) % % Where a is of size Dxn, b is of size Dxm (or empty), C is of size nxm. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-12-13. function C = sq_dist(a, b) if nargin<1 || nargin>3 || nargout>1, error('Wrong number of arguments.'); end bsx = exist('bsxfun','builtin'); % since Matlab R2007a 7.4.0 and Octave 3.0 if ~bsx, bsx = exist('bsxfun'); end % bsxfun is not yet "builtin" in Octave [D, n] = size(a); % Computation of a^2 - 2*a*b + b^2 is less stable than (a-b)^2 because numerical % precision can be lost when both a and b have very large absolute value and the % same sign. For that reason, we subtract the mean from the data beforehand to % stabilise the computations. This is OK because the squared error is % independent of the mean. if nargin==1 % subtract mean mu = mean(a,2); if bsx a = bsxfun(@minus,a,mu); else a = a - repmat(mu,1,size(a,2)); end b = a; m = n; else [d, m] = size(b); if d ~= D, error('Error: column lengths must agree.'); end mu = (m/(n+m))*mean(b,2) + (n/(n+m))*mean(a,2); if bsx a = bsxfun(@minus,a,mu); b = bsxfun(@minus,b,mu); else a = a - repmat(mu,1,n); b = b - repmat(mu,1,m); end end if bsx % compute squared distances C = bsxfun(@plus,sum(a.*a,1)',bsxfun(@minus,sum(b.*b,1),2*a'*b)); else C = repmat(sum(a.*a,1)',1,m) + repmat(sum(b.*b,1),n,1) - 2*a'*b; end C = max(C,0); % numerical noise can cause C to negative i.e. C > -1e-14 gpml-matlab-v3.6-2015-07-07/util/cov_deriv_sq_dist.m000644 000765 000024 00000003562 12240130516 021727 0ustar00carlstaff000000 000000 % Compute derivative k'(x^p,x^q) of a stationary covariance k(d2) (ard or iso) % w.r.t. to squared distance d2 = (x^p - x^q)'*inv(P)*(x^p - x^q) measure. Here % P is either diagonal with ARD parameters ell_1^2,...,ell_D^2 where D is the % dimension of the input space or ell^2 times the unit matrix for isotropic % covariance. % The derivatives can only be computed for one of the following eight % covariance functions: cov{Matern|PP|RQ|SE}{iso|ard}. % % Copyright (c) by Hannes Nickisch, 2013-10-28. % % See also INFFITC.M, COVFITC.M. function Kp = cov_deriv_sq_dist(cov,hyp,x,z) if nargin<4, z = []; end % make sure, z exists xeqz = numel(z)==0; dg = strcmp(z,'diag') && numel(z)>0; % determine mode if iscell(cov), covstr = cov{1}; else covstr = cov; end if ~ischar(covstr), covstr = func2str(covstr); end if numel([strfind(covstr,'iso'),strfind(covstr,'ard')])==0 error('Only iso|ard covariances allowed for derivatives w.r.t. xu.') elseif numel([strfind(covstr,'covLIN'); strfind(covstr,'covGabor'); strfind(covstr,'covPER')])>0 error('Gabor|LIN|PER covariances not allowed for derivatives w.r.t. xu.') end [n,D] = size(x); if numel(strfind(covstr,'iso')), id = 1:D; else id = 1; end % *iso covariance ell1 = exp(hyp(1)); % first characteristic length scale Kp = feval(cov{:},hyp,x,z,1); % use derivative w.r.t. log(ell(1)) % precompute squared distances if dg % vector kxx d2 = zeros(n,1); else if xeqz % symmetric matrix Kxx d2 = sq_dist(x(:,id)'/ell1); else % cross covariances Kxz d2 = sq_dist(x(:,id)'/ell1,z(:,id)'/ell1); end end Kp = -1/2*Kp./d2; Kp(d2==0) = 0;gpml-matlab-v3.6-2015-07-07/util/make.m000644 000765 000024 00000004447 12301637047 017152 0ustar00carlstaff000000 000000 % For compilation under Windows/Octave you need to install the package % octavex.x-headers. % Under Linux/Octave, compilation works out of the box. % % There is a big variety of platforms running Matlab/Octave around and the % script is tested for: % % Linux 32bit / Matlab R2008b, 7.7 % Linux 64bit / Matlab R2010b, 7.11 % Windows 7 32bit / Matlab R14, 2004, 7.0 % Windows 7 64bit / Matlab R2011b, 7.13 % Windows XP 32bit / Matlab R2010b, 7.11 % OS X 10.7.5 / Matlab R2013b % OS X 10.9 / Matlab R2013b % % Linux 32bit / Octave 3.2.4 % OS X 10.9.1 / Octave 3.8.0 built with Homebrew % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2014-02-13. OCTAVE = exist('OCTAVE_VERSION') ~= 0; % check if we run Matlab or Octave me = mfilename; % what is my filename mydir = which(me); mydir = mydir(1:end-2-numel(me)); % where am I located cd(mydir) fprintf('Compiling solve_chol.c ...\n') if OCTAVE % Octave if ismac % Accelerate framework needed on OS X fprintf('mkoctfile --mex solve_chol.c "-Wl,-framework,Accelerate"\n') mkoctfile --mex solve_chol.c '-Wl,-framework,Accelerate' else fprintf('mkoctfile --mex solve_chol.c\n') mkoctfile --mex solve_chol.c end delete solve_chol.o else % Matlab if ispc % Windows try % take care of old Matlab versions cc = mex.getCompilerConfigurations; cc = lower(cc.Manufacturer);% compiler catch cc = 'lcc'; end if numel(strfind(computer,'64')) % 64 bit ospath = ['extern/lib/win64/',cc]; else % 32 bit ospath = ['extern/lib/win32/',cc]; end comp_cmd = 'mex -O solve_chol.c -output solve_chol'; fprintf('%s "../%s"\n',comp_cmd,[ospath,'/libmwlapack.lib']) eval([comp_cmd,' ''',matlabroot,'/',ospath,'/libmwlapack.lib','''']) else fprintf('mex -O -lmwlapack solve_chol.c\n') eval('mex -O -lmwlapack solve_chol.c') end end fprintf('Done!\n')gpml-matlab-v3.6-2015-07-07/util/logphi.m000644 000765 000024 00000004325 12550665722 017521 0ustar00carlstaff000000 000000 % Safe computation of logphi(z) = log(normcdf(z)) and its derivatives % dlogphi(z) = normpdf(x)/normcdf(x). % The function is based on index 5725 in Hart et al. and gsl_sf_log_erfc_e. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-11-13. function [lp,dlp,d2lp,d3lp] = logphi(z) z = real(z); % support for real arguments only lp = zeros(size(z)); % allocate memory id1 = z.*z<0.0492; % first case: close to zero lp0 = -z(id1)/sqrt(2*pi); c = [ 0.00048204; -0.00142906; 0.0013200243174; 0.0009461589032; -0.0045563339802; 0.00556964649138; 0.00125993961762116; -0.01621575378835404; 0.02629651521057465; -0.001829764677455021; 2*(1-pi/3); (4-pi)/3; 1; 1]; f = 0; for i=1:14, f = lp0.*(c(i)+f); end, lp(id1) = -2*f-log(2); id2 = z<-11.3137; % second case: very small r = [ 1.2753666447299659525; 5.019049726784267463450; 6.1602098531096305441; 7.409740605964741794425; 2.9788656263939928886 ]; q = [ 2.260528520767326969592; 9.3960340162350541504; 12.048951927855129036034; 17.081440747466004316; 9.608965327192787870698; 3.3690752069827527677 ]; num = 0.5641895835477550741; for i=1:5, num = -z(id2).*num/sqrt(2) + r(i); end den = 1.0; for i=1:6, den = -z(id2).*den/sqrt(2) + q(i); end e = num./den; lp(id2) = log(e/2) - z(id2).^2/2; id3 = ~id2 & ~id1; lp(id3) = log(erfc(-z(id3)/sqrt(2))/2); % third case: rest if nargout>1 % compute first derivative dlp = zeros(size(z)); % allocate memory dlp( id2) = abs(den./num) * sqrt(2/pi); % strictly positive first derivative dlp(~id2) = exp(-z(~id2).*z(~id2)/2-lp(~id2))/sqrt(2*pi); % safe computation if nargout>2 % compute second derivative d2lp = -dlp.*abs(z+dlp); % strictly negative second derivative if nargout>3 % compute third derivative d3lp = -d2lp.*abs(z+2*dlp)-dlp; % strictly positive third derivative end end endgpml-matlab-v3.6-2015-07-07/util/unwrap.m000644 000765 000024 00000001213 11422240325 017525 0ustar00carlstaff000000 000000 % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse rewrap.m. function v = unwrap(s) v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s))); % alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; unwrap(s{i})]; end end % other types are ignored gpml-matlab-v3.6-2015-07-07/util/minimize_v2.m000644 000765 000024 00000027260 12441263152 020460 0ustar00carlstaff000000 000000 % minimize.m - minimize a smooth differentiable multivariate function using % LBFGS (Limited memory LBFGS) or CG (Conjugate Gradients) % Usage: [X, fX, i] = minimize(X, F, p, other, ... ) % where % X is an initial guess (any type: vector, matrix, cell array, struct) % F is the objective function (function pointer or name) % p parameters - if p is a number, it corresponds to p.length below % p.length allowed 1) # linesearches or 2) if -ve minus # func evals % p.method minimization method, 'BFGS', 'LBFGS' or 'CG' % p.verbosity 0 quiet, 1 line, 2 line + warnings (default), 3 graphical % p.mem number of directions used in LBFGS (default 100) % other, ... other parameters, passed to the function F % X returned minimizer % fX vector of function values showing minimization progress % i final number of linesearches or function evaluations % The function F must take the following syntax [f, df] = F(X, other, ...) % where f is the function value and df its partial derivatives. The types of X % and df must be identical (vector, matrix, cell array, struct, etc). % % Copyright (C) 1996 - 2011 by Carl Edward Rasmussen, 2011-10-13. % Permission is hereby granted, free of charge, to any person OBTAINING A COPY % OF THIS SOFTWARE AND ASSOCIATED DOCUMENTATION FILES (THE "SOFTWARE"), TO DEAL % IN THE SOFTWARE WITHOUT RESTRICTION, INCLUDING WITHOUT LIMITATION THE RIGHTS % to use, copy, modify, merge, publish, distribute, sublicense, and/or sell % copies of the Software, and to permit persons to whom the Software is % furnished to do so, subject to the following conditions: % % The above copyright notice and this permission notice shall be included in % all copies or substantial portions of the Software. % % THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR % IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, % FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE % AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER % LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, % OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE % SOFTWARE. function [X, fX, i] = minimize(X, F, p, varargin) if isnumeric(p), p = struct('length', p); end % convert p to struct if p.length > 0, p.S = 'linesearch #'; else p.S = 'function evaluation #'; end; x = unwrap(X); % convert initial guess to vector if ~isfield(p,'method'), if length(x) > 1000, p.method = @LBFGS; else p.method = @BFGS; end; end % set default method if ~isfield(p,'verbosity'), p.verbosity = 2; end % default 1 line text output if ~isfield(p,'MFEPLS'), p.MFEPLS = 10; end % Max Func Evals Per Line Search if ~isfield(p,'MSR'), p.MSR = 100; end % Max Slope Ratio default f(F, X, varargin{:}); % set up the function f [fx, dfx] = f(x); % initial function value and derivatives if p.verbosity, printf('Initial Function Value %4.6e\r', fx); end if p.verbosity > 2, clf; subplot(211); hold on; xlabel(p.S); ylabel('function value'); plot(p.length < 0, fx, '+'); drawnow; end [x, fX, i] = feval(p.method, x, fx, dfx, p); % minimize using direction method X = rewrap(X, x); % convert answer to original representation if p.verbosity, printf('\n'); end function [x, fx, i] = CG(x0, fx0, dfx0, p) if ~isfield(p, 'SIG'), p.SIG = 0.1; end % default for line search quality i = p.length < 0; ok = 0; % initialize resource counter r = -dfx0; s = -r'*r; b = -1/(s-1); bs = -1; fx = fx0; % steepest descent while i < abs(p.length) b = b*bs/min(b*s,bs/p.MSR); % suitable initial step size using slope ratio [x, b, fx0, dfx, i] = lineSearch(x0, fx0, dfx0, r, s, b, i, p); if i < 0 % if line search failed i = -i; if ok, ok = 0; r = -dfx; else break; end % try steepest or stop else ok = 1; bs = b*s; % record step times slope (for slope ratio method) r = (dfx'*(dfx-dfx0))/(dfx0'*dfx0)*r - dfx; % Polack-Ribiere CG end s = r'*dfx; if s >= 0, r = -dfx; s = r'*dfx; ok = 0; end % slope must be -ve x0 = x; dfx0 = dfx; fx = [fx; fx0]; % replace old values with new ones end function [x, fx, i] = BFGS(x0, fx0, dfx0, p) if ~isfield(p, 'SIG'), p.SIG = 0.5; end % default for line search quality i = p.length < 0; ok = 0; % initialize resource counter x = x0; fx = fx0; r = -dfx0; s = -r'*r; b = -1/(s-1); H = eye(length(x0)); while i < abs(p.length) [x, b, fx0, dfx, i] = lineSearch(x0, fx0, dfx0, r, s, b, i, p); if i < 0 i = -i; if ok, ok = 0; else break; end; % try steepest or stop else ok = 1; t = x - x0; y = dfx - dfx0; ty = t'*y; Hy = H*y; H = H + (ty+y'*Hy)/ty^2*t*t' - 1/ty*Hy*t' - 1/ty*t*Hy'; % BFGS update end r = -H*dfx; s = r'*dfx; x0 = x; dfx0 = dfx; fx = [fx; fx0]; end function [x, fx, i] = LBFGS(x0, fx0, dfx0, p) if ~isfield(p, 'SIG'), p.SIG = 0.5; end % default for line search quality n = length(x0); k = 0; ok = 0; x = x0; fx = fx0; bs = -1/p.MSR; if isfield(p, 'mem'), m = p.mem; else m = min(100, n); end % set memory size a = zeros(1, m); t = zeros(n, m); y = zeros(n, m); % allocate memory i = p.length < 0; % initialize resource counter while i < abs(p.length) q = dfx0; for j = rem(k-1:-1:max(0,k-m),m)+1 a(j) = t(:,j)'*q/rho(j); q = q-a(j)*y(:,j); end if k == 0, r = -q/(q'*q); else r = -t(:,j)'*y(:,j)/(y(:,j)'*y(:,j))*q; end for j = rem(max(0,k-m):k-1,m)+1 r = r-t(:,j)*(a(j)+y(:,j)'*r/rho(j)); end s = r'*dfx0; if s >= 0, r = -dfx0; s = r'*dfx0; k = 0; ok = 0; end b = bs/min(bs,s/p.MSR); % suitable initial step size (usually 1) if isnan(r) | isinf(r) % if nonsense direction i = -i; % try steepest or stop else [x, b, fx0, dfx, i] = lineSearch(x0, fx0, dfx0, r, s, b, i, p); end if i < 0 % if line search failed i = -i; if ok, ok = 0; k = 0; else break; end % try steepest or stop else j = rem(k,m)+1; t(:,j) = x-x0; y(:,j) = dfx-dfx0; rho(j) = t(:,j)'*y(:,j); ok = 1; k = k+1; bs = b*s; end x0 = x; dfx0 = dfx; fx = [fx; fx0]; % replace and add values end function [x, a, fx, df, i] = lineSearch(x0, f0, df0, d, s, a, i, p) if p.length < 0, LIMIT = min(p.MFEPLS, -i-p.length); else LIMIT = p.MFEPLS; end p0.x = 0.0; p0.f = f0; p0.df = df0; p0.s = s; p1 = p0; % init p0 and p1 j = 0; p3.x = a; wp(p0, p.SIG, 0); % set step & Wolfe-Powell conditions if p.verbosity > 2 A = [-a a]/5; nd = norm(d); subplot(212); hold off; plot(0, f0, 'k+'); hold on; plot(nd*A, f0+s*A, 'k-'); xlabel('distance in line search direction'); ylabel('function value'); end while 1 % keep extrapolating as long as necessary ok = 0; while ~ok & j < LIMIT try % try, catch and bisect to safeguard extrapolation evaluation j = j+1; [p3.f p3.df] = f(x0+p3.x*d); p3.s = p3.df'*d; ok = 1; if isnan(p3.f+p3.s) | isinf(p3.f+p3.s) error('Objective function returned Inf or NaN',''); end; catch if p.verbosity > 1, printf('\n'); warning(lasterr); end % warn or silence p3.x = (p1.x+p3.x)/2; ok = 0; p3.f = NaN; % bisect, and retry end end if p.verbosity > 2 plot(nd*p3.x, p3.f, 'b+'); plot(nd*(p3.x+A), p3.f+p3.s*A, 'b-'); drawnow end if wp(p3) | j >= LIMIT, break; end % done? p0 = p1; p1 = p3; % move points back one unit p3.x = p0.x + minCubic(p1.x-p0.x, p1.f-p0.f, p0.s, p1.s, 1); % cubic extrap end while 1 % keep interpolating as long as necessary if p1.f > p3.f, p2 = p3; else p2 = p1; end % make p2 the best so far if wp(p2) > 1 | j >= LIMIT, break; end % done? p2.x = p1.x + minCubic(p3.x-p1.x, p3.f-p1.f, p1.s, p3.s, 0); % cubic interp j = j+1; [p2.f p2.df] = f(x0+p2.x*d); p2.s = p2.df'*d; if p.verbosity > 2 plot(nd*p2.x, p2.f, 'r+'); plot(nd*(p2.x+A), p2.f+p2.s*A, 'r'); drawnow end if wp(p2) > -1 & p2.s > 0 | wp(p2) < -1, p3 = p2; else p1 = p2; end % bracket end x = x0+p2.x*d; fx = p2.f; df = p2.df; a = p2.x; % return the value found if p.length < 0, i = i+j; else i = i+1; end % count func evals or line searches if p.verbosity, printf('%s %6i; value %4.6e\r', p.S, i, fx); end if wp(p2) < 2, i = -i; end % indicate faliure if p.verbosity > 2 if i>0, plot(norm(d)*p2.x, fx, 'go'); end subplot(211); plot(abs(i), fx, '+'); drawnow; end function z = minCubic(x, df, s0, s1, extr) % minimizer of approximating cubic INT = 0.1; EXT = 5.0; % interpolate and extrapolation limits A = -6*df+3*(s0+s1)*x; B = 3*df-(2*s0+s1)*x; if B<0, z = s0*x/(s0-s1); else z = -s0*x*x/(B+sqrt(B*B-A*s0*x)); end if extr % are we extrapolating? if ~isreal(z) | ~isfinite(z) | z < x | z > x*EXT, z = EXT*x; end % fix bad z z = max(z, (1+INT)*x); % extrapolate by at least INT else % else, we are interpolating if ~isreal(z) | ~isfinite(z) | z < 0 | z > x, z = x/2; end; % fix bad z z = min(max(z, INT*x), (1-INT)*x); % at least INT away from the boundaries end function y = wp(p, SIG, RHO) persistent a b c sig rho; if nargin == 3 % if three arguments, then set up the Wolfe-Powell conditions a = RHO*p.s; b = p.f; c = -SIG*p.s; sig = SIG; rho = RHO; y= 0; else if p.f > a*p.x+b % function value too large? if a > 0, y = -1; else y = -2; end else if p.s < -c, y = 0; elseif p.s > c, y = 1; else y = 2; end % if sig*abs(p.s) > c, a = rho*p.s; b = p.f-a*p.x; c = sig*abs(p.s); end end end function [fx, dfx] = f(varargin) persistent F p; if nargout == 0 p = varargin; if ischar(p{1}), F = str2func(p{1}); else F = p{1}; end else [fx, dfx] = F(rewrap(p{2}, varargin{1}), p{3:end}); dfx = unwrap(dfx); end function v = unwrap(s) % extract num elements of s (any type) into v (vector) v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s))); % alphabetize, conv to cell, recurse elseif iscell(s) % cell array elements are for i = 1:numel(s), v = [v; unwrap(s{i})]; end % handled sequentially end % other types are ignored function [s v] = rewrap(s, v) % map elements of v (vector) onto s (any type) if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed function printf(varargin) fprintf(varargin{:}); if exist('fflush','builtin'), fflush(stdout); end gpml-matlab-v3.6-2015-07-07/util/brentmin.m000644 000765 000024 00000007574 11422240325 020047 0ustar00carlstaff000000 000000 %% BRENTMIN: Brent's minimization method in one dimension function [xmin,fmin,funccount,varargout] = ... brentmin(xlow,xupp,Nitmax,tol,f,nout,varargin) % code taken from % § 10.2 Parabolic Interpolation and Brent's Method in One Dimension % Press, Teukolsky, Vetterling & Flannery % Numerical Recipes in C, Cambridge University Press, 2002 % % [xmin,fmin,funccout,varargout] = BRENTMIN(xlow,xupp,Nit,tol,f,nout,varargin) % Given a function f, and given a search interval this routine isolates % the minimum of fractional precision of about tol using Brent's method. % % INPUT % ----- % xlow,xupp: search interval such that xlow<=xmin<=xupp % Nitmax: maximum number of function evaluations made by the routine % tol: fractional precision % f: [y,varargout{:}] = f(x,varargin{:}) is the function % nout: no. of outputs of f (in varargout) in addition to the y value % % OUTPUT % ------ % fmin: minimal function value % xmin: corresponding abscissa-value % funccount: number of function evaluations made % varargout: additional outputs of f at optimum % % Copyright (c) by Hannes Nickisch 2010-01-10. if nargin<6, nout = 0; end varargout = cell(nout,1); % tolerance is no smaller than machine's floating point precision tol = max(tol,eps); % Evaluate endpoints fa = f(xlow,varargin{:}); fb = f(xupp,varargin{:}); funccount = 2; % number of function evaluations % Compute the start point seps = sqrt(eps); c = 0.5*(3.0 - sqrt(5.0));% golden ratio a = xlow; b = xupp; v = a + c*(b-a); w = v; xf = v; d = 0.0; e = 0.0; x = xf; [fx,varargout{:}] = f(x,varargin{:}); funccount = funccount + 1; fv = fx; fw = fx; xm = 0.5*(a+b); tol1 = seps*abs(xf) + tol/3.0; tol2 = 2.0*tol1; % Main loop while ( abs(xf-xm) > (tol2 - 0.5*(b-a)) ) gs = 1; % Is a parabolic fit possible if abs(e) > tol1 % Yes, so fit parabola gs = 0; r = (xf-w)*(fx-fv); q = (xf-v)*(fx-fw); p = (xf-v)*q-(xf-w)*r; q = 2.0*(q-r); if q > 0.0, p = -p; end q = abs(q); r = e; e = d; % Is the parabola acceptable if ( (abs(p)q*(a-xf)) && (p= xm, e = a-xf; else e = b-xf; end d = c*e; end % The function must not be evaluated too close to xf si = sign(d) + (d == 0); x = xf + si * max( abs(d), tol1 ); [fu,varargout{:}] = f(x,varargin{:}); funccount = funccount + 1; % Update a, b, v, w, x, xm, tol1, tol2 if fu <= fx if x >= xf, a = xf; else b = xf; end v = w; fv = fw; w = xf; fw = fx; xf = x; fx = fu; else % fu > fx if x < xf, a = x; else b = x; end if ( (fu <= fw) || (w == xf) ) v = w; fv = fw; w = x; fw = fu; elseif ( (fu <= fv) || (v == xf) || (v == w) ) v = x; fv = fu; end end xm = 0.5*(a+b); tol1 = seps*abs(xf) + tol/3.0; tol2 = 2.0*tol1; if funccount >= Nitmax % typically we should not get here % warning(sprintf(['Maximum number of iterations (%d) exceeded:', ... % 'precision is not guaranteed'],Nitmax)) % fprintf('[%1.3f,%1.3f,%1.3f]\n',xlow,xf,xupp) break end end % while % check that endpoints are less than the minimum found if ( (fa < fx) && (fa <= fb) ) xf = xlow; fx = fa; elseif fb < fx xf = xupp; fx = fb; end fmin = fx; xmin = xf; gpml-matlab-v3.6-2015-07-07/util/minimize.m000644 000765 000024 00000025667 12215541640 020062 0ustar00carlstaff000000 000000 function [X, fX, i] = minimize(X, f, length, varargin) % Minimize a differentiable multivariate function using conjugate gradients. % % Usage: [X, fX, i] = minimize(X, f, length, P1, P2, P3, ... ) % % X initial guess; may be of any type, including struct and cell array % f the name or pointer to the function to be minimized. The function % f must return two arguments, the value of the function, and it's % partial derivatives wrt the elements of X. The partial derivative % must have the same type as X. % length length of the run; if it is positive, it gives the maximum number of % line searches, if negative its absolute gives the maximum allowed % number of function evaluations. Optionally, length can have a second % component, which will indicate the reduction in function value to be % expected in the first line-search (defaults to 1.0). % P1, P2, ... parameters are passed to the function f. % % X the returned solution % fX vector of function values indicating progress made % i number of iterations (line searches or function evaluations, % depending on the sign of "length") used at termination. % % The function returns when either its length is up, or if no further progress % can be made (ie, we are at a (local) minimum, or so close that due to % numerical problems, we cannot get any closer). NOTE: If the function % terminates within a few iterations, it could be an indication that the % function values and derivatives are not consistent (ie, there may be a bug in % the implementation of your "f" function). % % The Polack-Ribiere flavour of conjugate gradients is used to compute search % directions, and a line search using quadratic and cubic polynomial % approximations and the Wolfe-Powell stopping criteria is used together with % the slope ratio method for guessing initial step sizes. Additionally a bunch % of checks are made to make sure that exploration is taking place and that % extrapolation will not be unboundedly large. % % See also: checkgrad % % Copyright (C) 2001 - 2010 by Carl Edward Rasmussen, 2010-01-03 INT = 0.1; % don't reevaluate within 0.1 of the limit of the current bracket EXT = 3.0; % extrapolate maximum 3 times the current step-size MAX = 20; % max 20 function evaluations per line search RATIO = 10; % maximum allowed slope ratio SIG = 0.1; RHO = SIG/2; % SIG and RHO are the constants controlling the Wolfe- % Powell conditions. SIG is the maximum allowed absolute ratio between % previous and new slopes (derivatives in the search direction), thus setting % SIG to low (positive) values forces higher precision in the line-searches. % RHO is the minimum allowed fraction of the expected (from the slope at the % initial point in the linesearch). Constants must satisfy 0 < RHO < SIG < 1. % Tuning of SIG (depending on the nature of the function to be optimized) may % speed up the minimization; it is probably not worth playing much with RHO. % The code falls naturally into 3 parts, after the initial line search is % started in the direction of steepest descent. 1) we first enter a while loop % which uses point 1 (p1) and (p2) to compute an extrapolation (p3), until we % have extrapolated far enough (Wolfe-Powell conditions). 2) if necessary, we % enter the second loop which takes p2, p3 and p4 chooses the subinterval % containing a (local) minimum, and interpolates it, unil an acceptable point % is found (Wolfe-Powell conditions). Note, that points are always maintained % in order p0 <= p1 <= p2 < p3 < p4. 3) compute a new search direction using % conjugate gradients (Polack-Ribiere flavour), or revert to steepest if there % was a problem in the previous line-search. Return the best value so far, if % two consecutive line-searches fail, or whenever we run out of function % evaluations or line-searches. During extrapolation, the "f" function may fail % either with an error or returning Nan or Inf, and minimize should handle this % gracefully. if max(size(length)) == 2, red=length(2); length=length(1); else red=1; end if length>0, S='Linesearch'; else S='Function evaluation'; end i = 0; % zero the run length counter ls_failed = 0; % no previous line search has failed [f0 df0] = feval(f, X, varargin{:}); % get function value and gradient Z = X; X = unwrap(X); df0 = unwrap(df0); fprintf('%s %6i; Value %4.6e\r', S, i, f0); if exist('fflush','builtin') fflush(stdout); end fX = f0; i = i + (length<0); % count epochs?! s = -df0; d0 = -s'*s; % initial search direction (steepest) and slope x3 = red/(1-d0); % initial step is red/(|s|+1) while i < abs(length) % while not finished i = i + (length>0); % count iterations?! X0 = X; F0 = f0; dF0 = df0; % make a copy of current values if length>0, M = MAX; else M = min(MAX, -length-i); end while 1 % keep extrapolating as long as necessary x2 = 0; f2 = f0; d2 = d0; f3 = f0; df3 = df0; success = 0; while ~success && M > 0 try M = M - 1; i = i + (length<0); % count epochs?! [f3 df3] = feval(f, rewrap(Z,X+x3*s), varargin{:}); df3 = unwrap(df3); if isnan(f3) || isinf(f3) || any(isnan(df3)+isinf(df3)), error(' '),end success = 1; catch % catch any error which occured in f x3 = (x2+x3)/2; % bisect and try again end end if f3 < F0, X0 = X+x3*s; F0 = f3; dF0 = df3; end % keep best values d3 = df3'*s; % new slope if d3 > SIG*d0 || f3 > f0+x3*RHO*d0 || M == 0 % are we done extrapolating? break end x1 = x2; f1 = f2; d1 = d2; % move point 2 to point 1 x2 = x3; f2 = f3; d2 = d3; % move point 3 to point 2 A = 6*(f1-f2)+3*(d2+d1)*(x2-x1); % make cubic extrapolation B = 3*(f2-f1)-(2*d1+d2)*(x2-x1); x3 = x1-d1*(x2-x1)^2/(B+sqrt(B*B-A*d1*(x2-x1))); % num. error possible, ok! if ~isreal(x3) || isnan(x3) || isinf(x3) || x3 < 0 % num prob | wrong sign? x3 = x2*EXT; % extrapolate maximum amount elseif x3 > x2*EXT % new point beyond extrapolation limit? x3 = x2*EXT; % extrapolate maximum amount elseif x3 < x2+INT*(x2-x1) % new point too close to previous point? x3 = x2+INT*(x2-x1); end end % end extrapolation while (abs(d3) > -SIG*d0 || f3 > f0+x3*RHO*d0) && M > 0 % keep interpolating if d3 > 0 || f3 > f0+x3*RHO*d0 % choose subinterval x4 = x3; f4 = f3; d4 = d3; % move point 3 to point 4 else x2 = x3; f2 = f3; d2 = d3; % move point 3 to point 2 end if f4 > f0 x3 = x2-(0.5*d2*(x4-x2)^2)/(f4-f2-d2*(x4-x2)); % quadratic interpolation else A = 6*(f2-f4)/(x4-x2)+3*(d4+d2); % cubic interpolation B = 3*(f4-f2)-(2*d2+d4)*(x4-x2); x3 = x2+(sqrt(B*B-A*d2*(x4-x2)^2)-B)/A; % num. error possible, ok! end if isnan(x3) || isinf(x3) x3 = (x2+x4)/2; % if we had a numerical problem then bisect end x3 = max(min(x3, x4-INT*(x4-x2)),x2+INT*(x4-x2)); % don't accept too close [f3 df3] = feval(f, rewrap(Z,X+x3*s), varargin{:}); df3 = unwrap(df3); if f3 < F0, X0 = X+x3*s; F0 = f3; dF0 = df3; end % keep best values M = M - 1; i = i + (length<0); % count epochs?! d3 = df3'*s; % new slope end % end interpolation if abs(d3) < -SIG*d0 && f3 < f0+x3*RHO*d0 % if line search succeeded X = X+x3*s; f0 = f3; fX = [fX' f0]'; % update variables fprintf('%s %6i; Value %4.6e\r', S, i, f0); if exist('fflush','builtin') fflush(stdout); end s = (df3'*df3-df0'*df3)/(df0'*df0)*s - df3; % Polack-Ribiere CG direction df0 = df3; % swap derivatives d3 = d0; d0 = df0'*s; if d0 > 0 % new slope must be negative s = -df0; d0 = -s'*s; % otherwise use steepest direction end x3 = x3 * min(RATIO, d3/(d0-realmin)); % slope ratio but max RATIO ls_failed = 0; % this line search did not fail else X = X0; f0 = F0; df0 = dF0; % restore best point so far if ls_failed || i > abs(length) % line search failed twice in a row break; % or we ran out of time, so we give up end s = -df0; d0 = -s'*s; % try steepest x3 = 1/(1-d0); ls_failed = 1; % this line search failed end end X = rewrap(Z,X); fprintf('\n'); if exist('fflush','builtin') fflush(stdout); end function v = unwrap(s) % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse rewrap.m. v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s))); % alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; unwrap(s{i})]; end end % other types are ignored function [s v] = rewrap(s, v) % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also unwrap.m. if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed gpml-matlab-v3.6-2015-07-07/util/minimize_lbfgsb_objfun.m000644 000765 000024 00000005207 11422240325 022723 0ustar00carlstaff000000 000000 function y = minimize_lbfgsb_objfun(X,varargin) % extract input arguments varargin = varargin{1}; strctX = varargin{2}; f = varargin{1}; % global variables serve as communication interface between calls global minimize_lbfgsb_iteration_number global minimize_lbfgsb_objective global minimize_lbfgsb_gradient global minimize_lbfgsb_X recompute = 0; if minimize_lbfgsb_iteration_number==0 recompute = 1; else if norm(unwrap(X)-unwrap(minimize_lbfgsb_X))>1e-10 recompute = 1; end end if recompute [y,G] = feval(f,rewrap(strctX,X),varargin{3:end}); else y = minimize_lbfgsb_objective(minimize_lbfgsb_iteration_number); G = minimize_lbfgsb_gradient; end % increase global counter, memorise objective function, gradient and position minimize_lbfgsb_iteration_number = minimize_lbfgsb_iteration_number+1; minimize_lbfgsb_objective(minimize_lbfgsb_iteration_number,1) = y; minimize_lbfgsb_gradient = G; minimize_lbfgsb_X = X; % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse rewrap.m. function v = unwrap(s) v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s)));% alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; unwrap(s{i})]; end end % other types are ignored % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also unwrap.m. function [s v] = rewrap(s, v) if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed gpml-matlab-v3.6-2015-07-07/util/gauher.m000644 000765 000024 00000004305 12076017514 017501 0ustar00carlstaff000000 000000 % compute abscissas and weight factors for Gaussian-Hermite quadrature % % CALL: [x,w] = gauher(N) % % x = base points (abscissas) % w = weight factors % N = number of base points (abscissas) (integrates an up to (2N-1)th order % polynomial exactly) % % p(x)=exp(-x^2/2)/sqrt(2*pi), a =-Inf, b = Inf % % The Gaussian Quadrature integrates a (2n-1)th order % polynomial exactly and the integral is of the form % b N % Int ( p(x)* F(x) ) dx = Sum ( w_j* F( x_j ) ) % a j=1 % % this procedure uses the coefficients a(j), b(j) of the % recurrence relation % % b p (x) = (x - a ) p (x) - b p (x) % j j j j-1 j-1 j-2 % % for the various classical (normalized) orthogonal polynomials, % and the zero-th moment % % 1 = integral w(x) dx % % of the given polynomial's weight function w(x). Since the % polynomials are orthonormalized, the tridiagonal matrix is % guaranteed to be symmetric. function [x,w]=gauher(N) if N==20 % return precalculated values x=[ -7.619048541679757;-6.510590157013656;-5.578738805893203; -4.734581334046057;-3.943967350657318;-3.18901481655339 ; -2.458663611172367;-1.745247320814127;-1.042945348802751; -0.346964157081356; 0.346964157081356; 1.042945348802751; 1.745247320814127; 2.458663611172367; 3.18901481655339 ; 3.943967350657316; 4.734581334046057; 5.578738805893202; 6.510590157013653; 7.619048541679757]; w=[ 0.000000000000126; 0.000000000248206; 0.000000061274903; 0.00000440212109 ; 0.000128826279962; 0.00183010313108 ; 0.013997837447101; 0.061506372063977; 0.161739333984 ; 0.260793063449555; 0.260793063449555; 0.161739333984 ; 0.061506372063977; 0.013997837447101; 0.00183010313108 ; 0.000128826279962; 0.00000440212109 ; 0.000000061274903; 0.000000000248206; 0.000000000000126 ]; else b = sqrt( (1:N-1)/2 )'; [V,D] = eig( diag(b,1) + diag(b,-1) ); w = V(1,:)'.^2; x = sqrt(2)*diag(D); end gpml-matlab-v3.6-2015-07-07/util/minimize_v1.m000644 000765 000024 00000025702 12441263152 020456 0ustar00carlstaff000000 000000 function [X, fX, i] = minimize_old(X, f, length, varargin) % Minimize a differentiable multivariate function using conjugate gradients. % % Usage: [X, fX, i] = minimize(X, f, length, P1, P2, P3, ... ) % % X initial guess; may be of any type, including struct and cell array % f the name or pointer to the function to be minimized. The function % f must return two arguments, the value of the function, and it's % partial derivatives wrt the elements of X. The partial derivative % must have the same type as X. % length length of the run; if it is positive, it gives the maximum number of % line searches, if negative its absolute gives the maximum allowed % number of function evaluations. Optionally, length can have a second % component, which will indicate the reduction in function value to be % expected in the first line-search (defaults to 1.0). % P1, P2, ... parameters are passed to the function f. % % X the returned solution % fX vector of function values indicating progress made % i number of iterations (line searches or function evaluations, % depending on the sign of "length") used at termination. % % The function returns when either its length is up, or if no further progress % can be made (ie, we are at a (local) minimum, or so close that due to % numerical problems, we cannot get any closer). NOTE: If the function % terminates within a few iterations, it could be an indication that the % function values and derivatives are not consistent (ie, there may be a bug in % the implementation of your "f" function). % % The Polack-Ribiere flavour of conjugate gradients is used to compute search % directions, and a line search using quadratic and cubic polynomial % approximations and the Wolfe-Powell stopping criteria is used together with % the slope ratio method for guessing initial step sizes. Additionally a bunch % of checks are made to make sure that exploration is taking place and that % extrapolation will not be unboundedly large. % % See also: checkgrad % % Copyright (C) 2001 - 2010 by Carl Edward Rasmussen, 2010-01-03 INT = 0.1; % don't reevaluate within 0.1 of the limit of the current bracket EXT = 3.0; % extrapolate maximum 3 times the current step-size MAX = 20; % max 20 function evaluations per line search RATIO = 10; % maximum allowed slope ratio SIG = 0.1; RHO = SIG/2; % SIG and RHO are the constants controlling the Wolfe- % Powell conditions. SIG is the maximum allowed absolute ratio between % previous and new slopes (derivatives in the search direction), thus setting % SIG to low (positive) values forces higher precision in the line-searches. % RHO is the minimum allowed fraction of the expected (from the slope at the % initial point in the linesearch). Constants must satisfy 0 < RHO < SIG < 1. % Tuning of SIG (depending on the nature of the function to be optimized) may % speed up the minimization; it is probably not worth playing much with RHO. % The code falls naturally into 3 parts, after the initial line search is % started in the direction of steepest descent. 1) we first enter a while loop % which uses point 1 (p1) and (p2) to compute an extrapolation (p3), until we % have extrapolated far enough (Wolfe-Powell conditions). 2) if necessary, we % enter the second loop which takes p2, p3 and p4 chooses the subinterval % containing a (local) minimum, and interpolates it, unil an acceptable point % is found (Wolfe-Powell conditions). Note, that points are always maintained % in order p0 <= p1 <= p2 < p3 < p4. 3) compute a new search direction using % conjugate gradients (Polack-Ribiere flavour), or revert to steepest if there % was a problem in the previous line-search. Return the best value so far, if % two consecutive line-searches fail, or whenever we run out of function % evaluations or line-searches. During extrapolation, the "f" function may fail % either with an error or returning Nan or Inf, and minimize should handle this % gracefully. if max(size(length)) == 2, red=length(2); length=length(1); else red=1; end if length>0, S='Linesearch'; else S='Function evaluation'; end i = 0; % zero the run length counter ls_failed = 0; % no previous line search has failed [f0 df0] = feval(f, X, varargin{:}); % get function value and gradient Z = X; X = unwrap(X); df0 = unwrap(df0); fprintf('%s %6i; Value %4.6e\r', S, i, f0); if exist('fflush','builtin') fflush(stdout); end fX = f0; i = i + (length<0); % count epochs?! s = -df0; d0 = -s'*s; % initial search direction (steepest) and slope x3 = red/(1-d0); % initial step is red/(|s|+1) while i < abs(length) % while not finished i = i + (length>0); % count iterations?! X0 = X; F0 = f0; dF0 = df0; % make a copy of current values if length>0, M = MAX; else M = min(MAX, -length-i); end while 1 % keep extrapolating as long as necessary x2 = 0; f2 = f0; d2 = d0; f3 = f0; df3 = df0; success = 0; while ~success && M > 0 try M = M - 1; i = i + (length<0); % count epochs?! [f3 df3] = feval(f, rewrap(Z,X+x3*s), varargin{:}); df3 = unwrap(df3); if isnan(f3) || isinf(f3) || any(isnan(df3)+isinf(df3)), error(''), end success = 1; catch % catch any error which occured in f x3 = (x2+x3)/2; % bisect and try again end end if f3 < F0, X0 = X+x3*s; F0 = f3; dF0 = df3; end % keep best values d3 = df3'*s; % new slope if d3 > SIG*d0 || f3 > f0+x3*RHO*d0 || M == 0 % are we done extrapolating? break end x1 = x2; f1 = f2; d1 = d2; % move point 2 to point 1 x2 = x3; f2 = f3; d2 = d3; % move point 3 to point 2 A = 6*(f1-f2)+3*(d2+d1)*(x2-x1); % make cubic extrapolation B = 3*(f2-f1)-(2*d1+d2)*(x2-x1); x3 = x1-d1*(x2-x1)^2/(B+sqrt(B*B-A*d1*(x2-x1))); % num. error possible, ok! if ~isreal(x3) || isnan(x3) || isinf(x3) || x3 < 0 % num prob | wrong sign? x3 = x2*EXT; % extrapolate maximum amount elseif x3 > x2*EXT % new point beyond extrapolation limit? x3 = x2*EXT; % extrapolate maximum amount elseif x3 < x2+INT*(x2-x1) % new point too close to previous point? x3 = x2+INT*(x2-x1); end end % end extrapolation while (abs(d3) > -SIG*d0 || f3 > f0+x3*RHO*d0) && M > 0 % keep interpolating if d3 > 0 || f3 > f0+x3*RHO*d0 % choose subinterval x4 = x3; f4 = f3; d4 = d3; % move point 3 to point 4 else x2 = x3; f2 = f3; d2 = d3; % move point 3 to point 2 end if f4 > f0 x3 = x2-(0.5*d2*(x4-x2)^2)/(f4-f2-d2*(x4-x2)); % quadratic interpolation else A = 6*(f2-f4)/(x4-x2)+3*(d4+d2); % cubic interpolation B = 3*(f4-f2)-(2*d2+d4)*(x4-x2); x3 = x2+(sqrt(B*B-A*d2*(x4-x2)^2)-B)/A; % num. error possible, ok! end if isnan(x3) || isinf(x3) x3 = (x2+x4)/2; % if we had a numerical problem then bisect end x3 = max(min(x3, x4-INT*(x4-x2)),x2+INT*(x4-x2)); % don't accept too close [f3 df3] = feval(f, rewrap(Z,X+x3*s), varargin{:}); df3 = unwrap(df3); if f3 < F0, X0 = X+x3*s; F0 = f3; dF0 = df3; end % keep best values M = M - 1; i = i + (length<0); % count epochs?! d3 = df3'*s; % new slope end % end interpolation if abs(d3) < -SIG*d0 && f3 < f0+x3*RHO*d0 % if line search succeeded X = X+x3*s; f0 = f3; fX = [fX' f0]'; % update variables fprintf('%s %6i; Value %4.6e\r', S, i, f0); if exist('fflush','builtin') fflush(stdout); end s = (df3'*df3-df0'*df3)/(df0'*df0)*s - df3; % Polack-Ribiere CG direction df0 = df3; % swap derivatives d3 = d0; d0 = df0'*s; if d0 > 0 % new slope must be negative s = -df0; d0 = -s'*s; % otherwise use steepest direction end x3 = x3 * min(RATIO, d3/(d0-realmin)); % slope ratio but max RATIO ls_failed = 0; % this line search did not fail else X = X0; f0 = F0; df0 = dF0; % restore best point so far if ls_failed || i > abs(length) % line search failed twice in a row break; % or we ran out of time, so we give up end s = -df0; d0 = -s'*s; % try steepest x3 = 1/(1-d0); ls_failed = 1; % this line search failed end end X = rewrap(Z,X); fprintf('\n'); if exist('fflush','builtin') fflush(stdout); end % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse rewrap.m. function v = unwrap(s) v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s))); % alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; unwrap(s{i})]; end end % other types are ignored % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also unwrap.m. function [s v] = rewrap(s, v) if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed gpml-matlab-v3.6-2015-07-07/util/rewrap.m000644 000765 000024 00000001766 11422240325 017526 0ustar00carlstaff000000 000000 % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also unwrap.m. function [s v] = rewrap(s, v) if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed gpml-matlab-v3.6-2015-07-07/util/lik_epquad.m000644 000765 000024 00000003126 12230176225 020341 0ustar00carlstaff000000 000000 % Compute infEP part of a likelihood function based on the infLaplace part using % Gaussian-Hermite quadrature. % % The function is used in GLM likelihoods such as likPoisson, likGamma, likBeta % and likInvGauss. % % Copyright (c) by Hannes Nickisch, 2013-10-16. function varargout = lik_epquad(lik,hyp,y,mu,s2) n = max([length(y),length(mu),length(s2)]); on = ones(n,1); N = 20; [t,w] = gauher(N); oN = ones(1,N); lw = ones(n,1)*log(w'); y = y(:).*on; mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only [lpi,dlpi,d2lpi] = feval(lik{:},hyp,y*oN,sig*t'+mu*oN,[],'infLaplace'); lZ = s(lpi+lw); dlZ = {}; d2lZ = {}; if nargout>1 % 1st derivative wrt mean % Using p*dlp=dp, p=exp(lp), Z=sum_i wi*pi, dZ = sum_i wi*dpi we obtain % dlZ = sum_i exp(lpi-lZ+lwi)*dlpi = sum_i ai*dlpi. a = exp(lpi - lZ*oN + lw); dlZ = sum(a.*dlpi,2); if nargout>2 % 2nd derivative wrt mean % Using d2lZ=(d2Z*Z-dZ^2)/Z^2 <=> d2Z=Z*(d2lZ+dlZ^2) and % d2Z = sum_i wi*d2Zi, we get d2lZ = sum_i ai*(d2lpi+dlpi^2)-dlZ^2. d2lZ = sum(a.*(d2lpi+dlpi.*dlpi),2) - dlZ.*dlZ; end end varargout = {lZ,dlZ,d2lZ}; % computes y = log( sum(exp(x),2) ), the softmax in a numerically safe way by % subtracting the row maximum to avoid cancelation after taking the exp % the sum is done along the rows function [y,x] = s(logx) N = size(logx,2); max_logx = max(logx,[],2); % we have all values in the log domain, and want to calculate a sum x = exp(logx-max_logx*ones(1,N)); y = log(sum(x,2)) + max_logx;gpml-matlab-v3.6-2015-07-07/util/binaryGP.m000644 000765 000024 00000011547 11422240325 017737 0ustar00carlstaff000000 000000 function varargout = binaryGP(hyper, approx, covfunc, lik, x, y, xstar) % Approximate binary Gaussian Process classification. Two modes are possible: % training or testing: if no test cases are supplied, then the approximate % negative log marginal likelihood and its partial derivatives wrt the % hyperparameters is computed; this mode is used to fit the hyperparameters. If % test cases are given, then the test set predictive probabilities are % returned. Exact inference is intractible, the function uses a specified % approximation method (see approximations.m), flexible covariance functions % (see covFunctions.m) and likelihood functions (see likelihoods.m). % % usage: [nlZ, dnlZ ] = binaryGP(hyper, approx, covfunc, lik, x, y); % or: [p,mu,s2,nlZ] = binaryGP(hyper, approx, covfunc, lik, x, y, xstar); % % where: % % hyper is a column vector of hyperparameters % approx is a function specifying an approximation method for inference % covfunc is the name of the covariance function (see below) % lik is the name of the likelihood function % x is a n by D matrix of training inputs % y is a (column) vector (of size n) of binary +1/-1 targets % xstar is a nn by D matrix of test inputs % nlZ is the returned value of the negative log marginal likelihood % dnlZ is a (column) vector of partial derivatives of the negative % log marginal likelihood wrt each hyperparameter % p is a (column) vector (of length nn) of predictive probabilities % mu is a (column) vector (of length nn) of predictive latent means % s2 is a (column) vector (of length nn) of predictive latent variances % % The length of the vector of hyperparameters depends on the covariance % function, as specified by the "covfunc" input to the function, specifying the % name of a covariance function. A number of different covariance function are % implemented, and it is not difficult to add new ones. See covFunctions.m for % the details. % % The "lik" input argument specifies the name of the likelihood function (see % likelihoods.m). % % The "approx" input argument to the function specifies an approximation method % (see approximations.m). An approximation method returns a representation of % the approximate Gaussian posterior. Usually, the approximate posterior admits % the form N(m=K*alpha, V=inv(inv(K)+W)), where alpha is a vector and W is % diagonal. The approximation method returns: % % alpha is a (sparse or full column vector) containing inv(K)*m, where K % is the prior covariance matrix and m the approx posterior mean % sW is a (sparse or full column) vector containing diagonal of sqrt(W) % the approximate posterior covariance matrix is inv(inv(K)+W) % L is a (sparse or full) matrix, L = chol(sW*K*sW+eye(n)) % % In cases where the approximate posterior variance does not admit the form % V=inv(inv(K)+W) with diagonal W, L contains instead -inv(K+inv(W)), and sW % is unused. % % The alpha parameter may be sparse. In that case sW and L can either be sparse % or full (retaining only the non-zero rows and columns, as indicated by the % sparsity structure of alpha). The L paramter is allowed to be empty, in % which case it will be computed. % % The function can conveniently be used with the "minimize" function to train % a Gaussian Process, eg: % % [hyper, fX, i] = minimize(hyper, 'binaryGP', length, 'approxEP', 'covSEiso', 'logistic', x, y); % % where "length" gives the length of the run: if it is positive, it gives the % maximum number of line searches, if negative its absolute gives the maximum % allowed number of function evaluations. % % Copyright (c) 2007 Carl Edward Rasmussen and Hannes Nickisch, 2010-07-21. if nargin<6 || nargin>7 disp('Usage: [nlZ, dnlZ ] = binaryGP(hyper,approx,covfunc,lik,x,y);') disp(' or: [p,mu,s2,nlZ] = binaryGP(hyper,approx,covfunc,lik,x,y,xstar);') return end % Note, this function is just a wrapper provided for backward compatibility, % the functionality is now provided by the more general gp function. if strcmp(lik,'logistic') lik = @likLogistic; elseif strcmpi(lik,'cumgauss') lik = @likErf; else error('Allowable likelihoods: logistic and cumGauss.'); end if strcmp(approx,'approxLA') inf = @infLaplace; elseif strcmp(approx,'approxEP') inf = @infEP; else error('Allowable approximations: approxLA and approxEP.'); end mean = @meanZero; varargout = cell(nargout, 1); % allocate the right number of output arguments hyp.cov = hyper; if nargin==6 [varargout{:}] = gp(hyp,inf,mean,covfunc,lik,x,y); if nargout>1, varargout{2} = varargout{2}.cov; end else [ymu,ys2,fmu,fs2] = gp(hyp,inf,mean,covfunc,lik,x,y,xstar); if nargout>0, varargout{1} = (1+ymu)/2; end if nargout>1, varargout{2} = fmu; end if nargout>2, varargout{3} = fs2; end if nargout>3, varargout{4} = gp(hyp,inf,mean,covfunc,lik,x,y); end % nlZ endgpml-matlab-v3.6-2015-07-07/util/elsympol.m000644 000765 000024 00000001273 11422240325 020063 0ustar00carlstaff000000 000000 % Evaluate the order R elementary symmetric polynomial Newton's identity aka % the Newton–Girard formulae: http://en.wikipedia.org/wiki/Newton's_identities % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-01-10. function E = elsympol(Z,R) % evaluate 'power sums' of the individual terms in Z sz = size(Z); P = zeros([sz(1:2),R]); for r=1:R, P(:,:,r) = sum(Z.^r,3); end E = zeros([sz(1:2),R+1]); % E(:,:,r+1) yields polynomial r E(:,:,1) = ones(sz(1:2)); if R==0, return, end % init recursion E(:,:,2) = P(:,:,1); if R==1, return, end % init recursion for r=2:R for i=1:r E(:,:,r+1) = E(:,:,r+1) + P(:,:,i).*E(:,:,r+1-i)*(-1)^(i-1)/r; end end gpml-matlab-v3.6-2015-07-07/util/glm_invlink_logistic.m000644 000765 000024 00000001256 12230176225 022433 0ustar00carlstaff000000 000000 % Compute the log intensity for the inverse link function g(f) = log(1+exp(f))). % % The function is used in GLM likelihoods such as likPoisson, likGamma, likBeta % and likInvGauss. % % Copyright (c) by Hannes Nickisch, 2013-10-16. function [lg,dlg,d2lg,d3lg] = glm_invlink_logistic(f) l1pef = max(0,f) + log(1+exp(-abs(f))); % safely compute log(1+exp(f)) lg = log(l1pef); id = f<-15; lg(id) = f(id); % fix log(log(1+exp(f))) limits if nargout>1 sm = 1./(1+exp(-f)); dlg = sm./l1pef; dlg(f<-15) = 1; if nargout>2 sp = 1./(1+exp(f)); d2lg = dlg.*(sp-dlg); if nargout>2 d3lg = d2lg.*(sp-2*dlg) - dlg.*sp.*sm; end end endgpml-matlab-v3.6-2015-07-07/util/lbfgsb/matlabscalar.cpp000644 000765 000024 00000001730 11422240325 022430 0ustar00carlstaff000000 000000 #include "matlabscalar.h" #include "matlabexception.h" // Function definitions. // ----------------------------------------------------------------- double& getMatlabScalar (const mxArray* ptr) { if (!mxIsDouble(ptr)) throw MatlabException("Matlab array must be of type double"); if (mxGetNumberOfElements(ptr) != 1) throw MatlabException("The Matlab array must be a scalar"); return *mxGetPr(ptr); } double& createMatlabScalar (mxArray*& ptr) { ptr = mxCreateDoubleScalar(0); return *mxGetPr(ptr); } // Function definitions for class MatlabScalar. // --------------------------------------------------------------- MatlabScalar::MatlabScalar (const mxArray* ptr) : x(getMatlabScalar(ptr)) { } MatlabScalar::MatlabScalar (mxArray*& ptr, double value) : x(createMatlabScalar(ptr)) { x = value; } MatlabScalar::MatlabScalar (MatlabScalar& source) : x(source.x) { } MatlabScalar& MatlabScalar::operator= (double value) { x = value; return *this; } gpml-matlab-v3.6-2015-07-07/util/lbfgsb/program.cpp000644 000765 000024 00000007675 12240130516 021466 0ustar00carlstaff000000 000000 #include "program.h" #include // Function definitions. // ----------------------------------------------------------------- // Copy a C-style string (a null-terminated character array) to a // non-C-style string (a simple character array). The length of the // destination character array is given by "ndest". If the source is // shorter than the destination, the destination is padded with blank // characters. void copyCStrToCharArray (const char* source, char* dest, int ndest) { // Get the length of the source C string. int nsource = strlen(source); // Only perform the copy if the source can fit into the destination. if (nsource < ndest) { // Copy the string. strcpy(dest,source); // Fill in the rest of the string with blanks. for (int i = nsource; i < ndest; i++) dest[i] = ' '; } } // Return true if the two strings are the same. The second input // argument must be a C-style string (a null-terminated character // array), but the first input argument need not be one. We only // compare the two strings up to the length of "cstr". bool strIsEqualToCStr (const char* str, const char* cstr) { // Get the length of the C string. int n = strlen(cstr); return !strncmp(str,cstr,n); } // Function definitions for class Program. // ----------------------------------------------------------------- Program::Program (int n, double* x, double* lb, double* ub, int* btype, int m, int maxiter, double factr, double pgtol) { this->n = n; this->x = x; this->lb = lb; this->ub = ub; this->btype = btype; this->m = m; this->maxiter = maxiter; this->factr = factr; this->pgtol = pgtol; this->iprint = defaultprintlevel; this->owner = false; initStructures(); } Program::Program (int n, int m, int maxiter, double factr, double pgtol) { this->n = n; this->x = new double[n]; this->lb = new double[n]; this->ub = new double[n]; this->btype = new int[n]; this->m = m; this->maxiter = maxiter; this->factr = factr; this->pgtol = pgtol; this->iprint = defaultprintlevel; this->owner = true; initStructures(); } Program::~Program() { if (owner) { delete[] x; delete[] lb; delete[] ub; delete[] btype; } delete[] g; delete[] wa; delete[] iwa; } SolverExitStatus Program::runSolver() { SolverExitStatus status = success; // The return value. // Initialize the objective function and gradient to zero. f = 0; for (int i = 0; i < n; i++) g[i] = 0; // This initial call sets up the structures for L-BFGS. callLBFGS("START"); // Repeat until we've reached the maximum number of iterations. int t = 0; while (true) { // Do something according to the "task" from the previous call to // L-BFGS. if (strIsEqualToCStr(task,"FG")) { // Evaluate the objective function and the gradient of the // objective at the current point. f = computeObjective(n,x); computeGradient(n,x,g); } else if (strIsEqualToCStr(task,"NEW_X")) { // Go to the next iteration and call the iterative callback // routine. t++; iterCallback(t,x,f); // If we've reached the maximum number of iterations, terminate // the optimization. if (t == maxiter) { callLBFGS("STOP"); break; } } else if (strIsEqualToCStr(task,"CONV")) break; else if (strIsEqualToCStr(task,"ABNO")) { status = abnormalTermination; break; } else if (strIsEqualToCStr(task,"ERROR")) { status = errorOnInput; break; } // Call L-BFGS again. callLBFGS(); } return status; } void Program::callLBFGS (const char* cmd) { if (cmd) copyCStrToCharArray(cmd,task,60); setulb_(&n,&m,x,lb,ub,btype,&f,g,&factr,&pgtol,wa,iwa,task,&iprint, csave,lsave,isave,dsave); } void Program::initStructures() { f = 0; g = new double[n]; wa = new double[(2*m + 5)*n + 12*m*(m + 1)]; iwa = new int[3*n]; } gpml-matlab-v3.6-2015-07-07/util/lbfgsb/matlabstring.cpp000644 000765 000024 00000002032 12313000560 022460 0ustar00carlstaff000000 000000 #include #include "matlabstring.h" #include "matlabexception.h" // Function definitions. // ----------------------------------------------------------------- char* copystring (const char* source) { int n = strlen(source); // The length of the string. char* dest = new char[n+1]; // The return value. strcpy(dest,source); return dest; } // Function definitions for class MatlabString. // ----------------------------------------------------------------- MatlabString::MatlabString (const mxArray* ptr) { s = 0; // Check to make sure the Matlab array is a string. if (!mxIsChar(ptr)) throw MatlabException("Matlab array must be a string (of type CHAR)"); // Get the string passed as a Matlab array. s = mxArrayToString(ptr); if (s == 0) throw MatlabException("Unable to obtain string from Matlab array"); } MatlabString::MatlabString (const MatlabString& source) { s = 0; // Copy the source string. s = copystring(source.s); } MatlabString::~MatlabString() { if (s) mxFree(s); } gpml-matlab-v3.6-2015-07-07/util/lbfgsb/LICENSE000644 000765 000024 00000077330 12343310245 020316 0ustar00carlstaff000000 000000 GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007 Copyright (C) 2007 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The GNU General Public License is a free, copyleft license for software and other kinds of works. The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. We, the Free Software Foundation, use the GNU General Public License for most of our software; it applies also to any other work released this way by its authors. You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others. For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. Developers that use the GNU GPL protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License giving you legal permission to copy, distribute and/or modify it. For the developers' and authors' protection, the GPL clearly explains that there is no warranty for this free software. For both users' and authors' sake, the GPL requires that modified versions be marked as changed, so that their problems will not be attributed erroneously to authors of previous versions. Some devices are designed to deny users access to install or run modified versions of the software inside them, although the manufacturer can do so. This is fundamentally incompatible with the aim of protecting users' freedom to change the software. The systematic pattern of such abuse occurs in the area of products for individuals to use, which is precisely where it is most unacceptable. Therefore, we have designed this version of the GPL to prohibit the practice for those products. If such problems arise substantially in other domains, we stand ready to extend this provision to those domains in future versions of the GPL, as needed to protect the freedom of users. Finally, every program is threatened constantly by software patents. States should not allow patents to restrict development and use of software on general-purpose computers, but in those that do, we wish to avoid the special danger that patents applied to a free program could make it effectively proprietary. To prevent this, the GPL assures that patents cannot be used to render the program non-free. The precise terms and conditions for copying, distribution and modification follow. TERMS AND CONDITIONS 0. Definitions. "This License" refers to version 3 of the GNU General Public License. "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. A "covered work" means either the unmodified Program or a work based on the Program. To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 1. Source Code. The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. The Corresponding Source for a work in source code form is that same work. 2. Basic Permissions. All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 3. Protecting Users' Legal Rights From Anti-Circumvention Law. No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 4. Conveying Verbatim Copies. You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 5. Conveying Modified Source Versions. You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 6. Conveying Non-Source Forms. You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. "Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. 7. Additional Terms. "Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or d) Limiting the use for publicity purposes of names of licensors or authors of the material; or e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. 8. Termination. You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. 9. Acceptance Not Required for Having Copies. You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. 10. Automatic Licensing of Downstream Recipients. Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. 11. Patents. A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. 12. No Surrender of Others' Freedom. If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. 13. Use with the GNU Affero General Public License. Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU Affero General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such. 14. Revised Versions of this License. The Free Software Foundation may publish revised and/or new versions of the GNU General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU General Public License, you may choose any version ever published by the Free Software Foundation. If the Program specifies that a proxy can decide which future versions of the GNU General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. 15. Disclaimer of Warranty. THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. Limitation of Liability. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 17. Interpretation of Sections 15 and 16. If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. END OF TERMS AND CONDITIONS gpml-matlab-v3.6-2015-07-07/util/lbfgsb/matlabexception.h000644 000765 000024 00000001545 11422240325 022632 0ustar00carlstaff000000 000000 #ifndef INCLUDE_MATLABEXCEPTION #define INCLUDE_MATLABEXCEPTION #include // Class MatlabException // ----------------------------------------------------------------- // This class just makes it easier for me to throw exceptions. Its // functionality really has nothing to do with MATLAB. class MatlabException : public std::exception { public: MatlabException (const char* message) throw(); ~MatlabException() throw() { }; // The copy constructor makes a shallow copy. MatlabException (const MatlabException& source) throw(); // The copy assignment operator makes a shallow copy as well. MatlabException& operator= (const MatlabException& source); // Return the message string. virtual const char* what () const throw() { return message; }; private: const char* message; // The error message. }; #endif gpml-matlab-v3.6-2015-07-07/util/lbfgsb/Makefile000644 000765 000024 00000006734 12441263152 020754 0ustar00carlstaff000000 000000 # for 'make nomex' you need to provide the variables MEX_SUFFIX and MATLAB_LIB # for 'make mex' you need to provide the variable MEX and have mex set up # for 'make oct' you need to provide the variable OCT and OCTAVE_INCLUDE ### 1) Choose Matlab configuration ############################################ ### path to your Matlab installation ### you can find it by the commands 'locate matlab' or 'find / -name "matlab"' MATLAB_HOME=/usr/local/bin/matlab ### For 32 bit Linux, uncomment the following three lines. -------------------- MEX = $(MATLAB_HOME)/bin/mex MEX_SUFFIX = mexglx MATLAB_LIB = -L$(MATLAB_HOME)/bin/glnx86 -lmex ### For 64 bit Linux, uncomment the following three lines. -------------------- #MEX = $(MATLAB_HOME)/bin/mex #MEX_SUFFIX = mexa64 #MATLAB_LIB = -L$(MATLAB_HOME)/bin/glnxa64 -lmex ### For Mac OS X, uncomment the following three lines. ------------------------ #MEX = /Applications/MATLAB7/bin/mex #MEX_SUFFIX = mexmac #MATLAB_LIB = -L$(MATLAB_HOME)/bin/maci -lmex ### 2) Choose Octave configuration ############################################ OCTAVE_INCLUDE = /usr/local/include/octave-3.2.4/octave OCT = mkoctfile ### 3) Choose L-BFGS-B configuration ########################################## ### Uncomment the following 2 lines for LBFGSB 2.4. --------------------------- LBFGSB_OBJS = solver_2_4.o LIBS = ### Uncomment the following 2 lines for LFBGSB 3.0 with blas libs provided. --- #LBFGSB_OBJS = solver_3_0.o linpack.o timer.o blas.o #LIBS = ### Uncomment the following 2 lines for LFBGSB 3.0 own blas libs. ------------- #LBFGSB_OBJS = solver_3_0.o linpack.o timer.o #LIBS = -lblas ### 4) Choose FORTRAN compiler ################################################ FFLAGS = -O3 -fPIC -fexceptions -Wall -g -Wno-uninitialized ### To work with f77, uncomment the following two lines. ---------------------- #F77 = f77 #FORTRAN_LIB = g2c ### To use gfortran, uncomment the following two lines. ----------------------- F77 = gfortran FORTRAN_LIB = gfortran ### 5) Choose C++ compiler #################################################### CXX = g++ CFLAGS = -O3 -fPIC -pthread -Wall -Werror -ansi -ffast-math -fomit-frame-pointer ### 6) If you need to specify extra directories where to look for libraries, ### write the paths of the directories, separated by a white space LIBDIRS = #LIBDIRS = ./ /usr/lib/gcc/x86_64-linux-gnu/4.8/ ############################################################################### ### Do not edit below ######################################################### ############################################################################### TARGET = lbfgsb OBJS = $(LBFGSB_OBJS) matlabexception.o matlabscalar.o matlabstring.o \ matlabmatrix.o arrayofmatrices.o program.o matlabprogram.o \ lbfgsb.o %.o: %.cpp $(CXX) $(CFLAGS) -I$(OCTAVE_INCLUDE) -I$(MATLAB_HOME)/extern/include -o $@ -c $^ %.o: %.f $(F77) $(FFLAGS) -o $@ -c $^ all: nomex mex: $(TARGET)_mex copy tidy nomex: $(TARGET)_nomex copy tidy octave: $(TARGET)_octave copy tidy oct: octave $(TARGET)_mex: $(OBJS) $(MEX) -cxx CXX=$(CXX) CC=$(CXX) FC=$(FCC) LD=$(CXX) -l$(FORTRAN_LIB) -lm \ -O -output $(TARGET) $^ $(TARGET)_nomex: $(OBJS) $(CXX) $^ -shared -o $(TARGET).$(MEX_SUFFIX) $(MATLAB_LIB) $(addprefix -L,$(LIBDIRS)) -l$(FORTRAN_LIB) -lm $(TARGET)_octave: $(OBJS) $(OCT) --mex -o $(TARGET).mex $^ $(LIBS)$ copy: mv $(TARGET).mex* .. tidy: rm -f *.o clean: tidy rm -f ../$(TARGET).mex* gpml-matlab-v3.6-2015-07-07/util/lbfgsb/lbfgsb.cpp000644 000765 000024 00000010024 12313000560 021230 0ustar00carlstaff000000 000000 #include "mex.h" #ifndef __linux__ #include "matrix.h" #endif #include "arrayofmatrices.h" #include "matlabexception.h" #include "matlabscalar.h" #include "matlabstring.h" #include "matlabmatrix.h" #include "arrayofmatrices.h" #include "program.h" #include "matlabprogram.h" #include #include extern void _main(); // Constants. // ----------------------------------------------------------------- const int minNumInputArgs = 5; // Function definitions. // ----------------------------------------------------------------- void mexFunction (int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) try { // Check to see if we have the correct number of input and output // arguments. if (nrhs < minNumInputArgs) throw MatlabException("Incorrect number of input arguments"); // Get the starting point for the variables. This is specified in // the first input argument. The variables must be either a single // matrix or a cell array of matrices. int k = 0; // The index of the current input argument. ArrayOfMatrices x0(prhs[k++]); // Create the output, which stores the solution obtained from // running IPOPT. There should be as many output arguments as cell // entries in X. if (nlhs != x0.length()) throw MatlabException("Incorrect number of output arguments"); ArrayOfMatrices x(plhs,x0); // Load the lower and upper bounds on the variables as // ArrayOfMatrices objects. They should have the same structure as // the ArrayOfMatrices object "x". ArrayOfMatrices lb(prhs[k++]); ArrayOfMatrices ub(prhs[k++]); // Check to make sure the bounds make sense. if (lb != x || ub != x) throw MatlabException("Input arguments LB and UB must have the same \ structure as X"); // Get the Matlab callback functions. MatlabString objFunc(prhs[k++]); MatlabString gradFunc(prhs[k++]); // Get the auxiliary data. const mxArray* auxData; const mxArray* ptr = prhs[k++]; if (nrhs > 5) { if (mxIsEmpty(ptr)) auxData = 0; else auxData = ptr; } else auxData = 0; // Get the intermediate callback function. MatlabString* iterFunc; ptr = prhs[k++]; if (nrhs > 6) { if (mxIsEmpty(ptr)) iterFunc = 0; else iterFunc = new MatlabString(ptr); } else iterFunc = 0; // Set the options for the L-BFGS algorithm to their defaults. int maxiter = defaultmaxiter; int m = defaultm; double factr = defaultfactr; double pgtol = defaultpgtol; // Process the remaining input arguments, which set options for // the IPOPT algorithm. while (k < nrhs) { // Get the option label from the Matlab input argument. MatlabString optionLabel(prhs[k++]); if (k < nrhs) { // Get the option value from the Matlab input argument. MatlabScalar optionValue(prhs[k++]); double value = optionValue; if (!strcmp(optionLabel,"maxiter")) maxiter = (int) value; else if (!strcmp(optionLabel,"m")) m = (int) value; else if (!strcmp(optionLabel,"factr")) factr = value / mxGetEps(); else if (!strcmp(optionLabel,"pgtol")) pgtol = value; else { if (iterFunc) delete iterFunc; throw MatlabException("Nonexistent option"); } } } // Create a new instance of the optimization problem. x = x0; MatlabProgram program(x,lb,ub,&objFunc,&gradFunc,iterFunc, (mxArray*) auxData,m,maxiter,factr,pgtol); // Run the L-BFGS-B solver. SolverExitStatus exitStatus = program.runSolver(); if (exitStatus == abnormalTermination) { if (iterFunc) delete iterFunc; throw MatlabException("Solver unable to satisfy convergence \ criteria due to abnormal termination"); } else if (exitStatus == errorOnInput) { if (iterFunc) delete iterFunc; throw MatlabException("Invalid inputs to L-BFGS routine"); } // Free the dynamically allocated memory. if (iterFunc) delete iterFunc; } catch (std::exception& error) { mexErrMsgTxt(error.what()); } gpml-matlab-v3.6-2015-07-07/util/lbfgsb/matlabstring.h000644 000765 000024 00000002200 12313000560 022122 0ustar00carlstaff000000 000000 #ifndef INCLUDE_MATLABSTRING #define INCLUDE_MATLABSTRING #include "mex.h" #include // Function declarations. // ----------------------------------------------------------------- // Copy a C-style string (i.e. a null-terminated character array). char* copystring (const char* source); // Class MatlabString. // ----------------------------------------------------------------- // This class encapsulates read-only access to a MATLAB character // array. class MatlabString { public: // The constructor accepts as input a pointer to a Matlab array, // which must be a valid string (array of type CHAR). explicit MatlabString (const mxArray* ptr); // The copy constructor makes a full copy of the source string. MatlabString (const MatlabString& source); // The destructor. ~MatlabString(); // Conversion operator for null-terminated string. operator const char* () const { return s; }; protected: char* s; // The null-terminated string. // The copy assignment operator is not proper, thus remains // protected. MatlabString& operator= (const MatlabString& source) { return *this; }; }; #endif gpml-matlab-v3.6-2015-07-07/util/lbfgsb/matlabprogram.h000644 000765 000024 00000005501 11422240325 022277 0ustar00carlstaff000000 000000 #ifndef INCLUDE_MATLABPROGRAM #define INCLUDE_MATLABPROGRAM #include "mex.h" #include "program.h" #include "matlabscalar.h" #include "matlabstring.h" #include "arrayofmatrices.h" // Class MatlabProgram. // ----------------------------------------------------------------- // This is an implementation of the abstract class Program. class MatlabProgram: public Program { public: // On input, "variables" should contain the initial point for the // optimization routine. If no iterative callback function is // specified, "iterFunc" may be set to 0. Also, if no auxiliary data // is needed, it may also be set to 0. MatlabProgram (ArrayOfMatrices& variables, const ArrayOfMatrices& lowerbounds, const ArrayOfMatrices& upperbounds, const MatlabString* objFunc, const MatlabString* gradFunc, const MatlabString* iterFunc, mxArray* auxData, int m = defaultm, int maxiter = defaultmaxiter, double factr = defaultfactr, double pgtol = defaultpgtol); // The destructor. virtual ~MatlabProgram(); // These provide definitions for the pure virtual functions of the // abstract parent class. virtual double computeObjective (int n, double* x); virtual void computeGradient (int n, double* x, double* g); virtual void iterCallback (int t, double* x, double f); // Run the solver. Upon completion, the solution is stored in // "variables". SolverExitStatus runSolver(); protected: ArrayOfMatrices& variables; // Storage for the initial value and // solution. const MatlabString* objFunc; // The objective callback function. const MatlabString* gradFunc; // The gradient callback function. const MatlabString* iterFunc; // The iterative callback function. ArrayOfMatrices* varMatlab; // Inputs to the Matlab callback mxArray** varInputs; // functions representing the // current values of the variables. MatlabScalar* fMatlab; // Input to the Matlab callback // functions representing the // current value of the objective. MatlabScalar* tMatlab; // Input to the Matlab callback // functions representing the // current iteration. int numInputsObjFunc; // The number of inputs passed to the // objective callback function. int numInputsGradFunc; // The number of inputs passed to the // gradient callback function. int numInputsIterFunc; // The number of inputs passed to the // iterative callback function. mxArray** inputsObjFunc; // The inputs passed to the objective // callback function. mxArray** inputsGradFunc; // The inputs passed to the gradient // callback function. mxArray** inputsIterFunc; // The inputs passed to the iterative // callback function. }; #endif gpml-matlab-v3.6-2015-07-07/util/lbfgsb/blas.f000644 000765 000024 00000014162 12343310245 020373 0ustar00carlstaff000000 000000 c c L-BFGS-B is released under the “New BSD License” (aka “Modified BSD c License” or “3-clause license”) c Please read attached file License.txt c double precision function dnrm2(n,x,incx) integer n,incx double precision x(n) c ********** c c Function dnrm2 c c Given a vector x of length n, this function calculates the c Euclidean norm of x with stride incx. c c The function statement is c c double precision function dnrm2(n,x,incx) c c where c c n is a positive integer input variable. c c x is an input array of length n. c c incx is a positive integer variable that specifies the c stride of the vector. c c Subprograms called c c FORTRAN-supplied ... abs, max, sqrt c c MINPACK-2 Project. February 1991. c Argonne National Laboratory. c Brett M. Averick. c c ********** integer i double precision scale dnrm2 = 0.0d0 scale = 0.0d0 do 10 i = 1, n, incx scale = max(scale, abs(x(i))) 10 continue if (scale .eq. 0.0d0) return do 20 i = 1, n, incx dnrm2 = dnrm2 + (x(i)/scale)**2 20 continue dnrm2 = scale*sqrt(dnrm2) return end c====================== The end of dnrm2 =============================== subroutine daxpy(n,da,dx,incx,dy,incy) c c constant times a vector plus a vector. c uses unrolled loops for increments equal to one. c jack dongarra, linpack, 3/11/78. c double precision dx(*),dy(*),da integer i,incx,incy,ix,iy,m,mp1,n c if(n.le.0)return if (da .eq. 0.0d0) return if(incx.eq.1.and.incy.eq.1)go to 20 c c code for unequal increments or equal increments c not equal to 1 c ix = 1 iy = 1 if(incx.lt.0)ix = (-n+1)*incx + 1 if(incy.lt.0)iy = (-n+1)*incy + 1 do 10 i = 1,n dy(iy) = dy(iy) + da*dx(ix) ix = ix + incx iy = iy + incy 10 continue return c c code for both increments equal to 1 c c c clean-up loop c 20 m = mod(n,4) if( m .eq. 0 ) go to 40 do 30 i = 1,m dy(i) = dy(i) + da*dx(i) 30 continue if( n .lt. 4 ) return 40 mp1 = m + 1 do 50 i = mp1,n,4 dy(i) = dy(i) + da*dx(i) dy(i + 1) = dy(i + 1) + da*dx(i + 1) dy(i + 2) = dy(i + 2) + da*dx(i + 2) dy(i + 3) = dy(i + 3) + da*dx(i + 3) 50 continue return end c====================== The end of daxpy =============================== subroutine dcopy(n,dx,incx,dy,incy) c c copies a vector, x, to a vector, y. c uses unrolled loops for increments equal to one. c jack dongarra, linpack, 3/11/78. c double precision dx(*),dy(*) integer i,incx,incy,ix,iy,m,mp1,n c if(n.le.0)return if(incx.eq.1.and.incy.eq.1)go to 20 c c code for unequal increments or equal increments c not equal to 1 c ix = 1 iy = 1 if(incx.lt.0)ix = (-n+1)*incx + 1 if(incy.lt.0)iy = (-n+1)*incy + 1 do 10 i = 1,n dy(iy) = dx(ix) ix = ix + incx iy = iy + incy 10 continue return c c code for both increments equal to 1 c c c clean-up loop c 20 m = mod(n,7) if( m .eq. 0 ) go to 40 do 30 i = 1,m dy(i) = dx(i) 30 continue if( n .lt. 7 ) return 40 mp1 = m + 1 do 50 i = mp1,n,7 dy(i) = dx(i) dy(i + 1) = dx(i + 1) dy(i + 2) = dx(i + 2) dy(i + 3) = dx(i + 3) dy(i + 4) = dx(i + 4) dy(i + 5) = dx(i + 5) dy(i + 6) = dx(i + 6) 50 continue return end c====================== The end of dcopy =============================== double precision function ddot(n,dx,incx,dy,incy) c c forms the dot product of two vectors. c uses unrolled loops for increments equal to one. c jack dongarra, linpack, 3/11/78. c double precision dx(*),dy(*),dtemp integer i,incx,incy,ix,iy,m,mp1,n c ddot = 0.0d0 dtemp = 0.0d0 if(n.le.0)return if(incx.eq.1.and.incy.eq.1)go to 20 c c code for unequal increments or equal increments c not equal to 1 c ix = 1 iy = 1 if(incx.lt.0)ix = (-n+1)*incx + 1 if(incy.lt.0)iy = (-n+1)*incy + 1 do 10 i = 1,n dtemp = dtemp + dx(ix)*dy(iy) ix = ix + incx iy = iy + incy 10 continue ddot = dtemp return c c code for both increments equal to 1 c c c clean-up loop c 20 m = mod(n,5) if( m .eq. 0 ) go to 40 do 30 i = 1,m dtemp = dtemp + dx(i)*dy(i) 30 continue if( n .lt. 5 ) go to 60 40 mp1 = m + 1 do 50 i = mp1,n,5 dtemp = dtemp + dx(i)*dy(i) + dx(i + 1)*dy(i + 1) + * dx(i + 2)*dy(i + 2) + dx(i + 3)*dy(i + 3) + dx(i + 4)*dy(i + 4) 50 continue 60 ddot = dtemp return end c====================== The end of ddot ================================ subroutine dscal(n,da,dx,incx) c c scales a vector by a constant. c uses unrolled loops for increment equal to one. c jack dongarra, linpack, 3/11/78. c modified 3/93 to return if incx .le. 0. c double precision da,dx(*) integer i,incx,m,mp1,n,nincx c if( n.le.0 .or. incx.le.0 )return if(incx.eq.1)go to 20 c c code for increment not equal to 1 c nincx = n*incx do 10 i = 1,nincx,incx dx(i) = da*dx(i) 10 continue return c c code for increment equal to 1 c c c clean-up loop c 20 m = mod(n,5) if( m .eq. 0 ) go to 40 do 30 i = 1,m dx(i) = da*dx(i) 30 continue if( n .lt. 5 ) return 40 mp1 = m + 1 do 50 i = mp1,n,5 dx(i) = da*dx(i) dx(i + 1) = da*dx(i + 1) dx(i + 2) = da*dx(i + 2) dx(i + 3) = da*dx(i + 3) dx(i + 4) = da*dx(i + 4) 50 continue return end c====================== The end of dscal =============================== gpml-matlab-v3.6-2015-07-07/util/lbfgsb/README000644 000765 000024 00000051032 12343310245 020160 0ustar00carlstaff000000 000000 # A MATLAB interface for L-BFGS-B ###Update As of March 24, 2014, the MATLAB code supports the latest version of the L-BFGS-B solver (version 3.0), and is compatible with [GNU Octave](https://www.gnu.org/software/octave). Thank you to [Jose Vallet](http://github.com/josombio) for providing these updates. ###Overview L-BFGS-B is a collection of Fortran 77 routines for solving large-scale nonlinear optimization problems with bound constraints on the variables. One of the key features of the nonlinear solver is that knowledge of the Hessian is not required; the solver computes search directions by keeping track of a quadratic model of the objective function with a limited-memory BFGS (Broyden-Fletcher-Goldfarb-Shanno) approximation to the Hessian *(see Note #1)*. The algorithm was developed by Ciyou Zhu, Richard Byrd and Jorge Nocedal. For more information, go to the [original distribution site](http://www.ece.northwestern.edu/~nocedal/lbfgsb.html) for the L-BFGS-B software package. I've designed an interface to the L-BFGS-B solver so that it can be called like any other function in MATLAB *(see Note #2)*. See the text below for more information on installing and calling this function in MATLAB. Along the way, I've also developed a C++ class that encapsulates all the "messy" details in executing the L-BFGS-B code. See below for instructions on how to incorporate this C++ class info your own code. This code has been tested in MATLAB and Octave on Linux, and is integrated into [GPML](http://www.gaussianprocess.org/gpml/code). It has also been executed successfully in Windows; see below. ###License Copyright (c) 2013, 2014 Peter Carbonetto The lbfgsb-matlab project repository is free software: you can redistribute it and/or modify it under the terms of the [GNU General Public License](http://www.gnu.org/licenses/gpl.html) as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but **without any warranty**; without even the implied warranty of **merchantability** or **fitness for a particular purpose**. See [LICENSE](LICENSE) for more details. ###Installation Follow these steps to compile and install **lbfgsb-matlab** for MATLAB and Octave. Some of these steps are applicable to MATLAB or Octave only, and you can skip them if you want to install the interface for only one of them. These installation instructions assume you have Linux, although similar steps should also work for other UNIX-based operating systems, such as Mac OS X. These instructions also assume you have [GNU Make](http://www.gnu.org/software/make) installed. (We have successfully compiled this source code with gcc versions 4.6.3 and 4.6.4, under Ubuntu Linux 12.10 and 13.10, and have successfully run the MATLAB scripts in Octave 3.6.2, 3.6.4, 3.8.0 and 3.8.1.) **For Windows users:** If you have a Windows operating system, Guillaume Jacquenot has generously provided [instructions](INSTALL.WINDOWS.txt) for compiling the software on Windows with [gnumex](http://gnumex.sourceforge.net) and Mingw. These instructions are slightly out of date, and suggestions for improving these installation instructions are welcome. We will create a MEX file, which is basically a file that contains a routine that can be called from MATLAB as if it were a built-it function. To learn about MEX files, I refer you to [this document](http://www.mathworks.com/support/tech-notes/1600/1605.html) on the MathWorks website. **Download the source code.** Clone or fork the this repository, or download this repository as a [ZIP file](http://github.com/pcarbo/lbfgsb-matlab/archive/master.zip) and unpack the ZIP file. The [src](src) subdirectory contains some MATLAB files (ending in .m), some C++ source and header files (ending in .h and .cpp), a single Fortran 77 source file, [solver.f](src/solver.f), containing the L-BFGS-B routines, and a [Makefile](src/Makefile). I've included a version of the Fortran routines that is more recent than what is available for download at the [distribution site](http://www.ece.northwestern.edu/~nocedal/lbfgsb.html) at Northwestern University. In the following, we will refer to the directory containing your local copy of the repository as LBFGSB_HOME. Wherever you find LBFGSB_HOME, you should substitute it with your own directory. **Install the C++ and Fortran 77 compilers.** In order to build the MEX file, you will need both a C++ compiler and a Fortran 77 compiler. Unfortunately, you can't just use any compiler. You have to use the precise one supported by MATLAB. For instance, if you are running Mac OS X 7.2 on a Linux operating system, you will need to install [GNU compiler collection](http://www.gnu.org/software/gcc) (GCC) version 3.4.4. Even if you already have a compiler installed on Linux, it may be the wrong version, and if you use the wrong version things could go horribly wrong. It is important that you use the correct version of the compiler, otherwise you will encounter linking errors. Refer to [this document](http://www.mathworks.com/support/tech-notes/1600/1601.html) to find out which compiler is supported by your version of MATLAB. You may also be able to use the **apt-get** program to install gfortran, as in: sudo apt-get install build-essential gfortran **Install Octave.** You will of course need a working installation of Matlab or Octave. In the case of Octave, if you don't have it the easiest way to install it is from the repositories using **apt-get**: sudo apt-get install octave If you want to use a more recent version of Octave, you will have to [install Octave from the source code](http://www.gnu.org/software/octave/doc/interpreter/Installation.html). After that, find the path to the mkoctfile compiler (in the bin directory) and the include directory. You will need those paths for steps below. Additionally you will need the Octave development tools, libs and header files to compile MEX files: sudo apt-get install liboctave-dev **Configure MATLAB.** Next, you need to set up and configure MATLAB to build MEX Files. This is explained quite nicely in [this MathWorks support document](http://www.mathworks.com/support/tech-notes/1600/1605.html). **Simple compilation procedure for MATLAB.** (Thanks to Stefan Harmeling.) You might be able to build and compile the MEX File from source code simply by calling MATLAB's mex program with the C++ and Fortran source files as inputs, like so: mex -output lbfgsb *.cpp solver.f If that doesn't work (e.g. you get linking errors, or it makes MATLAB crash when you try to call it), then you may have to follow the more complicated installation instructions below. **Modify the Makefile.** You are almost ready to build the MEX file. But before you do so, you need to edit the Makefile to coincide with your system setup. Edit the Makefile located inside the LBFGSB_HOME/src directory, following the instructions provided in this file. With this Makefile you may create: - A mex file for Matlab using the mex compiler (mex taget). - A mex file for Matlab using your C compiler (nomex target). - A mex file for Octave using mkoctfile (oct target). Regarding the compilation for Octave, the variable **OCTAVE_INCLUDE** is the directory where the Octave header files required for development are installed. Make sure that it points to the correct directory corresponding to your Octave installation. **OCT** is the command to call the mkoctfile compiler. If mkoctfile is in the path, simply leave it as it is. If not, include in the variable the full path to it. If you have several versions of Octave installed, make sure that OCT points to the mkoctfile program that corresponds to the Octave version you would like to use. For MATLAB, the variable **MEX** is the executable called to build the MEX file. The variables **CXX** and **F77** must be the names of your C++ and Fortran 77 compilers. **MEXSUFFIX** is the MEX file extension specific to your platform. (For instance, the extension for Linux is **mexglx**.) The variable **MATLAB_HOME** must be the base directory of your MATLAB installation. Finally, **CFLAGS** and **FFLAGS** are options passed to the C++ and Fortran compilers, respectively. Often these flags coincide with your MEX options file (see [here](http://www.mathworks.com/support/tech-notes/1600/1605.html) for more information on the options file). But often the settings in the MEX option file are incorrect, and so the options must be set manually. MATLAB requires some special compilation flags for various reasons, one being that it requires position-independent code. These instructions are vague (I apologize), and this step may require a bit of trial and error until before you get it right. **INSTALLDIR** is the installation directory relative to the LBFGSB_HOME/src directory (see Installation below). It may be helpful to look at the GCC documentation in order to understand what these various compiler flags mean. **Build the MEX file.** If you are in the directory containing all the source files, calling **make** at the command prompt will first compile the Fortran and C++ source files into object code (.o files). After that, the make program calls the MEX script, which links all the object files together into a single MEX file. Call **make** from the LBFGSB_HOME/src directory using the appropriate target, one of: make oct make nomex make mex If you didn't get any errors, then you are ready to try out the bound-constrained solver in MATLAB or Octave. Note that even if you didn't get any errors, there's still a possibility that you didn't link the MEX file properly, in which case executing the MEX file will cause MATLAB to crash. But there's only one way to find out: the hard way. **Final installation steps.** Installing the mex file is as simple as placing it wherever MATLAB or Octave can find it; that is, in its "path". There are many ways to do this. Whatever approach you take, note that the mex file does not contain the usage documentation, which is provided in lbfgsb.m. So wherever you place lbfgsb.mex, make sure that you include lbfgsb.m in the same directory. Perhaps the simplest approach to copy lbfgsb.mex and lbfgsb.m to a a specific folder. Alternatively, make install will create the .mex and .m files in the INSTALLDIR directory, and this directory will be automatically created if it does not exist. Once you are in MATLAB or Octave, add the installation directory to the MATLAB or Octave path can be done using the command **addpath**. Note that once you exit MATLAB or Octave the updated path is not preserved, and you will need to use addpath in every new session. You can add the addpath command to ~/.octaverc or ~/matlab/startup. In this way, the path will automatically be updated every time you start a new session. ###A brief tutorial I've written a short script [examplehs038.m](src/examplehs038.m) which demonstrates how to call **lbfgsb** for solving a very small optimization problem, the Hock & Schittkowski test problem #38 **(Note #3)**. The optimization problem has 4 variables. They are bounded from below by -10 and from above by +10. We've set the starting point to (-3, -1, -3, -1). We've written two MATLAB functions for computing the objective function and the gradient of the objective function at a given point. These functions can be found in the files [computeObjectiveHS038.m](src/computeObjectiveHS038.m) and [computeGradientHS038.m](src/computeGradientHS038.m). If you run the script, the solver should very quickly progress toward the point (1, 1, 1, 1). The objective is 0 at this point. The basic MATLAB function call is lbfgsb(x0,lb,ub,objfunc,gradfunc) The first input argument **x0** declares the starting point for the solver. The second and third input arguments define the lower and upper bounds on the variables, respectively. If an entry of **lb** is set to **-Inf**, then this is equivalent to no lower bound for that entry (the same applies to upper bounds, except that it must be **+Inf** instead). The next two input arguments must be the names of MATLAB routines (M-files). The first routine must calculate the objective function at the current point. The output must be a scalar representing the objective evaluated at the current point. The second function must compute the gradient of the objective at the current point. The input is the same as **objfunc**, but it must return as many outputs as there are inputs. For a complete guide on using the MATLAB interface, type **help lbfgsb** in MATLAB. The script [exampleldaimages.m](src/exampleldaimages.m) is a more complicated example. It uses the L-BFGS-B solver to compute some statistics of a posterior distribution for a text analysis model called latent Dirichlet allication (LDA). The model is used to estimate the topics for a collection of documents. The optimization problem is to compute an approximation to the posterior, since it is not possible to compute the posterior exactly. In this case, the optimization problem corresponds to minimizing the distance between the true posterior distribution and an approximate distribution. For more information, see the paper of Blei, Ng and Jordan, 2003 **(Note #4)**. In order to run this example, you will need to install the [lightspeed](http://research.microsoft.com/~minka/software/lightspeed) for MATLAB developed by Tom Minka at Microsoft Research. The script starts by generating a bunch of canonical images that represent topics; if a pixel is white then a word at that position is more likely to appear in that topic. Note that the order of the pixels in the image is unimportant, since LDA is a "bag of words" model. In another figure we show a small sample of the data generated from the topics. Each image represents a document, and the pixels in the image depict the word proportions in that document **(Note #5)**. Next, the script runs the L-BFGS-B solver to find a local minimum to the variational objective. Buried in the M-file [mflda.m](src/mflda.m) is the call to the solver: [xi gamma Phi] = lbfgsb({xi gamma Phi},... {repmat(lb,W,K) repmat(lb,K,D) repmat(lb,K,M)},... {repmat(inf,W,K) repmat(inf,K,D) repmat(inf,K,M)},... 'computeObjectiveMFLDA','computeGradientMFLDA',... {nu eta L w lnZconst},'callbackMFLDA',... 'm',m_lbfgs,'factr',1e-12,'pgtol',... tolerance,'maxiter',maxiter); This example will give us the opportunity to demonstrate some of the more complicated aspects of the our MATLAB interface. The first input to **lbfgsb** sets the initial iterate of the optimization algorithm. Notice that we've passed a cell array with entries that are matrices. Indeed, it is possible to pass either a matrix or a cell array. Whatever structure is passed, the other input arguments (and outputs) must also abide by this structure. The lower and upper bounds on variables are also cell arrays. All the upper bounds are set to infinity, which means that the variables are only bounded from below. It is instructive to examine the callback routines. They look like: f = computeObjectiveMFLDA(xi,gamma,Phi,auxdata) [dxi, dgamma, dPhi] = ... computeGradientMFLDA(xi,gamma,Phi,auxdata) Notice that each entry to the cell array is its own input argument to the callback routines. The same applies for the output arguments of the gradient callback function. The sixth input argument is a cell array that contains auxiliary data. This data will be passed to the MATLAB functions that evaluate the objective and gradient. The seventh input argument specifies a callback routine that is called exactly once for every iteration. This callback routine is most useful for examining the progress of the solver. The remaining input arguments are label/value pairs that override some of the default settings of L-BFGS-B. After converging to a solution, we display topic samples drawn from the variational approximation. A good solution should come close to recovering the true topics from which the data was generated (although they might not be in the same order). Don't be surprised if it takes many iterations before the optimization algorithm converges on a local minimum. (In repeated trials, I found it as many as 2000 iterations to converge to a stationary point of the objective.) This particular problem demonstrates the limitations of limited-memory quasi-Newton approximations to the Hessian; the low storage requirements can come at the cost of slow convergence to the solution. ###The C++ interface One of the nice byproducts of writing a MATLAB interface for L-BFGS-B is that I've ended up with a neat little C++ class that encapsulates the nuts and bolts of executing the solver. A brief description of the Program class can be found in the header file [program.h](src/program.h). "Program" is an abstract class, which means that its impossible to instantiate a Program object. This is because the class has member functions that aren't yet implemented. These are called pure virtual functions. In order to use the Program class, one needs to define another class that inherits the Program class and that implements the pure virtual functions. The class MatlabProgram (in [matlabprogram.h](src/matlabprogram.h)) is an example of such a class. The new child class must declare and implement these two functions: virtual double computeObjective (int n, double* x); virtual void computeGradient (int n, double* x, double* g); See the header file for a detailed description of these functions. The only remaining detail is calling the Program constructor. After that, it is just a matter of declaring a new object of type MyProgram then calling the function runSolver. ###Overview of source files solver.f
Fortran 77 source for the L-BFGS-B solver routines. program.h
program.cpp
 Header and source file for the C++ interface. array.h
arrayofmatrices.h
matlabexception.h
matlabmatrix.h
matlabprogram.h
matlabscalar.h
matlabstring.h
arrayofmatrices.cpp
matlabexception.cpp
matlabmatrix.cpp
matlabprogram.cpp
matlabscalar.cpp
matlabstring.cpp
lbfgsb.cpp
 Header and source files for the MATLAB interface. lbfgsb.m
Usage instructions for the MEX file. examplehs038.m
MATLAB script for solving the H&S test example #38. computeObjectiveHS038.m
computeGradientHS038.m
genericcallback.m
 MATLAB functions used by the script [examplehs038.m](src/examplehs038.m). exampleldaimages.m
MATLAB script that generates synthetic documents and topics, then computes a mean field variational approximation to the posterior distribution of the latent Dirichlet allocation model, then displays the result. mflda.m
This function computes a mean field variational approximation to the posterior distribution of the latent Dirichlet allocation model by minimizing the distance between the variational distribution and the true distribution. It uses the L-BFGS-B to minimize the objective function subject to the bound constraints. The objective function also acts as a lower bound on the logarithm of the denominator that appears from the application of Bayes rule **(Note #4)**. callbackMFLDA.m
computeObjectiveMFLDA.m
computeGradientMFLDA.m
computelnZ.m
computelnZconst.m
computem.m
dirichletrnd.m
gammarnd.m
createsyntheticdata.m
mfldainit.m
 Some functions used by [exampleldaimages.m](src/exampleldaimages.m) and [mflda.m](src/mflda.m). ###Credits MATLAB interface Developed by
[Peter Carbonetto](http://www.cs.ubc.ca/spider/pcarbo)
Dept. of Human Genetics
University of Chicago Support for Octave contributed by
[Jose Vallet](http://github.com/josombio)
School of Electrical Engineering
Aalto University Thank you to Ciyou Zhu, Richard Byrd, Jorge Nocedal and Jose Luis for making their L-BFGS-B Fortran code available, and to Hannes Nickisch for ... ###Notes 1. Ciyou Zhu, Richard H. Byrd, Peihuang Lu and Jorge Nocedal (1997). Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization. *ACM Transactions on Mathematical Software* 23(4): 550-560. 2. [Another MATLAB interface](http://www.cs.toronto.edu/~liam/software.shtml) for the L-BFGS-B routines has been developed by Liam Stewart at the University of Toronto. 3. Willi Hock and Klaus Schittkowski (1981). *Test Examples for Nonlinear Programming Codes.* Lecture Notes in Economics and Mathematical Systems, Vol. 187, Springer-Verlag. 4. David M. Blei, Andrew Y. Ng, Michael I. Jordan (2003). Latent Dirichlet allocation. *Journal of Machine Learning Research* Vol3: 993-1022. 5. Tom L. Griffiths and Mark Steyvers (2004). Finding scientific topics. *Proceedings of the National Academy of Sciences* 101: 5228-5235. gpml-matlab-v3.6-2015-07-07/util/lbfgsb/timer.f000644 000765 000024 00000002130 12343310245 020562 0ustar00carlstaff000000 000000 c c L-BFGS-B is released under the “New BSD License” (aka “Modified BSD c License” or “3-clause license”) c Please read attached file License.txt c subroutine timer(ttime) double precision ttime c real temp c c This routine computes cpu time in double precision; it makes use of c the intrinsic f90 cpu_time therefore a conversion type is c needed. c c J.L Morales Departamento de Matematicas, c Instituto Tecnologico Autonomo de Mexico c Mexico D.F. c c J.L Nocedal Department of Electrical Engineering and c Computer Science. c Northwestern University. Evanston, IL. USA c c January 21, 2011 c temp = sngl(ttime) call cpu_time(temp) ttime = dble(temp) return end gpml-matlab-v3.6-2015-07-07/util/lbfgsb/linpack.f000644 000765 000024 00000014274 12343310245 021077 0ustar00carlstaff000000 000000 c c L-BFGS-B is released under the “New BSD License” (aka “Modified BSD c License” or “3-clause license”) c Please read attached file License.txt c subroutine dpofa(a,lda,n,info) integer lda,n,info double precision a(lda,*) c c dpofa factors a double precision symmetric positive definite c matrix. c c dpofa is usually called by dpoco, but it can be called c directly with a saving in time if rcond is not needed. c (time for dpoco) = (1 + 18/n)*(time for dpofa) . c c on entry c c a double precision(lda, n) c the symmetric matrix to be factored. only the c diagonal and upper triangle are used. c c lda integer c the leading dimension of the array a . c c n integer c the order of the matrix a . c c on return c c a an upper triangular matrix r so that a = trans(r)*r c where trans(r) is the transpose. c the strict lower triangle is unaltered. c if info .ne. 0 , the factorization is not complete. c c info integer c = 0 for normal return. c = k signals an error condition. the leading minor c of order k is not positive definite. c c linpack. this version dated 08/14/78 . c cleve moler, university of new mexico, argonne national lab. c c subroutines and functions c c blas ddot c fortran sqrt c c internal variables c double precision ddot,t double precision s integer j,jm1,k c begin block with ...exits to 40 c c do 30 j = 1, n info = j s = 0.0d0 jm1 = j - 1 if (jm1 .lt. 1) go to 20 do 10 k = 1, jm1 t = a(k,j) - ddot(k-1,a(1,k),1,a(1,j),1) t = t/a(k,k) a(k,j) = t s = s + t*t 10 continue 20 continue s = a(j,j) - s c ......exit if (s .le. 0.0d0) go to 40 a(j,j) = sqrt(s) 30 continue info = 0 40 continue return end c====================== The end of dpofa =============================== subroutine dtrsl(t,ldt,n,b,job,info) integer ldt,n,job,info double precision t(ldt,*),b(*) c c c dtrsl solves systems of the form c c t * x = b c or c trans(t) * x = b c c where t is a triangular matrix of order n. here trans(t) c denotes the transpose of the matrix t. c c on entry c c t double precision(ldt,n) c t contains the matrix of the system. the zero c elements of the matrix are not referenced, and c the corresponding elements of the array can be c used to store other information. c c ldt integer c ldt is the leading dimension of the array t. c c n integer c n is the order of the system. c c b double precision(n). c b contains the right hand side of the system. c c job integer c job specifies what kind of system is to be solved. c if job is c c 00 solve t*x=b, t lower triangular, c 01 solve t*x=b, t upper triangular, c 10 solve trans(t)*x=b, t lower triangular, c 11 solve trans(t)*x=b, t upper triangular. c c on return c c b b contains the solution, if info .eq. 0. c otherwise b is unaltered. c c info integer c info contains zero if the system is nonsingular. c otherwise info contains the index of c the first zero diagonal element of t. c c linpack. this version dated 08/14/78 . c g. w. stewart, university of maryland, argonne national lab. c c subroutines and functions c c blas daxpy,ddot c fortran mod c c internal variables c double precision ddot,temp integer case,j,jj c c begin block permitting ...exits to 150 c c check for zero diagonal elements. c do 10 info = 1, n c ......exit if (t(info,info) .eq. 0.0d0) go to 150 10 continue info = 0 c c determine the task and go to it. c case = 1 if (mod(job,10) .ne. 0) case = 2 if (mod(job,100)/10 .ne. 0) case = case + 2 go to (20,50,80,110), case c c solve t*x=b for t lower triangular c 20 continue b(1) = b(1)/t(1,1) if (n .lt. 2) go to 40 do 30 j = 2, n temp = -b(j-1) call daxpy(n-j+1,temp,t(j,j-1),1,b(j),1) b(j) = b(j)/t(j,j) 30 continue 40 continue go to 140 c c solve t*x=b for t upper triangular. c 50 continue b(n) = b(n)/t(n,n) if (n .lt. 2) go to 70 do 60 jj = 2, n j = n - jj + 1 temp = -b(j+1) call daxpy(j,temp,t(1,j+1),1,b(1),1) b(j) = b(j)/t(j,j) 60 continue 70 continue go to 140 c c solve trans(t)*x=b for t lower triangular. c 80 continue b(n) = b(n)/t(n,n) if (n .lt. 2) go to 100 do 90 jj = 2, n j = n - jj + 1 b(j) = b(j) - ddot(jj-1,t(j+1,j),1,b(j+1),1) b(j) = b(j)/t(j,j) 90 continue 100 continue go to 140 c c solve trans(t)*x=b for t upper triangular. c 110 continue b(1) = b(1)/t(1,1) if (n .lt. 2) go to 130 do 120 j = 2, n b(j) = b(j) - ddot(j-1,t(1,j),1,b(1),1) b(j) = b(j)/t(j,j) 120 continue 130 continue 140 continue 150 continue return end c====================== The end of dtrsl =============================== gpml-matlab-v3.6-2015-07-07/util/lbfgsb/arrayofmatrices.h000644 000765 000024 00000004144 11422240325 022644 0ustar00carlstaff000000 000000 #ifndef INCLUDE_ARRAYOFMATRICES #define INCLUDE_ARRAYOFMATRICES #include "array.h" #include "matlabmatrix.h" #include "mex.h" // Class ArrayOfMatrices. // ----------------------------------------------------------------- class ArrayOfMatrices : public Array { public: // This version of the constructor behaves just like its parent. explicit ArrayOfMatrices (int length) : Array(length) { }; // This constructor creates an array of matrices from a Matlab // array. It accepts either a matrix in double precision, or a cell // array with entries that are matrices. explicit ArrayOfMatrices (const mxArray* ptr); // This constructor creates an array of matrices from a collection // of Matlab arrays. The Matlab arrays must be matrices. ArrayOfMatrices (const mxArray* ptrs[], int numptrs); // This constructor creates an array of matrices and the // associated Matlab structures. The Matlab structures are // matrices. The second input argument acts as a template for the // creation of the matrices, but the data from "model" is not // actually copied into the new ArrayOfMatrices object. It is up // to the user to make sure that the array of mxArray pointers has // enough room for the pointers to the Matlab arrays. ArrayOfMatrices (mxArray* ptrs[], const ArrayOfMatrices& model); // This constructor creates an array of matrices using the second // input argument as a model. The input argument "data" contains // the element data. Note that the information is NOT copied from // the model! ArrayOfMatrices (double* data, const ArrayOfMatrices& model); // The copy constructor makes a shallow copy of the source object. ArrayOfMatrices (const ArrayOfMatrices& source); // The destructor. ~ArrayOfMatrices(); // Copy assignment operator that observes the same behaviour as // the Array copy assignment operator. ArrayOfMatrices& operator= (const ArrayOfMatrices& source); // Return the total number of elements in all the matrices. int numelems() const; protected: static int getnummatlabmatrices (const mxArray* ptr); }; #endif gpml-matlab-v3.6-2015-07-07/util/lbfgsb/array.h000644 000765 000024 00000010714 11422240325 020567 0ustar00carlstaff000000 000000 #ifndef INCLUDE_ARRAY #define INCLUDE_ARRAY #include "matlabexception.h" #include // Function definitions. // ----------------------------------------------------------------- template void copymemory (const Type* source, Type* dest, int length) { memcpy(dest,source,sizeof(Type)*length); } // Class Array // ----------------------------------------------------------------- // An Array object stores a collection of ordered elements. The key // aspect of Array objects is that they do not necessarily achieve // encapsulation; they do not necessarily retain independent // ownership of their data. That is, the data could be modified // externally by another agent. This behaviour is determined by the // data member "owner". If "owner" is false, the array does not take // care of allocation and deallocation of the data in memory. // // Copy assignment behaves different than usual---it copies the // data, but requires that the destination already have the proper // resources allocated in memory. // // The copy constructor performs a shallow copy. template class Array { public: // This constructor allocates memory for an array of elements of // the specificed length. explicit Array (int length); // This constructor is set to point to an already existing // array. As such, it does not gain ownership of the elements. Array (Type* data, int length); // The copy constructor performs a shallow copy, so that both // arrays share the same copy of the data. Array (const Array& source); // The destructor. ~Array(); // Set all the elements of the array to the specified value. void setvalue (const Type& value); // Copy the elements to from the source array. Both the source and // destination array must have the same length. void inject (const Type* source); void inject (const Array& source); Array& operator= (const Type* source); Array& operator= (const Array& source); // Get the number of elements in the array. int length() const { return n; }; // Elements of an array A can be accessed the subscript operator; // e.g. A[i]. For the sake of efficiency, no checks are made to // ensure that the subscripting is legal. It is up to the user to // ensure that i is non-negative and less than A.length(). Note // that subscripts, as per C++ convention, start at 0. Type& operator[] (int i); const Type& operator[] (int i) const; // Returns true if the two arrays have the same dimensions // (i.e. the same lengths). bool operator== (const Array& a) const { return n == a.n; }; bool operator!= (const Array& a) const { return !(*this == a); }; protected: Type* elems; int n; // The number of elements in the array. bool owner; // Whether the object has ownership of the data. }; // Function definitions for class Array. // ----------------------------------------------------------------- template Array::Array (int length) { n = length; owner = true; elems = 0; elems = new Type[length]; } template Array::Array (Type* data, int length) { n = length; owner = false; elems = data; } template Array::Array (const Array& source) { n = source.n; owner = false; elems = source.elems; } template Array::~Array () { if (owner && elems) delete[] elems; } template void Array::setvalue (const Type& value) { for (int i = 0; i < n; i++) elems[i] = value; } template void Array::inject (const Type* source) { // Check for self-assignment. If there is self-assignment, there // is no need to copy because the objects share the same data! if (elems != source) copymemory(source,elems,n); } template void Array::inject (const Array& source) { if (n != source.n) throw MatlabException("Unable to perform copy; arrays have \ different lengths"); inject(source.elems); } template Array& Array::operator= (const Type* source) { inject(source); return *this; } template Array& Array::operator= (const Array& source) { inject(source); return *this; } template Type& Array::operator[] (int i) { return elems[i]; } template const Type& Array::operator[] (int i) const { return elems[i]; } #endif gpml-matlab-v3.6-2015-07-07/util/lbfgsb/solver_3_0.f000644 000765 000024 00000376670 12313000560 021435 0ustar00carlstaff000000 000000 c c L-BFGS-B is released under the “New BSD License” (aka “Modified BSD License” c or “3-clause license”) c Please read attached file License.txt c c=========== L-BFGS-B (version 3.0. April 25, 2011 =================== c c This is a modified version of L-BFGS-B. Minor changes in the updated c code appear preceded by a line comment as follows c c c-jlm-jn c c Major changes are described in the accompanying paper: c c Jorge Nocedal and Jose Luis Morales, Remark on "Algorithm 778: c L-BFGS-B: Fortran Subroutines for Large-Scale Bound Constrained c Optimization" (2011). To appear in ACM Transactions on c Mathematical Software, c c The paper describes an improvement and a correction to Algorithm 778. c It is shown that the performance of the algorithm can be improved c significantly by making a relatively simple modication to the subspace c minimization phase. The correction concerns an error caused by the use c of routine dpmeps to estimate machine precision. c c The total work space **wa** required by the new version is c c 2*m*n + 11m*m + 5*n + 8*m c c the old version required c c 2*m*n + 12m*m + 4*n + 12*m c c c J. Nocedal Department of Electrical Engineering and c Computer Science. c Northwestern University. Evanston, IL. USA c c c J.L Morales Departamento de Matematicas, c Instituto Tecnologico Autonomo de Mexico c Mexico D.F. Mexico. c c March 2011 c c============================================================================= subroutine setulb(n, m, x, l, u, nbd, f, g, factr, pgtol, wa, iwa, + task, iprint, csave, lsave, isave, dsave) character*60 task, csave logical lsave(4) integer n, m, iprint, + nbd(n), iwa(3*n), isave(44) double precision f, factr, pgtol, x(n), l(n), u(n), g(n), c c-jlm-jn + wa(2*m*n + 5*n + 11*m*m + 8*m), dsave(29) c ************ c c Subroutine setulb c c This subroutine partitions the working arrays wa and iwa, and c then uses the limited memory BFGS method to solve the bound c constrained optimization problem by calling mainlb. c (The direct method will be used in the subspace minimization.) c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c x is a double precision array of dimension n. c On entry x is an approximation to the solution. c On exit x is the current approximation. c c l is a double precision array of dimension n. c On entry l is the lower bound on x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound on x. c On exit u is unchanged. c c nbd is an integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, and c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c f is a double precision variable. c On first entry f is unspecified. c On final exit f is the value of the function at x. c c g is a double precision array of dimension n. c On first entry g is unspecified. c On final exit g is the value of the gradient at x. c c factr is a double precision variable. c On entry factr >= 0 is specified by the user. The iteration c will stop when c c (f^k - f^{k+1})/max{|f^k|,|f^{k+1}|,1} <= factr*epsmch c c where epsmch is the machine precision, which is automatically c generated by the code. Typical values for factr: 1.d+12 for c low accuracy; 1.d+7 for moderate accuracy; 1.d+1 for extremely c high accuracy. c On exit factr is unchanged. c c pgtol is a double precision variable. c On entry pgtol >= 0 is specified by the user. The iteration c will stop when c c max{|proj g_i | i = 1, ..., n} <= pgtol c c where pg_i is the ith component of the projected gradient. c On exit pgtol is unchanged. c c wa is a double precision working array of length c (2mmax + 5)nmax + 12mmax^2 + 12mmax. c c iwa is an integer working array of length 3nmax. c c task is a working string of characters of length 60 indicating c the current job when entering and quitting this subroutine. c c iprint is an integer variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c csave is a working string of characters of length 60. c c lsave is a logical working array of dimension 4. c On exit with 'task' = NEW_X, the following information is c available: c If lsave(1) = .true. then the initial X has been replaced by c its projection in the feasible set; c If lsave(2) = .true. then the problem is constrained; c If lsave(3) = .true. then each variable has upper and lower c bounds; c c isave is an integer working array of dimension 44. c On exit with 'task' = NEW_X, the following information is c available: c isave(22) = the total number of intervals explored in the c search of Cauchy points; c isave(26) = the total number of skipped BFGS updates before c the current iteration; c isave(30) = the number of current iteration; c isave(31) = the total number of BFGS updates prior the current c iteration; c isave(33) = the number of intervals explored in the search of c Cauchy point in the current iteration; c isave(34) = the total number of function and gradient c evaluations; c isave(36) = the number of function value or gradient c evaluations in the current iteration; c if isave(37) = 0 then the subspace argmin is within the box; c if isave(37) = 1 then the subspace argmin is beyond the box; c isave(38) = the number of free variables in the current c iteration; c isave(39) = the number of active constraints in the current c iteration; c n + 1 - isave(40) = the number of variables leaving the set of c active constraints in the current iteration; c isave(41) = the number of variables entering the set of active c constraints in the current iteration. c c dsave is a double precision working array of dimension 29. c On exit with 'task' = NEW_X, the following information is c available: c dsave(1) = current 'theta' in the BFGS matrix; c dsave(2) = f(x) in the previous iteration; c dsave(3) = factr*epsmch; c dsave(4) = 2-norm of the line search direction vector; c dsave(5) = the machine precision epsmch generated by the code; c dsave(7) = the accumulated time spent on searching for c Cauchy points; c dsave(8) = the accumulated time spent on c subspace minimization; c dsave(9) = the accumulated time spent on line search; c dsave(11) = the slope of the line search function at c the current point of line search; c dsave(12) = the maximum relative step length imposed in c line search; c dsave(13) = the infinity norm of the projected gradient; c dsave(14) = the relative step length in the line search; c dsave(15) = the slope of the line search function at c the starting point of the line search; c dsave(16) = the square of the 2-norm of the line search c direction vector. c c Subprograms called: c c L-BFGS-B Library ... mainlb. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: a c limited memory FORTRAN code for solving bound constrained c optimization problems'', Tech. Report, NAM-11, EECS Department, c Northwestern University, 1994. c c (Postscript files of these papers are available via anonymous c ftp to eecs.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ c-jlm-jn integer lws,lr,lz,lt,ld,lxp,lwa, + lwy,lsy,lss,lwt,lwn,lsnd if (task .eq. 'START') then isave(1) = m*n isave(2) = m**2 isave(3) = 4*m**2 isave(4) = 1 ! ws m*n isave(5) = isave(4) + isave(1) ! wy m*n isave(6) = isave(5) + isave(1) ! wsy m**2 isave(7) = isave(6) + isave(2) ! wss m**2 isave(8) = isave(7) + isave(2) ! wt m**2 isave(9) = isave(8) + isave(2) ! wn 4*m**2 isave(10) = isave(9) + isave(3) ! wsnd 4*m**2 isave(11) = isave(10) + isave(3) ! wz n isave(12) = isave(11) + n ! wr n isave(13) = isave(12) + n ! wd n isave(14) = isave(13) + n ! wt n isave(15) = isave(14) + n ! wxp n isave(16) = isave(15) + n ! wa 8*m endif lws = isave(4) lwy = isave(5) lsy = isave(6) lss = isave(7) lwt = isave(8) lwn = isave(9) lsnd = isave(10) lz = isave(11) lr = isave(12) ld = isave(13) lt = isave(14) lxp = isave(15) lwa = isave(16) call mainlb(n,m,x,l,u,nbd,f,g,factr,pgtol, + wa(lws),wa(lwy),wa(lsy),wa(lss), wa(lwt), + wa(lwn),wa(lsnd),wa(lz),wa(lr),wa(ld),wa(lt),wa(lxp), + wa(lwa), + iwa(1),iwa(n+1),iwa(2*n+1),task,iprint, + csave,lsave,isave(22),dsave) return end c======================= The end of setulb ============================= subroutine mainlb(n, m, x, l, u, nbd, f, g, factr, pgtol, ws, wy, + sy, ss, wt, wn, snd, z, r, d, t, xp, wa, + index, iwhere, indx2, task, + iprint, csave, lsave, isave, dsave) implicit none character*60 task, csave logical lsave(4) integer n, m, iprint, nbd(n), index(n), + iwhere(n), indx2(n), isave(23) double precision f, factr, pgtol, + x(n), l(n), u(n), g(n), z(n), r(n), d(n), t(n), c-jlm-jn + xp(n), + wa(8*m), + ws(n, m), wy(n, m), sy(m, m), ss(m, m), + wt(m, m), wn(2*m, 2*m), snd(2*m, 2*m), dsave(29) c ************ c c Subroutine mainlb c c This subroutine solves bound constrained optimization problems by c using the compact formula of the limited memory BFGS updates. c c n is an integer variable. c On entry n is the number of variables. c On exit n is unchanged. c c m is an integer variable. c On entry m is the maximum number of variable metric c corrections allowed in the limited memory matrix. c On exit m is unchanged. c c x is a double precision array of dimension n. c On entry x is an approximation to the solution. c On exit x is the current approximation. c c l is a double precision array of dimension n. c On entry l is the lower bound of x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound of x. c On exit u is unchanged. c c nbd is an integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c f is a double precision variable. c On first entry f is unspecified. c On final exit f is the value of the function at x. c c g is a double precision array of dimension n. c On first entry g is unspecified. c On final exit g is the value of the gradient at x. c c factr is a double precision variable. c On entry factr >= 0 is specified by the user. The iteration c will stop when c c (f^k - f^{k+1})/max{|f^k|,|f^{k+1}|,1} <= factr*epsmch c c where epsmch is the machine precision, which is automatically c generated by the code. c On exit factr is unchanged. c c pgtol is a double precision variable. c On entry pgtol >= 0 is specified by the user. The iteration c will stop when c c max{|proj g_i | i = 1, ..., n} <= pgtol c c where pg_i is the ith component of the projected gradient. c On exit pgtol is unchanged. c c ws, wy, sy, and wt are double precision working arrays used to c store the following information defining the limited memory c BFGS matrix: c ws, of dimension n x m, stores S, the matrix of s-vectors; c wy, of dimension n x m, stores Y, the matrix of y-vectors; c sy, of dimension m x m, stores S'Y; c ss, of dimension m x m, stores S'S; c yy, of dimension m x m, stores Y'Y; c wt, of dimension m x m, stores the Cholesky factorization c of (theta*S'S+LD^(-1)L'); see eq. c (2.26) in [3]. c c wn is a double precision working array of dimension 2m x 2m c used to store the LEL^T factorization of the indefinite matrix c K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c c where E = [-I 0] c [ 0 I] c c snd is a double precision working array of dimension 2m x 2m c used to store the lower triangular part of c N = [Y' ZZ'Y L_a'+R_z'] c [L_a +R_z S'AA'S ] c c z(n),r(n),d(n),t(n), xp(n),wa(8*m) are double precision working arrays. c z is used at different times to store the Cauchy point and c the Newton point. c xp is used to safeguard the projected Newton direction c c sg(m),sgo(m),yg(m),ygo(m) are double precision working arrays. c c index is an integer working array of dimension n. c In subroutine freev, index is used to store the free and fixed c variables at the Generalized Cauchy Point (GCP). c c iwhere is an integer working array of dimension n used to record c the status of the vector x for GCP computation. c iwhere(i)=0 or -3 if x(i) is free and has bounds, c 1 if x(i) is fixed at l(i), and l(i) .ne. u(i) c 2 if x(i) is fixed at u(i), and u(i) .ne. l(i) c 3 if x(i) is always fixed, i.e., u(i)=x(i)=l(i) c -1 if x(i) is always free, i.e., no bounds on it. c c indx2 is an integer working array of dimension n. c Within subroutine cauchy, indx2 corresponds to the array iorder. c In subroutine freev, a list of variables entering and leaving c the free set is stored in indx2, and it is passed on to c subroutine formk with this information. c c task is a working string of characters of length 60 indicating c the current job when entering and leaving this subroutine. c c iprint is an INTEGER variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c csave is a working string of characters of length 60. c c lsave is a logical working array of dimension 4. c c isave is an integer working array of dimension 23. c c dsave is a double precision working array of dimension 29. c c c Subprograms called c c L-BFGS-B Library ... cauchy, subsm, lnsrlb, formk, c c errclb, prn1lb, prn2lb, prn3lb, active, projgr, c c freev, cmprlb, matupd, formt. c c Minpack2 Library ... timer c c Linpack Library ... dcopy, ddot. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: FORTRAN c Subroutines for Large Scale Bound Constrained Optimization'' c Tech. Report, NAM-11, EECS Department, Northwestern University, c 1994. c c [3] R. Byrd, J. Nocedal and R. Schnabel "Representations of c Quasi-Newton Matrices and their use in Limited Memory Methods'', c Mathematical Programming 63 (1994), no. 4, pp. 129-156. c c (Postscript files of these papers are available via anonymous c ftp to eecs.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ logical prjctd,cnstnd,boxed,updatd,wrk character*3 word integer i,k,nintol,itfile,iback,nskip, + head,col,iter,itail,iupdat, + nseg,nfgv,info,ifun, + iword,nfree,nact,ileave,nenter double precision theta,fold,ddot,dr,rr,tol, + xstep,sbgnrm,ddum,dnorm,dtd,epsmch, + cpu1,cpu2,cachyt,sbtime,lnscht,time1,time2, + gd,gdold,stp,stpmx,time double precision one,zero parameter (one=1.0d0,zero=0.0d0) if (task .eq. 'START') then epsmch = epsilon(one) call timer(time1) c Initialize counters and scalars when task='START'. c for the limited memory BFGS matrices: col = 0 head = 1 theta = one iupdat = 0 updatd = .false. iback = 0 itail = 0 iword = 0 nact = 0 ileave = 0 nenter = 0 fold = zero dnorm = zero cpu1 = zero gd = zero stpmx = zero sbgnrm = zero stp = zero gdold = zero dtd = zero c for operation counts: iter = 0 nfgv = 0 nseg = 0 nintol = 0 nskip = 0 nfree = n ifun = 0 c for stopping tolerance: tol = factr*epsmch c for measuring running time: cachyt = 0 sbtime = 0 lnscht = 0 c 'word' records the status of subspace solutions. word = '---' c 'info' records the termination information. info = 0 itfile = 8 if (iprint .ge. 1) then c open a summary file 'iterate.dat' open (8, file = 'iterate.dat', status = 'unknown') endif c Check the input arguments for errors. call errclb(n,m,factr,l,u,nbd,task,info,k) if (task(1:5) .eq. 'ERROR') then call prn3lb(n,x,f,task,iprint,info,itfile, + iter,nfgv,nintol,nskip,nact,sbgnrm, + zero,nseg,word,iback,stp,xstep,k, + cachyt,sbtime,lnscht) return endif call prn1lb(n,m,l,u,x,iprint,itfile,epsmch) c Initialize iwhere & project x onto the feasible set. call active(n,l,u,nbd,x,iwhere,iprint,prjctd,cnstnd,boxed) c The end of the initialization. else c restore local variables. prjctd = lsave(1) cnstnd = lsave(2) boxed = lsave(3) updatd = lsave(4) nintol = isave(1) itfile = isave(3) iback = isave(4) nskip = isave(5) head = isave(6) col = isave(7) itail = isave(8) iter = isave(9) iupdat = isave(10) nseg = isave(12) nfgv = isave(13) info = isave(14) ifun = isave(15) iword = isave(16) nfree = isave(17) nact = isave(18) ileave = isave(19) nenter = isave(20) theta = dsave(1) fold = dsave(2) tol = dsave(3) dnorm = dsave(4) epsmch = dsave(5) cpu1 = dsave(6) cachyt = dsave(7) sbtime = dsave(8) lnscht = dsave(9) time1 = dsave(10) gd = dsave(11) stpmx = dsave(12) sbgnrm = dsave(13) stp = dsave(14) gdold = dsave(15) dtd = dsave(16) c After returning from the driver go to the point where execution c is to resume. if (task(1:5) .eq. 'FG_LN') goto 666 if (task(1:5) .eq. 'NEW_X') goto 777 if (task(1:5) .eq. 'FG_ST') goto 111 if (task(1:4) .eq. 'STOP') then if (task(7:9) .eq. 'CPU') then c restore the previous iterate. call dcopy(n,t,1,x,1) call dcopy(n,r,1,g,1) f = fold endif goto 999 endif endif c Compute f0 and g0. task = 'FG_START' c return to the driver to calculate f and g; reenter at 111. goto 1000 111 continue nfgv = 1 c Compute the infinity norm of the (-) projected gradient. call projgr(n,l,u,nbd,x,g,sbgnrm) if (iprint .ge. 1) then write (6,1002) iter,f,sbgnrm write (itfile,1003) iter,nfgv,sbgnrm,f endif if (sbgnrm .le. pgtol) then c terminate the algorithm. task = 'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL' goto 999 endif c ----------------- the beginning of the loop -------------------------- 222 continue if (iprint .ge. 99) write (6,1001) iter + 1 iword = -1 c if (.not. cnstnd .and. col .gt. 0) then c skip the search for GCP. call dcopy(n,x,1,z,1) wrk = updatd nseg = 0 goto 333 endif cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Compute the Generalized Cauchy Point (GCP). c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc call timer(cpu1) call cauchy(n,x,l,u,nbd,g,indx2,iwhere,t,d,z, + m,wy,ws,sy,wt,theta,col,head, + wa(1),wa(2*m+1),wa(4*m+1),wa(6*m+1),nseg, + iprint, sbgnrm, info, epsmch) if (info .ne. 0) then c singular triangular system detected; refresh the lbfgs memory. if(iprint .ge. 1) write (6, 1005) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. call timer(cpu2) cachyt = cachyt + cpu2 - cpu1 goto 222 endif call timer(cpu2) cachyt = cachyt + cpu2 - cpu1 nintol = nintol + nseg c Count the entering and leaving variables for iter > 0; c find the index set of free and active variables at the GCP. call freev(n,nfree,index,nenter,ileave,indx2, + iwhere,wrk,updatd,cnstnd,iprint,iter) nact = n - nfree 333 continue c If there are no free variables or B=theta*I, then c skip the subspace minimization. if (nfree .eq. 0 .or. col .eq. 0) goto 555 cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Subspace minimization. c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc call timer(cpu1) c Form the LEL^T factorization of the indefinite c matrix K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c where E = [-I 0] c [ 0 I] if (wrk) call formk(n,nfree,index,nenter,ileave,indx2,iupdat, + updatd,wn,snd,m,ws,wy,sy,theta,col,head,info) if (info .ne. 0) then c nonpositive definiteness in Cholesky factorization; c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1006) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. call timer(cpu2) sbtime = sbtime + cpu2 - cpu1 goto 222 endif c compute r=-Z'B(xcp-xk)-Z'g (using wa(2m+1)=W'(xcp-x) c from 'cauchy'). call cmprlb(n,m,x,g,ws,wy,sy,wt,z,r,wa,index, + theta,col,head,nfree,cnstnd,info) if (info .ne. 0) goto 444 c-jlm-jn call the direct method. call subsm( n, m, nfree, index, l, u, nbd, z, r, xp, ws, wy, + theta, x, g, col, head, iword, wa, wn, iprint, info) 444 continue if (info .ne. 0) then c singular triangular system detected; c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1005) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. call timer(cpu2) sbtime = sbtime + cpu2 - cpu1 goto 222 endif call timer(cpu2) sbtime = sbtime + cpu2 - cpu1 555 continue cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Line search and optimality tests. c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c Generate the search direction d:=z-x. do 40 i = 1, n d(i) = z(i) - x(i) 40 continue call timer(cpu1) 666 continue call lnsrlb(n,l,u,nbd,x,f,fold,gd,gdold,g,d,r,t,z,stp,dnorm, + dtd,xstep,stpmx,iter,ifun,iback,nfgv,info,task, + boxed,cnstnd,csave,isave(22),dsave(17)) if (info .ne. 0 .or. iback .ge. 20) then c restore the previous iterate. call dcopy(n,t,1,x,1) call dcopy(n,r,1,g,1) f = fold if (col .eq. 0) then c abnormal termination. if (info .eq. 0) then info = -9 c restore the actual number of f and g evaluations etc. nfgv = nfgv - 1 ifun = ifun - 1 iback = iback - 1 endif task = 'ABNORMAL_TERMINATION_IN_LNSRCH' iter = iter + 1 goto 999 else c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1008) if (info .eq. 0) nfgv = nfgv - 1 info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. task = 'RESTART_FROM_LNSRCH' call timer(cpu2) lnscht = lnscht + cpu2 - cpu1 goto 222 endif else if (task(1:5) .eq. 'FG_LN') then c return to the driver for calculating f and g; reenter at 666. goto 1000 else c calculate and print out the quantities related to the new X. call timer(cpu2) lnscht = lnscht + cpu2 - cpu1 iter = iter + 1 c Compute the infinity norm of the projected (-)gradient. call projgr(n,l,u,nbd,x,g,sbgnrm) c Print iteration information. call prn2lb(n,x,f,g,iprint,itfile,iter,nfgv,nact, + sbgnrm,nseg,word,iword,iback,stp,xstep) goto 1000 endif 777 continue c Test for termination. if (sbgnrm .le. pgtol) then c terminate the algorithm. task = 'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL' goto 999 endif ddum = max(abs(fold), abs(f), one) if ((fold - f) .le. tol*ddum) then c terminate the algorithm. task = 'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH' if (iback .ge. 10) info = -5 c i.e., to issue a warning if iback>10 in the line search. goto 999 endif c Compute d=newx-oldx, r=newg-oldg, rr=y'y and dr=y's. do 42 i = 1, n r(i) = g(i) - r(i) 42 continue rr = ddot(n,r,1,r,1) if (stp .eq. one) then dr = gd - gdold ddum = -gdold else dr = (gd - gdold)*stp call dscal(n,stp,d,1) ddum = -gdold*stp endif if (dr .le. epsmch*ddum) then c skip the L-BFGS update. nskip = nskip + 1 updatd = .false. if (iprint .ge. 1) write (6,1004) dr, ddum goto 888 endif cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Update the L-BFGS matrix. c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc updatd = .true. iupdat = iupdat + 1 c Update matrices WS and WY and form the middle matrix in B. call matupd(n,m,ws,wy,sy,ss,d,r,itail, + iupdat,col,head,theta,rr,dr,stp,dtd) c Form the upper half of the pds T = theta*SS + L*D^(-1)*L'; c Store T in the upper triangular of the array wt; c Cholesky factorize T to J*J' with c J' stored in the upper triangular of wt. call formt(m,wt,sy,ss,col,theta,info) if (info .ne. 0) then c nonpositive definiteness in Cholesky factorization; c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1007) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. goto 222 endif c Now the inverse of the middle matrix in B is c [ D^(1/2) O ] [ -D^(1/2) D^(-1/2)*L' ] c [ -L*D^(-1/2) J ] [ 0 J' ] 888 continue c -------------------- the end of the loop ----------------------------- goto 222 999 continue call timer(time2) time = time2 - time1 call prn3lb(n,x,f,task,iprint,info,itfile, + iter,nfgv,nintol,nskip,nact,sbgnrm, + time,nseg,word,iback,stp,xstep,k, + cachyt,sbtime,lnscht) 1000 continue c Save local variables. lsave(1) = prjctd lsave(2) = cnstnd lsave(3) = boxed lsave(4) = updatd isave(1) = nintol isave(3) = itfile isave(4) = iback isave(5) = nskip isave(6) = head isave(7) = col isave(8) = itail isave(9) = iter isave(10) = iupdat isave(12) = nseg isave(13) = nfgv isave(14) = info isave(15) = ifun isave(16) = iword isave(17) = nfree isave(18) = nact isave(19) = ileave isave(20) = nenter dsave(1) = theta dsave(2) = fold dsave(3) = tol dsave(4) = dnorm dsave(5) = epsmch dsave(6) = cpu1 dsave(7) = cachyt dsave(8) = sbtime dsave(9) = lnscht dsave(10) = time1 dsave(11) = gd dsave(12) = stpmx dsave(13) = sbgnrm dsave(14) = stp dsave(15) = gdold dsave(16) = dtd 1001 format (//,'ITERATION ',i5) 1002 format + (/,'At iterate',i5,4x,'f= ',1p,d12.5,4x,'|proj g|= ',1p,d12.5) 1003 format (2(1x,i4),5x,'-',5x,'-',3x,'-',5x,'-',5x,'-',8x,'-',3x, + 1p,2(1x,d10.3)) 1004 format (' ys=',1p,e10.3,' -gs=',1p,e10.3,' BFGS update SKIPPED') 1005 format (/, +' Singular triangular system detected;',/, +' refresh the lbfgs memory and restart the iteration.') 1006 format (/, +' Nonpositive definiteness in Cholesky factorization in formk;',/, +' refresh the lbfgs memory and restart the iteration.') 1007 format (/, +' Nonpositive definiteness in Cholesky factorization in formt;',/, +' refresh the lbfgs memory and restart the iteration.') 1008 format (/, +' Bad direction in the line search;',/, +' refresh the lbfgs memory and restart the iteration.') return end c======================= The end of mainlb ============================= subroutine active(n, l, u, nbd, x, iwhere, iprint, + prjctd, cnstnd, boxed) logical prjctd, cnstnd, boxed integer n, iprint, nbd(n), iwhere(n) double precision x(n), l(n), u(n) c ************ c c Subroutine active c c This subroutine initializes iwhere and projects the initial x to c the feasible set if necessary. c c iwhere is an integer array of dimension n. c On entry iwhere is unspecified. c On exit iwhere(i)=-1 if x(i) has no bounds c 3 if l(i)=u(i) c 0 otherwise. c In cauchy, iwhere is given finer gradations. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer nbdd,i double precision zero parameter (zero=0.0d0) c Initialize nbdd, prjctd, cnstnd and boxed. nbdd = 0 prjctd = .false. cnstnd = .false. boxed = .true. c Project the initial x to the easible set if necessary. do 10 i = 1, n if (nbd(i) .gt. 0) then if (nbd(i) .le. 2 .and. x(i) .le. l(i)) then if (x(i) .lt. l(i)) then prjctd = .true. x(i) = l(i) endif nbdd = nbdd + 1 else if (nbd(i) .ge. 2 .and. x(i) .ge. u(i)) then if (x(i) .gt. u(i)) then prjctd = .true. x(i) = u(i) endif nbdd = nbdd + 1 endif endif 10 continue c Initialize iwhere and assign values to cnstnd and boxed. do 20 i = 1, n if (nbd(i) .ne. 2) boxed = .false. if (nbd(i) .eq. 0) then c this variable is always free iwhere(i) = -1 c otherwise set x(i)=mid(x(i), u(i), l(i)). else cnstnd = .true. if (nbd(i) .eq. 2 .and. u(i) - l(i) .le. zero) then c this variable is always fixed iwhere(i) = 3 else iwhere(i) = 0 endif endif 20 continue if (iprint .ge. 0) then if (prjctd) write (6,*) + 'The initial X is infeasible. Restart with its projection.' if (.not. cnstnd) + write (6,*) 'This problem is unconstrained.' endif if (iprint .gt. 0) write (6,1001) nbdd 1001 format (/,'At X0 ',i9,' variables are exactly at the bounds') return end c======================= The end of active ============================= subroutine bmv(m, sy, wt, col, v, p, info) integer m, col, info double precision sy(m, m), wt(m, m), v(2*col), p(2*col) c ************ c c Subroutine bmv c c This subroutine computes the product of the 2m x 2m middle matrix c in the compact L-BFGS formula of B and a 2m vector v; c it returns the product in p. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c sy is a double precision array of dimension m x m. c On entry sy specifies the matrix S'Y. c On exit sy is unchanged. c c wt is a double precision array of dimension m x m. c On entry wt specifies the upper triangular matrix J' which is c the Cholesky factor of (thetaS'S+LD^(-1)L'). c On exit wt is unchanged. c c col is an integer variable. c On entry col specifies the number of s-vectors (or y-vectors) c stored in the compact L-BFGS formula. c On exit col is unchanged. c c v is a double precision array of dimension 2col. c On entry v specifies vector v. c On exit v is unchanged. c c p is a double precision array of dimension 2col. c On entry p is unspecified. c On exit p is the product Mv. c c info is an integer variable. c On entry info is unspecified. c On exit info = 0 for normal return, c = nonzero for abnormal return when the system c to be solved by dtrsl is singular. c c Subprograms called: c c Linpack ... dtrsl. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,k,i2 double precision sum if (col .eq. 0) return c PART I: solve [ D^(1/2) O ] [ p1 ] = [ v1 ] c [ -L*D^(-1/2) J ] [ p2 ] [ v2 ]. c solve Jp2=v2+LD^(-1)v1. p(col + 1) = v(col + 1) do 20 i = 2, col i2 = col + i sum = 0.0d0 do 10 k = 1, i - 1 sum = sum + sy(i,k)*v(k)/sy(k,k) 10 continue p(i2) = v(i2) + sum 20 continue c Solve the triangular system call dtrsl(wt,m,col,p(col+1),11,info) if (info .ne. 0) return c solve D^(1/2)p1=v1. do 30 i = 1, col p(i) = v(i)/sqrt(sy(i,i)) 30 continue c PART II: solve [ -D^(1/2) D^(-1/2)*L' ] [ p1 ] = [ p1 ] c [ 0 J' ] [ p2 ] [ p2 ]. c solve J^Tp2=p2. call dtrsl(wt,m,col,p(col+1),01,info) if (info .ne. 0) return c compute p1=-D^(-1/2)(p1-D^(-1/2)L'p2) c =-D^(-1/2)p1+D^(-1)L'p2. do 40 i = 1, col p(i) = -p(i)/sqrt(sy(i,i)) 40 continue do 60 i = 1, col sum = 0.d0 do 50 k = i + 1, col sum = sum + sy(k,i)*p(col+k)/sy(i,i) 50 continue p(i) = p(i) + sum 60 continue return end c======================== The end of bmv =============================== subroutine cauchy(n, x, l, u, nbd, g, iorder, iwhere, t, d, xcp, + m, wy, ws, sy, wt, theta, col, head, p, c, wbp, + v, nseg, iprint, sbgnrm, info, epsmch) implicit none integer n, m, head, col, nseg, iprint, info, + nbd(n), iorder(n), iwhere(n) double precision theta, epsmch, + x(n), l(n), u(n), g(n), t(n), d(n), xcp(n), + wy(n, col), ws(n, col), sy(m, m), + wt(m, m), p(2*m), c(2*m), wbp(2*m), v(2*m) c ************ c c Subroutine cauchy c c For given x, l, u, g (with sbgnrm > 0), and a limited memory c BFGS matrix B defined in terms of matrices WY, WS, WT, and c scalars head, col, and theta, this subroutine computes the c generalized Cauchy point (GCP), defined as the first local c minimizer of the quadratic c c Q(x + s) = g's + 1/2 s'Bs c c along the projected gradient direction P(x-tg,l,u). c The routine returns the GCP in xcp. c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c x is a double precision array of dimension n. c On entry x is the starting point for the GCP computation. c On exit x is unchanged. c c l is a double precision array of dimension n. c On entry l is the lower bound of x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound of x. c On exit u is unchanged. c c nbd is an integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, and c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c g is a double precision array of dimension n. c On entry g is the gradient of f(x). g must be a nonzero vector. c On exit g is unchanged. c c iorder is an integer working array of dimension n. c iorder will be used to store the breakpoints in the piecewise c linear path and free variables encountered. On exit, c iorder(1),...,iorder(nleft) are indices of breakpoints c which have not been encountered; c iorder(nleft+1),...,iorder(nbreak) are indices of c encountered breakpoints; and c iorder(nfree),...,iorder(n) are indices of variables which c have no bound constraits along the search direction. c c iwhere is an integer array of dimension n. c On entry iwhere indicates only the permanently fixed (iwhere=3) c or free (iwhere= -1) components of x. c On exit iwhere records the status of the current x variables. c iwhere(i)=-3 if x(i) is free and has bounds, but is not moved c 0 if x(i) is free and has bounds, and is moved c 1 if x(i) is fixed at l(i), and l(i) .ne. u(i) c 2 if x(i) is fixed at u(i), and u(i) .ne. l(i) c 3 if x(i) is always fixed, i.e., u(i)=x(i)=l(i) c -1 if x(i) is always free, i.e., it has no bounds. c c t is a double precision working array of dimension n. c t will be used to store the break points. c c d is a double precision array of dimension n used to store c the Cauchy direction P(x-tg)-x. c c xcp is a double precision array of dimension n used to return the c GCP on exit. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c ws, wy, sy, and wt are double precision arrays. c On entry they store information that defines the c limited memory BFGS matrix: c ws(n,m) stores S, a set of s-vectors; c wy(n,m) stores Y, a set of y-vectors; c sy(m,m) stores S'Y; c wt(m,m) stores the c Cholesky factorization of (theta*S'S+LD^(-1)L'). c On exit these arrays are unchanged. c c theta is a double precision variable. c On entry theta is the scaling factor specifying B_0 = theta I. c On exit theta is unchanged. c c col is an integer variable. c On entry col is the actual number of variable metric c corrections stored so far. c On exit col is unchanged. c c head is an integer variable. c On entry head is the location of the first s-vector (or y-vector) c in S (or Y). c On exit col is unchanged. c c p is a double precision working array of dimension 2m. c p will be used to store the vector p = W^(T)d. c c c is a double precision working array of dimension 2m. c c will be used to store the vector c = W^(T)(xcp-x). c c wbp is a double precision working array of dimension 2m. c wbp will be used to store the row of W corresponding c to a breakpoint. c c v is a double precision working array of dimension 2m. c c nseg is an integer variable. c On exit nseg records the number of quadratic segments explored c in searching for the GCP. c c sg and yg are double precision arrays of dimension m. c On entry sg and yg store S'g and Y'g correspondingly. c On exit they are unchanged. c c iprint is an INTEGER variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c sbgnrm is a double precision variable. c On entry sbgnrm is the norm of the projected gradient at x. c On exit sbgnrm is unchanged. c c info is an integer variable. c On entry info is 0. c On exit info = 0 for normal return, c = nonzero for abnormal return when the the system c used in routine bmv is singular. c c Subprograms called: c c L-BFGS-B Library ... hpsolb, bmv. c c Linpack ... dscal dcopy, daxpy. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: FORTRAN c Subroutines for Large Scale Bound Constrained Optimization'' c Tech. Report, NAM-11, EECS Department, Northwestern University, c 1994. c c (Postscript files of these papers are available via anonymous c ftp to eecs.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ logical xlower,xupper,bnded integer i,j,col2,nfree,nbreak,pointr, + ibp,nleft,ibkmin,iter double precision f1,f2,dt,dtm,tsum,dibp,zibp,dibp2,bkmin, + tu,tl,wmc,wmp,wmw,ddot,tj,tj0,neggi,sbgnrm, + f2_org double precision one,zero parameter (one=1.0d0,zero=0.0d0) c Check the status of the variables, reset iwhere(i) if necessary; c compute the Cauchy direction d and the breakpoints t; initialize c the derivative f1 and the vector p = W'd (for theta = 1). if (sbgnrm .le. zero) then if (iprint .ge. 0) write (6,*) 'Subgnorm = 0. GCP = X.' call dcopy(n,x,1,xcp,1) return endif bnded = .true. nfree = n + 1 nbreak = 0 ibkmin = 0 bkmin = zero col2 = 2*col f1 = zero if (iprint .ge. 99) write (6,3010) c We set p to zero and build it up as we determine d. do 20 i = 1, col2 p(i) = zero 20 continue c In the following loop we determine for each variable its bound c status and its breakpoint, and update p accordingly. c Smallest breakpoint is identified. do 50 i = 1, n neggi = -g(i) if (iwhere(i) .ne. 3 .and. iwhere(i) .ne. -1) then c if x(i) is not a constant and has bounds, c compute the difference between x(i) and its bounds. if (nbd(i) .le. 2) tl = x(i) - l(i) if (nbd(i) .ge. 2) tu = u(i) - x(i) c If a variable is close enough to a bound c we treat it as at bound. xlower = nbd(i) .le. 2 .and. tl .le. zero xupper = nbd(i) .ge. 2 .and. tu .le. zero c reset iwhere(i). iwhere(i) = 0 if (xlower) then if (neggi .le. zero) iwhere(i) = 1 else if (xupper) then if (neggi .ge. zero) iwhere(i) = 2 else if (abs(neggi) .le. zero) iwhere(i) = -3 endif endif pointr = head if (iwhere(i) .ne. 0 .and. iwhere(i) .ne. -1) then d(i) = zero else d(i) = neggi f1 = f1 - neggi*neggi c calculate p := p - W'e_i* (g_i). do 40 j = 1, col p(j) = p(j) + wy(i,pointr)* neggi p(col + j) = p(col + j) + ws(i,pointr)*neggi pointr = mod(pointr,m) + 1 40 continue if (nbd(i) .le. 2 .and. nbd(i) .ne. 0 + .and. neggi .lt. zero) then c x(i) + d(i) is bounded; compute t(i). nbreak = nbreak + 1 iorder(nbreak) = i t(nbreak) = tl/(-neggi) if (nbreak .eq. 1 .or. t(nbreak) .lt. bkmin) then bkmin = t(nbreak) ibkmin = nbreak endif else if (nbd(i) .ge. 2 .and. neggi .gt. zero) then c x(i) + d(i) is bounded; compute t(i). nbreak = nbreak + 1 iorder(nbreak) = i t(nbreak) = tu/neggi if (nbreak .eq. 1 .or. t(nbreak) .lt. bkmin) then bkmin = t(nbreak) ibkmin = nbreak endif else c x(i) + d(i) is not bounded. nfree = nfree - 1 iorder(nfree) = i if (abs(neggi) .gt. zero) bnded = .false. endif endif 50 continue c The indices of the nonzero components of d are now stored c in iorder(1),...,iorder(nbreak) and iorder(nfree),...,iorder(n). c The smallest of the nbreak breakpoints is in t(ibkmin)=bkmin. if (theta .ne. one) then c complete the initialization of p for theta not= one. call dscal(col,theta,p(col+1),1) endif c Initialize GCP xcp = x. call dcopy(n,x,1,xcp,1) if (nbreak .eq. 0 .and. nfree .eq. n + 1) then c is a zero vector, return with the initial xcp as GCP. if (iprint .gt. 100) write (6,1010) (xcp(i), i = 1, n) return endif c Initialize c = W'(xcp - x) = 0. do 60 j = 1, col2 c(j) = zero 60 continue c Initialize derivative f2. f2 = -theta*f1 f2_org = f2 if (col .gt. 0) then call bmv(m,sy,wt,col,p,v,info) if (info .ne. 0) return f2 = f2 - ddot(col2,v,1,p,1) endif dtm = -f1/f2 tsum = zero nseg = 1 if (iprint .ge. 99) + write (6,*) 'There are ',nbreak,' breakpoints ' c If there are no breakpoints, locate the GCP and return. if (nbreak .eq. 0) goto 888 nleft = nbreak iter = 1 tj = zero c------------------- the beginning of the loop ------------------------- 777 continue c Find the next smallest breakpoint; c compute dt = t(nleft) - t(nleft + 1). tj0 = tj if (iter .eq. 1) then c Since we already have the smallest breakpoint we need not do c heapsort yet. Often only one breakpoint is used and the c cost of heapsort is avoided. tj = bkmin ibp = iorder(ibkmin) else if (iter .eq. 2) then c Replace the already used smallest breakpoint with the c breakpoint numbered nbreak > nlast, before heapsort call. if (ibkmin .ne. nbreak) then t(ibkmin) = t(nbreak) iorder(ibkmin) = iorder(nbreak) endif c Update heap structure of breakpoints c (if iter=2, initialize heap). endif call hpsolb(nleft,t,iorder,iter-2) tj = t(nleft) ibp = iorder(nleft) endif dt = tj - tj0 if (dt .ne. zero .and. iprint .ge. 100) then write (6,4011) nseg,f1,f2 write (6,5010) dt write (6,6010) dtm endif c If a minimizer is within this interval, locate the GCP and return. if (dtm .lt. dt) goto 888 c Otherwise fix one variable and c reset the corresponding component of d to zero. tsum = tsum + dt nleft = nleft - 1 iter = iter + 1 dibp = d(ibp) d(ibp) = zero if (dibp .gt. zero) then zibp = u(ibp) - x(ibp) xcp(ibp) = u(ibp) iwhere(ibp) = 2 else zibp = l(ibp) - x(ibp) xcp(ibp) = l(ibp) iwhere(ibp) = 1 endif if (iprint .ge. 100) write (6,*) 'Variable ',ibp,' is fixed.' if (nleft .eq. 0 .and. nbreak .eq. n) then c all n variables are fixed, c return with xcp as GCP. dtm = dt goto 999 endif c Update the derivative information. nseg = nseg + 1 dibp2 = dibp**2 c Update f1 and f2. c temporarily set f1 and f2 for col=0. f1 = f1 + dt*f2 + dibp2 - theta*dibp*zibp f2 = f2 - theta*dibp2 if (col .gt. 0) then c update c = c + dt*p. call daxpy(col2,dt,p,1,c,1) c choose wbp, c the row of W corresponding to the breakpoint encountered. pointr = head do 70 j = 1,col wbp(j) = wy(ibp,pointr) wbp(col + j) = theta*ws(ibp,pointr) pointr = mod(pointr,m) + 1 70 continue c compute (wbp)Mc, (wbp)Mp, and (wbp)M(wbp)'. call bmv(m,sy,wt,col,wbp,v,info) if (info .ne. 0) return wmc = ddot(col2,c,1,v,1) wmp = ddot(col2,p,1,v,1) wmw = ddot(col2,wbp,1,v,1) c update p = p - dibp*wbp. call daxpy(col2,-dibp,wbp,1,p,1) c complete updating f1 and f2 while col > 0. f1 = f1 + dibp*wmc f2 = f2 + 2.0d0*dibp*wmp - dibp2*wmw endif f2 = max(epsmch*f2_org,f2) if (nleft .gt. 0) then dtm = -f1/f2 goto 777 c to repeat the loop for unsearched intervals. else if(bnded) then f1 = zero f2 = zero dtm = zero else dtm = -f1/f2 endif c------------------- the end of the loop ------------------------------- 888 continue if (iprint .ge. 99) then write (6,*) write (6,*) 'GCP found in this segment' write (6,4010) nseg,f1,f2 write (6,6010) dtm endif if (dtm .le. zero) dtm = zero tsum = tsum + dtm c Move free variables (i.e., the ones w/o breakpoints) and c the variables whose breakpoints haven't been reached. call daxpy(n,tsum,d,1,xcp,1) 999 continue c Update c = c + dtm*p = W'(x^c - x) c which will be used in computing r = Z'(B(x^c - x) + g). if (col .gt. 0) call daxpy(col2,dtm,p,1,c,1) if (iprint .gt. 100) write (6,1010) (xcp(i),i = 1,n) if (iprint .ge. 99) write (6,2010) 1010 format ('Cauchy X = ',/,(4x,1p,6(1x,d11.4))) 2010 format (/,'---------------- exit CAUCHY----------------------',/) 3010 format (/,'---------------- CAUCHY entered-------------------') 4010 format ('Piece ',i3,' --f1, f2 at start point ',1p,2(1x,d11.4)) 4011 format (/,'Piece ',i3,' --f1, f2 at start point ', + 1p,2(1x,d11.4)) 5010 format ('Distance to the next break point = ',1p,d11.4) 6010 format ('Distance to the stationary point = ',1p,d11.4) return end c====================== The end of cauchy ============================== subroutine cmprlb(n, m, x, g, ws, wy, sy, wt, z, r, wa, index, + theta, col, head, nfree, cnstnd, info) logical cnstnd integer n, m, col, head, nfree, info, index(n) double precision theta, + x(n), g(n), z(n), r(n), wa(4*m), + ws(n, m), wy(n, m), sy(m, m), wt(m, m) c ************ c c Subroutine cmprlb c c This subroutine computes r=-Z'B(xcp-xk)-Z'g by using c wa(2m+1)=W'(xcp-x) from subroutine cauchy. c c Subprograms called: c c L-BFGS-B Library ... bmv. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,j,k,pointr double precision a1,a2 if (.not. cnstnd .and. col .gt. 0) then do 26 i = 1, n r(i) = -g(i) 26 continue else do 30 i = 1, nfree k = index(i) r(i) = -theta*(z(k) - x(k)) - g(k) 30 continue call bmv(m,sy,wt,col,wa(2*m+1),wa(1),info) if (info .ne. 0) then info = -8 return endif pointr = head do 34 j = 1, col a1 = wa(j) a2 = theta*wa(col + j) do 32 i = 1, nfree k = index(i) r(i) = r(i) + wy(k,pointr)*a1 + ws(k,pointr)*a2 32 continue pointr = mod(pointr,m) + 1 34 continue endif return end c======================= The end of cmprlb ============================= subroutine errclb(n, m, factr, l, u, nbd, task, info, k) character*60 task integer n, m, info, k, nbd(n) double precision factr, l(n), u(n) c ************ c c Subroutine errclb c c This subroutine checks the validity of the input data. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i double precision one,zero parameter (one=1.0d0,zero=0.0d0) c Check the input arguments for errors. if (n .le. 0) task = 'ERROR: N .LE. 0' if (m .le. 0) task = 'ERROR: M .LE. 0' if (factr .lt. zero) task = 'ERROR: FACTR .LT. 0' c Check the validity of the arrays nbd(i), u(i), and l(i). do 10 i = 1, n if (nbd(i) .lt. 0 .or. nbd(i) .gt. 3) then c return task = 'ERROR: INVALID NBD' info = -6 k = i endif if (nbd(i) .eq. 2) then if (l(i) .gt. u(i)) then c return task = 'ERROR: NO FEASIBLE SOLUTION' info = -7 k = i endif endif 10 continue return end c======================= The end of errclb ============================= subroutine formk(n, nsub, ind, nenter, ileave, indx2, iupdat, + updatd, wn, wn1, m, ws, wy, sy, theta, col, + head, info) integer n, nsub, m, col, head, nenter, ileave, iupdat, + info, ind(n), indx2(n) double precision theta, wn(2*m, 2*m), wn1(2*m, 2*m), + ws(n, m), wy(n, m), sy(m, m) logical updatd c ************ c c Subroutine formk c c This subroutine forms the LEL^T factorization of the indefinite c c matrix K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c where E = [-I 0] c [ 0 I] c The matrix K can be shown to be equal to the matrix M^[-1]N c occurring in section 5.1 of [1], as well as to the matrix c Mbar^[-1] Nbar in section 5.3. c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c nsub is an integer variable c On entry nsub is the number of subspace variables in free set. c On exit nsub is not changed. c c ind is an integer array of dimension nsub. c On entry ind specifies the indices of subspace variables. c On exit ind is unchanged. c c nenter is an integer variable. c On entry nenter is the number of variables entering the c free set. c On exit nenter is unchanged. c c ileave is an integer variable. c On entry indx2(ileave),...,indx2(n) are the variables leaving c the free set. c On exit ileave is unchanged. c c indx2 is an integer array of dimension n. c On entry indx2(1),...,indx2(nenter) are the variables entering c the free set, while indx2(ileave),...,indx2(n) are the c variables leaving the free set. c On exit indx2 is unchanged. c c iupdat is an integer variable. c On entry iupdat is the total number of BFGS updates made so far. c On exit iupdat is unchanged. c c updatd is a logical variable. c On entry 'updatd' is true if the L-BFGS matrix is updatd. c On exit 'updatd' is unchanged. c c wn is a double precision array of dimension 2m x 2m. c On entry wn is unspecified. c On exit the upper triangle of wn stores the LEL^T factorization c of the 2*col x 2*col indefinite matrix c [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c c wn1 is a double precision array of dimension 2m x 2m. c On entry wn1 stores the lower triangular part of c [Y' ZZ'Y L_a'+R_z'] c [L_a+R_z S'AA'S ] c in the previous iteration. c On exit wn1 stores the corresponding updated matrices. c The purpose of wn1 is just to store these inner products c so they can be easily updated and inserted into wn. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c ws, wy, sy, and wtyy are double precision arrays; c theta is a double precision variable; c col is an integer variable; c head is an integer variable. c On entry they store the information defining the c limited memory BFGS matrix: c ws(n,m) stores S, a set of s-vectors; c wy(n,m) stores Y, a set of y-vectors; c sy(m,m) stores S'Y; c wtyy(m,m) stores the Cholesky factorization c of (theta*S'S+LD^(-1)L') c theta is the scaling factor specifying B_0 = theta I; c col is the number of variable metric corrections stored; c head is the location of the 1st s- (or y-) vector in S (or Y). c On exit they are unchanged. c c info is an integer variable. c On entry info is unspecified. c On exit info = 0 for normal return; c = -1 when the 1st Cholesky factorization failed; c = -2 when the 2st Cholesky factorization failed. c c Subprograms called: c c Linpack ... dcopy, dpofa, dtrsl. c c c References: c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: a c limited memory FORTRAN code for solving bound constrained c optimization problems'', Tech. Report, NAM-11, EECS Department, c Northwestern University, 1994. c c (Postscript files of these papers are available via anonymous c ftp to eecs.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer m2,ipntr,jpntr,iy,is,jy,js,is1,js1,k1,i,k, + col2,pbegin,pend,dbegin,dend,upcl double precision ddot,temp1,temp2,temp3,temp4 double precision one,zero parameter (one=1.0d0,zero=0.0d0) c Form the lower triangular part of c WN1 = [Y' ZZ'Y L_a'+R_z'] c [L_a+R_z S'AA'S ] c where L_a is the strictly lower triangular part of S'AA'Y c R_z is the upper triangular part of S'ZZ'Y. if (updatd) then if (iupdat .gt. m) then c shift old part of WN1. do 10 jy = 1, m - 1 js = m + jy call dcopy(m-jy,wn1(jy+1,jy+1),1,wn1(jy,jy),1) call dcopy(m-jy,wn1(js+1,js+1),1,wn1(js,js),1) call dcopy(m-1,wn1(m+2,jy+1),1,wn1(m+1,jy),1) 10 continue endif c put new rows in blocks (1,1), (2,1) and (2,2). pbegin = 1 pend = nsub dbegin = nsub + 1 dend = n iy = col is = m + col ipntr = head + col - 1 if (ipntr .gt. m) ipntr = ipntr - m jpntr = head do 20 jy = 1, col js = m + jy temp1 = zero temp2 = zero temp3 = zero c compute element jy of row 'col' of Y'ZZ'Y do 15 k = pbegin, pend k1 = ind(k) temp1 = temp1 + wy(k1,ipntr)*wy(k1,jpntr) 15 continue c compute elements jy of row 'col' of L_a and S'AA'S do 16 k = dbegin, dend k1 = ind(k) temp2 = temp2 + ws(k1,ipntr)*ws(k1,jpntr) temp3 = temp3 + ws(k1,ipntr)*wy(k1,jpntr) 16 continue wn1(iy,jy) = temp1 wn1(is,js) = temp2 wn1(is,jy) = temp3 jpntr = mod(jpntr,m) + 1 20 continue c put new column in block (2,1). jy = col jpntr = head + col - 1 if (jpntr .gt. m) jpntr = jpntr - m ipntr = head do 30 i = 1, col is = m + i temp3 = zero c compute element i of column 'col' of R_z do 25 k = pbegin, pend k1 = ind(k) temp3 = temp3 + ws(k1,ipntr)*wy(k1,jpntr) 25 continue ipntr = mod(ipntr,m) + 1 wn1(is,jy) = temp3 30 continue upcl = col - 1 else upcl = col endif c modify the old parts in blocks (1,1) and (2,2) due to changes c in the set of free variables. ipntr = head do 45 iy = 1, upcl is = m + iy jpntr = head do 40 jy = 1, iy js = m + jy temp1 = zero temp2 = zero temp3 = zero temp4 = zero do 35 k = 1, nenter k1 = indx2(k) temp1 = temp1 + wy(k1,ipntr)*wy(k1,jpntr) temp2 = temp2 + ws(k1,ipntr)*ws(k1,jpntr) 35 continue do 36 k = ileave, n k1 = indx2(k) temp3 = temp3 + wy(k1,ipntr)*wy(k1,jpntr) temp4 = temp4 + ws(k1,ipntr)*ws(k1,jpntr) 36 continue wn1(iy,jy) = wn1(iy,jy) + temp1 - temp3 wn1(is,js) = wn1(is,js) - temp2 + temp4 jpntr = mod(jpntr,m) + 1 40 continue ipntr = mod(ipntr,m) + 1 45 continue c modify the old parts in block (2,1). ipntr = head do 60 is = m + 1, m + upcl jpntr = head do 55 jy = 1, upcl temp1 = zero temp3 = zero do 50 k = 1, nenter k1 = indx2(k) temp1 = temp1 + ws(k1,ipntr)*wy(k1,jpntr) 50 continue do 51 k = ileave, n k1 = indx2(k) temp3 = temp3 + ws(k1,ipntr)*wy(k1,jpntr) 51 continue if (is .le. jy + m) then wn1(is,jy) = wn1(is,jy) + temp1 - temp3 else wn1(is,jy) = wn1(is,jy) - temp1 + temp3 endif jpntr = mod(jpntr,m) + 1 55 continue ipntr = mod(ipntr,m) + 1 60 continue c Form the upper triangle of WN = [D+Y' ZZ'Y/theta -L_a'+R_z' ] c [-L_a +R_z S'AA'S*theta] m2 = 2*m do 70 iy = 1, col is = col + iy is1 = m + iy do 65 jy = 1, iy js = col + jy js1 = m + jy wn(jy,iy) = wn1(iy,jy)/theta wn(js,is) = wn1(is1,js1)*theta 65 continue do 66 jy = 1, iy - 1 wn(jy,is) = -wn1(is1,jy) 66 continue do 67 jy = iy, col wn(jy,is) = wn1(is1,jy) 67 continue wn(iy,iy) = wn(iy,iy) + sy(iy,iy) 70 continue c Form the upper triangle of WN= [ LL' L^-1(-L_a'+R_z')] c [(-L_a +R_z)L'^-1 S'AA'S*theta ] c first Cholesky factor (1,1) block of wn to get LL' c with L' stored in the upper triangle of wn. call dpofa(wn,m2,col,info) if (info .ne. 0) then info = -1 return endif c then form L^-1(-L_a'+R_z') in the (1,2) block. col2 = 2*col do 71 js = col+1 ,col2 call dtrsl(wn,m2,col,wn(1,js),11,info) 71 continue c Form S'AA'S*theta + (L^-1(-L_a'+R_z'))'L^-1(-L_a'+R_z') in the c upper triangle of (2,2) block of wn. do 72 is = col+1, col2 do 74 js = is, col2 wn(is,js) = wn(is,js) + ddot(col,wn(1,is),1,wn(1,js),1) 74 continue 72 continue c Cholesky factorization of (2,2) block of wn. call dpofa(wn(col+1,col+1),m2,col,info) if (info .ne. 0) then info = -2 return endif return end c======================= The end of formk ============================== subroutine formt(m, wt, sy, ss, col, theta, info) integer m, col, info double precision theta, wt(m, m), sy(m, m), ss(m, m) c ************ c c Subroutine formt c c This subroutine forms the upper half of the pos. def. and symm. c T = theta*SS + L*D^(-1)*L', stores T in the upper triangle c of the array wt, and performs the Cholesky factorization of T c to produce J*J', with J' stored in the upper triangle of wt. c c Subprograms called: c c Linpack ... dpofa. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,j,k,k1 double precision ddum double precision zero parameter (zero=0.0d0) c Form the upper half of T = theta*SS + L*D^(-1)*L', c store T in the upper triangle of the array wt. do 52 j = 1, col wt(1,j) = theta*ss(1,j) 52 continue do 55 i = 2, col do 54 j = i, col k1 = min(i,j) - 1 ddum = zero do 53 k = 1, k1 ddum = ddum + sy(i,k)*sy(j,k)/sy(k,k) 53 continue wt(i,j) = ddum + theta*ss(i,j) 54 continue 55 continue c Cholesky factorize T to J*J' with c J' stored in the upper triangle of wt. call dpofa(wt,m,col,info) if (info .ne. 0) then info = -3 endif return end c======================= The end of formt ============================== subroutine freev(n, nfree, index, nenter, ileave, indx2, + iwhere, wrk, updatd, cnstnd, iprint, iter) integer n, nfree, nenter, ileave, iprint, iter, + index(n), indx2(n), iwhere(n) logical wrk, updatd, cnstnd c ************ c c Subroutine freev c c This subroutine counts the entering and leaving variables when c iter > 0, and finds the index set of free and active variables c at the GCP. c c cnstnd is a logical variable indicating whether bounds are present c c index is an integer array of dimension n c for i=1,...,nfree, index(i) are the indices of free variables c for i=nfree+1,...,n, index(i) are the indices of bound variables c On entry after the first iteration, index gives c the free variables at the previous iteration. c On exit it gives the free variables based on the determination c in cauchy using the array iwhere. c c indx2 is an integer array of dimension n c On entry indx2 is unspecified. c On exit with iter>0, indx2 indicates which variables c have changed status since the previous iteration. c For i= 1,...,nenter, indx2(i) have changed from bound to free. c For i= ileave+1,...,n, indx2(i) have changed from free to bound. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer iact,i,k nenter = 0 ileave = n + 1 if (iter .gt. 0 .and. cnstnd) then c count the entering and leaving variables. do 20 i = 1, nfree k = index(i) c write(6,*) ' k = index(i) ', k c write(6,*) ' index = ', i if (iwhere(k) .gt. 0) then ileave = ileave - 1 indx2(ileave) = k if (iprint .ge. 100) write (6,*) + 'Variable ',k,' leaves the set of free variables' endif 20 continue do 22 i = 1 + nfree, n k = index(i) if (iwhere(k) .le. 0) then nenter = nenter + 1 indx2(nenter) = k if (iprint .ge. 100) write (6,*) + 'Variable ',k,' enters the set of free variables' endif 22 continue if (iprint .ge. 99) write (6,*) + n+1-ileave,' variables leave; ',nenter,' variables enter' endif wrk = (ileave .lt. n+1) .or. (nenter .gt. 0) .or. updatd c Find the index set of free and active variables at the GCP. nfree = 0 iact = n + 1 do 24 i = 1, n if (iwhere(i) .le. 0) then nfree = nfree + 1 index(nfree) = i else iact = iact - 1 index(iact) = i endif 24 continue if (iprint .ge. 99) write (6,*) + nfree,' variables are free at GCP ',iter + 1 return end c======================= The end of freev ============================== subroutine hpsolb(n, t, iorder, iheap) integer iheap, n, iorder(n) double precision t(n) c ************ c c Subroutine hpsolb c c This subroutine sorts out the least element of t, and puts the c remaining elements of t in a heap. c c n is an integer variable. c On entry n is the dimension of the arrays t and iorder. c On exit n is unchanged. c c t is a double precision array of dimension n. c On entry t stores the elements to be sorted, c On exit t(n) stores the least elements of t, and t(1) to t(n-1) c stores the remaining elements in the form of a heap. c c iorder is an integer array of dimension n. c On entry iorder(i) is the index of t(i). c On exit iorder(i) is still the index of t(i), but iorder may be c permuted in accordance with t. c c iheap is an integer variable specifying the task. c On entry iheap should be set as follows: c iheap .eq. 0 if t(1) to t(n) is not in the form of a heap, c iheap .ne. 0 if otherwise. c On exit iheap is unchanged. c c c References: c Algorithm 232 of CACM (J. W. J. Williams): HEAPSORT. c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c ************ integer i,j,k,indxin,indxou double precision ddum,out if (iheap .eq. 0) then c Rearrange the elements t(1) to t(n) to form a heap. do 20 k = 2, n ddum = t(k) indxin = iorder(k) c Add ddum to the heap. i = k 10 continue if (i.gt.1) then j = i/2 if (ddum .lt. t(j)) then t(i) = t(j) iorder(i) = iorder(j) i = j goto 10 endif endif t(i) = ddum iorder(i) = indxin 20 continue endif c Assign to 'out' the value of t(1), the least member of the heap, c and rearrange the remaining members to form a heap as c elements 1 to n-1 of t. if (n .gt. 1) then i = 1 out = t(1) indxou = iorder(1) ddum = t(n) indxin = iorder(n) c Restore the heap 30 continue j = i+i if (j .le. n-1) then if (t(j+1) .lt. t(j)) j = j+1 if (t(j) .lt. ddum ) then t(i) = t(j) iorder(i) = iorder(j) i = j goto 30 endif endif t(i) = ddum iorder(i) = indxin c Put the least member in t(n). t(n) = out iorder(n) = indxou endif return end c====================== The end of hpsolb ============================== subroutine lnsrlb(n, l, u, nbd, x, f, fold, gd, gdold, g, d, r, t, + z, stp, dnorm, dtd, xstep, stpmx, iter, ifun, + iback, nfgv, info, task, boxed, cnstnd, csave, + isave, dsave) character*60 task, csave logical boxed, cnstnd integer n, iter, ifun, iback, nfgv, info, + nbd(n), isave(2) double precision f, fold, gd, gdold, stp, dnorm, dtd, xstep, + stpmx, x(n), l(n), u(n), g(n), d(n), r(n), t(n), + z(n), dsave(13) c ********** c c Subroutine lnsrlb c c This subroutine calls subroutine dcsrch from the Minpack2 library c to perform the line search. Subroutine dscrch is safeguarded so c that all trial points lie within the feasible region. c c Subprograms called: c c Minpack2 Library ... dcsrch. c c Linpack ... dtrsl, ddot. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ********** integer i double precision ddot,a1,a2 double precision one,zero,big parameter (one=1.0d0,zero=0.0d0,big=1.0d+10) double precision ftol,gtol,xtol parameter (ftol=1.0d-3,gtol=0.9d0,xtol=0.1d0) if (task(1:5) .eq. 'FG_LN') goto 556 dtd = ddot(n,d,1,d,1) dnorm = sqrt(dtd) c Determine the maximum step length. stpmx = big if (cnstnd) then if (iter .eq. 0) then stpmx = one else do 43 i = 1, n a1 = d(i) if (nbd(i) .ne. 0) then if (a1 .lt. zero .and. nbd(i) .le. 2) then a2 = l(i) - x(i) if (a2 .ge. zero) then stpmx = zero else if (a1*stpmx .lt. a2) then stpmx = a2/a1 endif else if (a1 .gt. zero .and. nbd(i) .ge. 2) then a2 = u(i) - x(i) if (a2 .le. zero) then stpmx = zero else if (a1*stpmx .gt. a2) then stpmx = a2/a1 endif endif endif 43 continue endif endif if (iter .eq. 0 .and. .not. boxed) then stp = min(one/dnorm, stpmx) else stp = one endif call dcopy(n,x,1,t,1) call dcopy(n,g,1,r,1) fold = f ifun = 0 iback = 0 csave = 'START' 556 continue gd = ddot(n,g,1,d,1) if (ifun .eq. 0) then gdold=gd if (gd .ge. zero) then c the directional derivative >=0. c Line search is impossible. write(6,*)' ascent direction in projection gd = ', gd info = -4 return endif endif call dcsrch(f,gd,stp,ftol,gtol,xtol,zero,stpmx,csave,isave,dsave) xstep = stp*dnorm if (csave(1:4) .ne. 'CONV' .and. csave(1:4) .ne. 'WARN') then task = 'FG_LNSRCH' ifun = ifun + 1 nfgv = nfgv + 1 iback = ifun - 1 if (stp .eq. one) then call dcopy(n,z,1,x,1) else do 41 i = 1, n x(i) = stp*d(i) + t(i) 41 continue endif else task = 'NEW_X' endif return end c======================= The end of lnsrlb ============================= subroutine matupd(n, m, ws, wy, sy, ss, d, r, itail, + iupdat, col, head, theta, rr, dr, stp, dtd) integer n, m, itail, iupdat, col, head double precision theta, rr, dr, stp, dtd, d(n), r(n), + ws(n, m), wy(n, m), sy(m, m), ss(m, m) c ************ c c Subroutine matupd c c This subroutine updates matrices WS and WY, and forms the c middle matrix in B. c c Subprograms called: c c Linpack ... dcopy, ddot. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer j,pointr double precision ddot double precision one parameter (one=1.0d0) c Set pointers for matrices WS and WY. if (iupdat .le. m) then col = iupdat itail = mod(head+iupdat-2,m) + 1 else itail = mod(itail,m) + 1 head = mod(head,m) + 1 endif c Update matrices WS and WY. call dcopy(n,d,1,ws(1,itail),1) call dcopy(n,r,1,wy(1,itail),1) c Set theta=yy/ys. theta = rr/dr c Form the middle matrix in B. c update the upper triangle of SS, c and the lower triangle of SY: if (iupdat .gt. m) then c move old information do 50 j = 1, col - 1 call dcopy(j,ss(2,j+1),1,ss(1,j),1) call dcopy(col-j,sy(j+1,j+1),1,sy(j,j),1) 50 continue endif c add new information: the last row of SY c and the last column of SS: pointr = head do 51 j = 1, col - 1 sy(col,j) = ddot(n,d,1,wy(1,pointr),1) ss(j,col) = ddot(n,ws(1,pointr),1,d,1) pointr = mod(pointr,m) + 1 51 continue if (stp .eq. one) then ss(col,col) = dtd else ss(col,col) = stp*stp*dtd endif sy(col,col) = dr return end c======================= The end of matupd ============================= subroutine prn1lb(n, m, l, u, x, iprint, itfile, epsmch) integer n, m, iprint, itfile double precision epsmch, x(n), l(n), u(n) c ************ c c Subroutine prn1lb c c This subroutine prints the input data, initial point, upper and c lower bounds of each variable, machine precision, as well as c the headings of the output. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i if (iprint .ge. 0) then write (6,7001) epsmch write (6,*) 'N = ',n,' M = ',m if (iprint .ge. 1) then write (itfile,2001) epsmch write (itfile,*)'N = ',n,' M = ',m write (itfile,9001) if (iprint .gt. 100) then write (6,1004) 'L =',(l(i),i = 1,n) write (6,1004) 'X0 =',(x(i),i = 1,n) write (6,1004) 'U =',(u(i),i = 1,n) endif endif endif 1004 format (/,a4, 1p, 6(1x,d11.4),/,(4x,1p,6(1x,d11.4))) 2001 format ('RUNNING THE L-BFGS-B CODE',/,/, + 'it = iteration number',/, + 'nf = number of function evaluations',/, + 'nseg = number of segments explored during the Cauchy search',/, + 'nact = number of active bounds at the generalized Cauchy point' + ,/, + 'sub = manner in which the subspace minimization terminated:' + ,/,' con = converged, bnd = a bound was reached',/, + 'itls = number of iterations performed in the line search',/, + 'stepl = step length used',/, + 'tstep = norm of the displacement (total step)',/, + 'projg = norm of the projected gradient',/, + 'f = function value',/,/, + ' * * *',/,/, + 'Machine precision =',1p,d10.3) 7001 format ('RUNNING THE L-BFGS-B CODE',/,/, + ' * * *',/,/, + 'Machine precision =',1p,d10.3) 9001 format (/,3x,'it',3x,'nf',2x,'nseg',2x,'nact',2x,'sub',2x,'itls', + 2x,'stepl',4x,'tstep',5x,'projg',8x,'f') return end c======================= The end of prn1lb ============================= subroutine prn2lb(n, x, f, g, iprint, itfile, iter, nfgv, nact, + sbgnrm, nseg, word, iword, iback, stp, xstep) character*3 word integer n, iprint, itfile, iter, nfgv, nact, nseg, + iword, iback double precision f, sbgnrm, stp, xstep, x(n), g(n) c ************ c c Subroutine prn2lb c c This subroutine prints out new information after a successful c line search. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,imod c 'word' records the status of subspace solutions. if (iword .eq. 0) then c the subspace minimization converged. word = 'con' else if (iword .eq. 1) then c the subspace minimization stopped at a bound. word = 'bnd' else if (iword .eq. 5) then c the truncated Newton step has been used. word = 'TNT' else word = '---' endif if (iprint .ge. 99) then write (6,*) 'LINE SEARCH',iback,' times; norm of step = ',xstep write (6,2001) iter,f,sbgnrm if (iprint .gt. 100) then write (6,1004) 'X =',(x(i), i = 1, n) write (6,1004) 'G =',(g(i), i = 1, n) endif else if (iprint .gt. 0) then imod = mod(iter,iprint) if (imod .eq. 0) write (6,2001) iter,f,sbgnrm endif if (iprint .ge. 1) write (itfile,3001) + iter,nfgv,nseg,nact,word,iback,stp,xstep,sbgnrm,f 1004 format (/,a4, 1p, 6(1x,d11.4),/,(4x,1p,6(1x,d11.4))) 2001 format + (/,'At iterate',i5,4x,'f= ',1p,d12.5,4x,'|proj g|= ',1p,d12.5) 3001 format(2(1x,i4),2(1x,i5),2x,a3,1x,i4,1p,2(2x,d7.1),1p,2(1x,d10.3)) return end c======================= The end of prn2lb ============================= subroutine prn3lb(n, x, f, task, iprint, info, itfile, + iter, nfgv, nintol, nskip, nact, sbgnrm, + time, nseg, word, iback, stp, xstep, k, + cachyt, sbtime, lnscht) character*60 task character*3 word integer n, iprint, info, itfile, iter, nfgv, nintol, + nskip, nact, nseg, iback, k double precision f, sbgnrm, time, stp, xstep, cachyt, sbtime, + lnscht, x(n) c ************ c c Subroutine prn3lb c c This subroutine prints out information when either a built-in c convergence test is satisfied or when an error message is c generated. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i if (task(1:5) .eq. 'ERROR') goto 999 if (iprint .ge. 0) then write (6,3003) write (6,3004) write(6,3005) n,iter,nfgv,nintol,nskip,nact,sbgnrm,f if (iprint .ge. 100) then write (6,1004) 'X =',(x(i),i = 1,n) endif if (iprint .ge. 1) write (6,*) ' F =',f endif 999 continue if (iprint .ge. 0) then write (6,3009) task if (info .ne. 0) then if (info .eq. -1) write (6,9011) if (info .eq. -2) write (6,9012) if (info .eq. -3) write (6,9013) if (info .eq. -4) write (6,9014) if (info .eq. -5) write (6,9015) if (info .eq. -6) write (6,*)' Input nbd(',k,') is invalid.' if (info .eq. -7) + write (6,*)' l(',k,') > u(',k,'). No feasible solution.' if (info .eq. -8) write (6,9018) if (info .eq. -9) write (6,9019) endif if (iprint .ge. 1) write (6,3007) cachyt,sbtime,lnscht write (6,3008) time if (iprint .ge. 1) then if (info .eq. -4 .or. info .eq. -9) then write (itfile,3002) + iter,nfgv,nseg,nact,word,iback,stp,xstep endif write (itfile,3009) task if (info .ne. 0) then if (info .eq. -1) write (itfile,9011) if (info .eq. -2) write (itfile,9012) if (info .eq. -3) write (itfile,9013) if (info .eq. -4) write (itfile,9014) if (info .eq. -5) write (itfile,9015) if (info .eq. -8) write (itfile,9018) if (info .eq. -9) write (itfile,9019) endif write (itfile,3008) time endif endif 1004 format (/,a4, 1p, 6(1x,d11.4),/,(4x,1p,6(1x,d11.4))) 3002 format(2(1x,i4),2(1x,i5),2x,a3,1x,i4,1p,2(2x,d7.1),6x,'-',10x,'-') 3003 format (/, + ' * * *',/,/, + 'Tit = total number of iterations',/, + 'Tnf = total number of function evaluations',/, + 'Tnint = total number of segments explored during', + ' Cauchy searches',/, + 'Skip = number of BFGS updates skipped',/, + 'Nact = number of active bounds at final generalized', + ' Cauchy point',/, + 'Projg = norm of the final projected gradient',/, + 'F = final function value',/,/, + ' * * *') 3004 format (/,3x,'N',4x,'Tit',5x,'Tnf',2x,'Tnint',2x, + 'Skip',2x,'Nact',5x,'Projg',8x,'F') 3005 format (i5,2(1x,i6),(1x,i6),(2x,i4),(1x,i5),1p,2(2x,d10.3)) 3007 format (/,' Cauchy time',1p,e10.3,' seconds.',/ + ' Subspace minimization time',1p,e10.3,' seconds.',/ + ' Line search time',1p,e10.3,' seconds.') 3008 format (/,' Total User time',1p,e10.3,' seconds.',/) 3009 format (/,a60) 9011 format (/, +' Matrix in 1st Cholesky factorization in formk is not Pos. Def.') 9012 format (/, +' Matrix in 2st Cholesky factorization in formk is not Pos. Def.') 9013 format (/, +' Matrix in the Cholesky factorization in formt is not Pos. Def.') 9014 format (/, +' Derivative >= 0, backtracking line search impossible.',/, +' Previous x, f and g restored.',/, +' Possible causes: 1 error in function or gradient evaluation;',/, +' 2 rounding errors dominate computation.') 9015 format (/, +' Warning: more than 10 function and gradient',/, +' evaluations in the last line search. Termination',/, +' may possibly be caused by a bad search direction.') 9018 format (/,' The triangular system is singular.') 9019 format (/, +' Line search cannot locate an adequate point after 20 function',/ +,' and gradient evaluations. Previous x, f and g restored.',/, +' Possible causes: 1 error in function or gradient evaluation;',/, +' 2 rounding error dominate computation.') return end c======================= The end of prn3lb ============================= subroutine projgr(n, l, u, nbd, x, g, sbgnrm) integer n, nbd(n) double precision sbgnrm, x(n), l(n), u(n), g(n) c ************ c c Subroutine projgr c c This subroutine computes the infinity norm of the projected c gradient. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i double precision gi double precision one,zero parameter (one=1.0d0,zero=0.0d0) sbgnrm = zero do 15 i = 1, n gi = g(i) if (nbd(i) .ne. 0) then if (gi .lt. zero) then if (nbd(i) .ge. 2) gi = max((x(i)-u(i)),gi) else if (nbd(i) .le. 2) gi = min((x(i)-l(i)),gi) endif endif sbgnrm = max(sbgnrm,abs(gi)) 15 continue return end c======================= The end of projgr ============================= subroutine subsm ( n, m, nsub, ind, l, u, nbd, x, d, xp, ws, wy, + theta, xx, gg, + col, head, iword, wv, wn, iprint, info ) implicit none integer n, m, nsub, col, head, iword, iprint, info, + ind(nsub), nbd(n) double precision theta, + l(n), u(n), x(n), d(n), xp(n), xx(n), gg(n), + ws(n, m), wy(n, m), + wv(2*m), wn(2*m, 2*m) c ********************************************************************** c c This routine contains the major changes in the updated version. c The changes are described in the accompanying paper c c Jose Luis Morales, Jorge Nocedal c "Remark On Algorithm 788: L-BFGS-B: Fortran Subroutines for Large-Scale c Bound Constrained Optimization". Decemmber 27, 2010. c c J.L. Morales Departamento de Matematicas, c Instituto Tecnologico Autonomo de Mexico c Mexico D.F. c c J, Nocedal Department of Electrical Engineering and c Computer Science. c Northwestern University. Evanston, IL. USA c c January 17, 2011 c c ********************************************************************** c c c Subroutine subsm c c Given xcp, l, u, r, an index set that specifies c the active set at xcp, and an l-BFGS matrix B c (in terms of WY, WS, SY, WT, head, col, and theta), c this subroutine computes an approximate solution c of the subspace problem c c (P) min Q(x) = r'(x-xcp) + 1/2 (x-xcp)' B (x-xcp) c c subject to l<=x<=u c x_i=xcp_i for all i in A(xcp) c c along the subspace unconstrained Newton direction c c d = -(Z'BZ)^(-1) r. c c The formula for the Newton direction, given the L-BFGS matrix c and the Sherman-Morrison formula, is c c d = (1/theta)r + (1/theta*2) Z'WK^(-1)W'Z r. c c where c K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c c Note that this procedure for computing d differs c from that described in [1]. One can show that the matrix K is c equal to the matrix M^[-1]N in that paper. c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c nsub is an integer variable. c On entry nsub is the number of free variables. c On exit nsub is unchanged. c c ind is an integer array of dimension nsub. c On entry ind specifies the coordinate indices of free variables. c On exit ind is unchanged. c c l is a double precision array of dimension n. c On entry l is the lower bound of x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound of x. c On exit u is unchanged. c c nbd is a integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, and c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c x is a double precision array of dimension n. c On entry x specifies the Cauchy point xcp. c On exit x(i) is the minimizer of Q over the subspace of c free variables. c c d is a double precision array of dimension n. c On entry d is the reduced gradient of Q at xcp. c On exit d is the Newton direction of Q. c c xp is a double precision array of dimension n. c used to safeguard the projected Newton direction c c xx is a double precision array of dimension n c On entry it holds the current iterate c On output it is unchanged c gg is a double precision array of dimension n c On entry it holds the gradient at the current iterate c On output it is unchanged c c ws and wy are double precision arrays; c theta is a double precision variable; c col is an integer variable; c head is an integer variable. c On entry they store the information defining the c limited memory BFGS matrix: c ws(n,m) stores S, a set of s-vectors; c wy(n,m) stores Y, a set of y-vectors; c theta is the scaling factor specifying B_0 = theta I; c col is the number of variable metric corrections stored; c head is the location of the 1st s- (or y-) vector in S (or Y). c On exit they are unchanged. c c iword is an integer variable. c On entry iword is unspecified. c On exit iword specifies the status of the subspace solution. c iword = 0 if the solution is in the box, c 1 if some bound is encountered. c c wv is a double precision working array of dimension 2m. c c wn is a double precision array of dimension 2m x 2m. c On entry the upper triangle of wn stores the LEL^T factorization c of the indefinite matrix c c K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c where E = [-I 0] c [ 0 I] c On exit wn is unchanged. c c iprint is an INTEGER variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c info is an integer variable. c On entry info is unspecified. c On exit info = 0 for normal return, c = nonzero for abnormal return c when the matrix K is ill-conditioned. c c Subprograms called: c c Linpack dtrsl. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer pointr,m2,col2,ibd,jy,js,i,j,k double precision alpha, xk, dk, temp1, temp2 double precision one,zero parameter (one=1.0d0,zero=0.0d0) c double precision dd_p if (nsub .le. 0) return if (iprint .ge. 99) write (6,1001) c Compute wv = W'Zd. pointr = head do 20 i = 1, col temp1 = zero temp2 = zero do 10 j = 1, nsub k = ind(j) temp1 = temp1 + wy(k,pointr)*d(j) temp2 = temp2 + ws(k,pointr)*d(j) 10 continue wv(i) = temp1 wv(col + i) = theta*temp2 pointr = mod(pointr,m) + 1 20 continue c Compute wv:=K^(-1)wv. m2 = 2*m col2 = 2*col call dtrsl(wn,m2,col2,wv,11,info) if (info .ne. 0) return do 25 i = 1, col wv(i) = -wv(i) 25 continue call dtrsl(wn,m2,col2,wv,01,info) if (info .ne. 0) return c Compute d = (1/theta)d + (1/theta**2)Z'W wv. pointr = head do 40 jy = 1, col js = col + jy do 30 i = 1, nsub k = ind(i) d(i) = d(i) + wy(k,pointr)*wv(jy)/theta + + ws(k,pointr)*wv(js) 30 continue pointr = mod(pointr,m) + 1 40 continue call dscal( nsub, one/theta, d, 1 ) c c----------------------------------------------------------------- c Let us try the projection, d is the Newton direction iword = 0 call dcopy ( n, x, 1, xp, 1 ) c do 50 i=1, nsub k = ind(i) dk = d(i) xk = x(k) if ( nbd(k) .ne. 0 ) then c if ( nbd(k).eq.1 ) then ! lower bounds only x(k) = max( l(k), xk + dk ) if ( x(k).eq.l(k) ) iword = 1 else c if ( nbd(k).eq.2 ) then ! upper and lower bounds xk = max( l(k), xk + dk ) x(k) = min( u(k), xk ) if ( x(k).eq.l(k) .or. x(k).eq.u(k) ) iword = 1 else c if ( nbd(k).eq.3 ) then ! upper bounds only x(k) = min( u(k), xk + dk ) if ( x(k).eq.u(k) ) iword = 1 end if end if end if c else ! free variables x(k) = xk + dk end if 50 continue c if ( iword.eq.0 ) then go to 911 end if c c check sign of the directional derivative c dd_p = zero do 55 i=1, n dd_p = dd_p + (x(i) - xx(i))*gg(i) 55 continue if ( dd_p .gt.zero ) then call dcopy( n, xp, 1, x, 1 ) write(6,*) ' Positive dir derivative in projection ' write(6,*) ' Using the backtracking step ' else go to 911 endif c c----------------------------------------------------------------- c alpha = one temp1 = alpha ibd = 0 do 60 i = 1, nsub k = ind(i) dk = d(i) if (nbd(k) .ne. 0) then if (dk .lt. zero .and. nbd(k) .le. 2) then temp2 = l(k) - x(k) if (temp2 .ge. zero) then temp1 = zero else if (dk*alpha .lt. temp2) then temp1 = temp2/dk endif else if (dk .gt. zero .and. nbd(k) .ge. 2) then temp2 = u(k) - x(k) if (temp2 .le. zero) then temp1 = zero else if (dk*alpha .gt. temp2) then temp1 = temp2/dk endif endif if (temp1 .lt. alpha) then alpha = temp1 ibd = i endif endif 60 continue if (alpha .lt. one) then dk = d(ibd) k = ind(ibd) if (dk .gt. zero) then x(k) = u(k) d(ibd) = zero else if (dk .lt. zero) then x(k) = l(k) d(ibd) = zero endif endif do 70 i = 1, nsub k = ind(i) x(k) = x(k) + alpha*d(i) 70 continue cccccc 911 continue if (iprint .ge. 99) write (6,1004) 1001 format (/,'----------------SUBSM entered-----------------',/) 1004 format (/,'----------------exit SUBSM --------------------',/) return end c====================== The end of subsm =============================== subroutine dcsrch(f,g,stp,ftol,gtol,xtol,stpmin,stpmax, + task,isave,dsave) character*(*) task integer isave(2) double precision f,g,stp,ftol,gtol,xtol,stpmin,stpmax double precision dsave(13) c ********** c c Subroutine dcsrch c c This subroutine finds a step that satisfies a sufficient c decrease condition and a curvature condition. c c Each call of the subroutine updates an interval with c endpoints stx and sty. The interval is initially chosen c so that it contains a minimizer of the modified function c c psi(stp) = f(stp) - f(0) - ftol*stp*f'(0). c c If psi(stp) <= 0 and f'(stp) >= 0 for some step, then the c interval is chosen so that it contains a minimizer of f. c c The algorithm is designed to find a step that satisfies c the sufficient decrease condition c c f(stp) <= f(0) + ftol*stp*f'(0), c c and the curvature condition c c abs(f'(stp)) <= gtol*abs(f'(0)). c c If ftol is less than gtol and if, for example, the function c is bounded below, then there is always a step which satisfies c both conditions. c c If no step can be found that satisfies both conditions, then c the algorithm stops with a warning. In this case stp only c satisfies the sufficient decrease condition. c c A typical invocation of dcsrch has the following outline: c c task = 'START' c 10 continue c call dcsrch( ... ) c if (task .eq. 'FG') then c Evaluate the function and the gradient at stp c goto 10 c end if c c NOTE: The user must no alter work arrays between calls. c c The subroutine statement is c c subroutine dcsrch(f,g,stp,ftol,gtol,xtol,stpmin,stpmax, c task,isave,dsave) c where c c f is a double precision variable. c On initial entry f is the value of the function at 0. c On subsequent entries f is the value of the c function at stp. c On exit f is the value of the function at stp. c c g is a double precision variable. c On initial entry g is the derivative of the function at 0. c On subsequent entries g is the derivative of the c function at stp. c On exit g is the derivative of the function at stp. c c stp is a double precision variable. c On entry stp is the current estimate of a satisfactory c step. On initial entry, a positive initial estimate c must be provided. c On exit stp is the current estimate of a satisfactory step c if task = 'FG'. If task = 'CONV' then stp satisfies c the sufficient decrease and curvature condition. c c ftol is a double precision variable. c On entry ftol specifies a nonnegative tolerance for the c sufficient decrease condition. c On exit ftol is unchanged. c c gtol is a double precision variable. c On entry gtol specifies a nonnegative tolerance for the c curvature condition. c On exit gtol is unchanged. c c xtol is a double precision variable. c On entry xtol specifies a nonnegative relative tolerance c for an acceptable step. The subroutine exits with a c warning if the relative difference between sty and stx c is less than xtol. c On exit xtol is unchanged. c c stpmin is a double precision variable. c On entry stpmin is a nonnegative lower bound for the step. c On exit stpmin is unchanged. c c stpmax is a double precision variable. c On entry stpmax is a nonnegative upper bound for the step. c On exit stpmax is unchanged. c c task is a character variable of length at least 60. c On initial entry task must be set to 'START'. c On exit task indicates the required action: c c If task(1:2) = 'FG' then evaluate the function and c derivative at stp and call dcsrch again. c c If task(1:4) = 'CONV' then the search is successful. c c If task(1:4) = 'WARN' then the subroutine is not able c to satisfy the convergence conditions. The exit value of c stp contains the best point found during the search. c c If task(1:5) = 'ERROR' then there is an error in the c input arguments. c c On exit with convergence, a warning or an error, the c variable task contains additional information. c c isave is an integer work array of dimension 2. c c dsave is a double precision work array of dimension 13. c c Subprograms called c c MINPACK-2 ... dcstep c c MINPACK-1 Project. June 1983. c Argonne National Laboratory. c Jorge J. More' and David J. Thuente. c c MINPACK-2 Project. October 1993. c Argonne National Laboratory and University of Minnesota. c Brett M. Averick, Richard G. Carter, and Jorge J. More'. c c ********** double precision zero,p5,p66 parameter(zero=0.0d0,p5=0.5d0,p66=0.66d0) double precision xtrapl,xtrapu parameter(xtrapl=1.1d0,xtrapu=4.0d0) logical brackt integer stage double precision finit,ftest,fm,fx,fxm,fy,fym,ginit,gtest, + gm,gx,gxm,gy,gym,stx,sty,stmin,stmax,width,width1 c Initialization block. if (task(1:5) .eq. 'START') then c Check the input arguments for errors. if (stp .lt. stpmin) task = 'ERROR: STP .LT. STPMIN' if (stp .gt. stpmax) task = 'ERROR: STP .GT. STPMAX' if (g .ge. zero) task = 'ERROR: INITIAL G .GE. ZERO' if (ftol .lt. zero) task = 'ERROR: FTOL .LT. ZERO' if (gtol .lt. zero) task = 'ERROR: GTOL .LT. ZERO' if (xtol .lt. zero) task = 'ERROR: XTOL .LT. ZERO' if (stpmin .lt. zero) task = 'ERROR: STPMIN .LT. ZERO' if (stpmax .lt. stpmin) task = 'ERROR: STPMAX .LT. STPMIN' c Exit if there are errors on input. if (task(1:5) .eq. 'ERROR') return c Initialize local variables. brackt = .false. stage = 1 finit = f ginit = g gtest = ftol*ginit width = stpmax - stpmin width1 = width/p5 c The variables stx, fx, gx contain the values of the step, c function, and derivative at the best step. c The variables sty, fy, gy contain the value of the step, c function, and derivative at sty. c The variables stp, f, g contain the values of the step, c function, and derivative at stp. stx = zero fx = finit gx = ginit sty = zero fy = finit gy = ginit stmin = zero stmax = stp + xtrapu*stp task = 'FG' goto 1000 else c Restore local variables. if (isave(1) .eq. 1) then brackt = .true. else brackt = .false. endif stage = isave(2) ginit = dsave(1) gtest = dsave(2) gx = dsave(3) gy = dsave(4) finit = dsave(5) fx = dsave(6) fy = dsave(7) stx = dsave(8) sty = dsave(9) stmin = dsave(10) stmax = dsave(11) width = dsave(12) width1 = dsave(13) endif c If psi(stp) <= 0 and f'(stp) >= 0 for some step, then the c algorithm enters the second stage. ftest = finit + stp*gtest if (stage .eq. 1 .and. f .le. ftest .and. g .ge. zero) + stage = 2 c Test for warnings. if (brackt .and. (stp .le. stmin .or. stp .ge. stmax)) + task = 'WARNING: ROUNDING ERRORS PREVENT PROGRESS' if (brackt .and. stmax - stmin .le. xtol*stmax) + task = 'WARNING: XTOL TEST SATISFIED' if (stp .eq. stpmax .and. f .le. ftest .and. g .le. gtest) + task = 'WARNING: STP = STPMAX' if (stp .eq. stpmin .and. (f .gt. ftest .or. g .ge. gtest)) + task = 'WARNING: STP = STPMIN' c Test for convergence. if (f .le. ftest .and. abs(g) .le. gtol*(-ginit)) + task = 'CONVERGENCE' c Test for termination. if (task(1:4) .eq. 'WARN' .or. task(1:4) .eq. 'CONV') goto 1000 c A modified function is used to predict the step during the c first stage if a lower function value has been obtained but c the decrease is not sufficient. if (stage .eq. 1 .and. f .le. fx .and. f .gt. ftest) then c Define the modified function and derivative values. fm = f - stp*gtest fxm = fx - stx*gtest fym = fy - sty*gtest gm = g - gtest gxm = gx - gtest gym = gy - gtest c Call dcstep to update stx, sty, and to compute the new step. call dcstep(stx,fxm,gxm,sty,fym,gym,stp,fm,gm, + brackt,stmin,stmax) c Reset the function and derivative values for f. fx = fxm + stx*gtest fy = fym + sty*gtest gx = gxm + gtest gy = gym + gtest else c Call dcstep to update stx, sty, and to compute the new step. call dcstep(stx,fx,gx,sty,fy,gy,stp,f,g, + brackt,stmin,stmax) endif c Decide if a bisection step is needed. if (brackt) then if (abs(sty-stx) .ge. p66*width1) stp = stx + p5*(sty - stx) width1 = width width = abs(sty-stx) endif c Set the minimum and maximum steps allowed for stp. if (brackt) then stmin = min(stx,sty) stmax = max(stx,sty) else stmin = stp + xtrapl*(stp - stx) stmax = stp + xtrapu*(stp - stx) endif c Force the step to be within the bounds stpmax and stpmin. stp = max(stp,stpmin) stp = min(stp,stpmax) c If further progress is not possible, let stp be the best c point obtained during the search. if (brackt .and. (stp .le. stmin .or. stp .ge. stmax) + .or. (brackt .and. stmax-stmin .le. xtol*stmax)) stp = stx c Obtain another function and derivative. task = 'FG' 1000 continue c Save local variables. if (brackt) then isave(1) = 1 else isave(1) = 0 endif isave(2) = stage dsave(1) = ginit dsave(2) = gtest dsave(3) = gx dsave(4) = gy dsave(5) = finit dsave(6) = fx dsave(7) = fy dsave(8) = stx dsave(9) = sty dsave(10) = stmin dsave(11) = stmax dsave(12) = width dsave(13) = width1 return end c====================== The end of dcsrch ============================== subroutine dcstep(stx,fx,dx,sty,fy,dy,stp,fp,dp,brackt, + stpmin,stpmax) logical brackt double precision stx,fx,dx,sty,fy,dy,stp,fp,dp,stpmin,stpmax c ********** c c Subroutine dcstep c c This subroutine computes a safeguarded step for a search c procedure and updates an interval that contains a step that c satisfies a sufficient decrease and a curvature condition. c c The parameter stx contains the step with the least function c value. If brackt is set to .true. then a minimizer has c been bracketed in an interval with endpoints stx and sty. c The parameter stp contains the current step. c The subroutine assumes that if brackt is set to .true. then c c min(stx,sty) < stp < max(stx,sty), c c and that the derivative at stx is negative in the direction c of the step. c c The subroutine statement is c c subroutine dcstep(stx,fx,dx,sty,fy,dy,stp,fp,dp,brackt, c stpmin,stpmax) c c where c c stx is a double precision variable. c On entry stx is the best step obtained so far and is an c endpoint of the interval that contains the minimizer. c On exit stx is the updated best step. c c fx is a double precision variable. c On entry fx is the function at stx. c On exit fx is the function at stx. c c dx is a double precision variable. c On entry dx is the derivative of the function at c stx. The derivative must be negative in the direction of c the step, that is, dx and stp - stx must have opposite c signs. c On exit dx is the derivative of the function at stx. c c sty is a double precision variable. c On entry sty is the second endpoint of the interval that c contains the minimizer. c On exit sty is the updated endpoint of the interval that c contains the minimizer. c c fy is a double precision variable. c On entry fy is the function at sty. c On exit fy is the function at sty. c c dy is a double precision variable. c On entry dy is the derivative of the function at sty. c On exit dy is the derivative of the function at the exit sty. c c stp is a double precision variable. c On entry stp is the current step. If brackt is set to .true. c then on input stp must be between stx and sty. c On exit stp is a new trial step. c c fp is a double precision variable. c On entry fp is the function at stp c On exit fp is unchanged. c c dp is a double precision variable. c On entry dp is the the derivative of the function at stp. c On exit dp is unchanged. c c brackt is an logical variable. c On entry brackt specifies if a minimizer has been bracketed. c Initially brackt must be set to .false. c On exit brackt specifies if a minimizer has been bracketed. c When a minimizer is bracketed brackt is set to .true. c c stpmin is a double precision variable. c On entry stpmin is a lower bound for the step. c On exit stpmin is unchanged. c c stpmax is a double precision variable. c On entry stpmax is an upper bound for the step. c On exit stpmax is unchanged. c c MINPACK-1 Project. June 1983 c Argonne National Laboratory. c Jorge J. More' and David J. Thuente. c c MINPACK-2 Project. October 1993. c Argonne National Laboratory and University of Minnesota. c Brett M. Averick and Jorge J. More'. c c ********** double precision zero,p66,two,three parameter(zero=0.0d0,p66=0.66d0,two=2.0d0,three=3.0d0) double precision gamma,p,q,r,s,sgnd,stpc,stpf,stpq,theta sgnd = dp*(dx/abs(dx)) c First case: A higher function value. The minimum is bracketed. c If the cubic step is closer to stx than the quadratic step, the c cubic step is taken, otherwise the average of the cubic and c quadratic steps is taken. if (fp .gt. fx) then theta = three*(fx - fp)/(stp - stx) + dx + dp s = max(abs(theta),abs(dx),abs(dp)) gamma = s*sqrt((theta/s)**2 - (dx/s)*(dp/s)) if (stp .lt. stx) gamma = -gamma p = (gamma - dx) + theta q = ((gamma - dx) + gamma) + dp r = p/q stpc = stx + r*(stp - stx) stpq = stx + ((dx/((fx - fp)/(stp - stx) + dx))/two)* + (stp - stx) if (abs(stpc-stx) .lt. abs(stpq-stx)) then stpf = stpc else stpf = stpc + (stpq - stpc)/two endif brackt = .true. c Second case: A lower function value and derivatives of opposite c sign. The minimum is bracketed. If the cubic step is farther from c stp than the secant step, the cubic step is taken, otherwise the c secant step is taken. else if (sgnd .lt. zero) then theta = three*(fx - fp)/(stp - stx) + dx + dp s = max(abs(theta),abs(dx),abs(dp)) gamma = s*sqrt((theta/s)**2 - (dx/s)*(dp/s)) if (stp .gt. stx) gamma = -gamma p = (gamma - dp) + theta q = ((gamma - dp) + gamma) + dx r = p/q stpc = stp + r*(stx - stp) stpq = stp + (dp/(dp - dx))*(stx - stp) if (abs(stpc-stp) .gt. abs(stpq-stp)) then stpf = stpc else stpf = stpq endif brackt = .true. c Third case: A lower function value, derivatives of the same sign, c and the magnitude of the derivative decreases. else if (abs(dp) .lt. abs(dx)) then c The cubic step is computed only if the cubic tends to infinity c in the direction of the step or if the minimum of the cubic c is beyond stp. Otherwise the cubic step is defined to be the c secant step. theta = three*(fx - fp)/(stp - stx) + dx + dp s = max(abs(theta),abs(dx),abs(dp)) c The case gamma = 0 only arises if the cubic does not tend c to infinity in the direction of the step. gamma = s*sqrt(max(zero,(theta/s)**2-(dx/s)*(dp/s))) if (stp .gt. stx) gamma = -gamma p = (gamma - dp) + theta q = (gamma + (dx - dp)) + gamma r = p/q if (r .lt. zero .and. gamma .ne. zero) then stpc = stp + r*(stx - stp) else if (stp .gt. stx) then stpc = stpmax else stpc = stpmin endif stpq = stp + (dp/(dp - dx))*(stx - stp) if (brackt) then c A minimizer has been bracketed. If the cubic step is c closer to stp than the secant step, the cubic step is c taken, otherwise the secant step is taken. if (abs(stpc-stp) .lt. abs(stpq-stp)) then stpf = stpc else stpf = stpq endif if (stp .gt. stx) then stpf = min(stp+p66*(sty-stp),stpf) else stpf = max(stp+p66*(sty-stp),stpf) endif else c A minimizer has not been bracketed. If the cubic step is c farther from stp than the secant step, the cubic step is c taken, otherwise the secant step is taken. if (abs(stpc-stp) .gt. abs(stpq-stp)) then stpf = stpc else stpf = stpq endif stpf = min(stpmax,stpf) stpf = max(stpmin,stpf) endif c Fourth case: A lower function value, derivatives of the same sign, c and the magnitude of the derivative does not decrease. If the c minimum is not bracketed, the step is either stpmin or stpmax, c otherwise the cubic step is taken. else if (brackt) then theta = three*(fp - fy)/(sty - stp) + dy + dp s = max(abs(theta),abs(dy),abs(dp)) gamma = s*sqrt((theta/s)**2 - (dy/s)*(dp/s)) if (stp .gt. sty) gamma = -gamma p = (gamma - dp) + theta q = ((gamma - dp) + gamma) + dy r = p/q stpc = stp + r*(sty - stp) stpf = stpc else if (stp .gt. stx) then stpf = stpmax else stpf = stpmin endif endif c Update the interval which contains a minimizer. if (fp .gt. fx) then sty = stp fy = fp dy = dp else if (sgnd .lt. zero) then sty = stx fy = fx dy = dx endif stx = stp fx = fp dx = dp endif c Compute the new step. stp = stpf return end gpml-matlab-v3.6-2015-07-07/util/lbfgsb/solver_2_4.f000644 000765 000024 00000423410 12313000560 021421 0ustar00carlstaff000000 000000 c================ L-BFGS-B (version 2.4) ========================== subroutine setulb(n, m, x, l, u, nbd, f, g, factr, pgtol, wa, iwa, + task, iprint, csave, lsave, isave, dsave) character*60 task, csave logical lsave(4) integer n, m, iprint, + nbd(n), iwa(3*n), isave(44) double precision f, factr, pgtol, x(n), l(n), u(n), g(n), + wa(2*m*n+4*n+11*m*m+8*m), dsave(29) c ************ c c Subroutine setulb c c This subroutine partitions the working arrays wa and iwa, and c then uses the limited memory BFGS method to solve the bound c constrained optimization problem by calling mainlb. c (The direct method will be used in the subspace minimization.) c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c x is a double precision array of dimension n. c On entry x is an approximation to the solution. c On exit x is the current approximation. c c l is a double precision array of dimension n. c On entry l is the lower bound on x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound on x. c On exit u is unchanged. c c nbd is an integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, and c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c f is a double precision variable. c On first entry f is unspecified. c On final exit f is the value of the function at x. c c g is a double precision array of dimension n. c On first entry g is unspecified. c On final exit g is the value of the gradient at x. c c factr is a double precision variable. c On entry factr >= 0 is specified by the user. The iteration c will stop when c c (f^k - f^{k+1})/max{|f^k|,|f^{k+1}|,1} <= factr*epsmch c c where epsmch is the machine precision, which is automatically c generated by the code. Typical values for factr: 1.d+12 for c low accuracy; 1.d+7 for moderate accuracy; 1.d+1 for extremely c high accuracy. c On exit factr is unchanged. c c pgtol is a double precision variable. c On entry pgtol >= 0 is specified by the user. The iteration c will stop when c c max{|proj g_i | i = 1, ..., n} <= pgtol c c where pg_i is the ith component of the projected gradient. c On exit pgtol is unchanged. c c wa is a double precision working array of length c (2mmax + 4)nmax + 11mmax^2 + 8mmax. c c iwa is an integer working array of length 3nmax. c c task is a working string of characters of length 60 indicating c the current job when entering and quitting this subroutine. c c iprint is an integer variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c csave is a working string of characters of length 60. c c lsave is a logical working array of dimension 4. c On exit with 'task' = NEW_X, the following information is c available: c If lsave(1) = .true. then the initial X has been replaced by c its projection in the feasible set; c If lsave(2) = .true. then the problem is constrained; c If lsave(3) = .true. then each variable has upper and lower c bounds; c c isave is an integer working array of dimension 44. c On exit with 'task' = NEW_X, the following information is c available: c isave(22) = the total number of intervals explored in the c search of Cauchy points; c isave(26) = the total number of skipped BFGS updates before c the current iteration; c isave(30) = the number of current iteration; c isave(31) = the total number of BFGS updates prior the current c iteration; c isave(33) = the number of intervals explored in the search of c Cauchy point in the current iteration; c isave(34) = the total number of function and gradient c evaluations; c isave(36) = the number of function value or gradient c evaluations in the current iteration; c if isave(37) = 0 then the subspace argmin is within the box; c if isave(37) = 1 then the subspace argmin is beyond the box; c isave(38) = the number of free variables in the current c iteration; c isave(39) = the number of active constraints in the current c iteration; c n + 1 - isave(40) = the number of variables leaving the set of c active constraints in the current iteration; c isave(41) = the number of variables entering the set of active c constraints in the current iteration. c c dsave is a double precision working array of dimension 29. c On exit with 'task' = NEW_X, the following information is c available: c dsave(1) = current 'theta' in the BFGS matrix; c dsave(2) = f(x) in the previous iteration; c dsave(3) = factr*epsmch; c dsave(4) = 2-norm of the line search direction vector; c dsave(5) = the machine precision epsmch generated by the code; c dsave(7) = the accumulated time spent on searching for c Cauchy points; c dsave(8) = the accumulated time spent on c subspace minimization; c dsave(9) = the accumulated time spent on line search; c dsave(11) = the slope of the line search function at c the current point of line search; c dsave(12) = the maximum relative step length imposed in c line search; c dsave(13) = the infinity norm of the projected gradient; c dsave(14) = the relative step length in the line search; c dsave(15) = the slope of the line search function at c the starting point of the line search; c dsave(16) = the square of the 2-norm of the line search c direction vector. c c Subprograms called: c c L-BFGS-B Library ... mainlb. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: a c limited memory FORTRAN code for solving bound constrained c optimization problems'', Tech. Report, NAM-11, EECS Department, c Northwestern University, 1994. c c (Postscript files of these papers are available via anonymous c ftp to ece.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer l1,l2,l3,lws,lr,lz,lt,ld,lwa,lwy,lsy,lss,lwt,lwn,lsnd if (task .eq. 'START') then isave(1) = m*n isave(2) = m**2 isave(3) = 4*m**2 isave(4) = 1 isave(5) = isave(4) + isave(1) isave(6) = isave(5) + isave(1) isave(7) = isave(6) + isave(2) isave(8) = isave(7) + isave(2) isave(9) = isave(8) isave(10) = isave(9) + isave(2) isave(11) = isave(10) + isave(3) isave(12) = isave(11) + isave(3) isave(13) = isave(12) + n isave(14) = isave(13) + n isave(15) = isave(14) + n isave(16) = isave(15) + n endif l1 = isave(1) l2 = isave(2) l3 = isave(3) lws = isave(4) lwy = isave(5) lsy = isave(6) lss = isave(7) lwt = isave(9) lwn = isave(10) lsnd = isave(11) lz = isave(12) lr = isave(13) ld = isave(14) lt = isave(15) lwa = isave(16) call mainlb(n,m,x,l,u,nbd,f,g,factr,pgtol, + wa(lws),wa(lwy),wa(lsy),wa(lss),wa(lwt), + wa(lwn),wa(lsnd),wa(lz),wa(lr),wa(ld),wa(lt), + wa(lwa),iwa(1),iwa(n+1),iwa(2*n+1),task,iprint, + csave,lsave,isave(22),dsave) return end c======================= The end of setulb ============================= subroutine mainlb(n, m, x, l, u, nbd, f, g, factr, pgtol, ws, wy, + sy, ss, wt, wn, snd, z, r, d, t, wa, + index, iwhere, indx2, task, iprint, + csave, lsave, isave, dsave) character*60 task, csave logical lsave(4) integer n, m, iprint, nbd(n), index(n), + iwhere(n), indx2(n), isave(23) double precision f, factr, pgtol, + x(n), l(n), u(n), g(n), z(n), r(n), d(n), t(n), + wa(8*m), ws(n, m), wy(n, m), sy(m, m), ss(m, m), + wt(m, m), wn(2*m, 2*m), snd(2*m, 2*m), dsave(29) c ************ c c Subroutine mainlb c c This subroutine solves bound constrained optimization problems by c using the compact formula of the limited memory BFGS updates. c c n is an integer variable. c On entry n is the number of variables. c On exit n is unchanged. c c m is an integer variable. c On entry m is the maximum number of variable metric c corrections allowed in the limited memory matrix. c On exit m is unchanged. c c x is a double precision array of dimension n. c On entry x is an approximation to the solution. c On exit x is the current approximation. c c l is a double precision array of dimension n. c On entry l is the lower bound of x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound of x. c On exit u is unchanged. c c nbd is an integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c f is a double precision variable. c On first entry f is unspecified. c On final exit f is the value of the function at x. c c g is a double precision array of dimension n. c On first entry g is unspecified. c On final exit g is the value of the gradient at x. c c factr is a double precision variable. c On entry factr >= 0 is specified by the user. The iteration c will stop when c c (f^k - f^{k+1})/max{|f^k|,|f^{k+1}|,1} <= factr*epsmch c c where epsmch is the machine precision, which is automatically c generated by the code. c On exit factr is unchanged. c c pgtol is a double precision variable. c On entry pgtol >= 0 is specified by the user. The iteration c will stop when c c max{|proj g_i | i = 1, ..., n} <= pgtol c c where pg_i is the ith component of the projected gradient. c On exit pgtol is unchanged. c c ws, wy, sy, and wt are double precision working arrays used to c store the following information defining the limited memory c BFGS matrix: c ws, of dimension n x m, stores S, the matrix of s-vectors; c wy, of dimension n x m, stores Y, the matrix of y-vectors; c sy, of dimension m x m, stores S'Y; c ss, of dimension m x m, stores S'S; c wt, of dimension m x m, stores the Cholesky factorization c of (theta*S'S+LD^(-1)L'); see eq. c (2.26) in [3]. c c wn is a double precision working array of dimension 2m x 2m c used to store the LEL^T factorization of the indefinite matrix c K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c c where E = [-I 0] c [ 0 I] c c snd is a double precision working array of dimension 2m x 2m c used to store the lower triangular part of c N = [Y' ZZ'Y L_a'+R_z'] c [L_a +R_z S'AA'S ] c c z(n),r(n),d(n),t(n),wa(8*m) are double precision working arrays. c z is used at different times to store the Cauchy point and c the Newton point. c c c index is an integer working array of dimension n. c In subroutine freev, index is used to store the free and fixed c variables at the Generalized Cauchy Point (GCP). c c iwhere is an integer working array of dimension n used to record c the status of the vector x for GCP computation. c iwhere(i)=0 or -3 if x(i) is free and has bounds, c 1 if x(i) is fixed at l(i), and l(i) .ne. u(i) c 2 if x(i) is fixed at u(i), and u(i) .ne. l(i) c 3 if x(i) is always fixed, i.e., u(i)=x(i)=l(i) c -1 if x(i) is always free, i.e., no bounds on it. c c indx2 is an integer working array of dimension n. c Within subroutine cauchy, indx2 corresponds to the array iorder. c In subroutine freev, a list of variables entering and leaving c the free set is stored in indx2, and it is passed on to c subroutine formk with this information. c c task is a working string of characters of length 60 indicating c the current job when entering and leaving this subroutine. c c iprint is an INTEGER variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c csave is a working string of characters of length 60. c c lsave is a logical working array of dimension 4. c c isave is an integer working array of dimension 23. c c dsave is a double precision working array of dimension 29. c c c Subprograms called c c L-BFGS-B Library ... cauchy, subsm, lnsrlb, formk, c c errclb, prn1lb, prn2lb, prn3lb, active, projgr, c c freev, cmprlb, matupd, formt. c c Minpack2 Library ... timer, dpmeps. c c Linpack Library ... dcopy, ddot. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: FORTRAN c Subroutines for Large Scale Bound Constrained Optimization'' c Tech. Report, NAM-11, EECS Department, Northwestern University, c 1994. c c [3] R. Byrd, J. Nocedal and R. Schnabel "Representations of c Quasi-Newton Matrices and their use in Limited Memory Methods'', c Mathematical Programming 63 (1994), no. 4, pp. 129-156. c c (Postscript files of these papers are available via anonymous c ftp to ece.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ logical prjctd,cnstnd,boxed,updatd,wrk character*3 word integer i,k,nintol,itfile,iback,nskip, + head,col,iter,itail,iupdat, + nint,nfgv,info,ifun, + iword,nfree,nact,ileave,nenter double precision theta,fold,ddot,dr,rr,tol,dpmeps, + xstep,sbgnrm,ddum,dnorm,dtd,epsmch, + cpu1,cpu2,cachyt,sbtime,lnscht,time1,time2, + gd,gdold,stp,stpmx,time double precision one,zero parameter (one=1.0d0,zero=0.0d0) if (task .eq. 'START') then call timer(time1) c Generate the current machine precision. epsmch = dpmeps() fold = 0.0d0 dnorm = 0.0d0 cpu1 = 0.0d0 gd = 0.0d0 sbgnrm = 0.0d0 stp = 0.0d0 stpmx = 0.0d0 gdold = 0.0d0 dtd = 0.0d0 c Initialize counters and scalars when task='START'. c for the limited memory BFGS matrices: col = 0 head = 1 theta = one iupdat = 0 updatd = .false. iback = 0 itail = 0 ifun = 0 iword = 0 nact = 0 ileave = 0 nenter = 0 c for operation counts: iter = 0 nfgv = 0 nint = 0 nintol = 0 nskip = 0 nfree = n c for stopping tolerance: tol = factr*epsmch c for measuring running time: cachyt = 0 sbtime = 0 lnscht = 0 c 'word' records the status of subspace solutions. word = '---' c 'info' records the termination information. info = 0 itfile = 0 if (iprint .ge. 1) then c open a summary file 'iterate.dat' open (8, file = 'iterate.dat', status = 'unknown') itfile = 8 endif c Check the input arguments for errors. call errclb(n,m,factr,l,u,nbd,task,info,k) if (task(1:5) .eq. 'ERROR') then call prn3lb(n,x,f,task,iprint,info,itfile, + iter,nfgv,nintol,nskip,nact,sbgnrm, + zero,nint,word,iback,stp,xstep,k, + cachyt,sbtime,lnscht) return endif call prn1lb(n,m,l,u,x,iprint,itfile,epsmch) c Initialize iwhere & project x onto the feasible set. call active(n,l,u,nbd,x,iwhere,iprint,prjctd,cnstnd,boxed) c The end of the initialization. else c restore local variables. prjctd = lsave(1) cnstnd = lsave(2) boxed = lsave(3) updatd = lsave(4) nintol = isave(1) itfile = isave(3) iback = isave(4) nskip = isave(5) head = isave(6) col = isave(7) itail = isave(8) iter = isave(9) iupdat = isave(10) nint = isave(12) nfgv = isave(13) info = isave(14) ifun = isave(15) iword = isave(16) nfree = isave(17) nact = isave(18) ileave = isave(19) nenter = isave(20) theta = dsave(1) fold = dsave(2) tol = dsave(3) dnorm = dsave(4) epsmch = dsave(5) cpu1 = dsave(6) cachyt = dsave(7) sbtime = dsave(8) lnscht = dsave(9) time1 = dsave(10) gd = dsave(11) stpmx = dsave(12) sbgnrm = dsave(13) stp = dsave(14) gdold = dsave(15) dtd = dsave(16) c After returning from the driver go to the point where execution c is to resume. if (task(1:5) .eq. 'FG_LN') goto 666 if (task(1:5) .eq. 'NEW_X') goto 777 if (task(1:5) .eq. 'FG_ST') goto 111 if (task(1:4) .eq. 'STOP') then if (task(7:9) .eq. 'CPU') then c restore the previous iterate. call dcopy(n,t,1,x,1) call dcopy(n,r,1,g,1) f = fold endif goto 999 endif endif c Compute f0 and g0. task = 'FG_START' c return to the driver to calculate f and g; reenter at 111. goto 1000 111 continue nfgv = 1 c Compute the infinity norm of the (-) projected gradient. call projgr(n,l,u,nbd,x,g,sbgnrm) if (iprint .ge. 1) then write (6,1002) iter,f,sbgnrm write (itfile,1003) iter,nfgv,sbgnrm,f endif if (sbgnrm .le. pgtol) then c terminate the algorithm. task = 'CONVERGENCE: NORM OF PROJECTED GRADIENT <= PGTOL' goto 999 endif c ----------------- the beginning of the loop -------------------------- 222 continue if (iprint .ge. 99) write (6,1001) iter + 1 iword = -1 c if (.not. cnstnd .and. col .gt. 0) then c skip the search for GCP. call dcopy(n,x,1,z,1) wrk = updatd nint = 0 goto 333 endif cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Compute the Generalized Cauchy Point (GCP). c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc call timer(cpu1) call cauchy(n,x,l,u,nbd,g,indx2,iwhere,t,d,z, + m,wy,ws,sy,wt,theta,col,head, + wa(1),wa(2*m+1),wa(4*m+1),wa(6*m+1),nint, + iprint,sbgnrm,info,epsmch) if (info .ne. 0) then c singular triangular system detected; refresh the lbfgs memory. if(iprint .ge. 1) write (6, 1005) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. call timer(cpu2) cachyt = cachyt + cpu2 - cpu1 goto 222 endif call timer(cpu2) cachyt = cachyt + cpu2 - cpu1 nintol = nintol + nint c Count the entering and leaving variables for iter > 0; c find the index set of free and active variables at the GCP. call freev(n,nfree,index,nenter,ileave,indx2, + iwhere,wrk,updatd,cnstnd,iprint,iter) nact = n - nfree 333 continue c If there are no free variables or B=theta*I, then c skip the subspace minimization. if (nfree .eq. 0 .or. col .eq. 0) goto 555 cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Subspace minimization. c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc call timer(cpu1) c Form the LEL^T factorization of the indefinite c matrix K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c where E = [-I 0] c [ 0 I] if (wrk) call formk(n,nfree,index,nenter,ileave,indx2,iupdat, + updatd,wn,snd,m,ws,wy,sy,theta,col,head,info) if (info .ne. 0) then c nonpositive definiteness in Cholesky factorization; c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1006) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. call timer(cpu2) sbtime = sbtime + cpu2 - cpu1 goto 222 endif c compute r=-Z'B(xcp-xk)-Z'g (using wa(2m+1)=W'(xcp-x) c from 'cauchy'). call cmprlb(n,m,x,g,ws,wy,sy,wt,z,r,wa,index, + theta,col,head,nfree,cnstnd,info) if (info .ne. 0) goto 444 c call the direct method. call subsm(n,m,nfree,index,l,u,nbd,z,r,ws,wy,theta, + col,head,iword,wa,wn,iprint,info) 444 continue if (info .ne. 0) then c singular triangular system detected; c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1005) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. call timer(cpu2) sbtime = sbtime + cpu2 - cpu1 goto 222 endif call timer(cpu2) sbtime = sbtime + cpu2 - cpu1 555 continue cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Line search and optimality tests. c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c Generate the search direction d:=z-x. do 40 i = 1, n d(i) = z(i) - x(i) 40 continue call timer(cpu1) 666 continue call lnsrlb(n,l,u,nbd,x,f,fold,gd,gdold,g,d,r,t,z,stp,dnorm, + dtd,xstep,stpmx,iter,ifun,iback,nfgv,info,task, + boxed,cnstnd,csave,isave(22),dsave(17)) if (info .ne. 0 .or. iback .ge. 20) then c restore the previous iterate. call dcopy(n,t,1,x,1) call dcopy(n,r,1,g,1) f = fold if (col .eq. 0) then c abnormal termination. if (info .eq. 0) then info = -9 c restore the actual number of f and g evaluations etc. nfgv = nfgv - 1 ifun = ifun - 1 iback = iback - 1 endif task = 'ABNORMAL_TERMINATION_IN_LNSRCH' iter = iter + 1 goto 999 else c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1008) if (info .eq. 0) nfgv = nfgv - 1 info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. task = 'RESTART_FROM_LNSRCH' call timer(cpu2) lnscht = lnscht + cpu2 - cpu1 goto 222 endif else if (task(1:5) .eq. 'FG_LN') then c return to the driver for calculating f and g; reenter at 666. goto 1000 else c calculate and print out the quantities related to the new X. call timer(cpu2) lnscht = lnscht + cpu2 - cpu1 iter = iter + 1 c Compute the infinity norm of the projected (-)gradient. call projgr(n,l,u,nbd,x,g,sbgnrm) c Print iteration information. call prn2lb(n,x,f,g,iprint,itfile,iter,nfgv,nact, + sbgnrm,nint,word,iword,iback,stp,xstep) goto 1000 endif 777 continue c Test for termination. if (sbgnrm .le. pgtol) then c terminate the algorithm. task = 'CONVERGENCE: NORM OF PROJECTED GRADIENT <= PGTOL' goto 999 endif ddum = max(abs(fold), abs(f), one) if ((fold - f) .le. tol*ddum) then c terminate the algorithm. task = 'CONVERGENCE: REL_REDUCTION_OF_F <= FACTR*EPSMCH' if (iback .ge. 10) info = -5 c i.e., to issue a warning if iback>10 in the line search. goto 999 endif c Compute d=newx-oldx, r=newg-oldg, rr=y'y and dr=y's. do 42 i = 1, n r(i) = g(i) - r(i) 42 continue rr = ddot(n,r,1,r,1) if (stp .eq. one) then dr = gd - gdold ddum = -gdold else dr = (gd - gdold)*stp call dscal(n,stp,d,1) ddum = -gdold*stp endif if (dr .le. epsmch*ddum) then c skip the L-BFGS update. nskip = nskip + 1 updatd = .false. if (iprint .ge. 1) write (6,1004) dr, ddum goto 888 endif cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Update the L-BFGS matrix. c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc updatd = .true. iupdat = iupdat + 1 c Update matrices WS and WY and form the middle matrix in B. call matupd(n,m,ws,wy,sy,ss,d,r,itail, + iupdat,col,head,theta,rr,dr,stp,dtd) c Form the upper half of the pds T = theta*SS + L*D^(-1)*L'; c Store T in the upper triangular of the array wt; c Cholesky factorize T to J*J' with c J' stored in the upper triangular of wt. call formt(m,wt,sy,ss,col,theta,info) if (info .ne. 0) then c nonpositive definiteness in Cholesky factorization; c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1007) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. goto 222 endif c Now the inverse of the middle matrix in B is c [ D^(1/2) O ] [ -D^(1/2) D^(-1/2)*L' ] c [ -L*D^(-1/2) J ] [ 0 J' ] 888 continue c -------------------- the end of the loop ----------------------------- goto 222 999 continue call timer(time2) time = time2 - time1 call prn3lb(n,x,f,task,iprint,info,itfile, + iter,nfgv,nintol,nskip,nact,sbgnrm, + time,nint,word,iback,stp,xstep,k, + cachyt,sbtime,lnscht) 1000 continue c Save local variables. lsave(1) = prjctd lsave(2) = cnstnd lsave(3) = boxed lsave(4) = updatd isave(1) = nintol isave(3) = itfile isave(4) = iback isave(5) = nskip isave(6) = head isave(7) = col isave(8) = itail isave(9) = iter isave(10) = iupdat isave(12) = nint isave(13) = nfgv isave(14) = info isave(15) = ifun isave(16) = iword isave(17) = nfree isave(18) = nact isave(19) = ileave isave(20) = nenter dsave(1) = theta dsave(2) = fold dsave(3) = tol dsave(4) = dnorm dsave(5) = epsmch dsave(6) = cpu1 dsave(7) = cachyt dsave(8) = sbtime dsave(9) = lnscht dsave(10) = time1 dsave(11) = gd dsave(12) = stpmx dsave(13) = sbgnrm dsave(14) = stp dsave(15) = gdold dsave(16) = dtd 1001 format (//,'ITERATION ',i5) 1002 format + (/,'At iterate',i5,4x,'f= ',1p,d12.5,4x,'|proj g|= ',1p,d12.5) 1003 format (2(1x,i4),5x,'-',5x,'-',3x,'-',5x,'-',5x,'-',8x,'-',3x, + 1p,2(1x,d10.3)) 1004 format (' ys=',1p,e10.3,' -gs=',1p,e10.3,' BFGS update SKIPPED') 1005 format (/, +' Singular triangular system detected;',/, +' refresh the lbfgs memory and restart the iteration.') 1006 format (/, +' Nonpositive definiteness in Cholesky factorization in formk;',/, +' refresh the lbfgs memory and restart the iteration.') 1007 format (/, +' Nonpositive definiteness in Cholesky factorization in formt;',/, +' refresh the lbfgs memory and restart the iteration.') 1008 format (/, +' Bad direction in the line search;',/, +' refresh the lbfgs memory and restart the iteration.') return end c======================= The end of mainlb ============================= subroutine active(n, l, u, nbd, x, iwhere, iprint, + prjctd, cnstnd, boxed) logical prjctd, cnstnd, boxed integer n, iprint, nbd(n), iwhere(n) double precision x(n), l(n), u(n) c ************ c c Subroutine active c c This subroutine initializes iwhere and projects the initial x to c the feasible set if necessary. c c iwhere is an integer array of dimension n. c On entry iwhere is unspecified. c On exit iwhere(i)=-1 if x(i) has no bounds c 3 if l(i)=u(i) c 0 otherwise. c In cauchy, iwhere is given finer gradations. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer nbdd,i double precision zero parameter (zero=0.0d0) c Initialize nbdd, prjctd, cnstnd and boxed. nbdd = 0 prjctd = .false. cnstnd = .false. boxed = .true. c Project the initial x to the easible set if necessary. do 10 i = 1, n if (nbd(i) .gt. 0) then if (nbd(i) .le. 2 .and. x(i) .le. l(i)) then if (x(i) .lt. l(i)) then prjctd = .true. x(i) = l(i) endif nbdd = nbdd + 1 else if (nbd(i) .ge. 2 .and. x(i) .ge. u(i)) then if (x(i) .gt. u(i)) then prjctd = .true. x(i) = u(i) endif nbdd = nbdd + 1 endif endif 10 continue c Initialize iwhere and assign values to cnstnd and boxed. do 20 i = 1, n if (nbd(i) .ne. 2) boxed = .false. if (nbd(i) .eq. 0) then c this variable is always free iwhere(i) = -1 c otherwise set x(i)=mid(x(i), u(i), l(i)). else cnstnd = .true. if (nbd(i) .eq. 2 .and. u(i) - l(i) .le. zero) then c this variable is always fixed iwhere(i) = 3 else iwhere(i) = 0 endif endif 20 continue if (iprint .ge. 0) then if (prjctd) write (6,*) + 'The initial X is infeasible. Restart with its projection.' if (.not. cnstnd) + write (6,*) 'This problem is unconstrained.' endif if (iprint .gt. 0) write (6,1001) nbdd 1001 format (/,'At X0 ',i9,' variables are exactly at the bounds') return end c======================= The end of active ============================= subroutine bmv(m, sy, wt, col, v, p, info) integer m, col, info double precision sy(m, m), wt(m, m), v(2*col), p(2*col) c ************ c c Subroutine bmv c c This subroutine computes the product of the 2m x 2m middle matrix c in the compact L-BFGS formula of B and a 2m vector v; c it returns the product in p. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c sy is a double precision array of dimension m x m. c On entry sy specifies the matrix S'Y. c On exit sy is unchanged. c c wt is a double precision array of dimension m x m. c On entry wt specifies the upper triangular matrix J' which is c the Cholesky factor of (thetaS'S+LD^(-1)L'). c On exit wt is unchanged. c c col is an integer variable. c On entry col specifies the number of s-vectors (or y-vectors) c stored in the compact L-BFGS formula. c On exit col is unchanged. c c v is a double precision array of dimension 2col. c On entry v specifies vector v. c On exit v is unchanged. c c p is a double precision array of dimension 2col. c On entry p is unspecified. c On exit p is the product Mv. c c info is an integer variable. c On entry info is unspecified. c On exit info = 0 for normal return, c = nonzero for abnormal return when the system c to be solved by dtrsl is singular. c c Subprograms called: c c Linpack ... dtrsl. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,k,i2 double precision sum if (col .eq. 0) return c PART I: solve [ D^(1/2) O ] [ p1 ] = [ v1 ] c [ -L*D^(-1/2) J ] [ p2 ] [ v2 ]. c solve Jp2=v2+LD^(-1)v1. p(col + 1) = v(col + 1) do 20 i = 2, col i2 = col + i sum = 0.0d0 do 10 k = 1, i - 1 sum = sum + sy(i,k)*v(k)/sy(k,k) 10 continue p(i2) = v(i2) + sum 20 continue c Solve the triangular system call dtrsl(wt,m,col,p(col+1),11,info) if (info .ne. 0) return c solve D^(1/2)p1=v1. cc do 30 i = 1, col cc p(i) = v(i)/sqrt(sy(i,i)) cc30 continue c PART II: solve [ -D^(1/2) D^(-1/2)*L' ] [ p1 ] = [ p1 ] c [ 0 J' ] [ p2 ] [ p2 ]. c solve J^Tp2=p2. call dtrsl(wt,m,col,p(col+1),01,info) if (info .ne. 0) return c compute p1=-D^(-1/2)(p1-D^(-1/2)L'p2) c =-D^(-1/2)p1+D^(-1)L'p2. do 40 i = 1, col cc p(i) = -p(i)/sqrt(sy(i,i)) combined with do 30 loop cc into the next line p(i) = -v(i)/sy(i,i) 40 continue do 60 i = 1, col sum = 0.d0 do 50 k = i + 1, col sum = sum + sy(k,i)*p(col+k)/sy(i,i) 50 continue p(i) = p(i) + sum 60 continue return end c======================== The end of bmv =============================== subroutine cauchy(n, x, l, u, nbd, g, iorder, iwhere, t, d, xcp, + m, wy, ws, sy, wt, theta, col, head, p, c, wbp, + v, nint, iprint, sbgnrm, info, epsmch) integer n, m, head, col, nint, iprint, info, + nbd(n), iorder(n), iwhere(n) double precision theta, epsmch, + x(n), l(n), u(n), g(n), t(n), d(n), xcp(n), + wy(n, col), ws(n, col), sy(m, m), + wt(m, m), p(2*m), c(2*m), wbp(2*m), v(2*m) c ************ c c Subroutine cauchy c c For given x, l, u, g (with sbgnrm > 0), and a limited memory c BFGS matrix B defined in terms of matrices WY, WS, WT, and c scalars head, col, and theta, this subroutine computes the c generalized Cauchy point (GCP), defined as the first local c minimizer of the quadratic c c Q(x + s) = g's + 1/2 s'Bs c c along the projected gradient direction P(x-tg,l,u). c The routine returns the GCP in xcp. c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c x is a double precision array of dimension n. c On entry x is the starting point for the GCP computation. c On exit x is unchanged. c c l is a double precision array of dimension n. c On entry l is the lower bound of x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound of x. c On exit u is unchanged. c c nbd is an integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, and c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c g is a double precision array of dimension n. c On entry g is the gradient of f(x). g must be a nonzero vector. c On exit g is unchanged. c c iorder is an integer working array of dimension n. c iorder will be used to store the breakpoints in the piecewise c linear path and free variables encountered. On exit, c iorder(1),...,iorder(nleft) are indices of breakpoints c which have not been encountered; c iorder(nleft+1),...,iorder(nbreak) are indices of c encountered breakpoints; and c iorder(nfree),...,iorder(n) are indices of variables which c have no bound constraits along the search direction. c c iwhere is an integer array of dimension n. c On entry iwhere indicates only the permanently fixed (iwhere=3) c or free (iwhere= -1) components of x. c On exit iwhere records the status of the current x variables. c iwhere(i)=-3 if x(i) is free and has bounds, but is not moved c 0 if x(i) is free and has bounds, and is moved c 1 if x(i) is fixed at l(i), and l(i) .ne. u(i) c 2 if x(i) is fixed at u(i), and u(i) .ne. l(i) c 3 if x(i) is always fixed, i.e., u(i)=x(i)=l(i) c -1 if x(i) is always free, i.e., it has no bounds. c c t is a double precision working array of dimension n. c t will be used to store the break points. c c d is a double precision array of dimension n used to store c the Cauchy direction P(x-tg)-x. c c xcp is a double precision array of dimension n used to return the c GCP on exit. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c ws, wy, sy, and wt are double precision arrays. c On entry they store information that defines the c limited memory BFGS matrix: c ws(n,m) stores S, a set of s-vectors; c wy(n,m) stores Y, a set of y-vectors; c sy(m,m) stores S'Y; c wt(m,m) stores the c Cholesky factorization of (theta*S'S+LD^(-1)L'). c On exit these arrays are unchanged. c c theta is a double precision variable. c On entry theta is the scaling factor specifying B_0 = theta I. c On exit theta is unchanged. c c col is an integer variable. c On entry col is the actual number of variable metric c corrections stored so far. c On exit col is unchanged. c c head is an integer variable. c On entry head is the location of the first s-vector c (or y-vector) in S (or Y). c On exit col is unchanged. c c p is a double precision working array of dimension 2m. c p will be used to store the vector p = W^(T)d. c c c is a double precision working array of dimension 2m. c c will be used to store the vector c = W^(T)(xcp-x). c c wbp is a double precision working array of dimension 2m. c wbp will be used to store the row of W corresponding c to a breakpoint. c c v is a double precision working array of dimension 2m. c c nint is an integer variable. c On exit nint records the number of quadratic segments explored c in searching for the GCP. c c iprint is an INTEGER variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c sbgnrm is a double precision variable. c On entry sbgnrm is the norm of the projected gradient at x. c On exit sbgnrm is unchanged. c c info is an integer variable. c On entry info is 0. c On exit info = 0 for normal return, c = nonzero for abnormal return when the the system c used in routine bmv is singular. c c Subprograms called: c c L-BFGS-B Library ... hpsolb, bmv. c c Linpack ... dscal dcopy, daxpy. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: FORTRAN c Subroutines for Large Scale Bound Constrained Optimization'' c Tech. Report, NAM-11, EECS Department, Northwestern University, c 1994. c c (Postscript files of these papers are available via anonymous c ftp to ece.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ logical xlower,xupper,bnded integer i,j,col2,nfree,nbreak,pointr, + ibp,nleft,ibkmin,iter double precision f1,f2,dt,dtm,tsum,dibp,zibp,dibp2,bkmin, + tu,tl,wmc,wmp,wmw,ddot,tj,tj0,neggi,sbgnrm, + f2_org double precision one,zero parameter (one=1.0d0,zero=0.0d0) c Check the status of the variables, reset iwhere(i) if necessary; c compute the Cauchy direction d and the breakpoints t; initialize c the derivative f1 and the vector p = W'd (for theta = 1). if (sbgnrm .le. zero) then if (iprint .ge. 0) write (6,*) 'Subgnorm = 0. GCP = X.' call dcopy(n,x,1,xcp,1) return endif bnded = .true. nfree = n + 1 nbreak = 0 ibkmin = 0 bkmin = zero col2 = 2*col f1 = zero if (iprint .ge. 99) write (6,3010) c We set p to zero and build it up as we determine d. do 20 i = 1, col2 p(i) = zero 20 continue c In the following loop we determine for each variable its bound c status and its breakpoint, and update p accordingly. c Smallest breakpoint is identified. do 50 i = 1, n neggi = -g(i) if (iwhere(i) .ne. 3 .and. iwhere(i) .ne. -1) then c if x(i) is not a constant and has bounds, c compute the difference between x(i) and its bounds. if (nbd(i) .le. 2) tl = x(i) - l(i) if (nbd(i) .ge. 2) tu = u(i) - x(i) c If a variable is close enough to a bound c we treat it as at bound. xlower = nbd(i) .le. 2 .and. tl .le. zero xupper = nbd(i) .ge. 2 .and. tu .le. zero c reset iwhere(i). iwhere(i) = 0 if (xlower) then if (neggi .le. zero) iwhere(i) = 1 else if (xupper) then if (neggi .ge. zero) iwhere(i) = 2 else if (abs(neggi) .le. zero) iwhere(i) = -3 endif endif pointr = head if (iwhere(i) .ne. 0 .and. iwhere(i) .ne. -1) then d(i) = zero else d(i) = neggi f1 = f1 - neggi*neggi c calculate p := p - W'e_i* (g_i). do 40 j = 1, col p(j) = p(j) + wy(i,pointr)* neggi p(col + j) = p(col + j) + ws(i,pointr)*neggi pointr = mod(pointr,m) + 1 40 continue if (nbd(i) .le. 2 .and. nbd(i) .ne. 0 + .and. neggi .lt. zero) then c x(i) + d(i) is bounded; compute t(i). nbreak = nbreak + 1 iorder(nbreak) = i t(nbreak) = tl/(-neggi) if (nbreak .eq. 1 .or. t(nbreak) .lt. bkmin) then bkmin = t(nbreak) ibkmin = nbreak endif else if (nbd(i) .ge. 2 .and. neggi .gt. zero) then c x(i) + d(i) is bounded; compute t(i). nbreak = nbreak + 1 iorder(nbreak) = i t(nbreak) = tu/neggi if (nbreak .eq. 1 .or. t(nbreak) .lt. bkmin) then bkmin = t(nbreak) ibkmin = nbreak endif else c x(i) + d(i) is not bounded. nfree = nfree - 1 iorder(nfree) = i if (abs(neggi) .gt. zero) bnded = .false. endif endif 50 continue c The indices of the nonzero components of d are now stored c in iorder(1),...,iorder(nbreak) and iorder(nfree),...,iorder(n). c The smallest of the nbreak breakpoints is in t(ibkmin)=bkmin. if (theta .ne. one) then c complete the initialization of p for theta not= one. call dscal(col,theta,p(col+1),1) endif c Initialize GCP xcp = x. call dcopy(n,x,1,xcp,1) if (nbreak .eq. 0 .and. nfree .eq. n + 1) then c is a zero vector, return with the initial xcp as GCP. if (iprint .gt. 100) write (6,1010) (xcp(i), i = 1, n) return endif c Initialize c = W'(xcp - x) = 0. do 60 j = 1, col2 c(j) = zero 60 continue c Initialize derivative f2. f2 = -theta*f1 f2_org = f2 if (col .gt. 0) then call bmv(m,sy,wt,col,p,v,info) if (info .ne. 0) return f2 = f2 - ddot(col2,v,1,p,1) endif dtm = -f1/f2 tsum = zero nint = 1 if (iprint .ge. 99) + write (6,*) 'There are ',nbreak,' breakpoints ' c If there are no breakpoints, locate the GCP and return. if (nbreak .eq. 0) goto 888 nleft = nbreak iter = 1 tj = zero c------------------- the beginning of the loop ------------------------- 777 continue c Find the next smallest breakpoint; c compute dt = t(nleft) - t(nleft + 1). tj0 = tj if (iter .eq. 1) then c Since we already have the smallest breakpoint we need not do c heapsort yet. Often only one breakpoint is used and the c cost of heapsort is avoided. tj = bkmin ibp = iorder(ibkmin) else if (iter .eq. 2) then c Replace the already used smallest breakpoint with the c breakpoint numbered nbreak > nlast, before heapsort call. if (ibkmin .ne. nbreak) then t(ibkmin) = t(nbreak) iorder(ibkmin) = iorder(nbreak) endif c Update heap structure of breakpoints c (if iter=2, initialize heap). endif call hpsolb(nleft,t,iorder,iter-2) tj = t(nleft) ibp = iorder(nleft) endif dt = tj - tj0 if (dt .ne. zero .and. iprint .ge. 100) then write (6,4011) nint,f1,f2 write (6,5010) dt write (6,6010) dtm endif c If a minimizer is within this interval, c locate the GCP and return. if (dtm .lt. dt) goto 888 c Otherwise fix one variable and c reset the corresponding component of d to zero. tsum = tsum + dt nleft = nleft - 1 iter = iter + 1 dibp = d(ibp) d(ibp) = zero if (dibp .gt. zero) then zibp = u(ibp) - x(ibp) xcp(ibp) = u(ibp) iwhere(ibp) = 2 else zibp = l(ibp) - x(ibp) xcp(ibp) = l(ibp) iwhere(ibp) = 1 endif if (iprint .ge. 100) write (6,*) 'Variable ',ibp,' is fixed.' if (nleft .eq. 0 .and. nbreak .eq. n) then c all n variables are fixed, c return with xcp as GCP. dtm = dt goto 999 endif c Update the derivative information. nint = nint + 1 dibp2 = dibp**2 c Update f1 and f2. c temporarily set f1 and f2 for col=0. f1 = f1 + dt*f2 + dibp2 - theta*dibp*zibp f2 = f2 - theta*dibp2 if (col .gt. 0) then c update c = c + dt*p. call daxpy(col2,dt,p,1,c,1) c choose wbp, c the row of W corresponding to the breakpoint encountered. pointr = head do 70 j = 1,col wbp(j) = wy(ibp,pointr) wbp(col + j) = theta*ws(ibp,pointr) pointr = mod(pointr,m) + 1 70 continue c compute (wbp)Mc, (wbp)Mp, and (wbp)M(wbp)'. call bmv(m,sy,wt,col,wbp,v,info) if (info .ne. 0) return wmc = ddot(col2,c,1,v,1) wmp = ddot(col2,p,1,v,1) wmw = ddot(col2,wbp,1,v,1) c update p = p - dibp*wbp. call daxpy(col2,-dibp,wbp,1,p,1) c complete updating f1 and f2 while col > 0. f1 = f1 + dibp*wmc f2 = f2 + 2.0d0*dibp*wmp - dibp2*wmw endif f2 = max(epsmch*f2_org,f2) if (nleft .gt. 0) then dtm = -f1/f2 goto 777 c to repeat the loop for unsearched intervals. else if(bnded) then f1 = zero f2 = zero dtm = zero else dtm = -f1/f2 endif c------------------- the end of the loop ------------------------------- 888 continue if (iprint .ge. 99) then write (6,*) write (6,*) 'GCP found in this segment' write (6,4010) nint,f1,f2 write (6,6010) dtm endif if (dtm .le. zero) dtm = zero tsum = tsum + dtm c Move free variables (i.e., the ones w/o breakpoints) and c the variables whose breakpoints haven't been reached. call daxpy(n,tsum,d,1,xcp,1) 999 continue c Update c = c + dtm*p = W'(x^c - x) c which will be used in computing r = Z'(B(x^c - x) + g). if (col .gt. 0) call daxpy(col2,dtm,p,1,c,1) if (iprint .gt. 100) write (6,1010) (xcp(i),i = 1,n) if (iprint .ge. 99) write (6,2010) 1010 format ('Cauchy X = ',/,(4x,1p,6(1x,d11.4))) 2010 format (/,'---------------- exit CAUCHY----------------------',/) 3010 format (/,'---------------- CAUCHY entered-------------------') 4010 format ('Piece ',i3,' --f1, f2 at start point ',1p,2(1x,d11.4)) 4011 format (/,'Piece ',i3,' --f1, f2 at start point ', + 1p,2(1x,d11.4)) 5010 format ('Distance to the next break point = ',1p,d11.4) 6010 format ('Distance to the stationary point = ',1p,d11.4) return end c====================== The end of cauchy ============================== subroutine cmprlb(n, m, x, g, ws, wy, sy, wt, z, r, wa, index, + theta, col, head, nfree, cnstnd, info) logical cnstnd integer n, m, col, head, nfree, info, index(n) double precision theta, + x(n), g(n), z(n), r(n), wa(4*m), + ws(n, m), wy(n, m), sy(m, m), wt(m, m) c ************ c c Subroutine cmprlb c c This subroutine computes r=-Z'B(xcp-xk)-Z'g by using c wa(2m+1)=W'(xcp-x) from subroutine cauchy. c c Subprograms called: c c L-BFGS-B Library ... bmv. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,j,k,pointr double precision a1,a2 if (.not. cnstnd .and. col .gt. 0) then do 26 i = 1, n r(i) = -g(i) 26 continue else do 30 i = 1, nfree k = index(i) r(i) = -theta*(z(k) - x(k)) - g(k) 30 continue call bmv(m,sy,wt,col,wa(2*m+1),wa(1),info) if (info .ne. 0) then info = -8 return endif pointr = head do 34 j = 1, col a1 = wa(j) a2 = theta*wa(col + j) do 32 i = 1, nfree k = index(i) r(i) = r(i) + wy(k,pointr)*a1 + ws(k,pointr)*a2 32 continue pointr = mod(pointr,m) + 1 34 continue endif return end c======================= The end of cmprlb ============================= subroutine errclb(n, m, factr, l, u, nbd, task, info, k) character*60 task integer n, m, info, k, nbd(n) double precision factr, l(n), u(n) c ************ c c Subroutine errclb c c This subroutine checks the validity of the input data. c c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i double precision zero parameter (zero=0.0d0) c Check the input arguments for errors. if (n .le. 0) task = 'ERROR: N .LE. 0' if (m .le. 0) task = 'ERROR: M .LE. 0' if (factr .lt. zero) task = 'ERROR: FACTR .LT. 0' c Check the validity of the arrays nbd(i), u(i), and l(i). do 10 i = 1, n if (nbd(i) .lt. 0 .or. nbd(i) .gt. 3) then c return task = 'ERROR: INVALID NBD' info = -6 k = i endif if (nbd(i) .eq. 2) then if (l(i) .gt. u(i)) then c return task = 'ERROR: NO FEASIBLE SOLUTION' info = -7 k = i endif endif 10 continue return end c======================= The end of errclb ============================= subroutine formk(n, nsub, ind, nenter, ileave, indx2, iupdat, + updatd, wn, wn1, m, ws, wy, sy, theta, col, + head, info) integer n, nsub, m, col, head, nenter, ileave, iupdat, + info, ind(n), indx2(n) double precision theta, wn(2*m, 2*m), wn1(2*m, 2*m), + ws(n, m), wy(n, m), sy(m, m) logical updatd c ************ c c Subroutine formk c c This subroutine forms the LEL^T factorization of the indefinite c c matrix K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c where E = [-I 0] c [ 0 I] c The matrix K can be shown to be equal to the matrix M^[-1]N c occurring in section 5.1 of [1], as well as to the matrix c Mbar^[-1] Nbar in section 5.3. c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c nsub is an integer variable c On entry nsub is the number of subspace variables in free set. c On exit nsub is not changed. c c ind is an integer array of dimension nsub. c On entry ind specifies the indices of subspace variables. c On exit ind is unchanged. c c nenter is an integer variable. c On entry nenter is the number of variables entering the c free set. c On exit nenter is unchanged. c c ileave is an integer variable. c On entry indx2(ileave),...,indx2(n) are the variables leaving c the free set. c On exit ileave is unchanged. c c indx2 is an integer array of dimension n. c On entry indx2(1),...,indx2(nenter) are the variables entering c the free set, while indx2(ileave),...,indx2(n) are the c variables leaving the free set. c On exit indx2 is unchanged. c c iupdat is an integer variable. c On entry iupdat is the total number of BFGS updates made so far. c On exit iupdat is unchanged. c c updatd is a logical variable. c On entry 'updatd' is true if the L-BFGS matrix is updatd. c On exit 'updatd' is unchanged. c c wn is a double precision array of dimension 2m x 2m. c On entry wn is unspecified. c On exit the upper triangle of wn stores the LEL^T factorization c of the 2*col x 2*col indefinite matrix c [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c c wn1 is a double precision array of dimension 2m x 2m. c On entry wn1 stores the lower triangular part of c [Y' ZZ'Y L_a'+R_z'] c [L_a+R_z S'AA'S ] c in the previous iteration. c On exit wn1 stores the corresponding updated matrices. c The purpose of wn1 is just to store these inner products c so they can be easily updated and inserted into wn. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c ws, wy, sy, and wtyy are double precision arrays; c theta is a double precision variable; c col is an integer variable; c head is an integer variable. c On entry they store the information defining the c limited memory BFGS matrix: c ws(n,m) stores S, a set of s-vectors; c wy(n,m) stores Y, a set of y-vectors; c sy(m,m) stores S'Y; c wtyy(m,m) stores the Cholesky factorization c of (theta*S'S+LD^(-1)L') c theta is the scaling factor specifying B_0 = theta I; c col is the number of variable metric corrections stored; c head is the location of the 1st s- (or y-) vector in S (or Y). c On exit they are unchanged. c c info is an integer variable. c On entry info is unspecified. c On exit info = 0 for normal return; c = -1 when the 1st Cholesky factorization failed; c = -2 when the 2st Cholesky factorization failed. c c Subprograms called: c c Linpack ... dcopy, dpofa, dtrsl. c c c References: c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: a c limited memory FORTRAN code for solving bound constrained c optimization problems'', Tech. Report, NAM-11, EECS Department, c Northwestern University, 1994. c c (Postscript files of these papers are available via anonymous c ftp to ece.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer m2,ipntr,jpntr,iy,is,jy,js,is1,js1,k1,i,k, + col2,pbegin,pend,dbegin,dend,upcl double precision ddot,temp1,temp2,temp3,temp4 double precision zero parameter (zero=0.0d0) c Form the lower triangular part of c WN1 = [Y' ZZ'Y L_a'+R_z'] c [L_a+R_z S'AA'S ] c where L_a is the strictly lower triangular part of S'AA'Y c R_z is the upper triangular part of S'ZZ'Y. if (updatd) then if (iupdat .gt. m) then c shift old part of WN1. do 10 jy = 1, m - 1 js = m + jy call dcopy(m-jy,wn1(jy+1,jy+1),1,wn1(jy,jy),1) call dcopy(m-jy,wn1(js+1,js+1),1,wn1(js,js),1) call dcopy(m-1,wn1(m+2,jy+1),1,wn1(m+1,jy),1) 10 continue endif c put new rows in blocks (1,1), (2,1) and (2,2). pbegin = 1 pend = nsub dbegin = nsub + 1 dend = n iy = col is = m + col ipntr = head + col - 1 if (ipntr .gt. m) ipntr = ipntr - m jpntr = head do 20 jy = 1, col js = m + jy temp1 = zero temp2 = zero temp3 = zero c compute element jy of row 'col' of Y'ZZ'Y do 15 k = pbegin, pend k1 = ind(k) temp1 = temp1 + wy(k1,ipntr)*wy(k1,jpntr) 15 continue c compute elements jy of row 'col' of L_a and S'AA'S do 16 k = dbegin, dend k1 = ind(k) temp2 = temp2 + ws(k1,ipntr)*ws(k1,jpntr) temp3 = temp3 + ws(k1,ipntr)*wy(k1,jpntr) 16 continue wn1(iy,jy) = temp1 wn1(is,js) = temp2 wn1(is,jy) = temp3 jpntr = mod(jpntr,m) + 1 20 continue c put new column in block (2,1). jy = col jpntr = head + col - 1 if (jpntr .gt. m) jpntr = jpntr - m ipntr = head do 30 i = 1, col is = m + i temp3 = zero c compute element i of column 'col' of R_z do 25 k = pbegin, pend k1 = ind(k) temp3 = temp3 + ws(k1,ipntr)*wy(k1,jpntr) 25 continue ipntr = mod(ipntr,m) + 1 wn1(is,jy) = temp3 30 continue upcl = col - 1 else upcl = col endif c modify the old parts in blocks (1,1) and (2,2) due to changes c in the set of free variables. ipntr = head do 45 iy = 1, upcl is = m + iy jpntr = head do 40 jy = 1, iy js = m + jy temp1 = zero temp2 = zero temp3 = zero temp4 = zero do 35 k = 1, nenter k1 = indx2(k) temp1 = temp1 + wy(k1,ipntr)*wy(k1,jpntr) temp2 = temp2 + ws(k1,ipntr)*ws(k1,jpntr) 35 continue do 36 k = ileave, n k1 = indx2(k) temp3 = temp3 + wy(k1,ipntr)*wy(k1,jpntr) temp4 = temp4 + ws(k1,ipntr)*ws(k1,jpntr) 36 continue wn1(iy,jy) = wn1(iy,jy) + temp1 - temp3 wn1(is,js) = wn1(is,js) - temp2 + temp4 jpntr = mod(jpntr,m) + 1 40 continue ipntr = mod(ipntr,m) + 1 45 continue c modify the old parts in block (2,1). ipntr = head do 60 is = m + 1, m + upcl jpntr = head do 55 jy = 1, upcl temp1 = zero temp3 = zero do 50 k = 1, nenter k1 = indx2(k) temp1 = temp1 + ws(k1,ipntr)*wy(k1,jpntr) 50 continue do 51 k = ileave, n k1 = indx2(k) temp3 = temp3 + ws(k1,ipntr)*wy(k1,jpntr) 51 continue if (is .le. jy + m) then wn1(is,jy) = wn1(is,jy) + temp1 - temp3 else wn1(is,jy) = wn1(is,jy) - temp1 + temp3 endif jpntr = mod(jpntr,m) + 1 55 continue ipntr = mod(ipntr,m) + 1 60 continue c Form the upper triangle of WN = [D+Y' ZZ'Y/theta -L_a'+R_z' ] c [-L_a +R_z S'AA'S*theta] m2 = 2*m do 70 iy = 1, col is = col + iy is1 = m + iy do 65 jy = 1, iy js = col + jy js1 = m + jy wn(jy,iy) = wn1(iy,jy)/theta wn(js,is) = wn1(is1,js1)*theta 65 continue do 66 jy = 1, iy - 1 wn(jy,is) = -wn1(is1,jy) 66 continue do 67 jy = iy, col wn(jy,is) = wn1(is1,jy) 67 continue wn(iy,iy) = wn(iy,iy) + sy(iy,iy) 70 continue c Form the upper triangle of c WN= [ LL' L^-1(-L_a'+R_z')] c [(-L_a +R_z)L'^-1 S'AA'S*theta ] c first Cholesky factor (1,1) block of wn to get LL' c with L' stored in the upper triangle of wn. call dpofa(wn,m2,col,info) if (info .ne. 0) then info = -1 return endif c then form L^-1(-L_a'+R_z') in the (1,2) block. col2 = 2*col do 71 js = col+1 ,col2 call dtrsl(wn,m2,col,wn(1,js),11,info) 71 continue c Form S'AA'S*theta + (L^-1(-L_a'+R_z'))'L^-1(-L_a'+R_z') in the c upper triangle of (2,2) block of wn. do 72 is = col+1, col2 do 74 js = is, col2 wn(is,js) = wn(is,js) + ddot(col,wn(1,is),1,wn(1,js),1) 74 continue 72 continue c Cholesky factorization of (2,2) block of wn. call dpofa(wn(col+1,col+1),m2,col,info) if (info .ne. 0) then info = -2 return endif return end c======================= The end of formk ============================== subroutine formt(m, wt, sy, ss, col, theta, info) integer m, col, info double precision theta, wt(m, m), sy(m, m), ss(m, m) c ************ c c Subroutine formt c c This subroutine forms the upper half of the pos. def. and symm. c T = theta*SS + L*D^(-1)*L', stores T in the upper triangle c of the array wt, and performs the Cholesky factorization of T c to produce J*J', with J' stored in the upper triangle of wt. c c Subprograms called: c c Linpack ... dpofa. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,j,k,k1 double precision ddum double precision zero parameter (zero=0.0d0) c Form the upper half of T = theta*SS + L*D^(-1)*L', c store T in the upper triangle of the array wt. do 52 j = 1, col wt(1,j) = theta*ss(1,j) 52 continue do 55 i = 2, col do 54 j = i, col k1 = min(i,j) - 1 ddum = zero do 53 k = 1, k1 ddum = ddum + sy(i,k)*sy(j,k)/sy(k,k) 53 continue wt(i,j) = ddum + theta*ss(i,j) 54 continue 55 continue c Cholesky factorize T to J*J' with c J' stored in the upper triangle of wt. call dpofa(wt,m,col,info) if (info .ne. 0) then info = -3 endif return end c======================= The end of formt ============================== subroutine freev(n, nfree, index, nenter, ileave, indx2, + iwhere, wrk, updatd, cnstnd, iprint, iter) integer n, nfree, nenter, ileave, iprint, iter, + index(n), indx2(n), iwhere(n) logical wrk, updatd, cnstnd c ************ c c Subroutine freev c c This subroutine counts the entering and leaving variables when c iter > 0, and finds the index set of free and active variables c at the GCP. c c cnstnd is a logical variable indicating whether bounds are present c c index is an integer array of dimension n c for i=1,...,nfree, index(i) are the indices of free variables c for i=nfree+1,...,n, index(i) are the indices of bound variables c On entry after the first iteration, index gives c the free variables at the previous iteration. c On exit it gives the free variables based on the determination c in cauchy using the array iwhere. c c indx2 is an integer array of dimension n c On entry indx2 is unspecified. c On exit with iter>0, indx2 indicates which variables c have changed status since the previous iteration. c For i= 1,...,nenter, indx2(i) have changed from bound to free. c For i= ileave+1,...,n, indx2(i) have changed from free to bound. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer iact,i,k nenter = 0 ileave = n + 1 if (iter .gt. 0 .and. cnstnd) then c count the entering and leaving variables. do 20 i = 1, nfree k = index(i) if (iwhere(k) .gt. 0) then ileave = ileave - 1 indx2(ileave) = k if (iprint .ge. 100) write (6,*) + 'Variable ',k,' leaves the set of free variables' endif 20 continue do 22 i = 1 + nfree, n k = index(i) if (iwhere(k) .le. 0) then nenter = nenter + 1 indx2(nenter) = k if (iprint .ge. 100) write (6,*) + 'Variable ',k,' enters the set of free variables' endif 22 continue if (iprint .ge. 99) write (6,*) + n+1-ileave,' variables leave; ',nenter,' variables enter' endif wrk = (ileave .lt. n+1) .or. (nenter .gt. 0) .or. updatd c Find the index set of free and active variables at the GCP. nfree = 0 iact = n + 1 do 24 i = 1, n if (iwhere(i) .le. 0) then nfree = nfree + 1 index(nfree) = i else iact = iact - 1 index(iact) = i endif 24 continue if (iprint .ge. 99) write (6,*) + nfree,' variables are free at GCP ',iter + 1 return end c======================= The end of freev ============================== subroutine hpsolb(n, t, iorder, iheap) integer iheap, n, iorder(n) double precision t(n) c ************ c c Subroutine hpsolb c c This subroutine sorts out the least element of t, and puts the c remaining elements of t in a heap. c c n is an integer variable. c On entry n is the dimension of the arrays t and iorder. c On exit n is unchanged. c c t is a double precision array of dimension n. c On entry t stores the elements to be sorted, c On exit t(n) stores the least elements of t, and t(1) to t(n-1) c stores the remaining elements in the form of a heap. c c iorder is an integer array of dimension n. c On entry iorder(i) is the index of t(i). c On exit iorder(i) is still the index of t(i), but iorder may be c permuted in accordance with t. c c iheap is an integer variable specifying the task. c On entry iheap should be set as follows: c iheap .eq. 0 if t(1) to t(n) is not in the form of a heap, c iheap .ne. 0 if otherwise. c On exit iheap is unchanged. c c c References: c Algorithm 232 of CACM (J. W. J. Williams): HEAPSORT. c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c ************ integer i,j,k,indxin,indxou double precision ddum,out if (iheap .eq. 0) then c Rearrange the elements t(1) to t(n) to form a heap. do 20 k = 2, n ddum = t(k) indxin = iorder(k) c Add ddum to the heap. i = k 10 continue if (i.gt.1) then j = i/2 if (ddum .lt. t(j)) then t(i) = t(j) iorder(i) = iorder(j) i = j goto 10 endif endif t(i) = ddum iorder(i) = indxin 20 continue endif c Assign to 'out' the value of t(1), the least member of the heap, c and rearrange the remaining members to form a heap as c elements 1 to n-1 of t. if (n .gt. 1) then i = 1 out = t(1) indxou = iorder(1) ddum = t(n) indxin = iorder(n) c Restore the heap 30 continue j = i+i if (j .le. n-1) then if (t(j+1) .lt. t(j)) j = j+1 if (t(j) .lt. ddum ) then t(i) = t(j) iorder(i) = iorder(j) i = j goto 30 endif endif t(i) = ddum iorder(i) = indxin c Put the least member in t(n). t(n) = out iorder(n) = indxou endif return end c====================== The end of hpsolb ============================== subroutine lnsrlb(n, l, u, nbd, x, f, fold, gd, gdold, g, d, r, t, + z, stp, dnorm, dtd, xstep, stpmx, iter, ifun, + iback, nfgv, info, task, boxed, cnstnd, csave, + isave, dsave) character*60 task, csave logical boxed, cnstnd integer n, iter, ifun, iback, nfgv, info, + nbd(n), isave(2) double precision f, fold, gd, gdold, stp, dnorm, dtd, xstep, + stpmx, x(n), l(n), u(n), g(n), d(n), r(n), t(n), + z(n), dsave(13) c ********** c c Subroutine lnsrlb c c This subroutine calls subroutine dcsrch from the Minpack2 library c to perform the line search. Subroutine dscrch is safeguarded so c that all trial points lie within the feasible region. c c Subprograms called: c c Minpack2 Library ... dcsrch. c c Linpack ... dtrsl, ddot. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ********** integer i double precision ddot,a1,a2 double precision one,zero,big parameter (one=1.0d0,zero=0.0d0,big=1.0d+10) double precision ftol,gtol,xtol parameter (ftol=1.0d-3,gtol=0.9d0,xtol=0.1d0) if (task(1:5) .eq. 'FG_LN') goto 556 dtd = ddot(n,d,1,d,1) dnorm = sqrt(dtd) c Determine the maximum step length. stpmx = big if (cnstnd) then if (iter .eq. 0) then stpmx = one else do 43 i = 1, n a1 = d(i) if (nbd(i) .ne. 0) then if (a1 .lt. zero .and. nbd(i) .le. 2) then a2 = l(i) - x(i) if (a2 .ge. zero) then stpmx = zero else if (a1*stpmx .lt. a2) then stpmx = a2/a1 endif else if (a1 .gt. zero .and. nbd(i) .ge. 2) then a2 = u(i) - x(i) if (a2 .le. zero) then stpmx = zero else if (a1*stpmx .gt. a2) then stpmx = a2/a1 endif endif endif 43 continue endif endif if (iter .eq. 0 .and. .not. boxed) then stp = min(one/dnorm, stpmx) else stp = one endif call dcopy(n,x,1,t,1) call dcopy(n,g,1,r,1) fold = f ifun = 0 iback = 0 csave = 'START' 556 continue gd = ddot(n,g,1,d,1) if (ifun .eq. 0) then gdold=gd if (gd .ge. zero) then c the directional derivative >=0. c Line search is impossible. info = -4 return endif endif call dcsrch(f,gd,stp,ftol,gtol,xtol,zero,stpmx,csave,isave,dsave) xstep = stp*dnorm if (csave(1:4) .ne. 'CONV' .and. csave(1:4) .ne. 'WARN') then task = 'FG_LNSRCH' ifun = ifun + 1 nfgv = nfgv + 1 iback = ifun - 1 if (stp .eq. one) then call dcopy(n,z,1,x,1) else do 41 i = 1, n x(i) = stp*d(i) + t(i) 41 continue endif else task = 'NEW_X' endif return end c======================= The end of lnsrlb ============================= subroutine matupd(n, m, ws, wy, sy, ss, d, r, itail, + iupdat, col, head, theta, rr, dr, stp, dtd) integer n, m, itail, iupdat, col, head double precision theta, rr, dr, stp, dtd, d(n), r(n), + ws(n, m), wy(n, m), sy(m, m), ss(m, m) c ************ c c Subroutine matupd c c This subroutine updates matrices WS and WY, and forms the c middle matrix in B. c c Subprograms called: c c Linpack ... dcopy, ddot. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer j,pointr double precision ddot double precision one parameter (one=1.0d0) c Set pointers for matrices WS and WY. if (iupdat .le. m) then col = iupdat itail = mod(head+iupdat-2,m) + 1 else itail = mod(itail,m) + 1 head = mod(head,m) + 1 endif c Update matrices WS and WY. call dcopy(n,d,1,ws(1,itail),1) call dcopy(n,r,1,wy(1,itail),1) c Set theta=yy/ys. theta = rr/dr c Form the middle matrix in B. c update the upper triangle of SS, c and the lower triangle of SY: if (iupdat .gt. m) then c move old information do 50 j = 1, col - 1 call dcopy(j,ss(2,j+1),1,ss(1,j),1) call dcopy(col-j,sy(j+1,j+1),1,sy(j,j),1) 50 continue endif c add new information: the last row of SY c and the last column of SS: pointr = head do 51 j = 1, col - 1 sy(col,j) = ddot(n,d,1,wy(1,pointr),1) ss(j,col) = ddot(n,ws(1,pointr),1,d,1) pointr = mod(pointr,m) + 1 51 continue if (stp .eq. one) then ss(col,col) = dtd else ss(col,col) = stp*stp*dtd endif sy(col,col) = dr return end c======================= The end of matupd ============================= subroutine prn1lb(n, m, l, u, x, iprint, itfile, epsmch) integer n, m, iprint, itfile double precision epsmch, x(n), l(n), u(n) c ************ c c Subroutine prn1lb c c This subroutine prints the input data, initial point, upper and c lower bounds of each variable, machine precision, as well as c the headings of the output. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i if (iprint .ge. 0) then write (6,7001) epsmch write (6,*) 'N = ',n,' M = ',m if (iprint .ge. 1) then write (itfile,2001) epsmch write (itfile,*)'N = ',n,' M = ',m write (itfile,9001) if (iprint .gt. 100) then write (6,1004) 'L =',(l(i),i = 1,n) write (6,1004) 'X0 =',(x(i),i = 1,n) write (6,1004) 'U =',(u(i),i = 1,n) endif endif endif 1004 format (/,a4, 1p, 6(1x,d11.4),/,(4x,1p,6(1x,d11.4))) 2001 format ('RUNNING THE L-BFGS-B CODE',/,/, + 'it = iteration number',/, + 'nf = number of function evaluations',/, + 'nint = number of segments explored during the Cauchy search',/, + 'nact = number of active bounds at the generalized Cauchy point' + ,/, + 'sub = manner in which the subspace minimization terminated:' + ,/,' con = converged, bnd = a bound was reached',/, + 'itls = number of iterations performed in the line search',/, + 'stepl = step length used',/, + 'tstep = norm of the displacement (total step)',/, + 'projg = norm of the projected gradient',/, + 'f = function value',/,/, + ' * * *',/,/, + 'Machine precision =',1p,d10.3) 7001 format ('RUNNING THE L-BFGS-B CODE',/,/, + ' * * *',/,/, + 'Machine precision =',1p,d10.3) 9001 format (/,3x,'it',3x,'nf',2x,'nint',2x,'nact',2x,'sub',2x,'itls', + 2x,'stepl',4x,'tstep',5x,'projg',8x,'f') return end c======================= The end of prn1lb ============================= subroutine prn2lb(n, x, f, g, iprint, itfile, iter, nfgv, nact, + sbgnrm, nint, word, iword, iback, stp, xstep) character*3 word integer n, iprint, itfile, iter, nfgv, nact, nint, + iword, iback double precision f, sbgnrm, stp, xstep, x(n), g(n) c ************ c c Subroutine prn2lb c c This subroutine prints out new information after a successful c line search. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,imod c 'word' records the status of subspace solutions. if (iword .eq. 0) then c the subspace minimization converged. word = 'con' else if (iword .eq. 1) then c the subspace minimization stopped at a bound. word = 'bnd' else if (iword .eq. 5) then c the truncated Newton step has been used. word = 'TNT' else word = '---' endif if (iprint .ge. 99) then write (6,*) 'LINE SEARCH',iback,' times; norm of step = ',xstep write (6,2001) iter,f,sbgnrm if (iprint .gt. 100) then write (6,1004) 'X =',(x(i), i = 1, n) write (6,1004) 'G =',(g(i), i = 1, n) endif else if (iprint .gt. 0) then imod = mod(iter,iprint) if (imod .eq. 0) write (6,2001) iter,f,sbgnrm endif if (iprint .ge. 1) write (itfile,3001) + iter,nfgv,nint,nact,word,iback,stp,xstep,sbgnrm,f 1004 format (/,a4, 1p, 6(1x,d11.4),/,(4x,1p,6(1x,d11.4))) 2001 format + (/,'At iterate',i5,4x,'f= ',1p,d12.5,4x,'|proj g|= ',1p,d12.5) 3001 format(2(1x,i4),2(1x,i5),2x,a3,1x,i4,1p,2(2x,d7.1),1p,2(1x,d10.3)) return end c======================= The end of prn2lb ============================= subroutine prn3lb(n, x, f, task, iprint, info, itfile, + iter, nfgv, nintol, nskip, nact, sbgnrm, + time, nint, word, iback, stp, xstep, k, + cachyt, sbtime, lnscht) character*60 task character*3 word integer n, iprint, info, itfile, iter, nfgv, nintol, + nskip, nact, nint, iback, k double precision f, sbgnrm, time, stp, xstep, cachyt, sbtime, + lnscht, x(n) c ************ c c Subroutine prn3lb c c This subroutine prints out information when either a built-in c convergence test is satisfied or when an error message is c generated. c c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i if (task(1:5) .eq. 'ERROR') goto 999 if (iprint .ge. 0) then write (6,3003) write (6,3004) write(6,3005) n,iter,nfgv,nintol,nskip,nact,sbgnrm,f if (iprint .ge. 100) then write (6,1004) 'X =',(x(i),i = 1,n) endif if (iprint .ge. 1) write (6,*) ' F =',f endif 999 continue if (iprint .ge. 0) then write (6,3009) task if (info .ne. 0) then if (info .eq. -1) write (6,9011) if (info .eq. -2) write (6,9012) if (info .eq. -3) write (6,9013) if (info .eq. -4) write (6,9014) if (info .eq. -5) write (6,9015) if (info .eq. -6) write (6,*)' Input nbd(',k,') is invalid.' if (info .eq. -7) + write (6,*)' l(',k,') > u(',k,'). No feasible solution.' if (info .eq. -8) write (6,9018) if (info .eq. -9) write (6,9019) endif if (iprint .ge. 1) write (6,3007) cachyt,sbtime,lnscht write (6,3008) time if (iprint .ge. 1) then if (info .eq. -4 .or. info .eq. -9) then write (itfile,3002) + iter,nfgv,nint,nact,word,iback,stp,xstep endif write (itfile,3009) task if (info .ne. 0) then if (info .eq. -1) write (itfile,9011) if (info .eq. -2) write (itfile,9012) if (info .eq. -3) write (itfile,9013) if (info .eq. -4) write (itfile,9014) if (info .eq. -5) write (itfile,9015) if (info .eq. -8) write (itfile,9018) if (info .eq. -9) write (itfile,9019) endif write (itfile,3008) time endif endif 1004 format (/,a4, 1p, 6(1x,d11.4),/,(4x,1p,6(1x,d11.4))) 3002 format(2(1x,i4),2(1x,i5),2x,a3,1x,i4,1p,2(2x,d7.1),6x,'-',10x,'-') 3003 format (/, + ' * * *',/,/, + 'Tit = total number of iterations',/, + 'Tnf = total number of function evaluations',/, + 'Tnint = total number of segments explored during', + ' Cauchy searches',/, + 'Skip = number of BFGS updates skipped',/, + 'Nact = number of active bounds at final generalized', + ' Cauchy point',/, + 'Projg = norm of the final projected gradient',/, + 'F = final function value',/,/, + ' * * *') 3004 format (/,3x,'N',3x,'Tit',2x,'Tnf',2x,'Tnint',2x, + 'Skip',2x,'Nact',5x,'Projg',8x,'F') 3005 format (i5,2(1x,i4),(1x,i6),(2x,i4),(1x,i5),1p,2(2x,d10.3)) 3007 format (/,' Cauchy time',1p,e10.3,' seconds.',/ + ' Subspace minimization time',1p,e10.3,' seconds.',/ + ' Line search time',1p,e10.3,' seconds.') 3008 format (/,' Total User time',1p,e10.3,' seconds.',/) 3009 format (/,a60) 9011 format (/, +' Matrix in 1st Cholesky factorization in formk is not Pos. Def.') 9012 format (/, +' Matrix in 2st Cholesky factorization in formk is not Pos. Def.') 9013 format (/, +' Matrix in the Cholesky factorization in formt is not Pos. Def.') 9014 format (/, +' Derivative >= 0, backtracking line search impossible.',/, +' Previous x, f and g restored.',/, +' Possible causes: 1 error in function or gradient evaluation;',/, +' 2 rounding errors dominate computation.') 9015 format (/, +' Warning: more than 10 function and gradient',/, +' evaluations in the last line search. Termination',/, +' may possibly be caused by a bad search direction.') 9018 format (/,' The triangular system is singular.') 9019 format (/, +' Line search cannot locate an adequate point after 20 function',/ +,' and gradient evaluations. Previous x, f and g restored.',/, +' Possible causes: 1 error in function or gradient evaluation;',/, +' 2 rounding error dominate computation.') return end c======================= The end of prn3lb ============================= subroutine projgr(n, l, u, nbd, x, g, sbgnrm) integer n, nbd(n) double precision sbgnrm, x(n), l(n), u(n), g(n) c ************ c c Subroutine projgr c c This subroutine computes the infinity norm of the projected c gradient. c c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i double precision gi double precision zero parameter (zero=0.0d0) sbgnrm = zero do 15 i = 1, n gi = g(i) if (nbd(i) .ne. 0) then if (gi .lt. zero) then if (nbd(i) .ge. 2) gi = max((x(i)-u(i)),gi) else if (nbd(i) .le. 2) gi = min((x(i)-l(i)),gi) endif endif sbgnrm = max(sbgnrm,abs(gi)) 15 continue return end c======================= The end of projgr ============================= subroutine subsm(n, m, nsub, ind, l, u, nbd, x, d, ws, wy, theta, + col, head, iword, wv, wn, iprint, info) integer n, m, nsub, col, head, iword, iprint, info, + ind(nsub), nbd(n) double precision theta, + l(n), u(n), x(n), d(n), + ws(n, m), wy(n, m), + wv(2*m), wn(2*m, 2*m) c ************ c c Subroutine subsm c c Given xcp, l, u, r, an index set that specifies c the active set at xcp, and an l-BFGS matrix B c (in terms of WY, WS, SY, WT, head, col, and theta), c this subroutine computes an approximate solution c of the subspace problem c c (P) min Q(x) = r'(x-xcp) + 1/2 (x-xcp)' B (x-xcp) c c subject to l<=x<=u c x_i=xcp_i for all i in A(xcp) c c along the subspace unconstrained Newton direction c c d = -(Z'BZ)^(-1) r. c c The formula for the Newton direction, given the L-BFGS matrix c and the Sherman-Morrison formula, is c c d = (1/theta)r + (1/theta*2) Z'WK^(-1)W'Z r. c c where c K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c c Note that this procedure for computing d differs c from that described in [1]. One can show that the matrix K is c equal to the matrix M^[-1]N in that paper. c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c nsub is an integer variable. c On entry nsub is the number of free variables. c On exit nsub is unchanged. c c ind is an integer array of dimension nsub. c On entry ind specifies the coordinate indices of free variables. c On exit ind is unchanged. c c l is a double precision array of dimension n. c On entry l is the lower bound of x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound of x. c On exit u is unchanged. c c nbd is a integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, and c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c x is a double precision array of dimension n. c On entry x specifies the Cauchy point xcp. c On exit x(i) is the minimizer of Q over the subspace of c free variables. c c d is a double precision array of dimension n. c On entry d is the reduced gradient of Q at xcp. c On exit d is the Newton direction of Q. c c ws and wy are double precision arrays; c theta is a double precision variable; c col is an integer variable; c head is an integer variable. c On entry they store the information defining the c limited memory BFGS matrix: c ws(n,m) stores S, a set of s-vectors; c wy(n,m) stores Y, a set of y-vectors; c theta is the scaling factor specifying B_0 = theta I; c col is the number of variable metric corrections stored; c head is the location of the 1st s- (or y-) vector in S (or Y). c On exit they are unchanged. c c iword is an integer variable. c On entry iword is unspecified. c On exit iword specifies the status of the subspace solution. c iword = 0 if the solution is in the box, c 1 if some bound is encountered. c c wv is a double precision working array of dimension 2m. c c wn is a double precision array of dimension 2m x 2m. c On entry the upper triangle of wn stores the LEL^T factorization c of the indefinite matrix c c K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c where E = [-I 0] c [ 0 I] c On exit wn is unchanged. c c iprint is an INTEGER variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c info is an integer variable. c On entry info is unspecified. c On exit info = 0 for normal return, c = nonzero for abnormal return c when the matrix K is ill-conditioned. c c Subprograms called: c c Linpack dtrsl. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ logical temp1_updated integer pointr,m2,col2,ibd,jy,js,i,j,k double precision alpha,dk,temp1,temp2 double precision one,zero parameter (one=1.0d0,zero=0.0d0) if (nsub .le. 0) return if (iprint .ge. 99) write (6,1001) c Compute wv = W'Zd. pointr = head do 20 i = 1, col temp1 = zero temp2 = zero do 10 j = 1, nsub k = ind(j) temp1 = temp1 + wy(k,pointr)*d(j) temp2 = temp2 + ws(k,pointr)*d(j) 10 continue wv(i) = temp1 wv(col + i) = theta*temp2 pointr = mod(pointr,m) + 1 20 continue c Compute wv:=K^(-1)wv. m2 = 2*m col2 = 2*col call dtrsl(wn,m2,col2,wv,11,info) if (info .ne. 0) return do 25 i = 1, col wv(i) = -wv(i) 25 continue call dtrsl(wn,m2,col2,wv,01,info) if (info .ne. 0) return c Compute d = (1/theta)d + (1/theta**2)Z'W wv. pointr = head do 40 jy = 1, col js = col + jy do 30 i = 1, nsub k = ind(i) d(i) = d(i) + wy(k,pointr)*wv(jy)/theta + + ws(k,pointr)*wv(js) 30 continue pointr = mod(pointr,m) + 1 40 continue do 50 i = 1, nsub d(i) = d(i)/theta 50 continue c Backtrack to the feasible region. alpha = one temp1 = alpha do 60 i = 1, nsub k = ind(i) dk = d(i) if (nbd(k) .ne. 0) then temp1_updated = .false. if (dk .lt. zero .and. nbd(k) .le. 2) then temp2 = l(k) - x(k) if (temp2 .ge. zero) then temp1 = zero temp1_updated = .true. else if (dk*alpha .lt. temp2) then temp1 = temp2/dk temp1_updated = .true. endif else if (dk .gt. zero .and. nbd(k) .ge. 2) then temp2 = u(k) - x(k) if (temp2 .le. zero) then temp1 = zero temp1_updated = .true. else if (dk*alpha .gt. temp2) then temp1 = temp2/dk temp1_updated = .true. endif endif cc logical variable temp1_updated added to eliminate unexpected cc trigger of the if statement due to possible difference between cc hardware precision and double precision. if (temp1_updated .and. temp1 .lt. alpha) then alpha = temp1 ibd = i endif endif 60 continue if (alpha .lt. one) then dk = d(ibd) k = ind(ibd) if (dk .gt. zero) then x(k) = u(k) d(ibd) = zero else if (dk .lt. zero) then x(k) = l(k) d(ibd) = zero endif endif do 70 i = 1, nsub k = ind(i) x(k) = x(k) + alpha*d(i) 70 continue if (iprint .ge. 99) then if (alpha .lt. one) then write (6,1002) alpha else write (6,*) 'SM solution inside the box' end if if (iprint .gt.100) write (6,1003) (x(i),i=1,n) endif if (alpha .lt. one) then iword = 1 else iword = 0 endif if (iprint .ge. 99) write (6,1004) 1001 format (/,'----------------SUBSM entered-----------------',/) 1002 format ( 'ALPHA = ',f7.5,' backtrack to the BOX') 1003 format ('Subspace solution X = ',/,(4x,1p,6(1x,d11.4))) 1004 format (/,'----------------exit SUBSM --------------------',/) return end c====================== The end of subsm =============================== subroutine dcsrch(f,g,stp,ftol,gtol,xtol,stpmin,stpmax, + task,isave,dsave) character*(*) task integer isave(2) double precision f,g,stp,ftol,gtol,xtol,stpmin,stpmax double precision dsave(13) c ********** c c Subroutine dcsrch c c This subroutine finds a step that satisfies a sufficient c decrease condition and a curvature condition. c c Each call of the subroutine updates an interval with c endpoints stx and sty. The interval is initially chosen c so that it contains a minimizer of the modified function c c psi(stp) = f(stp) - f(0) - ftol*stp*f'(0). c c If psi(stp) <= 0 and f'(stp) >= 0 for some step, then the c interval is chosen so that it contains a minimizer of f. c c The algorithm is designed to find a step that satisfies c the sufficient decrease condition c c f(stp) <= f(0) + ftol*stp*f'(0), c c and the curvature condition c c abs(f'(stp)) <= gtol*abs(f'(0)). c c If ftol is less than gtol and if, for example, the function c is bounded below, then there is always a step which satisfies c both conditions. c c If no step can be found that satisfies both conditions, then c the algorithm stops with a warning. In this case stp only c satisfies the sufficient decrease condition. c c A typical invocation of dcsrch has the following outline: c c task = 'START' c 10 continue c call dcsrch( ... ) c if (task .eq. 'FG') then c Evaluate the function and the gradient at stp c goto 10 c end if c c NOTE: The user must no alter work arrays between calls. c c The subroutine statement is c c subroutine dcsrch(f,g,stp,ftol,gtol,xtol,stpmin,stpmax, c task,isave,dsave) c where c c f is a double precision variable. c On initial entry f is the value of the function at 0. c On subsequent entries f is the value of the c function at stp. c On exit f is the value of the function at stp. c c g is a double precision variable. c On initial entry g is the derivative of the function at 0. c On subsequent entries g is the derivative of the c function at stp. c On exit g is the derivative of the function at stp. c c stp is a double precision variable. c On entry stp is the current estimate of a satisfactory c step. On initial entry, a positive initial estimate c must be provided. c On exit stp is the current estimate of a satisfactory step c if task = 'FG'. If task = 'CONV' then stp satisfies c the sufficient decrease and curvature condition. c c ftol is a double precision variable. c On entry ftol specifies a nonnegative tolerance for the c sufficient decrease condition. c On exit ftol is unchanged. c c gtol is a double precision variable. c On entry gtol specifies a nonnegative tolerance for the c curvature condition. c On exit gtol is unchanged. c c xtol is a double precision variable. c On entry xtol specifies a nonnegative relative tolerance c for an acceptable step. The subroutine exits with a c warning if the relative difference between sty and stx c is less than xtol. c On exit xtol is unchanged. c c stpmin is a double precision variable. c On entry stpmin is a nonnegative lower bound for the step. c On exit stpmin is unchanged. c c stpmax is a double precision variable. c On entry stpmax is a nonnegative upper bound for the step. c On exit stpmax is unchanged. c c task is a character variable of length at least 60. c On initial entry task must be set to 'START'. c On exit task indicates the required action: c c If task(1:2) = 'FG' then evaluate the function and c derivative at stp and call dcsrch again. c c If task(1:4) = 'CONV' then the search is successful. c c If task(1:4) = 'WARN' then the subroutine is not able c to satisfy the convergence conditions. The exit value of c stp contains the best point found during the search. c c If task(1:5) = 'ERROR' then there is an error in the c input arguments. c c On exit with convergence, a warning or an error, the c variable task contains additional information. c c isave is an integer work array of dimension 2. c c dsave is a double precision work array of dimension 13. c c Subprograms called c c MINPACK-2 ... dcstep c c MINPACK-1 Project. June 1983. c Argonne National Laboratory. c Jorge J. More' and David J. Thuente. c c MINPACK-2 Project. October 1993. c Argonne National Laboratory and University of Minnesota. c Brett M. Averick, Richard G. Carter, and Jorge J. More'. c c ********** double precision zero,p5,p66 parameter(zero=0.0d0,p5=0.5d0,p66=0.66d0) double precision xtrapl,xtrapu parameter(xtrapl=1.1d0,xtrapu=4.0d0) logical brackt integer stage double precision finit,ftest,fm,fx,fxm,fy,fym,ginit,gtest, + gm,gx,gxm,gy,gym,stx,sty,stmin,stmax,width,width1 c Initialization block. if (task(1:5) .eq. 'START') then c Check the input arguments for errors. if (stp .lt. stpmin) task = 'ERROR: STP .LT. STPMIN' if (stp .gt. stpmax) task = 'ERROR: STP .GT. STPMAX' if (g .ge. zero) task = 'ERROR: INITIAL G .GE. ZERO' if (ftol .lt. zero) task = 'ERROR: FTOL .LT. ZERO' if (gtol .lt. zero) task = 'ERROR: GTOL .LT. ZERO' if (xtol .lt. zero) task = 'ERROR: XTOL .LT. ZERO' if (stpmin .lt. zero) task = 'ERROR: STPMIN .LT. ZERO' if (stpmax .lt. stpmin) task = 'ERROR: STPMAX .LT. STPMIN' c Exit if there are errors on input. if (task(1:5) .eq. 'ERROR') return c Initialize local variables. brackt = .false. stage = 1 finit = f ginit = g gtest = ftol*ginit width = stpmax - stpmin width1 = width/p5 c The variables stx, fx, gx contain the values of the step, c function, and derivative at the best step. c The variables sty, fy, gy contain the value of the step, c function, and derivative at sty. c The variables stp, f, g contain the values of the step, c function, and derivative at stp. stx = zero fx = finit gx = ginit sty = zero fy = finit gy = ginit stmin = zero stmax = stp + xtrapu*stp task = 'FG' goto 1000 else c Restore local variables. if (isave(1) .eq. 1) then brackt = .true. else brackt = .false. endif stage = isave(2) ginit = dsave(1) gtest = dsave(2) gx = dsave(3) gy = dsave(4) finit = dsave(5) fx = dsave(6) fy = dsave(7) stx = dsave(8) sty = dsave(9) stmin = dsave(10) stmax = dsave(11) width = dsave(12) width1 = dsave(13) endif c If psi(stp) <= 0 and f'(stp) >= 0 for some step, then the c algorithm enters the second stage. ftest = finit + stp*gtest if (stage .eq. 1 .and. f .le. ftest .and. g .ge. zero) + stage = 2 c Test for warnings. if (brackt .and. (stp .le. stmin .or. stp .ge. stmax)) + task = 'WARNING: ROUNDING ERRORS PREVENT PROGRESS' if (brackt .and. stmax - stmin .le. xtol*stmax) + task = 'WARNING: XTOL TEST SATISFIED' if (stp .eq. stpmax .and. f .le. ftest .and. g .le. gtest) + task = 'WARNING: STP = STPMAX' if (stp .eq. stpmin .and. (f .gt. ftest .or. g .ge. gtest)) + task = 'WARNING: STP = STPMIN' cc New warning statement added to eliminate the unexpected case cc of stp=stx due to possible difference between hardware precision cc and double precision. if (stp .eq. stx) + task = 'WARNING: STP = STX' c Test for convergence. if (f .le. ftest .and. abs(g) .le. gtol*(-ginit)) + task = 'CONVERGENCE' c Test for termination. if (task(1:4) .eq. 'WARN' .or. task(1:4) .eq. 'CONV') goto 1000 c A modified function is used to predict the step during the c first stage if a lower function value has been obtained but c the decrease is not sufficient. if (stage .eq. 1 .and. f .le. fx .and. f .gt. ftest) then c Define the modified function and derivative values. fm = f - stp*gtest fxm = fx - stx*gtest fym = fy - sty*gtest gm = g - gtest gxm = gx - gtest gym = gy - gtest c Call dcstep to update stx, sty, and to compute the new step. call dcstep(stx,fxm,gxm,sty,fym,gym,stp,fm,gm, + brackt,stmin,stmax) c Reset the function and derivative values for f. fx = fxm + stx*gtest fy = fym + sty*gtest gx = gxm + gtest gy = gym + gtest else c Call dcstep to update stx, sty, and to compute the new step. call dcstep(stx,fx,gx,sty,fy,gy,stp,f,g, + brackt,stmin,stmax) endif c Decide if a bisection step is needed. if (brackt) then if (abs(sty-stx) .ge. p66*width1) stp = stx + p5*(sty - stx) width1 = width width = abs(sty-stx) endif c Set the minimum and maximum steps allowed for stp. if (brackt) then stmin = min(stx,sty) stmax = max(stx,sty) else stmin = stp + xtrapl*(stp - stx) stmax = stp + xtrapu*(stp - stx) endif c Force the step to be within the bounds stpmax and stpmin. stp = max(stp,stpmin) stp = min(stp,stpmax) c If further progress is not possible, let stp be the best c point obtained during the search. if (brackt .and. (stp .le. stmin .or. stp .ge. stmax) + .or. (brackt .and. stmax-stmin .le. xtol*stmax)) stp = stx c Obtain another function and derivative. task = 'FG' 1000 continue c Save local variables. if (brackt) then isave(1) = 1 else isave(1) = 0 endif isave(2) = stage dsave(1) = ginit dsave(2) = gtest dsave(3) = gx dsave(4) = gy dsave(5) = finit dsave(6) = fx dsave(7) = fy dsave(8) = stx dsave(9) = sty dsave(10) = stmin dsave(11) = stmax dsave(12) = width dsave(13) = width1 end c====================== The end of dcsrch ============================== subroutine dcstep(stx,fx,dx,sty,fy,dy,stp,fp,dp,brackt, + stpmin,stpmax) logical brackt double precision stx,fx,dx,sty,fy,dy,stp,fp,dp,stpmin,stpmax c ********** c c Subroutine dcstep c c This subroutine computes a safeguarded step for a search c procedure and updates an interval that contains a step that c satisfies a sufficient decrease and a curvature condition. c c The parameter stx contains the step with the least function c value. If brackt is set to .true. then a minimizer has c been bracketed in an interval with endpoints stx and sty. c The parameter stp contains the current step. c The subroutine assumes that if brackt is set to .true. then c c min(stx,sty) < stp < max(stx,sty), c c and that the derivative at stx is negative in the direction c of the step. c c The subroutine statement is c c subroutine dcstep(stx,fx,dx,sty,fy,dy,stp,fp,dp,brackt, c stpmin,stpmax) c c where c c stx is a double precision variable. c On entry stx is the best step obtained so far and is an c endpoint of the interval that contains the minimizer. c On exit stx is the updated best step. c c fx is a double precision variable. c On entry fx is the function at stx. c On exit fx is the function at stx. c c dx is a double precision variable. c On entry dx is the derivative of the function at c stx. The derivative must be negative in the direction of c the step, that is, dx and stp - stx must have opposite c signs. c On exit dx is the derivative of the function at stx. c c sty is a double precision variable. c On entry sty is the second endpoint of the interval that c contains the minimizer. c On exit sty is the updated endpoint of the interval that c contains the minimizer. c c fy is a double precision variable. c On entry fy is the function at sty. c On exit fy is the function at sty. c c dy is a double precision variable. c On entry dy is the derivative of the function at sty. c On exit dy is the derivative of the function at the exit sty. c c stp is a double precision variable. c On entry stp is the current step. If brackt is set to .true. c then on input stp must be between stx and sty. c On exit stp is a new trial step. c c fp is a double precision variable. c On entry fp is the function at stp c On exit fp is unchanged. c c dp is a double precision variable. c On entry dp is the the derivative of the function at stp. c On exit dp is unchanged. c c brackt is an logical variable. c On entry brackt specifies if a minimizer has been bracketed. c Initially brackt must be set to .false. c On exit brackt specifies if a minimizer has been bracketed. c When a minimizer is bracketed brackt is set to .true. c c stpmin is a double precision variable. c On entry stpmin is a lower bound for the step. c On exit stpmin is unchanged. c c stpmax is a double precision variable. c On entry stpmax is an upper bound for the step. c On exit stpmax is unchanged. c c MINPACK-1 Project. June 1983 c Argonne National Laboratory. c Jorge J. More' and David J. Thuente. c c MINPACK-2 Project. October 1993. c Argonne National Laboratory and University of Minnesota. c Brett M. Averick and Jorge J. More'. c c ********** double precision zero,p66,two,three parameter(zero=0.0d0,p66=0.66d0,two=2.0d0,three=3.0d0) double precision gamma,p,q,r,s,sgnd,stpc,stpf,stpq,theta sgnd = dp*(dx/abs(dx)) c First case: A higher function value. The minimum is bracketed. c If the cubic step is closer to stx than the quadratic step, the c cubic step is taken, otherwise the average of the cubic and c quadratic steps is taken. if (fp .gt. fx) then theta = three*(fx - fp)/(stp - stx) + dx + dp s = max(abs(theta),abs(dx),abs(dp)) gamma = s*sqrt((theta/s)**2 - (dx/s)*(dp/s)) if (stp .lt. stx) gamma = -gamma p = (gamma - dx) + theta q = ((gamma - dx) + gamma) + dp r = p/q stpc = stx + r*(stp - stx) stpq = stx + ((dx/((fx - fp)/(stp - stx) + dx))/two)* + (stp - stx) if (abs(stpc-stx) .lt. abs(stpq-stx)) then stpf = stpc else stpf = stpc + (stpq - stpc)/two endif brackt = .true. c Second case: A lower function value and derivatives of opposite c sign. The minimum is bracketed. If the cubic step is farther from c stp than the secant step, the cubic step is taken, otherwise the c secant step is taken. else if (sgnd .lt. zero) then theta = three*(fx - fp)/(stp - stx) + dx + dp s = max(abs(theta),abs(dx),abs(dp)) gamma = s*sqrt((theta/s)**2 - (dx/s)*(dp/s)) if (stp .gt. stx) gamma = -gamma p = (gamma - dp) + theta q = ((gamma - dp) + gamma) + dx r = p/q stpc = stp + r*(stx - stp) stpq = stp + (dp/(dp - dx))*(stx - stp) if (abs(stpc-stp) .gt. abs(stpq-stp)) then stpf = stpc else stpf = stpq endif brackt = .true. c Third case: A lower function value, derivatives of the same sign, c and the magnitude of the derivative decreases. else if (abs(dp) .lt. abs(dx)) then c The cubic step is computed only if the cubic tends to infinity c in the direction of the step or if the minimum of the cubic c is beyond stp. Otherwise the cubic step is defined to be the c secant step. theta = three*(fx - fp)/(stp - stx) + dx + dp s = max(abs(theta),abs(dx),abs(dp)) c The case gamma = 0 only arises if the cubic does not tend c to infinity in the direction of the step. gamma = s*sqrt(max(zero,(theta/s)**2-(dx/s)*(dp/s))) if (stp .gt. stx) gamma = -gamma p = (gamma - dp) + theta q = (gamma + (dx - dp)) + gamma r = p/q if (r .lt. zero .and. gamma .ne. zero) then stpc = stp + r*(stx - stp) else if (stp .gt. stx) then stpc = stpmax else stpc = stpmin endif stpq = stp + (dp/(dp - dx))*(stx - stp) if (brackt) then c A minimizer has been bracketed. If the cubic step is c closer to stp than the secant step, the cubic step is c taken, otherwise the secant step is taken. if (abs(stpc-stp) .lt. abs(stpq-stp)) then stpf = stpc else stpf = stpq endif if (stp .gt. stx) then stpf = min(stp+p66*(sty-stp),stpf) else stpf = max(stp+p66*(sty-stp),stpf) endif else c A minimizer has not been bracketed. If the cubic step is c farther from stp than the secant step, the cubic step is c taken, otherwise the secant step is taken. if (abs(stpc-stp) .gt. abs(stpq-stp)) then stpf = stpc else stpf = stpq endif stpf = min(stpmax,stpf) stpf = max(stpmin,stpf) endif c Fourth case: A lower function value, derivatives of the c same sign, and the magnitude of the derivative does not c decrease. If the minimum is not bracketed, the step is either c stpmin or stpmax, otherwise the cubic step is taken. else if (brackt) then theta = three*(fp - fy)/(sty - stp) + dy + dp s = max(abs(theta),abs(dy),abs(dp)) gamma = s*sqrt((theta/s)**2 - (dy/s)*(dp/s)) if (stp .gt. sty) gamma = -gamma p = (gamma - dp) + theta q = ((gamma - dp) + gamma) + dy r = p/q stpc = stp + r*(sty - stp) stpf = stpc else if (stp .gt. stx) then stpf = stpmax else stpf = stpmin endif endif c Update the interval which contains a minimizer. if (fp .gt. fx) then sty = stp fy = fp dy = dp else if (sgnd .lt. zero) then sty = stx fy = fx dy = dx endif stx = stp fx = fp dx = dp endif c Compute the new step. stp = stpf end c====================== The end of dcstep ============================== subroutine timer(ttime) double precision ttime c ********* c c Subroutine timer c c This subroutine is used to determine user time. In a typical c application, the user time for a code segment requires calls c to subroutine timer to determine the initial and final time. c c The subroutine statement is c c subroutine timer(ttime) c c where c c ttime is an output variable which specifies the user time. c c Argonne National Laboratory and University of Minnesota. c MINPACK-2 Project. c c Modified October 1990 by Brett M. Averick. c c ********** real temp real tarray(2) real etime c The first element of the array tarray specifies user time temp = etime(tarray) ttime = dble(tarray(1)) return end c====================== The end of timer =============================== double precision function dnrm2(n,x,incx) integer n,incx double precision x(n) c ********** c c Function dnrm2 c c Given a vector x of length n, this function calculates the c Euclidean norm of x with stride incx. c c The function statement is c c double precision function dnrm2(n,x,incx) c c where c c n is a positive integer input variable. c c x is an input array of length n. c c incx is a positive integer variable that specifies the c stride of the vector. c c Subprograms called c c FORTRAN-supplied ... abs, max, sqrt c c MINPACK-2 Project. February 1991. c Argonne National Laboratory. c Brett M. Averick. c c ********** integer i double precision scale dnrm2 = 0.0d0 scale = 0.0d0 do 10 i = 1, n, incx scale = max(scale, abs(x(i))) 10 continue if (scale .eq. 0.0d0) return do 20 i = 1, n, incx dnrm2 = dnrm2 + (x(i)/scale)**2 20 continue dnrm2 = scale*sqrt(dnrm2) return end c====================== The end of dnrm2 =============================== double precision function dpmeps() c ********** c c Subroutine dpeps c c This subroutine computes the machine precision parameter c dpmeps as the smallest floating point number such that c 1 + dpmeps differs from 1. c c This subroutine is based on the subroutine machar described in c c W. J. Cody, c MACHAR: A subroutine to dynamically determine machine parameters, c ACM Trans. Math. Soft., 14, 1988, pages 303-311. c c The subroutine statement is: c c subroutine dpeps(dpmeps) c c where c c dpmeps is a double precision variable. c On entry dpmeps need not be specified. c On exit dpmeps is the machine precision. c c MINPACK-2 Project. February 1991. c Argonne National Laboratory and University of Minnesota. c Brett M. Averick. c c ******* integer i,ibeta,irnd,it,itemp,negep double precision a,b,beta,betain,betah,temp,tempa,temp1, + zero,one,two data zero,one,two /0.0d0,1.0d0,2.0d0/ c determine ibeta, beta ala malcolm. a = one b = one 10 continue a = a + a temp = a + one temp1 = temp - a if (temp1 - one .eq. zero) go to 10 20 continue b = b + b temp = a + b itemp = int(temp - a) if (itemp .eq. 0) go to 20 ibeta = itemp beta = dble(ibeta) c determine it, irnd. it = 0 b = one 30 continue it = it + 1 b = b * beta temp = b + one temp1 = temp - b if (temp1 - one .eq. zero) go to 30 irnd = 0 betah = beta/two temp = a + betah if (temp - a .ne. zero) irnd = 1 tempa = a + beta temp = tempa + betah if ((irnd .eq. 0) .and. (temp - tempa .ne. zero)) irnd = 2 c determine dpmeps. negep = it + 3 betain = one/beta a = one do 40 i = 1, negep a = a*betain 40 continue 50 continue temp = one + a if (temp - one .ne. zero) go to 60 a = a*beta go to 50 60 continue dpmeps = a if ((ibeta .eq. 2) .or. (irnd .eq. 0)) go to 70 a = (a*(one + a))/two temp = one + a if (temp - one .ne. zero) dpmeps = a 70 return end c====================== The end of dpmeps ============================== subroutine daxpy(n,da,dx,incx,dy,incy) c c constant times a vector plus a vector. c uses unrolled loops for increments equal to one. c jack dongarra, linpack, 3/11/78. c double precision dx(*),dy(*),da integer i,incx,incy,ix,iy,m,mp1,n c if(n.le.0)return if (da .eq. 0.0d0) return if(incx.eq.1.and.incy.eq.1)go to 20 c c code for unequal increments or equal increments c not equal to 1 c ix = 1 iy = 1 if(incx.lt.0)ix = (-n+1)*incx + 1 if(incy.lt.0)iy = (-n+1)*incy + 1 do 10 i = 1,n dy(iy) = dy(iy) + da*dx(ix) ix = ix + incx iy = iy + incy 10 continue return c c code for both increments equal to 1 c c c clean-up loop c 20 m = mod(n,4) if( m .eq. 0 ) go to 40 do 30 i = 1,m dy(i) = dy(i) + da*dx(i) 30 continue if( n .lt. 4 ) return 40 mp1 = m + 1 do 50 i = mp1,n,4 dy(i) = dy(i) + da*dx(i) dy(i + 1) = dy(i + 1) + da*dx(i + 1) dy(i + 2) = dy(i + 2) + da*dx(i + 2) dy(i + 3) = dy(i + 3) + da*dx(i + 3) 50 continue return end c====================== The end of daxpy =============================== subroutine dcopy(n,dx,incx,dy,incy) c c copies a vector, x, to a vector, y. c uses unrolled loops for increments equal to one. c jack dongarra, linpack, 3/11/78. c double precision dx(*),dy(*) integer i,incx,incy,ix,iy,m,mp1,n c if(n.le.0)return if(incx.eq.1.and.incy.eq.1)go to 20 c c code for unequal increments or equal increments c not equal to 1 c ix = 1 iy = 1 if(incx.lt.0)ix = (-n+1)*incx + 1 if(incy.lt.0)iy = (-n+1)*incy + 1 do 10 i = 1,n dy(iy) = dx(ix) ix = ix + incx iy = iy + incy 10 continue return c c code for both increments equal to 1 c c c clean-up loop c 20 m = mod(n,7) if( m .eq. 0 ) go to 40 do 30 i = 1,m dy(i) = dx(i) 30 continue if( n .lt. 7 ) return 40 mp1 = m + 1 do 50 i = mp1,n,7 dy(i) = dx(i) dy(i + 1) = dx(i + 1) dy(i + 2) = dx(i + 2) dy(i + 3) = dx(i + 3) dy(i + 4) = dx(i + 4) dy(i + 5) = dx(i + 5) dy(i + 6) = dx(i + 6) 50 continue return end c====================== The end of dcopy =============================== double precision function ddot(n,dx,incx,dy,incy) c c forms the dot product of two vectors. c uses unrolled loops for increments equal to one. c jack dongarra, linpack, 3/11/78. c double precision dx(*),dy(*),dtemp integer i,incx,incy,ix,iy,m,mp1,n c ddot = 0.0d0 dtemp = 0.0d0 if(n.le.0)return if(incx.eq.1.and.incy.eq.1)go to 20 c c code for unequal increments or equal increments c not equal to 1 c ix = 1 iy = 1 if(incx.lt.0)ix = (-n+1)*incx + 1 if(incy.lt.0)iy = (-n+1)*incy + 1 do 10 i = 1,n dtemp = dtemp + dx(ix)*dy(iy) ix = ix + incx iy = iy + incy 10 continue ddot = dtemp return c c code for both increments equal to 1 c c c clean-up loop c 20 m = mod(n,5) if( m .eq. 0 ) go to 40 do 30 i = 1,m dtemp = dtemp + dx(i)*dy(i) 30 continue if( n .lt. 5 ) go to 60 40 mp1 = m + 1 do 50 i = mp1,n,5 dtemp = dtemp + dx(i)*dy(i) + dx(i + 1)*dy(i + 1) + * dx(i + 2)*dy(i + 2) + dx(i + 3)*dy(i + 3) + dx(i + 4)*dy(i + 4) 50 continue 60 ddot = dtemp return end c====================== The end of ddot ================================ subroutine dpofa(a,lda,n,info) integer lda,n,info double precision a(lda,*) c c dpofa factors a double precision symmetric positive definite c matrix. c c dpofa is usually called by dpoco, but it can be called c directly with a saving in time if rcond is not needed. c (time for dpoco) = (1 + 18/n)*(time for dpofa) . c c on entry c c a double precision(lda, n) c the symmetric matrix to be factored. only the c diagonal and upper triangle are used. c c lda integer c the leading dimension of the array a . c c n integer c the order of the matrix a . c c on return c c a an upper triangular matrix r so that a = trans(r)*r c where trans(r) is the transpose. c the strict lower triangle is unaltered. c if info .ne. 0 , the factorization is not complete. c c info integer c = 0 for normal return. c = k signals an error condition. the leading minor c of order k is not positive definite. c c linpack. this version dated 08/14/78 . c cleve moler, university of new mexico, argonne national lab. c c subroutines and functions c c blas ddot c fortran sqrt c c internal variables c double precision ddot,t double precision s integer j,jm1,k c begin block with ...exits to 40 c c do 30 j = 1, n info = j s = 0.0d0 jm1 = j - 1 if (jm1 .lt. 1) go to 20 do 10 k = 1, jm1 t = a(k,j) - ddot(k-1,a(1,k),1,a(1,j),1) t = t/a(k,k) a(k,j) = t s = s + t*t 10 continue 20 continue s = a(j,j) - s c ......exit if (s .le. 0.0d0) go to 40 a(j,j) = sqrt(s) 30 continue info = 0 40 continue return end c====================== The end of dpofa =============================== subroutine dscal(n,da,dx,incx) c c scales a vector by a constant. c uses unrolled loops for increment equal to one. c jack dongarra, linpack, 3/11/78. c modified 3/93 to return if incx .le. 0. c double precision da,dx(*) integer i,incx,m,mp1,n,nincx c if( n.le.0 .or. incx.le.0 )return if(incx.eq.1)go to 20 c c code for increment not equal to 1 c nincx = n*incx do 10 i = 1,nincx,incx dx(i) = da*dx(i) 10 continue return c c code for increment equal to 1 c c c clean-up loop c 20 m = mod(n,5) if( m .eq. 0 ) go to 40 do 30 i = 1,m dx(i) = da*dx(i) 30 continue if( n .lt. 5 ) return 40 mp1 = m + 1 do 50 i = mp1,n,5 dx(i) = da*dx(i) dx(i + 1) = da*dx(i + 1) dx(i + 2) = da*dx(i + 2) dx(i + 3) = da*dx(i + 3) dx(i + 4) = da*dx(i + 4) 50 continue return end c====================== The end of dscal =============================== subroutine dtrsl(t,ldt,n,b,job,info) integer ldt,n,job,info double precision t(ldt,*),b(*) c c c dtrsl solves systems of the form c c t * x = b c or c trans(t) * x = b c c where t is a triangular matrix of order n. here trans(t) c denotes the transpose of the matrix t. c c on entry c c t double precision(ldt,n) c t contains the matrix of the system. the zero c elements of the matrix are not referenced, and c the corresponding elements of the array can be c used to store other information. c c ldt integer c ldt is the leading dimension of the array t. c c n integer c n is the order of the system. c c b double precision(n). c b contains the right hand side of the system. c c job integer c job specifies what kind of system is to be solved. c if job is c c 00 solve t*x=b, t lower triangular, c 01 solve t*x=b, t upper triangular, c 10 solve trans(t)*x=b, t lower triangular, c 11 solve trans(t)*x=b, t upper triangular. c c on return c c b b contains the solution, if info .eq. 0. c otherwise b is unaltered. c c info integer c info contains zero if the system is nonsingular. c otherwise info contains the index of c the first zero diagonal element of t. c c linpack. this version dated 08/14/78 . c g. w. stewart, university of maryland, argonne national lab. c c subroutines and functions c c blas daxpy,ddot c fortran mod c c internal variables c double precision ddot,temp integer case,j,jj c c begin block permitting ...exits to 150 c c check for zero diagonal elements. c do 10 info = 1, n c ......exit if (t(info,info) .eq. 0.0d0) go to 150 10 continue info = 0 c c determine the task and go to it. c case = 1 if (mod(job,10) .ne. 0) case = 2 if (mod(job,100)/10 .ne. 0) case = case + 2 go to (20,50,80,110), case c c solve t*x=b for t lower triangular c 20 continue b(1) = b(1)/t(1,1) if (n .lt. 2) go to 40 do 30 j = 2, n temp = -b(j-1) call daxpy(n-j+1,temp,t(j,j-1),1,b(j),1) b(j) = b(j)/t(j,j) 30 continue 40 continue go to 140 c c solve t*x=b for t upper triangular. c 50 continue b(n) = b(n)/t(n,n) if (n .lt. 2) go to 70 do 60 jj = 2, n j = n - jj + 1 temp = -b(j+1) call daxpy(j,temp,t(1,j+1),1,b(1),1) b(j) = b(j)/t(j,j) 60 continue 70 continue go to 140 c c solve trans(t)*x=b for t lower triangular. c 80 continue b(n) = b(n)/t(n,n) if (n .lt. 2) go to 100 do 90 jj = 2, n j = n - jj + 1 b(j) = b(j) - ddot(jj-1,t(j+1,j),1,b(j+1),1) b(j) = b(j)/t(j,j) 90 continue 100 continue go to 140 c c solve trans(t)*x=b for t upper triangular. c 110 continue b(1) = b(1)/t(1,1) if (n .lt. 2) go to 130 do 120 j = 2, n b(j) = b(j) - ddot(j-1,t(1,j),1,b(1),1) b(j) = b(j)/t(j,j) 120 continue 130 continue 140 continue 150 continue return end c====================== The end of dtrsl =============================== gpml-matlab-v3.6-2015-07-07/util/lbfgsb/matlabexception.cpp000644 000765 000024 00000001000 11422240325 023147 0ustar00carlstaff000000 000000 #include "matlabexception.h" // Function definitions for class MatlabException. // ----------------------------------------------------------------- MatlabException::MatlabException (const char* message) throw() : exception() { this->message = message; } MatlabException::MatlabException (const MatlabException& source) throw() : exception() { message = source.message; } MatlabException& MatlabException::operator= (const MatlabException& source) { message = source.message; return *this; } gpml-matlab-v3.6-2015-07-07/util/lbfgsb/arrayofmatrices.cpp000644 000765 000024 00000004741 12343310245 023204 0ustar00carlstaff000000 000000 #ifdef __linux__ #include "mex.h" #else #include "matrix.h" #endif #include "arrayofmatrices.h" // Function definitions for class ArrayOfMatrices. // ----------------------------------------------------------------- ArrayOfMatrices::ArrayOfMatrices (const mxArray* ptr) : Array(getnummatlabmatrices(ptr)) { setvalue(0); if (mxIsCell(ptr)) // Fill in the entries of the array with the entries of the Matlab // cell array. for (int i = 0; i < n; i++) elems[i] = new Matrix(mxGetCell(ptr,i)); else // Fill in the single entry of the array with the Matlab matrix. elems[0] = new Matrix(ptr); } ArrayOfMatrices::ArrayOfMatrices (const mxArray* ptrs[], int numptrs) : Array(numptrs) { setvalue(0); for (int i = 0; i < numptrs; i++) elems[i] = new Matrix(ptrs[i]); } ArrayOfMatrices::ArrayOfMatrices (mxArray* ptrs[], const ArrayOfMatrices& source) : Array(source.length()) { setvalue(0); // Create the new Matlab matrices and copy the data from the // source. for (int i = 0; i < n; i++) { int h = source[i]->height(); int w = source[i]->width(); elems[i] = new Matrix(ptrs[i],h,w); *elems[i] = *source[i]; } } ArrayOfMatrices::ArrayOfMatrices (const ArrayOfMatrices& source) : Array(source) { } ArrayOfMatrices::ArrayOfMatrices (double* data, const ArrayOfMatrices& model) : Array(model.length()) { setvalue(0); // Repeat for each matrix. for (int i = 0; i < n; i++) { const Matrix* A = model[i]; elems[i] = new Matrix(data,A->height(),A->width()); data += A->length(); } } ArrayOfMatrices::~ArrayOfMatrices() { for (int i = 0; i < n; i++) if (elems[i]) delete elems[i]; } int ArrayOfMatrices::numelems() const { int m = 0; // The return value. // Repeat for each matrix. for (int i = 0; i < n; i++) m += elems[i]->length(); return m; } ArrayOfMatrices& ArrayOfMatrices::operator= (const ArrayOfMatrices& source) { // Copy each matrix. for (int i = 0; i < n; i++) *elems[i] = *source.elems[i]; return *this; } int ArrayOfMatrices::getnummatlabmatrices (const mxArray* ptr) { int n; // The return value. if (mxIsCell(ptr)) n = mxGetNumberOfElements(ptr); else if (mxGetNumberOfDimensions(ptr) == 2 && mxIsDouble(ptr)) n = 1; else throw MatlabException("The Matlab array must either be a cell \ array or a double precision matrix"); return n; } gpml-matlab-v3.6-2015-07-07/util/lbfgsb/program.h000644 000765 000024 00000011647 11422240325 021126 0ustar00carlstaff000000 000000 #ifndef INCLUDE_PROGRAM #define INCLUDE_PROGRAM // Type definitions. // ----------------------------------------------------------------- // This defines the possible results of running the L-BFGS-B solver. enum SolverExitStatus { success, // The algorithm has converged to a stationary // point or has reached the maximum number of // iterations. abnormalTermination, // The algorithm has terminated abnormally and // was unable to satisfy the convergence // criteria. errorOnInput // The routine has detected an error in the // input parameters. }; // Constants. // ----------------------------------------------------------------- const int defaultm = 5; const int defaultmaxiter = 100; const double defaultfactr = 1e7; const double defaultpgtol = 1e-5; const int defaultprintlevel = -1; // Function declarations. // ----------------------------------------------------------------- // This is the L-BFGS-B routine implemented in Fortran 77. extern "C" void setulb_ (int* n, int* m, double x[], double l[], double u[], int nbd[], double* f, double g[], double* factr, double* pgtol, double wa[], int iwa[], char task[], int* iprint, char csave[], bool lsave[], int isave[], double dsave[]); // Class Program. // ----------------------------------------------------------------- // This class encapsulates execution of the L-BFGS-B routine for // solving a nonlinear optimization problem with bound constraints // using limited-memory approximations to the Hessian. // // This is an abstract class since some of the class methods have not // been implemented (they are "pure virtual" functions). In order to // use this class, one needs to define a child class and provide // definitions for the pure virtual methods. class Program { public: // The first input argument "n" is the number of variables, and "x" // must be set to the suggested starting point for the optimization // algorithm. See the L-BFGS-B documentation for more information on // the inputs to the constructor. Program (int n, double* x, double* lb, double* ub, int* btype, int m = defaultm, int maxiter = defaultmaxiter, double factr = defaultfactr, double pgtol = defaultpgtol); // This is the same constructor as above, except that the // appropriate amount of memory is dynamically allocated for the // variables, the bounds, and the bound types. The destructor also // makes sure that the memory is properly deallocated, but not so // for the other constructor. Program (int n, int m = defaultm, int maxiter = defaultmaxiter, double factr = defaultfactr, double pgtol = defaultpgtol); // The destructor. virtual ~Program(); // An implementation of this method should return the value of the // objective at the current value of the variables "x", where "n" is // the number of variables. virtual double computeObjective (int n, double* x) = 0; // An implementation of this method should fill in the values of the // gradient "g" and the current point "x". virtual void computeGradient (int n, double* x, double* g) = 0; // The child class may optionally override this method which is // called once per iteration of the L-BFGS-B routine. The input // arguments are the current iteration "t", the current value of the // variables "x" and the current value of the objective "f". virtual void iterCallback (int t, double* x, double f) { }; // Run the solver. Upon completion, the solution is stored in "x". SolverExitStatus runSolver(); protected: // The copy constructor and copy assignment operator are kept // private so that they are not used. Program (const Program& source) { }; Program& operator= (const Program& source) { return *this; }; int n; // The number of variables. double* x; // The current point. double* lb; // The lower bounds. double* ub; // The upper bounds. int* btype; // The bound types. private: // Execute a single step the L-BFGS-B solver routine. void callLBFGS (const char* cmd = 0); // Initialize some of the structures used by the L-BFGS-B routine. void initStructures(); bool owner; // If true, then the blocks of memory pointed to by // x, lb, ub and btype must be deallocated in the // destructor. double f; // The value of the objective. double* g; // The value of the gradient. int iprint; // The print level. int maxiter; // The maximum number of iterations. double pgtol; // Convergence parameter passed to L-BFGS-B. double factr; // Convergence parameter passed to L-BFGS-B. int m; // The number of variable corrections to the // limited-memory approximation to the Hessian. // These are structures used by the L-BFGS-B routine. double* wa; int* iwa; char task[60]; char csave[60]; bool lsave[4]; int isave[44]; double dsave[29]; }; #endif gpml-matlab-v3.6-2015-07-07/util/lbfgsb/matlabscalar.h000644 000765 000024 00000002264 11422240325 022100 0ustar00carlstaff000000 000000 #ifndef INCLUDE_MATLABSCALAR #define INCLUDE_MATLABSCALAR #include "mex.h" // Class MatlabScalar // ----------------------------------------------------------------- // The main appeal of this class is that one can create a scalar // object that accesses a MATLAB array. // // Note that the copy assignment operator is not valid for this class // because we cannot reassign a reference. class MatlabScalar { public: // This constructor accepts as input a pointer to a Matlab array // which must be a scalar in double precision. explicit MatlabScalar (const mxArray* ptr); // This constructor creates a new Matlab array which is a scalar // in double precision. MatlabScalar (mxArray*& ptr, double value); // The copy constructor. MatlabScalar (MatlabScalar& source); // The destructor. ~MatlabScalar() { }; // Access the value of the scalar. operator const double () const { return x; }; // Assign the value of the scalar. MatlabScalar& operator= (double value); protected: double& x; // The copy assignment operator is kept protected because it is // invalid. MatlabScalar& operator= (const MatlabScalar& source) { return *this; }; }; #endif gpml-matlab-v3.6-2015-07-07/util/lbfgsb/matlabmatrix.cpp000644 000765 000024 00000003156 11422240325 022473 0ustar00carlstaff000000 000000 #include "matlabmatrix.h" #include "matlabexception.h" // Function definitions. // ----------------------------------------------------------------- double* getMatlabMatrixDouble (const mxArray* ptr) { if (mxGetNumberOfDimensions(ptr) != 2) throw MatlabException("Matlab array must be a matrix"); if (!mxIsDouble(ptr)) throw MatlabException("Matlab array must be of type double"); return mxGetPr(ptr); } double* createMatlabMatrixDouble (mxArray*& ptr, int height, int width) { ptr = mxCreateDoubleMatrix(height,width,mxREAL); return mxGetPr(ptr); } // Function definitions for class Matrix. // ----------------------------------------------------------------- Matrix::Matrix (int height, int width) : Array(height*width) { h = height; w = width; } Matrix::Matrix (double* data, int height, int width) : Array(data,height*width) { h = height; w = width; } Matrix::Matrix (const mxArray* ptr) : Array(getMatlabMatrixDouble(ptr),mxGetNumberOfElements(ptr)) { h = mxGetM(ptr); w = mxGetN(ptr); } Matrix::Matrix (mxArray*& ptr, int height, int width) : Array(createMatlabMatrixDouble(ptr,height,width), height*width) { h = height; w = width; } Matrix::Matrix (const Matrix& source) : Array(source) { h = source.h; w = source.w; } Matrix& Matrix::operator= (const Matrix& source) { inject(source); return *this; } bool Matrix::operator== (const Matrix& X) const { return (h == X.h) && (w == X.w); } double& Matrix::entry (int r, int c) { return elems[h*c + r]; } double Matrix::entry (int r, int c) const { return elems[h*c + r]; } gpml-matlab-v3.6-2015-07-07/util/lbfgsb/matlabmatrix.h000644 000765 000024 00000004562 11422240325 022142 0ustar00carlstaff000000 000000 #ifndef INCLUDE_MATLABMATRIX #define INCLUDE_MATLABMATRIX #include "array.h" #include "mex.h" // Class Matrix // --------------------------------------------------------------- // A matrix object stores its elements in column-major format, as in // Fortran and Matlab. This means that columns are stored one after // another. For example, the matrix // // 1 2 3 // 4 5 6 // // is stored in memory as // // 1 4 2 5 3 6 // // Like Array objects, Matrix objects are not necessarily // encapsulated, and they exhibit analogous behaviour. class Matrix : public Array { public: // This constructor allocates memory for matrix of the specified // heigth and width. Matrix (int height, int width); // This constructor basically follows the lead of the analagous // constructor for the Array class. Matrix (double* data, int height, int width); // This constructor retrieves a matrix from a Matlab array. This // particular constructor is only defined for Matrix. // Since Matlab handles storage, the object created by this // constructor is not encapsulated. explicit Matrix (const mxArray* ptr); // This constructor creates a new Matlab array as a side effect. // Since Matlab handles storage, the object created by this // constructor is not encapsulated. Matrix (mxArray*& ptr, int height, int width); // The copy constructor makes a shallow copy of the data. Matrix (const Matrix& source); // The destructor. ~Matrix() { }; // Copy assignment operator that observes the same behaviour as // the Array copy assignment operator. Matrix& operator= (const Matrix& source); // Get the height and width of the matrix. int height() const { return h; }; int width () const { return w; }; // Returns true if the two matrices have the same dimensions // (i.e. the same height and width). bool operator== (const Matrix& X) const; bool operator!= (const Matrix& X) const { return !(*this == X); }; // If X is an object of type Matrix, X.entry(r,c) accesses the // entry of the rth row and cth column. double entry (int r, int c) const; double& entry (int r, int c); double operator() (int r, int c) const { return entry(r,c); }; double& operator() (int r, int c) { return entry(r,c); }; protected: int h; // The height of the matrix. int w; // The width of the matrix. }; #endif gpml-matlab-v3.6-2015-07-07/util/lbfgsb/matlabprogram.cpp000644 000765 000024 00000016452 11422240325 022641 0ustar00carlstaff000000 000000 #include "matlabprogram.h" #include "array.h" // Function definitions. // ----------------------------------------------------------------- // Copy the matrix elements to the destination array. It is assumed // that sufficient memory has been allocated for the destination // array. The copying preserves the column-major ordering of the // matrix elements. void copyElemsFromMatrix (const Matrix& source, double* dest) { Matrix destmatrix(dest,source.height(),source.width()); destmatrix = source; } // Copy the elements from the source array into the matrix. void copyElemsToMatrix (const double* source, Matrix& dest) { dest.inject(source); } // Copy the elements contained in all the matrices to the destination // array. It is assumed that sufficient memory has been allocated for // the destination array. The copying preserves the column-major // ordering of the matrix elements. void copyElemsFromArrayOfMatrices (const ArrayOfMatrices& source, double* dest) { // Repeat for each matrix. for (int i = 0; i < source.length(); i++) { copyElemsFromMatrix(*source[i],dest); dest += source[i]->length(); } } // Copy the elements from the source array into the array of matrices. void copyElemsToArrayOfMatrices (const double* source, ArrayOfMatrices& dest) { // Repeat for each matrix. for (int i = 0; i < dest.length(); i++) { copyElemsToMatrix(source,*dest[i]); source += dest[i]->length(); } } int getBoundType (double& lb, double& ub) { bool haslb = !mxIsInf(lb); bool hasub = !mxIsInf(ub); int btype = haslb + 3*hasub - 2*(haslb && hasub); lb = haslb ? lb : 0; ub = hasub ? ub : 0; return btype; } // Function definitions for class MatlabProgram. // ----------------------------------------------------------------- MatlabProgram::MatlabProgram (ArrayOfMatrices& variables, const ArrayOfMatrices& lowerbounds, const ArrayOfMatrices& upperbounds, const MatlabString* objFunc, const MatlabString* gradFunc, const MatlabString* iterFunc, mxArray* auxData, int m, int maxiter, double factr, double pgtol) : Program(variables.numelems(),0,0,0,0,m,maxiter,factr,pgtol), variables(variables) { x = new double[n]; lb = new double[n]; ub = new double[n]; btype = new int[n]; this->objFunc = objFunc; this->gradFunc = gradFunc; this->iterFunc = iterFunc; // Copy some pieces of information from the input arguments to the // respective program structures. copyElemsFromArrayOfMatrices(variables,x); copyElemsFromArrayOfMatrices(lowerbounds,lb); copyElemsFromArrayOfMatrices(upperbounds,ub); // Set the bound types. for (int i = 0; i < n; i++) btype[i] = getBoundType(lb[i],ub[i]); // Create a Matlab array for the variables. int nv = variables.length(); varInputs = new mxArray*[nv]; varMatlab = new ArrayOfMatrices(varInputs,variables); // Set up the inputs to the objective callback function. numInputsObjFunc = nv + (bool) auxData; inputsObjFunc = new mxArray*[numInputsObjFunc]; copymemory(varInputs,inputsObjFunc,nv); if (auxData) inputsObjFunc[nv] = auxData; // Set up the inputs to the gradient callback function. numInputsGradFunc = numInputsObjFunc; inputsGradFunc = inputsObjFunc; // If necessary, set up the inputs to the iterative callback // function. The first two inputs are the iteration and the function // objective. The next of inputs are the values of the variables // then, optionally, the auxiliary data. if (iterFunc) { numInputsIterFunc = 2 + nv + (bool) auxData; inputsIterFunc = new mxArray*[numInputsIterFunc]; mxArray** ptr = inputsIterFunc; tMatlab = new MatlabScalar(*ptr++,0); fMatlab = new MatlabScalar(*ptr++,0); copymemory(varInputs,ptr,nv); ptr += nv; if (auxData) *ptr = auxData; } } MatlabProgram::~MatlabProgram() { delete[] x; delete[] lb; delete[] ub; delete[] btype; // Deallocate the Matlab arrays. int nv = variables.length(); for (int i = 0; i < nv; i++) mxDestroyArray(varInputs[i]); delete varMatlab; delete[] varInputs; // If necessary, deallocate the additional Matlab arrays that act as // inputs to the iterative callback function. if (iterFunc) { delete fMatlab; delete tMatlab; // delete[] inputsIterFunc; } } double MatlabProgram::computeObjective (int n, double* x) { int nlhs = 1; // The number of outputs from Matlab. mxArray* plhs[nlhs]; // The outputs from the Matlab routine. // Copy the current value of the optimization variables. copyElemsToArrayOfMatrices(x,*varMatlab); // Call the designated Matlab routine for evaluating the objective // function. It takes as input the values of the variables, and // returns a single output, the value of the objective function. if (mexCallMATLAB(nlhs,plhs,numInputsObjFunc,inputsObjFunc,*objFunc)) throw MatlabException("Evaluation of objective function failed in \ call to MATLAB routine"); // Get the result passed back from the Matlab routine. MatlabScalar matlabOutput(plhs[0]); double objective = matlabOutput; // Free the dynamically allocated memory. mxDestroyArray(plhs[0]); return objective; } void MatlabProgram::computeGradient (int n, double* x, double* g) { int nlhs = variables.length(); // The number of outputs from Matlab. mxArray* plhs[nlhs]; // The outputs from the Matlab routine. // Copy the current value of the optimization variables. copyElemsToArrayOfMatrices(x,*varMatlab); // Call the designated Matlab routine for computing the gradient // of the objective function. It takes as input the values of the // variables, and returns as many outputs corresponding to the // partial derivatives of the objective function with respect to // the variables at their curent values. if (mexCallMATLAB(nlhs,plhs,numInputsGradFunc,inputsGradFunc,*gradFunc)) throw MatlabException("Evaluation of objective gradient failed in \ call to MATLAB routine"); // Get the result passed back from the Matlab routine. ArrayOfMatrices matlabOutput((const mxArray**) plhs,nlhs); for (int i = 0; i < variables.length(); i++) if (matlabOutput[i]->length() != variables[i]->length()) throw MatlabException("Invalid gradient passed back from MATLAB \ routine"); copyElemsFromArrayOfMatrices(matlabOutput,g); // Free the dynamically allocated Matlab outputs. for (int i = 0; i < variables.length(); i++) mxDestroyArray(plhs[i]); } void MatlabProgram::iterCallback (int t, double* x, double f) { if (iterFunc) { // Copy the current iteration, the current value of the objective // function, and the current value of the optimization variables. *tMatlab = t; *fMatlab = f; copyElemsToArrayOfMatrices(x,*varMatlab); // Call the Matlab iterative callback routine. Since there are no // outputs, there is no memory to deallocate. if (mexCallMATLAB(0,0,numInputsIterFunc,inputsIterFunc,*iterFunc)) throw MatlabException("Call to MATLAB iterative callback function \ failed"); } } SolverExitStatus MatlabProgram::runSolver() { SolverExitStatus status; // The return value. status = Program::runSolver(); // Copy the solution to the class member "variables". copyElemsToArrayOfMatrices(x,variables); return status; } gpml-matlab-v3.6-2015-07-07/mean/meanConst.m000644 000765 000024 00000001217 11447142622 020117 0ustar00carlstaff000000 000000 function A = meanConst(hyp, x, i) % Constant mean function. The mean function is parameterized as: % % m(x) = c % % The hyperparameter is: % % hyp = [ c ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-08-04. % % See also MEANFUNCTIONS.M. if nargin<2, A = '1'; return; end % report number of hyperparameters if numel(hyp)~=1, error('Exactly one hyperparameter needed.'), end c = hyp; if nargin==2 A = c*ones(size(x,1),1); % evaluate mean else if i==1 A = ones(size(x,1),1); % derivative else A = zeros(size(x,1),1); end endgpml-matlab-v3.6-2015-07-07/mean/meanSum.m000644 000765 000024 00000003511 11447142622 017574 0ustar00carlstaff000000 000000 function A = meanSum(mean, hyp, x, i) % meanSum - compose a mean function as the sum of other mean functions. % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls other mean functions to do the actual work. % % m(x) = \sum_i m_i(x) % % Copyright (c) by Carl Edward Rasmussen & Hannes Nickisch 2010-08-04. % % See also MEANFUNCTIONS.M. for ii = 1:numel(mean) % iterate over mean functions f = mean(ii); if iscell(f{:}), f = f{:}; end % expand cell array if necessary j(ii) = cellstr(feval(f{:})); % collect number hypers end if nargin<3 % report number of parameters A = char(j(1)); for ii=2:length(mean), A = [A, '+', char(j(ii))]; end; return end [n,D] = size(x); v = []; % v vector indicates to which mean parameters belong for ii = 1:length(mean), v = [v repmat(ii, 1, eval(char(j(ii))))]; end if nargin==3 % compute mean vector A = zeros(n,1); % allocate space for ii = 1:length(mean) % iteration over summand functions f = mean(ii); if iscell(f{:}), f = f{:}; end % expand cell array if needed A = A + feval(f{:}, hyp(v==ii), x); % accumulate means end else % compute derivative vector if i<=length(v) ii = v(i); % which mean function j = sum(v(1:i)==ii); % which parameter in that mean f = mean(ii); if iscell(f{:}), f = f{:}; end % dereference cell array if necessary A = feval(f{:}, hyp(v==ii), x, j); % compute derivative else A = zeros(n,1); end endgpml-matlab-v3.6-2015-07-07/mean/meanPoly.m000644 000765 000024 00000002436 12277357505 017772 0ustar00carlstaff000000 000000 function A = meanPoly(d, hyp, x, i) % meanPoly - compose a mean function as a polynomial. % % The degree d has to be a strictly positive integer. % % m(x) = sum_i=1..D sum_j=1..d a_ij * x_i^j % % The hyperparameter is: % % hyp = [ a_11 % a_21 % .. % a_D1 % a_12 % a_22 % .. % a_Dd] % % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls other mean function to do the actual work. % % Copyright (c) by Hannes Nickisch 2013-11-02. % % See also MEANFUNCTIONS.M. d = max(abs(floor(d)),1); % positive integer degree if nargin<3, A = ['D*',int2str(d)]; return; end % report number of hyperparams [n,D] = size(x); a = reshape(hyp,D,d); A = zeros(n,1); % allocate memory if nargin==3 % compute mean vector for i=1:d, A = A + (x.^i)*a(:,i); end % evaluate mean else % compute derivative vector if i<=d*D j = mod(i-1,D)+1; % which dimension A = x(:,j).^((i-j)/D+1); % derivative else A = zeros(n,1); end endgpml-matlab-v3.6-2015-07-07/mean/meanPref.m000644 000765 000024 00000001414 12441263115 017720 0ustar00carlstaff000000 000000 function A = meanPref(mean, hyp, x, varargin) % meanPref - mean function for preference learning. % % m(x) = m_0(x1)-m_0(x2), where x1=x(:,1:D), x2=x(:,D+1:2*D), D = size(x,2)/2. % % The hyperparameters are: % % hyp = [ hyp_m0 ] % % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls another mean function to do the actual work. % % See Collaborative Gaussian Processes for Preference Learning, NIPS 2014. % % Copyright (c) by Hannes Nickisch and Roman Garnett, 2014-11-01. % % See also MEANFUNCTIONS.M and COVPREF.M. if nargin<3, A = strrep(feval(mean{:}),'D','D/2'); return; end % no of params A = feval(mean{:}, hyp, x(:,1 :end/2), varargin{:}) ... - feval(mean{:}, hyp, x(:,1+end/2:end ), varargin{:}); gpml-matlab-v3.6-2015-07-07/mean/meanZero.m000644 000765 000024 00000000603 11422240323 017734 0ustar00carlstaff000000 000000 function A = meanZero(hyp, x, i) % Zero mean function. The mean function does not have any parameters. % % m(x) = 0 % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-01-10. % % See also MEANFUNCTIONS.M. if nargin<2, A = '0'; return; end % report number of hyperparameters A = zeros(size(x,1),1); % derivative and mean gpml-matlab-v3.6-2015-07-07/mean/meanPow.m000644 000765 000024 00000001713 12240130514 017564 0ustar00carlstaff000000 000000 function A = meanPow(d, mean, hyp, x, i) % meanPow - compose a mean function as the power of another mean function m0. % % The degree d has to be a strictly positive integer. % % m(x) = m_0(x) ^ d % % The hyperparameter is: % % hyp = [ ] % % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls other mean function to do the actual work. % % Copyright (c) by Carl Edward Rasmussen & Hannes Nickisch 2013-11-02. % % See also MEANFUNCTIONS.M. d = max(abs(floor(d)),1); % positive integer degree only if nargin<4 % report number of parameters A = feval(mean{:}); return end [n,D] = size(x); if nargin==4 % compute mean vector A = feval(mean{:},hyp,x).^d; else % compute derivative vector A = ( d*feval(mean{:},hyp,x).^(d-1) ).* feval(mean{:},hyp,x,i); endgpml-matlab-v3.6-2015-07-07/mean/meanMask.m000644 000765 000024 00000002136 12240130514 017712 0ustar00carlstaff000000 000000 function A = meanMask(mask, mean, hyp, x, i) % Apply a mean function to a subset of the dimensions only. The subset can % either be specified by a 0/1 mask by a boolean mask or by an index set. % % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls another mean function to do the actual work. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-11-02. % % See also MEANFUNCTIONS.M. nh_string = feval(mean{:}); % number of hyperparameters of the full mean if max(mask)<2 && length(mask)>1, mask = find(mask); end % convert 1/0->index D = length(mask); % masked dimension if nargin<4, A = num2str(eval(nh_string)); return; end % number of parameters if eval(nh_string)~=length(hyp) % check hyperparameters error('number of hyperparameters does not match size of masked data') end if nargin==4 A = feval(mean{:}, hyp, x(:,mask)); else % compute derivative A = feval(mean{:}, hyp, x(:,mask), i); end gpml-matlab-v3.6-2015-07-07/mean/meanLinear.m000644 000765 000024 00000001333 12240130514 020227 0ustar00carlstaff000000 000000 function A = meanLinear(hyp, x, i) % Linear mean function. The mean function is parameterized as: % % m(x) = sum_i a_i * x_i; % % The hyperparameter is: % % hyp = [ a_1 % a_2 % .. % a_D ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-01-10. % % See also MEANFUNCTIONS.M. if nargin<2, A = 'D'; return; end % report number of hyperparameters [n,D] = size(x); if any(size(hyp)~=[D,1]), error('Exactly D hyperparameters needed.'), end a = hyp(:); if nargin==2 A = x*a; % evaluate mean else if i<=D A = x(:,i); % derivative else A = zeros(n,1); end endgpml-matlab-v3.6-2015-07-07/mean/meanOne.m000644 000765 000024 00000000746 11447142622 017560 0ustar00carlstaff000000 000000 function A = meanOne(hyp, x, i) % One mean function. The mean function does not have any parameters. % % m(x) = 1 % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-08-04. % % See also MEANFUNCTIONS.M. if nargin<2, A = '0'; return; end % report number of hyperparameters if nargin==2 A = ones(size(x,1),1); % evaluate mean else A = zeros(size(x,1),1); % derivative endgpml-matlab-v3.6-2015-07-07/mean/meanGPexact.m000644 000765 000024 00000004547 12441263115 020371 0ustar00carlstaff000000 000000 function A = meanGPexact(mean,cov,x,y, hypz,z,i) % Mean function being the predictive mean of a GP model: % % m(z) = posterior mean of GP at location z as given by % m(z) = gp(hyp,@infExact,mean,cov,@likGauss,x,y, z) where % hyp.mean = hyp_mean; hyp.lik = log(sn); hyp.cov = hyp.cov; % % The hyperparameters are: % % hypz = [ hyp_cov % log(sn) % hyp_mean ] % % where hyp_cov are the covariance function hyperparameters, sn is the % noise variance of the Gaussian likelihood and hyp_mean are the mean % function hyperparameters. % % Copyright (c) by Hannes Nickisch, 2014-11-01. % % See also MEANFUNCTIONS.M and MEANGP.M. if nargin<4, error('GP must be specified.'), end % check for dimension if isempty(mean), mean = @meanZero; end % set default and make cell if ~iscell(mean), mean = {mean}; end if isempty(cov), cov = @covSEiso; end if ~iscell(cov), cov = {cov}; end nms = feval(mean{:}); ncs = feval(cov{:}); % number of hyperparameter string if nargin<6, A = [ncs,'+1+',nms]; return, end % report number of hyperparameters [nz,D] = size(z); n = size(x,1); nc = eval(ncs); nm = eval(nms); hyp = rewrap(struct('cov',zeros(nc,1),'lik',0,'mean',zeros(nm,1)),hypz); m = feval(mean{:},hyp.mean,x); mz = feval(mean{:},hyp.mean,z); K = feval(cov{:}, hyp.cov, x); kz = feval(cov{:}, hyp.cov, x,z); sn2 = exp(2*hyp.lik); % noise variance of likGauss if sn2<1e-6 % very tiny sn2 can lead to numerical trouble L = chol(K+sn2*eye(n)); sl = 1; % Cholesky factor of covariance with noise else L = chol(K/sn2+eye(n)); sl = sn2; % Cholesky factor of B end iKs = @(t) solve_chol(L,t)/sl; % iKs(t) = (K+sn2*eye(n))\t alpha = iKs(y-m); if nargin==6 % eval posterior mean A = mz+kz'*alpha; else if i<=nc % covariance function hypers dK = feval(cov{:},hyp.cov,x,[],i); dkz = feval(cov{:},hyp.cov,x,z,i); A = dkz'*alpha-kz'*iKs(dK*alpha); elseif i==nc+1 % likelihood function parameter sn A = -2*sn2*kz'*iKs(alpha); else % mean function parameter dm = feval(mean{:},hyp.mean,x,i-nc-1); dmz = feval(mean{:},hyp.mean,z,i-nc-1); A = dmz-kz'*iKs(dm); end end gpml-matlab-v3.6-2015-07-07/mean/meanGP.m000644 000765 000024 00000001307 12441263115 017333 0ustar00carlstaff000000 000000 function A = meanGP(hyp,inf,mean,cov,lik,x,y, hypz,z,i) % Mean function being the predictive mean of a GP model: % % m(z) = posterior mean of another GP at location z as given by % m(z) = gp(hyp,inf,mean,cov,lik,x,y, z) % % The hyperparameters are: % % hypz = [ ] % % Copyright (c) by Hannes Nickisch, 2014-11-01. % % See also MEANFUNCTIONS.M and MEANGPEXACT.M. if nargin<7, error('GP must be specified.'), end % check for dimension if nargin<9, A = '0'; return, end % report number of hyperparameters if nargin==9 A = gp(hyp,inf,mean,cov,lik,x,y, z); % evaluate posterior mean else A = zeros(size(x,1),1); % derivative end gpml-matlab-v3.6-2015-07-07/mean/meanDiscrete.m000644 000765 000024 00000001633 12441263115 020571 0ustar00carlstaff000000 000000 function A = meanDiscrete(s, hyp, x, i) % Mean function for discrete inputs x. Given a function defined on the % integers 1,2,3,..,s, the mean function is parametrized as: % % m(x) = mu_x, % % where mu is a fixed vector of length s. % % This implementation assumes that the inputs x are given as integers % between 1 and s, which simply index the provided vector. % % The hyperparameters are: % % hyp = [ mu_1 % mu_2 % .. % mu_s ] % % Copyright (c) by Roman Garnett, 2014-08-14. % % See also COVDISCRETE.M, MEANFUNCTIONS.M. if nargin==0, error('s must be specified.'), end % check for dimension if nargin<=2, A = num2str(s); return; end % report number of hyperparameters mu = hyp(:); if nargin==3 A = mu(x(:)); % evaluate mean else A = zeros(numel(x),1); % derivative A(x==i) = 1; endgpml-matlab-v3.6-2015-07-07/mean/meanNN.m000644 000765 000024 00000001414 12441263115 017337 0ustar00carlstaff000000 000000 function A = meanNN(c,m, hyp, x, i) % Nearest neighbor mean function. The mean function is parameterized as: % % m(z) = m_j, j = arg min_i d(ci,x) where d is the Euclidean distance and ci is % the ith cluster center. % % The hyperparameters are: % % hyp = [ ] % % Copyright (c) by Hannes Nickisch, 2014-12-04. % % See also MEANFUNCTIONS.M. if nargin<4, A = '0'; return; end % report number of hyperparameters if numel(hyp)~=0, error('No hyperparameters needed for this model.'), end if nargin==4 % evaluate mean [junk,j] = min(sq_dist(c',x')); A = m(j); A = A(:); else A = zeros(size(x,1),1); % derivative endgpml-matlab-v3.6-2015-07-07/mean/meanProd.m000644 000765 000024 00000003754 11447142622 017745 0ustar00carlstaff000000 000000 function A = meanProd(mean, hyp, x, i) % meanProd - compose a mean function as the product of other mean functions. % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls other mean functions to do the actual work. % % m(x) = \prod_i m_i(x) % % Copyright (c) by Carl Edward Rasmussen & Hannes Nickisch 2010-08-04. % % See also MEANFUNCTIONS.M. for ii = 1:numel(mean) % iterate over mean functions f = mean(ii); if iscell(f{:}), f = f{:}; end % expand cell array if necessary j(ii) = cellstr(feval(f{:})); % collect number hypers end if nargin<3 % report number of parameters A = char(j(1)); for ii=2:length(mean), A = [A, '+', char(j(ii))]; end; return end [n,D] = size(x); v = []; % v vector indicates to which mean parameters belong for ii = 1:length(mean), v = [v repmat(ii, 1, eval(char(j(ii))))]; end A = ones(n,1); % allocate space if nargin==3 % compute mean vector for ii = 1:length(mean) % iteration over summand functions f = mean(ii); if iscell(f{:}), f = f{:}; end % expand cell array if needed A = A.*feval(f{:}, hyp(v==ii), x); % accumulate means end else % compute derivative vector if i<=length(v) ii = v(i); % which mean function j = sum(v(1:i)==ii); % which parameter in that mean for jj = 1:length(mean) f = mean(jj); if iscell(f{:}), f = f{:}; end % dereference cell array if necessary if jj==ii A = A .* feval(f{:}, hyp(v==jj), x, j); % multiply derivative else A = A .* feval(f{:}, hyp(v==jj), x); % multiply mean end end else A = zeros(n,1); end endgpml-matlab-v3.6-2015-07-07/mean/meanScale.m000644 000765 000024 00000001622 12441263115 020054 0ustar00carlstaff000000 000000 function A = meanScale(mean, hyp, x, i) % meanScale - compose a mean function as a scaled version of another one. % % m(x) = a * m_0(x) % % The hyperparameters are: % % hyp = [ a; % hyp_m0 ] % % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls other mean functions to do the actual work. % % Copyright (c) by Carl Edward Rasmussen & Hannes Nickisch 2014-11-01. % % See also MEANFUNCTIONS.M. if nargin<3 % report number of parameters A = [feval(mean{:}),'+1']; return end [n,D] = size(x); a = hyp(1); if nargin==3 % compute mean vector A = a*feval(mean{:},hyp(2:end),x); else % compute derivative vector if i==1 A = feval(mean{:},hyp(2:end),x); else A = a*feval(mean{:},hyp(2:end),x,i-1); end end