gpml-matlab-v3.5-2014-12-08/000775 000765 000024 00000000000 12526350576 015076 5ustar00carlstaff000000 000000 gpml-matlab-v3.5-2014-12-08/.octaverc000644 000765 000024 00000000010 11414675757 016700 0ustar00carlstaff000000 000000 startup gpml-matlab-v3.5-2014-12-08/priorDistributions.m000664 000765 000024 00000003767 12526350575 021206 0ustar00carlstaff000000 000000 % prior distributions to be used for hyperparameters of Gaussian processes % using infPrior. % There are two different kinds of prior distributions: simple and composite: % % simple prior distributions: % % priorGauss - univariate Gaussian % priorLaplace - univariate Laplace % priorT - univariate Student's t % % priorSmoothBox1 - univariate interval (linear decay in log domain) % priorSmoothBox2 - univariate interval (quadr. decay in log domain) % % priorGamma - univariate Gamma, IR+ % priorWeibull - univariate Weibull, IR+ % priorInvGauss - univariate Inverse Gaussian, IR+ % priorLogNormal - univariate Log-normal, IR+ % % priorClamped or - fix hyperparameter to its current value by setting % priorDelta derivatives to zero, no effect on marginal likelihood % % priorGaussMulti - multivariate Gauss % priorLaplaceMulti - multivariate Laplace % priorTMulti - multivariate Student's t % % priorClampedMulti or - fix hyperparameter to its current value by setting % priorDeltaMulti derivatives to zero, no effect on marginal likelihood % % composite prior distributions (see explanation at the bottom): % % priorMix - nonnegative mixture of priors % priorTransform - prior on g(t) rather than t % % Naming convention: all prior distributions are named "prior/prior*.m". % % % 1) With only a fixed input arguments: % % r = priorNAME(par1,par2,parN) % % The function returns a random sample from the distribution for e.g. % random restarts, simulations or optimisation initialisation. % % 2) With one additional input arguments: % % [lp,dlp] = priorNAME(par1,par2,parN, t) % % The function returns the log density at location t along with its first % derivative. % % See also doc/usagePrior.m, inf/infPrior.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-05-18. % File automatically generated using noweb. gpml-matlab-v3.5-2014-12-08/gp.m000664 000765 000024 00000023062 12526350575 015664 0ustar00carlstaff000000 000000 function [varargout] = gp(hyp, inf, mean, cov, lik, x, y, xs, ys) % Gaussian Process inference and prediction. The gp function provides a % flexible framework for Bayesian inference and prediction with Gaussian % processes for scalar targets, i.e. both regression and binary % classification. The prior is Gaussian process, defined through specification % of its mean and covariance function. The likelihood function is also % specified. Both the prior and the likelihood may have hyperparameters % associated with them. % % Two modes are possible: training or prediction: if no test cases are % supplied, then the negative log marginal likelihood and its partial % derivatives w.r.t. the hyperparameters is computed; this mode is used to fit % the hyperparameters. If test cases are given, then the test set predictive % probabilities are returned. Usage: % % training: [nlZ dnlZ ] = gp(hyp, inf, mean, cov, lik, x, y); % prediction: [ymu ys2 fmu fs2 ] = gp(hyp, inf, mean, cov, lik, x, y, xs); % or: [ymu ys2 fmu fs2 lp] = gp(hyp, inf, mean, cov, lik, x, y, xs, ys); % % where: % % hyp struct of column vectors of mean/cov/lik hyperparameters % inf function specifying the inference method % mean prior mean function % cov prior covariance function % lik likelihood function % x n by D matrix of training inputs % y column vector of length n of training targets % xs ns by D matrix of test inputs % ys column vector of length nn of test targets % % nlZ returned value of the negative log marginal likelihood % dnlZ struct of column vectors of partial derivatives of the negative % log marginal likelihood w.r.t. mean/cov/lik hyperparameters % ymu column vector (of length ns) of predictive output means % ys2 column vector (of length ns) of predictive output variances % fmu column vector (of length ns) of predictive latent means % fs2 column vector (of length ns) of predictive latent variances % lp column vector (of length ns) of log predictive probabilities % % post struct representation of the (approximate) posterior % 3rd output in training mode or 6th output in prediction mode % can be reused in prediction mode gp(.., cov, lik, x, post, xs,..) % % See also infMethods.m, meanFunctions.m, covFunctions.m, likFunctions.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-05-18. % File automatically generated using noweb. if nargin<7 || nargin>9 disp('Usage: [nlZ dnlZ ] = gp(hyp, inf, mean, cov, lik, x, y);') disp(' or: [ymu ys2 fmu fs2 ] = gp(hyp, inf, mean, cov, lik, x, y, xs);') disp(' or: [ymu ys2 fmu fs2 lp] = gp(hyp, inf, mean, cov, lik, x, y, xs, ys);') return end if isempty(mean), mean = {@meanZero}; end % set default mean if ischar(mean) || isa(mean, 'function_handle'), mean = {mean}; end % make cell if isempty(cov), error('Covariance function cannot be empty'); end % no default if ischar(cov) || isa(cov,'function_handle'), cov = {cov}; end % make cell cstr = cov{1}; if isa(cstr,'function_handle'), cstr = func2str(cstr); end if strcmp(cstr,'covFITC') && isfield(hyp,'xu'), cov{3} = hyp.xu; end %use hyp.xu if isempty(inf) % set default inference method if strcmp(cstr,'covFITC'), inf = {@infFITC}; else inf = {@infExact}; end end if ischar(inf), inf = str2func(inf); end % convert into function handle if ischar(inf) || isa(inf,'function_handle'), inf = {inf}; end % make cell istr = inf{1}; if isa(istr,'function_handle'), istr = func2str(istr); end if strcmp(istr,'infPrior') istr = inf{2}; if isa(istr,'function_handle'), istr = func2str(istr); end end if strcmp(cstr,'covFITC') % only infFITC* are possible if isempty(strfind(istr,'infFITC')==1) error('Only infFITC* are possible inference algorithms') end end % only one possible class of inference algorithms if isempty(lik), lik = {@likGauss}; end % set default lik if ischar(lik) || isa(lik,'function_handle'), lik = {lik}; end % make cell lstr = lik{1}; if isa(lstr,'function_handle'), lstr = func2str(lstr); end D = size(x,2); if strcmp(cstr,'covGrid') % only infGrid* are possible if isempty(strfind(istr,'infGrid')==1) error('Only infGrid* are possible inference algorithms') end D = 0; xg = cov{3}; p = numel(xg); for i=1:p, D = D+size(xg{i},2); end % dims end if ~isfield(hyp,'mean'), hyp.mean = []; end % check the hyp specification if eval(feval(mean{:})) ~= numel(hyp.mean) error('Number of mean function hyperparameters disagree with mean function') end if ~isfield(hyp,'cov'), hyp.cov = []; end if eval(feval(cov{:})) ~= numel(hyp.cov) error('Number of cov function hyperparameters disagree with cov function') end if ~isfield(hyp,'lik'), hyp.lik = []; end if eval(feval(lik{:})) ~= numel(hyp.lik) error('Number of lik function hyperparameters disagree with lik function') end try % call the inference method % issue a warning if a classification likelihood is used in conjunction with % labels different from +1 and -1 if strcmp(lstr,'likErf') || strcmp(lstr,'likLogistic') if ~isstruct(y) uy = unique(y); if any( uy~=+1 & uy~=-1 ) warning('You try classification with labels different from {+1,-1}') end end end if nargin>7 % compute marginal likelihood and its derivatives only if needed if isstruct(y) post = y; % reuse a previously computed posterior approximation else post = feval(inf{:}, hyp, mean, cov, lik, x, y); end else if nargout<=1 [post nlZ] = feval(inf{:}, hyp, mean, cov, lik, x, y); dnlZ = {}; else [post nlZ dnlZ] = feval(inf{:}, hyp, mean, cov, lik, x, y); end end catch msgstr = lasterr; if nargin>7, error('Inference method failed [%s]', msgstr); else warning('Inference method failed [%s] .. attempting to continue',msgstr) dnlZ = struct('cov',0*hyp.cov, 'mean',0*hyp.mean, 'lik',0*hyp.lik); varargout = {NaN, dnlZ}; return % continue with a warning end end if nargin==7 % if no test cases are provided varargout = {nlZ, dnlZ, post}; % report -log marg lik, derivatives and post else alpha = post.alpha; L = post.L; sW = post.sW; if issparse(alpha) % handle things for sparse representations nz = alpha ~= 0; % determine nonzero indices if issparse(L), L = full(L(nz,nz)); end % convert L and sW if necessary if issparse(sW), sW = full(sW(nz)); end else nz = true(size(alpha,1),1); end % non-sparse representation if isempty(L) % in case L is not provided, we compute it K = feval(cov{:}, hyp.cov, x(nz,:)); L = chol(eye(sum(nz))+sW*sW'.*K); end %verify whether L contains valid Cholesky decomposition or something different Lchol = isnumeric(L) && all(all(tril(L,-1)==0)&diag(L)'>0&isreal(diag(L))'); ns = size(xs,1); % number of data points if strcmp(cstr,'covGrid'), xs = covGrid('idx2dat',cov{3},xs); end % expand xs nperbatch = 1000; % number of data points per mini batch nact = 0; % number of already processed test data points ymu = zeros(ns,1); ys2 = ymu; fmu = ymu; fs2 = ymu; lp = ymu; % allocate mem while nact use Cholesky parameters (alpha,sW,L) V = L'\(repmat(sW,1,length(id)).*Ks); fs2(id) = kss - sum(V.*V,1)'; % predictive variances else % L is not triangular => use alternative parametrisation if isnumeric(L), LKs = L*Ks; else LKs = L(Ks); end % matrix or callback fs2(id) = kss + sum(Ks.*LKs,1)'; % predictive variances end fs2(id) = max(fs2(id),0); % remove numerical noise i.e. negative variances Fs2 = repmat(fs2(id),1,N); % we have multiple values in case of sampling if nargin<9 [Lp, Ymu, Ys2] = feval(lik{:},hyp.lik,[], Fmu(:),Fs2(:)); else Ys = repmat(ys(id),1,N); [Lp, Ymu, Ys2] = feval(lik{:},hyp.lik,Ys(:),Fmu(:),Fs2(:)); end lp(id) = sum(reshape(Lp, [],N),2)/N; % log probability; sample averaging ymu(id) = sum(reshape(Ymu,[],N),2)/N; % predictive mean ys|y and .. ys2(id) = sum(reshape(Ys2,[],N),2)/N; % .. variance nact = id(end); % set counter to index of last processed data point end if nargin<9 varargout = {ymu, ys2, fmu, fs2, [], post}; % assign output arguments else varargout = {ymu, ys2, fmu, fs2, lp, post}; end end gpml-matlab-v3.5-2014-12-08/mean/000775 000765 000024 00000000000 12526350576 016016 5ustar00carlstaff000000 000000 gpml-matlab-v3.5-2014-12-08/util/000775 000765 000024 00000000000 12526350576 016053 5ustar00carlstaff000000 000000 gpml-matlab-v3.5-2014-12-08/likFunctions.m000664 000765 000024 00000010312 12526350575 017720 0ustar00carlstaff000000 000000 % likelihood functions are provided to be used by the gp.m function: % % likErf (Error function, classification, probit regression) % likLogistic (Logistic, classification, logit regression) % likUni (Uniform likelihood, classification) % % likGauss (Gaussian, regression) % likGaussWarp (Warped Gaussian, regression) % likGumbel (Gumbel likelihood for extremal values) % likLaplace (Laplacian or double exponential, regression) % likSech2 (Sech-square, regression) % likT (Student's t, regression) % % likPoisson (Poisson regression, count data) % likGamma (Nonnegative regression, positive data) % likExp (Nonnegative regression, positive data) % likInvGauss (Nonnegative regression, positive data) % likBeta (Beta regression, interval data) % % likMix (Mixture of individual covariance functions) % % The likelihood functions have three possible modes, the mode being selected % as follows (where "lik" stands for any likelihood function in "lik/lik*.m".): % % 1) With one or no input arguments: [REPORT NUMBER OF HYPERPARAMETERS] % % s = lik OR s = lik(hyp) % % The likelihood function returns a string telling how many hyperparameters it % expects, using the convention that "D" is the dimension of the input space. % For example, calling "likLogistic" returns the string '0'. % % % 2) With three or four input arguments: [PREDICTION MODE] % % lp = lik(hyp, y, mu) OR [lp, ymu, ys2] = lik(hyp, y, mu, s2) % % This allows to evaluate the predictive distribution. Let p(y_*|f_*) be the % likelihood of a test point and N(f_*|mu,s2) an approximation to the posterior % marginal p(f_*|x_*,x,y) as returned by an inference method. The predictive % distribution p(y_*|x_*,x,y) is approximated by. % q(y_*) = \int N(f_*|mu,s2) p(y_*|f_*) df_* % % lp = log( q(y) ) for a particular value of y, if s2 is [] or 0, this % corresponds to log( p(y|mu) ) % ymu and ys2 the mean and variance of the predictive marginal q(y) % note that these two numbers do not depend on a particular % value of y % All vectors have the same size. % % % 3) With five or six input arguments, the fifth being a string [INFERENCE MODE] % % [varargout] = lik(hyp, y, mu, s2, inf) OR % [varargout] = lik(hyp, y, mu, s2, inf, i) % % There are three cases for inf, namely a) infLaplace, b) infEP and c) infVB. % The last input i, refers to derivatives w.r.t. the ith hyperparameter. % % a1) [lp,dlp,d2lp,d3lp] = lik(hyp, y, f, [], 'infLaplace') % lp, dlp, d2lp and d3lp correspond to derivatives of the log likelihood % log(p(y|f)) w.r.t. to the latent location f. % lp = log( p(y|f) ) % dlp = d log( p(y|f) ) / df % d2lp = d^2 log( p(y|f) ) / df^2 % d3lp = d^3 log( p(y|f) ) / df^3 % % a2) [lp_dhyp,dlp_dhyp,d2lp_dhyp] = lik(hyp, y, f, [], 'infLaplace', i) % returns derivatives w.r.t. to the ith hyperparameter % lp_dhyp = d log( p(y|f) ) / ( dhyp_i) % dlp_dhyp = d^2 log( p(y|f) ) / (df dhyp_i) % d2lp_dhyp = d^3 log( p(y|f) ) / (df^2 dhyp_i) % % % b1) [lZ,dlZ,d2lZ] = lik(hyp, y, mu, s2, 'infEP') % let Z = \int p(y|f) N(f|mu,s2) df then % lZ = log(Z) % dlZ = d log(Z) / dmu % d2lZ = d^2 log(Z) / dmu^2 % % b2) [dlZhyp] = lik(hyp, y, mu, s2, 'infEP', i) % returns derivatives w.r.t. to the ith hyperparameter % dlZhyp = d log(Z) / dhyp_i % % % c1) [b,z] = lik(hyp, y, [], ga, 'infVB') % ga is the variance of a Gaussian lower bound to the likelihood p(y|f). % p(y|f) \ge exp( b*(f+z) - (f+z).^2/(2*ga) - h(ga)/2 ) \propto N(f|b*ga-z,ga) % The function returns the linear part b and z. % % Cumulative likelihoods are designed for binary classification. Therefore, they % only look at the sign of the targets y; zero values are treated as +1. % % Some examples for valid likelihood functions: % lik = @likLogistic; % lik = {'likMix',{'likUni',@likErf}} % lik = {@likPoisson,'logistic'}; % % See the help for the individual likelihood for the computations specific to % each likelihood function. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-05-18. % File automatically generated using noweb. gpml-matlab-v3.5-2014-12-08/covFunctions.m000664 000765 000024 00000013571 12526350575 017742 0ustar00carlstaff000000 000000 % covariance functions to be use by Gaussian process functions. There are two % different kinds of covariance functions: simple and composite: % % simple covariance functions: % covConst - covariance for constant functions % covCos - sine periodic covariance function (1d) with unit period % covLIN - linear covariance function without parameters % covLINard - linear covariance function with ARD % covLINiso - linear covariance function % covLINone - linear covariance function with bias % covMaternard - Matern covariance function with nu=1/2, 3/2 or 5/2 with ARD % covMaterniso - Matern covariance function with nu=1/2, 3/2 or 5/2 % covNNone - neural network covariance function % covNoise - independent covariance function (i.e. white noise) % covPeriodic - smooth periodic covariance function (1d) % covPeriodicNoDC - as above but with zero DC component and properly scaled % covPoly - polynomial covariance function % covPPard - piecewise polynomial covariance function (compact support) % covPPiso - piecewise polynomial covariance function (compact support) % covRQard - rational quadratic covariance function with ARD % covRQiso - isotropic rational quadratic covariance function % covSEard - squared exponential covariance function with ARD % covSEiso - isotropic squared exponential covariance function % covSEisoU - same as above but without latent scale % covSEvlen - spatially varying lengthscale squared exponential % covSEfact - factor analysis squared exponential covariance function % covSM - spectral mixture covariance function % covGaborard - Gabor covariance function with ARD % covGaborsio - isotropic Gabor covariance function % covDiscrete - precomputed covariance for discrete data % % composite (meta) covariance functions (see explanation at the bottom): % covScale - scaled version of a covariance function % covProd - products of covariance functions % covSum - sums of covariance functions % covADD - additive covariance function % covMask - mask some dimensions of the data % covPERard - make ARD stationary covariance periodic % covPERiso - make isotropic stationary covariance periodic % covPref - difference covariance for preference learning % % special purpose (wrapper) covariance functions % covFITC - to be used in conjunction with infFITC* for large scale % inference problems; any covariance can be wrapped by % covFITC such that the FITC approximation is applicable % covGrid - to be used in conjunction with infGrid* for large scale % inference problems on grids resulting Kronecker structure % % Naming convention: all covariance functions are named "cov/cov*.m". A trailing % "iso" means isotropic, "ard" means Automatic Relevance Determination, and % "one" means that the distance measure is parameterized by a single parameter. % % The covariance functions are written according to a special convention where % the exact behaviour depends on the number of input and output arguments % passed to the function. If you want to add new covariance functions, you % should follow this convention if you want them to work with the function gp. % There are four different ways of calling the covariance functions: % % 1) With no (or one) input argument(s): % % s = cov % % The covariance function returns a string s telling how many hyperparameters it % expects, using the convention that "D" is the dimension of the input space. % For example, calling "covRQard" returns the string '(D+2)'. % % 2) With two input arguments: % % K = cov(hyp, x) equivalent to K = cov(hyp, x, []) % % The function computes and returns the covariance matrix where hyp are % the hyperparameters and x is an n by D matrix of cases, where % D is the dimension of the input space. The returned covariance matrix is of % size n by n. % % 3) With three input arguments: % % Ks = cov(hyp, x, xs) % kss = cov(hyp, xs, 'diag') % % The function computes test set covariances; kss is a vector of self covariances % for the test cases in xs (of length ns) and Ks is an (n by ns) matrix of cross % covariances between training cases x and test cases xs. % % 4) With four input arguments: % % dKi = cov(hyp, x, [], i) % dKsi = cov(hyp, x, xs, i) % dkssi = cov(hyp, xs, 'diag', i) % % The function computes and returns the partial derivatives of the % covariance matrices with respect to hyp(i), i.e. with % respect to the hyperparameter number i. % % Covariance functions can be specified in two ways: either as a string % containing the name of the covariance function or using a cell array. For % example: % % cov = 'covRQard'; % cov = {'covRQard'}; % cov = {@covRQard}; % % are supported. Only the second and third form using the cell array can be used % for specifying composite covariance functions, made up of several % contributions. For example: % % cov = {'covScale', {'covRQiso'}}; % cov = {'covSum', {'covRQiso','covSEard','covNoise'}}; % cov = {'covProd',{'covRQiso','covSEard','covNoise'}}; % cov = {'covMask',{mask,'covSEiso'}} % q=1; cov = {'covPPiso',q}; % d=3; cov = {'covPoly',d}; % cov = {'covADD',{[1,2],'covSEiso'}}; % cov = {@covFITC, {@covSEiso}, u}; where u are the inducing inputs % % specifies a covariance function which is the sum of three contributions. To % find out how many hyperparameters this covariance function requires, we do: % % feval(cov{:}) % % which returns the string '3+(D+1)+1' (i.e. the 'covRQiso' contribution uses % 3 parameters, the 'covSEard' uses D+1 and 'covNoise' a single parameter). % % See also doc/usageCov.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-05-18. % File automatically generated using noweb. gpml-matlab-v3.5-2014-12-08/README000644 000765 000024 00000037665 12526350575 015774 0ustar00carlstaff000000 000000 GAUSSIAN PROCESS REGRESSION AND CLASSIFICATION Toolbox version 3.5 for GNU Octave 3.2.x and Matlab 7.x Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2014-12-08. 0) HOW TO READ ============== If you want to get started without further delay, then read section 1) below and jump right to the examples in doc/index.html. 1) ABOUT THESE PROGRAMS ======================= This collection of matlab programs implements and demonstrates some of the algorithms described in a) the book by Rasmussen and Williams: "Gaussian Processes for Machine Learning", the MIT Press 2006, in b) the article by Nickisch and Rasmussen: "Approximations for Binary Gaussian Process Classification", JMLR 2008, in c) the article by Candela and Rasmussen: "A Unifying View of Sparse Approximate Gaussian Process Regression", JMLR 2005, in d) the paper by Murray, Adams and Mackay: "Elliptical slice sampling", AISTATS 2010, in e) the report by Neal: "Annealed Importance Sampling", UToronto 1998, in f) the paper by Naish-Guzman and Holden: "The Generalized FITC Approximation", NIPS, 2007, in g) the paper by Duvenaud, Nickisch and Rasmussen: "Additive Gaussian Processes", NIPS, 2011, in h) the paper by Wilson and Adams: "Gaussian Process Kernels for Pattern Discovery and Extrapolation", ICML, 2013, and in i) the paper Snelson, Rasmussen and Ghahramani: "Warped Gaussian Processes", NIPS, 2003. There are 7 code subdirectories: cov, doc, inf, lik, mean, prior and util. cov/ contains covariance functions cov*.m => see covFunctions.m doc/ contains an index.html file providing documentation. This information is also available from http://www.GaussianProcess.org/gpml/code. Usage of mean, cov, classification and regression is demonstrated in usage*.m. Further details can be found in the developer documentation manual.pdf. inf/ contains the inference algorithms inf*.m => see infMethods.m lik/ contains the likelihood functions lik*.m => see likFunctions.m mean/ contains the mean functions mean*.m => see meanFunctions.m prior/ contains the hyperparameter prior distributions prior*.m => see priorDistributions.m util/ contains optimisation routines, backward compatibility programs and small auxiliary scripts Before running the demos or any other part of the code, you should execute startup.m to add all necessary directories to your path. In Octave, you can also use the .octaverc file. 2) ABOUT MEX FILES ================== Some of the programs make use of the MEX facility in matlab for more efficient implementation. However, if you don't know about how to compile MEX files, you do not need to worry about this -- the code runs anyway. If you do compile the MEX files, this is automatically detected, and the program will run more efficiently. If you do not have a compiler installed on your system, you might want to consult [3]. Two components of the toolbox can be accelerated by MEX. First, there is an auxiliary functions solve_chol.m having a MEX equivalent. This can be compiled by either executing util/make.m from the Matlab/Octave command line. In addition to that, we provide a Makefile for Matlab in util/ that you can run by simply typing make in your shell after having supplied your Matlab path and your operating system. We recommend the make.m script because it works both under Matlab and Octave. Second, in order to use the L-BFGS minimiser for hyperparameter optimisation as an equivalent to minimize.m, you have to compile Peter Carbonetto's "Matlab interface for L-BFGS-B" [1] which was extended to also work for Octave by José Vallet. The challenge here is the Fortran 77 code. We provide a Makefile suitable for Linux 32/64 bit and Mac whenever you have 1) g77 or 2) gfortran properly set up. The latter is prefered on current Ubuntu systems. Under Ubuntu, you can achieve this by installing the packages fort77 and gfortran, respectively. Note that the code is GNU licensed. Compilation is done by editing util/lbfgsb/Makefile. In any case, you need to provide $MATLAB_HOME which can be found by the commands 'locate matlab' or 'find / -name "matlab"'. You can choose between two compilation modes: a) without mex utility by Matlab [default] provide $MEX_SUFFIX and $MATLAB_LIB, then type 'make nomex' b) using the mex utility by Matlab provide $MEX, then type 'make mex' c) compilation for Octave provide Octave variables and type 'make oct' In Ubuntu 10.04 LTS, the libg2c library needed for both 1)+a) and 1)+b) is not included per default. If 'ls /usr/lib/libg2c.*' does not list anything this is the case on your machine. You then whant to install the packages gcc-3.4-base and libg2c0 e.g. from http://packages.ubuntu.com/hardy/gcc-3.4-base and http://packages.ubuntu.com/hardy/libg2c0. After installation, you have to create a symbolic link by 'cd /usr/lib' and 'ln -s libg2c.so.0 libg2c.so'. There were reports of problems running the mex files on recent 64bit Ubuntu since the libgfortran.3.0.0 that comes with matlab ($(MATLAB_HOME)/sys/os/glnxa64/libgfortran.so.3.0.0) is not the same as the libgfortran.3.0.0 that comes with the ubuntu package libgfortran3 and that is used when linking. So running the mex file results in a version mismatch error. The issue can be solved by changing the name of to the matlab file and linking to the ubuntu file instead: in $(MATLAB_HOME)/sys/os/glnxa64 run 'sudo mv libgfortran.3.0.0 libgfortran.3.0.0_matlab' 'sudo ln -s /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0 libgfortran.3.0.0' Thanks to Giampiero Salvi for posting the workaround. [1] https://github.com/pcarbo/lbfgsb-matlab [2] http://www.mathworks.com/support/compilers/R2010a 3) CURRENT VERSION ================== The current version of the programs is 3.5. Previous versions of the code are available at http://www.gaussianprocess.org/gpml/code/oldcode.html. 4) DIFFERENCES TO PREVIOUS VERSIONS =================================== NEW in version 3.5, 2014-12-08 ------------------------------ - mechanism for specifying hyperparameter priors (together with Roman Garnett and José Vallet) - new inference method inf/infGrid allowing efficient inference for data defined on a Cartesian grid (together with Andrew Wilson) - new mean/cov functions for preference learning: meanPref/covPref - new mean/cov functions for non-vectorial data: meanDiscrete/covDiscrete - new piecewise constant nearest neighbor mean function: meanNN - new mean functions being predictions from GPs: meanGP and meanGPexact - new covariance function for standard additive noise: covEye - new covariance function for factor analysis: covSEfact - new covariance function with varying length scale : covSEvlen - make covScale more general to scaling with a function instead of a scalar - bugfix in covGabor* and covSM (due to Andrew Gordon Wilson) - bugfix in lik/likBeta.m (suggested by Dali Wei) - bugfix in solve_chol.c (due to Todd Small) - bugfix in FITC inference mode (due to Joris Mooij) where the wrong mode for post.L was chosen when using infFITC and post.L being a diagonal matrix - bugfix in infVB marginal likelihood for likLogistic with nonzero mean function (reported by James Lloyd) - removed the combination likErf/infVB as it yields a bad posterior approximation and lacks theoretical justification - Matlab and Octave compilation for L-BFGS-B v2.4 and the more recent L-BFGS-B v3.0 (contributed by José Vallet) - smaller bugfixes in gp.m (due to Joris Mooij and Ernst Kloppenburg) - bugfix in lik/likBeta.m (due to Dali Wei) - updated use of logphi in lik/likErf - bugfix in util/solve_chol.c where a typing issue occured on OS X (due to Todd Small) - bugfix due to Bjørn Sand Jensen noticing that cov_deriv_sq_dist.m was missing in the distribution - bugfix in infFITC_EP for ttau->inf (suggested by Ryan Turner) NEW in version 3.4, 2013-11-11 ------------------------------ - new likelihood function for warped GPs: likGaussWarp - new exponential likelihood function: likExp - new Weibull likelihood function: likWeibull - new Gumbel likelihood function for extremal value regression: likGumbel - new polynomial mean function: meanPoly - included derivatives w.r.t. inducing points xu in infFITC, infFITC_Laplace, infFITC_EP - allow for zero noise variance for infExact NEW in version 3.3, 2013-10-19 ------------------------------ - new variational inference algorithm based on direct KL minimisation: infKL - improved inf/infVB double loop scheme so that only very few likelihood properties are required; infVB is now internally a sequence of infLaplace runs - improved inf/infLaplace to be more generic so that optimisers other than scaled Newton can be used - improved inf/infEP so that the internal variables (mu,Sigma) now represent the current posterior approximation - added three new generalised linear model likelihoods: gamma, beta, inverse Gaussian - new covariance functions: spectral mixture covSM, covGaboriso and covGaborard - new meta covariance function to turn a stationary covariance into a periodic covariance function covPERard, covPERiso - new periodic covariance function with zero DC component and correct scaling covPeriodicNoDC New likelihood functions in lik/: - likGamma.m, likBeta.m, likInvGauss.m New inference functions in inf/: - infKL.m New covariance functions in cov/: - covSM.m, covGaboriso.m, covGaborard.m (contributed by Andrew Gordon Wilson) - covPPard.m, covMaternard.m, covLINiso.m - covCos.m, covPeriodicNoDC.m (contributed by James Robert Lloyd) - covPERiso.m, covPERard.m NEW in version 3.2, 2013-01-15 ------------------------------ - added inf/infFITC_Laplace.m and inf/infFITC_EP.m for large scale approximate inference - added inf/infMCMC.m to allow for Markov Chain Monte Carlo sampling and Annealed Importance Sampling; we currently support two samplers: Elliptical Slice Sampling (ESS) and Hybrid Monte Carlo (HMC) - improved accuracy of the digamma function lik/likT.m/dloggamma from three to ten digits precision (thanks to Edward Challis) - added gfortran support to util/lbfgsb/Makefile (thanks to Ernst Kloppenburg) - fixed a stupid bug in cov/covFITC.m (thanks to Krzysztof Chalupka) - fixed a severe stability bug in the derivatives of inf/infFITC.m (thanks to Andrew McHutchon) - changed scalars in util/solve_chol.c to long to provide 64bit compatibility (thanks to Ernst Kloppenburg) - added infLOO to allow pseudo-likelihood optimisation instead of the marginal likelihood - fixed a performance issue in util/sq_dist.m (thanks to Krzysztof Chalupka) - fixed a bug causing numerical problems in infFITC and covFITC whenever Kuu is close to singular (thanks to Joris M. Mooij) - fixed an issue in covNoise (thanks to Daniel Marthaler) - fixed implicit derivative w.r.t. likelihood in infLaplace* - added two more likelihood functions likMix.m for mixtures and likUni for classification noise - slightly changed the interface for the likelihood functions - added likPoisson for Poisson regression of count data NEW in version 3.1, 2010-10-10 ------------------------------ - following a suggestion by Ed Snelson we now support FITC regression - cov/covFITC.m and inf/infFITC.m have been added with Ed Snelson's help - the covariance interface was slightly changed to make that possible - changed parametrisation in cov/covPeriodic.m (thanks to Philipp Hennig) NEW in version 3.0, 2010-07-23 ------------------------------ A major code reorganisation effort did take place in the current release. First, classification and regression are now done by a single file gp.m which is completely generic in the likelihood. The previous regression program gpr.m corresponds to gp.m with Gaussian likelihood. Several other likelihoods for robust regresssion were added. Further, the code now supports mean functions with a similar specification mechanism as already used by the covariance functions. Previous implementations correspond to using meanZero.m. We merged the covariance functions covMatern3iso.m and covMatern5.iso into a single covMaterniso.m and added a covariance function for additive functions as well as the possibility to use only certain components of the data. Finally, we included covPPiso.m a piecewise polynomial covariance function with compact support. New likelihood functions in lik/: likGauss.m, likLaplace.m, likLogistic.m and likT.m New mean functions in mean/: meanConst.m, meanLinear.m, meanPow.m, meanProd.m, meanSum.m and meanZero.m New covariance functions in cov/: covADD.m, covMask.m, covPPiso.m The gprSRPP.m function which previously provided "Subset of Regressors" and the "Projected Process" approximation has now been removed. NEW in version 2.1, 2007-07-25 ------------------------------ covConst.m: fixed a bug which caused an error in the derivative of the log marginal likelihood for certain combinations of covariance functions and approximation methods. (Thanks to Antonio Eleuteri for reporting the problem) gauher.m: added the function "gauher.m" which was mistakenly missing from the previous release. This caused an error for certain combinations of approximation method and likelihood function. logistic.m: modified the approximation of moments calculation to use a mixture of cumulative Gaussian, rather than Gauss-Hermite quadrature, as the former turns out to be more accurate. NEW in version 2.0, 2007-06-25 ------------------------------ Some code restructuring has taken place for the classification code to make it more modular, to facilitate addition of new likelihood functions and approximations methods. Now, all classification is done using the binaryGP function, which (among other things) takes an approximation method and a likelihood function as an arguments. Thus, binaryGP replaces both binaryEPGP and binaryLapaceGP, although wrapper functions are still provided for backward compatibility. This gives added flexibility: now EP can also be used wth the logistic likelihood function (implemented using Gauss-Hermite quadrature). approxEP.m: New file, containing the Expectation Propagation approximation method, which was previously contained in binaryEPGP.m approxLA.m: New file, containing Laplaces approximation method, which was previously contained in binaryLaplace.m approximations.m: New file, help for the approximation methods. binaryEPGP.m: This file has been replaced by a wrapper (for backward compatibility) which calls the more general binaryGP function. binaryGP.m: New general function to do binary classification. binaryLaplaceGP.m: This file has been replaced by a wrapper (for backward compatibility) which calls the more general binaryGP function. covMatern3iso.m, covMatern5iso.m, covNNone.m, covRQard.m, covRQiso.m, cosSEard, covSEiso: now check more carefully, that persistent variables have the correct sizes, and some variable names have been modified. cumGauss.m: New file, containing code for the cumulative Gaussian likelihood function likelihoods.m: New file, help for likelihood functions logistic.m: New file, logistic likelihood NEW in version 1.3, 2006-09-08 ------------------------------ covRQard.m: bugfix: replaced x with x' and z with z' in line 36 covRQiso.m: bugfix: replaced x with x' and z with z' in line 28 minimize.m: correction: replaced "error()" with "error('')", and made a few cosmetic changes binaryEPGP.m: added the line "lml = -n*log(2);" in line 77. This change should be largely inconsequential, but occationally may save things when the covariance matrix is exceptionally badly conditioned. NEW in version 1.2, 2006-05-10 ------------------------------ added the "erfint" function to "binaryLaplaceGP.m". The erfint function was missing by mistake, preventing the use of the "logistic" likelihood. NEW in version 1.1, 2006-04-12 ------------------------------ added files: "covProd.m" and "covPeriodic.m" changes: "covSEiso.m" was changed slightly to avoid the use of persistent variables NEW in version 1.0, 2006-03-29 ------------------------------ initial version shipped with the book gpml-matlab-v3.5-2014-12-08/inf/000775 000765 000024 00000000000 12526350576 015652 5ustar00carlstaff000000 000000 gpml-matlab-v3.5-2014-12-08/prior/000775 000765 000024 00000000000 12526350576 016231 5ustar00carlstaff000000 000000 gpml-matlab-v3.5-2014-12-08/infMethods.m000664 000765 000024 00000005264 12526350575 017362 0ustar00carlstaff000000 000000 % Inference methods: Compute the (approximate) posterior for a Gaussian process. % Methods currently implemented include: % % infExact Exact inference (only possible with Gaussian likelihood) % infLaplace Laplace's Approximation % infEP Expectation Propagation % infVB Variational Bayes Approximation % infKL Kullback-Leibler optimal Approximation % % infFITC Large scale regression with approximate covariance matrix % infFITC_Laplace Large scale inference with approximate covariance matrix % infFITC_EP Large scale inference with approximate covariance matrix % % infMCMC Markov Chain Monte Carlo and Annealed Importance Sampling % We offer two samplers. % - hmc: Hybrid Monte Carlo % - ess: Elliptical Slice Sampling % No derivatives w.r.t. to hyperparameters are provided. % % infLOO Leave-One-Out predictive probability and Least-Squares Approxim. % infPrior Perform inference with hyperparameter prior. % % The interface to the approximation methods is the following: % % function [post nlZ dnlZ] = inf..(hyp, cov, lik, x, y) % % where: % % hyp is a struct of hyperparameters % cov is the name of the covariance function (see covFunctions.m) % lik is the name of the likelihood function (see likFunctions.m) % x is a n by D matrix of training inputs % y is a (column) vector (of size n) of targets % % nlZ is the returned value of the negative log marginal likelihood % dnlZ is a (column) vector of partial derivatives of the negative % log marginal likelihood w.r.t. each hyperparameter % post struct representation of the (approximate) posterior containing % alpha is a (sparse or full column vector) containing inv(K)*(mu-m), % where K is the prior covariance matrix, m the prior mean, % and mu the approx posterior mean % sW is a (sparse or full column) vector containing diagonal of sqrt(W) % the approximate posterior covariance matrix is inv(inv(K)+W) % L is a (sparse or full) matrix, L = chol(sW*K*sW+eye(n)) % % Usually, the approximate posterior to be returned admits the form % N(mu=m+K*alpha, V=inv(inv(K)+W)), where alpha is a vector and W is diagonal; % if not, then L contains instead -inv(K+inv(W)), and sW is unused. % % For more information on the individual approximation methods and their % implementations, see the separate inf??.m files. See also gp.m % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-05-18. % File automatically generated using noweb. gpml-matlab-v3.5-2014-12-08/meanFunctions.m000664 000765 000024 00000003727 12526350575 020075 0ustar00carlstaff000000 000000 % mean functions to be use by Gaussian process functions. There are two % different kinds of mean functions: simple and composite: % % simple mean functions: % % meanZero - zero mean function % meanOne - one mean function % meanConst - constant mean function % meanLinear - linear mean function % meanPoly - polynomial mean function % meanDiscrete - precomputed mean for discrete data % meanGP - predictive mean of another GP % meanGPexact - predictive mean of a regression GP % meanNN - nearest neighbor mean function % % composite covariance functions (see explanation at the bottom): % % meanScale - scaled version of a mean function % meanPow - power of a mean function % meanProd - products of mean functions % meanSum - sums of mean functions % meanMask - mask some dimensions of the data % meanPref - difference mean for preference learning % % Naming convention: all mean functions are named "mean/mean*.m". % % % 1) With no or only a single input argument: % % s = meanNAME or s = meanNAME(hyp) % % The mean function returns a string s telling how many hyperparameters hyp it % expects, using the convention that "D" is the dimension of the input space. % For example, calling "meanLinear" returns the string 'D'. % % 2) With two input arguments: % % m = meanNAME(hyp, x) % % The function computes and returns the mean vector where hyp are the % hyperparameters and x is an n by D matrix of cases, where D is the dimension % of the input space. The returned mean vector is of size n by 1. % % 3) With three input arguments: % % dm = meanNAME(hyp, x, i) % % The function computes and returns the n by 1 vector of partial derivatives % of the mean vector w.r.t. hyp(i) i.e. hyperparameter number i. % % See also doc/usageMean.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-05-18. % File automatically generated using noweb. gpml-matlab-v3.5-2014-12-08/lik/000775 000765 000024 00000000000 12526350576 015655 5ustar00carlstaff000000 000000 gpml-matlab-v3.5-2014-12-08/doc/000775 000765 000024 00000000000 12526350576 015643 5ustar00carlstaff000000 000000 gpml-matlab-v3.5-2014-12-08/Copyright000644 000765 000024 00000003456 12437551244 016773 0ustar00carlstaff000000 000000 GAUSSIAN PROCESS REGRESSION AND CLASSIFICATION Toolbox version 3.5 for GNU Octave 3.2.x and Matlab 7.x The code is released under the FreeBSD License. Copyright (c) 2005-2015 Carl Edward Rasmussen & Hannes Nickisch. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY CARL EDWARD RASMUSSEN & HANNES NICKISCH ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL CARL EDWARD RASMUSSEN & HANNES NICKISCH OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. The views and conclusions contained in the software and documentation are those of the authors and should not be interpreted as representing official policies, either expressed or implied, of Carl Edward Rasmussen & Hannes Nickisch. The code and associated documentation is available from http://gaussianprocess.org/gpml/code. gpml-matlab-v3.5-2014-12-08/cov/000775 000765 000024 00000000000 12526350576 015665 5ustar00carlstaff000000 000000 gpml-matlab-v3.5-2014-12-08/startup.m000644 000765 000024 00000001440 12436547121 016745 0ustar00carlstaff000000 000000 % startup script to make Octave/Matlab aware of the GPML package % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2014-11-30. disp(['executing gpml startup script...']); OCT = exist('OCTAVE_VERSION') ~= 0; % check if we run Matlab or Octave me = mfilename; % what is my filename mydir = which(me); mydir = mydir(1:end-2-numel(me)); % where am I located if OCT && numel(mydir)==2 if strcmp(mydir,'./'), mydir = [pwd,mydir(2:end)]; end end % OCTAVE 3.0.x relative, MATLAB and newer have absolute path addpath(mydir(1:end-1)) addpath([mydir,'cov']) addpath([mydir,'doc']) addpath([mydir,'inf']) addpath([mydir,'lik']) addpath([mydir,'mean']) addpath([mydir,'prior']) addpath([mydir,'util']) clear me mydirgpml-matlab-v3.5-2014-12-08/cov/covPERiso.m000644 000765 000024 00000006111 12403307417 017677 0ustar00carlstaff000000 000000 function K = covPERiso(cov, hyp, x, z, i) % Stationary periodic covariance function for an isotropic stationary covariance % function k0 such as covMaterniso, covPPiso, covRQiso and covSEiso. % Isotropic stationary means that the covariance function k0(x,z) depends on the % data points x,z only through the squared distance % dxz = (x-z)'*inv(P)*(x-z) where the P matrix is ell^2 times the unit matrix. % The covariance function is parameterized as: % % k(x,z) = k0(u(x),u(z)), u(x) = [sin(pi*x/p); cos(pi*x/p)] % % where the period p belongs to covPERiso and hyp0 belong to k0: % % hyp = [ log(p) % hyp0 ] % % The first hyperparameter of k0 is the log lengthscale hyp0(1) = log(ell). % Note that for k0 = covSEiso and D = 1, a faster alternative is covPeriodic. % % Copyright (c) by Hannes Nickisch, 2013-10-15. % % See also COVFUNCTIONS.M. nocov = false; % default case when no cov argument is provided if nargin==0, cov = {@covSEiso}; nocov = true; end % default case if isnumeric(cov) % detect old version where the cov parameter was missing % i <- z, z <- x, x <- hyp, hyp <- cov if nargin>3, i = z; end if nargin>2, z = x; end if nargin>1, x = hyp; end hyp = cov; cov = {@covSEiso}; nocov = true; end if nocov && nargin<2 || ~nocov && nargin<3 % report number of parameters K = ['(1+',feval(cov{:}),')']; return end if nocov && nargin<3 || ~nocov && nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); p = exp(hyp(1)); if nocov && nargin<4 || ~nocov && nargin<5 [x,z] = u(x,z,p,dg); % apply the embedding u:IR^D->IR^2*D K = feval(cov{:},hyp(2:end),x,z); else if i==1 if dg % compute distance d d = zeros([n,1,D]); else if xeqz % symmetric matrix Kxx d = repmat(reshape(x,n,1,D),[1,n, 1])-repmat(reshape(x,1,n, D),[n,1,1]); else % cross covariances Kxz nz = size(z,1); d = repmat(reshape(x,n,1,D),[1,nz,1])-repmat(reshape(z,1,nz,D),[n,1,1]); end end d = 2*pi*d/p; dD2_dlp = -2*sum(sin(d).*d,3); % derivative dD2/dlog(p) [x,z] = u(x,z,p,dg); % apply the embedding u:IR^D->IR^2*D if dg % compute squared distances D2 = zeros(n,1); else if xeqz, D2 = sq_dist(x'); else D2 = sq_dist(x',z'); end end % reconstruct derivative w.r.t. D2 from derivative w.r.t. log(ell) dK_dD2 = feval(cov{:},hyp(2:end),x,z,1)./(-2*D2); dK_dD2(D2<1e-12) = 0; K = dK_dD2.*dD2_dlp; % apply chain rule else [x,z] = u(x,z,p,dg); % apply the embedding u:IR^D->IR^2*D K = feval(cov{:},hyp(2:end),x,z,i-1); end end function [x,z] = u(x,z,p,dg) % apply the embedding u:IR^D->IR^2*D x = 2*pi*x/p; x = [sin(x), cos(x)]; if numel(z)>0 && ~dg, z = 2*pi*z/p; z = [sin(z), cos(z)]; endgpml-matlab-v3.5-2014-12-08/cov/covPref.m000664 000765 000024 00000002233 12425364001 017431 0ustar00carlstaff000000 000000 function K = covPref(cov, hyp, x, z, varargin) % covPref - covariance function for preference learning. The covariance % function corresponds to a prior on f(x1) - f(x2). % % k(x,z) = k_0(x1,z1) + k_0(x2,z2) - k_0(x1,z2) - k_0(x2,z1). % % The hyperparameters are: % % hyp = [ hyp_k0 ] % % For more help on design of covariance functions, try "help covFunctions". % % See Collaborative Gaussian Processes for Preference Learning, NIPS 2014. % % Copyright (c) by Hannes Nickisch and Roman Garnett, 2014-11-01. % % See also COVFUNCTIONS.M. if nargin<3, K = strrep(feval(cov{:}),'D','D/2'); return; end % no of params x1 = x(:,1:end/2); x2 = x(:,1+end/2:end); if nargin<4 || isempty(z), z = x; end if strcmp(z,'diag') n = size(x,1); diag12 = zeros(n,1); for i=1:n, diag12(i) = feval(cov{:},hyp,x1(i,:),x2(i,:),varargin{:}); end K = feval(cov{:},hyp,x1,z,varargin{:}) ... + feval(cov{:},hyp,x2,z,varargin{:}) - 2*diag12; else z1 = z(:,1:end/2); z2 = z(:,1+end/2:end); K = feval(cov{:},hyp,x1,z1,varargin{:}) ... + feval(cov{:},hyp,x2,z2,varargin{:}) ... - feval(cov{:},hyp,x1,z2,varargin{:}) ... - feval(cov{:},hyp,x2,z1,varargin{:}); end gpml-matlab-v3.5-2014-12-08/cov/covLIN.m000644 000765 000024 00000002036 12403307704 017161 0ustar00carlstaff000000 000000 function K = covLIN(hyp, x, z, i) % Linear covariance function. The covariance function is parameterized as: % % k(x^p,x^q) = x^p'*x^q % % The are no hyperparameters: % % hyp = [ ] % % Note that there is no bias or scale term; use covConst to add these. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. if nargin<2, K = '0'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode % compute inner products if dg % vector kxx K = sum(x.*x,2); else if xeqz % symmetric matrix Kxx K = x*x'; else % cross covariances Kxz K = x*z'; end end if nargin>3 % derivatives error('Unknown hyperparameter') endgpml-matlab-v3.5-2014-12-08/cov/covLINard.m000644 000765 000024 00000003011 12403307674 017650 0ustar00carlstaff000000 000000 function K = covLINard(hyp, x, z, i) % Linear covariance function with Automatic Relevance Determination (ARD). The % covariance function is parameterized as: % % k(x^p,x^q) = x^p'*inv(P)*x^q % % where the P matrix is diagonal with ARD parameters ell_1^2,...,ell_D^2, where % D is the dimension of the input space. The hyperparameters are: % % hyp = [ log(ell_1) % log(ell_2) % .. % log(ell_D) ] % % Note that there is no bias term; use covConst to add a bias. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. if nargin<2, K = 'D'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode ell = exp(hyp); [n,D] = size(x); x = x*diag(1./ell); % precompute inner products if dg % vector kxx K = sum(x.*x,2); else if xeqz % symmetric matrix Kxx K = x*x'; else % cross covariances Kxz z = z*diag(1./ell); K = x*z'; end end if nargin>3 % derivatives if i<=D if dg K = -2*x(:,i).*x(:,i); else if xeqz K = -2*x(:,i)*x(:,i)'; else K = -2*x(:,i)*z(:,i)'; end end else error('Unknown hyperparameter') end endgpml-matlab-v3.5-2014-12-08/cov/covPoly.m000644 000765 000024 00000002717 12403307402 017463 0ustar00carlstaff000000 000000 function K = covPoly(d, hyp, x, z, i) % Polynomial covariance function. The covariance function is parameterized as: % % k(x^p,x^q) = sf^2 * ( c + (x^p)'*(x^q) )^d % % The hyperparameters are: % % hyp = [ log(c) % log(sf) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. if nargin<3, K = '2'; return; end % report number of parameters if nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode c = exp(hyp(1)); % inhomogeneous offset sf2 = exp(2*hyp(2)); % signal variance if d~=max(1,fix(d)), error('only nonzero integers allowed for d'), end % degree % precompute inner products if dg % vector kxx K = sum(x.*x,2); else if xeqz % symmetric matrix Kxx K = x*x'; else % cross covariances Kxz K = x*z'; end end if nargin<5 % covariances K = sf2*( c + K ).^d; else % derivatives if i==1 K = c*d*sf2*( c + K ).^(d-1); elseif i==2 K = 2*sf2*( c + K ).^d; else error('Unknown hyperparameter') end endgpml-matlab-v3.5-2014-12-08/cov/covLINone.m000644 000765 000024 00000002511 12403307655 017666 0ustar00carlstaff000000 000000 function K = covLINone(hyp, x, z, i) % Linear covariance function with a single hyperparameter. The covariance % function is parameterized as: % % k(x^p,x^q) = (x^p'*x^q + 1)/t^2; % % where the P matrix is t2 times the unit matrix. The second term plays the % role of the bias. The hyperparameter is: % % hyp = [ log(t) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. if nargin<2, K = '1'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode it2 = exp(-2*hyp); % t2 inverse % precompute inner products if dg % vector kxx K = sum(x.*x,2); else if xeqz % symmetric matrix Kxx K = x*x'; else % cross covariances Kxz K = x*z'; end end if nargin<4 % covariances K = it2*(1+K); else % derivatives if i==1 K = -2*it2*(1+K); else error('Unknown hyperparameter') end end gpml-matlab-v3.5-2014-12-08/cov/covLINiso.m000644 000765 000024 00000002524 12403307664 017703 0ustar00carlstaff000000 000000 function K = covLINiso(hyp, x, z, i) % Linear covariance function with Automatic Relevance Determination (ARD). The % covariance function is parameterized as: % % k(x^p,x^q) = x^p'*inv(P)*x^q % % where the P matrix is ell^2 times the unit matrix. The hyperparameters are: % % hyp = [ log(ell) ] % % Note that there is no bias term; use covConst to add a bias. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-10-13. % % See also COVFUNCTIONS.M. if nargin<2, K = '1'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode ell = exp(hyp(1)); x = x/ell; % precompute inner products if dg % vector kxx K = sum(x.*x,2); else if xeqz % symmetric matrix Kxx K = x*x'; else % cross covariances Kxz z = z/ell; K = x*z'; end end if nargin>3 % derivatives if i==1 if dg K = -2*sum(x.*x,2); else if xeqz K = -2*x*x'; else K = -2*x*z'; end end else error('Unknown hyperparameter') end endgpml-matlab-v3.5-2014-12-08/cov/covPERard.m000644 000765 000024 00000007004 12403307517 017656 0ustar00carlstaff000000 000000 function K = covPERard(cov, hyp, x, z, i) % Stationary periodic covariance function for a stationary covariance function % k0 such as covMaternard, covPPard, covRQard and covSEard. % Stationary means that the covariance function k0(x,z) depends on the % data points x,z only through the squared distance % dxz = (x-z)'*inv(P)*(x-z) where the P matrix is diagonal with ARD parameters % ell_1^2,...,ell_D^2, where D is the dimension of the input space. % The covariance function is parameterized as: % % k(x,z) = k0(u(x),u(z)), u(x) = [sin(pi*x/p); cos(pi*x/p)] % % where the period p belongs to covPERiso and hyp0 belong to k0: % % hyp = [ log(p_1) % log(p_2) % . % log(p_D) % hyp0 ] % % The first D hyperparameters of k0 are the log lengthscales such that % hyp0(i) = log(ell(i)) for i=1..D. % Note that for k0 = covSEard and D = 1, a faster alternative is covPeriodic. % % Copyright (c) by Hannes Nickisch, 2013-10-16. % % See also COVFUNCTIONS.M. nocov = false; % default case when no cov argument is provided if nargin==0, cov = {@covSEard}; nocov = true; end % default case if isnumeric(cov) % detect old version where the cov parameter was missing % i <- z, z <- x, x <- hyp, hyp <- cov if nargin>3, i = z; end if nargin>2, z = x; end if nargin>1, x = hyp; end hyp = cov; cov = {@covSEard}; nocov = true; end if nocov && nargin<2 || ~nocov && nargin<3 % report number of parameters K = ['(D+',feval(cov{:}),')']; return end if nocov && nargin<3 || ~nocov && nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); p = exp(hyp(1:D)); lell = hyp(D+(1:D)); hyp0 = [lell; lell; hyp(2*D+1:end)]; if nocov && nargin<4 || ~nocov && nargin<5 [x,z] = u(x,z,p,dg); % apply the embedding u:IR^D->IR^2*D K = feval(cov{:},hyp0,x,z); else if i<=D if dg % compute distance d di = zeros([n,1]); else if xeqz % symmetric matrix Kxx di = repmat(reshape(x(:,i),n, 1),[1, n])... -repmat(reshape(x(:,i),1, n),[n, 1]); else % cross covariances Kxz nz = size(z,1); di = repmat(reshape(x(:,i),n, 1),[1,nz])... -repmat(reshape(z(:,i),1,nz),[n, 1]); end end di = 2*pi*di/p(i); dD2_dlpi = -2*sin(di).*di; % derivative dD2i/dlog(p(i)) [x,z] = u(x,z,p,dg); % apply the embedding u:IR^D->IR^2*D if dg % compute squared distances D2 = zeros(n,1); else if xeqz D2 = sq_dist(x(:,[i,i+D])'); else D2 = sq_dist(x(:,[i,i+D])',z(:,[i,i+D])'); end end % reconstruct derivative w.r.t. D2i from derivative w.r.t. log(ell(i)) dK_dD2 = feval(cov{:},hyp0,x,z,i) + feval(cov{:},hyp0,x,z,i+D); dK_dD2 = dK_dD2./(-2*D2); dK_dD2(D2<1e-12) = 0; K = dK_dD2.*dD2_dlpi; % apply chain rule else [x,z] = u(x,z,p,dg); % apply the embedding u:IR^D->IR^2*D if i<=2*D K = feval(cov{:},hyp0,x,z,i-D) + feval(cov{:},hyp0,x,z,i); else K = feval(cov{:},hyp0,x,z,i); end end end function [x,z] = u(x,z,p,dg) % apply the embedding u:IR^D->IR^2*D x = 2*pi*x*diag(1./p); x = [sin(x), cos(x)]; if numel(z)>0 && ~dg, z = 2*pi*z*diag(1./p); z = [sin(z), cos(z)]; endgpml-matlab-v3.5-2014-12-08/cov/covSEisoU.m000644 000765 000024 00000002760 12403307143 017707 0ustar00carlstaff000000 000000 function K = covSEisoU(hyp, x, z, i) % Squared Exponential covariance function with isotropic distance measure with % unit magnitude. The covariance function is parameterized as: % % k(x^p,x^q) = exp(-(x^p - x^q)'*inv(P)*(x^p - x^q)/2) % % where the P matrix is ell^2 times the unit matrix and sf2 is the signal % variance. The hyperparameters are: % % hyp = [ log(ell) ] % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. if nargin<2, K = '1'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode ell = exp(hyp(1)); % characteristic length scale % precompute squared distances if dg % vector kxx K = zeros(size(x,1),1); else if xeqz % symmetric matrix Kxx K = sq_dist(x'/ell); else % cross covariances Kxz K = sq_dist(x'/ell,z'/ell); end end if nargin<4 % covariances K = exp(-K/2); else % derivatives if i==1 K = exp(-K/2).*K; else error('Unknown hyperparameter') end endgpml-matlab-v3.5-2014-12-08/cov/covConst.m000644 000765 000024 00000002172 12403307122 017620 0ustar00carlstaff000000 000000 function K = covConst(hyp, x, z, i) % covariance function for a constant function. The covariance function is % parameterized as: % % k(x^p,x^q) = s2; % % The scalar hyperparameter is: % % hyp = [ log(sqrt(s2)) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. if nargin<2, K = '1'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode s2 = exp(2*hyp); % s2 n = size(x,1); if dg % vector kxx K = s2*ones(n,1); else if xeqz % symmetric matrix Kxx K = s2*ones(n); else % cross covariances Kxz K = s2*ones(n,size(z,1)); end end if nargin>3 % derivatives if i==1 K = 2*K; else error('Unknown hyperparameter') end endgpml-matlab-v3.5-2014-12-08/cov/covADD.m000644 000765 000024 00000007060 12410610223 017120 0ustar00carlstaff000000 000000 function K = covADD(cov, hyp, x, z, i) % Additive covariance function using a 1d base covariance function % cov(x^p,x^q;hyp) with individual hyperparameters hyp. % % k(x^p,x^q) = \sum_{r \in R} sf_r \sum_{|I|=r} % \prod_{i \in I} cov(x^p_i,x^q_i;hyp_i) % % hyp = [ hyp_1 % hyp_2 % ... % hyp_D % log(sf_R(1)) % ... % log(sf_R(end)) ] % % where hyp_d are the parameters of the 1d covariance function which are shared % over the different values of R(1) to R(end). % % Please see the paper Additive Gaussian Processes by Duvenaud, Nickisch and % Rasmussen, NIPS, 2011 for details. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. R = cov{1}; nh = eval(feval(cov{2})); % number of hypers per individual covariance nr = numel(R); % number of different degrees of interaction if nargin<3 % report number of hyper parameters K = ['D*', int2str(nh), '+', int2str(nr)]; return end if nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); % dimensionality sf2 = exp( 2*hyp(D*nh+(1:nr)) ); % signal variances of individual degrees Kd = Kdim(cov{2},hyp,x,z); % evaluate dimensionwise covariances K if nargin<5 % covariances EE = elsympol(Kd,max(R)); % Rth elementary symmetric polynomials K = 0; for ii=1:nr, K = K + sf2(ii)*EE(:,:,R(ii)+1); end % sf2 weighted sum else % derivatives if i <= D*nh % individual covariance function parameters j = fix(1+(i-1)/nh); % j is the dimension of the hyperparameter if dg, zj='diag'; else if xeqz, zj=[]; else zj=z(:,j); end, end dKj = feval(cov{2},hyp(nh*(j-1)+(1:nh)),x(:,j),zj,i-(j-1)*nh); % other dK=0 % the final derivative is a sum of multilinear terms, so if only one term % depends on the hyperparameter under consideration, we can factorise it % out and compute the sum with one degree less E = elsympol(Kd(:,:,[1:j-1,j+1:D]),max(R)-1); % R-1th elementary sym polyn K = 0; for ii=1:nr, K = K + sf2(ii)*E(:,:,R(ii)); end % sf2 weighted sum K = dKj.*K; elseif i <= D*nh+nr EE = elsympol(Kd,max(R)); % Rth elementary symmetric polynomials j = i-D*nh; K = 2*sf2(j)*EE(:,:,R(j)+1); % rest of the sf2 weighted sum else error('Unknown hyperparameter') end end % evaluate dimensionwise covariances K function K = Kdim(cov,hyp,x,z) [n,D] = size(x); % dimensionality nh = eval(feval(cov)); % number of hypers per individual covariance if nargin<4, z = []; end % make sure, z exists xeqz = numel(z)==0; dg = strcmp(z,'diag') && numel(z)>0; % determine mode if dg % allocate memory K = zeros(n,1,D); else if xeqz, K = zeros(n,n,D); else K = zeros(n,size(z,1),D); end end for d=1:D hyp_d = hyp(nh*(d-1)+(1:nh)); % hyperparamter of dimension d if dg K(:,:,d) = feval(cov,hyp_d,x(:,d),'diag'); else if xeqz K(:,:,d) = feval(cov,hyp_d,x(:,d)); else K(:,:,d) = feval(cov,hyp_d,x(:,d),z(:,d)); end end endgpml-matlab-v3.5-2014-12-08/cov/covGaboriso.m000644 000765 000024 00000004431 12411325370 020303 0ustar00carlstaff000000 000000 function K = covGaboriso(hyp, x, z, i) % Gabor covariance function with length scale ell and period p. The % covariance function is parameterized as: % % k(x,z) = h(x-z) with h(t) = exp(-t'*t/(2*ell^2))*cos(2*pi*sum(t)/p). % % The hyperparameters are: % % hyp = [ log(ell) % log(p) ] % % Note that covSM implements a weighted sum of Gabor covariance functions, but % using an alternative (spectral) parameterization. % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Hannes Nickisch, 2014-09-26. % % See also COVFUNCTIONS.M, COVGABORARD.M, COVSM.M. if nargin<2, K = '2'; return; end % report no of params if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); % dimensionality ell = exp(hyp(1)); % length scale p = exp(hyp(2)); % period if dg % compute squared distance d2 d2 = zeros(n,1); else if xeqz % symmetric matrix Kxx d2 = sq_dist(x'/ell); else % cross covariances Kxz d2 = sq_dist(x'/ell,z'/ell); end end dp = zeros(size(d2)); % init sum(t)/p computation if ~dg if xeqz % symmetric matrix Kxx for d=1:D, dp = dp + (x(:,d)*ones(1,size(x,1))-ones(n,1)*x(:,d)')/p; end else % cross covariances Kxz for d=1:D, dp = dp + (x(:,d)*ones(1,size(z,1))-ones(n,1)*z(:,d)')/p; end end end K = exp(-d2/2); if nargin<4 % covariances K = cos(2*pi*dp).*K; else % derivatives if i==1 % length scale K = d2.*cos(2*pi*dp).*K; elseif i==2 % period K = 2*pi*dp.*sin(2*pi*dp).*K; else error('Unknown hyperparameter') end endgpml-matlab-v3.5-2014-12-08/cov/covGaborard.m000644 000765 000024 00000005473 12411325365 020272 0ustar00carlstaff000000 000000 function K = covGaborard(hyp, x, z, i) % Gabor covariance function with length scales ell and periods p. The covariance % function is parameterized as: % % k(x,z) = h(x-z), h(t) = exp(-sum(t.^2./(2*ell.^2)))*cos(2*pi*sum(t./p)). % % The hyperparameters are: % % hyp = [ log(ell_1) % log(ell_2) % .. % log(ell_D) % log(p_1) % log(p_2) % .. % log(p_D) ] % % For more help on design of covariance functions, try "help covFunctions". % % Note that covSM implements a weighted sum of Gabor covariance functions, but % using an alternative (spectral) parameterization. % % Copyright (c) by Hannes Nickisch, 2014-09-26. % % See also COVFUNCTIONS.M, COVGABORISO, COVSM.M. if nargin<2, K = '2*D'; return; end % report no of params if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); % dimensionality ell = exp(hyp(1:D)); % length scale p = exp(hyp(D+1:2*D)); % period if dg % compute squared distance d2 d2 = zeros(size(x,1),1); else if xeqz % symmetric matrix Kxx d2 = sq_dist(diag(1./ell)*x'); else % cross covariances Kxz d2 = sq_dist(diag(1./ell)*x',diag(1./ell)*z'); end end dp = zeros(size(d2)); % init sum(t)/p computation if ~dg if xeqz % symmetric matrix Kxx for d=1:D, dp = dp + (x(:,d)*ones(1,size(x,1))-ones(n,1)*x(:,d)')/p(d); end else % cross covariances Kxz for d=1:D, dp = dp + (x(:,d)*ones(1,size(z,1))-ones(n,1)*z(:,d)')/p(d); end end end K = exp(-d2/2); if nargin<4 % covariance K = cos(2*pi*dp).*K; else % derivatives d = mod(i-1,D)+1; if dg, K = K*0; end % handle diagonal case if xeqz % symmetric matrix Kxx dd = x(:,d)*ones(1,size(x,1))-ones(n,1)*x(:,d)'; else % cross covariances Kxz dd = x(:,d)*ones(1,size(z,1))-ones(n,1)*z(:,d)'; end if i<=D % length scale K = (dd/ell(d)).^2.*cos(2*pi*dp).*K; elseif i<=2*D % period K = 2*pi*dd/p(d).*sin(2*pi*dp).*K; else error('Unknown hyperparameter') end endgpml-matlab-v3.5-2014-12-08/cov/covSM.m000644 000765 000024 00000011263 12410631060 017051 0ustar00carlstaff000000 000000 function K = covSM(Q, hyp, x, z, i) % Gaussian Spectral Mixture covariance function. The covariance function % parametrization depends on the sign of Q. % % Let t(Dx1) be an offset vector in dataspace e.g. t = x_i - z_j. Then w(DxP) % are the weights and m(Dx|Q|) = 1/p, v(Dx|Q|) = (2*pi*ell)^-2 are spectral % means (frequencies) and variances, where p is the period and ell the length % scale of the Gabor function h(t2v,tm) given by the expression % h(t2v,tm) = exp(-2*pi^2*t2v).*cos(2*pi*tm) % % Then, the two covariances are obtained as follows: % % SM, spectral mixture: Q>0 => P = 1 % k(x_i,z_j) = w'*h(v'*(t.*t),m'*t) % % SMP, spectral mixture product: Q<0 => P = D % k(x_i,z_j) = prod(w'*h(T*T*v,T*m)), T = diag(t) % % The hyperparameters are: % % hyp = [ log(w(:)) % log(m(:)) % log(sqrt(v(:))) ] % % For more help on design of covariance functions, try "help covFunctions". % % Note that the spectral density H(s) = F[ h(t) ] of covGaboriso is given by % H(s) = N(s|m,v)/2 + N(s|-m,v)/2 where m=1/p is the mean and v=(2*pi*ell)^-2 % is the variance of a symmetric Gaussian mixture. Hence the covGaboriso % covariance forms a basis for the class of stationary covariances since a % weighted sum of covGaboriso covariances corresponds to an isotropic % location-scale mixture of a symmetric Gaussian mixture in the spectral domain. % % Internally, covSM constructs a weighted sum of products of 1d covGaboriso % covariances using covMask, covProd, covScale and covSum. % % For more details, see % 1) Gaussian Process Kernels for Pattern Discovery and Extrapolation, % ICML, 2013, by Andrew Gordon Wilson and Ryan Prescott Adams. % 2) GPatt: Fast Multidimensional Pattern Extrapolation with Gaussian % Processes, arXiv 1310.5288, 2013, by Andrew Gordon Wilson, Elad Gilboa, % Arye Nehorai and John P. Cunningham, and % http://mlg.eng.cam.ac.uk/andrew/pattern % % For Q>0, covSM corresponds to Eq. 12 in Ref (1) % For Q<0, covSM corresponds to Eq. 14 in Ref (2) (but w here = w^2 in (14)) % % Copyright (c) by Andrew Gordon Wilson and Hannes Nickisch, 2014-09-24. % % See also COVFUNCTIONS.M, COVGABORISO.M, COVGABORARD.M. smp = Q<0; Q = abs(Q); % switch between covSM and covSMP mode if nargin<3 % report no of parameters if smp, K = '3*D*'; else K = '(1+2*D)*'; end, K = [K,sprintf('%d',Q)]; return end if nargin<4, z = []; end % make sure, z exists D = size(x,2); P = smp*D+(1-smp); % dimensionality, P=D or P=1 lw = reshape(hyp( 1:P*Q) ,P,Q); % log mixture weights lm = reshape(hyp(P*Q+ (1:D*Q)),D,Q); % log spectral means ls = reshape(hyp(P*Q+D*Q+(1:D*Q)),D,Q); % log spectral standard deviations % In the following, we construct nested cell arrays to finally obtain either if smp % 1) the product of weighted sums of 1d covGabor covariance functions or fac = cell(1,D); for d=1:D add = cell(1,Q); % a) addends for weighted sum of univariate Gabor functions for q=1:Q, add{q} = {'covScale',{'covMask',{d,{'covGaboriso'}}}}; end fac{d} = {'covSum',add}; % b) combine addends into sum end cov = {'covProd',fac}; % c) combine factors into product else % 2) the weighted sum of multivariate covGaborard covariance functions. % weighted sum of multivariate Gabor functions add = cell(1,Q); for q=1:Q, add{q} = {'covScale',{'covGaborard'}}; end cov = {'covSum',add}; % combine into sum end if smp % assemble hyp; covGabor is parametrised using -ls-log(2*pi) and -lm hyp = [lw(:)'/2; -ls(:)'-log(2*pi); -lm(:)']; else hyp = [lw/2; -ls-log(2*pi); -lm ]; end if nargin<5 % evaluation of the covariance K = feval(cov{:},hyp(:),x,z); else % We compute the indices j in the new hyperparameter vector hyp. The % correction constants c are needed because some hyperparameters in hyp are % powers of hyperparameters accepted by covSM. if i<=P*Q % derivatives w.r.t. w c = 0.5; if smp, j = 1+3*(i-1); else j = 1+(i-1)*(2*D+1); end elseif i<=(P+ D)*Q % derivatives w.r.t. m c = -1.0; j = i- P *Q; [j1,j2] = ind2sub([D,Q],j); if smp, j = 3+3*(j-1); else j = 1+j1+D+(j2-1)*(2*D+1); end elseif i<=(P+2*D)*Q % derivatives w.r.t. v c = -1.0; j = i-(P+D)*Q; [j1,j2] = ind2sub([D,Q],j); if smp, j = 2+3*(j-1); else j = 1+j1+ (j2-1)*(2*D+1); end else error('Unknown hyperparameter') end K = c*feval(cov{:},hyp(:),x,z,j); endgpml-matlab-v3.5-2014-12-08/cov/covFITC.m000644 000765 000024 00000003341 12403307740 017264 0ustar00carlstaff000000 000000 function [K,Kuu,Ku] = covFITC(cov, xu, hyp, x, z, i) % Covariance function to be used together with the FITC approximation. % % The function allows for more than one output argument and does not respect the % interface of a proper covariance function. In fact, it wraps a proper % covariance function such that it can be used together with infFITC.m. % Instead of outputing the full covariance, it returns cross-covariances between % the inputs x, z and the inducing inputs xu as needed by infFITC.m % % Copyright (c) by Ed Snelson, Carl Edward Rasmussen % and Hannes Nickisch, 2011-11-02. % % See also COVFUNCTIONS.M, INFFITC.M. if nargin<4, K = feval(cov{:}); return, end if nargin<5, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode if size(xu,2) ~= size(x,2) error('Dimensionality of inducing inputs must match training inputs'); end if nargin<6 % covariances if dg K = feval(cov{:},hyp,x,'diag'); else if xeqz K = feval(cov{:},hyp,x,'diag'); if nargout>1, Kuu = feval(cov{:},hyp,xu); end if nargout>2, Ku = feval(cov{:},hyp,xu,x); end else K = feval(cov{:},hyp,xu,z); end end else % derivatives if dg K = feval(cov{:},hyp,x,'diag',i); else if xeqz K = feval(cov{:},hyp,x,'diag',i); if nargout>1, Kuu = feval(cov{:},hyp,xu,[],i); end if nargout>2, Ku = feval(cov{:},hyp,xu,x,i); end else K = feval(cov{:},hyp,xu,z,i); end end endgpml-matlab-v3.5-2014-12-08/cov/covPeriodic.m000644 000765 000024 00000003204 12403307504 020271 0ustar00carlstaff000000 000000 function K = covPeriodic(hyp, x, z, i) % Stationary covariance function for a smooth periodic function, with period p % in 1d (see covPERiso and covPERard for multivariate data): % % k(x,z) = sf^2 * exp( -2*sin^2( pi*||x-z||/p )/ell^2 ) % % where the hyperparameters are: % % hyp = [ log(ell) % log(p) % log(sf) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2011-01-05. % % See also COVFUNCTIONS.M. if nargin<2, K = '3'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); if D>1, error('Covariance is defined for 1d data only.'), end ell = exp(hyp(1)); p = exp(hyp(2)); sf2 = exp(2*hyp(3)); % precompute distances if dg % vector kxx K = zeros(size(x,1),1); else if xeqz % symmetric matrix Kxx K = sqrt(sq_dist(x')); else % cross covariances Kxz K = sqrt(sq_dist(x',z')); end end K = pi*K/p; if nargin<4 % covariances K = sin(K)/ell; K = K.*K; K = sf2*exp(-2*K); else % derivatives if i==1 K = sin(K)/ell; K = K.*K; K = 4*sf2*exp(-2*K).*K; elseif i==2 R = sin(K)/ell; K = 4*sf2/ell*exp(-2*R.*R).*R.*cos(K).*K; elseif i==3 K = sin(K)/ell; K = K.*K; K = 2*sf2*exp(-2*K); else error('Unknown hyperparameter') end endgpml-matlab-v3.5-2014-12-08/cov/covNNone.m000644 000765 000024 00000004016 12403307552 017555 0ustar00carlstaff000000 000000 function K = covNNone(hyp, x, z, i) % Neural network covariance function with a single parameter for the distance % measure. The covariance function is parameterized as: % % k(x^p,x^q) = sf2 * asin(x^p'*P*x^q / sqrt[(1+x^p'*P*x^p)*(1+x^q'*P*x^q)]) % % where the x^p and x^q vectors on the right hand side have an added extra bias % entry with unit value. P is ell^-2 times the unit matrix and sf2 controls the % signal variance. The hyperparameters are: % % hyp = [ log(ell) % log(sqrt(sf2) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. if nargin<2, K = '2'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode n = size(x,1); ell2 = exp(2*hyp(1)); sf2 = exp(2*hyp(2)); sx = 1 + sum(x.*x,2); if dg % vector kxx K = sx./(sx+ell2); else if xeqz % symmetric matrix Kxx S = 1 + x*x'; K = S./(sqrt(ell2+sx)*sqrt(ell2+sx)'); else % cross covariances Kxz S = 1 + x*z'; sz = 1 + sum(z.*z,2); K = S./(sqrt(ell2+sx)*sqrt(ell2+sz)'); end end if nargin<4 % covariances K = sf2*asin(K); else % derivatives if i==1 % lengthscale if dg V = K; else vx = sx./(ell2+sx); if xeqz V = repmat(vx/2,1,n) + repmat(vx'/2,n,1); else vz = sz./(ell2+sz); nz = size(z,1); V = repmat(vx/2,1,nz) + repmat(vz'/2,n,1); end end K = -2*sf2*(K-K.*V)./sqrt(1-K.*K); elseif i==2 % magnitude K = 2*sf2*asin(K); else error('Unknown hyperparameter') end endgpml-matlab-v3.5-2014-12-08/cov/covEye.m000664 000765 000024 00000002576 12403653021 017270 0ustar00carlstaff000000 000000 function K = covEye(hyp, x, z, i) % Independent covariance function, i.e. "white noise", with unit variance. % The covariance function is specified as: % % k(x^p,x^q) = \delta(p,q) % % \delta(p,q) is a Kronecker delta function which is 1 iff p=q and zero % otherwise in mode 1). % In cross covariance mode 2) two data points x_p and z_q are considered equal % if their difference norm |x_p-z_q| is less than eps, the machine precision. % The hyperparameters are: % % hyp = [ ] % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Hannes Nickisch, 2014-09-05. % % See also COVFUNCTIONS.M. tol = eps; % threshold on the norm when two vectors are considered to be equal if nargin<2, K = '0'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists dg = strcmp(z,'diag'); % determine mode n = size(x,1); % precompute raw if dg % vector kxx K = ones(n,1); else if isempty(z) % symmetric matrix Kxx K = eye(n); else % cross covariances Kxz K = double(sq_dist(x',z')1, mask = find(mask); end % convert 1/0->index D = length(mask); % masked dimension if nargin<3, K = num2str(eval(nh_string)); return, end % number of parameters if nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode if eval(nh_string)~=length(hyp) % check hyperparameters error('number of hyperparameters does not match size of masked data') end if nargin<5 % covariances if dg K = feval(cov{:}, hyp, x(:,mask), 'diag'); else if xeqz K = feval(cov{:}, hyp, x(:,mask)); else K = feval(cov{:}, hyp, x(:,mask), z(:,mask)); end end else % derivatives if i <= eval(nh_string) if dg K = feval(cov{:}, hyp, x(:,mask), 'diag', i); else if xeqz K = feval(cov{:}, hyp, x(:,mask), [], i); else K = feval(cov{:}, hyp, x(:,mask), z(:,mask), i); end end else error('Unknown hyperparameter') end endgpml-matlab-v3.5-2014-12-08/cov/covSEfact.m000664 000765 000024 00000006042 12500641125 017703 0ustar00carlstaff000000 000000 function K = covSEfact(d, hyp, x, z, i) % Factor analysis squared exponential covariance. % % The covariance function is parameterized as: % % k(x,z) = sf^2 * covSEiso(x*L',z*L'), % % where x of size (n,D) and z of size (nz,D) are the inputs and, L is an % embedding matrix of size (dxD), and sf is the signal standard deviation. % The low-rank matrix L'*L is the metric used to measure distances between % data inputs x an z; dist(x,z) = ||x-z||_{L'*L} % % The hyperparameters are: % hyp = [ hypL; % log(sf) ], % % where hypL is the vectorized L matrix. % The conversion between hypL and L is achieved by the following code: % % conversion L -> hypL % >> if d==1, diagL = L(1); else diagL = diag(L); end % >> L(1:d+1:d*d) = log(diagL); hyp = L(triu(true(d,D))); % % conversion hypL -> L % >> L = zeros(d,D); L(triu(true(d,D))) = hyp(:); % >> if d==1, diagL = L(1); else diagL = diag(L); end % >> L(1:d+1:d*d) = exp(diagL); % % hypL = [ log(L_11) % L_21 % log(L_22) % L_31 % L_32 % .. % log(L_dd) % .. % L_dD] % % Copyright (c) by Roman Garnett & Hannes Nickisch, 2014-08-15. % % See also COVFUNCTIONS.M. if nargin==0, error('d must be specified.'), end nh = sprintf('(D*%d - %d*(%d-1)/2 + 1)',d,d,d); % number of hyperparam string if nargin<3, K = nh; return; end % report number of hyperparameters if nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode if xeqz, z = x; end % make sure we have a valid z [n,D] = size(x); % dimensionality if d>D, error('We need d<=D.'), end nh = eval(nh); sf = exp(hyp(nh)); hyp = hyp(1:nh-1); % bring hypers in correct shape L = zeros(d,D); L(triu(true(d,D))) = hyp(:); % embedding matrix if d==1, diagL = L(1); else diagL = diag(L); end % properly handle limit case L(1:d+1:d*d) = exp(diagL); xl = x*L'; % transform input data if dg, zl = 'diag'; else if xeqz, zl = xl; else zl = z*L'; end end if nargin<5 % covariance, call covSEiso on the embedded points K = covSEiso([0;log(sf)],xl,zl); else if i==nh % derivative w.r.t. log signal standard deviation K = covSEiso([0;log(sf)],xl,zl,2); else td = d*(d+1)/2; % determine indices in L if i>td col = ceil((i-td)/d); col = col+ceil((sqrt(8*(i-d*col)+1)-1)/2); row = mod(i-d*(d+1)/2-1,d)+1; else col = ceil((sqrt(8*i+1)-1)/2); row = i-col*(col-1)/2; end if dg K = zeros(n,1); else K = covSEiso([0;log(sf)],xl,zl); % derivative w.r.t. log lengthscale m = size(z,1); F = x(:,col)*ones(1,m)-ones(n,1)*z(:,col)'; K = -K.*F.*(xl(:,row)*ones(1,m)-ones(n,1)*zl(:,row)'); if row==col, K = L(row,col)*K; end % diagonal is in the log domain end end end gpml-matlab-v3.5-2014-12-08/cov/covPPard.m000644 000765 000024 00000006022 12466202235 017546 0ustar00carlstaff000000 000000 function K = covPPard(v, hyp, x, z, i) % Piecewise polynomial covariance function with compact support, v = 0,1,2,3. % The covariance functions are 2v times contin. diff'ble and the corresponding % processes are hence v times mean-square diffble. The covariance function is: % % k(x^p,x^q) = s2f * max(1-r,0)^(j+v) * f(r,j) with j = floor(D/2)+v+1 % % where r is the distance sqrt((x^p-x^q)'*inv(P)*(x^p-x^q)), and the P matrix % is diagonal with ARD parameters ell_1^2,...,ell_D^2, where D is the dimension % of the input space and sf2 is the signal variance. The hyperparameters are: % % hyp = [ log(ell_1) % log(ell_2) % .. % log(ell_D) % log(sf) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-02-09. % % See also COVFUNCTIONS.M. if nargin<3, K = '(D+1)'; return; end % report number of parameters if nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); ell = exp(hyp(1:D)); sf2 = exp(2*hyp(D+1)); if all(v~=[0,1,2,3]), error('only 0,1,2 and 3 allowed for v'), end % degree j = floor(D/2)+v+1; % exponent switch v case 0, f = @(r,j) 1; df = @(r,j) 0; case 1, f = @(r,j) 1 + (j+1)*r; df = @(r,j) (j+1); case 2, f = @(r,j) 1 + (j+2)*r + ( j^2+ 4*j+ 3)/ 3*r.^2; df = @(r,j) (j+2) + 2*( j^2+ 4*j+ 3)/ 3*r; case 3, f = @(r,j) 1 + (j+3)*r + (6*j^2+36*j+45)/15*r.^2 ... + (j^3+9*j^2+23*j+15)/15*r.^3; df = @(r,j) (j+3) + 2*(6*j^2+36*j+45)/15*r ... + (j^3+9*j^2+23*j+15)/ 5*r.^2; end cs = @(r,e) (r<1).*max(1-r,0).^e; pp = @(r,j,v,f) cs(r,j+v ).* f(r,j); dpp = @(r,j,v,f) r.*cs(r,j+v-1).*( f(r,j)*(j+v) - max(1-r,0).*df(r,j) ); % precompute squared distances if dg % vector kxx K = zeros(size(x,1),1); else if xeqz % symmetric matrix Kxx K = sqrt( sq_dist(diag(1./ell)*x') ); else % cross covariances Kxz K = sqrt( sq_dist(diag(1./ell)*x',diag(1./ell)*z') ); end end if nargin<5 % covariances K = sf2*pp( K, j, v, f ); else % derivatives if i<=D % length scale parameter if dg Ki = zeros(size(x,1),1); else if xeqz Ki = sq_dist(1/ell(i)*x(:,i)'); else Ki = sq_dist(1/ell(i)*x(:,i)',1/ell(i)*z(:,i)'); end end K = sf2*dpp( K, j, v, f ).*Ki./K.^2; K(Ki<1e-12) = 0; % fix limit case elseif i==D+1 % magnitude parameter K = 2*sf2*pp( K, j, v, f ); else error('Unknown hyperparameter') end endgpml-matlab-v3.5-2014-12-08/cov/covScale.m000664 000765 000024 00000005743 12403653021 017574 0ustar00carlstaff000000 000000 function K = covScale(cov, lsf, hyp, x, z, i) % covScale - compose a covariance function as a scaled version of another % one to model functions of the form f(x) = sf(x) f0(x), where sf(x) is a % scaling function determining the function's standard deviation given f0(x) % is normalised. % % The covariance function is parameterized as: % k(x,z) = sf(x) * k_0(x,z) * sf(z) % with an important special case being % k(x,z) = sf^2 * k_0(x,z). % % You can either use K = covScale(cov, lsf, hyp, x, z, i) where the log scaling % function lsf is a GPML mean function with hyperparameters hyp_sf yielding % hyp = [ hyp_lsf % hyp_cov ] % as hyperparameters % or you can use covScale(cov, hyp, x, z, i) to perform % rescaling by a scalar value sf specified as an additional variable yielding % hyp = [ log(sf) % hyp_cov ] % as hyperparameters. % % Copyright (c) by Carl Edward Rasmussen, Hannes Nickisch & Roman Garnett % 2014-09-05. % % See also COVFUNCTIONS.M. if nargin==0, error('cov function must be specified'), end if nargin<=1, lsf = []; end, narg = nargin; % set a default value if isnumeric(lsf)&&~isempty(lsf) % shift parameters if sf contains actually hyp if nargin>4, i = z; end if nargin>3, z = x; end if nargin>2, x = hyp; end if nargin>1, hyp = lsf; end narg = nargin+1; lsf = []; end % below we us narg instead of nargin to be independent of the parameter shift if isempty(lsf), ssf = '1'; else ssf = feval(lsf{:}); end % number of hypers if narg<4, K = ['(',feval(cov{:}),'+',ssf,')']; return, end if narg<5, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); % dimension of input data nsf = eval(ssf); hyp_lsf = hyp(1:nsf); % number of params, split hyp ncov = eval(feval(cov{:})); hyp_cov = hyp(nsf+(1:ncov)); scalar = isempty(lsf); if scalar, sf = exp(hyp_lsf); end if scalar, sfx = sf; else sfx = exp(feval(lsf{:},hyp_lsf,x)); end if dg K = sfx.*sfx; else if xeqz, sfz = sfx; else if scalar, sfz = sf; else sfz = exp(feval(lsf{:},hyp_lsf,z)); end, end K = sfx*sfz'; end if narg<6 % covariances K = K.*feval(cov{:},hyp_cov,x,z); else % derivatives if i>nsf % wrt covariance function hyperparameters K = K.*feval(cov{:},hyp_cov,x,z,i-nsf); else % wrt scaling function hyperparameters K = feval(cov{:},hyp_cov,x,z); if scalar, dsfx = sfx; else dsfx = sfx.*feval(lsf{:},hyp_lsf,x,i); end if dg K = (2*sfx.*dsfx).*K; else if xeqz, dsfz = dsfx; else if scalar, dsfz = sfz; else dsfz = sfz.*feval(lsf{:},hyp_lsf,z,i); end end K = (sfx*dsfz'+dsfx*sfz').*K; end end endgpml-matlab-v3.5-2014-12-08/cov/covRQiso.m000644 000765 000024 00000003473 12403307320 017574 0ustar00carlstaff000000 000000 function K = covRQiso(hyp, x, z, i) % Rational Quadratic covariance function with isotropic distance measure. The % covariance function is parameterized as: % % k(x^p,x^q) = sf^2 * [1 + (x^p - x^q)'*inv(P)*(x^p - x^q)/(2*alpha)]^(-alpha) % % where the P matrix is ell^2 times the unit matrix, sf2 is the signal % variance and alpha is the shape parameter for the RQ covariance. The % hyperparameters are: % % hyp = [ log(ell) % log(sf) % log(alpha) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. if nargin<2, K = '3'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode ell = exp(hyp(1)); sf2 = exp(2*hyp(2)); alpha = exp(hyp(3)); % precompute squared distances if dg % vector kxx D2 = zeros(size(x,1),1); else if xeqz % symmetric matrix Kxx D2 = sq_dist(x'/ell); else % cross covariances Kxz D2 = sq_dist(x'/ell,z'/ell); end end if nargin<4 % covariances K = sf2*(1+0.5*D2/alpha).^(-alpha); else % derivatives if i==1 % length scale parameter K = sf2*(1+0.5*D2/alpha).^(-alpha-1).*D2; elseif i==2 % magnitude parameter K = 2*sf2*(1+0.5*D2/alpha).^(-alpha); elseif i==3 K = (1+0.5*D2/alpha); K = sf2*K.^(-alpha).*(0.5*D2./K - alpha*log(K)); else error('Unknown hyperparameter') end endgpml-matlab-v3.5-2014-12-08/cov/covNoise.m000644 000765 000024 00000003321 12403307542 017612 0ustar00carlstaff000000 000000 function K = covNoise(hyp, x, z, i) % Independent covariance function, i.e. "white noise", with specified variance. % The covariance function is specified as: % % k(x^p,x^q) = s2 * \delta(p,q) % % where s2 is the noise variance and \delta(p,q) is a Kronecker delta function % which is 1 iff p=q and zero otherwise in mode 1). % In cross covariance mode 2) two data points x_p and z_q are considered equal % if their difference norm |x_p-z_q| is less than eps, the machine precision. % The hyperparameter is: % % hyp = [ log(sqrt(s2)) ] % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-05-15. % % See also COVFUNCTIONS.M. tol = eps; % threshold on the norm when two vectors are considered to be equal if nargin<2, K = '1'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists dg = strcmp(z,'diag'); % determine mode n = size(x,1); s2 = exp(2*hyp); % noise variance % precompute raw if dg % vector kxx K = ones(n,1); else if numel(z)==0 % symmetric matrix Kxx K = eye(n); else % cross covariances Kxz K = double(sq_dist(x',z')1, error('Covariance is defined for 1d data only.'), end p = exp(hyp(1)); sf2 = exp(2*hyp(2)); % precompute distances if dg % vector kxx K = zeros(size(x,1),1); else if xeqz % symmetric matrix Kxx K = repmat(x,1,n) - repmat(x',n,1); else % cross covariances Kxz K = repmat(x,1,size(z,1)) - repmat(z',n,1); end end K = 2*pi*K/p; if nargin<4 % covariances K = sf2*cos(K); else % derivatives if i==1 K = sf2*sin(K).*K; elseif i==2 K = 2*sf2*cos(K); else error('Unknown hyperparameter') end endgpml-matlab-v3.5-2014-12-08/cov/covMaternard.m000664 000765 000024 00000005144 12403307634 020463 0ustar00carlstaff000000 000000 function K = covMaternard(d, hyp, x, z, i) % Matern covariance function with nu = d/2 and with Automatic Relevance % Determination (ARD) distance measure. For d=1 the function is also known as % the exponential covariance function or the Ornstein-Uhlenbeck covariance % in 1d. The covariance function is: % % k(x^p,x^q) = sf^2 * f( sqrt(d)*r ) * exp(-sqrt(d)*r) % % with f(t)=1 for d=1, f(t)=1+t for d=3 and f(t)=1+t+t²/3 for d=5. % Here r is the distance sqrt((x^p-x^q)'*inv(P)*(x^p-x^q)), where the P matrix % is diagonal with ARD parameters ell_1^2,...,ell_D^2, where D is the dimension % of the input space and sf2 is the signal variance. The hyperparameters are: % % hyp = [ log(ell_1) % log(ell_2) % .. % log(ell_D) % log(sf) ] % % Copyright (c) by Hannes Nickisch, 2013-10-13. % % See also COVFUNCTIONS.M. if nargin<3, K = '(D+1)'; return; end % report number of parameters if nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); ell = exp(hyp(1:D)); sf2 = exp(2*hyp(D+1)); if all(d~=[1,3,5]), error('only 1, 3 and 5 allowed for d'), end % degree switch d case 1, f = @(t) 1; df = @(t) 1./t; % df(t) = (f(t)-f'(t))/t case 3, f = @(t) 1 + t; df = @(t) 1; case 5, f = @(t) 1 + t.*(1+t/3); df = @(t) (1+t)/3; end m = @(t,f) f(t).*exp(-t); dm = @(t,f) df(t).*exp(-t); % precompute distances if dg % vector kxx K = zeros(size(x,1),1); else if xeqz % symmetric matrix Kxx K = sq_dist(diag(sqrt(d)./ell)*x'); else % cross covariances Kxz K = sq_dist(diag(sqrt(d)./ell)*x',diag(sqrt(d)./ell)*z'); end end if nargin<5 % covariances K = sf2*m(sqrt(K),f); else % derivatives if i<=D % length scale parameter if dg Ki = zeros(size(x,1),1); else if xeqz Ki = sq_dist(sqrt(d)/ell(i)*x(:,i)'); else Ki = sq_dist(sqrt(d)/ell(i)*x(:,i)',sqrt(d)/ell(i)*z(:,i)'); end end K = sf2*dm(sqrt(K),f).*Ki; K(Ki<1e-12) = 0; % fix limit case for d=1 elseif i==D+1 % magnitude parameter K = 2*sf2*m(sqrt(K),f); else error('Unknown hyperparameter') end endgpml-matlab-v3.5-2014-12-08/cov/covDiscrete.m000664 000765 000024 00000004541 12403307100 020274 0ustar00carlstaff000000 000000 function K = covDiscrete(s, hyp, x, z, i) % Covariance function for discrete inputs. Given a function defined on the % integers 1,2,3,..,s, the covariance function is parameterized as: % % k(x,z) = K_{xz}, % % where K is a matrix of size (s x s). % % This implementation assumes that the inputs x and z are given as integers % between 1 and s, which simply index the matrix K. % % The hyperparameters specify the upper-triangular part of the Cholesky factor % of K, where the (positive) diagonal elements are specified by their logarithm. % If L = chol(K), so K = L'*L, then the hyperparameters are: % % hyp = [ log(L_11) % L_21 % log(L_22) % L_31 % L_32 % .. % log(L_ss) ] % % The hyperparameters hyp can be generated from K using: % L = chol(K); L(1:(s+1):end) = log(diag(L)); % hyp = L(triu(true(s))); % % The covariance matrix K is obtained from the hyperparameters hyp by: % L = zeros(s); L(triu(true(s))) = hyp(:); % L(1:(s+1):end) = exp(diag(L)); K = L'*L; % % This parametrization allows unconstrained optimization of K. % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Roman Garnett, 2014-08-14. % % See also MEANDISCRETE.M, COVFUNCTIONS.M. if nargin==0, error('s must be specified.'), end % check for dimension if nargin<3, K = num2str(s*(s+1)/2); return; end % report number of parameters if nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode if xeqz, z = x; end % make sure we have a valid z L = zeros(s); L(triu(true(s))) = hyp(:); % build Cholesky factor L(1:(s+1):end) = exp(diag(L)); if nargin<5 A = L'*L; % A is a placeholder for K to avoid a name clash with the return arg if dg K = A(sub2ind(size(A),x,x)); else K = A(x,z); end else col = ceil((sqrt(8*i+1)-1)/2); row = i-col*(col-1)/2; % indices by tri-root dL = zeros(s); % derivative of Cholesky factor if (row == col) dL(row,col) = L(row,col); % diagonal entries have exp transformation else dL(row,col) = 1; end dK = dL'*L; dK = dK+dK'; % derivative of sxs covariance matrix if dg K = dK(sub2ind(size(dK),x,x)); else K = dK(x,z); end endgpml-matlab-v3.5-2014-12-08/cov/covSEiso.m000644 000765 000024 00000003166 12403307156 017567 0ustar00carlstaff000000 000000 function K = covSEiso(hyp, x, z, i) % Squared Exponential covariance function with isotropic distance measure. The % covariance function is parameterized as: % % k(x^p,x^q) = sf^2 * exp(-(x^p - x^q)'*inv(P)*(x^p - x^q)/2) % % where the P matrix is ell^2 times the unit matrix and sf^2 is the signal % variance. The hyperparameters are: % % hyp = [ log(ell) % log(sf) ] % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. if nargin<2, K = '2'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode ell = exp(hyp(1)); % characteristic length scale sf2 = exp(2*hyp(2)); % signal variance % precompute squared distances if dg % vector kxx K = zeros(size(x,1),1); else if xeqz % symmetric matrix Kxx K = sq_dist(x'/ell); else % cross covariances Kxz K = sq_dist(x'/ell,z'/ell); end end if nargin<4 % covariances K = sf2*exp(-K/2); else % derivatives if i==1 K = sf2*exp(-K/2).*K; elseif i==2 K = 2*sf2*exp(-K/2); else error('Unknown hyperparameter') end endgpml-matlab-v3.5-2014-12-08/cov/covGrid.m000664 000765 000024 00000023525 12476264572 017453 0ustar00carlstaff000000 000000 function [K,Mx,xe] = covGrid(cov, xg, hyp, x, z, i) % covGrid - Kronecker covariance function based on a grid. % % The grid g is represented by its p axes xg = {x1,x2,..xp}. An axis xi is of % size (ni,di) and the grid g has size (n1,n2,..,np,D), where D=d1+d2+..+dp. % Hence, the grid contains N=n1*n2*..*np data points. The axes do neither need % to be sorted in any way nor do they need to be 1d i.e. ni>=1. % % The covGrid function can be used to expand the cell array xg into an expanded % multivariate grid xe of size (N,D) via: % [xe,nx,Dx] = covGrid('expand',xg); => mode 1) % The operation can be reverted by: % xg = covGrid('factor',{xe,ng,Dg}); => mode 2) % % Given scattered data x of size (n,D), we can create a grid xg covering % the support of x using: % xg = covGrid('create',x,equi,k); => mode 3) % The flag equi (default value 1) can be used to enforce an equispaced % grid. The integer k, indicates the number of grid points per dimension. If % k is a real number from (0,1], then the number of grid points equals % k*numel(unique(x(:,1))). % We require at least two different components per dimension. % % The variables v={x,z} can either be a) grid indices or b) data points. % a) The variable v has size (nv,1) and contains integers from [1,N]. Then % the datapoints are obtained as g2 = reshape(g,N,D); v = g2(v,:). % b) The variable v has size (nv,D) and directly represents the data points. % At the moment, this is only supported for z. % % An arbitrary data point x -- be it an index vector of size (n,1) or a data % point of size (n,D) -- is converted into a regular data point xx of % size (n,D) by: % xx = covGrid('idx2dat',xg,x); => mode 4) % If x is already of size (n,D), xx will simply equal x. % % The resulting covariance matrix is given by: % K = kron( kron(...,K{2}), K{1} ) = K_p x .. x K_2 x K_1. % % The hyperparameters are: % hyp = [ hyp_1 % hyp_2 % .. % hyp_p ], % % Copyright (c) by Hannes Nickisch and Andrew Wilson 2015-03-05. % % See also COVFUNCTIONS.M, INFGRID.M. if nargin<2, error('Not enough parameters provided.'), end % mode 1) expand axes xg representation into full grid x if strcmp(cov,'expand') % call: [xe,nx,Dx] = covGrid('expand',xg); [K,Mx,xe] = expandgrid(xg); return % mode 2) factor full x grid into axes representation xg elseif strcmp(cov,'factor') % call: xg = covGrid('factor',{xe,ng,Dg}); K = factorgrid(xg{:}); return % mode 3) create axes representation xg from scattered data elseif strcmp(cov,'create') % call: xg = covGrid('create',x,equi,k); if nargin<3, equi = 1; else equi = hyp; end % set default values if nargin<4, k = 1; else k = x; end, x = xg; % set input params p = size(x,2); xg = cell(p,1); % allocate result if numel(k)>0, k = ones(p,1).*k(:); end % enforce vector-valued k for j=1:p % iterate over dimensions u = sort(unique(x(:,j))); if numel(u)<2, error('Two few unique points.'),end if isempty(k) % determine number of grid points if equi ngj = ceil( (u(end)-u(1))/min(abs(diff(u))) ); % use minimum spacing else ngj = numel(u); end elseif 0<=k(j) && k(j)<=1 ngj = ceil(k(j)*numel(u)); else ngj = k(j); end du = (u(end)-u(1))/ngj; bu = [u(1)-5*du, u(end)+5*du]; if equi % equispaced grid xg{j} = linspace(bu(1),bu(2),max(ngj,5))'; % at least 5 grid points else % non-equispaced grid [idx,xgj] = kmeans(u,min(numel(u),ngj-2)); xgj = sort(xgj(:))'; % cluster nb = ngj-numel(xgj); nb1 = floor(nb/2); nb2 = nb - nb1; % size of boundary xg1 = linspace(bu(1),xgj(1),nb1+1); xg2 = linspace(xgj(end),bu(2),nb2+1); xg{j} = [xg1(1:nb1), xgj, xg2(1+(1:nb2))]'; end end K = xg; return % mode 4) convert possible index vector into data space elseif strcmp(cov,'idx2dat') % call: xx = covGrid('idx2dat',xg,x); N = 1; for i=1:numel(xg), N = N*size(xg{i},1); end if isidx(hyp,N), xe = expandgrid(xg); K = xe(hyp,:); else K = hyp; end, return end % mode 0) regular covariance function computations p = numel(xg); ng = zeros(p,1); Dg = zeros(p,1); % number of Kronecker factors if numel(cov)~=p, error('We require p factors.'), end for ii = 1:p % iterate over covariance functions [ng(ii),Dg(ii)] = size(xg{ii}); f = cov(ii); if iscell(f{:}), f = f{:}; end % expand cell array if necessary D = Dg(ii); j(ii) = cellstr(num2str(eval(feval(f{:})))); % collect nbr hypers end if nargin<4 % report number of parameters K = char(j(1)); for ii=2:length(cov), K = [K, '+', char(j(ii))]; end, return end if nargin<5, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode v = []; % v vector indicates to which covariance parameters belong for ii = 1:p, v = [v repmat(ii, 1, eval(char(j(ii))))]; end if nargin==6 && i>length(v), error('Unknown hyperparameter'), end equi = true(p,1); % grid along dimension is equispaced -> Toeplitz structure for ii=1:p if ng(ii)>1 % diagnose Toeplitz structure if data is linearly increasing dev = abs(diff(xg{ii})-ones(ng(ii)-1,1)*(xg{ii}(2,:)-xg{ii}(1,:))); equi(ii) = max(dev(:))<1e-9; end end N = prod(ng); n = size(x,1); D = sum(Dg); % expanded grid and data dimension ix = isidx(x,N); % determine whether x is an index or a data array if ~ix && nargout>1 % off-grid interpolation error('No off-grid values for x allowed.') end if dg % evaluate as full dense vector for diagonal covariance case K = 1; % xg is not assumed to form a grid for z = 'diag' for ii = 1:length(cov) % iteration over factor functions f = cov(ii); if iscell(f{:}), f = f{:}; end % expand cell array if necessary d = sum(Dg(1:ii-1))+(1:Dg(ii)); % dimensions of interest if nargin<6, i = 0; vi = 0; else vi = v(i); end; % which covariance function if i<=length(v) if ix, xii = xg{ii}; else xii = x(:,d); end % switch Kronecker/plain prod if ii==vi j = sum(v(1:i)==vi); % which parameter in that covariance Kj = feval(f{:}, hyp(v==ii), xii, z, j); % deriv Kronecker factor else Kj = feval(f{:}, hyp(v==ii), xii, z); % plain Kronecker factor end if ix, K = kron(K,Kj); else K = K.*Kj; end % switch Kronecker/plain prod else error('Unknown hyperparameter') end end if ix, K = K(x); end, return end if isidx(z,N), iz = z; z = covGrid('expand',xg); z = z(iz,:); end % expand z K = cell(p,1); % covariance Kronecker factors for ii = 1:length(cov) % iteration over factor functions f = cov(ii); if iscell(f{:}), f = f{:}; end % expand cell array if necessary d = sum(Dg(1:ii-1))+(1:Dg(ii)); % dimensions of interest if isnumeric(z) && ~isempty(z) % cross terms zd = z(:,d); elseif xeqz && equi(ii) % we have Toeplitz structure zd = xg{ii}(1,:); else % symmetric matrix zd = z; end if nargin<6, i = 0; vi = 0; else vi = v(i); end; % which covariance function if i<=length(v) if ii==vi j = sum(v(1:i)==vi); % which parameter in that covariance K{ii} = feval(f{:}, hyp(v==ii), xg{ii}, zd, j); % deriv Kronecker factor else K{ii} = feval(f{:}, hyp(v==ii), xg{ii}, zd); % plain Kronecker factor end else error('Unknown hyperparameter') end if xeqz && equi(ii), K{ii} = {'toep',K{ii}}; end % make Toeplitz explicit end if ~xeqz % expand cross terms Ks = K; K = Ks{1}; for ii = 2:p, K = kron1(Ks{ii},K); end if ix, if numel(x)~=N || max(abs(x-(1:N)'))>0, K = K(x,:); end else K = Mx*K; end end if nargout>1, if ix, Mx = sparse(1:n,x,1,n,N); end, end if nargout>2, xe = covGrid('expand',xg); end % perform kron along first dimension only % the code is equivalent to the following loop % z = zeros(size(x,1)*size(y,1),size(x,2)); % for i=1:size(z,2), z(:,i) = kron(x(:,i),y(:,i)); end function z = kron1(x,y) nx = size(x,1); ny = size(y,1); z = repmat(reshape(x,1,nx,[]),[ny,1,1]).*repmat(reshape(y,ny,1,[]),[1,nx,1]); z = reshape(z,nx*ny,[]); function r = isidx(i,N) % check whether i represents an integer index vector r = false; if numel(i)>0 && ~strcmp(i,'diag') && size(i,2)==1 && ndims(i)==2 if max(abs(i-floor(i)))<1e-13 if 03 % derivatives if i<=D % length scale parameters if dg K = K*0; else if xeqz K = K.*sq_dist(x(:,i)'/ell(i)); else K = K.*sq_dist(x(:,i)'/ell(i),z(:,i)'/ell(i)); end end elseif i==D+1 % magnitude parameter K = 2*K; else error('Unknown hyperparameter') end endgpml-matlab-v3.5-2014-12-08/cov/covPeriodicNoDC.m000644 000765 000024 00000007056 12231260315 021003 0ustar00carlstaff000000 000000 function K = covPeriodicNoDC(hyp, x, z, i) % Stationary covariance function for a smooth periodic function, with period p: % % k(x,x') = sf^2 * [k0(pi*(x-x')/p) - f(ell)] / [1 - f(ell)] % with k0(t) = exp( -2*sin^2(t)/ell^2 ) and f(ell) = \int 0..pi k0(t) dt. % % The constant (DC component) has been removed and marginal variance is sf^2. % The hyperparameters are: % % hyp = [ log(ell) % log(p) % log(sf) ] % % Note that covPeriodicNoDC converges to covCos as ell goes to infinity. % % Copyright (c) by James Robert Lloyd and Hannes Nickisch 2013-10-21. % % See also COVFUNCTIONS.M, COVCOS.M. if nargin<2, K = '3'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = numel(z)==0; dg = strcmp(z,'diag') && numel(z)>0; % determine mode [n,D] = size(x); if D>1, error('Covariance is defined for 1d data only.'), end ell = exp(hyp(1)); p = exp(hyp(2)); sf2 = exp(2*hyp(3)); % precompute distances if dg % vector kxx K = zeros(size(x,1),1); else if xeqz % symmetric matrix Kxx K = sqrt(sq_dist(x')); else % cross covariances Kxz K = sqrt(sq_dist(x',z')); end end K = 2*pi*K/p; if nargin<4 % covariances K = sf2*covD(K,ell); else if i==1 if ell>1e4 % limit for ell->infty K = zeros(size(K)); % no further progress in ell possible elseif 1/ell^2<3.75 cK = cos(K); ecK = exp(cK/ell^2); b0 = besseli(0,1/ell^2); b1 = besseli(1,1/ell^2); K = 2*(exp(1/ell^2)-ecK )*b1 ... - 2*(exp(1/ell^2)-ecK.*cK)*b0 ... + 4*exp(2*(cos(K/2)/ell).^2).*sin(K/2).^2; K = sf2/(ell*(exp(1/ell^2)-b0))^2 * K; else cK = cos(K); ecK = exp((cK-1)/ell^2); b0 = embi0(1/ell^2); b1 = embi1(1/ell^2); K = 2*(1-ecK)*b1 - 2*(1-ecK.*cK)*b0 ... + 4*exp(2*(cos(K/2).^2-1)/ell^2).*sin(K/2).^2; K = sf2/(ell*(1-b0))^2 * K; end elseif i==2 if ell>1e4 % limit for ell->infty K = sf2*sin(K).*K; elseif 1/ell^2<3.75 K = exp(cos(K)/ell^2).*sin(K).*K; K = sf2/ell^2/(exp(1/ell^2)-besseli(0,1/ell^2))*K; else K = exp((cos(K)-1)/ell^2).*sin(K).*K; K = sf2/ell^2/(1-embi0(1/ell^2))*K; end elseif i==3 K = 2*sf2*covD(K,ell); else error('Unknown hyperparameter') end end function K = covD(D,ell) % evaluate covariances from distances if ell>1e4 % limit for ell->infty K = cos(D); elseif 1/ell^2<3.75 K = exp(cos(D)/ell^2); b0 = besseli(0,1/ell^2); K = (K-b0)/(exp(1/ell^2)-b0); else K = exp((cos(D)-1)/ell^2); b0 = embi0(1/ell^2); K = (K-b0)/(1-b0); end function y = embi0(x) % = exp(-x)*besseli(0,x) => 9.8.2 Abramowitz & Stegun y = 3.75/x; y = 0.39894228 + 0.01328592*y + 0.00225319*y^2 - 0.00157565*y^3 ... + 0.00916281*y^4 - 0.02057706*y^5 + 0.02635537*y^6 - 0.01647633*y^7 ... + 0.00392377*y^8; y = y/sqrt(x); function y = embi1(x) % = exp(-x)*besseli(1,x) => 9.8.4 Abramowitz & Stegun y = 3.75/x; y = 0.39894228 - 0.03988024*y - 0.00362018*y^2 + 0.00163801*y^3 ... - 0.01031555*y^4 + 0.02282967*y^5 - 0.02895312*y^6 + 0.01787654*y^7 ... - 0.00420059*y^8; y = y/sqrt(x);gpml-matlab-v3.5-2014-12-08/cov/covMaterniso.m000664 000765 000024 00000003765 12403307624 020515 0ustar00carlstaff000000 000000 function K = covMaterniso(d, hyp, x, z, i) % Matern covariance function with nu = d/2 and isotropic distance measure. For % d=1 the function is also known as the exponential covariance function or the % Ornstein-Uhlenbeck covariance in 1d. The covariance function is: % % k(x^p,x^q) = sf^2 * f( sqrt(d)*r ) * exp(-sqrt(d)*r) % % with f(t)=1 for d=1, f(t)=1+t for d=3 and f(t)=1+t+t²/3 for d=5. % Here r is the distance sqrt((x^p-x^q)'*inv(P)*(x^p-x^q)), P is ell times % the unit matrix and sf2 is the signal variance. The hyperparameters are: % % hyp = [ log(ell) % log(sf) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-09-10. % % See also COVFUNCTIONS.M. if nargin<3, K = '2'; return; end % report number of parameters if nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode ell = exp(hyp(1)); sf2 = exp(2*hyp(2)); if all(d~=[1,3,5]), error('only 1, 3 and 5 allowed for d'), end % degree switch d case 1, f = @(t) 1; df = @(t) 1; % df(t) = f(t)-f'(t) case 3, f = @(t) 1 + t; df = @(t) t; case 5, f = @(t) 1 + t.*(1+t/3); df = @(t) t.*(1+t)/3; end m = @(t,f) f(t).*exp(-t); dm = @(t,f) df(t).*exp(-t).*t; % precompute distances if dg % vector kxx K = zeros(size(x,1),1); else if xeqz % symmetric matrix Kxx K = sq_dist(sqrt(d)/ell*x'); else % cross covariances Kxz K = sq_dist(sqrt(d)/ell*x',sqrt(d)/ell*z'); end end if nargin<5 % covariances K = sf2*m(sqrt(K),f); else % derivatives if i==1 K = sf2*dm(sqrt(K),f); elseif i==2 K = 2*sf2*m(sqrt(K),f); else error('Unknown hyperparameter') end endgpml-matlab-v3.5-2014-12-08/doc/demoGrid.m000664 000765 000024 00000002424 12440633735 017551 0ustar00carlstaff000000 000000 disp('See http://www.gaussianprocess.org/gpml/code/matlab/doc/ for details.') disp('Hit any key to continue...'); pause clear all, close all write_fig = 0; xg = {linspace(-2,2,120)',linspace(-3,3,120)'}; % construct a grid [xe,ng] = covGrid('expand',xg); y = sin(xe(:,2)) + xe(:,1); x = (1:prod(ng))'; % generate training data xgs = {linspace(-4,4,100)',linspace(-6,6,110)'}; y = y + 0.01*gpml_randn(1,size(y)); % add some noise [xs,ns] = covGrid('expand',xgs); cov = {{@covSEiso},{@covSEiso}}; covg = {@covGrid,cov,xg}; % set up a GP hyp.cov = zeros(4,1); hyp.mean = []; hyp.lik = log(0.1); hyp = minimize(hyp,@gp,-50,@infGrid,[],covg,[],x,y); % optimise hyperparameters opt.cg_maxit = 200; opt.cg_tol = 5e-3; % perform inference and prediction post = infGrid(hyp,{@meanZero},covg,'likGauss',x,y,opt); post.L = @(a) a; ym = gp(hyp, @infGrid,[],covg,[],x,post,xs); cl = [min(ym(:)), max(ym(:))]; subplot(121), imagesc(xg{1}, xg{2}, reshape(y, ng'),cl), title('data y') grid on, set(gca,'xtick',-2:2), set(gca,'ytick',-3:3) subplot(122), imagesc(xgs{1},xgs{2},reshape(ym,ns'),cl), title('prediction ym') grid on, set(gca,'xtick',-4:4), set(gca,'ytick',-6:6) rectangle('Position',[-2,-3,4,6]) if write_fig, print -depsc f9.eps; endgpml-matlab-v3.5-2014-12-08/doc/f8.gif000644 000765 000024 00000034552 12075734530 016651 0ustar00carlstaff000000 000000 GIF89aA/?O_/o?O_o/?O_oo_O?/!Created with GIMP,A H*\ȰÇ#JHEI\VƎ CQɓHjDɲ%&+]ʜfL8s*YRO2 [5 FgJf5׏a;S%ŸeF(ʷkXy( 蛃X(p艇"bpad "8h$:fbIF(@FP9G[W;WoWUQpz /,m%.Ye7E/ϼי87ʼ׋7]wX̠_O\޲V$E(F/)8 g!oH%{Ԙ`,ЁHEnCD߀QP` :.D4HC-‰"!:"yA)1`xD#!cg&fqP a;bQ 7)FAE/;5ưBq,d%!1I^ҍP9IKO/޻_1)b%V&m)4)a^2Ō"1'JA[ēIrdfxN5:c)LMlSKۣ/*~SCKu92xS Y8cx=<?r~$AaPF"Ԙ$C(IBE @JGwjOQ"wzS K9Nt3"l&D< [8uDJֲZRaJP@­)TiѩFª ׾|j@aDC "cGe+!4I7)+ hW(X"v_ _,U1VI4Me0JYIVL_fuF%yzpJӶ=;[ZJ4lVK nZA﷉pNj3ѸD%Z5Z]A\wͭ׶+ ih-y!v^/3,b|)@bWSmuƭaݔ`dE(NW#=3ar!TAh7{E>ib>.XJ1'0Iwq!| k #SKNOʘʛ&L^v` B'\$lE[9O?t-eI G>82FAYG?:Œ4})MK?!+"tWv^׼^FQjT:7l\I@[Z-;<8v'LmeNӰij٪iM֠;km޾v.Vqۦc=t[ӕk_;QY9o\4جxWc3 #+d`<&.7Ϸ\oIݼK؆_B9F<@H/:m~B2sjC 1@Q`peN/q OX`V uEԽlk=Y)~vZp;wgc{;YTc?{8|)v.|!$a}~MXzYwʦ@)kE_r&=#[XϾ{yԿ~p#KQyM4&qv\]wY6~E Gys$Wgzٗ2 bD5!#h%apw"x w !@XeP GA#&4סlTI.80Ȅg'p^wU:eЃsvA8`DZ&VńX&%"gNn8l[~ס7!eqA$gQPjX2*R tXshMx\gUnA4H|tUhc_'88/1H)Xhb`R(}Ds|7{֖8 AP4,&B]ƷzzE苘Sx֘g?b菼Z+1x$Iɋ8P1/4ny @\u^؅'u<ba0R ɋ\PXR,6=QHsk!Y&֎)U&ov}YA`b9dA15PlYPK9uyw~/k)y^Ypؤ _\I{ey0d=H\K؄n(YDu)bdF5^(*ɍD9ٚcɘ03/E(O樊T7wVYPyWx@};!&隮 ؆H 'SxAz9j9swyv}Ie~&pi pr'&5V737p婎 8yYQb U"꟔.y=p`x^2[iŢZ҉":I0Dvzi_7ؐ~C`jF=&ڋ&e L N wGE"E5 A@ea:cZ'.hI٠Siv金7yn5H< YککB~.J 'a+LIyR 5juG $txJ79 ZZ5 ǶI/o]Z[|ꫥz@U)ZiZڇneE$0*JXI : 8P[&*r^H{ ե: v S~zsZu|e &IY4[6{8[j;1Mep)z)I5_`09;TJ2>vXV_ zI)cu8KzO2T7k%-^iGzQWPEHsvEVsPTp۸* /O|0'BC|U=[=gҴjů<붎S+`rd6UT.u5gJ՗7!0,P{țƻg0{ [:(}%B1wv[5[HPq J{˼(ZzQ;_rTEY{}Yq)s[ȫ%*?h+V]s«qָP:7>g*~Fa"о[ 1)"UOLv*Eu_;;[q̪ \ѫ-0P5:E~5~Uuܖ=\^`B0Dfr6AÈ`s)s8Hr Q'a~<;,(@YY47Xi,A>RqUElGxG8™7ʌ6) ۶ʼL\L gY9V  }Yܼ<V!u#'O[2)[Hhm rC]MDӦ#*.Bs:,=]?-fdMRR:c7".,g:6D0!hZ] `zlUf$ݏ4 IHK3,;85YdY ԯ \Y1ɲ=u`:X Вݣ XUD8RAقp MBqt^b3]E;yjg__IE{H;88vQmJ-o]ߍԷTg$w7/C<5esR0n]SU#}CZ٩]+2[ST&dnBq\70$Tt^ f@?R皉(㮴0>څOB?FB/}D}J} @z|zQz\ Znި$ge_Q/a6А[@L=~Mud~x6Vn&'JIsr @.FnC%`e5遌'|ND!JP)'P=됌[P|enTDF{tc&^Hsz>N?Mʉۃ6סMmE+C>7!ol>>xqȝpݢn۸\\Y[@0> <#(+$vx@B?D~UΘ(Fl'';IG{imwiޕ18 Ei0#"-%Me-5atcq.^ݣ?..U Or8\nQ5_a p9/߼g?@Iΰ~EWsb UjB6r\;ah죌L?Ugq?~-σ3}Plu׮~ c'>_ۀ@(hH`8Ȩ)9IYiyYYyZ"êC)"++Hi(2jH*l,`L-h .>N޽Xx~ؾ/?H?=m/<,H+zA%… 4W Ä,2rG<mV.eJܱt$ȥL4k^bI6~7)4 P"[45)U'Vhg$%qqHy:ᴄRZsfBȥ͹t RV֝5QZBmBBT {oI|3i}KlK>軬[o[=rM~ JѬJkʵL2V^,QAjn^mE ]{>;u5zPy-)|j"nEa+? XIfIuބ]^1HyR{!&nDCqaV\&1W }! CXH8YMFBQ{HdN~o, `W${GY<]:Ј c 9a$EbW`UJ5$ 5C3 !s_UTDd,(&YdZC^if&(4iܶaѤ@)ڢgj)N8j٢ziri0[ =Ŋڏe!jLFoTZ2D`/]HVۚ)K/:/%ΘH*rw*oF:PBzkj느r2mRZoKr=J+( r7\x 0ȷ i@ }` jϣ.~l\rTKz|F6!5e+\5Yy%4OCzw~}uu<CocTWҙ3IO]uor8P5ߦNYuˌ綏${ ;eLJzn B!Go>AƓ5ZN}߬"3LІ43P8=n+0_ G7wCz=9(ǣƴGovSQ[q[g a.R9o{oL*"F7Q 8K*J/gŇBt*JhSYBHJGϱ(JqTYB`F @q]T ;%;PBT( j'I@b1f vEaB. *-А2 nK;M܁уtl%IH- Jdcx6S;R|Q89C(c#)1!Lt7a˓,%SRh B6ժgI p2$?lpFX֙LQ^Jo\PDdL;yKba`2bja/@""#ٟh&+7]MG&D-QYAъ%8I90PnxO`#2)% *=[A]͔MАfA;" ET9uFMR"luSgV~SS.j3!LzҌ3OK^ fð)eL،byt}]WE_JnSO*MJWJ*`%T$DS}D,)PE~}(XGx͛ E&ቛۅRGbA&69%M1.c2: ђԄGu F#Y>kwс8 @:f,99&;9\qx/ f$) UM0 (KyP6w1 L%88a ٰa9kW{y!{BZc9vЭh\1&wubvoB@ ʯr~/苻"YO)Q1$XI+)Z޹J8iiCM?qn WC/B%SEs=v㞲37Ԑ}`|.̀nY`$,5Ю6* o|ٵ2`Alm̥l&Aa&0_1hJmj_)۵EI)*%x| 5c lҔASDB\#^Ay 5ɗ䒄4o v8mk6 ecd|oc2Cy^*rlMۥ= rvh룴;۔y7E1ocbo~6H|,, uq31e3//~'tD"F6]jwûe7EGg:;߷ϲoOZu}'})ͅt5-~앚}1k'}~QR P_çW.br Lw7x '#Y4NKX HTF!y -O@$ FCzdTu7 UpH#W&(bFH2*,hwR8}*?c~vwK,w6azuzϒkXK({/Ph W1x'w&c%|;1{0Q`A|I膲kxQ5n$F%iRS814aHZ̧?h20(H(xV60!08%0xisA;NQ9$r7Ê|/ȍ݈h]x t8>悺*8WyC"8ONC(38uэIߨp~5s苍Qh gA{X I~X"A .6-W4aJ]A)9ybI@HRtxi2W9l(9h"Ψ QUzv$ɒW  Y'C;C !B}Ϩ9!>xh8"o q)s OYh Uv6*8yyI ['(tm%u6Wb(dtiiRTقxce;:-!ls(Mq_8&95%2ry Tɗ2yyyv#6q22bgyJ)s i™\[3'𘚈xh)xP|HޙIv Imb ˀI|A6b3%z"Y2بy1@%j'%*!`/ ."`mwwgơ^9#I+J}-ʋZ*X5jbRjC"1e.<"`<8z)LTJ'ʤ&.ڌ06 4cCoY(`yFblֱmo:P Fi;cy$zU 9#bCh"j0ꪯ ʨt=pJs*WFF<>_0|8$z?sdE"),&myJ:S > M 81+4>Uv%kSvIPת:c:6o7*"@$R~J1j^J4MF{ !d!TM trjtxSl'jeQFKzf`RdYПy!+5%[)Ȳ _pLaS %4<+R]H롎PP{HukvTnQ':Q*V49VB1fKնXEw Ke1Rۭ:QY6_yrx ]x껣CJy[ ^{˷}p4vFc oK?D \uYݓFa˾kzg ,:FzcqTBԻH6q )ڻ[3(P[MY˶ bKi,{ f[Ppyєm%羧e)TʋD-`GIlijԴ+[v"s} Y80EY G %=Ӷ(*sLG,$@Q\0VW YLԥѥKc#~0ՑYy'Tۏ<Ձ}yr'L5Gܷ Ԩ9Ke s,-KLkا~+lł,ƶyQ]vm=0a7 - ;[Hfԩ]jvKؓڒ]~#]Ew,M=mvEhڧ0njQ ݧUMbܣh-}ޢm-f=]IM- b]MěUٞ})${Lԉ ؍Mݕt^RBw߼ݙ>4]== lxvW`T߱ܽ$='[][.̼2[܇1&57 H_y想z+->g(7<*Xny.]2Eh1~[}nVPl)MPoD #`>yTZl ,\棘x %.~r~^30R6@t;,^ G쾵3m>;R`B=*vﭻjRkEo2l.[V.wA:/1#3촮UR=4>(H#_a/_d.Ν Ό~<K* I\c{_ +/ nT_D꙰}_g }3nFfD[o"?B@:}ތdV/OA:/?ϝN-X_ٯݯ_4 ȿ!? ]hj$HO#~A/8:A"A Aʢ !㽉A8ƋAX jDZgÔ[T_A.wG5)ݨ}!tnP@B#$澚Kd; !d$2eJhFsLdF}(ԫX-ӇRE-UBIRF*ڛR@WbRgBTӫV}LÄg]LҤ/ s,YSy̶ZFr*楮~6ٌsOʵs^_^u<}FH:HB}zS}s0n9fC_NphF"%)JB9}DW`owZ ڧ`(I`8Oi8al݈߅gn eeL FTdcup)O2ig9Ne!,ZV3m e\1YgqUǨ4]nGi饋ީVejȞYg/fi7V w(FbYff]КlRsqzHt*N"#V'=" VTHFV}FB^H*l:!r[;S#͞HlXMk(Mcj%0w շ" 7 \]G,/K(,*Κ.Vg$!5,SZ%N-,2,Ll4Rتf4 8/xg7Gy $GO,Z!\VӍvWu`֮ŭ]J(L! +GqLpV gHp̠b>HpФgu!GfFԣY t 0L_zP *!0ѓ**MF5ʻ5)pЬ`qOCU#zF٩G*L) 8*ՑP5+Q"R pզnj91IU;+;5ն"Ս\*쮀^W8౐d' ٷ4}AX]!4X6u$ʚ() N&WC,h5i?:,JGu f8^ O]iJַx*\rZC%5'/ԲgFs[ޖb]RuٖwM Gj(ͯ~߆N Y+Ki/+ߖH8ɻdy 7WG$LbR1MT !n07۸eFH\ٶ\9d66rckӔU!yV^$R,.*/r̶#y,߷NY%PsYr΀žL1G|ν9o3dZɨM'ͻ6fz=hVS#gMZZޜ)dîn}mw$[ظFqo|գ:~cAbڳsUc變 Ӕl?S> n{^lКaZFҿev?_`7'98N~;y@gL ri(OyO)K[㑨n蒻mOC|s̫T xpL6G|Nu،h2(XxؘڸHdk;Gt#Th'GզHu7t߈^(> (莕ӊW,%`Qh6x[Qxhfl&gȏhׄdbɐɐV 9HjxX .i!Y9wh'}=9?:)E5הH<0HW9HY=TVdX9敐 *n8:`[_f]GX\9Xky؍2ɒhli3[uVXșG{kIٗɘpAZIyٛ;gpml-matlab-v3.5-2014-12-08/doc/index.html000664 000765 000024 00000163655 12503453060 017643 0ustar00carlstaff000000 000000 Documentation for GPML Matlab Code

Documentation for GPML Matlab Code version 3.5

1) What?

The code provided here originally demonstrated the main algorithms from Rasmussen and Williams: Gaussian Processes for Machine Learning. It has since grown to allow more likelihood functions, further inference methods and a flexible framework for specifying GPs. Other GP packages can be found here.

The code is written by Carl Edward Rasmussen and Hannes Nickisch; it runs on both Octave 3.2.x and Matlab® 7.x and later. The code is based on previous versions written by Carl Edward Rasmussen and Chris Williams.

2) Download, Install and Documentation

All the code including demonstrations and html documentation can be downloaded in a tar or zip archive file.

Minor changes and incremental bugfixes to the current version are documented in the changelog, changes from previous versions are documented in README.

Please read the copyright notice.

After unpacking the tar or zip file you will find 7 subdirectories: cov, doc, inf, lik, mean, prior and util. It is not necessary to install anything to get started, just run the startup script to set your path.

Details about the directory contents and on how to compile mex files can be found in the README. The getting started guide is the remainder of the html file you are currently reading (also available at http://gaussianprocess.org/gpml/code/matlab/doc). A Developer's Guide containing technical documentation is found in manual.pdf, but for the casual user, the guide is below.

3) Theory

Gaussian Processes (GPs) can conveniently be used for Bayesian supervised learning, such as regression and classification. In its simplest form, GP inference can be implemented in a few lines of code. However, in practice, things typically get a little more complicated: you might want to use complicated covariance functions and mean functions, learn good values for hyperparameters, use non-Gaussian likelihood functions (rendering exact inference intractable), use approximate inference algorithms, or combinations of many or all of the above. This is what the GPML software package does.

Before going straight to the examples, just a brief note about the organization of the package. There are four types of objects which you need to know about:

Gaussian Process
A Gaussian Process is fully specified by a mean function and a covariance function. These functions are specified separately, and consist of a specification of a functional form as well as a set of parameters called hyperparameters, see below.
Mean functions
Several mean functions are available, all start with the four letters mean and reside in the mean directory. An overview is provided by the meanFunctions help function (type help meanFunctions to get help), and an example is the meanLinear function.
Covariance functions
There are many covariance functions available, all start with the three letters cov and reside in the cov directory. An overview is provided by the covFunctions help function (type help covFunctions to get help), and an example is the covSEard "Squared Exponential with Automatic Relevance Determination" covariance function.
For both mean functions and covariance functions, two types exist: simple and composite. Whereas simple types are specified by the function name (or function pointer), composite functions join together several components using cell arrays. Composite functions can be composed of other composite functions, allowing for very flexible and interesting structures.
Examples are given below and in the usageMean and usageCov functions.
Hyperparameters
GPs are typically specified using mean and covariance functions which have free parameters called hyperparameters. Also likelihood functions may have such parameters. These are encoded in a struct with the fields mean, cov and lik (some of which may be empty). When specifying hyperparameters, it is important that the number of elements in each of these struct fields, precisely match the number of parameters expected by the mean function, the covariance function and the likelihood functions respectively (otherwise an error will result). Hyperparameters whose natural domain is positive are represented by their logarithms. When working with FITC approximate inference, the inducing inputs xu can also be treated as hyperparameters for some common stationary covariances.
Hyperparameter Prior Distributions
When optimising the marginal likelihood w.r.t. hyperparameters, it is sometimes useful to softly constrain the hyperparameters by means of prior knowledge. A hyperprior is a probability distribution over individual or a group of hyperparameters. All hyperpriors begin with letters prior and reside in the prior directory. An overview is provided by the priorDistributions help function (type help priorDistributions to get help). Some examples are priorGauss and priorGaussMulti the uni- and multivariate Gaussian prior, priorSmoothBox1 and priorSmoothBox2 the univariate (smoothed) box prior with linear or quadratic decay, priorMix the mixture prior, and priorTransform allowing to place a prior on a transformation g(x) rather than on x directly. Further, we have the two equivalents priorDelta and priorClamp to exclude a parameter from optimisation.
Examples are given below and in the usagePrior function.
Likelihood Functions
The likelihood function specifies the probability of the observations given the latent function, i.e. the GP (and the hyperparameters). All likelihood functions begin with the three letters lik and reside in the lik directory. An overview is provided by the likFunctions help function (type help likFunctions to get help). Some examples are likGauss the Gaussian likelihood for regression, likPoisson the Poisson likelihood for count data, likGamma the Gamma likelihood for positive data, likBeta the Beta likelihood for interval data, and likLogistic the logistic function used in classification (a.k.a. logistic regression). Output dependent noise can be modeled using warped GPs implemented by likGaussWarp. As for the mean and covariance functions there is also a composite likelihood likMix that can be used to generate mixtures of other likelihood functions.
Examples are given below and in the usageLik function.
Inference Methods
The inference methods specify how to compute with the model, i.e. how to infer the (approximate) posterior process, how to find hyperparameters, evaluate the log marginal likelihood and how to make predictions. Inference methods all begin with the three letters inf and reside in the inf directory. An overview is provided by the infMethods help file (type help infMethods to get help). Some examples are infExact for exact inference (regression with Gaussian likelihood), infFITC and infGrid for large scale Gaussian regression, infEP for the Expectation Propagation algorithm, infVB and infKL for two variational approximations based on a lower bound to the marginal likelihood, or infFITC_Laplace for large scale Laplace's approximation.
Further usage examples are provided for both regression and classification. However, not all combinations of likelihood function and inference method are possible (e.g. you cannot do exact inference with a Laplace likelihood). An exhaustive compatibility matrix between likelihoods (rows) and inference methods (columns) is given in the table below:
Plain Regression Likelihoods
Inference GPML name Exact
infExact		 EP
infEP		 Laplace
infLaplace		 Variational Bayes
infVB		 Kullback Leibler
infKL		 Sampling
infMCMC		 LOO
infLOO		 type, output domain
 		 alternative names
 
Likelihood FITC
infFITC		 FITC-EP
infFITC_EP		 FITC-Laplace
infFITC_Laplace
Grid
infGrid		    
Gaussian likGauss $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ regression, IR  
Warped Gaussian likGaussWarp   $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ regression, IR  
Gumbel likGumbel     $\checkmark$   $\checkmark$ $\checkmark$ $\checkmark$ regression, IR extremal value regression
Sech squared likSech2   $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ regression, IR logistic distribution
Laplacian likLaplace   $\checkmark$   $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ regression, IR double exponential
Student's t likT     $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ regression, IR  
Composite Likelihoods
Inference GPML name Exact
infExact		 EP
infEP		 Laplace
infLaplace		 Variational Bayes
infVB		 Kullback Leibler
infKL		 Sampling
infMCMC		 LOO
infLOO		 type, output domain
 		 alternative names
 
Likelihood FITC
infFITC		 FITC-EP
infFITC_EP		 FITC-Laplace
infFITC_Laplace
Grid
infGrid		    
Mixture likMix   $\checkmark$ $\checkmark$   $\checkmark$ $\checkmark$ $\checkmark$ classification, ±1 and regression, IR mixing meta likelihood
Classification Likelihoods
Inference GPML name Exact
infExact		 EP
infEP		 Laplace
infLaplace		 Variational Bayes
infVB		 Kullback Leibler
infKL		 Sampling
infMCMC		 LOO
infLOO		 type, output domain
 		 alternative names
 
Likelihood FITC
infFITC		 FITC-EP
infFITC_EP		 FITC-Laplace
infFITC_Laplace
Grid
infGrid		    
Uniform likUni   $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ classification, ±1 label noise
Error function likErf   $\checkmark$ $\checkmark$   $\checkmark$ $\checkmark$ $\checkmark$ classification, ±1 probit regression
Logistic function likLogistic   $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ classification, ±1 logistic regression
logit regression
Generalized Linear Model Likelihoods
Inference GPML name Exact
infExact		 EP
infEP		 Laplace
infLaplace		 Variational Bayes
infVB		 Kullback Leibler
infKL		 Sampling
infMCMC		 LOO
infLOO		 type, output domain
 		 alternative names
 
Likelihood FITC
infFITC		 FITC-EP
infFITC_EP		 FITC-Laplace
infFITC_Laplace
Grid
infGrid		    
Weibull likWeibull     $\checkmark$     $\checkmark$ $\checkmark$ positive data, IR+\{0} nonnegative regression
Gamma likGamma     $\checkmark$     $\checkmark$ $\checkmark$ positive data, IR+\{0} nonnegative regression
Exponential likExp     $\checkmark$     $\checkmark$ $\checkmark$ positive data, IR+\{0} nonnegative regression
Inverse Gaussian likInvGauss     $\checkmark$     $\checkmark$ $\checkmark$ positive data, IR+\{0} nonnegative regression
Poisson likPoisson   ($\checkmark$)* $\checkmark$   $\checkmark$ $\checkmark$ $\checkmark$ count data, IN Poisson regression
Beta likBeta     $\checkmark$     $\checkmark$ $\checkmark$ interval data, [0,1] range regression

* EP might not converge in some cases since quadrature is used.

All of the objects described above are written in a modular way, so you can add functionality if you feel constrained despite the considerable flexibility provided. Details about how to do this are provided in the developer documentation.

Inference by MCMC sampling is the only inference method that cannot be used as a black box. Also gradient-based marginal likelihood optimisation is not possible with MCMC. Please see usageSampling for a toy example illustrating the usage of the implemented samplers.

4) Practice

Using the GPML package is simple. There is only one single function to call: gp, it does posterior inference, learns hyperparameters, computes the marginal likelihood and makes predictions. Generally, the gp function takes the following arguments: a hyperparameter struct, an inference method, a mean function, a covariance function, a likelihood function, training inputs, training targets, and possibly test cases. The exact computations done by the function is controlled by the number of input and output arguments in the call. Here is part of the help message for the gp function (follow the link to see the whole thing):

  function [varargout] = gp(hyp, inf, mean, cov, lik, x, y, xs, ys)

  [ ... snip ...]

  Two modes are possible: training or prediction: if no test cases are
  supplied, then the negative log marginal likelihood and its partial
  derivatives wrt the hyperparameters is computed; this mode is used to fit the
  hyperparameters. If test cases are given, then the test set predictive
  probabilities are returned. Usage:

    training: [nlZ dnlZ          ] = gp(hyp, inf, mean, cov, lik, x, y);
  prediction: [ymu ys2 fmu fs2   ] = gp(hyp, inf, mean, cov, lik, x, y, xs);
          or: [ymu ys2 fmu fs2 lp] = gp(hyp, inf, mean, cov, lik, x, y, xs, ys);

  [ .. snip ...]

Here x and y are training inputs and outputs, and xs and ys are test set inputs and outputs, nlZ is the negative log marginal likelihood and dnlZ its partial derivatives wrt the hyperparameters (which are used for training the hyperparameters). The prediction outputs are ymu and ys2 for test output mean and covariance, and fmu and fs2 are the equivalent quenteties for the corresponding latent variables. Finally, lp are the test output log probabilities.

Instead of exhaustively explaining all the possibilities, we will give two illustrative examples to give you the idea; one for regression and one for classification. You can either follow the example here on this page, or using the two scripts demoRegression and demoClassification (using the scripts, you still need to follow the explanation on this page).

3a) Regression

You can either follow the example here on this page, or use the script demoRegression.

This is a simple example, where we first generate n=20 data points from a GP, where the inputs are scalar (so that it is easy to plot what is going on). We then use various other GPs to make inferences about the underlying function.

First, generate some data from a Gaussian process (it is not essential to understand the details of this):

  clear all, close all
 
  meanfunc = {@meanSum, {@meanLinear, @meanConst}}; hyp.mean = [0.5; 1];
  covfunc = {@covMaterniso, 3}; ell = 1/4; sf = 1; hyp.cov = log([ell; sf]);
  likfunc = @likGauss; sn = 0.1; hyp.lik = log(sn);
 
  n = 20;
  x = gpml_randn(0.3, n, 1);
  K = feval(covfunc{:}, hyp.cov, x);
  mu = feval(meanfunc{:}, hyp.mean, x);
  y = chol(K)'*gpml_randn(0.15, n, 1) + mu + exp(hyp.lik)*gpml_randn(0.2, n, 1);

  plot(x, y, '+')

Above, we first specify the mean function meanfunc, covariance function covfunc of a GP and a likelihood function, likfunc. The corresponding hyperparameters are specified in the hyp structure:

The mean function is composite, adding (using meanSum function) a linear (meanLinear) and a constant (meanConst) to get an affine function. Note, how the different components are composed using cell arrays. The hyperparameters for the mean are given in hyp.mean and consists of a single (because the input will one dimensional, i.e. D=1) slope (set to 0.5) and an off-set (set to 1). The number and the order of these hyperparameters conform to the mean function specification. You can find out how many hyperparameters a mean (or covariance or likelihood function) expects by calling it without arguments, such as feval(meanfunc{:}). For more information on mean functions see meanFunctions and the directory mean/.

The covariance function is of the Matérn form with isotropic distance measure covMaterniso. This covariance function is also composite, as it takes a constant (related to the smoothness of the GP), which in this case is set to 3. The covariance function takes two hyperparameters, a characteristic length-scale ell and the standard deviation of the signal sf. Note, that these positive parameters are represented in hyp.cov using their logarithms. For more information on covariance functions see covFunctions and cov/.

Finally, the likelihood function is specified to be Gaussian. The standard deviation of the noise sn is set to 0.1. Again, the representation in the hyp.lik is given in terms of its logarithm. For more information about likelihood functions, see likFunctions and lik/.

Then, we generate a dataset with n=20 examples. The inputs x are drawn from a unit Gaussian (using the gpml_randn utility, which generates unit Gaussian pseudo random numbers with a specified seed). We then evaluate the covariance matrix K and the mean vector m by calling the corresponding functions with the hyperparameters and the input locations x. Finally, the targets y are computed by drawing randomly from a Gaussian with the desired covariance and mean and adding Gaussian noise with standard deviation exp(hyp.lik). The above code is a bit special because we explicitly call the mean and covariance functions (in order to generate samples from a GP); ordinarily, we would only directly call the gp function.

f1.gif

Let's ask the model to compute the (joint) negative log probability (density) nlml (also called marginal likelihood or evidence) and to generalize from the training data to other (test) inputs z:

  nlml = gp(hyp, @infExact, meanfunc, covfunc, likfunc, x, y)

  z = linspace(-1.9, 1.9, 101)';
  [m s2] = gp(hyp, @infExact, meanfunc, covfunc, likfunc, x, y, z);

  f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)]; 
  fill([z; flipdim(z,1)], f, [7 7 7]/8)
  hold on; plot(z, m); plot(x, y, '+')

The gp function is called with a struct of hyperparameters hyp, and inference method, in this case @infExact for exact inference and the mean, covariance and likelihood functions, as well as the inputs and outputs of the training data. With no test inputs, gp returns the negative log probability of the training data, in this example nlml=11.97.

To compute the predictions at test locations we add the test inputs z as a final argument, and gp returns the mean m variance s2 at the test location. The program is using algorithm 2.1 from the GPML book. Plotting the mean function plus/minus two standard deviations (corresponding to a 95% confidence interval):

f2.gif

Typically, we would not a priori know the values of the hyperparameters hyp, let alone the form of the mean, covariance or likelihood functions. So, let's pretend we didn't know any of this. We assume a particular structure and learn suitable hyperparameters:

  covfunc = @covSEiso; hyp2.cov = [0; 0]; hyp2.lik = log(0.1);

  hyp2 = minimize(hyp2, @gp, -100, @infExact, [], covfunc, likfunc, x, y);
  exp(hyp2.lik)
  nlml2 = gp(hyp2, @infExact, [], covfunc, likfunc, x, y)

  [m s2] = gp(hyp2, @infExact, [], covfunc, likfunc, x, y, z);
  f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)];
  fill([z; flipdim(z,1)], f, [7 7 7]/8)
  hold on; plot(z, m); plot(x, y, '+')

First, we guess that a squared exponential covariance function covSEiso may be suitable. This covariance function takes two hyperparameters: a characteristic length-scale and a signal standard deviation (magnitude). These hyperparameters are non-negative and represented by their logarithms; thus, initializing hyp2.cov to zero, correspond to unit characteristic length-scale and unit signal standard deviation. The likelihood hyperparameter in hyp2.lik is also initialized. We assume that the mean function is zero, so we simply ignore it (and when in the following we call gp, we give an empty argument for the mean function).

In the following line, we optimize over the hyperparameters, by minimizing the negative log marginal likelihood w.r.t. the hyperparameters. The third parameter in the call to minimize limits the number of function evaluations to a maximum of 100. The inferred noise standard deviation is exp(hyp2.lik)=0.15, somewhat larger than the one used to generate the data (0.1). The final negative log marginal likelihood is nlml2=14.13, showing that the joint probability (density) of the training data is about exp(14.13-11.97)=8.7 times smaller than for the setup actually generating the data. Finally, we plot the predictive distribution.

f3.gif

This plot shows clearly, that the model is indeed quite different from the generating process. This is due to the different specifications of both the mean and covariance functions. Below we'll try to do a better job, by allowing more flexibility in the specification.

Note that the confidence interval in this plot is the confidence for the distribution of the (noisy) data. If instead you want the confidence region for the underlying function, you should use the 3rd and 4th output arguments from gp as these refer to the latent process, rather than the data points.

  hyp.cov = [0; 0]; hyp.mean = [0; 0]; hyp.lik = log(0.1);
  hyp = minimize(hyp, @gp, -100, @infExact, meanfunc, covfunc, likfunc, x, y);
  [m s2] = gp(hyp, @infExact, meanfunc, covfunc, likfunc, x, y, z);
 
  f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)];
  fill([z; flipdim(z,1)], f, [7 7 7]/8)
  hold on; plot(z, m); plot(x, y, '+');

Here, we have changed the specification by adding the affine mean function. All the hyperparameters are learnt by optimizing the marginal likelihood.

f4.gif

This shows that a much better fit is achieved when allowing a mean function (although the covariance function is still different from that of the generating process).

Guiding marginal likelihood optimisation with a hyperprior

It can be usefull to put a prior distribution on (a part of) the hyperparameters. Sometimes, one may want to exclude some hyperparameters from the optimisation i.e. fix their values beforehand and treat them as constants.

In these cases, a hyperprior comes to bear. A hyperprior is specified by augmenting the inf parameter of gp.m In the regression before, we had inf = @infExact;. To put a Gaussian prior on the first mean hyperparameter hyp.mean(1) and a Laplacian prior on the second mean hyperparameter hyp.mean(2) and wished to fix the noise variance hyperparameter hyp.lik, we simple need to set up the corresponding prior structure as detailed below. 

  mu = 1.0; s2 = 0.01^2;
  prior.mean = {{@priorGauss,mu,s2}; {'priorLaplace',mu,s2}};
  prior.lik = {{@priorDelta}};
  inf = {@infPrior,@infExact,prior};
  hyp = minimize(hyp, @gp, -100, inf, meanfunc, covfunc, likfunc, x, y);

Further examples are provided in usagePrior.

Large scale regression using the FITC approximation

In case the number of training inputs x exceeds a few thousands, exact inference using infExact.m takes too long. We offer the FITC approximation based on a low-rank plus diagonal approximation to the exact covariance to deal with these cases. The general idea is to use inducing points u and to base the computations on cross-covariances between training, test and inducing points only.

Using the FITC approximation is very simple, we just have to wrap the covariance function covfunc into covFITC.m and call gp.m with the inference method infFITC.m as demonstrated by the following lines of code. 

  nu = fix(n/2); u = linspace(-1.3,1.3,nu)';
  covfuncF = {@covFITC, {covfunc}, u};
  [mF s2F] = gp(hyp, @infFITC, meanfunc, covfuncF, likfunc, x, y, z);

We define equispaced inducing points u that are shown in the figure as black circles. Note that the predictive variance is overestimated outside the support of the inducing inputs. In a multivariate example where densely sampled inducing inputs are infeasible, one can simply use a random subset of the training points. 

  nu = fix(n/2); iu = randperm(n); iu = iu(1:nu); u = x(iu,:);
f5.gif

Large scale regression exploiting the grid structure

A covariance function factorising over coordinates axes leads to a covariance matrix with Kronecker structure. This can be exploited to scale GPs beyond the O(n³) limit. If there are observations for all points of the grid, all computations are exact. The example below contains the most relevant code from the script demoGrid, where we extrapolate a pixel image beyond its boundaries.

For a comprehensive set of examples and more resources, see a website by Andrew Wilson.

We start off by setting up the training data and the GP on a [-2,2]x[-3,3] lattice with 14400 pixels -- a size where a usual dens GP would be computationally infeasible. Then we wrap the covariance functions for the axes into covGrid and specify the initial values for the hyperparameters. Note that covGrid can only be used in conjunction with infGrid. 

  xg = {linspace(-2,2,120)',linspace(-3,3,120)'}; xe = covGrid('expand',xg);
  x = (1:size(xe,1))'; y = sin(xe(:,2)) + xe(:,1) + 0.01*gpml_randn(1,size(x));
  cov = {{@covSEiso},{@covSEiso}}; covg = {@covGrid,cov,xg};
  hyp.cov = zeros(4,1); hyp.mean = []; hyp.lik = log(0.1);

As a next step, we perform hyperparameter optimisation as with a usual GP model. The inference method inGrid only supports likGauss, hence we can leave the field empty. 

  hyp = minimize(hyp,@gp,-50,@infGrid,@meanZero,covg,[],x,y);

Finally, we predict on a [-4,4]x[-6,6] lattice to see the result of the extrapolation. The figure shows on the left the training data and on the right the prediction. Note that, we also support partial observations. 

  xs = covGrid('expand',{linspace(-4,4,100)',linspace(-6,6,110)'});
  opt.cg_maxit = 200; opt.cg_tol = 5e-3;
  post = infGrid(hyp,{@meanZero},covg,'likGauss',x,y,opt); post.L = @(a) a;
  ym = gp(hyp, @infGrid,[],covg,[],x,post,xs);
f9.png

Exercises for the reader

Inference Methods
Try using Expectation Propagation instead of exact inference in the above, by exchanging @infExact with @infEP. You get exactly identical results, why?
Mean or Covariance
Try training a GP where the affine part of the function is captured by the covariance function instead of the mean function. That is, use a GP with no explicit mean function, but further additive contributions to the covariance. How would you expect the marginal likelihood to compare to the previous case?

3b) Classification

You can either follow the example here on this page, or use the script demoClassification.

The difference between regression and classification isn't of fundamental nature. We can use a Gaussian process latent function in essentially the same way, it is just that the Gaussian likelihood function often used for regression is inappropriate for classification. And since exact inference is only possible for Gaussian likelihood, we also need an alternative, approximate, inference method.

Here, we will demonstrate binary classification, using two partially overlapping Gaussian sources of data in two dimensions. First we generate the data:

  clear all, close all
 
  n1 = 80; n2 = 40;                   % number of data points from each class
  S1 = eye(2); S2 = [1 0.95; 0.95 1];           % the two covariance matrices
  m1 = [0.75; 0]; m2 = [-0.75; 0];                            % the two means
 
  x1 = bsxfun(@plus, chol(S1)'*gpml_randn(0.2, 2, n1), m1);
  x2 = bsxfun(@plus, chol(S2)'*gpml_randn(0.3, 2, n2), m2);
 
  x = [x1 x2]'; y = [-ones(1,n1) ones(1,n2)]';
  plot(x1(1,:), x1(2,:), 'b+'); hold on;
  plot(x2(1,:), x2(2,:), 'r+');

120 data points are generated from two Gaussians with different means and covariances. One Gaussian is isotropic and contains 2/3 of the data (blue), the other is highly correlated and contains 1/3 of the points (red). Note, that the labels for the targets are ±1 (and not 0/1).

In the plot, we superimpose the data points with the posterior equi-probability contour lines for the probability of class two given complete information about the generating mechanism

  [t1 t2] = meshgrid(-4:0.1:4,-4:0.1:4);
  t = [t1(:) t2(:)]; n = length(t);                 % these are the test inputs
  tmm = bsxfun(@minus, t, m1');
  p1 = n1*exp(-sum(tmm*inv(S1).*tmm/2,2))/sqrt(det(S1));
  tmm = bsxfun(@minus, t, m2');
  p2 = n2*exp(-sum(tmm*inv(S2).*tmm/2,2))/sqrt(det(S2));
  contour(t1, t2, reshape(p2./(p1+p2), size(t1)), [0.1:0.1:0.9]);
f6.gif

We specify a Gaussian process model as follows: a constant mean function, with initial parameter set to 0, a squared exponential with automatic relevance determination (ARD) covariance function covSEard. This covariance function has one characteristic length-scale parameter for each dimension of the input space, and a signal magnitude parameter, for a total of 3 parameters (as the input dimension is D=2). ARD with separate length-scales for each input dimension is a very powerful tool to learn which inputs are important for predictions: if length-scales are short, inputs are very important, and when they grow very long (compared to the spread of the data), the corresponding inputs will be largely ignored. Both length-scales and the signal magnitude are initialized to 1 (and represented in the log space). Finally, the likelihood function likErf has the shape of the error-function (or cumulative Gaussian), which doesn't take any hyperparameters (so hyp.lik does not exist).

  meanfunc = @meanConst; hyp.mean = 0;
  covfunc = @covSEard; ell = 1.0; sf = 1.0; hyp.cov = log([ell ell sf]);
  likfunc = @likErf;

  hyp = minimize(hyp, @gp, -40, @infEP, meanfunc, covfunc, likfunc, x, y);
  [a b c d lp] = gp(hyp, @infEP, meanfunc, covfunc, likfunc, x, y, t, ones(n, 1));

  plot(x1(1,:), x1(2,:), 'b+'); hold on; plot(x2(1,:), x2(2,:), 'r+')
  contour(t1, t2, reshape(exp(lp), size(t1)), [0.1:0.1:0.9]);

We train the hyperparameters using minimize, to minimize the negative log marginal likelihood. We allow for 40 function evaluations, and specify that inference should be done with the Expectation Propagation (EP) inference method @infEP, and pass the usual parameters. Training is done using algorithm 3.5 and 5.2 from the gpml book. When computing test probabilities, we call gp with additional test inputs, and as the last argument a vector of targets for which the log probabilities lp should be computed. The fist four output arguments of the function are mean and variance for the targets and corresponding latent variables respectively. The test set predictions are computed using algorithm 3.6 from the GPML book. The contour plot for the predictive distribution is shown below. Note, that the predictive probability is fairly close to the probabilities of the generating process in regions of high data density. Note also, that as you move away from the data, the probability approaches 1/3, the overall class probability.

f7.gif

Examining the two ARD characteristic length-scale parameters after learning, you will find that they are fairly similar, reflecting the fact that for this data set, both inputs important.

Large scale classification using the FITC approximation

In case the number of training inputs x exceeds a few hundreds, approximate inference using infLaplace.m, infEP.m and infVB.m takes too long. As in regression, we offer the FITC approximation based on a low-rank plus diagonal approximation to the exact covariance to deal with these cases. The general idea is to use inducing points u and to base the computations on cross-covariances between training, test and inducing points only.

Using the FITC approximation is very simple, we just have to wrap the covariance function covfunc into covFITC.m and call gp.m with the inference methods infFITC_Laplace.m and infFITC_EP.m as demonstrated by the following lines of code. 

  [u1,u2] = meshgrid(linspace(-2,2,5)); u = [u1(:),u2(:)]; clear u1; clear u2
  nu = size(u,1);
  covfuncF = {@covFITC, {covfunc}, u};
  inffunc = @infFITC_EP;                       % also @infFITC_Laplace is possible
  hyp = minimize(hyp, @gp, -40, inffunc, meanfunc, covfuncF, likfunc, x, y);
  [a b c d lp] = gp(hyp, inffunc, meanfunc, covfuncF, likfunc, x, y, t, ones(n,1));

We define equispaced inducing points u that are shown in the figure as black circles. Alternatively, a random subset of the training points can be used as inducing points.

f8.gif

Exercise for the reader

Inference Methods
Use the Laplace Approximation for inference @infLaplace (or @infFITC_Laplace in the large scale example), and compare the approximate marginal likelihood for the two methods. Which approximation is best?
Covariance Function
Try using the squared exponential with isotropic distance measure covSEiso instead of ARD distance measure covSEard. Which is best?

5) Acknowledgements

Innumerable colleagues have helped to improve this software. Some of these are: John Cunningham, Máté Lengyel, Joris Mooij, Iain Murray, David Duvenaud, Andrew McHutchon, Rowan McAllister, Daniel Marthaler, Giampiero Salvi, Mark van der Wilk, Marco Fraccaro, Dali Wei, Ernst Kloppenburg, Ryan Turner and Chris Williams. Especially Ed Snelson helped to improve the code and to include sparse approximations and Roman Garnett and José Vallet helped to include hyperparameter priors. The spectral mixture covariance function and the grid-based inference were contributed by Andrew Gordon Wilson and periodic covariances were added by James Robert Lloyd.

Last modified: March 23rd 2015 gpml-matlab-v3.5-2014-12-08/doc/changelog000644 000765 000024 00000000716 12503453137 017507 0ustar00carlstaff000000 000000 GAUSSIAN PROCESS REGRESSION AND CLASSIFICATION Toolbox version 3.5 for GNU Octave 3.2.x and Matlab 7.x Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-03-23. Changes and small incremental bug fixes applied since the last major release from 2014-12-08. - smal bugfixes in covGrid and infGrid - bugfix in predictive variance of likNegBinom due to Seth Flaxman - bugfix in infFITC_Laplace as suggested by Wu Lin - bugfix in covPP{iso,ard} gpml-matlab-v3.5-2014-12-08/doc/usageRegression.m000644 000765 000024 00000005271 12240501133 021145 0ustar00carlstaff000000 000000 % demonstrate usage of regression % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2013-10-16. clear all, close all %% SAY WHICH CODE WE WISH TO EXERCISE id = [1,1]; % use Gauss/Exact id = [1,2; 3,2; 4,2]; % compare Laplace id = [1,3; 2,3; 3,3]; % study EP id = [1,5; 2,5]; % look into KL (takes quite a while) id = [1,4; 2,4; 3,4; 4,4]; % deal with VB seed = 197; randn('seed',seed), rand('seed',seed) ntr = 50; nte = 1e4; % number of training and test points xtr = 10*sort(rand(ntr,1)); % sample dataset f = @(x) sin(x)+sqrt(x); % "true" underlying function sn = 0.2; ytr = f(xtr) + randn(ntr,1)*sn; % add Gaussian noise i = randperm(ntr); nout = 3; % add outliers ytr(i(1:nout)) = 5; xte = linspace(0,10,1e4)'; % support, we test our function on cov = {@covSEiso}; sf = 1; ell = 0.4; % setup the GP hyp0.cov = log([ell;sf]); mean = {@meanSum,{@meanLinear,@meanConst}}; a = 1/5; b = 1; % m(x) = a*x+b hyp0.mean = [a;b]; lik_list = {'likGauss','likLaplace','likSech2','likT'}; % possible likelihoods inf_list = {'infExact','infLaplace','infEP','infVB','infKL'}; % inference algs Ncg = 50; % number of conjugate gradient steps sdscale = 0.5; % how many sd wide should the error bars become? col = {'k',[.8,0,0],[0,.5,0],'b',[0,.75,.75],[.7,0,.5]}; % colors ymu{1} = f(xte); ys2{1} = sn^2; nlZ(1) = -Inf; for i=1:size(id,1) lik = lik_list{id(i,1)}; % setup the likelihood if strcmp(lik,'likT') nu = 4; hyp0.lik = log([nu-1;sqrt((nu-2)/nu)*sn]); else hyp0.lik = log(sn); end inf = inf_list{id(i,2)}; fprintf('OPT: %s/%s\n',lik_list{id(i,1)},inf_list{id(i,2)}) if Ncg==0 hyp = hyp0; else hyp = minimize(hyp0,'gp', -Ncg, inf, mean, cov, lik, xtr, ytr); % opt hypers end [ymu{i+1}, ys2{i+1}] = gp(hyp, inf, mean, cov, lik, xtr, ytr, xte); % predict [nlZ(i+1)] = gp(hyp, inf, mean, cov, lik, xtr, ytr); end figure, hold on for i=1:size(id,1)+1 plot(xte,ymu{i},'Color',col{i},'LineWidth',2) if i==1 leg = {'function'}; else leg{end+1} = sprintf('%s/%s -lZ=%1.2f',... lik_list{id(i-1,1)},inf_list{id(i-1,2)},nlZ(i)); end end for i=1:size(id,1)+1 ysd = sdscale*sqrt(ys2{i}); fill([xte;flipud(xte)],[ymu{i}+ysd;flipud(ymu{i}-ysd)],... col{i},'EdgeColor',col{i},'FaceAlpha',0.1,'EdgeAlpha',0.3); end for i=1:size(id,1)+1, plot(xte,ymu{i},'Color',col{i},'LineWidth',2), end plot(xtr,ytr,'k+'), plot(xtr,ytr,'ko'), legend(leg) gpml-matlab-v3.5-2014-12-08/doc/gpml_randn.m000644 000765 000024 00000002125 11417160122 020120 0ustar00carlstaff000000 000000 function x = gpml_randn(seed,varargin) % Generate pseudo-random numbers in a quick and dirty way. % The function makes sure, we obtain the same random numbers using Octave and % Matlab for the demo scripts. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2010-07-07. if nargin==2 sz = varargin{1}; else sz = zeros(1,nargin-1); for i=1:nargin-1 sz(i) = varargin{i}; end end n = prod(sz); N = ceil(n/2)*2; % minimal uniform random number generator for uniform deviates from [0,1] % by Park and Miller a = 7^5; m = 2^31-1; % using Schrage's algorithm q = fix(m/a); r = mod(m,a); % m = a*q+r u = zeros(N+1,1); u(1) = fix(seed*2^31); for i=2:N+1 % Schrage's algorithm for mod(a*u(i),m) u(i) = a*mod(u(i-1),q) - r*fix(u(i-1)/q); if u(i)<0, u(i) = u(i)+m; end end u = u(2:N+1)/2^31; % Box-Muller transform: Numerical Recipies, 2nd Edition, $7.2.8 % http://en.wikipedia.org/wiki/Box-Muller_transform w = sqrt(- 2*log(u(1:N/2))); % split into two groups x = [w.*cos(2*pi*u(N/2+1:N)); w.*sin(2*pi*u(N/2+1:N))]; x = reshape(x(1:n),sz); gpml-matlab-v3.5-2014-12-08/doc/usageCov.m000664 000765 000024 00000007063 12526350575 017602 0ustar00carlstaff000000 000000 % demonstrate usage of covariance functions % % See also covFunctions.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-05-18. % File automatically generated using noweb. clear all, close all n = 5; D = 3; x = randn(n,D); xs = randn(3,D); % create a data set % set up simple covariance functions cn = {'covNoise'}; sn = .1; hypn = log(sn); % one hyperparameter cc = {@covConst}; sf = 2; hypc = log(sf); % function handles OK ce = {@covEye}; hype = []; % identity cl = {@covLIN}; hypl = []; % linear is parameter-free cla = {'covLINard'}; L = rand(D,1); hypla = log(L); % linear (ARD) cli = {'covLINiso'}; l = rand(1); hypli = log(l); % linear iso clo = {@covLINone}; ell = .9; hyplo = log(ell); % linear with bias cp = {@covPoly,3}; c = 2; hypp = log([c;sf]); % third order poly cga = {@covSEard}; hypga = log([L;sf]); % Gaussian with ARD cgi = {'covSEiso'}; hypgi = log([ell;sf]); % isotropic Gaussian cgu = {'covSEisoU'}; hypgu = log(ell); % isotropic Gauss no scale cra = {'covRQard'}; al = 2; hypra = log([L;sf;al]); % ration. quad. cri = {@covRQiso}; hypri = log([ell;sf;al]); % isotropic cma = {@covMaternard,5}; hypma = log([ell;sf]); % Matern class d=5 cmi = {'covMaterniso',3}; hypmi = log([ell;sf]); % Matern class d=3 cnn = {'covNNone'}; hypnn = log([L;sf]); % neural network cpe = {'covPeriodic'}; p = 2; hyppe = log([ell;p;sf]); % periodic cpn = {'covPeriodicNoDC'}; p = 2; hyppe = log([ell;p;sf]); % w/o DC cpc = {'covCos'}; p = 2; hypcpc = log([p;sf]); % cosine cov cca = {'covPPard',3}; hypcc = hypgu;% compact support poly degree 3 cci = {'covPPiso',2}; hypcc = hypgi;% compact support poly degree 2 cgb = {@covGaboriso}; ell = 1; p = 1.2; hypgb=log([ell;p]); % Gabor Q = 2; w = ones(Q,1)/Q; m = rand(D,Q); v = rand(D,Q); csm = {@covSM,Q}; hypsm = log([w;m(:);v(:)]); % Spectral Mixture cvl = {@covSEvlen,{@meanLinear}}; hypvl = [1;2;1; 0]; % var lenscal s = 12; cds = {@covDiscrete,s}; % discrete covariance function L = randn(s); L = chol(L'*L); L(1:(s+1):end) = log(diag(L)); hypds = L(triu(true(s))); xd = randi([1,s],[n,1]); xsd = [1;3;6]; cfa = {@covSEfact,2}; hypfa = randn(D*2,1); % factor analysis % set up composite i.e. meta covariance functions csc = {'covScale',{cgu}}; hypsc = [log(3); hypgu]; % scale by 9 csu = {'covSum',{cn,cc,cl}}; hypsu = [hypn; hypc; hypl]; % sum cpr = {@covProd,{cc,cci}}; hyppr = [hypc; hypcc]; % product mask = [0,1,0]; % binary mask excluding all but the 2nd component cma = {'covMask',{mask,cgi{:}}}; hypma = hypgi; % isotropic periodic rational quadratic cpi = {'covPERiso',{@covRQiso}}; % periodic Matern with ARD cpa = {'covPERard',{@covMaternard,3}}; % additive based on SEiso using unary and pairwise interactions cad = {'covADD',{[1,2],'covSEiso'}}; % preference covariance with squared exponential base covariance cpr = {'covPref',{'covSEiso'}}; hyppr = [0;0]; xp = randn(n,2*D); xsp = randn(3,2*D); % 0) specify a covariance function % cov = cma; hyp = hypma; % cov = cci; hyp = hypcc; % cov = csm; hyp = hypsm; cov = cds; hyp = hypds; x = xd; xs = xsd; % cov = cfa; hyp = hypfa; % cov = cvl; hyp = hypvl; % cov = cpr; hyp = hyppr; x = xp; xs = xsp; % 1) query the number of parameters feval(cov{:}) % 2) evaluate the function on x feval(cov{:},hyp,x) % 3) evaluate the function on x and xs to get cross-terms kss = feval(cov{:},hyp,xs,'diag') Ks = feval(cov{:},hyp,x,xs) % 4) compute the derivatives w.r.t. to hyperparameter i i = 1; feval(cov{:},hyp,x,[],i) gpml-matlab-v3.5-2014-12-08/doc/usagePrior.m000664 000765 000024 00000006614 12526350575 020147 0ustar00carlstaff000000 000000 % demonstrate usage of prior distributions % % See also priorDistributions.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-05-18. % File automatically generated using noweb. clear all, close all % 1) specify some priors % a) univariate priors mu = 1.0; s2 = 0.01^2; nu = 3; pg = {@priorGauss,mu,s2}; % Gaussian prior pl = {'priorLaplace',mu,s2}; % Laplace prior pt = {@priorT,mu,s2,nu}; % Student's t prior p1 = {@priorSmoothBox1,0,3,15}; % smooth box constraints lin decay p2 = {@priorSmoothBox2,0,2,15}; % smooth box constraints qua decay pd = {'priorDelta'}; % fix value of prior exclude from optimisation pc = {@priorClamped}; % equivalent to above lam = 1.05; k = 2.5; pw = {@priorWeibull,lam,k}; % Weibull prior % b) meta priors pmx = {@priorMix,[0.5,0.5],{pg,pl}}; % mixture of two priors g = @exp; dg = @exp; ig = @log; ptr = {@priorTransform,g,dg,ig,pg}; % Gaussian in the exp domain % c) multivariate priors m = [1;2]; V = [2,1;1,2]; pG = {@priorGaussMulti,m,V}; % 2d Gaussian prior pD = {'priorDeltaMulti'}; % fix value of prior exclude from optim pC = {@priorClampedMulti}; % equivalent to above % 2) evaluation % pri = pt; hp = randn(1,3); % pri = pmx; hp = randn(1,3); % pri = ptr; hp = randn(1,3); pri = pG; hp = randn(2,3); % a) draw a sample from the prior feval(pri{:}) % b) evaluate prior and derivative if requires [lp,dlp] = feval(pri{:},hp) % 3) comprehensive example x = (0:0.1:10)'; y = 2*x+randn(size(x)); % generate training data mean = {@meanSum,{@meanConst,@meanLinear}}; % specify mean function cov = {@covSEiso}; lik = {@likGauss}; % specify covariance and lik hyp.cov = [log(1);log(1.2)]; hyp.lik = log(0.9); hyp.mean = [2;3]; par = {mean,cov,lik,x,y}; mfun = @minimize; % input for GP function % a) plain marginal likelihood optimisation (maximum likelihood) im = @infExact; % inference method hyp_plain = feval(mfun, hyp, @gp, -10, im, par{:}); % optimise % b) regularised optimisation (maximum a posteriori) with 1d priors prior.mean = {pg;pc}; % Gaussian prior for first, clamp second par prior.cov = {p1;[]}; % box prior for first, nothing for second par im = {@infPrior,@infExact,prior}; % inference method hyp_p1 = feval(mfun, hyp, @gp, -10, im, par{:}); % optimise % c) regularised optimisation (maximum a posteriori) with Nd priors prior = []; % clear the structure % multivariate Student's t prior on the first and second mean hyper prior.multi{1} = {@priorTMulti,[mu;mu],diag([s2,s2]),nu,... struct('mean',[1,2])}; % use hyper struct % Equivalent shortcut (same mu and s2 for all dimensions) prior.multi{1} = {@priorTMulti,mu,s2,nu,struct('mean',[1,2])}; % multivariate Gaussian prior jointly on 1st and 3rd hyper prior.multi{2} = {@priorGaussMulti,[mu;mu],diag([s2,s2]),... [1,3]}; % use unwrapped hyper vector % Equivalent shortcut (same mu and s2 for all dimensions) prior.multi{2} = {@priorGaussMulti,mu,s2,[1,3]}; im = {@infPrior,@infExact,prior}; % inference method hyp_pN = feval(mfun, hyp, @gp, -10, im, par{:}); % optimise [unwrap(hyp), unwrap(hyp_plain), unwrap(hyp_p1), unwrap(hyp_pN)] gpml-matlab-v3.5-2014-12-08/doc/README000644 000765 000024 00000037665 12441371762 016536 0ustar00carlstaff000000 000000 GAUSSIAN PROCESS REGRESSION AND CLASSIFICATION Toolbox version 3.5 for GNU Octave 3.2.x and Matlab 7.x Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2014-12-08. 0) HOW TO READ ============== If you want to get started without further delay, then read section 1) below and jump right to the examples in doc/index.html. 1) ABOUT THESE PROGRAMS ======================= This collection of matlab programs implements and demonstrates some of the algorithms described in a) the book by Rasmussen and Williams: "Gaussian Processes for Machine Learning", the MIT Press 2006, in b) the article by Nickisch and Rasmussen: "Approximations for Binary Gaussian Process Classification", JMLR 2008, in c) the article by Candela and Rasmussen: "A Unifying View of Sparse Approximate Gaussian Process Regression", JMLR 2005, in d) the paper by Murray, Adams and Mackay: "Elliptical slice sampling", AISTATS 2010, in e) the report by Neal: "Annealed Importance Sampling", UToronto 1998, in f) the paper by Naish-Guzman and Holden: "The Generalized FITC Approximation", NIPS, 2007, in g) the paper by Duvenaud, Nickisch and Rasmussen: "Additive Gaussian Processes", NIPS, 2011, in h) the paper by Wilson and Adams: "Gaussian Process Kernels for Pattern Discovery and Extrapolation", ICML, 2013, and in i) the paper Snelson, Rasmussen and Ghahramani: "Warped Gaussian Processes", NIPS, 2003. There are 7 code subdirectories: cov, doc, inf, lik, mean, prior and util. cov/ contains covariance functions cov*.m => see covFunctions.m doc/ contains an index.html file providing documentation. This information is also available from http://www.GaussianProcess.org/gpml/code. Usage of mean, cov, classification and regression is demonstrated in usage*.m. Further details can be found in the developer documentation manual.pdf. inf/ contains the inference algorithms inf*.m => see infMethods.m lik/ contains the likelihood functions lik*.m => see likFunctions.m mean/ contains the mean functions mean*.m => see meanFunctions.m prior/ contains the hyperparameter prior distributions prior*.m => see priorDistributions.m util/ contains optimisation routines, backward compatibility programs and small auxiliary scripts Before running the demos or any other part of the code, you should execute startup.m to add all necessary directories to your path. In Octave, you can also use the .octaverc file. 2) ABOUT MEX FILES ================== Some of the programs make use of the MEX facility in matlab for more efficient implementation. However, if you don't know about how to compile MEX files, you do not need to worry about this -- the code runs anyway. If you do compile the MEX files, this is automatically detected, and the program will run more efficiently. If you do not have a compiler installed on your system, you might want to consult [3]. Two components of the toolbox can be accelerated by MEX. First, there is an auxiliary functions solve_chol.m having a MEX equivalent. This can be compiled by either executing util/make.m from the Matlab/Octave command line. In addition to that, we provide a Makefile for Matlab in util/ that you can run by simply typing make in your shell after having supplied your Matlab path and your operating system. We recommend the make.m script because it works both under Matlab and Octave. Second, in order to use the L-BFGS minimiser for hyperparameter optimisation as an equivalent to minimize.m, you have to compile Peter Carbonetto's "Matlab interface for L-BFGS-B" [1] which was extended to also work for Octave by José Vallet. The challenge here is the Fortran 77 code. We provide a Makefile suitable for Linux 32/64 bit and Mac whenever you have 1) g77 or 2) gfortran properly set up. The latter is prefered on current Ubuntu systems. Under Ubuntu, you can achieve this by installing the packages fort77 and gfortran, respectively. Note that the code is GNU licensed. Compilation is done by editing util/lbfgsb/Makefile. In any case, you need to provide $MATLAB_HOME which can be found by the commands 'locate matlab' or 'find / -name "matlab"'. You can choose between two compilation modes: a) without mex utility by Matlab [default] provide $MEX_SUFFIX and $MATLAB_LIB, then type 'make nomex' b) using the mex utility by Matlab provide $MEX, then type 'make mex' c) compilation for Octave provide Octave variables and type 'make oct' In Ubuntu 10.04 LTS, the libg2c library needed for both 1)+a) and 1)+b) is not included per default. If 'ls /usr/lib/libg2c.*' does not list anything this is the case on your machine. You then whant to install the packages gcc-3.4-base and libg2c0 e.g. from http://packages.ubuntu.com/hardy/gcc-3.4-base and http://packages.ubuntu.com/hardy/libg2c0. After installation, you have to create a symbolic link by 'cd /usr/lib' and 'ln -s libg2c.so.0 libg2c.so'. There were reports of problems running the mex files on recent 64bit Ubuntu since the libgfortran.3.0.0 that comes with matlab ($(MATLAB_HOME)/sys/os/glnxa64/libgfortran.so.3.0.0) is not the same as the libgfortran.3.0.0 that comes with the ubuntu package libgfortran3 and that is used when linking. So running the mex file results in a version mismatch error. The issue can be solved by changing the name of to the matlab file and linking to the ubuntu file instead: in $(MATLAB_HOME)/sys/os/glnxa64 run 'sudo mv libgfortran.3.0.0 libgfortran.3.0.0_matlab' 'sudo ln -s /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0 libgfortran.3.0.0' Thanks to Giampiero Salvi for posting the workaround. [1] https://github.com/pcarbo/lbfgsb-matlab [2] http://www.mathworks.com/support/compilers/R2010a 3) CURRENT VERSION ================== The current version of the programs is 3.5. Previous versions of the code are available at http://www.gaussianprocess.org/gpml/code/oldcode.html. 4) DIFFERENCES TO PREVIOUS VERSIONS =================================== NEW in version 3.5, 2014-12-08 ------------------------------ - mechanism for specifying hyperparameter priors (together with Roman Garnett and José Vallet) - new inference method inf/infGrid allowing efficient inference for data defined on a Cartesian grid (together with Andrew Wilson) - new mean/cov functions for preference learning: meanPref/covPref - new mean/cov functions for non-vectorial data: meanDiscrete/covDiscrete - new piecewise constant nearest neighbor mean function: meanNN - new mean functions being predictions from GPs: meanGP and meanGPexact - new covariance function for standard additive noise: covEye - new covariance function for factor analysis: covSEfact - new covariance function with varying length scale : covSEvlen - make covScale more general to scaling with a function instead of a scalar - bugfix in covGabor* and covSM (due to Andrew Gordon Wilson) - bugfix in lik/likBeta.m (suggested by Dali Wei) - bugfix in solve_chol.c (due to Todd Small) - bugfix in FITC inference mode (due to Joris Mooij) where the wrong mode for post.L was chosen when using infFITC and post.L being a diagonal matrix - bugfix in infVB marginal likelihood for likLogistic with nonzero mean function (reported by James Lloyd) - removed the combination likErf/infVB as it yields a bad posterior approximation and lacks theoretical justification - Matlab and Octave compilation for L-BFGS-B v2.4 and the more recent L-BFGS-B v3.0 (contributed by José Vallet) - smaller bugfixes in gp.m (due to Joris Mooij and Ernst Kloppenburg) - bugfix in lik/likBeta.m (due to Dali Wei) - updated use of logphi in lik/likErf - bugfix in util/solve_chol.c where a typing issue occured on OS X (due to Todd Small) - bugfix due to Bjørn Sand Jensen noticing that cov_deriv_sq_dist.m was missing in the distribution - bugfix in infFITC_EP for ttau->inf (suggested by Ryan Turner) NEW in version 3.4, 2013-11-11 ------------------------------ - new likelihood function for warped GPs: likGaussWarp - new exponential likelihood function: likExp - new Weibull likelihood function: likWeibull - new Gumbel likelihood function for extremal value regression: likGumbel - new polynomial mean function: meanPoly - included derivatives w.r.t. inducing points xu in infFITC, infFITC_Laplace, infFITC_EP - allow for zero noise variance for infExact NEW in version 3.3, 2013-10-19 ------------------------------ - new variational inference algorithm based on direct KL minimisation: infKL - improved inf/infVB double loop scheme so that only very few likelihood properties are required; infVB is now internally a sequence of infLaplace runs - improved inf/infLaplace to be more generic so that optimisers other than scaled Newton can be used - improved inf/infEP so that the internal variables (mu,Sigma) now represent the current posterior approximation - added three new generalised linear model likelihoods: gamma, beta, inverse Gaussian - new covariance functions: spectral mixture covSM, covGaboriso and covGaborard - new meta covariance function to turn a stationary covariance into a periodic covariance function covPERard, covPERiso - new periodic covariance function with zero DC component and correct scaling covPeriodicNoDC New likelihood functions in lik/: - likGamma.m, likBeta.m, likInvGauss.m New inference functions in inf/: - infKL.m New covariance functions in cov/: - covSM.m, covGaboriso.m, covGaborard.m (contributed by Andrew Gordon Wilson) - covPPard.m, covMaternard.m, covLINiso.m - covCos.m, covPeriodicNoDC.m (contributed by James Robert Lloyd) - covPERiso.m, covPERard.m NEW in version 3.2, 2013-01-15 ------------------------------ - added inf/infFITC_Laplace.m and inf/infFITC_EP.m for large scale approximate inference - added inf/infMCMC.m to allow for Markov Chain Monte Carlo sampling and Annealed Importance Sampling; we currently support two samplers: Elliptical Slice Sampling (ESS) and Hybrid Monte Carlo (HMC) - improved accuracy of the digamma function lik/likT.m/dloggamma from three to ten digits precision (thanks to Edward Challis) - added gfortran support to util/lbfgsb/Makefile (thanks to Ernst Kloppenburg) - fixed a stupid bug in cov/covFITC.m (thanks to Krzysztof Chalupka) - fixed a severe stability bug in the derivatives of inf/infFITC.m (thanks to Andrew McHutchon) - changed scalars in util/solve_chol.c to long to provide 64bit compatibility (thanks to Ernst Kloppenburg) - added infLOO to allow pseudo-likelihood optimisation instead of the marginal likelihood - fixed a performance issue in util/sq_dist.m (thanks to Krzysztof Chalupka) - fixed a bug causing numerical problems in infFITC and covFITC whenever Kuu is close to singular (thanks to Joris M. Mooij) - fixed an issue in covNoise (thanks to Daniel Marthaler) - fixed implicit derivative w.r.t. likelihood in infLaplace* - added two more likelihood functions likMix.m for mixtures and likUni for classification noise - slightly changed the interface for the likelihood functions - added likPoisson for Poisson regression of count data NEW in version 3.1, 2010-10-10 ------------------------------ - following a suggestion by Ed Snelson we now support FITC regression - cov/covFITC.m and inf/infFITC.m have been added with Ed Snelson's help - the covariance interface was slightly changed to make that possible - changed parametrisation in cov/covPeriodic.m (thanks to Philipp Hennig) NEW in version 3.0, 2010-07-23 ------------------------------ A major code reorganisation effort did take place in the current release. First, classification and regression are now done by a single file gp.m which is completely generic in the likelihood. The previous regression program gpr.m corresponds to gp.m with Gaussian likelihood. Several other likelihoods for robust regresssion were added. Further, the code now supports mean functions with a similar specification mechanism as already used by the covariance functions. Previous implementations correspond to using meanZero.m. We merged the covariance functions covMatern3iso.m and covMatern5.iso into a single covMaterniso.m and added a covariance function for additive functions as well as the possibility to use only certain components of the data. Finally, we included covPPiso.m a piecewise polynomial covariance function with compact support. New likelihood functions in lik/: likGauss.m, likLaplace.m, likLogistic.m and likT.m New mean functions in mean/: meanConst.m, meanLinear.m, meanPow.m, meanProd.m, meanSum.m and meanZero.m New covariance functions in cov/: covADD.m, covMask.m, covPPiso.m The gprSRPP.m function which previously provided "Subset of Regressors" and the "Projected Process" approximation has now been removed. NEW in version 2.1, 2007-07-25 ------------------------------ covConst.m: fixed a bug which caused an error in the derivative of the log marginal likelihood for certain combinations of covariance functions and approximation methods. (Thanks to Antonio Eleuteri for reporting the problem) gauher.m: added the function "gauher.m" which was mistakenly missing from the previous release. This caused an error for certain combinations of approximation method and likelihood function. logistic.m: modified the approximation of moments calculation to use a mixture of cumulative Gaussian, rather than Gauss-Hermite quadrature, as the former turns out to be more accurate. NEW in version 2.0, 2007-06-25 ------------------------------ Some code restructuring has taken place for the classification code to make it more modular, to facilitate addition of new likelihood functions and approximations methods. Now, all classification is done using the binaryGP function, which (among other things) takes an approximation method and a likelihood function as an arguments. Thus, binaryGP replaces both binaryEPGP and binaryLapaceGP, although wrapper functions are still provided for backward compatibility. This gives added flexibility: now EP can also be used wth the logistic likelihood function (implemented using Gauss-Hermite quadrature). approxEP.m: New file, containing the Expectation Propagation approximation method, which was previously contained in binaryEPGP.m approxLA.m: New file, containing Laplaces approximation method, which was previously contained in binaryLaplace.m approximations.m: New file, help for the approximation methods. binaryEPGP.m: This file has been replaced by a wrapper (for backward compatibility) which calls the more general binaryGP function. binaryGP.m: New general function to do binary classification. binaryLaplaceGP.m: This file has been replaced by a wrapper (for backward compatibility) which calls the more general binaryGP function. covMatern3iso.m, covMatern5iso.m, covNNone.m, covRQard.m, covRQiso.m, cosSEard, covSEiso: now check more carefully, that persistent variables have the correct sizes, and some variable names have been modified. cumGauss.m: New file, containing code for the cumulative Gaussian likelihood function likelihoods.m: New file, help for likelihood functions logistic.m: New file, logistic likelihood NEW in version 1.3, 2006-09-08 ------------------------------ covRQard.m: bugfix: replaced x with x' and z with z' in line 36 covRQiso.m: bugfix: replaced x with x' and z with z' in line 28 minimize.m: correction: replaced "error()" with "error('')", and made a few cosmetic changes binaryEPGP.m: added the line "lml = -n*log(2);" in line 77. This change should be largely inconsequential, but occationally may save things when the covariance matrix is exceptionally badly conditioned. NEW in version 1.2, 2006-05-10 ------------------------------ added the "erfint" function to "binaryLaplaceGP.m". The erfint function was missing by mistake, preventing the use of the "logistic" likelihood. NEW in version 1.1, 2006-04-12 ------------------------------ added files: "covProd.m" and "covPeriodic.m" changes: "covSEiso.m" was changed slightly to avoid the use of persistent variables NEW in version 1.0, 2006-03-29 ------------------------------ initial version shipped with the book gpml-matlab-v3.5-2014-12-08/doc/checkmark.png000644 000765 000024 00000000306 11415166010 020256 0ustar00carlstaff000000 000000 PNG  IHDRa*PLTExxxlll```TTTHHH<<<$$$ tRNS@fJIDATc````d6 N`8,~e RS!Ke0O P`ʳ&] NIENDB`gpml-matlab-v3.5-2014-12-08/doc/demoRegression.m000644 000765 000024 00000011772 12075731017 021004 0ustar00carlstaff000000 000000 disp('See http://www.gaussianprocess.org/gpml/code/matlab/doc/ for details.') disp('Hit any key to continue...'); pause disp(' '); disp('clear all, close all') clear all, close all write_fig = 0; disp(' ') disp('meanfunc = {@meanSum, {@meanLinear, @meanConst}}; hyp.mean = [0.5; 1];') meanfunc = {@meanSum, {@meanLinear, @meanConst}}; hyp.mean = [0.5; 1]; disp('covfunc = {@covMaterniso, 3}; ell = 1/4; sf = 1; hyp.cov = log([ell; sf]);') covfunc = {@covMaterniso, 3}; ell = 1/4; sf = 1; hyp.cov = log([ell; sf]); disp('likfunc = @likGauss; sn = 0.1; hyp.lik = log(sn);') likfunc = @likGauss; sn = 0.1; hyp.lik = log(sn); disp(' ') disp('n = 20;') n = 20; disp('x = gpml_randn(0.3, n, 1);') x = gpml_randn(0.3, n, 1); disp('K = feval(covfunc{:}, hyp.cov, x);') K = feval(covfunc{:}, hyp.cov, x); disp('mu = feval(meanfunc{:}, hyp.mean, x);') mu = feval(meanfunc{:}, hyp.mean, x); disp('y = chol(K)''*gpml_randn(0.15, n, 1) + mu + exp(hyp.lik)*gpml_randn(0.2, n, 1);') y = chol(K)'*gpml_randn(0.15, n, 1) + mu + exp(hyp.lik)*gpml_randn(0.2, n, 1); figure(1) set(gca, 'FontSize', 24) disp(' '); disp('plot(x, y, ''+'')') plot(x, y, '+', 'MarkerSize', 12) axis([-1.9 1.9 -0.9 3.9]) grid on xlabel('input, x') ylabel('output, y') if write_fig, print -depsc f1.eps; end disp(' '); disp('Hit any key to continue...'); pause disp(' ') disp('nlml = gp(hyp, @infExact, meanfunc, covfunc, likfunc, x, y)') nlml = gp(hyp, @infExact, meanfunc, covfunc, likfunc, x, y) disp(' ') disp('z = linspace(-1.9, 1.9, 101)'';') z = linspace(-1.9, 1.9, 101)'; disp('[m s2] = gp(hyp, @infExact, meanfunc, covfunc, likfunc, x, y, z);') [m s2] = gp(hyp, @infExact, meanfunc, covfunc, likfunc, x, y, z); figure(2) set(gca, 'FontSize', 24) disp(' ') disp('f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)];') f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)]; disp('fill([z; flipdim(z,1)], f, [7 7 7]/8);') fill([z; flipdim(z,1)], f, [7 7 7]/8); disp('hold on; plot(z, m); plot(x, y, ''+'')') hold on; plot(z, m, 'LineWidth', 2); plot(x, y, '+', 'MarkerSize', 12) axis([-1.9 1.9 -0.9 3.9]) grid on xlabel('input, x') ylabel('output, y') if write_fig, print -depsc f2.eps; end disp(' '); disp('Hit any key to continue...'); pause disp(' ') disp('covfunc = @covSEiso; hyp2.cov = [0; 0]; hyp2.lik = log(0.1);') covfunc = @covSEiso; hyp2.cov = [0; 0]; hyp2.lik = log(0.1); disp('hyp2 = minimize(hyp2, @gp, -100, @infExact, [], covfunc, likfunc, x, y)') hyp2 = minimize(hyp2, @gp, -100, @infExact, [], covfunc, likfunc, x, y); disp(' ') disp('exp(hyp2.lik)') exp(hyp2.lik) disp('nlml2 = gp(hyp2, @infExact, [], covfunc, likfunc, x, y)') nlml2 = gp(hyp2, @infExact, [], covfunc, likfunc, x, y) disp('[m s2] = gp(hyp2, @infExact, [], covfunc, likfunc, x, y, z);') [m s2] = gp(hyp2, @infExact, [], covfunc, likfunc, x, y, z); disp(' ') figure(3) set(gca, 'FontSize', 24) f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)]; disp('fill([z; flipdim(z,1)], f, [7 7 7]/8)'); fill([z; flipdim(z,1)], f, [7 7 7]/8) disp('hold on; plot(z, m); plot(x, y, ''+'')'); hold on; plot(z, m, 'LineWidth', 2); plot(x, y, '+', 'MarkerSize', 12) grid on xlabel('input, x') ylabel('output, y') axis([-1.9 1.9 -0.9 3.9]) if write_fig, print -depsc f3.eps; end disp(' '); disp('Hit any key to continue...'); pause disp(' ') disp('hyp.cov = [0; 0]; hyp.mean = [0; 0]; hyp.lik = log(0.1);') hyp.cov = [0; 0]; hyp.mean = [0; 0]; hyp.lik = log(0.1); disp('hyp = minimize(hyp, @gp, -100, @infExact, meanfunc, covfunc, likfunc, x, y);') hyp = minimize(hyp, @gp, -100, @infExact, meanfunc, covfunc, likfunc, x, y); disp('[m s2] = gp(hyp, @infExact, meanfunc, covfunc, likfunc, x, y, z);') [m s2] = gp(hyp, @infExact, meanfunc, covfunc, likfunc, x, y, z); figure(4) set(gca, 'FontSize', 24) disp(' ') disp('f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)];') f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)]; disp('fill([z; flipdim(z,1)], f, [7 7 7]/8)') fill([z; flipdim(z,1)], f, [7 7 7]/8) disp('hold on; plot(z, m); plot(x, y, ''+'');') hold on; plot(z, m, 'LineWidth', 2); plot(x, y, '+', 'MarkerSize', 12) grid on xlabel('input, x') ylabel('output, y') axis([-1.9 1.9 -0.9 3.9]) if write_fig, print -depsc f4.eps; end disp(' '); disp('Hit any key to continue...'); pause disp('large scale regression using the FITC approximation') disp('nu = fix(n/2); u = linspace(-1.3,1.3,nu)'';') nu = fix(n/2); u = linspace(-1.3,1.3,nu)'; disp('covfuncF = {@covFITC, {covfunc}, u};') covfuncF = {@covFITC, {covfunc}, u}; disp('[mF s2F] = gp(hyp, @infFITC, meanfunc, covfuncF, likfunc, x, y, z);') [mF s2F] = gp(hyp, @infFITC, meanfunc, covfuncF, likfunc, x, y, z); figure(5) set(gca, 'FontSize', 24) disp(' ') disp('f = [mF+2*sqrt(s2F); flipdim(mF-2*sqrt(s2F),1)];') f = [mF+2*sqrt(s2F); flipdim(mF-2*sqrt(s2F),1)]; disp('fill([z; flipdim(z,1)], f, [7 7 7]/8)') fill([z; flipdim(z,1)], f, [7 7 7]/8) disp('hold on; plot(z, mF); plot(x, y, ''+'');') hold on; plot(z, mF, 'LineWidth', 2); plot(x, y, '+', 'MarkerSize', 12) disp('plot(u,1,''o'')') plot(u,1,'ko', 'MarkerSize', 12) grid on xlabel('input, x') ylabel('output, y') axis([-1.9 1.9 -0.9 3.9]) if write_fig, print -depsc f5.eps; end disp(' ')gpml-matlab-v3.5-2014-12-08/doc/usageClassification.m000644 000765 000024 00000005144 12240504511 021762 0ustar00carlstaff000000 000000 % demonstrate usage of classification % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2013-10-16. clear all, close all %% SAY WHICH CODE WE WISH TO EXERCISE id = [1,1]; % use Gauss/Exact id = [1,2; 2,2; 3,2]; % compare Laplace id = [1,3; 2,3; 3,3]; % study EP id = [1,5; 2,5]; % look into KL (takes quite a while) id = [1,4; 2,4; 3,4]; % deal with VB seed = 943; randn('seed',seed), rand('seed',seed) ntr = 50; nte = 1e4; % number of training and test points xtr = 10*sort(rand(ntr,1)); % sample dataset p = @(x) 1./(1+exp(-5*sin(x))); % "true" underlying probability ytr = 2*(p(xtr)>rand(ntr,1))-1; % draw labels +1/-1 i = randperm(ntr); nout = 3; % add outliers ytr(i(1:nout)) = -ytr(i(1:nout)); xte = linspace(0,10,1e4)'; % support, we test our function on cov = {@covSEiso}; sf = 1; ell = 0.7; % setup the GP hyp0.cov = log([ell;sf]); mean = {@meanZero}; % m(x) = 0 hyp0.mean = []; lik_list = {'likGauss','likErf','likLogistic'}; % allowable likelihoods inf_list = {'infExact','infLaplace','infEP','infVB','infKL'}; % inference algs Ncg = 50; % number of conjugate gradient steps sdscale = 0.5; % how many sd wide should the error bars become? col = {'k',[.8,0,0],[0,.5,0],'b',[0,.75,.75],[.7,0,.5]}; % colors ymu{1} = 2*p(xte)-1; ys2{1} = 0; for i=1:size(id,1) lik = lik_list(id(i,1)); % setup the likelihood if strcmp(lik,'likGauss') sn = .2; hyp0.lik = log(sn); else hyp0.lik = []; end inf = inf_list{id(i,2)}; fprintf('OPT: %s/%s\n',lik_list{id(i,1)},inf_list{id(i,2)}) hyp = minimize(hyp0,'gp', -Ncg, inf, mean, cov, lik, xtr, ytr); % opt hypers [ymu{i+1}, ys2{i+1}] = gp(hyp, inf, mean, cov, lik, xtr, ytr, xte); % predict [nlZ(i+1)] = gp(hyp, inf, mean, cov, lik, xtr, ytr); end figure, hold on for i=1:size(id,1)+1 plot(xte,ymu{i},'Color',col{i},'LineWidth',2) if i==1 leg = {'function'}; else leg{end+1} = sprintf('%s/%s -lZ=%1.2f',... lik_list{id(i-1,1)},inf_list{id(i-1,2)},nlZ(i)); end end for i=1:size(id,1)+1 ysd = sdscale*sqrt(ys2{i}); fill([xte;flipud(xte)],[ymu{i}+ysd;flipud(ymu{i}-ysd)],... col{i},'EdgeColor',col{i},'FaceAlpha',0.1,'EdgeAlpha',0.3); end for i=1:size(id,1)+1, plot(xte,ymu{i},'Color',col{i},'LineWidth',2), end plot(xtr,ytr,'k+'), plot(xtr,ytr,'ko'), legend(leg) gpml-matlab-v3.5-2014-12-08/doc/f9.png000664 000765 000024 00000053646 12440641245 016674 0ustar00carlstaff000000 000000 PNG  IHDR~͕ pHYs B(xtIME 5()"tEXtCreation Time06-Dec-2014 18:53:40ƴkb$tEXtSoftwareMATLAB, The Mathworks, Inc.R IDATxm8s\+\,\A  p+\A Wܮ @"9^t/(Q]f`9{&)X`a`I &)#Ƙ ;=lẁN )0I5Rp8g-]#\muAd~f[N*0d{w}%'>uЧt${!dApan=FGKNWQ?$Ft,@ ΀4Ѻ9I KRgX=V9}Lo@5kUHDdNh::Op4PI^\?)pOI>]gD7:b7{){$7GU_::0)̸XOVuX!"맕x,x)PI[IH `$ngASaxsC&iI` !?`)v+It 0˧q!LO$ ;n#s`do\>%/up}SdA!CtI#`e-(o M 0z,,M,m^7ы`IZTҏܿu`M$nuGUu7~. TҊ&Ws$ǒ6~3׽\ꕡ_]I2ͯhl9; |5t7l/(OAɯo> "_5O/xC }}N ș*\/Fܜ5YQ~W/q5W׹E-Xd6z!"=Jqߗ$}qhNgb{hIZgזۯJK e:Myt}KzVΧqDmޒ m,Kx/gcCԍ(ez-VIN$0:dۈ08|{yF0pJ~I)],*yow{lEBjGQWa?EJg\f\i(ΔH_Vg :)83D: ?0Y+M$Ε&Jr3xL~7}Y VM(BלmLD0$Z.1{p b͚,ol,wvL5!kV߯wpj_:1~6M8JmX9[ZgGvoZX'"PiL0l$3gl}~3_EPjW'ZP%s)O(?O=BժPb9-\O/L<{`΁D> ^B.Խ3g.β>Z$j/PGX-;ʢ:>A-iP| s2Z*yσOy3K,]nfԃ9I W&Gq?iÝf]-ߔ_3chIWCiz{F cީ[i:`.D"p}P%(E,e9R9 ]%K,^KCn,vov _/[nMs֠(oz0Ivq4GE~~gU( ?p%EЎrR+=uee[9>0K6)"f-tzvE:u[&_T~ \-u;C($+&9 l;|ȥUT\Pjċ_ImjX?BKQNm#eRnQ0L|جz%68/ܝU 7"Psq`d&Uscz}W~V:B2G?9US٧=& 9 0 _;))Xy| X$x/j|W)z R`$8x/x)*^d-+7ؾ75h}1rPK =ힽ:5ƌWY~e":ZF3cL)SocE}Şֶ`r_˓^B?  tk1(?Kv7nǣ:Fٓ>h_?Aw||F OI\Rܬٓʠ/޻) ܯμ=*xYNјW!-*:67Wng`)vů!^y2h柭?0XK0p?WPLg3X{Fn Q"ԄԖ̓.63)-*i>*dn2]{;4#xv\>/lyLIz9j)j e|z=Ò2\&"v0 /޻)lv|dQ KWd!63^{d&԰*i\Z}H5߳=Dl-cNޔC%ftkj7ȥr7ĤsZHi࣡ 7uK9t(EKڽJMcc*}86q0ujhjm6faړ|L(Xv!}w+EG%d[L9vl$].u$?x+7>>˹o({Ik'K'*QS!mV˥>7~PsEmJfֆk,= ĩKTK(Ba4+Ѿ?%<8(j7Ѣ˯;Tأs%h!k=bw~0 4s3ӛ.ؼ=~/YgﾴSc 3jd%4ltPv-͝H%]v9ƖL-N0Pepsfޔ:3Ǐ̢'ŭyi|j=7qI٦y0Eޤ/*m\x=d_wSPIqwRv1 e *Լ/OgP}ͯ..NHcLEZJfV' 袃jT.o.IƜ6f%R!aVgESNu}RN R+zJ,vJt[m/X1o^}z]~.sa?H.~6+W2;E|}y SOrPdP? X雹K,ǽOŝ!ho=Z9J6> Aio#t;)CSd[ -z`@%m%/c C(|.K.7w0_]bT%_q_U ^WJ.xE *|56IFnNv}i{٣ӛ,u--C[C;=E)擩!4φS3|&v}Oor;Lo*9S:N[$=mh~5ӫ*&w nn1ٝ{7ZRG˫3F+)1Fb?M2]vjKMg~3p'%ϑ+# n~ޛ_ "2 أ乼UG\frd3qe@\wah Mג?;tӛ { ;ЩE[lѮu|q6M"y*$Uю#]FB*Ӣ'C=k@Q"IN^"{ Ṽot-Lտ `YN¼7+wfj!9!+)1V*T JAOpKF^(} ;&${e~.94ӛxzgaSEf"GP-ܬk_zȫkj.qcKJZ-þN1\8WT6C ;5+ ֒?;$ݜ}+ g/%{Ϲ c\M2K @j1~/]yw1ٕ^ +<0D|tE&MYS0ULyQJɣC 96ˍߟUP:΋ I=ZTf9Rmӛ:B{LRL= MKbY0gA|\2'a Ҧ,]Fkbd߂#. ͭ?EܝԿtrk~WC T!Iiﻃx)?ᾙM3"Pw`Ӏ`aEUvCKt8*kߵP;,R՜r_7< 5(t$JZ)./!K38NǤ;4]^7=Nm+d: ]M`i0~6h`##ԺU|f-97"mDPdo ޘ>BN]K# ew1جjѬm{`aPC*W@(@~EyJVR3޴[-b>491L|Nȵ"Mdy5KW{Ա{v2 :y{ ;F oҘm6:NK:ݱg^AO_u}~ ?_/4?AKMG)Ot֎gͯgm|o+ֿ3hO{%;0޽1馒$cLRILT}M!w $Vj7o"Rŷu-}]o'9{H. wd \\q%*46+.~lPjm]Vo] "ߧj1}jXrTx,jrS8@|GTD{8qGM{tSI{D+)2ma_TpW2E]hS7= mJۮ6b5nM4LU\5(Fj$'K ټ8g^UX,v(gTI5eJsR>+o/Hؤi29E.G;֪DJ=b&95bz ~:\DzI/bښ{riPINkt\C/jh&=61DnN^R- [e&-Ôۍ*Stz;3jAq>3'nE.]M=Qͱ~<#wScF_Yb߄1 s&d G5&NΝ{Mp ϹhVlDXIFx܈# \Ɗ.nIJ"aOnj߮^[l̮!؛g*=^}of=4f*T-1о'kwG 3Fsȋ\O V-}3B3 1x9Y0(.[ 4J<̍|4B {s/y(xc`KwCVx27a/~L 2d|27\ A~.F':~$U¹dwù}NMl/_kPK_I5{((ۨ:Zn}Hy-SjIk׼7h"M.RI"8n;{!UyȓO_|-FtyZict|5O;1X\.>xݥ#DW'1nO5|5dHeO_n0Wd^h< ?O/lao u3{_U&m{U2¶[rݜנ6hCh_̶ JןqMv8UrF?"tSI`A0I)N s0IxI:)r7EN~:|Y~~_3T#,Zͥ?(Hj%}0w ߒRA}rʵ]͠k8&x|I'\4;8>q- 2o3SLohV|c~ K sxkGѬ. Xb `Ƽ]>qBgJPjӛ4iEsO.ZYTwէ͞42eudQXIDFG9G[=8 IDATɤ:ṠTqΊJIhLR=ޛ5Fqef(8rtSLojetu@GX`n)9)4ЎJ t&*Ǘ P. KL?5RXY%՟o[a_|&`s=>W 48M@9K̇J28;Y rp(w~.>;.m{suTQ놑meN*Rtk[p?+cI:9=u-s>TTSH2إLΣuG6iRks'e).[,p%F4s1^"?X cQŊ+?N沎bɊ&t9bq KL7p7bYLs>ZStr;%f\KRFty*~h_#NQrR9, ր< *QdY{TEʰΗ4G}IPKzeca_k.+ǗTCE# ͚g%pшYq(r$A3'_#f=Ke5Oa: KMޢr j87}ٺy6t-Q %l`\祫VQPѫ!PIg(N4 f8H8Y-T0Mo93rjzSv]]n@j!qʨ^WʜrF,$Y";ݔ*)h X1(ȱ},wa*l LJ *4TVZVSLU*:--3sK$*GKFA:2hO7)>)*%Ʃ\*BܳK?3MMqfmiD8`STx27KW4spc2{F@U-ϙ[t6Q T7 %!ZjMONŶL9..-9˯᳦et5v|/Xq? :gK1eXW .ɥ^^F%>|kS#YiRV;5?PIA3W= 3Fy j.gsI[4`plPrpGs+T2s̅ 1x/ȍ61"P#K;-MriEߑ-6~ dIܡ1? o~Q b ɜMvT[:qbA_lvDhn_R FCp{sn S4Ecם#Ӎ9w_Xpb bP@ǯY6h}ϛKZE"'0p?9 8;x6O8 ٺ=zc¦=jJ:9vvTq&y$ESQ mn)UyOl]֓9Σ&9ӛӛZgh`na'Mo1FDMA$S4Pn SN*&TR 73޴1KW v@%8dNRWT^ODzMVL6R>(9 "m*U8ii_AMglz 3h\ɥ+%ʧ%g /wJM6fێ;E,Z*fPdM ؑ=B4þ獼cŮ*lG:~#E)+ rklvG}弚˿ :8q/o þS}г hKZ*vܨŗ Tv`\xo;~/]pNlkP,TCBY̅?ҥ|tjm3+E͊(ٗ4[5vtCJqT%}4v6nD."S5/i)PIA+Y{6}+Bj|~-]pN4V]bZ}0vO&Z6B7MYK(*Ϩ$k _4&OS% (7ML؎^cuh6#$;EuYe>좃ʅȣ;mCM̀%Iڊ݃`Gʳ(|Iq߱Y /i2E[`_^-e>l/<2ڢNW2Pl~.]y7 gt-Q %{s`t|4~?x㓹Ś=k?\/O.󁎯?\ f+I+^?5W~jpcͭoo``I`;g5:ޘ'?y32^g?ʼWo`d{a͏SdɚZc S7O`5w#\žq=uo$pAw _`%ju\?!.attSgڠ_x265 EݾmtV!O &qA|bS=A}ݳa.\@Wx9 AZ _)R=NoRIǵޔn=]{H,+Nſ%Gy MyxV5(B̍]0/G-0I X?̜B7e˖Y׷8g2XTEU1/i4.u+AQш&鬔GM<څb?hQ]\Nhf(7g`tg?ٟ!?`MEI Z8kSTVڄa_uC:v%RKu6_R5{ >kf\|IRqpX`NtIi2O nQq뻘 oS0Oϕ 0/iV.-D% 嘸X7!=hOh_ n *`{tfY`. dQ`'XbK:+]\DDGiþ_K ϐw|DG2{?ʨK=6;{>G""cue^KqcGt7K~0w.@Gs>2|S-ͽ_r ¿˿0o悉n!ȥz+P+ cֿOFv6zLs ܺƒ׾\}-q0nA;g4Q';ͳbNqg[rEEp( E<5i6{c7!fR=0q7qmoˎ3j9Fhq-5@5L8&z(.1OoQaA.ʼX-*~+ޚYrgx:K7}v5"p` 5&IڣPIgE>[F$d둝>Y* #eG4N6R(LZF,S|W̭Văɍדk/A`)mO iq+R%feiz +3tgwfqtrP(L0&΀n/z\/]m{tr],JGOѱ-'BGqdT;Y6Xpz~EnޗZi>c5lvapHO1Uw%͟Z4%87=Fd28 }t)O #qʐv[8ʼ`ca   SdأrӛNLGItþܗ 7b9sርOјͰox=?p ZӅ50hֳޔ(:P kd!U漼EjNr=?U}ߒuVl~VQ LAG nӛp9ٌW7dMY7)[Z"$0[󘜚,m2vI:Ai>V *͠zz7̶ElunAsmyUKiSڈ5(ӛ/Z Ƃ`ۿ *<gJJqPQk41ZHs|Qff?f 0''X*g0p zLokҼ.]Kk|m:ہ.34 ^>6Q]s;>vԠqj|fTyvぎ+r{2xcϛ[hAr%_g̋_0m4!s݋# n $Q%}=*fq%>NJA]fSٛI{\0.huɠ9Yd:d.\0I'~#?a;ޙRͽø]{ȸ`j~"w~*wsld?Hm|ʼ?WAgw!?ٴ%Qj?*THk=u]u ?5ɉG(;^W`Q"5|5|=bbavWjF÷QPC *9V +/6@B~"gJ*54lz eMmܾr7AWn i4~o)|I{v@=Svt-Qn12T:ŵtΣs#? ɽҼQM>|IyģOL Dsa dݑnr@_qolq74|e;Zq4W ʩV&޻CWt1\}Q9:Z^:[̩+d5hz+ۇLx:1vɕQ E3ǭ&)h̀E;-Ax TҘ-A:>sE%%ԋ l6˥r\Z94h%0I[qTLRڔ.,QdVTEw֞bΨ9;E<9.kUC6%V1KWcأ+d(E|o0iZܼu._}H53x|V.9ki!=5ܕ7uEۏ-;KEӆgB(?5%\$alpzѽI7p؂mKK1k帴s4*"֒Jnj*)h k`*_i]&෨ɏ.K"O,MSvk5:h%')h Q,$}džSsgdR]ʼ$]6Ayi#gh}NIN=4ȗ4P0`n} TɁKʹ7;y%eS3V`ZN7խ-:^l`JcM65Ea n{HSe}RøuC +(vI/6˥×|6e!Sr)~T08GA9PIA+~*zX{p>`錾`pl.Cݛ4Mi]ׯ^&nhWIuّBꄮ wR7QZfNQ'5Kg`NN2Ouuc9I×)i0+ʜaH#[2˅ {6h`(}l}]9 {T=>1ߪsþٞ'?3&9>KEqUF ~_|aq4nc! Ll~<-]m{trzuסfL1R_qz-=MEԉ%cԔ5}8{? -xՇ9ޑb$+)5;4&}0{OEXFM x>j\ap,߫U 0IA+ww.޴x=^1&s=rL47j^灓h|_>(~\q%oGcpV?ף%  !  (UZ "2G31&&GNuJ:h Ja6\Q%F3|gU c<\(aszVBlcJGDU*iVx 06Fč{Pgr*5"HWՎb|uIIDATTŴ0ZF'GRc-*{wS% V?kï=zWR O~;Jy)9.n&-u)ui;%=DXd6pkO$^cω44t%6DR&l~%8kMKOorZ)B i|)21PQazM:YĕAN)Ol#XGa:S;&N\cVd$Sعaa˾uY LkFR;@@WR;! *ŭ*!VZ<$q҇=w6Vvu]iS[fVkbO?߮;_F ϼRbDP8*gмN/`΀G8^Chі\Pa킦)=Y,iR;zWIIU7&4l~.֠X[eT~FLQ*9u5.F0NUk T>M/L2B6$]jY|]RNhlGgؽkNԥ2?JnNCڱB Ķض'"\-wvo i=Gp]!G5G5X~Zmw<)Σr)!uJ\Kۏ{䣺Z):M ?ON '&Zdq(ykAۇhi#Ef(1k7{[m'TƓ1n{e0{0p?9@DoGK-dҋA1,*u+lNZqTsL VW14^&i675?Ãs;>G#ʢRR]Mad}幦P%{KK϶>5(w@F LV.^pR+%fb KW#ۣgi.Tͱ }?{cW l{ld3soJ >N*ן"[UnjͲ1`437vk}R^V+<&i+88tΣeأu>>lhDSzL?-Fc8Ujnh1s ڣd`nƼ/$taFAҼ +"ͳ1OgH[!s oJb%ȗ-|2`a8oH1]'tJrA.aGf\u+yQ%}Wg?lA7;4}S'#q/_F ,3'=7nPWIwcG? dƕ FE||m!(f|lPq?E٣Ƙ >#6RRxVԍyQq~kiO:tTy+e@yl2|,:7;}hqT2ƼK#"8#!@hQ/䔫qdywc"dKv^Ov( mT5@F<"]Lb:(3bGlh""'Bʮm.11)JQ1fuI5悙"SeI<*d3Du*UIi)ASazr ^jele"unRR/5bL w;+gcxvT<wSPI;`<Ymy^Z_:V4䧛b?u룕[ѢDyh{ƎOC*D3g_T :!:]M~*r[ؒ{)^O8/AUwO|.wd\@5(J ZDsض%E3A#XmK:-/QElpeQKRSEФd'gKb-7W{w0Ƽ+kx׷0ޡrϦ ^f+vCp",SiȪ\x#=ZwMa&8y9 m+ێ2>F8|ISϚ/Z~l~7`a]]-nb'<}kC.]ki@Ik ]PmGʢw];2+£ wS%Չ$P-f4{92r8(f5\K"ϼRɥVx +BfѼ콛J:9`+O \-eH"`o/3yf} Ma713V;':%)z|2: 67 [^A&)h`&;;p}e-°/7e]O׎3b>eO.)A`qHJg6 QzYL'OlӓrR3ߕC2b9dlL:%Ue 3]!{֑eרӛ} lSw{E ̶تOuFgK>*o,Mt* rRѴ=% =qUgs7Ƽoi8Q☊RYm4/4fO(n8F3BVxB%`zX3z+=F0K7͋6:♯4v<GvYqǔ.bj{>C*إ^t\^=>+a0^d7Y}tר*`;sSc0$}I26}yhTD?"A4hxQCѴB-#5.ŀGe37DQ L|mf`"ͻrc$s!2 DV !|Cw(_h8h\AtꬔVKF*Nf s!K8Z-+kDti^A*:en/;j<"4WȮKU(C"gg::F{I17KDMm{4."뚆=1#:bjUK礏>j))K$ՏMgQ lgqp7{_O3KD{x}g80A.]R(ZP ha\-G7b#(.C] w+hEn*l|`=<݌@3Ͼ]cvTi1Ƽ4 OQjLDS8E-u zgkt(ywi*bPI1;fTlw'Ujt z݅R.UU-Ј lssmJsIūt-6xŮݫ޴Jʽ9bjQu<h75-zр v&%35ۣ{\;Ւ /,]-y{$`zX3ؽ/G8jQħsɲکG){4*Tm\Nee|ut. dU>zʬzcԞN*4E ]Ibdő ;@{\|[Sxg9o r[`:zKc;s8w=ݛg>{⑉'&&-P!q܇ Wblk?hL|n]f̪GD_A-Pt.g"Iڎ߹iJTw mj8JcfF~HfFrNK,/=gWA%](םt~rRW7s n`n$&i_U'VMKU4.ZiJ,A'+Hm90' /d^ɼ"/yE&Lf`;/Qv܀f`^KACDo#mTy.#7ǨA'uށ+k$奾% bB )x.+|ޚ;1x/ǒSC!r^Ġ|'V4>Np=O^`+%&A0(98oT 7X>ÿ,jSd_!+ Agk_kޥ칯7`%칛"Lo?^n `YMףS@_LR0qͻn7u _c/y/Oz_8fnEQzW&Td7_;6 xv~_9/p>lg]zf> ,MmbQLҩ{ 6o_`/=> stream xڅWKo8WT@Ē{k4" Z{h,"&3ʖ{9p)m#ݼ߭_ H,y7Q+2*eRF67^'-R4@ R fu6 We$-{qU"\ =1$˃2yΤ YUU;Y=2Xq(?UkKLBߌʊEqƋSDDuEt oHH _Lr%` Z>X7]%:JRHv~ i*0!9Ir汱4ƿp;~4$NN=мV=$:(6aQ~0#)8+!T d|k4Qmuf%bm-1W'VPg*%?wnyP0kPUk _:$f3,'W?=fsX=a9yt0:*՞=-8;b U׾@|!/` Q9F&()=(0y]_t3m7eN7sL@ Zj0;lw&0As>1;nКXH"푺+(0j (yׇg.ppprs'Fڃ##zg PdJ7~Q;)7tfn `Zk$OAF@:tʻ\1-" h_t[m]J.akd*K3 Q,`+% [I-q=xPĚ5A=5a4U˱.&2,["IӭY-D-y4?sB}l8C@uibݲgnԪ}Bcx؅y3Yyν4ɬmk y U}Ʃ@JaD)ΰ#3t!)i-'|R>ӃӲi4#^yDS[Z,L( MK:bbz-~o,ՆOxp '&^^vUC[Dۨ=\T}* /$Z2?6:N|HRx~>!k|QړQMA tvΠ߱ET~iԛ7kgKQ-[d ࿎ׂsL5)=W?äƼ endstream endobj 2 0 obj << /Type /ObjStm /N 100 /First 810 /Length 1436 /Filter /FlateDecode >> stream xڭXnF}W[ W.#ԉQbik2k$Hʗ֭}9;srG"2"+JHD ?RAIhb%HJC6 $,C=4%dL0@ڡt:2xԂL,-$7d%K(:! pjyvLq $GNH+%XKӰE4"!EX.Z%I֡)LT6h=x RSxBx$H"i,aZq&>= Z)/9@vZrN{F")~:NF5UxaAdt!"RwHJ1ZG2!pID@$$ kv R; C,yF"&iĴ@2i?h&-\MZIR@63MƈI#[$<N: 8AX9V9d'/ 9( RhBeo9坧qI'||5CO羹+F+QO鰡 Vsgi9I˲*iƪ_DWk,xc=7m6ܤ{:W9{x8f6hf{ZetBGD^F0F ;.O-ajvm-ϗ} 57-'~a4=QcO-GNbZ؛l5tG%݇m/xؾնg"V>[E3J5s {Q>f cۉhnxFAP`)|<&  n _Η#}+.G`s:M?D-r8;qisYc!Aβ~B}N;]vj Zv5 s+]nC0݆a֛vFK-Z6/:osқŇah^]oZsoPl? gv݂ǢBs髲ʊjnƯŻZ\rSx5׌qpHԸrK}]*ܦ[xuMzTPx韚yJw endstream endobj 249 0 obj << /Length 1377 /Filter /FlateDecode >> stream x]sF++ vc׭=әEXʎj]vHE(S[<]G‹ϧg/)x@BLG'8s'Uܯɟӟ_ (}OH=!fqLdS2'XM$YZ Ӊ~\ sMϛӹUR $ww d~NxwVXquO}b7s"F5zF #!cq;Os7lrߧER"iY7՛1uQB A0×dVt 3?4q})%{@ۧ]½IY2K+ ke>eb qC/K`[C]׾Ϊodќ} L&mXM%3l ǛTeiaGҴXt KYZҌWkpapBCꤐ+'ph2\prZ4/{"b1~éqRNnq^r'I&#Q ]P;~&2jsnBvv:WVu(EE[}|=H+ʮMNnq F]/Y%^jY.%P ȶdaz{X.V˵J|w2FhUnj55R˝\ڝ""\79$RH+'(ǣttUR!&bY t{,Dѥ^̬NV?y;N$[[N횠SC]ġ#X|:nk3LS~4HmfnR+Eb4:Q&ioם̕cmJ Y1sЎ[.,QH["I 95} 9vABo2]$JXZȲudgms RLk{ uaԗ,U`d*6,g2ygUnD&eeWlʴ/Ŷ*|w8:Kj1odb |<ވ];UB"2I/OpĆGmuڋ̥,#Myó?Ny 0] ҃nCjօnXhUG]+E4ndКF!WkTnQ(J M,ߦM/˯4h40 [+oký@Fܷ Bԁ?sr/UʼnXA +(6Q%6.6A )#sCǁ8Ɏy K|ČJ.-X|#߮kcƸL~]L endstream endobj 258 0 obj << /Length 189 /Filter /FlateDecode >> stream x= 0὿dHX?8`ZEB7Af>;NhJuH*A'h%,!9hCnQk+۟TRW٤e/ kElv8FPdtiK<%Wm~,NM\oYkriQfp> stream xZIW-L,$g%s99%EίO@jK." Z**yn"_m.onwcՍ5Zd&*Wo04v]۵*V7 Snlͱ9>М+hUx|V72UVI=YLdgxR0y8)U0kDUӮ[ 8C#0 [cZGz| eifS\d snͽgmi/wm;]Ү`6wޅ7kKYK-r\3o=UçY ~DGaf n&I/,Ox;:xY!|UH۵-ϜG̦C7tݮL"U\VؠqOGH0͢X^0L18&,6 Ԑo[,lH+HB<401uX/JA R纥NUC`4; }8Zgi;iue@\FWִruu%?>@Mɖ|' Ǣp?gyVN=rsG]kX{(+KuLT:&Z.=cCA=m@/h^JKl:? SfTeޠ}rݰ$z1 Aq2ЭAX&ρBZ]K4?gljA5 &{>:P$7!p!vx-p:uY{642_xTzidp#Xq!n^c虉TP[LGF:>٠r@=Ϲy{`"ْ ˠxR#2DO ^d; bcJ籯ɛĜWS&9i4V hͦo݅ZS&=ItywIU8J gԤ8G]>FeD3Z'c㘊w288mH sk MMyASSeC05G e9}sKLa=ehgM`B>5ȋ.C*Qaz}k4R+"+'兒oƳޤ(Vtd`3I(.Dp_!*fֺ2+WOp51JWKx$'D:Ҭ0|` v;0 *O2b[|/:V?|~𷇳npȈ׵ i&(N3lpj O W<s$HoS=cMF8nG07 v Bs~?F{3`EP3w2:`rad={2iȢN+^>@*Dsܞ7(.iF+e~>ff-NE;to8X-sxXĊLv9&URoȕjKd zk)tx2,u_73t8anNL[ F9nH?֧u8ʼCߝOLfG3Y,˜*}J(={[U\&xDlME@='5y`J+vz2ݧ?N3fK|Z b]b M2T>=R$ѰjDl@;U1Tv'0ֱ9ȁ[զ=SfUۈGqΎlt @T ߰ݼBp!bԪ)awP]Ka: yWm6BAG*T' w#WF8i%W,J9)J0c.RQ ,rݲ|ՐUMf\RaQ  ^z ͈eYl u_h_J*_T^\vja:h6GEfNnx꬇Й s;6E#E :TrZ>['fV9_+f7MK\䅜rn|0+gv+$.*$SVPT3-)RF k˗ j'SM-YQ |/"kzpw> ;̆%(,jץXw)2.f2>?1l4?ȕ"~W撫i_"ZM4KD54f|vP*IlTƦ>4M Sx Q(!ѦOS Z17OZe )^΅.GZ7mk:ϯԥZOժԁdvE > stream xڝMo +8bLrT8ZʡɁ$ؖq[`uS/yg)zF}:uiWwJ"F5C2)J}ߪ՝P ڤcR1@-VK|d'6YU ` RB@ D H0!&b;rS?1}Y͂]5nef TmNrg{>3D}7.wQ}9t]0S%v%G7E_'"2̖TcEɝ0e0AF%be?ždr"-š/VIXŅnE^~_? /56x<'ބ7v/ Y3ir+SKwu+l\;0:); *,V`+6mswM endstream endobj 294 0 obj << /Length 3857 /Filter /FlateDecode >> stream x]Iϯ% \EetcK0\f9me7تqPLrm?Q([Dz}QMbH{멄$Q׈=Sǧǿ;ѱ”y'+^"McJYJ0.67=J┤ݗ*RĄjRU;U~^bfLCrye}Y44p0m-S#Y.[>Lϛlmo}-Z>FQt 󲔢hh5 [q*ebBDŽ^ߌ`ʶ_  S0$1KHs7~LeҘjF\}W0IR?LBxLh G]1zI^\ ^!*'󡟨٤2Z)_VM׾$݌i/NlgȢ^@j7޵&pgY5Q)$;\D+~#<274T-$s[]0tVA͡ &S*җ{"-[chiyddjBe:Nnjjc{ cf"S6[JL[*+bkW^*NrmbOE7-],2DH+88[gaҧƚTMV"R;%%Nw5c(H`Z){ХL24iB"U( ϟ[Ml)Ep']D)N` Mn QNSȡI N@>-äNIk3+9GpA:U X‡;֎Ɏ?j0/>L1MQ$RH|gVͳMnVl4i&ғ؅I!>5>=~&hnԪ(Fa:0mЩ$u~#go<NgFk{F^L'k!0a`=gyUrl?PF=vٿ>٬qןBy2ML -l bl,LgY8Y*l,:v*017"v""asa+T7Mc"[4J3Eޑ(ӆjYĊmm V$ ILM%)w,PT$gXRl,Zm1)z`M?aV=0!rău8^ryG<y}i`*bG'DeD2$^j<W7ԩ "H =ڬ*Y+\=HtW'|fT9`-1%5!G x}ėSWnSA$ @oM &})Z Ư  Ta ;txK2v -q3FCh5H3HMABpTӷMMH4ZdsmG'"ݢVr%eRX)3*Sk{C/g_B!kfC_yB ri[Z KHho`VA[c P ` HXaW,as0qO$WflqhEz4}w0נm dbx@ކ,m0 G~ZB:8Zҁ'Iٲ H9-G:Gj+G|TČ4)\$n2!]yv5 J }/-M#{, % n5 r Zm"4U4Qt`Xsip-U57:,<*uAukL_WXV'fM-uev]jIwjwCnAܝ-9v 7owU;qswNCRqh\;* , '`W"~cHSSfT~M.s; b%6`s0x$% K5F~ig$}۵G !q-{!(ܟ=w{GJQW1gLړrOZ{W_JڍC̰dLpv_oc&‘3&oߘ2m<܎ _@{xK"wKn ?V@m0fb/qZZkS, vd*o vq.<]cM0Lh9U2grΎɍv~Xz2ooQ(*tEKǨ-}[QdGz` W% nY2*iq>4oX,.]j4H>›ř98ϙydʰg^ɔ[:o)syܝK߹K߹4]҈}g@_;]w{"rv.{\0z {*nSe$g/LuJ,r[zol-#&zKϣssCx''Qfay+Yeė\*Ǡ Qg[lҌ VM='% ~^z"1lvgQМ.B2U׆7 ;"[8 {G8վiT1/%{{{\QJϿNVy}t/ލ_9>qN6q+Ԇ+Z y Þ;S8H B b"vf=Iqpe sX{q-}F*UUIA€< SX)٥%bxcmRXT椦E{>!Mq'`Nې̅`m]22 .4ٯ涗[s] k a HLX]``Cdqu+!8[2k/1 "69ؿ!Mu5{o0z`newy_RX둢HIe鑒ECoFC;X_&7I8ɩ~ Z1ó[jՍeKԁm.>a!hxp(>vD9jAN>zuK,f> stream xZ[T~_aݮe QHBnxXQv |uM9Þq)*լJIkSR킧 ϚzD$'3z"%KL%QaDo8 a U 4>k1*c)u ,ufJRAG4iW@jkNKKUb_H=|S5e|Pӆ.vkblLVM&AGk2 dVV% 4j8fX]-;0c`O f5Ajc` 5K(f`,:~K c6Jr/I+7P`cT3,`9'oszr/`!2vIJٰkꁟ.\'qOaСr!*,8hWklu=F+%%#b1HG}@^TZI j 9Jhzq ]/PҁLBۡ"db5߂Q#bT8MPm<ΕΝsf{o8];ٛWgo&^?ݿK ^/6 qoqF0nRϧ7y?M?~|zk^d '7kQ(:fAYy[){0N(޾|dMHR8{Jy -0F>"V`sAc1<]J{e8j:4xͪLsi4ևaWdžy=/D ?]kp( 07UaCO`O*s"B])桬oOjl+BJxPַ'aJ$X?wy( 0m"[B3e}SJ?aɊs Q"Dl j5{@="SȊ#Mb$#LUxgr^HN*,Ldjp5" Zkd Z]\չ9̌Gj @CCe72Z}kCk uRֳ'Yև0/Q D{lX$#5b\d8O7 @8QjGz_tAXJ\y bN] ,b8n䑩E*bG|eUL_i)"="^Ħq'P jSN:wNSu_ֆ y^H1!-6UsuEH*:7U$4! *H춪UxRbhD䈂uk*G9Ǧ[0N0 Ko+bB,3!pgxt4g+\_;/Ei1 J^MȤ}-ZkGAu~#{k^n9>Ňۿb]?~/u=>\A|\q>\VU]Q0vsH%=o?k~7Y,V8Ǖ5s(q"m={_ ]/+ReUnw\~/= $yHY1X/Do᪠8ހQP, I5Aqq80*8Aw5ŝ n 8wd6Up$qР8_5{L v`)'EЭzRk z23¹=eSk)tܩ@.b "Sj.0dnB>ˇh+Ge ~_B8*bݸJw + 醊eJ5BK3 endstream endobj 384 0 obj << /Length 4277 /Filter /FlateDecode >> stream x][s~_$tgmiN//LF(cPn$! REJew=!;WfzRw;*rW Qǂl$E!󃐏0oeE#rdJ}0I Nv%A]\[At-8 5Zi҉J^vDDK .Y[ c+Bg\tK3͜T]؛>j4.* }бzZ-1 3\y9o % VT9#ҼyF C.qK_Ǽ AĒxߪcnFth`V⤙H굛c$1`! */7 "`'=)wRB&ћ6"wђx$ @%I s|t>,/3#z@$+3BSb*qFچ6|XPL`īK&<~RUڍf(S}>Qj4 h8N>r*y/y*tӯpՉ@T5Uz V#a -ݏLo$4nW2E՘ p^]h{(X8~~}#.H(iI \+GE-+FgƦq!1i>RCc4P:c |JKx_ZHI[f'J7%_\8[IorfUrQ]s_,ee,'rޠ-7]_$CczSEh^[6&GJ6av՚J4}8?aSH'Z%벜Qj7qsOZHR}Q'wY%!o@XxؿPź-zD}`k9 |WÉUR7`1j7XwKڕF5ʠЕcu+H?@ {N)!ibPXKlf_<1LMۥ+Ri3?eIkjReObV jm~JY6[?Mv VSrǛdZ GӬ#8\p т[ҝ 6FX-35M:^z50*D/,VҔ,ɓlLI:rMޘ:L5]k%_fI2MVX˓v +S-Q_m-2T]O;0 HsZ( ׭ yRzXET*GT5O"𣰦q),T0֥({.d,5mn]o6LzUPȤE~gq쫟u0w`;MڌPgvnȀjezN[um z0泐܍{5O߭m~}_K Z+:ưh L̠Fj$04 M'zE"ƒVC>7!OQImiPwC`&դ6_ʒ;Z]J.2F5p/gv=KX'}P"o{=ۧ?,yY',eM=R]eqkC7;- S!w3;'R`$C]`^"[r:~dpRW J(f/D h>ܞeQ(itÆƣA8(YVv/gh`D2l ߙ)RNxP=9q/zer0ӧØ~d1}z.ϑde^)}EYhL`ĭqŇ' sAB/2NjU@m# P_ dm> tCe bW O\XbxѹF+Jdq>ӧ "ى+ep_x 1]adJlau̹8<[Wv/\iTDօ?OfΚ1U˭['S]s YA~[|a.1ϕ܄`] p$#ŚY5<=Chl@+ PXT%^rK.~y dTPI&o+%֜s,!7Q/ؽDca//ǖ=+ExP5|ϹNCa3&۵rQz[UP3S38YH^z;&|V~jWb2TVZR:nyxfV],LoX/ W~ٛ.IlSɧȯʋ\ q7ӽ Ns)l]3T% ϠWj1 Pg:ߵIGA<5cDIϻEDvge6H-rU"4nףĮ [MjVĺ=(i5"G& e[ v9ؿ-^ ܵ eU=W =W] BҢg=}Vg##@{"o ^=o4iDGK%zX#vv&$&$2c`)%є6U* ohć++\xg\̂1.ִtcejq3cnьۗ<0fߏ518io+[ ,x-vY֞46iY3kͱg!̚ d]1 rX{Y7. 8"Ozzz]F/Ex* {'aH5 Cj♀y8"$DX괴 pӇfs+> stream xYMo7 б=D+R)A؆4MMc䐏m8MI8tQk|H4ZH!s "h(ԊeFR%%ˡtrhrB߀U r9%T]K.Xق5B[C2˹jjY(4-h/t "qYEbv኎Y9)-p$0,8ՀS4p LjL#E!6x``Vh rAo 8K[kF6h2UT 8_I5.P6\sb"a@@0fJUWm`x"3IAN̘'\?!+N<('3s쭠ñZugU'a3[na(KKq@<[H5י$L.5 ZM}䣀\rm.  "8 {ѱ4(@x#BpRx:R;̗#Tr|9E/Gt+fo{ሡo/`"%g?ݹ9G9d"ɦpaX.ZbwZ:qb+%Ŕ:qsI:qYcwɒDBM#s.[lcdlع4Wds)J5JMzDI,*G>YKQCѨW4vf5u2ګ3Jѣmluƽu=+1}Wpv#0ja>+#'?g~@/y-ټ^6>x)q ʉL;rztq~?[|= }/f][,O?#ZWg'o_=Zzg59Gy}H л PGC \9oml؞s弥c+c;鈧#x:وg#x6وg#x6وg#^9w*;?={s: Rk7dT ֤֦ۂX>;ԏM03vʻ{H~koRԯ+*Me_컁YI8y$ 9 1 $z~.ҷ]AM~p< m3(%w~w[p>As(R^[.KAơ8TpK,R|3[ZoR%ЕkcC TW)1%ꓕӆ;"NXP]JdUl׈Rl^ɮLxqȵ,R.oj=ʵT=Cnji.̵=}NP*]BF%u۩.̵9ҷ.nu*$Zs,_ڥ[? GtsA0|K$JMlއ_yEʣgʣډ>Fc},u⺏I[>i endstream endobj 456 0 obj << /Length 3706 /Filter /FlateDecode >> stream x]۶ݿB/mx&Cӏig2~i?$=}\H )J"%8L<7GX,v$|HCw;䯯<ADŊc>yp3N&8lj>yaz?6%BbU֪KNqB(%'IIL'h{dSGwV7~HX?ť3(Oz}#ouk*"ywFhiaߨQkȣ<u) GX (CS)f^?~QECe ģ {/Zt3nM;"լ{铲n1NQ.Oih p !HӜ9z.QsosD 3bM5)iR;T8)B;+x^L*>U$X(zJ7` p` aG`=yW0gESEm+|Q2?KG]ens+[5bG sU+;W>F %} .4D[á7+yyWUUS`|MEcA1Db'z#*~c=A *0,N4]g}m#56c$f\-۽&4O Jz=B?3Lo../-X@B{pml_v21=۬(^BRé]X"P1VSK֢+YѫRPR +-V 1p9GI`F\Te45g;$Xr'R/$[.YpcV.!#HT9[RSd ؍YMZ߀~xVX\ ˔^Ê\w-=S] Rjuך@onV:LμU@ P  dD9'ľ,2]YTz@x:F&:?~qzX$@& )4xИ#nsA Nڀ~3 =~o8M lGCOCqޤ?6>m5 e њ1p0u ;l~Vg`(FTuȼOn=`Uf9ϫZӪ U.$Ua)PẎ= g!'so5>=3!1j 'jbF/f Z6``k포^2bmg->hcgwoq;L60-X`|'K<ۋm/Z^ۋz9j{it~Z|>ՅN97Ѭvvn-"ϝTUXg{<l?={nq--::,diGbi9oV}ZZKs4,vMBɯM%/>vS0cMF'Z<u>Zm[~@mMQj׋4O3JH xaWM2l<].yaY^-fCǢL_/6U!Gȧ\m d [y%Yz& 2ZL6ORCp/$H.hHEu]?in06l/܎F};2̀ YPX'/ju6bCĚu*3=UCc5@۹Wa@Y|tnH իmi?M91n|goK{3;TܬzVfu="N$iu>ri]#OgSb_.f7YG)00Dӝf ~jU1?6+C܈c@I hӵ&>bY'}7}Ɩ> BY̒d*Mw{sc$esH ΍8e vm,/lȮ~rv͑N\hn\fԱHznng#Ο Sxp<@L`'w۸w?s3/pRsD"qyBK(<+RM^&`7Hu ޓ%B m-;a<&IaAl%z )cuZIb{3}K_RHȣȳnD߄ @ہ> Wre'KJoNο Ÿ["ݸ$) vgH1tV9x*KL[MHDV뗿zM[ַ SLr 9,?~>;WA3>xqoQnW֕;昹GY(=hcs2 6IltkE p؃$?tNX:F;$?`A9~+ͣ[<'䙜1?]@_Vo408/:ujÉ32%uǵux32}Y`woko_teHhCuP$?̒֕Y@[Ɵ\6soWsTxdipc]mTA@`>hXxO Q]lvʍ#2*PQ{ 7"K*f~2fq]&C2##®" m4a\hJǸ.`x=ԕQ-zrY#0,V=~f>3t-3} 4Lz#\_T?(\撴 Yש?sNb)l1|l3 endstream endobj 514 0 obj << /Length 4888 /Filter /FlateDecode >> stream x][6~ϯ(RͶŢZ`>8fFo5L߾GlQ6'nP9;77OR?Q]|OȊ")r_esv!2x=iB_^7O4Odxvyyt@Y_^Y1ʶ$uo\,d ږ[E&ϡo^~_܍T/?}}EWWO;M$zAzH jVNVЉP&_ L ˕k{ݝtsSN8#׶M2g[1kUJ萀{ i;j~C\oIE0mKk0 |RhvvI k:mӺ<՘LT8~=+(+t)&IDDHF$xD$?DDG $*Y@WxKXMS %ad+6[&f:ޚV=9h nQJ(0961<.XfRbtLjq47Rntp87ʾaJ^CKgI )t}okR'3kxH-Z:Jmr_6BZ4s:<3GR`^` SB9R{!"R:PAFX=3l?,c}ר~7wrk@*-U`PqEXQn8U]n1UDx麩p3VƻND[-B 7zTMMY۷T7M_Vjucғ7We]fsiLЊTfM^S55k-5ͅiTsdUrXWLu cVr[~fY.7ri_^צ~vDfin4gy[6ۤw?{!Tg$ SK}"1i>ii<&,ٹ4 4M>4iQzw9%ʄ78`͙1 `[@U0/p`7AKFcl׆ܠW̼?2ȣu 0jk8nփFrx9,[X&Fz[XArq`EES OTiH*+Ugj11`L%[& })Q>C)Itsֱ5MT8S$(4ZhA'ʊ]*PoWP6PN\WP3~;g jþ^ + `V!28OXSER\N,1GvfXbiJE^yea{u-^T9Rf͔T!ZwJhLa* pj`ᦠ:=}g(e(tGf;Ia/ޟ{N@Ɉ&f{~o7TWqzGT{GyzN]aw@9.E4eQ <칈& %|/i}Gi>Kd9%03ȱFs߬yMU^MO';kUOHwMF&Jbީe4ĚH(K %@4mWaO A}Vz OmOFn_z6BX'mC/v̴Q),4HRu>vU!Q}xP$4'Q)9r" H8]+h271 r 7tqm2x `=~a~ o ~w6pRJ5a=hWU{% kt1"Ukr}8eO8 9hqal;X> C=OaѶVQ@^| !Q :Cq`i0DSߧO= ?]D>IJl- ^Mv#{Hsئ3VPc}o[g}=ho{蛁wzCtLpkĥC90Gol垡):Q g5s{]WV=PI6{Gݪ ,yg( ;*Ol;CMWu.!ާ0B8$†Fl:]O\)zZ V2=L۷Er +쌝~'x?@B}P>(9[$Jd؇$3~ (&zDO&Ƹ -cТ$CL\ȿ"dKEh&r.rRe,Ce`.4YoZ% +V2D2gv%QdXfmf=W~EgqnQ [0MxfP@;6=,{Q R;6tC.:T]?~xw*~%E5I0c7Gu [ @̾Hw$BXBiDal㫞Xg0.dEMޙ7_ Ww5Y|~qZM{ ,x+ EO` n]F==kTsU2No:IRSX&!]-$~cI5'(LCXĀ78]"9izImD bDL<։y\dr@Џ£jE`mC|/]BSNNƻF2){M%͍k"p7z^ïؘb++)a_BF i"!s3]Y?Uщ"rD:C:<;[k9m;sR*?5rNV~~,7tcq oyEt%%AEXsmWW8c*HCX'Y"WN6y 8wCd?DU<>倝ZPY;}w=f9ZAƮ.k(]f#{5;&؈SX4?=*-c.و6dl5)vu8)(|h|u.Af"D$i'ogJ}P"5GG.YbbCБ&ztqQXܸA;Y>, k{QMЎqʾShBsk SRW]sJ,bW?yNieK}ni6>Z䏓H>@HvnT"v7j 7!#}oUԡЇ{N,QSE&^"GȠ ϗd">yr/ck~m6kU'#igEP*-qLX$ΐS<5:9OSSkǧi|"¦;|a{wYe\K;C Lʣj\:錪n2te ̥CTKG:8y28V耪%z4VjJeI3jG@&~oc`x#t쉑ې]\ryI87?e#C[fc>G.LnY,ɲx("XR/")E"ʉ XVRtg[?h (55wkOr+Eff3&{WKT9GԻ |^ iޗ9.z@|DˇIn^4ّC``Bsb<ת}-/:֞Y#Cd4ZƼM 1$li0޳Rt'mclL08`!cME>-%GAE1O'ɲ!v8C_^Ypx,pCA?U&1I13(~KVvz1 +iZD+iÝeƚV^PX^! g'Y"j1(x;z|K o }|X;䡌v棧StC.$ _id'#Jc!<&B2?t)po2 atqP|@r F隄eȬz'^ӝmƧzx=_#qUră遉jc)}y8AYP~95eu*D_ .ffTjb[#h 3͂toOt~Y"%uEiW><]>E"!9%BXbO y1 endstream endobj 447 0 obj << /Type /ObjStm /N 100 /First 881 /Length 1547 /Filter /FlateDecode >> stream xYnTG}䁞B,H$A($XF /9u=c0Yӧkꪋ5)1D[V1쨑&%IN( =IoZ0jO# G5JDa%0Iqpwg8x$!w{`2L1^A@-+<&0uPRܳ +y Z*&`p0WYsFǪIh`d AKi0+&0*ABeA[fp9UJG9=℠hꫀk1 1 ƴ!~=(`2ٙwӡGPI.͆&\4y `BFpF.-ȫ-|ŠeBH,K({$Ecd?hoY4, to}D?(ƈ;jL7ČJ$"EC퐉FM@mW%ȑOCǒmҮnhҶbiҶb7i[4i[ۛm뚴MUXoBW%h30aQPi1, / lՁc,"JE?E?18s endstream endobj 569 0 obj << /Length 1551 /Filter /FlateDecode >> stream xZ[O8~Wmv|g}XiAZF09n M2'8tz7ȅYg u3NwQJa xCrh ǐx>At{FQjfs*d8Ȫ..n(شjPoc>1&D::BmrCgɽ B8kPiA̔\T`߉rq 9 {7P<bVnjW2?)~!5{~rs. k) @וv3 Fe'ia ı@.:Z3|#pl&qDE$j23p2[gL,\7ۑɖe)r*9f`#XmSBAnT@aUNf-fh*۷JHeyVVnq?F'G4tLogpeU\gN]YSr5)*7# X᪍D/σϠe 70FW{>k}X)}S*7֘תU}'@z%Ғ{qEUkOP81 pxag={˘Fpv=I|*&SW E4pIۤa.z K&"AmSu$B_ BDmv}7V&=BKȧxuiNa ?aC O## ZL -bEbepk=Rp_Q]Y9 xgo+9ys|k jƩF%h1K;C)O {.ufsd&I`0 yHL` _hş-Z=9 v8,+NTQq/iwl- ?G!AƳ=!dB54,y5Ђ(Z>R]iUp; _N0iOםUt3ٷ׬dgN:5Ucvl-f^[H:QnRu,ҭ5ਇ{;aNhv*I&=\_7~nmRfW ˬU-˃5 H4,U5-( X=o .u<@/P]mݍu2WTfDZ_䷺]& re$C[AQ<ܣ%e}Ծ aKQ? ]J{Wrfq}MÔv, g20eK,]x)m\hsY|K]{ţJϻʼn,Jo%%D {?D@1^Ֆ+*Sm߃#*J2M4E&C΁2K;>~ endstream endobj 590 0 obj << /Length 4284 /Filter /FlateDecode >> stream x][s6~mA6tqΤtd9m%ٱ/xRG@,5 Ee'|E~{ųuX4AGJEsS#yj 1ڂC`n"%nDAFjI<|rΙNG([k=Hrо#)KK!KX/M!X!drcʼn#s8G4텋$Sqɏ0WTUhu2Z =ֺv+ MqmW4C:ڟ3"yX/&;]D}aв #&:kأR1*G卪\ yl8%!.}oo_%Ak88E@C˃E ڕ]OKtN;A_=헟bbY:!vɺA zP*+0`f 03hs҂>9Y q`фh]yei)K` )Wxd+U0Q͎fDNq7O ˘'k8Kv臫g2]Q | +(ϡ|[g){Gp!A*^w )0+KDZ(Ծg2՞Z){ɛ{P  ŒhtP: !Ge$dǁ33rEr@gX3p Cwwz^G~߹gf - MFP#Mf]a 7df hU Af4G6H A zďw/Põh݋Nd;#M? d Ȥ~6cs-܏x%7^:׭j 7TֵjW 1J'H7ԏC$I{Nʾ C[ T?dѝ>՚Oғkv:N-ν]x[*'Έ2HlV=q:yMƯVv.n[ P1GgZl؆;h%-;J7ıSFN9>¾Ax!`JyNhc+1D3w#D>Gz 0& Ldꌩ pV;\I'*yb{JI'S@*+)" +94+O=hx k$tC84dl=GF)x&XΞ4G->Ry7.$_!hؐv="9SCn"9AITuJz T4?TE"H#noz ͹q%uțϬk7Nj0?)رO=zbGHK,RM(~9D>k<@Je!NSTv^%ByA#Vjҗ}2@JisE8S*q>ƒʦlwKtSKX_zN3p<8IoL*/`- QXWȫ{d yɂÊgVLYɧCVjpXxua!q=H3jfhqqJcڍ]ܓy ONJ( FU| +P AJxD&7RhK+T2~"n^ۚs/WO3*V{or~3k$[q[5ƦynU@T*|/)0e2'}FR5 UHU-=&#(5-SJ=*>㡇>ZH6mql3lOnCHpsނ ;}2C&XlԱcg_/3?OO"rڷ͈kuA7SCh-x_nx9bXLcDMϲ?lcCvGm`Zu1+IoA1?qOӪ@WWX&>զ 9ZCo?z/ oޥk v % B9PI'cH=Y9Iyy9#(}A3dAD8d_\ŸE,>hr8#+c=.'Y$Cy71ri#ִﹿ^Ò$Ιub͕T/ endstream endobj 566 0 obj << /Type /ObjStm /N 100 /First 872 /Length 1348 /Filter /FlateDecode >> stream xڭWَ7}cUŖFH,D"HYFr{aPsMH*$Ovw>.˧[iE ax+Par\c/{)]-}bEn5̵#H8 8%F)>W-Ǣ IeZ`_ @-VU&ƅ(lIaRc- 6G ;sIJe6o/8x0 #ڽ,z>ؽ'{-ųq :_w[gvF=ncL T4j3Kѡ~|kpbQd: Jfd Q1N,U`Pm6Wq u[iN5Zsq 7k*8$o2(NjVu-šW$/(ɋbEjEnba 32$+H"̞Evܖs/RZbIچ:D<*K\'ypZP[KQJQ1K1O#qHIc=#{g&ṳшR=(5kQtzRGEWuO2 endstream endobj 649 0 obj << /Length 4301 /Filter /FlateDecode >> stream xk۶3l& Ib4mnڙiFx'z\(K&]Ԓ$5N&8 `߻u/|1wգO/={d,̴н˫NCe$2ջIv+@Q/T0']%QPD: a@q)`ޛNd/`qOTe *ms/ &. f^/-a;v}&''pJ Q0ƝP 'np`---ޟkhS/﫽!b\U(<-8 1 lG.,>^Zm;4F:yH\Gˤa$4nA@Fm]O%趵0|gS_u8rVHtFLarⴚ|e= -"y\u`Oݰ[ڒ5qvSƈDD\8 4Ia3HV{kԙ7,J4vh'm(k``DGL,Ƕk S73L)e0\\4+& a_5«:q&sgz@]jg O+Vb^Y8L*,ZX9[zF*Ü8 K{y䶏qeu2F{ODr%% [&!P=-,xHhXr:BB zE*aM(#q$?|r"ks՝_ӭYN%`>*{aݕ]Avtoڽ\OWrt0x2sH-Q-9=Gv}`Dwnl*&`z-Ճqp9v]hm[! LUc~ Ǎ gm{pG+9jR]^Wt{@N .,gĪ&f$f(ʡ&(h)`\Kz|'/EeclmJʁw '1DU%CW("INWLRG64) 8ZaO$(;庨*Vjr m@yf@1)t36@""L@趵ҡI*ğJmjhL-nX6 vh Ї Se VU:Pq\0Z7@`mbm`mnm5ިL>{bEa"_fL87 % 22WV^5#|>gF}A+# !B9cW!j$z_ H[R ` ne~鱲$Nd=t%UDlQ) *IbV}ԴҔmQݛϐ,<4ee19RO- ;dSXwRg,jP-J!eBrH&d{[bLd%2ǘ{<603{%J:j/Nv Z/@dw[Ɇoq*Х~U'Pw"ek jL/[G'AYl수橥} ]ݐ;"v"O~5[ds4%7b=Ҧ~/Z2gev l~9EK[-c $Ij[Avr'5i{nZTUI{YdBl]ЅKKMVVWHSi; >SQ-+/R߳DN AYJ>Y=7bCS|t,[>ȃq%[mnWא1`23⩙EL?YSImp*s\H¢DoXx`+sZ5><6аåAZTZ}H$zT^WTSV^y?[ipY4͸ |hόY_Bb3+:$&V`Je-x}ߐ[As4?FHQowJncd<J_r<'),Db˗]c%XtYb:wc"+X4x=*I/̆+Y}jfe}'>s"n#-I5&ߖ.(g*սϱJNj[kk=Z@bOА_#^]CQ'ltRS6aq/A D#hwLLY&s#4 M&ۍ,F 36qӌ4(P0Rf],ؔ}L^~Zl_mqRSR_[m '5OL~bMzbnPzm7B_`VZD)d^G_~ֹUފ/, EUC ~Hu;p)AmnBD J9OaiYgk7 /=q7Z@O>0BNG%~ 'o$8<E }*> stream xk6{~Unc/"pIkH{A 4 Pf?Y IILCYp.׏p7ѓ Ln$WB<'^e _{_ہkQ9 ~o_ GdHlIj8dOCg~`Npskd (+f)~!2Z\VI"'K^Rm<#JsSL" kհh 3K3 -rk  YWpA!7^{a"[ >Fg^5^ޏvأD//Bx?Uf{_ f ]~ {EWvkR?+/{e#'v%,- K4tgߐ4Y*ۻҧ@V̮1"J:>@1e~9+9#R[`vmfqi,Se>4aG#%ǍT:J`ڲ" Z+0XĒOf)5p~Ra2\_+G-rq|Os5p3 k#I,/s÷l<)?AdpDjHrTr>yN倒C.d2Cͳ%`"i \!ٶYUecT"ǫbL0cXE,f[7 G_wM5!=gq"pP}V[ sGt_ ?0m90,n]zΪ VJjT's&R{uÐ~)4QO?0d]nP3P"GK`[^g2>XlQ !< H3!2# Kș୧t`D<oKjX2}6]a6ETμc=C}WkQ $mJ>O19< Pȶ*KZ%<þIC l!QEdؕ`.W'cKщh> I <ƻI ҈d`qO% o8`*i\h%M9MyE$&b#Nk,kgYa`5@YU㐷R7ו˜1?W zݏ߼Y*Hw'<5Kp A $4Іߦ>%R?O1{R[Ekϐ EXV*-r"1ް}0 F)d5a6x%%X]w,@_Q7~!泄B|{B2ku䨆ĥ!V HŭFŭCl=N(LXɀLX$dT&GpPJǸ(Aи!haFvDNjhpו/TMcv+7`t߷z9oڔs=2l '6蠝ӝkPwߕ7e e^n}̴/,Ȫx% m]O.OD̰WjVF _&@Uw75'loiig K[L/nUmyF{Þj{6BJv+5:<+}Oh6}5s.rvIb ?]u+JM͛^囦蟸.ٷ` V݅ :npv_wu2|)٦omɶ-EӍ6CmB`c'O{8ktvZWS\T)72 n*v7W'uK q\oa$c/%ͩ).OtHi *O#Nuݭ—Ǔ충,lʎnr7n"Ϲ A6|{^oO r-7M8Ϫi9v?3p&" ,;Vap5_yݱ=jtȓM&U!Xv벮7:x`\MU Uz-M:qsk5/5O05]ⅸ-vYn{0g,xLtT:iFo°iO$&,l|DzD_~LEdg-a8z#27]a'*viAwZFD&13MRRYںãxNzO{x oMwC`]<{A8{V$=sq B(0WVn*1Kmml-ZpN\P!PΪ}ԛe,rn6)l/mjrKm)pJγ= `GV'.`n<%8h ;Uj:- !B{^.8N_#8Akf⎧=MZ˫Fp/( ۯ"J~g#Lz ~Z nrS_PM7e--*WRV= 0߯qjm)(n5{ӷXYuoNFv!6Ż#;m>Ή endstream endobj 724 0 obj << /Length 4237 /Filter /FlateDecode >> stream x\ms۸_~ӈ;I/sifi^J]|J,%izSg ~G' h̹fY>?b~V#Dh(|yy4BY*c"Q@E# h^̊F+([v߱>clj???L^I{a`#J"#A⊍dna[*rV 4hV-4i:Ð+RI}Hw.4jqhTƌԶuH@xݟ-(vġN?bu|։AX'`P$n)kUC2or~ޅ@[]_Jmx_5ނmدa5 7DyXR#g-X4ZyĘ~|]^No+4q9q |\KX3jP[3r[Kenҁi-W /@Er2Sř!(OV|i__eBR%Xo~̮";6']fe6W ު4N4 Vv HOK\1I;'I=mz4 [rݡh^O56I8U=92#ư5R\OGٽomL{ {h4e_WtǴ" ǖ.΃%S}oi\ѭrmT73؊e'Ib]"1݆ZiL}`y^! O[ӳO5oCUOyp}#J31 T鈚:nY̚ M!Zˎ%ژ_]8xoxZ\, Zg5vaݡoCaZY5Ku&)  *ّxGi 05ՠi)2|Rͥ|3RS*$ʠD$$v& (:|\oKjR DKN?вAK5 -՝r DJe=& {k01UVS"Tr&|yaaB)ҥq_.'K߈jFփi-X)X%Q/ )<蔽o?8jyqs ߸+%so+gO^|4vW3vikH 7{ɍM =x 3o>pPۊkx,ƩNWlzt8/N[z:i \hq.B"FjL9=/ c}:nP>O |=%+Y&w&N[}?KQ^nAbv9+-wϟ5<ϏYlgh:Zղ vJfMV rpE7h2],ʾaP/p:kX!P߃ՔռGGhuY1WiR݌:nKd#c(章]dHưnM[Xe,aHezϣ=ï2յ(YZDYŪefy* [. J(:VƲlvb]ek ͞o=[&:*U mRd* $D*O%M/zo9fka!d:BXcHƴ< T!ȗxk=SL_,[z_,5[-h>Yp0xf +߼( 'i#OL]C~>&G]`8$DfW I\M-ò3kb1 T`b;8\D]% q+r]v%)p! =-wC$$p{qe6 T:]C8h F"W_ G>4AiqgSW0UMvsNsC|$%t0z ,e d 1HZA pԽ[բRoAw rԛ`iPX$^^r$S@b]T|gOnqB+; IM &b0q[ݨf7@S2= *_2ox9cCֵ{l ٨u%ga{2ah%ܞ|\Gp@4݋(Ab tӡRؙ Sb1Yi-@2㳗Da~U$3MM`u #_:kRzfl٘KKLtu@^vVu8U3۸ky9Tuc~2bN:X8Ī XV󀅀.I3k':XD"ɚ|ȥXOxIϵ濝K Қ5sޅv̮, ǕLp6 >E]Wh]t]If|\ӶF7Vh7߬'%΋(9!0Vcvźz_(+?Cc>?;/7z8>=|/aͫ pX$c!(t 29m^1~ ('*!1 ! _e~d Ԗ{Jz4}wfߊoA;vM$xO(nw0 euPFJ+\Ll8~ xn>`H<ۦtp< _uOU1 &@U%h1WahDlnb{M d i$'ͥmݹӘQSr1EpMc6]Dҝ2)9lORvQUis[VnB6IP2kI\K=kGIN^޷>]Qsu~Z]d,4=p\mn7!A> stream x[m۶_~itWN24iM2$3I<E.^H%c'L? @X>XzFf_=!'3Jtv};SlKY}ov}wԼ :1ۯL >=E :ߘ/R3mcSEaZnguδ:[>j} O0糗.@HԐBx~}/|͕U))`efhFy[7C i:2]3vޔ<Ҽf[ak5BHҤR‚m.\-2bRTf^f%ΊUwl35wxg^,jS]T˲]!ڴfnoLᶭvn"s &iEu1}smB,",5l̷ E`S .(_4s|Iͫ+3F\p7캒REDҝbCqo,#$4#':HXl;sƑkx Œ2njwh꼼v[v_4wXnNxLfUs>ag-zd7z:; 7m]j-o~қiH$9Rl%#Oi'e CEģwޠyX*NGVffl@n-:n1w*4X-a%@db[=t¨bʵJ33+ aP,x6"M,(ZX0m*7틷:]oc zU"s[u5"umD KHpxx3:mj':K>oȄ$bIh(Jurw8#ٝ]  ,b#!CvQk#z#$ҹ9P5g:jU'G4>7H1 #;h[(v3<lj~%&VP՛r|r-y(S,;6/hƘ> }Mq (4 :0>SEV*Jqzx 2BG]J>ub*NIcn% r޽yʓ"^Aa(K(d( >4644 4vɗO^=-'3:w E'QlyOdFXzuMg['2G#X2adSS""~gQ?^Yo~Ooe\N@d$>P:q#e2{-JAa ,'O/! ,)-~zVۛc% `q@Ql#Н3@ doOM"zy ۖ|!q*F$XwCIDƝd4KZK:dtVtL 8.iu͑żp@^$6qb7aOء*9{ ʉې֣ctH0MK{u$cڀ#.<}me."Q]_J\1]4X~~.wj7Z8?\eZ7ݸ7ee]^:̟%tؔl^:s䗢jlv^)Ī>GrH$Vb&~_W+,!j_e{D/1%DWyAjrUTy7dD0|gWօ>Џzit˦nL TYF}}^&qz}5mrȜ~:L|_%zݦ`Ewb07,0t9\$ :;&b Ch: 'q^@Ń[]zޭnzpqUv+pFws;wa9:;R L1V$jySwtb\qKpYk>2Frc-B)ѳ'xQ*B0SOFGxr8%F7zkwF0?T{T#h]A֮kzXLfQK3$5*y[LAc'2{:[C4JW1] :,΁~?b2L`}h/z bz[{>lE4xEq(px*bc8f )WJ%C^]ԙy\}t|ρ87ϧsF?K'LkŨLFݶƱԆmOv^DȱK@guįju9b?)*۞Ϻ{^ sP3boTu" g*"{0yWvhIANF{Zyh[(fHh20O? xz d 6 ^3Dw8w%H:6UaS3 :9SМS .;1 N8Yfm薛4N^7V|_.5i 6=y-("kQU:W;FiOJhNy_z63Cy<[#Po}TC:$wPÅۉd.XJ ۧ8RAFܻD<SƝqgdkf榰mLb'ZIdZۇA+',l2ëp$~hKmUμTfׅ|1 Z۝4NcqDU|*H ,;v+|"FtA!bO:Aa/#6)8 ߠT.s8L~pHsJ:JS>Ti8 ?S"> ;ͼ^M2NiYcep~d3zwΉs0Go;du{~=8 (T8^rI&.wtKx ROLʒc8@ǣkF.u]lGnq4YQ"eɅv4:.y 9C.iOہkzvZO1x6tc')IRjyZLn(O09Lp2>f'Y^L\1QT DF۱W2rH=/]._X' endstream endobj 744 0 obj << /Length 4888 /Filter /FlateDecode >> stream x<]ܶvd"*)Pr٤*gҞ/~uA䀳sAht7P[/R͋_|SfS7 -:,f1y`5k2DޗWK)E)rw%L+'E]6eC]4K>Tf]~WM1cb)S8 ic&0FzQ^6 KZ?fs~~bl&ROTD!tyXn16!4s4 3Hw,݈Ē E_޼Δ.BT^-V?.PPAne"_|3PZLyS%*ԘaY۞i\z\0->Nmԩ\I2HT 14(uyHDp"h9!8X~x Fv{AD}0)6n#ߎ4gI\*ATW$Gq5"S8m& %c\B - qf5mY֖*4㦽]-ϒW8fQHGPx}/yP<"hG:Odz~(ȼ@e >`+=[h Y&*z~(@`h݌7k&T1@!ľwESY˂&pEOE qߌ ЖJHU4zÞ\Mb5>pj#u8MQKE).0k|122vۈ7X$7(8.BKJ ph6' @i.CPۥ \AZ'lzSo/+GGnʵm׍za8pIEC74{m;nh1UXOc& M1-3˦E?Պx `#zi9I@xfͩ*2x~My~=?G(2H'1h-rb f\DP4A J~|dKP!MHeϭ1eAF'tnv)Dl@1GG$#i~׸oM^GKT8"ZĂ]byV&:bJəafoyx3˽~}˧}ܩ|\$'R\Gϑ7D"Dy|HUKp$kJ l|=* Z^TZ,/@8VH "!v@@1] Q5a`Vܚ˙"&pq RFbf]]weJÌ$3]NG²,5ZR.Y颦:"kukb[U4X}Քw[鿦a M|۶~T(sxц3Mwr 2ה#}@/M-T.%S䃗8bM-@G1sOfߋ7uŤnރbVwi>- g) <"8"X]t|(w5e o>\:P(; +1Ts;- ΰIIإG>%Dfq'_˭ ܡϺ! um˶ ]nSZUejS_,\k`YC 0(I-#M̦XY˽-l\7ǞێT5} ޻Fx?)Dh!dV!r} %C>fZKjfQO,bxk0y< Sp39#V!d).$]Pb!1<lp2^T /$ٴ496eMrrf;C*>h}'0<'@qz4{tyOxCE <ӌ٦h[DQJcCe@nmJt~+)#gC(-$Oo$mPpZڣi)aluYWmcyx)B=u7_cYjmD|0 bbȼÄK䡡w# e2\p*p=$v`e:r[wUQ$ G1RtY!W(R <->In/e;k'~)c ZMI>vx,ݖ,|U?$=x(Ɛ'\ho/zg&fPih^?A:xِuD> tLpg'X5m}\= ,4= ; sIK]!ѰJXb7 nՐԃcԭ`'vRogc!оbDY]3jvuuNTC]c~#E}wo&3'+ILz##=Wa_^4ge<΃; _Ӡ: Jv40z]WO8`j8Yg0j2е&ui`Wƙi4t0ACq W_ܕrU!ܢh/O+ A1H;9ʈηűJjæt~rzV+?` |3z<b~ CBgC g#F:$(pZ~YӺ}vzmw Z<3#>x@,Lb)Ӄl'tS%-[( {Xv2K(׬G O1{*b`3.>p$"rKST9}>Z&8mb}Jŝ$w*'EBL1r|cMJ1P(2￘a|F)Hx|eIxT u g* AU?Dlߨ|#v>9~$ݝp=,3A #UL T`kL~=s|UH]Yk:I\DEڂ?Of)庂鎂$cDٔ f ]2E.?I+wyQPFZ$M0$ 6\&1Uw{rIk=@6A9U;5zf፯8o8K3V/^]ApP}tft9(BlFX?#6lLyOf . Ŭ&F7z: {0FFN_:OPX$Yvճzu׫1S9SXu未?fJ@~pM ;\dKE-L N+IYnƎ~)M=9:pP`X.K̩ ɷ g>7#:T(`)˕)$lv_W/aאnWE7i{~˂[֥߲.x3:RC bOvbJw7] kP$+AW'-EF>q0y?#ThD-nB;eIGD Ukev>{7n\9FYݕơ2AKouBl(ʚWByKAHbЗ7/+9 endstream endobj 641 0 obj << /Type /ObjStm /N 100 /First 881 /Length 2138 /Filter /FlateDecode >> stream xYmo7_s9×!Fm9] +XgY}f-9^jg3pLلK19!O&bLqw W6!=x=5 9brd >r ĈAĠC`9R BJE0TL% SepTY*@. *'1 g  &' a,l~5 FY!L0G5Orī 8X-)YURTQ@jVa 蛊N akƓn ; og(HIqk@!UW}`@?,oG`.ꣂ6GX/.ۛ5/Q} ;tES1b4hbIĄ;. c/G9[x$*ԓH !T`tO̫ԧx\ EcG͏ M3#yl~w:먑zx~ ̈J8Q^h^7\.yn fj,Q{iMӼm47JŃv;ֿ>jewޕ~m?Lwc1ɤ{L3^]pL{5w:MuQr5nԿ?jN._Q}.}spޫbKk6@#[5^F?2O`þwۿxʿ:;}ۉR @E pr  ξW%^vsvEpVÛ0oJ,ݻ_ܥYlBEfͬ:[BJJ(l#|@!:C|4qxtkjQlid>De+aX$GEi|8$ق#ӘuUZVAeH2S,IR%I |JJm) #+qc.Wz98KR%TKd3[-ʫdY+qcEjE׃+Ce:QI,xhNV*YA%Nl%VȥT" ʝ@イ^Kpn+ر-RGPh¬:\U%Py%iX,ŕćJr%W!MJ+ *ym|꒪ `wPq AQ:QFIo}rzQGQvJEiGBEc &\1<*䶨uw:]٢+]٢+= a;UY{:QčP''hNe"JGvhfv a>x0}7CVOpW4#7ZѤF>&"^O<_a;yܡG^x]_%h!`%4}"[lg͊/ Q$*eMv~ͫ~U?ss}szyylaŧb|.-|:9.'.oŇ>[v Z@ (k2O˛Pl&l$l &!G9ՀW'#;4  z%|#_}['(d ,-=rz 6O>yCfMnEs<2n8Gnq/bP0ePSӯӻuŬ~%F u9"c}(}# ]ԞO'n}:i.=rG_ 7ޚ!iΖv>Xdv菔^9R;/L"9g|ٟft:og9gbum3ϠzYKڍCP u+|m?OG9FW_ң,€ӹwwZxC RވDdVO|jrƊۉ|^Uéln厍5HK$pZdAn]UzCqȣvLR$pò:5λy&v6vq|;ԚC6$h)[H#(Hr endstream endobj 750 0 obj << /Length 990 /Filter /FlateDecode >> stream xڅVM6Э4I}6DjeɠͿ rTm7Cqt&?n6I%1/2v(Q$Uڭ2adhЯDXݶ!ϛL]X$կkYj 5!>iLtB @w"75X ۄ5 D1܄d: pTӗN39yC{0kJGc+. P8s ObJtN0p|G'Zp93R@Nn|zK3 endstream endobj 755 0 obj << /Length 3703 /Filter /FlateDecode >> stream x\s۸߿Bؙ}hg7;lwtlQJ@eŦ(w 7}ZqIgonfgs1{MU RXw_,RFtJ䰞ͺzo߿&rF0pFLx20Ѯ?_dytu_4>ID3lPm#gr$r~nju{1:  #y=]q]7o m͆#IfBbDHsi?;G;C%h|U식Mt$ i"ڎjdG*+}o֋p,_b]{#;5^ (f}3F2泴`&@a*se!nClBQNdr)Hːа(|],f3%<_;r}(ÕI<UTBDU~&~{GRꑨ/-;buf+/TL#K_*=ŸOEDCragÓe u.8e0.qgӔG+ k9YvRyJG Cp=@]Kwp4/x)@M! 2v ~Ɉ{WT )W/94Pc%3rq^ pn':&Cb0Ch?Zx@YI6F>+,\b=mx';*]f}.y_޻ϋڗCYݹN kNRB8 Uu̫4H;r2֙]q{EoUYg}φ1Ot-0|7y-Ů~JӫӠml|6m8k/D#`b]WAeQ+Xx߫i`yf&GfU-j%_b16LVq]b=b_|\vnܑk%x\w[3](F,UpBr2!h-I#8`~~?#)j&"ٶkHf=P,6V]Pi>C$=eyX<b.Z A# T6~u 5c'=0+uѢ\m8F:;I#"7n*j#ՙ;-vs(P=%B"ɪȣ12TRZioO@+cb#iE B{q] VxkvSYBK0$hdAiШvG#Ca<H17sw[| mVAa@}e@yUeB2i>yc$`DP*x9*:ok]=$%hpL(x%*SE|@#]4% HS&3Dh/ vK?Y3 YlpCGrX@sGƌ=Ln#rSҭN#-x#Ȥ[)[σpQx@~ ⾴nj M&)(;&wl"㱌|>J6[)20PP#ќ n/Y F"ya~~3fЁXZ3@<|j:@*1z c@SSF M6)8m"%&6Tɻm)3NpĉRП! 򰲏–HىDG H>3&£SCCCO3|B'&4ʂAM r>^,Y|$Mm6&Y4V<&_bR]| E+mTp xj]ƹyŦ2 fe|G$ɱ"8uaY_RJCBb_t Z ףاqj.Gh3!LPY.J&ILZs⤾/:#*@_dp_I40C)sph @`df'$x񑠑r)-R*@SួHZh f\ڝ~2SeӼ5ڳ }"MT T*G#GC+B>V,sۍ!V$S(*)\`Ul͞=9oYO:lz"Z]BX8ADL@X! @^̦pG\zz }u3ו[Doew"wU9kH#h :x4'#3? 4Rx>r Qzѽ:})~;[3$6#ݐDԴ~|Z_3jp'U?:W},L:E4`\ lR v$:BWa Bk<mF]=F)P< S'ޓ,iV!tʘT[ @%UDfPE;Ul" aW-\ hR0ONt"#on`D?놄n?v֜ǝABWv38v?DA#HF #Q#a Xϴ'wǑ$.$7 -" |S.xouG#ÍlM3*D .C=d UbR 4g̲/- 5qusqBΞ̀v1NtGiR:]<!,/j)rE{JD#3]z˜[WTڸ&!lE&~I-)C~C"oňLY9F{%I6eS݉pCqn7׆>GFOS)}7 mT|ϚfPWbgHiwL4;/4Y]K(ַO 0OHMM> stream x]sB/ғ!> v&/4Lu3M@K%:"e.JM )X,㷻]2$w7zO$qddrq5Q"fjr'\L._"K_"Ileʫ)AU.|elJh^T_>y0;O٘ϼjjӻ*򦼬 VwwgB-B m5b^ eӮʩńXel2O3˵QC#Ѽ^U'}Wy^z Nf")UqKԷv4 9@A,G}2$J=JĊq=d~Mf2Yc. Č,Q1%zpFߡMJ5qC]$4 Mcʹe oDt3hhr<[=JS; Ĩ)SߚQ}='$fiVlJd= , fZn烪'T<@o@bdPAH,!tc2JEX8srJ";U#0skB1O9koS1@:̐1V ƙsN_B1I͑LiL%y#4x3st_>mBD2)KphsnLָ%` ΢ȗWگ-ʰ-P)lW z}ܔqĖ:$qºw@w0 *fI/ruX|9+GnyU}z켱\V2J%AهẀMf^y !ؐ% }@t3LPG̀ 9]Vu\/\>DD7ZH4;#{-nl<_7 jQݗ͐o 0B>Op6 m^H-8>n!/M=Psd,cq^d0Cj1YݧMΖ3h k`a*A4.q>6 G$&Qo[x(Gls@pHYȣT `>GP;`3!ݲtX25 8#,({|&(Ϣ/Kӣi 5-4f, : ۶>] ˲j`d)o&Ӟ( L985 <1Vϵ0b> yLIz>5V7VnAe3x˥op)l::͑nvz*_Us!UycJ|Ukp %ߑKqk0{YY˺ W+. px(t !9aݾ9m"HDV`1|jFr!\<h-W03vidwв,&ްp]: 2>\, ̏e^O׶Be~^|bf3\h,3/ɱL v@瓼9DDEG7yk ԿRlQ/z߭53,kǬ$L[n?VΜ1Ҁ4~=EnꪪcSl/ʼ2/ڼ}@νksS'iչiiuS*cPag.=ͱ*I~<=7[K\ui4T%a&lK4$sXX/E{=R (z֔ܮʹ3{S'败V|K+|^n.3t[l6t#6gG7$pvF>4uM4ҭ,VWx֎{d~Qm(,foMuS,>r:YVkSWTq! ^_ oKm-&7ȉwN^]V(U̽)R?+־KJvUXġ,~Y/DO*jGY"Y:sҀ\w֕1pR9E*c=@,R{mXdp-I;Қ'c(?ciI)x#^8f]8\^5|; f-%יVT3_b1d4j.ě <-ĊR=**-V{oyySa\;h8 -$Vף&~iITAi\uTKP>b"Cs%`/q0 <CBm\XJ?"3-&FoGL_ Q&W~% k Sn]s7H1G~>fQ>T7W{tGhTAl,|@ߌ)]=nvWzQ U?5tPCeRCih(77UF`[x7_ʌv)yTs:(~^W F>&e{~LӃgޝ"l*$/@=_{u귙3Nĝʫ|~%8L+[G(aBEw~Wú; UֻBRXxd5ew~P;щQ?|a 52$TU(Yx?rK^!:<4vAޓ4> stream x]۶0T)vD4`6ii6,pھ47i]BTfxlRȗ}.zqGe߻^yG_>z<{y?LN==~*bӿV Ž޲|nGN;賘P h:<ONVD2:$4~EіJo]jF#K{ y={ICG~w^{ Jw,5+C#Y1ӯ\b旤 ISLBDmݿ$9sBW4PkZWt}NrMw&UJ245cI<77{Kp̄@: cVj-;g[TqX;xS[r!#`%$F? oB҆'6BUY[fҟP}<}|U9>@Cfo`ŶBdKgEn꼪*A؞7(S㭵9vX[sA¿twR?hLuV.Y|4xqC71?23.ѵk (;R:|NT&_1-yxLkD ȶg5 CǓ9Ea8ix//18fvb_9" 4 Uk % ?6%x tS.UZoUGjZ26''R 51AZUtӽ^ 7׋8ޏri.ldl0vmcS<[bFVf¿5Z{q L-o7iwo8ݰ_3 ܼnrj.H]rAtoqL1ڢ5xmMN;KK|q1 56T7 DZꜹS99ԍڒ(詯RɷN! /R7P#42a`,ѶI/+D6~%yn!XHv+RX4VQYGT)rG(h&p /1C[/t Kx}Jݣ7Aqa7wŗsP^a$D3`pp01/*B57v8l@-I^TN 6ֹ7{cp-)wn]R{GFl^#B!KX lƊ4Eߓs2ѵc}Ɨr \!BŠ uK~ǔ(_ߖ_ߑwF"H(o]xE=wrkи0G"iQkQk}Z}M4j-;{2JrzPǴ#P9h풥jV>\=5D9B ,8@?I~xz%rZerSPj%%`\C ł)}E%{\C) 14سxvvq=?PX}=tw"z0Fg$ \M&$qԱVz@0(o8VljN#Rߺ9P~Ԧ<ԓ+T!ǁ!ꊒ> stream xX]o[G}ׯAs|p #@"mStw<(q$ACJtmf/}H.3I )H99=zg  >ׂ'JB@${RMe fk-dBZM:l Xz8qX8' P34̂/YK\[LM *V&LaM)!`+an70wJOR+%uUY3fB5p kj&B89UѤu𨙰'`Rz%FaBLlP̜SPdBXpAPj8}y0/ ٌ(B>N٬%E'Ă^;W+<[_u(anv~XlbY-O߭6ӏ۳g PΧv qW3;fk3׈%Hêܥ~v#*.+=btGbEF~.X/V"F,[sT78w2ǎ$q/F "I5,H|߭Q*{O`d-Y;`gÐ{0FEbT>F:$*Kc 7D.pE.KF5@AQc:Ow= ÏfM|ƿiJ_7S6ܑZ#5:FtQZڈJYDp&za!؉)n9|)`]rEP0x7AZ2}Җ [RyƅE*6ۅEeWuR]|#yNcNе8yKT8b K<&Mv#-bn۸9Q/v8vÊj]D߯AH_0&5ch]D!Qm?*cIAmjR"}6 cPDPaQP3f䕂˸Kt#؂Y+܆ksV5.XK%jYIǛqx[5-5I+ * G&'/ɛS#Dyq'5HQOA9JNZ ՙ$ou&>V͙Pcs&>ƠNqaN^Lݙ/A1*3Q |>^@Gc_ʱ,G endstream endobj 859 0 obj << /Length 6946 /Filter /FlateDecode >> stream x=ksFrW>LKxޏ\J|eo]b|^\Iq%JWdZO.}q]鞞~wπL&dKzRO_>sa'ˉ2T|/阮ߚȤN7/`Iq,vl% ~@?I&Ӡj:8 9k}q\e ݜ`:q[4T:RqJ*cϳ̐9ib@ܞ͘>$xw L?HYLWpK~6%dZ&λS-;ZPLנt[}|寛,@9=j<0NnzAB/]$`2f3o5Ei eP9xL- cP0hמ_K[O~Sz_eQ4Lx'ՀJyfz =G#(q^ֈ]_y$A䕄xCLv-=M2S{,& G}R>yx|GS>K}Gs3GA|D7=0>3>&Ft ᕪ>xg6#Z XE;=AwAe(}zMeZyE0*vudB}=V_}L""el_  PUBJRkk:Ө6MǼ<5tK_q76p5KT9?P.Y\˴P/d֫٧/nIeQ>&Y{,K ?W?w>L,q}ƕ>A&6ZԫnN\QQ _Y!޳s~ItWO|l]| 1 g0`K#S<{"e>}e= 5B)ݚz0 r*U-| ymxI%i${<}~Mwy9KNbOs?ƒ6ڳ~B7Yt1U&Wu^{Q&D]jTtE.=rqΥ_5:PV:pmyzkdmZ2ep *Q^/}ɕrO_u\eʳfGbN Ok7XΣK $3Y*%y)qr,ɥOa*0~ѬiM='l+~_u{>H.Ƴq_OަnRrs3g%$#azS?YU$"!N)ץIUYK%O ;E#wy#i@YF:S+%K?]̭~2V~\ #rdӼ?, uh֨0Z/\:3lM TKm[5/_ۂfhƘI9%iy!)0(j;v8-.>*5b{qv`) ~~fau]ޭ6%TM(),A$@ZU0JX/NS͝/Uh;uSvo榼q]f{L7u:f<韁CDz-cR(PTTW+IuHx5\| HfLA0_9D9׋s:]/6%R ;xgؖo]=U/bXWKem[G@8Se^V2G'Ԍ>Ya$1 %.5_)tgUaL+GA8oSDҘfq˅T*0hxp*aGK@(i_//6ʕO$7,cjR0΅àL r f#NyhHsGp;p$5@NDmh΢kK!M({zpkR#Z{(Q ϡ8h`)C|}?:! :>(`>g3F||`kdVcq*(bT߻OKߣ`2&x .LSF[z/cricBA&^"BDQ:-yUzZ(tٟh9 z*Bk&tVg=2YÃ5eQP% JXڳzpjq3` E-a ~Vۛ7 9'! xZI $i0u8ثϤ͌tvA1pSܑS{Cz5 xŻDƪųO34U+OΞJ7sG.擁h0L5;9cy'S]U@2sPACLQT H" 1JBG(2+ V#^ [80u~5'?+9mtx5 Ẃ20@=qvr+ I,TFNrIGY4'8trWrCGyfڃs<3pbk6e1T]6uM(t:?su/m: ".1D xi[h ՗h *;@mWzw>`FsV 4R>]8XD~Et?!$&03>4DoMaRf%KKp#{!Րդ.i[MYk)"M@n,b[;! 'Yz(6ڍK=.Љ'aj$SG;ft-FwK%n,LE0>\TlDa ƪF(\JʝQQ3HRƇVP Q]E‹LG/NhL2C=VA)5dpdFZ6aUa [/^ˬfQFJ V@ȭ'_hkR5/V7$pgD x3P*t<` 5-h;u8WhgfonɤMFk.>X%B:8bעKfIsD/\):t#-.E?ݣMi G͒RcvТDTSxeC 8Ijv)-?_7.HI]kJ@5vG|Y|۳J@gEQ)FEthԈ^B[\ݯkw_$mHkE}~1C51Yjhw"=֗f$T4Ҏ.z@= שoJ :xP}joxnvަ ` a焣a{AN.qH.~+EP#8%* `t+.bIB"N3j.8Ȩ@!qxD"1II[b€CtHBKVк-A#=] *Ʃ*,uH,Q55`+聐'0"-,qhU0&< u寫  !/M ǯŀ *Y/_ p`zi_e9^Of" dkZd*$#aǢ_;3oi(lžP+ԋԅua("007 q\OprLq*Zg-;SPk\ZS}kk9D=Gh (-XS+ΡVXtJ)hyYklX$lPQop@8BWפC$&R eD !2;W1bdw`Ĥnϯ+Zv.IOv+wetwkZxn!H#9oнu uY޺iڿ-CC٥ag6b$@QّG,"5T_q,lnZ\ia+ԝ!D7ga'UwO+,U.\{Y R$t>=1@ h~ԝQ`HLۨqp-TʠOr*@9HDEԵ6lrFf hMa\vV;І1p;WM=#'*Di}j]a9jiڹ^Q#j#=׻Sk',3 nUu,$qBcB6Pt }i8)`Pν8Zu,T2Bݪr>pH4P>0-瑂bd)rHŀ+4WKHT)d[v;WC1aJkȅǓ%H+L1arx;[xbVʟWo|Va,hTcNo7GYWʿ;U @WW 7fH4@# axoUs 6w΋<.Nj=[a>C|ZؼsmWZ"pןUbA"m慠]GTw' T С$Fqf&;&fN6 4~=И-sٷ̍[(.ձ̃´mQ=Yj\vUIOہHh- q#T5B87*%iJT)2ΚDS=s'aAI8E` #2CJn<3G[j~"9oQ?EÛe.Ÿw~[V8vFsVϮkI gUaqܥ,>ԻCtw[e7mb6vﶟM|#_ {y۞q2brב'EpFpJp:A(zBtXX7z6XnOSo~ծ9bZ_zqgoVSDۡ/Y{-ۇb+#bGHo}炤c8 5+ 7n `~}rKW {{gZcvMhg}:z !C 7P/V:Fm`~$/ h>sguo Z͝NbP͹%g֣ endstream endobj 889 0 obj << /Length 3492 /Filter /FlateDecode >> stream xkoF{~澗 4F)i`2a ɉ\>hV[V+ &w,?{vBi JҌ޻a}BSzE.&>{(K,Gz<_% ~z)Ԣ2Qj1E1 Ep1R8&l$NtzެAmP-٢0ky1].PT[b]:H?- E"SCHhnʀ'cOLX%Fr|oQ07\f hŠ(O1sЦPtzE}x5dđu:n"=T3GzόȒf5єnRpzsgH$x?yF{s㵾d~4I6I^ez˲Xם:(yYn8L,5ۨFMDSǐ X .ӑ9Kԃ}Ls&her>1 _Sld.iuXӋ>J } Eithi=LGB$%Nv'j6wsU.~Vd35Nժ0/(/Ӝs ,DT"|N002*2f"zĐ4Ŕn-l)]@ STD*)vO7㠅ѣPišnznڨNcQq (eV[e Zk}3BK(30*fPEpe % (,@ q`\dϴ0*UO%`xSX kQWnƁ? + Ylgqy+iSpY,NH10$pmȝ9pTneMb%i;`v4ؘU>gf* I*w7L3m QnRU9hq5E>v;<:^6R&pL9fU RHG 2WX`qllFzb :cڅ4Q*,$,0*$K^\񞶆/ V0FSeXnBJ!Of-EQXܜ/ 22ԸW"iUz8pinDY&7!.YuC"-)ؙmpfqy DAqIցK"܂X'@s2`=Q,'y~3n4^8;7uDA&:رh:Hkb)j"J={`1T>r s1G B'L΍A+t`W%GD+cxsN%Ww/bdƟ6n IfUչXTw v(a{I6\DҕBbEWh-ra]VpgNuXf+ ǭ!ҎތwZb2MPMtLh@ Ν1z'V%ʀ,mE!S+D@!'!c~^0tɨI3{IA*H9*^5&AagΨl`s&:yA:6C >IC;mi9lc3{JK0f>Ţs\+pn_NHrx}:#{[̝]o+g :Zsp`^P(+\;{'*„9Ucb[nĴ͋C3EGyՌ`xQH`+?&ٷ9߂dJO7.l7U*xo֫"ˎ基i%je2tUcZHȡt} }i1t/`f1}uL|^_6ʴ&x hxҾʛ% (Om³3a)Sqɏev֙nȸիwo2 S;ߵйr0)nˤ4RF+ [5)]Mmn8pVڧAw:x_0lP?-PA%K*)Mcu{czOX[2i%<[Bl]x@B~&z.z/$t ,:y׉Z Jd, ӠbGۡGlS 5xARsI`DH#&l)oC r`%m,Z$鼳?wkf SBaDv w$pn((A5"~4S_1;wAyL _F坼Ч;G&>"5\53gṆiPbFlFڛp ɏ6טE]r ͎(z"E EJrW YhzcTG(7r.'NNT{dG#5fPm1m >UlɓIT 諸ع"|%DP1G"HY;5RtVg`Km; ѨSfdISE*l|-wyV endstream endobj 940 0 obj << /Length 4803 /Filter /FlateDecode >> stream x[ȑSe(vP`g{:ڗyHsAP_vYYSH$T@_Ta,^> _KY篟}e-2?\,^/D"\$QQxYx? Vc~CD` -Z%Hǁ#b!5 o]Y0r f$5e"+p}ˏ$y-y,h*^ 6humqy{^:v`]Lj=h{*[պ0'a lGN\ 0ILc\yzթ2B%6^Žr2`{qiʶImKo_&9qNgxDG[#g$bumGhu9ZqemH pv\WbI9*I< ͕@ ą*[1r?1J C%Ο!drV!ΰ;bcDͫN^ xVhϲp^xEE+-rϐJ`P8L "b26P]9}EO??cE9jٯA{rvS~y dyQ Vq!dHHZT]Z7{եW[n0<8\V"~ZbB~|r:# -Rl +pE uթ C;:Mj{\nLZҕ )?ơnyE=~ } u5%{t P`ǂJR+;ePݠhV6>7tZj% 8kyW5Kyq4; Z'[$|l>噟h줭]y90"e ;=P(hDƺ|u.1(! 2~үY!(4 #7ߜ($Q_ì.>Sͱ[l>7Lp@IkUm|R)sc!2o6RlJ6˄%%/zeL0#`I,+Qd9gs0@UӜ: ka0TS>9(LtqI r33l79 ]s05]s05]35]sPH|!#E&/#\* <70N`24] d#h +4uRHrјMw$85|H.>iyBx=?*jD"lZbQpEþlm=${LޔEpN$iꢖze^ r4w,0|كPH=ntsP݊srΪCp3Izc4c$#4c4#$d jR1pNfͪC`fKALüa+?C `nDCEKaIKPnlK9ICMC9MCaMKǂ" `̵{[~Oo+\}_ymN[5lX0v;D!oҾK w=֐~9 ߰ OYQm7l߈߰E6ᝣrܽ ho A:yqKt sE6DOP:֧ϣ/&)a!coVi;@y) JGSPV2*կrn2 Fi{1=֠~ں -툦-55Ѵ&45&5EhWS4C,]/ cm\3o3q!{'Z DDkH8/@.vͷ y`vm322򈤥2򘦁2ae2AY$* as8bP px:4Fyf`30di0ιgTzQ3oA*4'y] yՖӫ$h%H-9)4 9G4-9 bܒT\4&ooD"|K2J~E >F&2p~f²6ZZֺ.N~છ_S/ C384ԄQbmn^ެ9uhbNkdB -3gaIxK͑[jlZlRD+r3?.t]83r@/LgpEЙepޮzYu(pNl|?8; 8aIxαp҈jppĸgGfIBΎ 'sNhEy"|K&gbp+6r7S5]b8!Dˊex3Zi2i*TWrz xo۫T͠XAi ZRKu.k},j}) 6WcH,{Lسel0i/9Fڪj+U*rCQD%p\>N-&QINj# KQ;E(O_'' v+%"B^ԊN&#++dil"gGr[L(lT&kkUUܧ_cTj2 OFj.LvRVm:SӾTL5zw؅Sʭ2k19~C{+9T=5 7YPU7 0oeBNx+".#֔(ktib6u^}lu^UM }oܼCRBqZuޣluq/N+Qu'm=jtC&i=O3gQVKn~7J:7q$?9Byj-!?#]1;e )μFl"[ۃڌ܁K&tx%<-j E]ZL>UV)-UU eZ|5̬Z)3Na ?0W<> stream xX[k7}_!Zf'-i \qh{f]:$'4͌d2K+&ZDb/ĽhtKwF/}6pS|00;/lXYy1RQ/8)~xcs] 5օMCb;jŖ?RN ԎV4e-8X8$!Yt.LOb֎-2j-epSeM6sU: TXק,#LIyqъf-_=_|,V_0bk|zpj~D6?z88W˼8\Au kKn^oOޜ],^6t|ni'WG2℘U9+lkU@6y^G~scSӟ<{|}wPΥ" N nk(l5akKa*BZ J8mZ28=G;횃*!o ;ia bK"IU5^tR|lՒ}(RX[ |ҒkK>`%_PI^{2=Ǟ |#'.{Hϱ(ףeW/,z]>TG> stream xے8,q[W2[P찳;/TN:q05ґlˎcdV]M&E(xϧ.#>BQD ]^b<) #F`f1~ҕxXd!WBeL=8%PrekӌN81DYugSJʝ,U9lE1ޏ'D1̹/9%/ʘ"'LHO Q`; 14qYV@rXM94᪉XkK_0tGFjA50"LB1)H/tkBP b A\^L("%͞~:e-p |P틒OrgzdI$?1L=S5%I`HO,-q[,@'-B?$FKv?{Pn{a svetv@9JU%j2DZ”OpJCx~uHDSq(tBڻI\8uZޱ'&Okq!d/NT&w`;) >+Gr,HCJ]m(7W2{ ?i>A+QPD(fD8'R!@d"y<()?I$Q8~u,Ii8\u|(+)k_C5ڏ ;K8+pn8X+"߃OK(0id}M-ֲNV^izK#㇈JBQPprCRr,HؠȴPA}-.5V4 ?⨱  X7S଄; ėrkcDܙ&m*!G~tG"x_|%8gPޖu`G u j@d羜<}L PfPN-%w:5-;kK.x,_Y~c$j:xL045Xk*@}LE} [lbZ-UZ .ݞEMde 0ٷzKCp#r&F!3$CZCIv ?ܟz;OPfvYV]l2ܘSI2W+6uƳES֩5@=^zt0KS#'=DV$| ޫ9liڠRj*+=!M@u{حfFAKY(S3+\Y)HɂLe9IYl`on}$gc5qS{Ш~N:vցQhn΂],AZ5# AI(2op]-Uv& 6[yeF^$*WiIQ9w.rIc\a3CN_&uJ4OP'/OPs7B?`; G=U.2]* deK9e +W!e@FȣaZJ>OtjoƔ[Y/k+u"h\M$oǖ΍}S$1c9`?DTtؖb{jh=m㨡n2N"l Ydb8aQl Y4m8r1/pSi%BʋEx\ YjQq$.H!~Iѳ9+@J"*p,O OV&gSx;ϡbk_䳶ܒ{_&,[weoaRʜ׻4ڛ t$Ѳib86 $DW>e,$ˤ.qS Y!X"ߚ$DbLR]dAVpdq6st\mg>+CF\k"[CjiaoR04X˞_cزëtLfsͥ~Qb7)NS۽↵4,pS a|݌7vzBN`6ݍqBeFdq5rufe弬,Mpݒ2Po?X\ż5Y>A ۥE::.ZwxJ"/*?h\ϖ ftO(g0h%VBIeT0UZyc6Fe^Z b&EBy\.@uc ~:}o {mުe溒<[ϗfdș?lan~LWlgڴ悗+]h 1v0 ~lgz3|hK2%@B&iI?<񣶅X*C:%F\\x2BNbmݶ.NR biS*Z]>?r endstream endobj 1040 0 obj << /Length 3035 /Filter /FlateDecode >> stream x\[o7~ϯˢ#'NfH<Enl+Yv%S!gĹHHtFz29$Ӑŀ >#j@I^޽h6ЂDOUNOp99?YLg Hʙbߏ1M~___M|!n#Mx>d:sY޸WΦ?.2M/-E9%x59w]Kh6W7jz=_n 9/' :[eRKQer.sNOz0sȅY9g|3M[qs~^v.PJ-q {DrYp7cw=szCLGcvW`n[S{Ӯvq nr+q:ѹE9%j 4vD;?0( L/S 4/?-,.G3D`ڜVs}BNp4%P4ReE6x}E^Td1KK߆#w<2L4)㌮SJv~1I50slc EMX?b47T)W3*[b; lW5K34PJ[)(QXߡp1xEߏ>[ q%-fI묳F`1C3ltxBZH3&ו ߵem|){*uX5d^Nڅ^S˛gI&g6mǚ-KLg㴤O_2aY@l8o|v[et%HJe'?隖/jmɡQMmXxqo!r3o=+ovV1Exi<*۶(Rbj޺`H5ؑ K߭Ee@YX! ,+\쯩 N/3rmGM8_rccx k4Jcmf>y8-ޓߊs2R\/zʢ[h_sXu1wi_%~U|eAam8 ~̃9?rF#4-- ?wګOQBC>[?{d8{l8Eν:nԒq9Fun4L>T/g) endstream endobj 959 0 obj << /Type /ObjStm /N 100 /First 921 /Length 1778 /Filter /FlateDecode >> stream xY]o\7}_݇jQhuR )$mC ]Oa;h~{ :<K+Q$EZRIRRw H&j&fײj-Iq&&Im%hKOV$3³ЖƀDK DHT3K za"nSsba(Չ6`.y-k΢ތ1η:Ǎ AcNG( kց<|պRDW_+K7n ,K#OBį쯀l'm.b\iu !֦;O LmZ`QP PIZ-]8)ۂlI/ UHX/Hi̷*!;C,* P9;Gaa-?juZ`8 ~io驖6F\|~LSerZ;Z,Rm'p5Y’N/Uxmk_3\֊cؚ:>/fY,z0KÍW{@Ό\ڿ'8.ws~'|O"iςn4rC@ɄքzNfAwf8TZiD8ɣvHR\[씩Ny>!ſ{;;():dpPI(GqZVtd$1{ CG0 j |44O|$v_Rax.sksmsE@/m`cˈ2sjAX`4gDs:"#.,1ǥ杗 5TAꝗ JĶp]k?oY|dt3oнW:Tm9wY"=*]6RVVWAVAWADXr$γlޝ]|q=!Z$0Zn>VoGvd@oUQEHzYI7 urk8 B q.\QPyxo-J R8֭[I[u?#|=v~#@؏wt(^QԺw+u>Kk8d"4bcHEQ\JM̓ZQn}jwqDE?@z endstream endobj 1089 0 obj << /Length 3625 /Filter /FlateDecode >> stream x]Y~_Aa+s8TmUJ)V\,݅cCsIREKLO@~onHsNF7w#IFQ3>~c-u"Qz1%">ٮYp~]a d2ZZ7;,F8F:<'@DdnO4̖dS`@|3ÿvm8d>/FGИwAAs$KG#*OMn&k5SF6 qaxʹ:&x\- P!ԋ;`tKSf{<@1Rt$|rXӯ33\C8K&iJ[u`oB|DJv>˧:20H{J ,3a2,-+N3W64?st`Y&s_[qp^Cf7,}޼a<,$& 0fW{tS=9f\\3nHcD0 H"BOM.czXhXڬT  F7( A@n͜Pa6sЙ:Z W6 m衏 y:O@fawS15KbE77p|@IXبZhSS!T>㚱3Eӓa9*,}d* sS7 (m8L8dFboy#rmHw,#k{*MpjIS7ֹsry18jy8w%~Âg+K~Z4} b^0 DCҕ@np)v L^_"YpBpP cA[NR1V(jw;x>0ٸ0"qY.`+E7d]g-/=l&4ǩS]W[ )(՟{K+|~+ x`j^6`.}{s?'5 endstream endobj 1095 0 obj << /Length 4117 /Filter /FlateDecode >> stream x\[~[~)I(Ңm H+쭒6ipF^}Jc<<\8|q/O-`xziŸ6Upaͅt˗+X)iln?ˋo$BpxD/V2ieMa}pAMk+wf]?\6ӷLWnsYzyןo6jžZW]WitB %Ls7{- ;Z3UHsGj둛w-L43Q<BUo*c_۰?9B۝B>Z'!&8^0L%nb:7ȲEhIId4Ey?-ghʛLЎQ=!cǙ>5YO \,Y%v&Q< gQP{ҊW, B'DH3mfFfO1DEJs`67M*dVC7M"57/,)ETS6/|kpZD=LX'O;֮42('.pƥ- {[ч&e鮡I)Zd%yx80C[p@Zp*#Gi<&4Cؠ0\TԍaBX rNN,L8qY^r5/]DbE 3_m3GoU!L_i3xـqGmlX6D l=tWo`TVUB0_\bvHzqJa"Ő>9DXS*%uD* Ki'uugBi.Fou~Bw℃N9xd)2f:,䧙@~hPQ0"coGKBD(A19=I/ټo sDJY~_]9z&}b曰q 葵hJ 1gzgd}KW,5'<"9wt-6)LO9}Dt^qۗX?}TW72a7㌻ rjJ')NUS> stream x]Ms6W(LTed+]砵XzlI̐#)$@ ~xcūO>JJpxՓ*'+TOWߝ{qo*iL |sE_}.jά TڦU9<7>u+ǂ㱁L;Uiuz -tdYm 58󓳟3^JiüճWg}ϫVz[zgk Z2!HeYjߔjDMۦpZ$ͮn7"XiiǔN d aS"TaJW$TdvL-3E.jҟGRt^>\]<}{kO/4ꖺiCԺZE \鷹sL:!O.|S7os:"-I3\ kG,H=:0P]]&y( w!so!sTiDlSsbhݾg8ƹMp)`Jm 6/v]UȌB_Z;. 7U w vBXCtb*mv qDE-T`8i,) H8Ө9IAۤTۗ !q] *UDIC>tjP*/u`Ps솣!U@;|Z[ѮLgt490R8uC;Ç(k)o%-< -u'xkJA 5C&x(61r.YQfFj@@usv3m!ȳN j^֍<&eKHf-`p!,=D-DK ~or,:g_Q!GI`@r@qv/3q^RBkfc`aᢘP I0FD/0P+ Z8A>L>ry41!rto%iY^ 4?La3 9bS%LNs'L{hrfQ zFS#4A>=S4y3/'+=B8K Gvl)}Mm`ll33,,qkP![`nVwo̭yBc}ڗ}xBap7T=ہ&H֒@EV1P$< b&Xv}3L5L4{=+FjDL]`nt10ɡ_я6:P1Vʚ( Gb^mp*5 TCXo6ʫHduJZJBTbg@dQ?b2*h "mKi>%.V%'ZNzy+2eҷ˞s`{\ \JsFPZN'es&OYNE9[@hc4M3 a>x4u='Iۋ!zuZ*^v%/rEeK0J/bXQjQ0RcTA%],BejC08":[J'gb4'#̪/Mݽ3b}& > ar0kÄTߧ\ kJOjɬrO'u嶙&&aG"Ť}fF-(vOm_>5c ndXԙw֦bؕ9]M6PMfIIr]rˮ1Pf<'=4;K'JjbDs-ӔA\Okiھ@OCIi kE 4-rQ6EkZ:߆6 Ut_5!\4i.9a+D&uWDD}faН;g^!K^^N]ջoonVo޽绻:)P_^|˅WC^\J`ZZ+Zp+eC_%Ɔ/q/U^=KwzMQ-f!i+6})˄K+e\&Xye+}d 26:rD:.A+R22Ri zͩu[ĬfUlM LiYOib mk赽sF{d8V ɢ#D,Cpd7Y2mJv7UYAiU9 r ccM2 FX&)r%SggdOex?u~`/]#9ZB梖AIGQ2D [dTTs)NU{\ǭv} jRE]p;}$%A%fyW>:}#4GG=t<״G¾**dww׏! ZƫqּOmi^fkWoam_֮v:<3` j-F(^%s'}!YsX(RQ)fH.cHkYpꋶC.ns\ LK`c [Xd"WL7K0Ajv«>(y;<8 ۛ"̧ Wc|;ԐL]c\TL#8m ү^R9ʐl!!f%uEZ՚[SQJIIQa7׹ )Qpj0׽fg[xbajJ?\ <TeIOxy#ƣEs5d oSkiJ7y`ΪȚ:Q\M$%fhaU FسU-:,"jP%|WK0FD AS"yZ-_"-.GʾD(Lv֙YrMk&) 'Ozz*Oky=͑ruo}JxqW>EEz*>W}~ &:yEl*b,YhR<_[ZXq^k!~Lf ü{Mxmab]W+Ll|^h4,*l*^SGA{{sc.&q\}'Q(7%褭C8n6RnX4bn#} !+)Tt;Z4 =˩iN%pDm8wNUj%S-#%eĶ#@XV2+G"3èx|/`TbT:&aTÝ66U`ie^ZM\\PEGG ˉbuFӲ(|n"&cTfu1CKS1mDB4jy v nH&S\"J7Wc@IyByq,5Xm"}-(lHvIH)1O"O}O9y ESW|кKJdEY)s*ܒV}&711k<2%2ɤϨ%d2U)F֨80n&COw+wٷ},/ =BlFG&u>>~GXqƫ1(ɋU_?hw?-m endstream endobj 1106 0 obj << /Length 3487 /Filter /FlateDecode >> stream x\[o~ׯX/~)` CF 4΃bY#F^t\.ɣ]rt9{7\8$~x3> {iŸ67Ow ϯnno]~f盫߯}aG]3z"ߥ3o Y&x 3r|&eFp|6Nμ Xz?葳?Lc'u34~vA˼g^Ά _qi33X0*ZU1d:N3yxŸ`R1+M' Z$Uj%(h\0 +3^$8%Z$0GQ+Twxʬ09 #11=Uf Ǵdjz7HxK! #T$`qU<_XYP.pN.x4"Jx]u?֗Þ:dl1!PR2vD BBI838{,+FzjcH}ub%ks2 satRCTOc-QMJ4yI$"&alk"$ VcH x`BB>3rK+GK" -@L JLfeR=8qs ;ĄcQWKR لE9C4 'i3l v$g<GVlx4"AZJ)z*%xP@r&@EBju12e2: ])` ۬&5 Y^Y΢(EF7)0BIJDhLL;-9܎cp< $v wr  )K-ۋW KyePSGH齄Q.LÒ!YO#+ZaD]dB:0ޔߏ0.ac݁ C>"5Nn7 {fiP>}Qy{@y~Res34]Ij)ptʣT񜅜w wn*<+z?$|5)2A5z&?D j Kw*8 bZ+ u0]{x d_LM_ r_coSd R8c?[G> oB>~ _njv?$1⦖LMNqSyQZ>X4VͅO舮BkSX̤x,PK9\i}viRS O^"2^ٜV<“ ;y)f:R~[DDZRGě90Ҁ } E JeqU :eX[f[Ķxs?)s;>7mw1\>Mi_FqhM!ցX#MM7nf]&ORt-M7K$W8 "34Q`FyE8:=c< E>;NRpJ]V,&E_-f|i5h5+]cQfF6OՆ oOo_ӹ۷ח/O'vvsW7>Ϫ_䮟rRNJ8 u,=v6wPU"&ȾKcd],<;ch ʹ eĠm;vNRal:jE)cbGN)F*Pjr6bTI)! S5Uuw}kVtHӷĬDѬfT`Y)gw*db'`Zv*-yh[" ?uP{ESMG|M}wI6= ,V+PvSk!scN32A鿴v*P  zֳ<ןfH:[)lЊr/Jc]=C7HFqQ9A/&`Ҝɩr|l yb{OA?z#cYح%^ꚡZFj68pimXܴj~$7mv{k U26I-P θ(Js'f٠i>*O.e.MDYL!C?o/ ?i˩9֢Jn[7'uڽ¬4I:Nԭ]ؽpn噫̳]ts&-<:,ܻgCM> stream x\IsW-`Yh$唜T*/)$h3= @\Lu[‹w/yϿ; YAoRz&D(̚Ͳyv}ΪfC˜7[ /4Z djP 웲Ͷ1nSfxsۅ^BTPC`W^,3l!6-bMſi F1x=B1aTcdl‹a^]g?acrWD殇]AH{ܰ Ӳ!^ifdc2aO[[+$4FJUs PcvPFRdD1o4kťS27ͳ]/1\sar`Q<(\0uX)'q Dd9|,֎Z 9 zΌU Vi)7X%c?`BP !X0d`S_qDK1Df?Kf?=\Ε]xuueNԍ 4 ^//;j|'b]VEQ$.]oW4P[Uwr8nvyW>Ynn[̻su}yX;bGi4YB "qlz]6oϷx096Fg3z,%}WUU qi\@wm5Z}Xש"Udn9R$$LBnM`UDqҘb^]2yCHŒi 8DX鐟vneƶ!v0Ӽwݐ`j߉A~O\>$l`qI5<*ͶEfowΒq͍ENyw"<#T5q1^2j>ɖ37s>Ĕܒ'qm-M&)Zs?l-ݐIIbR'5 Oj&+G->o>f xWF5m+FY"J2O,a~DLчpΟ) 1OY\bB2׷zsW1gݥW3EsU|L9wݻ-W%RWn9Qw+ʅv]FHibNPxܔ?]o6[/{zD/5KzU/ziLbW2L̔hNݮ~{\\ 6u1t\cܔMb~_-nn^Os9 jyn;HL0DEBv `ZL+]/蟒+P 9"x{oa I)|CB'M1vhza Ry (h1!H5irUH hE zْ:BJ섍`Gna>>Yu}$ [dxRexL(T8YGR=!lMh?au5d1fh(ӌs"ۼ[2pBeF(H{3BEόTV2"nm>uwB ڢ0 њXc>Cc1apQ կuh;af|al,v+jD[CbЛ>a9l`0G i(7pKݭş|~]FMG[_CFL᭖1*;Ě̝nT5mM8c$[Q* lYv(@tf~huh*ZEm@oX@U!jkZ~W@-RTUqd_M"tZ N<1MݙDZ-v醅ɸ"!E|wAGN;:b*GOwiQRZJG1`E !q$GO`j*h͔ hg䆣!xSOvծ ~6~CFw(mE+}*>B͹S|>Ħ8>~!A3>Z$7+ncm*Z2mTyv*li 0t=YXQzIN 7Ixd TsS1Z$F1 LQB G?Mp'ʭ$NZ?IF35>3^#FPD):*CeYQ%֍5|D19sƅ2JM,Bet^ O{szltoiu\yYoրܦ[ܲ>rrw x0v*n3y\XPq'TЪPHO韴HZ#nQq]FN"@D9(keb&>Jyi/= kwv}-nwݜ%(zUYJMW):bO]϶Q<&w|=QˮDp?~yv#Dl>}v51j:m7&w+TH[[h)Bg:whuL|k~_~58q$CGLjM;xOn 0mQiFۗ 88a:>CԴanvaxvz *(>Cu^ [Vc57Pݧ&̾k4hC/]m=%5cP?dP ZxLf=cК31?bEbt UՑkރO^ng_iѮQ92'z'"M 5nBYd77!D E m"MtfKdn[_7\ŬfP.rͪSdvguR2="W]$Q2^WE"R^$B }U6x=wŶ\_37vτ#91}l 6" ȩ[yBjpYvR*nJ2ʼn ~rwr<~:jĈ|`m4o> stream x]s#7r/bs;tNZT.'gJ!q711#`7L_gn:ht&ŝ\(+3աFN: 3FgV[xiT-y>#2 b+G5%un@zD쀭+@33AĀO }m#"?s*ReF7l܎1s5&$Zg‚!0$;_an.'x&JWT}I0|* 4y?%zD?i3 v IZ˸-}tG/Vθb/۾;g>1߫W+{=//zarX>K oou׿q~ޕa]vӦηi_r*>~M1YY+# `tk\(4-lkO/OwMfs,>Y!x17aω;Dβ&{Q=qx'$ړ&Pբ?OI]# <35+ w\ id ?=ؚX[/ݖBDvsUo(cwmOwf7r}'vyyU]qrDxϽ0KN֫ѨD6|;zXb}%m)6%&l.<|T6:qSZe_`͛Ady1VÒ*` mQ FoOn]S}hqa9!ȷD){#H4y(0Tz*~xq+_ݕe+|Qz3"6j}ν^Y[p}.BT(rm,|CL?a[ytwg_wǏbxT|Tf"JwrIXf+Q B=z]2MaGӫLZ*e迪~/4n-1`:}QǏ]OW0U0WJ]@ƾ<4A08&Ħd`%ũ33q.Ri@X4.$NGIkb6ǦŃQ' a KءRn3QA)} %EX`T( a(}!E`Ri]BJ_qљ|Ni9#ue{ $ی'f2pjbxOXrpbÙ/pot>+q{}\BF/pze?Ir9ЮBr[{dcau LmuàN6H'|[0,뭠xM)򻻀='+V x,,-C.W]26s06FϬ6jsHPm s YTTFLgD'rW FGe;<׏}}ל#9oe$1fm1GT$lD 1 fVNk?>MaQYﶧep&e@tÙKotI-0`MA_qp}uiǞjm U\(0Kِ {܅qȏ)rDPYxX:n|ނW"Y#LF5XpUw~Z=L\b2+@1wgv0!gʵBfi?OAlͨZWX{! #;g3/>()6GP kmE8ΓVnA,"mM Ievbᢣ ៩4? G ktS(g^ Z. 7Tc 2ɪjWd|Rj.PsFdKO'P󴘞gxȅ,d XT#MB@nAO`~r֪ ցQut+ PPW=uD2uv`Ƿ_v5/ǭc_4Q:#!0ӮyL8Ĭ>|~,1ѡS%NQ*c3,8AbLӻ;ASjv_)a\xɄ(9|ˎޯXuƱ+XuUЮkԉwӔ~#q<;*t~'x\h|ܰɮƬ:q4 O]Nj꘴9=M8s,JqSee e !gV[t ,8pxK [gZ.<'pMx垶 ,Ks6/9xl{4g7ӳVZvߌ7*~5ɋD:LY$X4f+RV,4nBH@?A KcUt&<9d(B*e  2T0{`LgY5^޳m NzKW;P)MXPO*ғݤ*H7 ˎ*'…o@T룻.:FNDn3m]0c{T@ox1W㵟xԲ/ x/204ce-7 zzW:p^+Ř#ىmsr` /p=Tu*-QxO"3BC%eF҉2S#H`L* ZIqPx.̨A4H!MYP;/ w=ѴuȄgsGSp+&לeև6׽-Owu!iŸ6lŸǝ˟WvK!}CgI.q6i}.^'(=*9awzeljn\ 3.$2nrW$kȻ5>w„&V.LzU<zEo3TYcG!K)H"+9B-dGQŕ(#fSMY^͸1ku.DŽ2bfsyPxw5&F⚯iֿ抸؛box .+?5Tx( ?o7eVNjat;׎]k~oekn:RWJ/ />K\ԑpok=Î%O~v%]M%_n ~]R+ 6cD6CI1tLb>wۿo?x ]cִ[4eѕީTlt"uQi|,9@ ߣ%_Qn/Cϔzw 06m Z,We'gUP{4{T9`S~˜w-tW J'4$ȕ0^pXؿK a KT* ;sb3؈BM!/to^mHZ+>9p1j-=[)td7u@hIlPr&|^BO9F 2$DciK6-62ʖd2ܜ|jӦ@pj?~ $]NFe>|+ ?z$ld9 33A'XZv{G5lso蛬"iӸY{ȭ  YVAw8():I-9 𛃡5JČ7z$j.zj_1 9 9*؛8,zᰃ_U*>x'WiAjӦOcj8Hc%CnUϵ%C*++Ӧی}8*:8/tU)#XT]5 endstream endobj 1122 0 obj << /Length 2923 /Filter /FlateDecode >> stream x\o6_{s'n[W{^ ID޳nw(ISmˎ]NŒșoG#<W["G# ]ߏ)bt}78~ۗˌ|6]*<]fq^%Ub9$|oʥK(ET0xlAv dUӳJD%-{x(ApA5Շrd zX(E12 M2آ*+Y%zQX!aH>74bZf1=,KO˗ie>FwDªfcQJaEa L5FD"hx20[!l6C>5T$eFw 0c[7z`?ʉFI\fpA$tȮ](4,G?* LM&0Ns֤vZ{nv6Pi؏0=)bۚCx\SE~M%)ӯ_` 0 ^wVYf6Z+nf!X_pȶq>!8L&.__Š848oi狏I[ibP&1RhfM{3@89gÝ6d12X*w+01#RjKkbvmj^pSK8mHi֦t!ZC]F/O J"bOp.`К Si1`%W YGqusnYa: &pq3f5T8Y7NpeS; 8hP r`T:,eٙDvĔRT@f) Z51WvGLG5ӘZL-4LfQđQr2E~Lf]w&&5ƿz1hyqL`+GjD`ՖiMbhw/+#@kvدyA- `g\X;ae2ư ~a 7% 98 Ig0qp$Ɵ'ec@_0̜K䳞$E-5vеr % yQXEa{QdEG(,(aIe}8>*Ho<1rV'W0eH63&V \G#]dS9c_ "{O3,s,ȎX" m)P~r*cM<}ivzډC-W5QChE?WUCq v= iٸK!Mto{P3g\7#ԙua$4m;vtVu%΢R!!߾CDsuM7,\kzf%(CeKjԈ,F\b{jbSѳ|hUvSP̺m02AWԻJ5SS PtZ 3/+^]$R©:†Κء0L Fc.Nj+ Qx8.;hvफ़Qe^GQ_PtE3ZdGk5 $ 哕 FsF۲7e($%G5Yvͦ@bSx)fMTNU+H&YܹD 1Srmo 45KBOj$&QI$ iN`>ZjrIjR(`P()a"mVT^POmy8WEgֻAa8*lίJ뺘>,e>5ì(-jpd"UcT 1vG1vKBZUs%O;E9Ba1єP Fk$,b(?!v2+~SꀷsV_zYM_@,P~9QO?m U9%/ە.`XJDliW `ˮ %q_yΪC\&:5U=u}fԤUwK8F3 B%%bHxhRm ;%a ?閼X8Y)Hy eB0D :5DC'$i6i+o?a 81jNyR4}x ;Qw7ZDz7p+NHR}ɂ6wࣄiJ93?igmuO͝ow> stream x][s~`Jg2I2f:i3M@KŘ]߃vBn_Dr a |{x _W lBх a.o,I]v,kwapBr\WU˳o~;Ù WHS((GK7G7RpD) +85GjЭՀRSP(F(HhPBN Cuj6F^փB!Iqj"<XIF E@P%-\_''Qoߟ^%)H+=DAY]p/{O/b=IFJF b" WI.1dj^$j?[MMqR;LIFzxhd'3RL+B=Z"Q518uWDmjH^&)H6Iu6'qk5+4QN4[O P叫{Q./-OE?kz] ~Ō_뵿~~"ruW; g޻o|R#ӛ̙Agky2;\S0 4$`7OuhЎz}7]Җh$y60Ar%>8P<+([<=`osH7 uVHqb|&;l(=݇S+ =-վY1H:d^e(#n6xiz9IX}o@ɾ~IS QGAƾI Nf] fuaɬs31:b2$ 4RWL6s8hv2탼`dO!J}T"ݧtU2Ts%r$ڲ?:&!U3`=KD\=R'( x4PIip g-34$Mͤ U/?#칌a%^<{+Fy6_;r1ml%xltEDXڡp SIM2+NwW+[\Y?f*nG,NZRpZlB.[WĚU`dB[2mFMAhYj[G|:#; 3b"R ΀۩E0*aŻa琥u ho=W m %xδyo=&O<;ԃ4L4h^F)p%MAzYvtiѵ+WeM  c g t#[ 7\ZȘjtPRŀi8~ n8~MeijBs_  ]Ci٣v8xgS>t0%%0O}$P!3Pùhv%4Z%һ-#nс{<.&jnF9'CLBzxqV,>w`3 kspS! aps}7I# iěd9u.d٨8PG|FI-;RG6â{ F dpf.dh9W3M+LKD KPPi|Nd& Xsd(t}ӼqoCfm PhՂDzEc/'zdS/ҹ(Avz־߽;%b BAg߈.Wڂ9q:48L1"Ld1\*h韐6nxC`YkX-)ؤ3XDME1z:_V,l%N{Pr!kS-heY[E39ή23&ECPLPDWQQ0e (@\Wi4sL6 'XH- n&#!;>=QdC?,2ws&\2l#TFE C7/$<'-`"g:Wyn# #+3!yg" 'S2YphD1Uehx~+s+x#խ>xhT_VA 8d im.MD "| <؋Z1ncr=֨уǸ΢Ǵ@% (%#pX#a 4 q8*l\$,ebN*qF8 8Q!e-0c?848z"!S#Hأ! BX-䜯nB/35Cۊ}!4%޲Imν6 ?J43U -g5VyJ#CP0Lw4\=H`8%v:#i7 % <걆L#WBt,:ٓ%h_|+вa&Q;gtHwk8"Hfo+*[vM%E;6Us{gD&׭9hl**J+mF鯢M\=d~c$eVMAGmǵS0"Rv@"!a,8#hqSajlFgcX=Q’I8 -g7D Ln&""E?F1gO5& <)CX3Jhqx)1YO[1LMm8ut1O㪘l3iCA=w#OL#!uD S8'j*B8ZŞaȇ{ʢ\] "rU"6W kogє}2ng.ܰЦt^&1@N1u!uݡUR-clfeL+`8~Ihhb } endstream endobj 1134 0 obj << /Length 3309 /Filter /FlateDecode >> stream x\KsWH=X48hT֫J@KX&> `JŤPw#:ً3Z><'P3R ˷3fZpB]^^Fye?y|^sZܞW_W݃&,^3T1Eb6f_}~q"j$kDK, Vpf˰D&VpkbU@WaՓ*etˮp7xaC}:~ziڬsLI '??|REo9o*i LLȘ 9_1J$b2QbƉ^1kXCfUGfJ!3[V_/CP]~Y!/~͝(r[ڄ?L=1#CMoECR\sv!a%<'HMno*w@OV02{Le(O%EDGX"ZQ3vX_-s ͣ7Z,R笢F͏{瘖_.ǏJ s,/E2[ܴ&\D!C̝u~[e/k;Jye9` ͛bupPb~/JWOyZos'DH#Έ}Ry\{ UDgkJ3 tU^Rs<-Ɗg/{xEg03X 9W^k?̿tj@g_~`5)iJ\;ˍ:Vv%1#tz٧ˢU<#S]TF*M^G)Dފ̏2:0%C_EtZ)pG ]9o:Y%UinDgSxPV7P-Ykl}hY\"~TY.hefI eS10 Xw,MB*L{bfDBX0}ij3 V9%Wo+w͕%1 67f}pk?E[!`5:cM47l]Czґ&DPYB?EDΊ@a4QC¶ O>loo=2h4R{7@j۫Cg>5\8*Z;)bbi[o ݪ>ِ* M =:-/]x)GCeLGŠ!0Dg <enU,^ڣA7P=>VN7c)l,ui>bu>ʺ^hur]9A&YkI\wh<6߽[|l$<a^ctG;={ V@%)gř--C@4Q>CD(F?'U >B.:vtfB(=wD=x.3V $d Fc8 Vr51a1inN.) Vk)6k*Fd|LYY*E 2ev\ 2l|O1+$|&aE4t*EP(C]gLcЛ tC :D);ꌣ(zxH5(zK7}y]F& 8aOG]4 XE#'OVUU~d;Λ@6 ;7wЛpjp.ZPuGygSw~]f>8ۇ~ǐ#s_?1y@>k~NMEuޟ 1A ,zE.B),/uN@ص|f^nqRtOۖUo*ųǼaݭ0] c;v C7n;Prݨ^f# B>}gb`WU_nu_L_%.*ME4rL`1:,1*##ר: }xQ>RbL#J:0E#ջD{{Uh@ѩa_pºޣrD;|iH_3좤TN熦Td0UIJ¢{+O,/&LX"Q(h3l5-\* 1*B N)a;xrܘrDG[G @Ȋ|$2Ovu@; Xd[Ki©|ShL 8G{3hԢrc4կyCr^i,^.,d /oM[ޟŃm~}K(=ovSYČ@<<` _CD ӽWR2ScN{¬Pri@KǕke++ [8 ܉MyL鿁XiWVpqlPWww_TՓ)-kj҈ '4-["Fabk¿ɐ!tڿ;]ntzCJxxqQ u$Xnh03t[25UJ鲜3jn8# ɓvOL ptYFL.GɽIGN(3*BN(K L+cxiMVoShL~MW,tK /!*,g endstream endobj 1140 0 obj << /Length 3535 /Filter /FlateDecode >> stream xks~NNއd&imk:h:1*IYr}{pAB6L-i#{4zI>|ybGWᑑ"I.Xe?L085Y|;'`:bi+!љHY{T3=]/*ky"u ૳>"d&.Ye":cw5c,*B CDX]0)Ӏ&V|]DɪmXq^ٯJ&kSn7gX|Ven58@.KPh4GZ\_m4UxU=&(˭vx\y196~W6Ų gHT u ]/bOmivsg'Ȫ.|,ܞx]H}؟^;U*ΧE|IDteeMZm`Tbl576PCl=mLMuڜxb>0d6k'\os}pEҴMXVb79'@)I@))+&T=Ě T#s"y厭Q::JQ&{70O/`:RRgpaw E90ZтvOuKf(bގT xɬޜ$59$>,j1llyQWŋ3'i1+HZWcr]2yĘ{׾N͝Ҭm+4vNHG~s6$ȏvP5|>[*' L2g,Z8xAxYwԹYIra+veE^߶s;q/~W:y~yri,cL^oi4v1JI"T:ur`;5a$afmuVeYTamc'݃"Wd փ{Zkd{-fb{ծJp*v D+g<!ۖzY]T/ũɝiL12{G`'R!mvgcgJ〖ެyW$;ǁ&A-iځ+hfK,^AK=a[rh | m]iŤ [BvmQ(0 MH`Ei: @'4TgK7t2lK$"hhJEX/h 㼨H}~Q㌈6OFKxS YTϡC50cxpp: ƵHxƵ;:S.LE&6O%b_w #p zBqHzz?YI }!{b>G?NӐOr9 b[m9ڷ >#hg$*LKhT؁07lۖߗR0( 5(Uzt7\~.qAEL90 ԩwӋeQvԩ5H/-qosFh(ՑШv7}#rzGJGqِŶA/k6}l<5SQs noσ]9K2S`t}II{l)<Ig[XuKp?C?9;Xâ0 i;g>g(8bg"",NWze;;; XDFxwՑJvvb1B:<0'söwD2js#cMFټD~G$_Bp`,hs&XԬ!`0 `-<2'rrOEG#Sx1^bd_ɦ⡆횰@SCE*@E.BF]iuMS&=LOіEH1bey4I5EFY4]ñݘ7z3ȣCzG)iw6 ~Qݰ>E*fBd Ɉ.fʈ,2FDC\Qj'=<Sξ_jҝlñ5zZQ*iE0ޔχ~Bg} ʾgz'5@7n,[k*#;G7(JsES{_TY?dBJ MSd+oz~0thLzGW{p|ޙ?JAidZyW;zu_oQHUפbw)GxܸxQKZQ6:>L8!FXvh5:wGK9_Sjn f8g6KxS0<qgEŒ7TEԽŭ艸EE }} oFЮxCKm5w.D, 8xCZu{ڭ j/Y05(:`vvx\77%:Y5 RY_LOM`ARW>=|nVD!S!{ch b@:ɡ!2 $ G#sΜy95Cl|%ݥ?\U3C>n{$΢+œ)G{+ɕd|]kOb2 %Czp@55#3r@ dR5,w 8Ua'$п|s>鴉eu!Đ&)MRN(3XȒ?Lo*e bnj&N.t*PSbZXv0/2ޑ^& rLd%Adz,$-|D.i2$i8B\[kBsT6N(ƴԖ/ȧ]XF ·8= h1ԞBdvx9Iju$ͤ˓ve endstream endobj 1085 0 obj << /Type /ObjStm /N 100 /First 949 /Length 1659 /Filter /FlateDecode >> stream xڭYn7}W}C07E Hb!0H eJH\쐜srɅl! љ\ 6aZ%i!O1>֚d لPk`(F^kL̵p2IeěK hH491(u|MIFX<&̂֡F`$4peԹ`^ >ɀ\ԵZFhx$:@$j%yB]RVb9vqc ,.ZSdDR lR[NuN'+`3@.:g<@񁣖Hf?C)$5/N[ oV15+ER}3Ucф*'"KDqBmD!5!*ĨK ː1rq`D a)X%bxê6pn,Naz-;gtZf iiҺk53B11Mӽƴ51݉% i_k"Oؕ,[Md;.zE\RS+aIGjykj~{rvt[ߵŭ0[w@3tXhUߟ,nV_?.nYXuտB&;DGd\:񌺭ltuiq] ƒ|LI4@H(B^"$T5VVo^6^r='ȣlBv)h !A΅8L# ]{E衢;yA% ow+ܵ.wL8-Pu!NClvQlBb('G;{&w֠Uh?.:, dh ۃ!sGX Yk&om(3}<D73$}Ml,q?0r+5%_08CM5߃q؞8!"!zסd "xKBw# |d{O06C?dxwxO0!q,+ن&4_y~yG48Eز,~,rZK6pG@dԇ> o.y endstream endobj 1193 0 obj << /Length 4268 /Filter /FlateDecode >> stream x][۶~P:N0q':ӴN1\ߏ/ރZ)+ .J2bW/6g_|[_!*W{c(^9#S`Jm&5#+=IJz ˯Jm)vw D)c蜉(RsݛSb_  4z\F0$1ӞJTt?M qhcH O'nF8b ]\<57¤ e3Gއj7KZ֚ \X U5[ Q4)h7K` Mq;02]2TbCa0[j{eHt7̵4\~zU-ڐ-^\IL@(]6'9Ʉ2yGẢSWr`#S'm(>L7 [UJzj!'72םqF39N dAWٚ\} s]8rgnpfJ5`j9nf6s*$J_"CJ;S(EU; 4Zg$n3"m7kIV <*SXUϊfiW]UiLֈZ*8&+PaF\-9;WRE..2 l"'YYB֥ǫs"% ; W1Yir-ާY;],!?wV6]r˲0y7W|֦᧬- l+lA<`lEUҭ6C*P1I½bpՅ\ { \4:Ќ4.xuDTF "\d%$Dc$aP}(<7LCnV1 "}ZL ʶ.u4Mz]XwMLǬ;=@yԐ?З?cDGE隓K\N ]Kq\u֩n[}BX]~viiSG{nօ5I%ž8`;:뫲t>a{cRM8|1jCCnQIJ!84K (8IpQW"kBQ G0~rA4Y@Wݽҿ]Tڭ8D-Ia]=]y$=9mf Ӷq8 y@]-aE S§}e۸j1 -BROZbƩ_Ot@z$ !-b X҂!^ g ڗ7{;7kߗnujs՘G8O(AB?Ad=DŽ)z7r>Labe?][;<%T櫂G6O|cH,z7#8eF…8rPڏ ~'It<v_Z' SӍN(錩h\Ad{c͇0Nh'Vtvkr31Kn0l&f)7RP;c iiz-],&F H2q,[8 M卋p ,ͻ$0Ps@.?wmk,w|̐ q>$n>YaLarƻ;~^ִ'О+ K>Yl,´)悰gŴĬ86B&xu.#BP#h3↖l|~\71512&rpmF|d|#g7T9eGa]fǑ\A֒}Yfzʮ+W͒vi~6ICӍ_á=vV3 ) جWHPyk2S݀v=R.akأl>u0x&1syl3e9 n1Y;"Ĭ2^8C옧j+Bڀ}p8:OP"\pbo <ơY$r.C31*j7!ԎGMMd>  O-IeSEonFǼ OH .GHlB {B jgt=2!6Wr־ym?myQzᒛ31|^ >^4AOK&G8-t8<SBvڠU o†Rs4R=K 0(7{t#͓vHP=3+6R| *'w BB!荐y;viv-mDDFF$4E Eg $6XpǠ'pi|zG|8qKkV'jFㄈ9tMXx#Y -y`A0àw"[(f:mg>^t u0"Kj" Oqpz8=v)1Ɖ#Jh'Dh LDu(nrM 횜m#LTB'4 eb:pq,O39Wp8%ߺgr聨i활Yρq_= )m4f`eXM=_۱erÈ` bad&bL.9+/Qz OdM&k߃Tgq-٭,!Jcn#3)Ғn h&<cUGYZۓnҀ440C6F2;HD2/$!զz\3-[Mqavߝ},#p8k)<Y|د7IK9i\;a!v ^!,A,3' 83?I?Pd0E|z9cW(o*fq%IVgTڄ2.b7Ȩ NQZGy˘uȶ5M)X^K/Mѵϛ#pqDZ7YNU'| fZ#|kyB{+&w jܺBM 7Slr/AnVf YZH!Gfz^4AQ(o{$\t[H2W$Zeׇ h{K\ŜÐ5@Ƴ/7l^^\ACO5Mht\ρ.*i$8 8 v~9B{VLsFp 3UF=mf3A xFψ8|qNb1^k>Ւf?sa5 7J$Iqp,cfžeYzN.hh/jΰ[gGy3e0M5˴ڲX Zؐ|m^KqMInFe@mޜw1q%͒C~Q犹Sޘ$B2[*Ʈ չ'>j~fk?$,* .2ݷɻ_~ɝ|=Gi$ &,FFuQ136՟YneNIYu=i&}OOj:~H t'A:e[1V_U&;%(wF`)yiv:,a.WwiYsnJP iHgW^Ƈ<֦mXG*%s=(1MR }\#zaSUHPjV5ܩϡSǝPE{r6t~t$ endstream endobj 1207 0 obj << /Length 3410 /Filter /FlateDecode >> stream x][o8~bNbiiY uh|Yi,/yHJ,ے|xv,Q$us'Ez] j_NT]O!{oz2zvOE;ח>ݟIf(34ާs%Yvsv.R-x t:9s MG xȆ@q Ns&q/ӅѴRBÄwP?#8ʃ8Ž灑)cn々},x M` S5pcq&v2N;ZvH-~+%}kv%ٕq0\i>P'w#wﮬ4Vlls 3gO9jJ[=&ޫ%3Z wm2Lc0Uui/lals "UX}ak;x r 65`C/@*hC@ TzѐܤU~jt6󬅯${3p+?3/3Q!rDI$ ?+jgIDC#;IѕGOp)o >y",Ϥ1@;}y.\zC0N[Xe@/rWʩ}> ܇# ib EˆQ\@`!)PhYj1ȓ=ir.Tfɀ ֚밝`;ҋʱy *ާf ۲ IVy/3?@_6nݸ{"-dMphPS^avm\=+rG.\@3@85ɯy-&U z XooWXoK#Yâ9-ωU:GaDI^!(&V2P]x Ik4=zc}XnnHya]D{:{/pq}FIg P 7JD܃ nR^7 ) qBAjm 8`LUd \N"=Dr&+o(Ҍ7RigQkY\/YhRMJ& ^B 'H jY[!4D$ɹ{qp>[%)ƒCEK]T !'JEy@P;c@-Er6n;HĖr^b5FFXupG z$$8TN0>[k䌏@|saasq K:!gԚUkxώn˴j^/cxjpNˌޕ!ԩD\ ȢBh阰yhӨzK=m1f5!L[8u(0e$1(r(~ɗB~}i]X |'{3Yv*)'F`JfS#M'<-Eܣk,LYeH~ƫ}lZ=!(G(N«𚜄.5(:+a}|<<.>>7-㪫|VUH/󳿲![j *7/yӉJrx5O¶A'ӳ&rx#OOO<݌yxy !&1Gs*w`KjBJ.-CcʎHYK̏ +[ Mou8,X:7myL})\N6]*i)!t{dGll_= qo>~֌s`;A"~w K@|ٵ07N8EC5%=sšAR7?Nl (T|crYx> stream xڵX[S7~ϯc=B⢅JB mW%(EL@vDK3;swƧS)R4T p.쒑dZLr$$e'1A4MRbN[ԹBS#iqLLs$!;1]RӬ4K`0LV)~!:gTxd8xeD..OR].+l4Wo7G`vaz 6ԃ9>99`c؍2`cJ1I8ɨC57fB>2lLI_PvvU:O˧OU\?VDZ_}m.gW~^yS:7;6Nxڎs ^o|;B?>Aos GWW~bysfuss&_N;T;dm =鳴|yIh֛V=S{wʏ^HE&j؟눸#r=92 Hmz%Xl,S₯>S!0jD b1'   VVEojGHQk1 c}<{ǡ{ڋ1azͮ'];:x;]_$cY͕ʲ1KkD:m3Gy 㙃c> stream x]Y8~_}X+SUgPm3EuX8{jmqrF}QFҠL6rbQ6LPUD&c&.䓑FpDgwF2xf6gdUѵ{9Zu:y_hrA TU~l,oRfP{\KFB I5$cdy׆ $8-]+` W1k|'s9z]4&P@ДeM7!;bzy时VJdi~H.ckZI 0O|fw#a^+{qoƐ,Uu ~07ZI9ʞ~3_hfImQ/ks(?3%h E75zז¨DJHAިĽ(ݧ5p\9qMwҕCWNA %pahΌkgZuynuӜnbF'7 t(BZ3ܕSWZju,($4!C|ނ4MSc,0'KW]i#Xt(iF p*(HēQjgwL\skզX.w e hrY˝m3G"An˙+GP8n48v8oA=ΟQƮHDl)h^+Dߕw?*"ˀO&/6' 'Ѵbe [kxx(/qyq2+Q=.`we n+V@Pھ;Dj4KtVD3Ad  .ޫOSn0-TGdXB~_~՜h{^8M[Zs̛ku >y83@NAf6"2?vAt>Y4xI0&}qo4֦ѴQddo|VmQ* eG SbD;chkXs`L `Ti ;e-CjurV;l( -ȁ@G_g|WM d0s%xcF4 ?\ߕsWN=Lv=yV CY's(Tdtf_PB#W^cI9/L%0{86>`26tzTfd$*FQNHۅ۸Ik(E\Pv4^} 1a 8A ^SRkjSVײ̪=>0=]9wd/BYd r іѧOa;}|z'FOo]G'x NN+CQNMqL::<W ]$S9Ld會cvr lam|'u^@[oV8ugfe?]+_*aQߍJ0%ر>=wpM#h挟*<]HHH#r!aISmUFeTqG︳Q38Hp3kgH~[ LIxJP̮+}̨f%v4+ptⳀlZ7-nk"y', `̏nPt %Lde#&`;vu+y 4@tLېXŦa+I"gCx zOݎ8]Q!!,(KPmө߿`U TI (ҫ3'_uA/`y>GlF%7qP2x4ipf[*[.MsKxS"U WݦH_@ļX rf:ӧ!w۽f\vҫFvV^hwT2k|u&M%DtyexUT ?@^rsNKM,G>W 5r\يwITc,p8V,.O5+Cm:xSUR|giHJ1bvYrŸҁ;{:l._~+'In,[bS7߹.VRP.Tc?nc*p"@'DO5p *#JbM4ǨQ'q"Yq-y| endstream endobj 1316 0 obj << /Length 3563 /Filter /FlateDecode >> stream x][o~ϯaZ!9. 46@ !FUIdgH(FZ̹gĝNy"v'%}W^św阮Q;*J%::8~Se?(tetfՏrqMcQ~FmIęFOxV:4hu]*-A~<\9+adP $zMh w Q@Np;NȠ2LH3Jв[(NL,wϾd~MO5xy$[YSĆ*zGgDB1ݼ.cN $*$*g)Q,5*!1Έ^m!jOoYQ H-TH/1\ޯj=i^[%[dqE3u:){%KR@^$|)aX{bN QC[I$w>&DMĸ,;R9^o=!:|WdHɆ\=rL\W)/ΈV6-7[ёs T8R99"w57s a&>Fnmƽ5NJa_mV7?镤%I5?@-f6[;}uij++E?`%cu㧐.© p).߻p.T1=cl6Ƹ$:j\kϿRo&VNtZIKp(8<`ճI0$e|F[~eQ [E-pEUe +;1b.FGaK$]"* ǘvet[b4[J*,ӜC^cUro̹A  tA%5$*vn-sf{D sf]KĖ}c[4\D(ԕԉ{+1|i_ wT Q߈EXSW.^Gk;|-Ӛ9L\ugщ2lp9(Fv{Tw(TE8Cd Bp .?TTM>7񕘊I.i%o#Yڍ?GiIK1iYݱT &LB;=vz-ݬnΊ9F` ~ QXү(];q*(01ȭIޔN%N:ֶ&[(Cz|v[|m*O?t~}ð:yp {_=uJ!y|xA'%J7nbf, b{hEm \0lWOQ(`[OyZj`5a Ki 1oul0JrTW>]KWl²k2ڃagɂPMvuӿdAF8l^uFkg/FU zbF|QuHYx#_>mǬ>enW/}-]`J, ̎L)'ڃ?X D61[RPc?}GjSƟ_?pȍC% ΍tILSqDh$1=<hNnjCZ&@۳P^ `dzbUHy?x4= wgzޱfO*4 'UEko}.T'ET_d:Of2NlRjEY,O+I/#6>Ir"۠N{76>>^ۨc޽5R(b|)V `vl;sW S?a'Su2vfvK0L ?=wD/ۅ]&[3VF/>e[op~ՁM;WUAP`V`'EܻwIOq`rv m1xk&>53=A<3Z:zj }P+1 V:cg_vVu0 oYreb.Gb #]:I5Ih s"Jn.X5 ;}Qx endstream endobj 1263 0 obj << /Type /ObjStm /N 100 /First 972 /Length 1430 /Filter /FlateDecode >> stream xڭXɎG +\]j`v` E8R0#4x#~zfY"VN%$5ԇBԍu7Zb7z:=# JsQ"Tcb4L%SSiԪ{jIRŰiJ5Wd+>N22;xZRګ؄[>BM 5p)';sbX}3Xmjb!_iGJ;wq@ 1b|H@CbR=d#X!L M#NRdV8EQ:}5 5X6}= HXPG!H?`!a&63yrd <;P^"DXQXJX NQGeC}:܇QC 1&2}Q!FCV1:Mbt>}Cݧ1q"msrY3"/K6ߒYn͛%c?bX8J,5bX\)JL+^}MxrQ檹K8e֒7՞4! ER4$S 2C13YL#Xud-{Kղ+G2ԥ9u1^_@cFƼb2Ӵ<۾Lס7,v|86˓۷ŜGWݿOgBgsz~_;;lgՍOOfy~zfxQ|y|;#\s_K<le2Dt ;3O>aܾ?ȯǸEz^2CgK^ Zm.7 C)y}F}4\V`-/ :2/_&f7XD5siA0se yDd(3,=\*RdFcZ,FFFExhSBK1$^w6+)BŢ`o2e(sA?Af[2Kg(@M(JV2 e"wlQU .Xu8UEjkTq-8֢bGU@= ]3cE/JTqn7 ;ͮOQGՅtmHn5>6:r=wlxmiLW~6 ?qn>w `HJYky`HJA,$J EYӵ5l0+Yky`X0z% =Q2COtD'&AfZ[R9>eyEK0eKCK_ky` -}0 2+t` D+z‡-oi*B endstream endobj 1380 0 obj << /Length 1837 /Filter /FlateDecode >> stream x\n6}W襨fd.I%(6ha:=lũQKR2vN}V5MI\C4aɫsKI]q^&eV*(Q2+Ht.3IwM9?5-$+'OW|6T!2nrjۆrm},Hځǟ&o;*i#tyY:$iT6 R95km:O3yoc@'$M%f3,,ђ ,i*>t:Ʊ 3 jaAjA,V, :6p4-Ď ^-]qJn," P-H *njF$Yi+7W8Wy@dXv`XRZm 0Fyz:F eaw40Of 6O$ t19h/`ڿOd=:]&с[ Y:&1* s.'40@&'Y$ZTҿ)67)Ho%htB^vIS/p~SHDYimK7Ӻ>*UuA;^3 {>]w|7W>%ՖI?<~rse;%iHN!ӯHH:(INAe8Hwq#$T5'e@ݍUKԵT H`ŗaNΉF h^kP3 N4֯|/;. .k]C%UꆢXP؇_cY9{v/uіWcYUBaqCI 4RA,DOHp1mfK"@pmȵ/,WıLh%Ŀܠu@F}`)vGaR`G,cXIFizSbʊe^rbfO;rߣ;:E,E5KZ* wuj>=x4~$C.9 "t›J .l"+E^]Njbݹʩ\PNw7/yn8ұՆϰ{pI媑hTt/]:*?}f`$U[uW+|Q1^Fe[bܯ>Z?o֍x% Ms[y>mXϷc̢=idBD[N))Rp[qVef_8]L-I)|q(Lz 2GZ$~r[eW/3eg1W gYJKI;\,/k"^v9& endstream endobj 1410 0 obj << /Length 10240 /Filter /FlateDecode >> stream x}Ys5Ǭ{F'c8|Gr31#btwSˍ'Pj"E|`j,\L$93R\QR_ivψg?.~:W{tuqUl1}~(.8S_nǷ8_VeQ~Gl.f_A%YNrj!g Vjح?y~~Ws~xS_\MUYwvE=_=Ewo~|_;M~՝Db&&[3yƘVd&>U'-M B֎} }4fR>Y=~BL\PI[Xz~qU_.6nuY~i^nSuX/o Ͼ{v ˘ԋ}GI^mwEfeyX(= 5^W~#L*" p&S*gr\2=xuf> "Yxdⲁa;mud4z] (4Q(&Q`J@LDƄzG>_5kTrQw|`}=ڇ[ֹyD)JN#޶[8)MZ IaU"SX:?4J> RE3nZ;Z5C2nQ^T‹A˛DŽIdߵ*"w |AFOx@yDAwʘw+.zW2}MQ;ڶ0FnON [ <Y gGp&482n_}(8Wno6CozkyI_7}r]7NiD#}A´XM5-rôF|3 J$M$'.i>9S6STL槉1fbQ,ӦrYwwRzڨa_JUKo22W\Fn׫M}Ő9aIbFvC0] @7{$!sA7cVOg(^ 5|I6Cvf;EEC}x)L⥁;U+^2!a;%﹔"|Zn{_W߸M5*TIk$' bsD ?J{Y 禡'ÁlFF.46z>as$B&[[johB>UE̽QkKE8wg"HVnXx")AN H=Mw7)G4ORѵ}-jlD|E fq/F:Hcr6%Poo4EQ؉ Ry0)U: 6_\/n۫šm!š]6 YB716]. 2T9H4F4OyUvw-U&Ml %eTfrf\0  0¥KXis/ yw!j qH[ɲQp.R Y}NڝhdU #L?jB(1%JL"az_}QXX YaXlb OKʌ UO︻!ZSTJvߐDEOOyX:I=w Avr'Ya4[զ*Uv#< `PV5bRo-w~鶅?2~;QLT D᫅Y/U8z0eUDlPr, d0;Fs x PBA#Ζ4clhTm;j$R5Iӎϭ6+9tghX|':貸ŽS80Ή[W-n_09Ms_sjv]lobǸ{_/z+bw0ׇ7ź`AG7G6 v}4 t;9ȸ+!^ؕ]Ȩ}p # ڱ|ah*e+%6 = ;U[&[-\ڞ` 0@8vM@ %D`=u-+NLim! ~]bPB:!\ `a]e`t)SF92Fw ?` %H9"m!%,0 K&~U0sф)Ws*wxy\G69iƴRD`~CcmaTO̽-_k) 2DhPGv7JJ:50)h}gCφЉ0mt`+lSmPe3 Jɏ5D1(FWRL8_ '!)0V_-.Uv|gvWK,<4r} 4E OFy |<)L!$y=OQcHOqgCe#+.`6|5l`;irȨ%)s m1Π*DށG~Z BYr$4-AP+׷94yl5M ܀0g}߈m] |]2+tj8;޹d: Y.?. x޻@IA5>>V`2y$P-lIe%r,:s#T\aL/=P':F&()kލi*b}EGzyGדm^wiI!l:w xhR'㣶OM#P: FQ6ro*8F&F FT9Ǐpb!Or4zVL3MD0M^xy|:C ZvO;U+m.]/: 2|Wפc(1B .na3:_{0!I#z4] ? 6f#Sd[!UB{$'ts=uBJ4A%(99;?DH#ѣcO6[";{.pZXGL(Le$B,fg%LKDݘ/Z_>O@%5%\B7V577|(#.4q }~s%gi-{s&潎0`6rLƦvAnv>)rP3SnDՌj~:u[ժ n''+?'J^h UEy@\&2X)QpC*]DlG|K{w5:QwSu=iAI0tiB d5}Y~qOxS,Mq@>ygn;Ƒnd~ޤEJD4ng~A L0]14ZB0\7 zraxMHa+q=a9{] TW!tͧZ^"Y&0 NnMYcݩep2]: =~[!9xڸtd杪WG5 XA ڂ:U( 6:` M|^ԏ&uAqsIE'v@} 5%D `>!=Fn4ТZ}U'_& q{wҩ+\e]x*3ɻ J?nг8-}u#ݧUT6;ۀ ĕ`ı"GO6.hX_䈺=+.)fm*bKugRH^qgl'i}\x}ӔU%&Ð`'4as5]:)uڥd̖;5p ?4/=; 1`0Zd!NBZE`ALxL_H՜ĿuSjƾjf՟4W)bK68:K'w5`tfoZbX٧(LZW-B_iPJ>Pzpfq./K{JDY$!>!Mwn]ҽ܂WrkXYrK7rG\iӏ04@yY.1E''(iGj !u$LcMoTI`"N4cRV q=C0_Tw:"[9DNRPŚWen.KUS:LSa 26RXyyz3qUYXʔ&}|姠&1 ?MDvhf ^Z'qDL'PZۼezqcG=?(N%uj{ fb]mNԹTϓgc]gº(u}6~i>;;=mcguy &M߂Yդ[0ِ[0 !g6͟rgd\Օ`uFԱAt^ߧ]_]Pժ]l` TUZAg!JW0AÑ2e/MwBX-d#&%gj.*-຺Nj̶}I}(aM ʝN=vIO߲nJ>>ydEEePJź[&%F/] s 詔`+Kniͮ8_0$"|),;xuDY h}pga:Њ6,o銃QAںJ1"('62{H=q2i`\\;4`HLS95"0u(J--@FRDѝI%^G҈7wЬEڸ՗e5fH[=f;Ь|@פs`YӋe|XVΆeXU=f=h4*°X S˶p[߯wyfȀZ*:.Y;k}/ln{]TW7K{U()}{S::K af5:A+ !O`k. %u1y䱘(S*SWQș|ұ7PTs2\K#,œ!lˏ @  ,-ζ-N n=lt-̨'@'v>L"pXttLMcq8W`v9RwǕ¹B1P<5NG.W4G4,,:JǸS3\ \ccswjf"շOs B܋j{Qg 9F&."rKl91Ys7N;gz]nH7 ɣA3MzA%*5v,Y9i?o1j`z,2i!@,Fھ,0$;L mYa^7/1"iðq*' I!>ibJ$G0CLAtB͹ XN\gYnQeV]qH91g]_ {P,z eA)bsYyUe=XrqX߿7Z5 2LS"ngƭDjM|j:6tvec908En | vDST2SG{a%쑭}7uCHEU~UO( H'=H+s:&rW9 W>*z 6~بD}k: J&u8˦^+0oQpŔgYOdR0.*)O35uNe1TD|zIC6djзE6G`$M?;}B ʈ#Ji?B O !1E2ۂ)`:o~쫿'ĎqXlV l5>o]vۚ(00A yAI~Q>촃XLX# TpzB&߱^sNq96[G=E.B9G۸ÏmNьL>3Sg&۰cuHn4`GB_gt'gS5+DYͨ#0A_nfiZU{],@-S%I#6j y&c־[L+aw2Ur2tC >c\L7M)_|>wXXQ\UyI׶UUM"2sPKi>=SDa`dobXR%bcyDw`daЇGAk >z.QmL!,ʻ5@cq:~`394Zg66ubMmo7qx[h7 ^23sl$FJ5 YQwt1] Hl_҈g]=Iend Fςj3~#y\Fg)|Ino"󼛱N]4YDw|┠0NӽVqLT2 g|_;b_vdau7}C?W1WrqU5fq}^5wYGk`+QQK]W4yw8uRå4Tit_" /|3@_uK[ʳ6`XD= n˂F0mQ&tUCpH[ lJM?,vjĻ}rD~"؏]\,AD\pP(y-y7LCiYT,s +G\!_6-n{MaQ=I4咺 cĊ:ݭ p%D1%կL##sXa"8yAG \HaCݡOaKxm]վն& S#2M}.m#Pl /ߥm^,& t?'L,yK Y70xy,@MCM}P/8c)9֜L}yЍrSr66 T$H km@a9us0fqodkk L&]o\cxdqunQQg _V;qQbj_e;7~'|޵Ϻ¹bSt\eՏ v*$tXH<3o֝e |[;l7_ lQҪfŢaؗ?i{|[اm_Ѻ*?,_||}Y~shbm 0(AjDŽ0[V9w]]rQh9hK;u$3ǘWވ0 . yxmNhm%+:/WbSgޕ97U0:xUV%) ?#ga\_f2w6AT'h硯~'ؔ endstream endobj 1418 0 obj << /Length 4602 /Filter /FlateDecode >> stream xr6=_NnDt&q֧4 %ۜHVx.)ʢ((qHrp88hA SJŀIt~< Sɐ`?~-d~"X=|Zȏ'e>g'C/>?Ipg'9bi"9z4bb&NI?H]O8X>1DŲι9$)Rt-gmIp1\-G+=+$> 5O\NPGz%AQ٦$r   i(%X`l\.F,0QOG㬀 'GiQ!~3Zh:;Sv6xYɑ[wGab (Q3P/ըebOh7mc az]xɐ"ςrUԿ'I===UMz[$^֛hx-Fj.noUs=Y\ψ)iؐL= S&Ob 4I5F h.qWlVX᯵IJ*v>$axd~KIES1-pcNH!,GU}+AE#1~O ~v2)E9RJK(GN$xEwa8dXdCQjO3r[C+PiF6c8w~wR 68BYBY6z)xoCq hM/i77d 5D]K&.3)NR,+0ZUﲑv.XZt m4c mBVg ZH1IJC@RJGBG5B Bm"VC, (v !3F'3}PHBBqV/#Q ܓgG}< XE zE/BH?2$:GgP__CsܙFcTOC 40; =d< H(,O,ŏkiuO"t 3Nvh!ۡqU_{_3(PN="/T˕η劦 QMX=r⨁(S o,*Zl]/ kCn׬e%-^lt5(}Y@!x,WMЦ$/rNvwesBiP5%0aG^  dSJLiu;0ojCi6О5ic(C^'C[ӻX(ƈ =?lB h$bҎ)P3b^y#/:n6x$BԓTNGP!PN5eg}I!=wWY7 6A,j'>}ԼΖ&blonq7^ s9@M{"?2"7߰X\[| ] L}= ! =04$jA31GQ'i=/Vt|I]JҢT:b[wld7&"_K+Ǚ{.}EƢ"fh;v)uכR |~oJDa(uUAȒLwH-4|o*L?88J>4 D i5!MeH4M G./R*5+B@ x2LOj_8n'n ؁CoM#tYC"n~B4$y4؞Z/ي}hłV9Ȧtq+cV9Tz@h7\#*pFGߋ=lⅽY *}yrP$NFډ~$pL[;u.':ne5w.% ܷJK/?"0׬A~ďzW<0yK!-mn/ -E1ǁfp q}ڡSRLҢ&BO)*ۗO6 ;3IĻ;ΡNcyan )HSOk#Җs6ʛ!1HlNݶK}D3GC7 FL!>|v#zS)(9r%SwÝ9ޝyG 6B\bC7rJeNr+/=H5;7x2k]H5[¶<%O#B-S L_YywhNX?j̨nu BSMI!I8|oK:zB?:?=6˦7V;s9N6QIu\?j0 Zj7Kȶ"xnOUf"d+<<*1&)i,MBX:nM9AOZrw>O N'Xԋ̍q?NsgNEa$ :O'ˌ"vOS;Ui]PY%vjm>z (ʹZp. yS/oqв+i7PZ92`` atsnיθ %F &$UĖX ?UX+-S<+Ivf_ 9S`ݓD9:2$zYnYD8r8`I V=u&5GRR/}i&-\t^yFp+oX"Bı;V[~O0Ip&IҪ<{B2 endstream endobj 1376 0 obj << /Type /ObjStm /N 100 /First 970 /Length 1620 /Filter /FlateDecode >> stream xY[o7}ϯcy=񥊐(D+P!$[FPIvd)S@Y{3x.ɅKH!+MIFIdFE6(p*[hxiCJ&6вJj ГI(t!00 I0TLnj&$2 )edMUkY[X79ttGAUu;u:F%PJh(:,mi]BTՄ(P!8 :XL b[;Clm:jtT]LL :z6t c$A0ـ`Z-=셳ʲ9 LŮal`*+ ·.veQƲ\ Z7 q][pVCۇ*ܖMMփbVSYIA2 \MFAȖ~[E8`!tjA : :D]:D:!U#D+-a8 z/L?K%JȮg0s%PeI̩ǒp)a]Q-kY{o8Fd)8%xmRluɌt2Wݻ${2sH@TŔIbN{{ք:兞.lִ7X^.ɾ_  (m!-1F1|9HsC/.tjkX̖FӓAF'|,zEIΈ܎i?L&fx3{Nѫz7Qm|˱oѫm %&'䘆KZܘ6`@◚`\$2{D^朣'W.NposI\Kl^ɱU/s'/3;%:`'3#d"e=ɓqu7UZþBVЛ|zΪ6=cÍW( 5Jʴm|ڟ~{ie\,_N ΏgˋOӛӳ)UgJruhVc>W:7%Xx}޸7Y0B ë\??Mb! .\`+%Y ^_Zdh,Vסr1')Ҩ>I@^kJWSA"}2>Ja}é}?0@|T?*+Ja]Tۻ\3 #4Y7z{2'l󦗞M`#DNpk8yQ2/3SL\uȳ1ytKNpKQQf!X{A endstream endobj 1470 0 obj << /Length 3642 /Filter /FlateDecode >> stream xko6~9|*|SjP=$zm5$?C YI8#ͥ(r87G$ >!/{?ij?"q]NTK'Z99MFB;g;$ޠ2{q:i8 eS%x-$/ILj߼>*3ѧ 3P [,RY^a~xInh$zv:e 78eiL]i"[7Xٖ_Tt ?DJQu"^{ K|mooonƩl6WuMfJV m9#e~j6TRXs$^z-k۞A& UQMVĥhJɰq]bCMxM<Ö>( zGҗ.}}YC"6Ⱦ5WXАXA EAH<G߆ vXAL z$'Gq\T) .qz_FDw0CN@AZ}4eMhY0eȣ31䀉HOl7-@,g9?n1$Gșǰ3kO:-Ò19gaքe u1ԢZFDݠvN^:5$,$^{Tቋ;L+ٓfeMIZZ)b[3pX'וt_nw`:ǕKOJѠ|ʱRlKDQ \ R-sΰ|ZvKR9}Un#I% a Jv4ߺeıekvۗ=-Ax,޶Q3qY˖CMgU MBQ}v[_ݢKQU≚wQs$fEu}rk]vg砬3 WYm}9pt,9x7U ǹs>1]>ccεP쉚?Q.jr=kOԼ0j>8g,@ANkbG m(=f,ZT(y'f+y4l8m_˟gP t0l~$$n&me$֩/z+ C[G2GQΫ [k;omُ|7Ï; 9/xnhxԝJzs,o|:`^8 e>4 -Ӡ0jpC?JS4N$5 ߚ쭳5=Oߏ0D`%).%tW0GÉ#pLz`JX4[VECH}v,FX/ZG6Yw B tBVF޹{{CKs>V{~y{!|dzZ{@x*b.VjfWL%wtkTy.U&)Iy<'1}|SL$ΒNR/+{`m;( B씦~'3!6I$CkѱTmXPk:Rt|u`r3[;l2xn:=FB\a0i,afAxz7 /-Ef*m)90FkuEQeNե}b8QB] #؞ܰ8~yR൬*6TӭT 4F5xg-g,rUҬNcAMXY8Zd,g<;!'U9]"+*cg|E'/W>YKMu Fڤ]Bӫ?a!ݗn0Wpv@ )ŲpڂtTݬ뫲-2rgl8gIG`EŤl^EoDf[[/?khLfThKF;4FVpaUmH4L%'Ge^o̅e*][XEچD0 jߴB;!ujEIQ|u0z}Юq5Л'eUY^YuaABB ` [|uvyt! Pd >;2C:Dv]RN~Avflǒ k%SBGAs;72Ӗ%l6+ܸ5? X^ٷ.rt8r'iI*_eN̰ʕy.v-E bɎz{1, =Cf&V5y*F#SwYcvmiz'(_z2XmӘm.G{)gM!$f,tұnT7WSFgoF],α J@p*+3.HŔPg^>K $wҋxKҋxj48i>导On]{:+Mr#шps^eco.;rˆ~4p(KWw OBIRsa+ gY\BUh Iա{<+ǩީyjaR<<`W479}yb ,޷ؒʔPߣr_yQAJo'5] +5HwEls6~'~VM~?ThJh^hɔqN[ endstream endobj 1515 0 obj << /Length 2624 /Filter /FlateDecode >> stream x][6~ϯ0 ī@&A Їl4cǨok{J|,9[sH cu9$ ziA|ϾzvRbjpu;<q"j8x]T b9mF&h4`/xo/,QȢIAu@=N .G#1yY*=~z6>j Q;:Δ}o.46ұ_ӈy mFОVD7Rɲ9` nbFŊ)e6+ 0r ;/a%0r=W"LL]1\4LV$QR<.e̒oG#]G[rURez~k*6@~bhZ8睠Zy*S)D&aB^oht8+U9D#C y{GA, ;`mw?읥 NiT;2H%A#2} M'u-u E?!h 3f /dvQե XD?b%lfF_HD+GØl`ҡ?+X%vL^#,dh0A$)H:>I)ʚE[yY;zN)Vn?{IpJxi ZQx:A 4pcƨDTUDw׾coQ ܹ3o jdt*u<-wLl UA 39L{}χ<([wHc`xPPYDY,$m@el-hٕ^"UBYZ< :r!zExE *,j wy{k1Mɬ"PrD]ÎYz` ڜ1*fy _;O[ȗtjTjoaq+#Zp#C.l42e˥"j{nn<e6Z _]gq lsWD["f-e;w'qHnؙdck@NѿFp?ENɤ_XM5;xBau5i/+.| گYW@{yhxhG7x-YҫfǁN7WUf[O%^ˎx><,wbО`Ͱe op3cPcj?0VyKd{qҤm`LeOا5>{$Gֻ 3pfY]l6S.f0#Ͳ8H!ۜYdiuz80 L)\*0A8ަQuJ6YP%$?dbv+}PlWU%NM^-'ӈe]v"Q2 %' L;rXIty\s+p"Y'3ZQv(X ++~LHG8i079((M$Gv[I3V)&XY+g9sdc/=y0^͊*| *=o_ҤbXW4fO6ga8Gg+ok.YEڑ85htYr JB.\Q$jGB;pNtTNH"u#ضԡ1DIufTix[jtGשF&1D5e  8UW!ۣ`/GylP$Xg 6U8+65]:F2T9ͼcv{;\ufVn9pB^wm&ǤBRDGpn!:` .k.`":ADWz- ! 89ǷHDБBxgBubC *r|oRJdB)ݱ@)Q-C)`' ( -zx Rwm@kTg7#GZkܑjߠF PT,G{>|u4Ϊ C {zG+nڽ{ٲ#y ٝ<; *u )TC}e0?阣ԲύW~#WP}[ЭvϕkBQPN{)gݨb13gsZǙt8n,>&L[vRAV|UhW#;󛘥O|!Ĥtb.U<0z' endstream endobj 1465 0 obj << /Type /ObjStm /N 100 /First 971 /Length 1415 /Filter /FlateDecode >> stream xڭX[k9 }_݇|!z!Bw)I yha)-3K.l&% 'K%Kǡ r.6G)jQ4%XES%RMt& Bv2vTWj+v5DSk$JUDA5qI 5w,.Su= a..u Iw Ǫ[dTT'GEq)$ M.fM]q,l:V+V PTMQ|4G Iu:drr8($uR@DHNT!6bRCj.%u E0j.+HH-MN9^t.#߂v< .˔XIbS"24JrM+$$IuH]Zt R\t > >jR|Ԧ:hQ|$hUuD\DJ>zP]":"$:8BT!٨.C>VswpϿ'_pONV;{; ܲo \ojߺrbj9VlLwDw:Q}V˵y"R<*jl'eF0udS2Z'6 K'\=KSדQ"J*7 wmTZC TxZ^oώ֗-׫\^Hl}+Ϫ}ûX@xG1[j1O6-:G+r]^\r<;Dv-े#*t!<~z{[>ÃGQ8hW9"tGbڐ.J ȖIӪi\j( so72qqC.F0.qom؊{ dwxoGX\\6p^x c1S§mԭ '#_1 '6|FAUhj9ujWK4 ulllGj6R=Ecv=MFp^`'kUDk)-'[ .~ ވҽ gJnYޚ62x)C` m؛~f $/w+D%"ɍn`n`E7־2ABmfP*u J P^$I/Z!F•^|<ֻK %K(0˜u}u 35OX;c]CuS5OX;c]Cu 35X> stream x]Y$5~W>etu+4Kc;0 c.UՕG,}ݥ8řb/ݕ:]\.:{=?fJٷ3gF B}{?nnYXc_l6jwYfV7 6?_voJ~wv|*PFM.MqS, ۗT„UXCAI&s$R}oBf>}L2yg`P%@P"VL6  a4ʱm9"zTmc+Ht`)S IBr8a*ŇbW{&%T"#Ve i"HS"шY&*-XebfdVw)V[j-RY-ra2S3 ̔L\f*LM*>P@0ҷ|Җ5ĀX4JF9B>jҰb:U1$&-xbIC|?(jH]|Lk 6zklXԉ1Za "Sos$P )jj#@5x6w,XPc;7v5;&B0k$SR*XJ&fm VI[:'oU&19[*T+RKB5bi h(Bcr( 沶."mwi Vmh4-_;QkT+=ˤt.>̃<{͍pr 9II;*#&+5I"d"NEe+@͗D0eKkHV꣤«X 5,( B4XF5$)2xbӬaEO`Md 3ugl9 VhkX̛zM/d.\%-Xu BJZ6GwTt#0"1'%(@0(MJx{XIQhX'oh E(ے1 $IeY o<8dP0k$!Ib9sY:bn6X YMD> jy?v@) !Y@ z IN+Z]2;shO f5X礒-2P *d`ځ  : pӀG==;Ĉ=xB /D>qبN{cYpXvP{* ІrJB VwO_o}T/ɻ7.ȸ\VBp  n4km_vvџ^"j~z-/'vt07{ͻHʞ3;IW3B)3 F]O.ۖ/)Wt;HׄPT83l=c\\Yk몍!q{@?< 3dLh4¡o:5hKA9i.4 cIC6^S1QeF2=}FZQm8~HЂQ\Ɣy/"[6g9Z*V2JvNF~GޫW(a@6􋮤k0Sr>rϬ#P66wUXÛN\<3T>Ԩ`h3~Dk"ޯy'TKdbm_2?gU:0aS:‘nw+Xur3O>ވG8ۙ6z IsxFXwgTXtynl]vM~G S<ǒ.ԉdИeL&t)2N\wgz\%)A q^-~O_ȽS"˫QGuX0GEci)MB˜"ݞCŋ|1%n77hb #17udeڊiA-fE#Xn4BSd6 ^G&mϙ1#5 e\ti/V݋iފ!3t'M\M&y3wf"#F[,X-vlCem)7|/"G_FsM_2Ѕ LuI{t̑7h" 6ttق_pUZ5[hbp-؄$"6kn"2=+l x+UڐR<-"\UYe'%W܉ %$ ByҰ\O$3vUs1}XZTbp\+}"GݯwAX;n0n֌0-9S(e>0Lu{) ZXo̽g9¬:] gcjT2q傼y"EҫJ).;Jij=$2`l'\$ ,VDT?AIŽXY.`XIx$R `JCLT.i)pa2\ϔN xɫTS@x~/ց& aߛ/n bsz=?L#NV՚H0 l0؟HVWSwKPL 028UX!>WԹ`GYlU@-5րx51Eʶ>gw^W=pKF?!aB  "sܺB).ɮ:s`QgnnmFn}_f:4͔L^` "Ra.J#D.R_7K-Fm)bX(FVi殼{ BՁתv\#Po[WNlYon_2튲qσ;q4C@(}EAٰ>$D"ob!n/ +qd|!@К>Ve+~9oiւ:de[ dq2ʸ@>R~0 u.LP碥8yhY6Q!ӭʦaq`L]o[ 1fo1\@΍B͛YctPjlmW$g]UEmmSW* oku{GKXm-̅$]gQ,UeKtYۣb1Օ: HW|{B׆1B7}PX|, %m6ff:0!nd;}F[zoa;wT$ >+s}ݷCuuў d4`l?fqGwi\0!1\Tl+O01q#f%cWqNF-!SH3S|7E-*j=n*ʂYe2h8<#pQjBQdeTPMO[nu}T^MHǽ/Tƫ$ձofoyby~w7 􀩷g+*1 3}7$ ;nxE5&=\m(/5 S^؂BBRl]aGMzVmA~M3b*>5`u@Z% lڋPcnCu'P *,yJ^h4tW2J4ݢcq d`kl{c2\֗bKkdguEqJ+43P8<0|N[U^9Y_^uL]OWoX/Ӓq\97 Jյ_ʰK0lG'ް;Rg|Jlܢ..5-Z34;:`MXmKe?k!x ,%+YwvڶBJ@/]]d+` VDl>o!F{=6t+'Zhs?dU TKAa_#eZ*xssw q^iy8\fxS- MAA3Dg{k8jsꨤ2T|@h ,w6'z{A ˳Q amKQ.+Hy(D2k=5TT> *%|*R]Qy*aM D1?;胫⟃l|W''bXctωpA%/h%mxR? ux(%&}a≝'̀@Y@-< D @}f lVDX;#-$teb"]ӆ"kiCKLL6x;Q ÕUeMd(6J_FKs'6-}e&AU1ک)#uQ5EkM|nmiA϶Ÿ!#` i1ڙ50>Jm *Q\x=I hѫ8@-!u<3ya;>v?$ vJv)9v>(X Nw(S~[ղ!ٳ: 'R&P:, 'cC= WPdu@@eʁY!rW#G_v蕉Jvt>}MK|KLҸŋX*Ȉ{xy:G^0[3?<,aݲx^PiY8ܻȧ!ܣ&d+m޵GBumpTO«. V=_3E(`1?|2$z|XcЀ ,>a5n|ngv=>.wowv<]"8,S޽:8WR=@^]u _5فni JbGuqx3 WRhYJo0sBȀ 0÷K_^-qޖ,$0K8CbǸ  FRcCet+mYW3'rXQW7 I1b:弖>'.'\Dg/&cJVV+2|޽pidEHWgXn-M[BC+ק&y>Y"Gu^=FR4dHC>x@=1\(tK`vwׇi-"E)Ϊ;].aVT<=~ ^!^?\{OpqedX51To?ֳg> endstream endobj 1574 0 obj << /Length 3767 /Filter /FlateDecode >> stream xێ6=_E5cE$EILPh@l,0d"Gg,y8h;2h6FDۋ'O_dDPD..G)1 .7A8qPEATE^.uѪmʳ1 3d1?Y\\6y]e3gcS4˦\.I[.fR.r¬0}^nŘB/َp6f, g$hLv^ J^.ht4'u}Ws)ޗIQU:7M3w522͈tZV5a8Ue ~GF|hsuʅnrӿA%-oθZ`6>@WԬ*׈Gokկ\iUʲ[)ůtC #YPU% TvR.t|W+YbٖucՄߧ$eHXޘEaF9Mc>}Sf\*]LCǶt<]YZiXV)di j"_# "zds /d4 fQ&HB ![[QHFSg6G9JѣތN 1LkHJ%m| iiJ /H`H [ 3Wm0R AkC;t@Rz4i <%8 @ Ͱ R6I: ^%s# ["ӗ:`}uOW׏6_PauϚt0;3ڴP9;]0K[aAaR0[oHV^,m;^Ǭ?KDwnQ݆f"OVY(|c@;h E p3Y!) Iw'0rAlXYs#ؚEOR`/ +iWصiN\38ͷ-& H?!Qo֐V+@9̝0' +RBAp {Ig#6QaGCm9m3iԀrOb~-$uisUP!ʉH)Þwǒ? ʞ3z]m^FDgcy,o$$ntא,hl !\-Ҍi>59G#(e4¢RXTJM :C@BC ˛< +v5YBrGۦ<[3Ӵ3 zK&HK3,_Ppz/TH6 m$gyjE< ̓ I"8]Ny SDar8%'ZHאN!z|%AmR%ߴ~3d}2O)2J~F:m ߺ$HtBfi!dJ#L 5tBz凈Qea<ɭLl͸DuZS$*XS $hb€Π)+8E?)Nѻp~ NkZ)˧85}D#^SjNTD~ad l0XL366oΐKD|=@ ҧ?~HᨳUn5OuÑRX sfLOѳm~vG1{@Z1U˱g+/.^( /OO>8_l@t7945戢Ç<р6d E51y=$Xxc.¤2% rO+[U 4A?~Gq#5"Zkh< z\-6R|tbO*#M4C\8~ of YbֳI킌iR߳nTzk ]TO)05 V%m@_uaS3@`]@?̗2)|>UnjK7h|@F"Q;%s?I?!̆h5rjc]stI#zBChst?|gMԲ/8 ͝@;}/N~ ˢEޗ`@bim7g꿀F* BEm'R#}. y] !=BdW6.E e%|댝j)BC׾U%#~#L&dC)PAq sH&k3$5vK<9aA{O $' >T_Pa쬛 *Q!4qه%`T2AgwkwRkޓzhutpN)Sd_u7hœxڠXQ:}!zxE| P۹1I(Fu 0o- U+3!6ۧkŮٝ#ݠ 8%;8X7TN :[$3h\9>'( ʼnu)Lh޺2ҵc)%Z, ~H왣':[ZKfd}x9ԑ'e&p 11yO&ؾH2X0=-N&zN037&^ߡ/Ĭ"rT L5(l;,c^<>֔t.~X+iKxf Ko<D6+>|"țEbVvl W> stream x[n7}W.ej( $Z$(RFXZYq81{/G%YМw;kXp9IV,7eժ:utҶlRvXZGVPr׺oKSrAO~4-I[\җ| 9okŋWouuZݶ}Y,18zٰO5̋TSSH2 zJh%wVϨ:czP1<вhS+8?Ю?937JY#j]_1 0)ǟq_϶4eh?4bÖf|wLdPcFi؜z9T*WO]TTdTL} F`\mϵnSaN +kƛ8A牚MҘ軮1F#b E ]R@B- >WGjIj@|Pר-ݡUd|6\3qHh#7F .yA&qiLNʵ6B!O!隸?ǹI.czdvpdęMMxH/kt魵̴ԁGCwShm񙨓 m`Sg<N4>H5.\y0Z AB '}.~dP8$:I0:EB5p~1;|wMkjOBX5T=S1x Է,NC>[Mw{(YXod+W֎%j%-GG%b u"Sᠨ^,bV ^+8Kdcϧ8g|ef<ݔ X*; H7l%Hu肆XD1V`a5-6^ /OoI#E9cw㝞ޡ~(ڈQr^k 1l_#!! ÜFnu4j5o{~FS(^%VD.I> stream xX[\7 ~_!>,K6,\ؤ! C.C ;ei}?yvL2ۈЇ#|G%[&-DM4Ҙ램NHdk$fqkwSDM DK"䌤-I9Jhr8NNM8zzsɪi4M(խ+[wi03WZK\D&iep327AT<5 Xƻ=th΃iOС:yay [sހ]&:zo:a8: "-%U?PJ&!1yq':t(L“b&:6_բF,nM:E%#1pUhV}FӢO+CDӢ-x}hѴpd  . Y=`"bfER Ѭ~1hH `΋JG4+fEݹD@;MeDCMcs0$L-7M_;LlP_=>_W9Q0Wx:G]TQp (]A'[kh5Gm W}άd"QheD%9n73i endstream endobj 1658 0 obj << /Length 156 /Filter /FlateDecode >> stream x%ɱ 0F*ܥo LӴdV08/Gp tVSq̚Ό3'E7J.}B!$%2Iy&YlHWȔxDX P9ԃX/*V&1;߳4;|7 endstream endobj 1666 0 obj << /Length 155 /Filter /FlateDecode >> stream x313R0P0U0S01CB.cI$r9yr\`W4K)YKE!P E ?0? J!DH" @ l%r38 H.WO@.E endstream endobj 1667 0 obj << /Length 194 /Filter /FlateDecode >> stream xE1JPHrW`ߠSL!he!V3,ͥd )S\_kYZK]?S*y_X_-6R< YV>]i;} 'ivJl&1r|$IF#[͛$h1A! M1u1bH@ffSؿH_mT endstream endobj 1671 0 obj << /Length 138 /Filter /FlateDecode >> stream x3631Q0P0V5T0P06VH1*2 M 2ɹ\N\ F&\@a.}O_T.}gC.}hCX.O 7>g Pa`D rpAQ\`1h@@ÁVpzrr=1 endstream endobj 1672 0 obj << /Length 114 /Filter /FlateDecode >> stream x3631Q0P0b#SccCB.C I$r9yr+Zp{E=}JJS ]  b<]00>``O0ZՓ+ V4q endstream endobj 1673 0 obj << /Length 171 /Filter /FlateDecode >> stream xɱ 0+ ?D'0iZM ftrC>Z1C 6wrZ,G8уā"Vz˯J-9#cDzŁEqʰ10;- 0q#1™3Zkg6T endstream endobj 1674 0 obj << /Length 170 /Filter /FlateDecode >> stream x-1 0t{wMtB`A'utP:֣3jox?> stream xM1 @D'`ࣽ?B+!Fp Ab!j)r4o2$X8)fIj7Φꃞ$k;v8^$bJ͂Uӊk}ܟg1fNL;v/P"w$cf :ޠBR!e}ƈ1dXƕFOy ([=9 endstream endobj 1683 0 obj << /Length 187 /Filter /FlateDecode >> stream x3Գ0S0P0W5P02S05SH1* @&P\.'O.pC .}8BIQi*S!BA, | 2 6  @b"*y0c`!TDp9@:U@A r |`u@=pՓ+ ; endstream endobj 1684 0 obj << /Length 188 /Filter /FlateDecode >> stream x3Գ0S0P0Q5P02V05SH1* *ɥ`hU()*Mw pV0wQ6T0tQ8A1> :;0c`!~4p96`Tvywu`@qzrrlO9 endstream endobj 1685 0 obj << /Length 188 /Filter /FlateDecode >> stream xm= @O,iraP hSZYZZ(Zg;$@:xwfL܎Æ[!Měd(ެw4H/dOz"t6%=8$2Ί[o@`K ( 5/ .P/ Eu% WV]6d<3SQ endstream endobj 1686 0 obj << /Length 186 /Filter /FlateDecode >> stream x3133S0P0R5T01P05RH1*26 \.'O.pcs.}0BIQi*S!BA,;`a$?``|w: 0@ &~L b<"o@ga`nn:4z.WO@.d endstream endobj 1687 0 obj << /Length 174 /Filter /FlateDecode >> stream xڍͱ 0[:խ4ISNZN⤎O˧2:5Er߽> stream xڥα@ .8%7`A+f(<EHh mӿQ8M HЄSRҁEj=ٟ0S6$Rd Ft܎ȲՌ8@/~/^kڠ} N5N:W& fOi),ϟ)pQ#mx^1ھo$\_XEq endstream endobj 1689 0 obj << /Length 166 /Filter /FlateDecode >> stream x33Q0P0V5R06U05PH1*26 (Ads< =\ %E\N @QhX.O ~10?o`? 30~a{ lTUc`G l sm`w! 'W 1b endstream endobj 1690 0 obj << /Length 125 /Filter /FlateDecode >> stream x33Q0P0bc#SCB.#S I$r9yr+r{E=}JJS ]  b<]X30 ?b70\\\+q endstream endobj 1691 0 obj << /Length 210 /Filter /FlateDecode >> stream x=ʿJP2{Vh ABujiSGˣk7?TF8}+Ssfc>x.QNK+})սJRT-<4W4#3&dod_W̼ DP!gꁄׄ\p#O:i 5pp#`W+P#<V- endstream endobj 1692 0 obj << /Length 216 /Filter /FlateDecode >> stream xe1jPDGl/-?QH*!2`\:#T!'U2kGt2}!'\ypv$۪-${,Y_d׵R;3Rmi qDҺ m&0ywuܻ&HzӀYh(PCofy\ 92%LayM~U endstream endobj 1693 0 obj << /Length 182 /Filter /FlateDecode >> stream xڍM 0ԭ sZ,]AP7JWMt50LM{6ܗ#&OBJ6h쎔9+LnI9߮l1a96r9{* m7KP<@B)0 *P /"H[PGĕ,:ZoS endstream endobj 1694 0 obj << /Length 202 /Filter /FlateDecode >> stream xڅ̱jQ+)aky BBA-b'>Or J̝i? 5FD[#aoJ?;Ӹ#vıt?dء# 5.CPy0{*׃A=3)@|TsxV#%o Z~wg\u`2&|^Je*W=L0 endstream endobj 1695 0 obj << /Length 216 /Filter /FlateDecode >> stream x]αj0hGн@k;!!I !dJ:nѣQ5d-߮\ۖۚ }m$]\zgP,1U |~P~ݰ[̞- d8Q^PQy(YfnbLH+t*]ẓ٩]3. \to#ܘfFf)]OУ4*J, (=詧73a endstream endobj 1696 0 obj << /Length 176 /Filter /FlateDecode >> stream xڍ1 @ ) nvL!bAPK!rD\G6B χIHB+S$VJkL -ib1 qFhܣH%گWrVWU>/džlb7jU/~|7= endstream endobj 1697 0 obj << /Length 234 /Filter /FlateDecode >> stream x5J@e) Hv^@\ XL!XpId-­+[ppSsEoTW",WZTs]Qq#2-P\_+~# +Fhq? 9N#L/z?"NQloF$RBIތ)d6v0S =mlŽ @7Z_}+ݲQ{SA:uX endstream endobj 1701 0 obj << /Length 323 /Filter /FlateDecode >> stream xڍJ@ƿbaLR|z5/jOԞAAMQ؂-+^'iHv!c"NVҔCA}k < endstream endobj 1702 0 obj << /Length 259 /Filter /FlateDecode >> stream xѽN@ pW"y#^.)j#JȀ GG#dP}0p;%cVe]]E+\׊?;O`{~p*T[n6HF\U#` }R>e@3|(Het%ضvS Oy2C{pW=hnG*"bI $H0AdLO2DZ9I[2EmÏ^ endstream endobj 1706 0 obj << /Length 105 /Filter /FlateDecode >> stream x33T0P0bCSsCB. 'r9yr+Xp{=}JJS ]  b<]lC@ 0@DY 'W e% endstream endobj 1707 0 obj << /Length 124 /Filter /FlateDecode >> stream x33T0P0S54V04SPH1*(B%s<,=\ %E\N @BA, ClP` l | 2   \\\M endstream endobj 1708 0 obj << /Length 90 /Filter /FlateDecode >> stream x33T0P0bCS CB. &r9yr+Xp{O_T.}gC.}hCX.O@Pgo9 endstream endobj 1709 0 obj << /Length 109 /Filter /FlateDecode >> stream x3632W0P0S0T0T01UH1*22 ɹ\N\ FF\ f\ %E\N \. ц \. xXQ \\\bQ endstream endobj 1710 0 obj << /Length 163 /Filter /FlateDecode >> stream x3134Q0P0bSsCB.c I$r9yr+s{E=}JJS ]*c<]710A; 5 0`d$@@IJ̡#C$'W ՘g endstream endobj 1711 0 obj << /Length 185 /Filter /FlateDecode >> stream xڍϽ0#lXjDD'㤎]ƣ M_rm9^,d/_=ԩ-퉕@v&#ۛWd@:n)CV%&0j I`sr6Ki>9魭g '+ or*4+vTm endstream endobj 1712 0 obj << /Length 137 /Filter /FlateDecode >> stream x3632T0P0V5T06R01UH1*2 (Ads<,=\ %E\N \. @b<]c u- ? @\ \=7 endstream endobj 1713 0 obj << /Length 221 /Filter /FlateDecode >> stream xEϽJ@ W0/p&y)+Gˣ72B:Oa?vvK)d>߹j4.,W޷Kp~JJPJAb -ȅd"H)ld07yDUl4fuNɣG/fHW&0/-~7sĸ-c@k endstream endobj 1714 0 obj << /Length 133 /Filter /FlateDecode >> stream x3130U0P0bCSCB.c I$r9yr+[p{E=}JJS ]  b<]&0000` ! `0Փ+ ɔN endstream endobj 1715 0 obj << /Length 221 /Filter /FlateDecode >> stream xe1j0bxf˻@")J@@=کC R5 =C=;]"> stream xڍ1 @YR nv6 n!he!Vjih'qGRZp5Lܔs<6lg  ^9l KQ߮z=g|a9Gt)B(PD 1 PBʇK|P\x0> endstream endobj 1717 0 obj << /Length 180 /Filter /FlateDecode >> stream x33T0P0R5T06W01SH1*26 (Cds<M=\ %E\N \. ц \. ?XN0 @Ar?``?h> stream x]1 @RnvA"+P,$(!E n1CXs_qxD:qeı,#e5$_l9eE2hKE T760= ӦAޝG4+ Ya|#xxЂf 8 endstream endobj 1719 0 obj << /Length 131 /Filter /FlateDecode >> stream x3235U0P0b#3sCB.# I$r9yr+p{E=}JJS ]*c<]?0acga0?`D?0o".WO@.y?B endstream endobj 1720 0 obj << /Length 146 /Filter /FlateDecode >> stream x33T0P0bcs3CB.c I$r9yr+p{E=}JJS ]  b<]000 "88{ ?8qV? ?&\= ZP endstream endobj 1721 0 obj << /Length 105 /Filter /FlateDecode >> stream x33T0P0bC 3CB.CS I$r9yr+r{E=}JJS ]  b<]BTՓ+ ": endstream endobj 1722 0 obj << /Length 137 /Filter /FlateDecode >> stream x33T0P0bcscCB.c I$r9yr+p{E=}JJS ]  b<]c`1~``">0Z0'r M endstream endobj 1723 0 obj << /Length 157 /Filter /FlateDecode >> stream x3632W0P0R5T06V06TH1*26PAc#Lr.'~PKW4K)YKE!P EACv ,yv`Q5 ?`Bd7`? Փ+ v?X endstream endobj 1724 0 obj << /Length 186 /Filter /FlateDecode >> stream xڍ= 0'}$ ], ftr'utPtp:v( ԡoOeLF4Ԙȥ"ՖvGL55.yDFآΈ }o J`J30V@ AaK}Ư/橀˥5 6?pq_Qn endstream endobj 1725 0 obj << /Length 169 /Filter /FlateDecode >> stream x%; 1F?p۩$:(+PmYf)H"x _u*[}.  ɖ)\ٟhRް-I/R&]/zIOVS6g5\쨞d-yvT"4h<, "2cA.-^I@aIO0hD8'yMk; endstream endobj 1726 0 obj << /Length 139 /Filter /FlateDecode >> stream x33T0P0R5T06W06PH1*26 (Bds<M=\ %E\N @BA, C ?b4"Q  A20~0pzrrMs endstream endobj 1727 0 obj << /Length 167 /Filter /FlateDecode >> stream x3632V0P0T5T06S0TH1*26 (Ces<M=\ %E\N \. ц \. 0?&01?`  @D2Q%=;g=p`#?``.WO@._x3 endstream endobj 1728 0 obj << /Length 189 /Filter /FlateDecode >> stream x= 0b/ X#~,I> stream x3632W0P0V5T06R01QH1*2 (Bds<,=\ %E\N \. ц \. ``~P~A@,~a`@ F6.{@"  wQ7 endstream endobj 1730 0 obj << /Length 116 /Filter /FlateDecode >> stream x3632W0P0bcCB.#c I$r9yr+s{E=}JJS ]  b<]xlj0ĦPqzrr- endstream endobj 1731 0 obj << /Length 205 /Filter /FlateDecode >> stream xE=j@'T,La]Xs'XE``ra $ 60MG#L! !){LOp\db(^4oHo8HJye隷55ja1R[!rҸ%B~q@$X&JDꉜėa`&hЎzf}Ѳ5J# endstream endobj 1732 0 obj << /Length 201 /Filter /FlateDecode >> stream xUαJ@_,8tIKB` A+ ʃS}fl}n4Lfn|HZh^Z̳{zډ)oՊj*up Q3X;#pC's0 s/ nRЉ~۞\%D~OIC endstream endobj 1733 0 obj << /Length 169 /Filter /FlateDecode >> stream x}ͱ @ BP:w> stream xڅ= @,Ct/ \@7b* V"j)(X٣(9e 1؈̛ua}pr&,};zG̜ÄXd -t!/كcU*).rx7ZrYzq 6d[|.l!ViB\[©&QJ3z@ endstream endobj 1735 0 obj << /Length 204 /Filter /FlateDecode >> stream xM=n@gm4{<H(TD@HIERJ m#lI`׼yW'θ䇜˂w9Xhq5/_Z67\,ȿjLy= W9f0g]D+sbʼn 64ioca\`ۍL{Ӥ^MqlF_R+ L endstream endobj 1745 0 obj << /Length 197 /Filter /FlateDecode >> stream xڍ `4w/Pj)MPԚ>#46_Gth =(TWC# |=yrϭ3;/ft싳^l,N+=u-',]ƠBR"/ w]OJ Hѐ4MJ0?_9.6վэ-iN͋eVL endstream endobj 1746 0 obj << /Length 196 /Filter /FlateDecode >> stream xڍ= @ GbVbh%GH"/Vef Ʃj?8$C(gbg(X]r;fwPL@ | ~nF <z/@:Mrp\3]8[FihHOҙAHVxuO endstream endobj 1747 0 obj << /Length 197 /Filter /FlateDecode >> stream x=;0DQ m#'1Q.@T@Ip'Xz&p:SN8qjhBq&,d p’r^ %mW|O w=ْ\%NfN‚R8Ԏ;?]Aq !帿;$EC3NMYBvҶ6n* UKe endstream endobj 1748 0 obj << /Length 97 /Filter /FlateDecode >> stream x31ӳP0P02PеP0P03VH1*2 (Bes<̹=\ %E\N @BA,BM?AM 'W (V endstream endobj 1749 0 obj << /Length 194 /Filter /FlateDecode >> stream xE10AH MR.@T@I< )) bex{6G9avὡũ4!H#8%5))z -lH1r-9 "HE H$;5ƚ2 @Ll)a7lI3G+lJ endstream endobj 1750 0 obj << /Length 110 /Filter /FlateDecode >> stream x31ӳP0P0TеP01Q03VH1*22(Bs<L=\ %E\N @BA,BQ? C GG\\\0oy endstream endobj 1751 0 obj << /Length 112 /Filter /FlateDecode >> stream x31ӳP0P0VеP0P03VH1*22 (Bds<L=\ %E\N @BA,B@ C \\\HB endstream endobj 1752 0 obj << /Length 180 /Filter /FlateDecode >> stream x31ӳP0P0UеP01R03VH1*26 (Bds<͸=\ %E\N @BA,A<@ fČ@ a@@D9e :ɑ $aU?pH\\\o endstream endobj 1753 0 obj << /Length 179 /Filter /FlateDecode >> stream x31ӳP0P0UеP01R03VH1*26 (Bds<͸=\ %E\N @BA,B@0"00ԃ@D0 @a"W" ",XL/=  p ISp endstream endobj 1754 0 obj << /Length 106 /Filter /FlateDecode >> stream x31ӳP0P0UеT01R5RH1*26 (C$s<͸=̹=}JJS ]  b<]L!W51 endstream endobj 1755 0 obj << /Length 143 /Filter /FlateDecode >> stream x31ӳP0P04S02U06V05SH1* !T*9ɓK?s{*r;8+. ц \. ?조1aPoP`L.D endstream endobj 1756 0 obj << /Length 180 /Filter /FlateDecode >> stream xڍN; @n!swCM*!Fp A+ RK E;!/^`%mȸ 0Ј2 1> stream x31ӳP0P04S54V06R04TH1*24 (s< M=\ %E\N \. ц \. ?aC??@P`4,r endstream endobj 1758 0 obj << /Length 96 /Filter /FlateDecode >> stream x31ӳP0P0@P!Ő H(`\.'O.pU()*Mw pV]zb<]\= endstream endobj 1759 0 obj << /Length 162 /Filter /FlateDecode >> stream x31ӳP0P0UеP01R03VH1*26 (Bds<͸=\ %E\N @BA,<b@N ?8$D D`#2f2X3Iq,63 *@'W yK/ endstream endobj 1760 0 obj << /Length 104 /Filter /FlateDecode >> stream x31ӳP0P0@dbUeh䃹`\.'O.pCC.}0BIQi*SPE!'EA0XA0Փ+ 9-I endstream endobj 1761 0 obj << /Length 111 /Filter /FlateDecode >> stream x31ӳP0P0V04W01Q0PH1*21PA#CLr.'~PKW4K)YwQ6T0tQz ?*1pՓ+ JS endstream endobj 1762 0 obj << /Length 254 /Filter /FlateDecode >> stream xڍ1N0E"49BD.!ҲH@Q V9#Xqabf4ҷoy5w|pw Raͫ~yzH==US53UKnC#[J  K(R( 4]7^/_2 (`\ <Lx\XgA7A:HÝְZjIKj)! "chI(QUJ{շ1Ge/]tG߹< endstream endobj 1763 0 obj << /Length 190 /Filter /FlateDecode >> stream x31ӳP0P0bSSsCB.1s<L=\ %E\N \. ц \. P߀ J2~~d|"N`%값 hL F'y,$33oAYՓ+ H06 endstream endobj 1764 0 obj << /Length 198 /Filter /FlateDecode >> stream x}ϱ 0 [|TkI Nj}>JcҘ 4蠄|4;.ˇ)Jq)+di#  3 bnA5o3bDTYk[z^DyÒ1 <§QSHhUsjD0N/QG<T]KDbh@C63K[xGj endstream endobj 1765 0 obj << /Length 230 /Filter /FlateDecode >> stream xڥбJ@/L i +PysQ%o镶={[r\/䶷\C#;"L E(JdG)23!_#2C[{GE{ʐ :Z2 fFb֘9e)QSFO?V2C鎾?9ru endstream endobj 1766 0 obj << /Length 197 /Filter /FlateDecode >> stream x31ӳP0P0bS3CB.C I$r9yr+r{E=}JJS. @-\. 700& @Y4$)&?H L2A :0Y&q RbbH.C _@|A! HC, !݈I endstream endobj 1767 0 obj << /Length 149 /Filter /FlateDecode >> stream x31ӳP0P0bSS3CB.C I$r9yr+r{E=}JJS. @-\. $BփI uD6`D2JOĥj2|$(47Ae\=WD endstream endobj 1768 0 obj << /Length 141 /Filter /FlateDecode >> stream x31ӳP0P0bS3CB.rAɹ\N\ &\@Q.}O_T.}gC.}hCX.O$3``'Lȁ|DAjD  \\\, endstream endobj 1769 0 obj << /Length 230 /Filter /FlateDecode >> stream xڕN0/?BՅv`b@Lб $R_.jKŊ-}oﳻͦTИJr&7R+Ly?ocv~K*^d`dPɑaDZN{8;@Ά:0GdzT 3#'d!Q M4 >15Ȏ×t*ć5 endstream endobj 1770 0 obj << /Length 105 /Filter /FlateDecode >> stream x31ӳP0P0bS3CB.c# I$r9yr+q{E=}JJS ]  b<] 0 %\\\6Qg? endstream endobj 1771 0 obj << /Length 231 /Filter /FlateDecode >> stream x]ϱn0` n#' v D$:1NClf1t#4Cd?Ka@?>ό/x7e`+/l2 .|w9be;U:.Jp΂Zг j@AR&B n ~x)[yF^}lEƪ# `"P0~? endstream endobj 1772 0 obj << /Length 126 /Filter /FlateDecode >> stream x31ӳP0P0bS3CB.rAɹ\N\ &\@Q.}O_T.}gC.}hCX.O``'!P:'`b\=jo endstream endobj 1773 0 obj << /Length 201 /Filter /FlateDecode >> stream xڭ1 @4 \kP1),J--!9D,,T]S[̃3nQ*9zK5.sWj9!!qSdaV o,cP$nPPBz@Q(>Zll/5.K=&Mإ(o9)[-_m0v`fs8 endstream endobj 1774 0 obj << /Length 199 /Filter /FlateDecode >> stream xe1 0-wӖZtP*AAAQPPRo7iqpT I( 8{~B&6}\9Ol[L,7@g@GEq;>:@8w^@8@X&as!eV^zH4 6Q25> stream x͐=@XL #V &naRK (҂.C l}/N竌BJh&)^PF ] 厹Fq(Eu1 }C$QtQZۂgmJ9Հe 7Fд?oaF k ,|_F&h endstream endobj 1776 0 obj << /Length 182 /Filter /FlateDecode >> stream xڭϱ 0H^{ӐZZ+AAA(}$]8N KM9&xg,\Od+ f.S0~ ,Ђ)qo19/"jB.P;UuDF 'aybhF4j-iMːO*"`a oƅt endstream endobj 1777 0 obj << /Length 217 /Filter /FlateDecode >> stream x1n@Џ(,M#\j%C* *I"Zo+M|k7c܀?WgiG^,#orI^/*J{rϒ._t<.^7!-i<Pqj,Q?vtDQ[ ba~WJ!IdGSI;% endstream endobj 1778 0 obj << /Length 218 /Filter /FlateDecode >> stream xڝн 1 Y|;? N⤎磝"b/Y>Jڤu)&cjiɈZ=qYh>&xޡG*hɘR. eى/".Ҭt ҪwVhO/o2C xBbMn7ݥ| "Ԁ3ï>$$J endstream endobj 1779 0 obj << /Length 250 /Filter /FlateDecode >> stream xu1N@E'rai=1IL,  DѶ. (ig?lncQiں'Tl=yE&lk\FZ,6KNZa| 9|t5iûH Jbz<rd'0 (9qp&8 %?cFi=H^Q #t)g/pxLkDυ3zA endstream endobj 1780 0 obj << /Length 127 /Filter /FlateDecode >> stream x31ӳP0P0bSS3CB.1s<L=\ %E\N \. ц \. D?`OY$$ ;R?$XՓ+ VX endstream endobj 1781 0 obj << /Length 174 /Filter /FlateDecode >> stream x1 @ ) fa n!he!Vjih-GL2 +7&.&RY S2sjOƠZKFe7?/4#ڂJ"nݯ;QO7ZB؈U$fMYD@ ϝf+;|WW endstream endobj 1782 0 obj << /Length 197 /Filter /FlateDecode >> stream x} @\|jU$A:uNձCQWQ /mU%v?,Ŝlriz mgCsb ۓ"[YyAm }$E'}2FZ_V7W?Wei f )XP>V*5yMg9V/zVJAf9'*LCx8 endstream endobj 1783 0 obj << /Length 210 /Filter /FlateDecode >> stream xڍϱj@_pK@{98Sj@-48 -rpnA0f#VX܏]і1ey8%dIL2~Ar<"9hM锳AitJweB# LX6vs`RaF$i4 ;QWytExݭ'{gk~϶,O*$=t %~ endstream endobj 1784 0 obj << /Length 186 /Filter /FlateDecode >> stream xڭ1 @ aM!L!he!Vji(Xͣ)ykk|x$ 0 S9|W> stream xu=N@ _b%79 H" * D[n&"ymafYy.\O:/wa\gVVOK{Ǵý~~|m]=(k}fϋ kEm&fhF hrá +'2ʉ3q4|PY؁0e齳s5\@e'XreSU4Q~MQd endstream endobj 1786 0 obj << /Length 206 /Filter /FlateDecode >> stream xڥϽ 0+->Z+S*AAAѹ}>b$*.bBz:ԥVDJQܣmT;fiTTf3:; :Yc6\;lhkb⍹/N-Z6*p|ZX?4>usn tn N2\KKv endstream endobj 1787 0 obj << /Length 205 /Filter /FlateDecode >> stream xڍn1 ]1%o )$n@S ZYG!i _ϲ=gzp;:٨T6{hh.DmyءQvF0`80cf̱b9)zA}T$"'S|_Q((M I +TPGey?4dѸYz1_ S endstream endobj 1788 0 obj << /Length 220 /Filter /FlateDecode >> stream xڝ; @ )isJE"b=A aS~] endstream endobj 1789 0 obj << /Length 216 /Filter /FlateDecode >> stream xu1N0E*ir ,-D $(VT@Iv(>–)VAaYO??V=ϝz`U6]oX?ݕvⷺ}qE XXͨ̎p[P0LhB M 4ESDiDf( DETHIc %)>/~Œ\r/_})oG endstream endobj 1790 0 obj << /Length 164 /Filter /FlateDecode >> stream x31ӳP0P0bSsCB.c3 I$r9yr+q{E=}JJS ]  b<]300? C=`cf ?F%5Ƅ@.N%\=CSt endstream endobj 1791 0 obj << /Length 275 /Filter /FlateDecode >> stream xڅ=N@ M_(E"T+*AD \%7!H9Ec{BHLid=RI'tT%=VjIM}h=<|ŕԱh UXiSQy :!1{.g t<A9Nt¿ɽ`n [Y'(3@ ~sPoi5E,b6y0ɬ1$V ٺ[Lz #h&;ij$^MR} ^x?m endstream endobj 1792 0 obj << /Length 165 /Filter /FlateDecode >> stream xɱ @ : Y k 7:9utPt>ZpcҘ(@>?1t>C1I0IF*x܂ڡA ʮv@F G` t>'C/fH= b賚'b6l Q"Di endstream endobj 1793 0 obj << /Length 137 /Filter /FlateDecode >> stream x31ӳP0P0bCSsCB.cc I$r9yr+s{E=}JJS ]  b<](B` D00 aDHpzrrȧYA endstream endobj 1794 0 obj << /Length 217 /Filter /FlateDecode >> stream xڭνn0pH' Q" vP+ċekdUGk?>48^iƏ%Ii?1B4,Ⱦr'd Wwc'/kL8TEk%t:u=|?Q ;DN d~U7 S[v0ؼ?bjv? k1N\*7V*=4#S endstream endobj 1795 0 obj << /Length 123 /Filter /FlateDecode >> stream x31ӳP0P0b#S3CB.c3 I$r9yr+q{E=}JJS ]  b<]``? ×0? 'W g endstream endobj 1796 0 obj << /Length 161 /Filter /FlateDecode >> stream x31ӳP0C CB.sD"9ɓK?\ĜK(ʥPRTʥ`ȥm`C}?  Yo`*?!*9=g!@d\= endstream endobj 1797 0 obj << /Length 159 /Filter /FlateDecode >> stream x1 @бa1[ZYZZ(ZoG 2΢]> stream xڍ1@E #0eV$&ZY+h+{4(- 㲘ڼOϛ$ͦ񄇚1'O6MvV6&U~{I7 ֤rkT dR" "/x"o"x Aā, Ң~~5oU9qNȩ9IR 3,hK` endstream endobj 1799 0 obj << /Length 221 /Filter /FlateDecode >> stream xڭбn0bt @Y"QPNt@hycs U.ɺϿm˧ > stream xڵ1 A i832VºSZYZZ(ZXYz#llXZO7荆d/9C;GtVibs0W,lQ9O=l1!洖}N)!0Z2-ygg"(.0P5tŷAUɲ+Y0\%-nYW endstream endobj 1801 0 obj << /Length 218 /Filter /FlateDecode >> stream xM1J`b`w.~7hXW0VbZ * vnUra!,ǔK-tgQ ->Gy劲p3%WtpK-Ϗ kxzX 33䎅rCF40@:b #LɂY.dČ 曶AȺ lB{,Zxώ`1K{+orSN~o' endstream endobj 1802 0 obj << /Length 160 /Filter /FlateDecode >> stream x31ӳP0P0R5T01P05PH1*26 \.'O.pcs.}0BIQi*Sm` $?` #$`'0   Sd.WO@.] endstream endobj 1803 0 obj << /Length 159 /Filter /FlateDecode >> stream x31ӳP0P0R5T01U0TH1*21 (@es<L=\ %E\N \. ц \. `,dF }H<00g?`G"?\=kqt endstream endobj 1804 0 obj << /Length 174 /Filter /FlateDecode >> stream x31ӳP0P0bScKCB.1s<L=\ %E\N \. ц \. 7P& eJ``$? @cg@%4*PFF2?F2~~F2?N7 H{ r V endstream endobj 1805 0 obj << /Length 195 /Filter /FlateDecode >> stream xuν @ > stream xmν 0C(vAAAѵͣ7Q|AwݤGr6&آt&=>'|z zzBQvi z0b zoU YUX)է-ؽFF'{DžyVJtlH!r&u]Ŋ;7RCSQ񋦠iwH>ʳh endstream endobj 1807 0 obj << /Length 237 /Filter /FlateDecode >> stream xeαN02D%{pҊ.TD$: &73Ea+RősƂ)eTQS9mr|IJҌ.kk* C秗{˫3Q&l [f۲cvӨh+켍 R PPÛLm55wۃQ?ڋ_"|v։&Ԋ*Z IM ]4O`9kb{0D>7k endstream endobj 1808 0 obj << /Length 171 /Filter /FlateDecode >> stream xڍ1 @ aM@ Fp A+ RK EۉG(2E:/u ͧB"IIR9|c#ʅgݺ+Kٕr%:/%!ԕIDeoKhѰj#0#0?Y` ` `]ГnS^yi endstream endobj 1809 0 obj << /Length 218 /Filter /FlateDecode >> stream xڥ1n@E?@#\ ^ c)ʅ*q"QQ8%Ŋ2[$r~y.9R3.#OcE_/T_ܙt_?g~)O)'o6`Pv*;k . , UPC< èzDNe{Υ]ɷ~+| 2%E_Iqhҁ x endstream endobj 1810 0 obj << /Length 143 /Filter /FlateDecode >> stream x31ӳP0PbSsCB.crAɹ\N\ \@Q.}O_T.}gC.}hCX.O@l``z 灸C??P szrrRZ endstream endobj 1811 0 obj << /Length 232 /Filter /FlateDecode >> stream xmN0Kxe' 0Y*E"L vd(~wH`O,+¯.wZt7j='(IB??v7ϭo^x# `0#,yB=:F0A.O= {řs2t 9FtJ:ZTTwHsͪTU!,)b")3t#}wo endstream endobj 1812 0 obj << /Length 239 /Filter /FlateDecode >> stream xMбN@ `G"yv~%-a H0u@LбCQn館?I}LeC-c%H00cRbL5hR"&\/d /N^J+Jx#jC(^ Nw6d`NV?1F3:=0+(-aO"{|ldy endstream endobj 1816 0 obj << /Length 343 /Filter /FlateDecode >> stream xŒJ@ϐbaV澀f7H6L!X}&_ `M$sN'))Sft:1er}g/(u\g4.'4ŜW;Y d U K MBxu[7w!P4m 7e +건z(3Za"Ϲ 6oݔ2~okyhuüx,u8U*U%TRh%`YCIv!9Z}m΂{Su4+m gcV'^oRZ endstream endobj 1817 0 obj << /Length 185 /Filter /FlateDecode >> stream x3736V0P0aK #CB.3# I$r9yr+q{E=}JJS ]  b<]C\4 @C{-4;fҌP] A 0hiF4J˃i`ae/F!]BބA endstream endobj 1818 0 obj << /Length 347 /Filter /FlateDecode >> stream xӽJ@ )Ӥ}EVSZ]!Vji(7  Orq|63`gO]!{-d> 9\d% c| Iq"!Y8O`Zƶ֌IRumnYuvLb\d_H׾\ƂTH3 I-sƫIaMUZRl,TʧXIۿa4Te* r;־<'l(u^ф W˝XÔ4ļ:Obn<*ߙ/nBn&_^ZV(,toJʤXXK endstream endobj 1819 0 obj << /Length 141 /Filter /FlateDecode >> stream xڳ03Q0P0bsK #CB.s# I$r9yr+q{E=}JJS ]  b<]A?b0i gh@ˀ'?X$&rzrr endstream endobj 1820 0 obj << /Length 115 /Filter /FlateDecode >> stream x313P0P0bc3 #CB.#K I$r9yr+Yr{E=}JJS ]  b<]3C (Փ+ O endstream endobj 1821 0 obj << /Length 160 /Filter /FlateDecode >> stream x3734W0P0a3 #CB.SK I$r9yr+Zr{E=}JJS ]  b<]S  Q~h8hv݀JPJCJz - xh.WO@.p endstream endobj 1822 0 obj << /Length 279 /Filter /FlateDecode >> stream xڍAK0~ L+%x Ƀxr; Mhf7vӃ$QAgi_]Ǻ> stream xн@ `.<}  b"NI4:(<#w=I qsI@$12 q1s7#ZzNW {#kQlq^@d% r<Q(6AYtɣ )*Q/XzڡndݱGzGuuںzϧ?ۯkulI3JynF&U=e|V % endstream endobj 1824 0 obj << /Length 180 /Filter /FlateDecode >> stream xڳ032S0P0bs  #CB.sc I$r9yr+s{E=}JJS ]  b<]C? ( 0C=UPe3@YH (b'jxe1b `5Y ?@N]˵ endstream endobj 1825 0 obj << /Length 253 /Filter /FlateDecode >> stream x}J0 9`^@輀e͢ ɃxR|> stream xڭ@'l&o`MF,@S^uX`}}}+Sg6ػW33_f8'3]G4fnP)P/XF,|Q׫OQiSfNIس @lUlhmt 3OV GMwڰMmps$r1'RVIƐ4 9{ qf"1w{4ï}fc endstream endobj 1827 0 obj << /Length 250 /Filter /FlateDecode >> stream xұn02XOr/@@`@J̀Svj }eNET6"%tlu:= (vH]z qqŀQG)g#O\FNcz4SzB%~|E}P Jw*,Ym3G†I4e ^aܢr:X|dwI沽%6Q!@ `rώ< j'~/Wf6ʎ 19 endstream endobj 1828 0 obj << /Length 236 /Filter /FlateDecode >> stream xڍбJ@H0MMa!L!heqX偊>>>–WqƋJd횳7Ӗ9wK~j镼pGSZ֒dլMldPdg߆cl?MY'G-IXE&SbDof{ jp endstream endobj 1829 0 obj << /Length 167 /Filter /FlateDecode >> stream x3637W0P0b CB.c HrW06r{*r;8+r(D*ry(000a```| ?00 ??H7Мtґ@ Fpzrrs endstream endobj 1830 0 obj << /Length 314 /Filter /FlateDecode >> stream xڝ;N0E’ { Q4DHLEJcD6)k;9>iVRVϢ.R'XQu)KR/oX_+n-DQ& ü&en[Z~ Cn#0)s0``_p`r*se(ڃ$-SnFȷ0a>2݄LcMp C~ޏ zpi6CB3Gt6Q:xF=fL}.Hysz)܀Eћ1()Qܗ6LY$\Vlv endstream endobj 1831 0 obj << /Length 178 /Filter /FlateDecode >> stream x3332S0P0bSK cCB.S1s<LM=\ %E\N \. ц \. @?BAU2 J{{ H1?R @5 T BS00R (0f?`\=] endstream endobj 1832 0 obj << /Length 130 /Filter /FlateDecode >> stream x363S0P0b#s CB.## I$r9yr+q{E=}JJS ]  b<]?00``E@Yr?H? #.WO@.6 \ endstream endobj 1833 0 obj << /Length 240 /Filter /FlateDecode >> stream x1N0PG)"M#'BViY$R AEt}=Ed=dD9kO:ӸXL7i7CO2\wd.枦.V. Q0Pvv#*<  QРY(P@q>@<ԁ%Xj"DHCHa`tߧmPg%US +#t1 }V= endstream endobj 1834 0 obj << /Length 110 /Filter /FlateDecode >> stream x363S0P0b#  cCB.#c I$r9yr+s{E=}JJS ]  b<]1GB" 1oqzrrz?pY endstream endobj 1835 0 obj << /Length 170 /Filter /FlateDecode >> stream xϱ 0 ): I}> stream xڭ0KHnB&N$Ltr0NUx4Gpd &&߯}j>]b UnBIJ9]! BkA > td# # :ګ^Ԅ/B/ (/)} f>,E,r3/FדG}oI\2i6C=,c1ҭ5yVvӲ{ endstream endobj 1837 0 obj << /Length 246 /Filter /FlateDecode >> stream xڽJ@ 97IB{Azoͣ#C8nM<߲l&bBt^^-!ftiëNk4+%죩or43)[c==GNIV9!3D!3NB#ϞmV($ tGH?J20{h7x> BJO^q)z)?=5.@t endstream endobj 1838 0 obj << /Length 167 /Filter /FlateDecode >> stream x3132W0P0aCS#CB.c3 I$r9yr+q{E=}JJS ]  b<]01?GF  ?0I@ `L0 " fsh'W c endstream endobj 1839 0 obj << /Length 242 /Filter /FlateDecode >> stream xUαJ@YR7p4 ; BBQ8Ȁ/6e8ccwf݉JtXQhQ]X/X"Ν\Xl^bqe, z~zb}yJvC'1{<+; /̴cwz c{NDhHl4 oC _zeJb;? [tZ/s endstream endobj 1840 0 obj << /Length 168 /Filter /FlateDecode >> stream x3132W0P0R5T0P07QH1*26 (Bds<M=\ %E\N \. ц \. ~TBNÉzH07ȃpI("0@"L|`z?В?r ~'W Bf endstream endobj 1841 0 obj << /Length 180 /Filter /FlateDecode >> stream x3332S0P0Q5T0T05RH1*25 (Cds<LM=\ %E\N \. ц \. 0TH1;Ŏ!P d=%P^@{xH1`oR$`(rzrr;7 endstream endobj 1842 0 obj << /Length 226 /Filter /FlateDecode >> stream xmбJA` bto+W1WZYUb)DQpvB$I 3g|4l441lf6ԍĤ[~x&5bMjr艺*R _H+p"VhXx&znP vDJpp8ML:>Q}mڌHC*t=c endstream endobj 1843 0 obj << /Length 277 /Filter /FlateDecode >> stream xmJ0)9}bu{XГ* =ͣQ&li&fe! yJKZdF2QJ.JNOo)1y )18&>?^1]Sɖ9[2 G0:0()ƬL f D /F0H"Xx4pBeq[n$#i}Wv .|P;O05PّǭQc8uuWqUk@Yyz ޔxy, endstream endobj 1844 0 obj << /Length 146 /Filter /FlateDecode >> stream x35ӳ4V0P04& )\\  ,ɥ`lΥU()*Mw pV0wQ6T0tQ``c9a"ꀘ`@?&~@X(A"r ŋtg endstream endobj 1845 0 obj << /Length 293 /Filter /FlateDecode >> stream xu1J@7Lbs3'0 fWL!XEa !ser%{Fccf?df8].|iKfׅ,ӫY&Ue5y}k??_L^PK;"Jbd([TOnO:PvS2PJ ~#݅w;14ZK-~-΋3z&ۣd*ިALʯR 5  Z0D+5 )) "׈k7r|,GVp(8b"U %|9|1Gla5׵3b endstream endobj 1846 0 obj << /Length 293 /Filter /FlateDecode >> stream xuѽN0> stream xڵj@=, 'nД-z)ОzAڛ/Eq.?,vI__v PV嚲Ԥ%ULy"}9!,a="M،> stream xڽѽJ@"M$;= +TL'̣#LnmR?Lv>6b 6J]UilUq2f.o첱U͏8lm<9 ~_yIŅmVd`F&tĆ0^8y Rÿ<|;#>>(%Z-dVZuwujۋ!kV}^7$E}6D$k85`&=Q6YVF9'4|=_ endstream endobj 1849 0 obj << /Length 304 /Filter /FlateDecode >> stream x}ѱN0KNJȀbF xFߙ|y~L2_ՄצZ^EvèC HZdV-dz@W%|k66@T1jڡZNkW~m4~(??%-D^VSAMn5+6ءZCb%c}Md#7ko}5bc!XO0'0? endstream endobj 1850 0 obj << /Length 228 /Filter /FlateDecode >> stream xڭпjAY7y;d WBL@#\"7B\<)2ůet2I,%M%;]݌ōg9ە6q. ۅ7/MZ]:z _N TMVAmm*[ TA\B !,hAS-/tX;φg/hlOg2']=y^ endstream endobj 1854 0 obj << /Length 227 /Filter /FlateDecode >> stream xڝ1n@ER,M @XHZ1."AE)%h{Š5@PѼȦs~7/0fB?[T5לR'ݼ;j̆tٖ, 1=Rl;p'6 w%޶٦|B@c2zVph fHJ"F$+QbE˜ZR endstream endobj 1855 0 obj << /Length 241 /Filter /FlateDecode >> stream xuJ@Yr̡y!u$b`=yOQТA}}>B94!ɿ[dLr:K×7^l$/bm}'W8+yt=q RRJ@q m\iǷ\:mK{t73-(#0q _@ⷿf0ң4/TKxyD)jq =F`J0\Լ#5Qe endstream endobj 1856 0 obj << /Length 198 /Filter /FlateDecode >> stream xڝ 0Eoq(PhP+AAQPQO}i,:IΔܓh4 b֜ 9yЙBf%HYAj͡&5}RłہTr@*MzKE΀N@F x-%08W\g-21鐹WZu2sw[Z,巷EVE\'hbD[ endstream endobj 1857 0 obj << /Length 199 /Filter /FlateDecode >> stream xڝO @Bx @ػ@N( rԪEAEAKQt#ps}x4'cZ{RTYpH*}K@ ]6XV /a& +̌Sv47fUEc]~s|lm[sIaU].Gz]H||-scL endstream endobj 1858 0 obj << /Length 161 /Filter /FlateDecode >> stream x313R0P0U52V01P0PH1*26(XBs<M=\ %E\N @BA, @D( DH '@`H0 MC\$:vmdZn'n\=- endstream endobj 1859 0 obj << /Length 110 /Filter /FlateDecode >> stream x3234R0PbC 3SCB.K 'r9yr+Xr{=}JJS ]  b<]d7z1}@>'W '2L endstream endobj 1860 0 obj << /Length 133 /Filter /FlateDecode >> stream x=1 @E? x+p'c#8ET,mv |>ӜdYx@=w7@p>Å-_ޭp [?"4'ҠK6ɟI&L2LJ%w9 {|ɉ$_ endstream endobj 1861 0 obj << /Length 132 /Filter /FlateDecode >> stream x=ɱ 0  :t AG˛i "~1WOjŊ/|:=PM-_Ul[Q6<*]+a˃.&dR 1YG$ endstream endobj 1862 0 obj << /Length 95 /Filter /FlateDecode >> stream x3234R0PbC KCB.K &r9yr+Xr{O_T.}gC.}hCX.Oz 0X [x endstream endobj 1863 0 obj << /Length 161 /Filter /FlateDecode >> stream x313R0P0U52V01P0PH1*26(XBs<M=\ %E\N @BA,`F0 $;`| NȀ 8a" DH F2-CXHeh6ix-cr E- endstream endobj 1864 0 obj << /Length 102 /Filter /FlateDecode >> stream x3234R0PbC cKCB.K 'r9yr+Xr{=}JJS ]  b<]d7`= S/'W " endstream endobj 1865 0 obj << /Length 142 /Filter /FlateDecode >> stream x3234R0PP5P0T0TH1*24PASsTr.'~PKW4K)YKE!P EA 30z` pxڂ!Փ+ |-s endstream endobj 1866 0 obj << /Length 166 /Filter /FlateDecode >> stream xe1 @Y>"w7+1] Fp A+ j)D:{=JhT, 30FMk6řᓦ+P*NxҒq\ 5Iq&YnIV׬d+E⟮>|>c*xp.|G.BR hiiK/2" endstream endobj 1867 0 obj << /Length 94 /Filter /FlateDecode >> stream xMɻ@@E|wGb BT(H4]])8+61|cZGHO@BJJ7" endstream endobj 1868 0 obj << /Length 165 /Filter /FlateDecode >> stream x3634Q0Bc3#K3cCB.# 9D&9ɓK?\ȂK(̥PRTʥ`ȥm`s3 ;CP$(q`ld6'W 1 endstream endobj 1869 0 obj << /Length 91 /Filter /FlateDecode >> stream x313R0B#cS#SCB. D"9ɓK?\ĄKCKW4K)YKE!P E?|\\\KF endstream endobj 1870 0 obj << /Length 123 /Filter /FlateDecode >> stream x313R0P0W05R0P03PH1*26PA Dr.'~BIQi*S!BA, 2,?`A7801pzrr<; endstream endobj 1871 0 obj << /Length 177 /Filter /FlateDecode >> stream x3532U0P0b 3CB.3 I$r9yr+q{E=}JJS ]*c<]1@  d0%d= u  fh gdՓ+ =ŀ endstream endobj 1872 0 obj << /Length 181 /Filter /FlateDecode >> stream xŐ @ A xxUp( vtr'utPtmh} bIbH `sȑ 3xćd|/'Js61Η|܎Ռ#o򌁎ԥ>Ŵ)mPN8J@ Gּ y[ʽVRl"紦 endstream endobj 1873 0 obj << /Length 157 /Filter /FlateDecode >> stream x3532U0P0b 3CB.3 I$r9yr+q{E=}JJS ]*c<]3w@20$)"@d=e0PPHZBOhr 5 endstream endobj 1874 0 obj << /Length 208 /Filter /FlateDecode >> stream x; 1?XS,vNH#.+Pͣ,-ĉZhk琄onכMxю>XmP8gQ8*ɄtRB: (Y(U ᔣ 3Ԯ LʉXSȼ)p~\'ȡ -#{<{^0?Ը[O(ćJ endstream endobj 1875 0 obj << /Length 344 /Filter /FlateDecode >> stream x]ϱJ@ )q+/IE @-(ښ}iEЗ'+Sɮ] |34۟L轉. ]\>"a=Y-k]2=DZL \έHyf r`AaG*3lK[nČe0 6A>#V~l*}GbKaKiDߋXDרEKӢ[.V7Me  XFWKjX :F,8nMњc VoIA,kMkaZ`*^j6qt\R8aD'=qr_E<N endstream endobj 1876 0 obj << /Length 248 /Filter /FlateDecode >> stream xmнj0P6%hHSB;e(Bұd!C_[ǾBNWd,,~r3}iwo;w>ؾt;Ŵ3+mk`> stream xڕѽ @ G0opz'A+AAA>Zѡܙ^2T)dJh-Q6/.w\ehd--gd;z=74b4)bΘ|!T0' 4, L*0V}Uv~ݷ'CdxxJDv5vwԁ?/uҹ |.uB)&) endstream endobj 1878 0 obj << /Length 269 /Filter /FlateDecode >> stream xڵJ@!E`8yHE p` A+ T,ɣQ)-[ww"ؙe|SǛN )9?fJEnYJᙶ .rR6L'R۫3H6'@hXkcznL 0>[DPiG Ѩ zCt`:D_drfZjF=c?`.-l[/Ǎ;b?Oy=^F.Dd/Z{ FF\n\3w* g7tMVXv endstream endobj 1879 0 obj << /Length 211 /Filter /FlateDecode >> stream xڵ10p lAĉ1D'㤎z@j %Ŕp6# 8CWt4G΀)|4.1!Bv> stream x}ѱ 0ДCpuz_`5ŮDQPQpOS *Rr<.p c4£+(erQep$A/*l> stream x3гP0P0bSS3#CB.SC I$r9yr+r{E=}JJS. @-\. bH700C(`B1P(|Teb P`BPP9J>4 B&!^@\=6 endstream endobj 1882 0 obj << /Length 268 /Filter /FlateDecode >> stream xڵJ0) B_`V4* ɃxR(}=ɔai I:S:<&Iso+ +nZT+T|TKQm=b%0V́a͖A;Cz\;Pf3b6~^\`pfg GDؿAGCF_> stream x3337U0P0b3S3#CB.3C I$r9yr+r{E=}JJS. @-\. ? t 0 lPoՓ+ _* endstream endobj 1884 0 obj << /Length 106 /Filter /FlateDecode >> stream x3633V0Pc3#CB.#3 I$r9yr+q{E=}JJS ]  b<]?``o7drzrra endstream endobj 1885 0 obj << /Length 170 /Filter /FlateDecode >> stream x1 @БbVj(2n6)F#9fÄ4᭤̇1lT+VI9_b1aIJr&S2"/:wԍ_OS#hG1;XD级[e endstream endobj 1886 0 obj << /Length 266 /Filter /FlateDecode >> stream x];N0RDb}RneHJ$@P'b;NA(s#L`$ɏ؞ln]*[gv~ԅGԜ]1g큳-8ڛ\otdd<KKMKMa; .0. >>N w&`>koFc7$կHo8={jp?53j~ M7wנuI}Ri"ӡ|V ! endstream endobj 1887 0 obj << /Length 144 /Filter /FlateDecode >> stream x353P0P0bS#3#CB. HrW0r{*r;8+r(D*ry(2`o^$3^L 0H9$Lց d Brzrr3nX endstream endobj 1888 0 obj << /Length 252 /Filter /FlateDecode >> stream x}j1 . Wo;OZXB{ -= Jt(ڄ$btUd5"|D~8k '9s|)e(gg5JҺ1*/G)g*G=Cl-[VҚCZ 7=+q,A wTÀ&u4-U(qhK$)n;%<.<2!WxS endstream endobj 1889 0 obj << /Length 250 /Filter /FlateDecode >> stream x]ѱJ@ )O`y),DPN:NEn}$!Grq63ͦXYbl1c7ƞ?ж7#zm)~zw{ix7!pu\ĵ khì>RQ|jbJg1T9XN`1,*/rpnLXb95#OSZʜ> stream xڵn01D%8AMPH@fN S(y (96c-ٟ"3(҄lFV@$Chw:;4ˏhSbM7 -oʼ.+aWy!a paN8(e~NHb+[&|EGMl#K!e_<۾zzP<vOlgԬEWGWWO} endstream endobj 1891 0 obj << /Length 175 /Filter /FlateDecode >> stream x333T0P0bS33#CB.S# I$r9yr+q{E=}JJS ]  b<]J쁢 ??0BC0eB} B1PG@\ٹ+ ` (V9(P$ endstream endobj 1892 0 obj << /Length 330 /Filter /FlateDecode >> stream xڵAK@Yk>; !ST)#B@"$n;*c~33Gl&8Op~wzR FZ*dS __U{F\nhG9SVt6JӘ_q- 8;g4Nu bi endstream endobj 1893 0 obj << /Length 230 /Filter /FlateDecode >> stream xڝ @ azyZںfNSu *꬏#x ~iHwfC} /.~?A .Dh~f q+vX+%H cȇ\'izhIi|Ӵ&:/?ռw~R2}6rӿC991uΪ9ɿ66-M쩥]e'(,G%Vɕb0` Ox1 endstream endobj 1894 0 obj << /Length 266 /Filter /FlateDecode >> stream xeϽJ@HMګ̾rw y),J--Gˣ#݂ˍ39TWEn0 *L2~5,'itvck]pXgͥy}y{:ۚ[t 6vZ5'@O6a&~`QLɤ䀄hADDND(An%=٨X }d*;Ad|HףMH+>o ߄k endstream endobj 1895 0 obj << /Length 158 /Filter /FlateDecode >> stream x3332W0P0b3#J1*2" \.'O.pS .}(BIQi*S!BA,0`0?do `L3c ` }L3 D3@hQ'bDc&@ endstream endobj 1896 0 obj << /Length 207 /Filter /FlateDecode >> stream xJ@?Py &^!`B=PBcAE[|4cZ/9avgaϮffzR}\^ 3f/U#n>wX\o;qbLk]Γ4І~,O~=[yR+>ɟ:ᑸgF#bn8&kufY f0AjekQ~uI endstream endobj 1897 0 obj << /Length 262 /Filter /FlateDecode >> stream xmбJAYiSݻ ^!he!B@-s >=`y֙]T;3 etd3zJ 킇Y<:G}GvǨkx|FS2oȹ qĕ+yD>8c8c\4{`*n;5[F6UEJ8YHrg@s/>p뭃Xl7i$S>~w_Wiuqyؤom endstream endobj 1898 0 obj << /Length 321 /Filter /FlateDecode >> stream xuK0+a;h;{A'a,7V4Nz|H&">NN1b\D8!/2Ih2IY.S`e^ Wg`w2]5(kv?=k@@# sG0qТ崖8n֩vI벒<=~ULlab[3'qݧe*&!R3- -*C7. )E{sܥ ). %sc^ CaĚh_eӯ0H:}󦃬\4e~8qC㠡vg穼 endstream endobj 1899 0 obj << /Length 294 /Filter /FlateDecode >> stream x]AK0+^= O`, Ƀzh(=P|It'RVۥ~/Nي/OiTm[UEs?T֥*Y5["v?p=w,~F&rX Bv{[.*:ob}LƔ= ihO[ z<;=J> P/%jBE_R.T, yƞ 3 tiԃf endstream endobj 1900 0 obj << /Length 251 /Filter /FlateDecode >> stream x1K@-f݀,b?B_T|ÿ%t_ endstream endobj 1901 0 obj << /Length 213 /Filter /FlateDecode >> stream xmο@/, ShuqZZ(ښ<$y<–!q,䎁 1v<qƖ혭᭡ٌcfiNmFzƤw>;Ռ 9ξ(35 !+PLpW.Pe@"Qmڢ i"1Ŕ"?OVHnqLUOUo*D6i|UԴiMעL endstream endobj 1902 0 obj << /Length 210 /Filter /FlateDecode >> stream xڽн @ B>Bzm=(vtr'utPvIK: #=vDzAmHJ]t9UgnHbR2pĻ~E;G3=hNa1/kFˈ܉Slx`p:FlTveV`9zTbr^MRV R':q@&x endstream endobj 1903 0 obj << /Length 193 /Filter /FlateDecode >> stream xڝ1 @E00Mntft` A+ RK E;19%Gu`ak?r}YЁ2+bvM6{*+K,Ȥr>bCz+險1CD(p.ܐ lQ4C!i(]W!pE#k%7)%c\_p0T78> endstream endobj 1904 0 obj << /Length 210 /Filter /FlateDecode >> stream xڽ= @ )sĬSZYZZDo7hB\gwJ)|浂F3"$;ԎhbR0 9IbcɄE:ŐdH 5:Pi=uek=BЫjn_t+k-JffLWn噞\y U;3ygz? endstream endobj 1905 0 obj << /Length 204 /Filter /FlateDecode >> stream xuο 0/t(`_@轀:YAAAMj-#8viQp0?K|6隌Nc8Sje57 N-鉌IS>N[ِ҃ / '+*F PWR7HU8##;o\]>K-AZ//>L^T^('N"nhAUhwdZ#=d# r!I endstream endobj 1906 0 obj << /Length 143 /Filter /FlateDecode >> stream x3233V0P0bcc3CB.c1s< =\ %E\N @QhX.O ?00``?<3c:f 1%P}Pszrr_ endstream endobj 1907 0 obj << /Length 261 /Filter /FlateDecode >> stream xڕ=N0_4>B|ȏ宴,)@T@IF(' 9e kb (H/˛tG}wȣ]>n~ndKuETgK՟秗{V'fӍk^".ٷtD\0_f+`Gt~΀~ܡLcxc㠤2%Ii(ئ4rB8F+ OƬܫ>Q=9'|V)X,i/ mh endstream endobj 1908 0 obj << /Length 165 /Filter /FlateDecode >> stream x3532T0P0b 3cCB.S I$r9yr+r{E=}JJS ]ry(0`$;dt"H @҆ 3g`D\$3ؑr y endstream endobj 1909 0 obj << /Length 124 /Filter /FlateDecode >> stream x3234R0Pc#3CCB.CK I$r9yr+Zr{E=}JJS|hCX.O  P001aqzrrHT endstream endobj 1910 0 obj << /Length 150 /Filter /FlateDecode >> stream x3233V0P5Q5T02P03TH1*2 (Xs<=\ %E\N \. ц \.   P( ulj3cP\=v9E endstream endobj 1911 0 obj << /Length 213 /Filter /FlateDecode >> stream xڽJ@`_@> stream x3234R0Pcc3cCB.#rAɹ\N\ F\@Q.}O_T.}gC.}hCX.O a0\=W endstream endobj 1913 0 obj << /Length 186 /Filter /FlateDecode >> stream xб 0  Ej3:9::( NGˣ:4qqé8Hķ)tJRWI8^0(v$kgfAuFX lYhFAQJ*˂Yu*>P'sx'`‚ʷs3 endstream endobj 1914 0 obj << /Length 154 /Filter /FlateDecode >> stream x3532T0P0b CB.S I$r9yr+r{E=}JJS ]ry(0` iH~`~ ?3 !d; \\\ep endstream endobj 1915 0 obj << /Length 188 /Filter /FlateDecode >> stream xڍ1 ` _qVdV8h֣;5I䅼Dq><Y>X:SwN'Js2c2 K^nG2jƖL[H5pG %BxʃAxNӃX:>ŴI=JRh4 V\_螡yNkPM endstream endobj 1916 0 obj << /Length 208 /Filter /FlateDecode >> stream xڭ1 @YR &] F0X`er%GH!u6 Z+ٿȧ>u!)P)N}fQrIJr2xޡc PƴSkLbF{z`)3Apڀ\A4ikh+/;Ň/׊y.L[ov3_nBk/cC:՘M$;| endstream endobj 1917 0 obj << /Length 223 /Filter /FlateDecode >> stream xڭϱjAbaH`B*EҔ)H!GGGuγFlbv;x$s ۘ7 }Mes֟4ɼMdN&2~<:ELtjtDEX pPYuwoڽmOX6JCT:knGC_ 8=Pח4%o endstream endobj 1918 0 obj << /Length 145 /Filter /FlateDecode >> stream x3634S0P0bccCB.c4H$r9yr+p{E=}JJS ]  b<]?~x?̟0~g 0cH`3szrrM[ endstream endobj 1919 0 obj << /Length 203 /Filter /FlateDecode >> stream x-AjP?d70sBtB[ܙʷyŷqRq,Q^i4d6Wd4&S/y&3[ْYqgc$Ovw x 4tHB8tmԨuUupAD#r&iNBKZӚ.8W endstream endobj 1920 0 obj << /Length 151 /Filter /FlateDecode >> stream x3634S0P0R5T06P05SH1*22 \.'O.p#s.}0BIQi*S!BA,Vl+313C1#T8fq{v r wSM6 endstream endobj 1921 0 obj << /Length 160 /Filter /FlateDecode >> stream x3532T0P0R5T0P01PH1*21 (Bds<LL=\ %E\N @B4РX.O `G%00a`f$Н l0A?? $@?P'W rjy endstream endobj 1922 0 obj << /Length 193 /Filter /FlateDecode >> stream x]= @YB\@71JL!he!B@- 19%GHi|{ G.?'T>.o=(D"壜q2|ݣ-䡜К6N(]9' {6*};:fViucfU)1[m?6*q_D endstream endobj 1923 0 obj << /Length 248 /Filter /FlateDecode >> stream xmϱJP? Ĝ{j@`N"QPQpNP|d63|p?=.%ww =Qjx>礯85eM:{SNH:asCºXWU<&.*;d (Faы> stream xMαJA?lq0lk!< 8B*ERKHzh(ufL4/Y_Ttz%RKxMGnܓtu|9}ۼ> stream x]J@; x%'S~\#^/4Iq1w-}<9&{@ 7lz P@?[VqtPA8.=փdFDb+8w:+cw9<<#Oʬj\ԯR*ٕmm`giM?AP endstream endobj 1926 0 obj << /Length 199 /Filter /FlateDecode >> stream x5= 1oI!Ls5"Z-n!he!Vjih9e`i!0 #vܗ|Ltԇ&% {Ov!d%_/#j9osv;*g7  $Oy $m}RK ****IQ $ Q&2JuWk D$_h^Қ8.G endstream endobj 1927 0 obj << /Length 191 /Filter /FlateDecode >> stream xڽ1 @EL2͚DL!he!Vjih'(9B -)fsofyH0d@iNjő!5T>'.&Ien(@*/SC^^$N-8b,(p OA-iU۹*m_ ڰ^!c9- @m endstream endobj 1928 0 obj << /Length 126 /Filter /FlateDecode >> stream x313R0Pbc 3CCB.#K I$r9yr+Yr{E=}JJS|hCX.O @@\ <yP\=sU endstream endobj 1929 0 obj << /Length 242 /Filter /FlateDecode >> stream xuJ@% f$,ăSZYHV2u@J&llDmwjR_@> ; l?hٝv* ؄'!nE5i>p {Nhd42FJgaZtEjZ!'Ēh } lV~hp endstream endobj 1930 0 obj << /Length 246 /Filter /FlateDecode >> stream xuN@ ed!/kBR$2 щ1c<==B ULp?77K%5ZGo?nY'[,vxK톞U/m#yTLP%d'd`ofAِ'~VN\'(uˈn(Eu,_ڡgŨxqGc/VJs5M#1%-~nn endstream endobj 1931 0 obj << /Length 185 /Filter /FlateDecode >> stream xڕϱ @ BP:w> stream xڥбj@ @_ 4z|k:ŐPdc! (= t:IÄS~¹ӑB)fgW)NnxfOnk醪9 mvarU();'Q/$C 3!`.z7l(ki?n!a먠luAIu2θ ҫq42BT]E endstream endobj 1933 0 obj << /Length 249 /Filter /FlateDecode >> stream xuϱJ@?LᾀSZY mr| 3J duM/|HհXhS-7YwkH~Ͳݍ~}~JR>Z=x~]賑ix o@ʺ\ur'x; endstream endobj 1934 0 obj << /Length 202 /Filter /FlateDecode >> stream xڕ; @A  M QL!he!9ZGHB;RXW?#l hS+z*ߛņBQK'jF;LA0Ӫ]WCȋz&\e (t XLݷ!)&e~R27ukd1y ]msXp endstream endobj 1935 0 obj << /Length 277 /Filter /FlateDecode >> stream x]J@EoH$j),J-- ɧͧ']ͪ x}|)TrBf1^զUٽʮYVY{#/*^J<?v+'@-0#" |'İ+Y X9"1fm)ӎz+~x/ȇ3FY g,ڏ@'DV{:RRh4zQc;uD*` "Ah^ᰥKp[Vݩoqg endstream endobj 1936 0 obj << /Length 252 /Filter /FlateDecode >> stream xu1J@?f. ]@lbځuSZY,Vj)h+ Bn+oayTg+ιW?Be9 e\֔]6e?R>炲 o 0@:)vr"yĂILiNיvFoi􇐃 `%<)ơYto%SK u+P ˴N2(vGU*DF I2v: endstream endobj 1940 0 obj << /Length 137 /Filter /FlateDecode >> stream x%; 1F;]]hL!he!Vjih7eIY@5`NKnn;[.>Yʬz8nQuĥ>W#D*L"QCĶ5e" ьwO)B endstream endobj 1941 0 obj << /Length 132 /Filter /FlateDecode >> stream x313T0P0S01T0P05TH1*26 (Bes< =\ %E\N @QhX.O 27??~0?P`G( endstream endobj 1942 0 obj << /Length 192 /Filter /FlateDecode >> stream xڅ1PDPl Ċ1D+ cmq@IA;WL0 v xlagnEt4'g'Ty!n{> stream xڅO; Pl {I*L!he!Vj)h-G,-$q̃T;LNuihuɗV'/2O4Ĭxq7 $$M | ,G\W{F9^ـ"J[|rY"ֱ4nT?pGrjݬc_e*[M* endstream endobj 1944 0 obj << /Length 167 /Filter /FlateDecode >> stream x313T0P0U0Q0T01SH1*26(%s<=\ %E\N \. ц \. L@$AD=$? ?@P&VV̌...SG;&.WO@.n= endstream endobj 1945 0 obj << /Length 96 /Filter /FlateDecode >> stream x313T0P0T5W02S0PH1*2 (Bes≮=\ %E\N \. ц \. (\\\&Q# endstream endobj 1946 0 obj << /Length 162 /Filter /FlateDecode >> stream x] 0->KNZ N⤎>cbMN8>] y GGbO%T2[0YFK&pOdLSAZZFHW 2"L}Tߩoﻭ "Іֺ? endstream endobj 1947 0 obj << /Length 114 /Filter /FlateDecode >> stream x313T0P04W5W01T0PH1*22(Bs<=\ %E\N \. ц \. a`?r 5ez endstream endobj 1948 0 obj << /Length 116 /Filter /FlateDecode >> stream x313T0P0V5W02W0PH1*22 (Bds<=\ %E\N \. ц \. c``pzrrlI endstream endobj 1949 0 obj << /Length 152 /Filter /FlateDecode >> stream x313T0P0U5W0T0PH1*26 (Bds<=\ %E\N \. ц \.  @N5 D؁{! ?8$ &> F0Tta*`gr W: endstream endobj 1950 0 obj << /Length 175 /Filter /FlateDecode >> stream xڵ 0DQXK'2҆  * D h%##6HWYM0p sf؜Tz2{XKf1)Kd*rdGR/RA-%a|ݠЂV$QoeUG+O;a endstream endobj 1951 0 obj << /Length 171 /Filter /FlateDecode >> stream xڵ 0EQ  miCp  (0 i~ϧ{~37 <& ~9JϓJu }s7&xܟnKœ(4^Jq^.JNQr?)F#PQ1H)3R;;J~.؆xC?ZOYb endstream endobj 1952 0 obj << /Length 104 /Filter /FlateDecode >> stream x313T0P0UеP0T5RH1*26 (A$s<≠=}JJS ]  b<]'W * endstream endobj 1953 0 obj << /Length 148 /Filter /FlateDecode >> stream x313T0P04U02R06P05TH1*24(YBs< M=\ %E\N \. ц \. ? 0`77g.`r j'. endstream endobj 1954 0 obj << /Length 171 /Filter /FlateDecode >> stream x313T0P0S0W0P01VH1*26(%s< =\ %E\N @QhX.OXǏ?1 ɁԀԂ2} pzrrxS endstream endobj 1955 0 obj << /Length 116 /Filter /FlateDecode >> stream x313T0P0V0S01T01QH1*26E-ɹ\N\ \@Q.}O_T.}gC.}hCX.O A-4v@ ù\=emH endstream endobj 1956 0 obj << /Length 136 /Filter /FlateDecode >> stream x313T0P04U54R0 R M F0\.'O.pC.}BIQi*S!BA,???PP'W ,5 endstream endobj 1957 0 obj << /Length 99 /Filter /FlateDecode >> stream x313T0P04F )\\@$lIr p{IO_T.}g E!'EA0XAՓ+ ; endstream endobj 1958 0 obj << /Length 157 /Filter /FlateDecode >> stream x313T0P0U5W0T0PH1*26 (Bds<=\ %E\N \. ц \. @#HD؁:Q'@&> f0d82>3 df Dpzrr@: endstream endobj 1959 0 obj << /Length 107 /Filter /FlateDecode >> stream x313T0P04F f )\\@ IrW04 s{*r;8+E]zb<]:\={-= endstream endobj 1960 0 obj << /Length 155 /Filter /FlateDecode >> stream x313T0P04U54R06P06SH1*24 (Xs< M=\ %E\N \. ц \. A# ?0` @.WO@.8 endstream endobj 1961 0 obj << /Length 110 /Filter /FlateDecode >> stream x313T0P0V04S01T06QH1*26 (Z@ds<͹=\ %E\N \. ц \.  \\\A endstream endobj 1962 0 obj << /Length 103 /Filter /FlateDecode >> stream x313T0P0W04S06W02TH1*2 (B$s<,=L=}JJS ]  b<]0 szrr$~ endstream endobj 1963 0 obj << /Length 117 /Filter /FlateDecode >> stream x313T0PT02W06U05RH1*22 ()Lr.'~8PKLz*r;8+r(D*ry(01l;cNJ l r \+ endstream endobj 1964 0 obj << /Length 247 /Filter /FlateDecode >> stream x5ϱN0 `G"y#/i+)qHt@1#ӥG#dLǿk.v5^6+j'{th﹌[襦4) paYY QD+`|.;1r ĚX67 !0Z6 sI1{8bgU3/BF ))s9r'A endstream endobj 1965 0 obj << /Length 239 /Filter /FlateDecode >> stream xڍ1N0Dg"o|$Q6ZZHPQ *!p!eU8i=opZ-uC玝|H?Я\~4wJ3޻MÍ?ε/2"P<> ufA@5ã`cO4s1d1gʮɧ:eP~Kٜ-˺QvOh9X܅H$% RM Zlmb dr)}A!> stream xuνJ@YRoyMry),J--$ba#*Āb8Gi+9/w]iF ftQ5 sʟjN\`v 1f!]b:ڣ5a&HzЃZ](&Dv) ZEֵ^mVvjRPkY-B4x1+ɛ>[OB:@|ӃFA:nKe4bڈq4Kmۘx~⃌ endstream endobj 1967 0 obj << /Length 191 /Filter /FlateDecode >> stream xڍ1 @EGR2'pa!F0Xr!E8 I 33;MR(oSJb:xEk%GU/hvd-Lkg4fAs \102w{(` UO\+rBZt%p#'*=J@ ҿϫFa;Whn I endstream endobj 1968 0 obj << /Length 184 /Filter /FlateDecode >> stream xm=` .߁1D'㤎]ċ8p n #~$(}L> stream x}0K:#pO`i1NI4 Kd0FMj\ijx@½%\PPGL2P[2;|=7P~K<Ls 9y|9#l K#vӜ_[ZCN _CF,a8[NXTQ endstream endobj 1970 0 obj << /Length 218 /Filter /FlateDecode >> stream xڝ1N@4QY AT (Ar 3AzWJ_kN|y9H/vI'Zun8-)\ؙBwoVWg)6r}Gݚ3J~ ZTMa.)- o̤/`tR27V֯ifhh`+-RN]dvg9 endstream endobj 1971 0 obj << /Length 183 /Filter /FlateDecode >> stream x313T0P0bCSCCB.c I$r9yr+[p{E=}JJS|hCX.OD|?b0 AD}&> f0H0b!On%rv?s?>  `szrrǁG endstream endobj 1972 0 obj << /Length 147 /Filter /FlateDecode >> stream x313T0P0b#SCCB.c HrW0r{*r;8+. ц \.    `|$lthvb)،6 Q .WO@.̌r endstream endobj 1973 0 obj << /Length 145 /Filter /FlateDecode >> stream x313T0P0bCSCCB.c I$r9yr+[p{E=}JJS|hCX.OH" $`@CLmQD !( ,x endstream endobj 1974 0 obj << /Length 227 /Filter /FlateDecode >> stream xڍ=N@\4PY AT(PR$ގk 7eUI"Q|{;5袥aC]8> stream x313T0P0b#SCCB.c HrW0r{*r;8+. ц \. ?c4 N%'W  endstream endobj 1976 0 obj << /Length 108 /Filter /FlateDecode >> stream x313T0P0bc SCCB.crAɹ\N\ \@Q.}O_T.}g E!P E >Փ+ HX~ endstream endobj 1977 0 obj << /Length 218 /Filter /FlateDecode >> stream xE=n@E.,MvNm M,#EPR%)SB9QPr.]lȢOLt&c&FRf1K~|U.k9s endstream endobj 1978 0 obj << /Length 123 /Filter /FlateDecode >> stream x313T0P0bCSCCB.cs I$r9yr+s{E=}JJS|hCX.OLŘN|? ?*f endstream endobj 1979 0 obj << /Length 177 /Filter /FlateDecode >> stream x313T0P0b#SCCB.c HrW0r{*r;8+. ц \.  B`W${1y 01h͇q|Fa  l?`!'W , endstream endobj 1980 0 obj << /Length 194 /Filter /FlateDecode >> stream xU-@%&c 迨 P$u[GEev K1h8&nL؃-;CFXA_>pi ?!&+R"c(ɉ(N+ƵGSroW\"Ϡ+tIߣmśh5| dXB]/qs| endstream endobj 1981 0 obj << /Length 170 /Filter /FlateDecode >> stream xŐ1 @ERxt)R-n!he!VB9EqW7seϨxAƘxң3U5ݮr 쀾"h `,T'uID x/H 9 Zpqol endstream endobj 1982 0 obj << /Length 174 /Filter /FlateDecode >> stream x313T0P0bSCCB.cs I$r9yr+s{E=}JJS|hCX.O0"370`H؃@`?#^^Q`Cƃ-Y  f $700 F"b\\\wN endstream endobj 1983 0 obj << /Length 209 /Filter /FlateDecode >> stream x1n0/ʀ! &HYj کC @9j1CNjKޠ{iˊs.y^,V\.x_ЉۜWH[KEԯ|9_do\g ƃHLd pLi'Ai ?NI i&tZ0^gȅX{cY701<5  endstream endobj 1984 0 obj << /Length 197 /Filter /FlateDecode >> stream xڕС0jrf{::"#a e0XvtmCOh)T^ aLiOvG ֤FscT,r0ʖSiNfEN`Y9Q3pqNN3O0n ZJ4&}5ty+A -ؼ+ԀW2>z endstream endobj 1985 0 obj << /Length 236 /Filter /FlateDecode >> stream xu1N@ E"a|$H" * DH$*\!G2HQwmT 娔DJsՠg?x#Um<>r\Iq+wn˜24wC0MLNLtA 9a=tC68yF̛aO2/a<&E>oxv endstream endobj 1986 0 obj << /Length 124 /Filter /FlateDecode >> stream x313T0P0b#SCCB.c HrW0r{*r;8+. ц \. @†H0 z(QՓ+ +T endstream endobj 1987 0 obj << /Length 167 /Filter /FlateDecode >> stream x1@G(LtYY +D ,ZZhq@IaGhf'_Ϭgɂ#}SqblF.b27+e=Z3bÏB&.ْ`9:Rs)U*H]J^w¤%HRQC/~*hGo8 endstream endobj 1988 0 obj << /Length 189 /Filter /FlateDecode >> stream xe;@!$p,1q ,ZZh.F5\5o2s? q98^}G|9^0vV2#kgdfAYL{NELi iww?>, ʐ ] xBi LHʛ1VL0JRaVuZ-Vi endstream endobj 1989 0 obj << /Length 197 /Filter /FlateDecode >> stream xڍϯ P#)>tœ &5m.b_CYN wzto,NvE69Wh .-rZeD/@sL@56Mo%n} :}v%$@FTiXz[V!zyM-+_X=Ey>J3CN.{K endstream endobj 1990 0 obj << /Length 192 /Filter /FlateDecode >> stream xڭ= @ )"U F0Xmb aҔ)®p)6 GqBQ@O[SQ6{ t&NExޡ9OA q@#~8 7ŝm'ch/m:^[ endstream endobj 1991 0 obj << /Length 182 /Filter /FlateDecode >> stream xڥϱ @ Y| j;:9::(}{{3!HŔĔ'tIio _Q[z>^WnEWtL(>a]Q3-c'4a|` BAI=EzNGKC8e  p&ȕ5 l endstream endobj 1992 0 obj << /Length 191 /Filter /FlateDecode >> stream xm= @ x Ղ?` A+ RK E[)S,;h%Xfh< }:ex\T:8^pVQ>EmqF;)C}FE$ sXBט^Hȃ@?|bezYETZ_q-`R!a~K<.Kj/\ endstream endobj 1993 0 obj << /Length 187 /Filter /FlateDecode >> stream xڝ= @g"#Xraˀ!N;GYg!BR@[]/w%ܔ|q&?,Lƹ+x"ҡ@yRx -0遍~*?umֽr!0e] EӐ`%Ж*sz endstream endobj 1994 0 obj << /Length 182 /Filter /FlateDecode >> stream xڍ1 @EIk9 n!he!Vjihh%GL2Φօ}g?ofǜlS>'t#k5?;2{Zd܆L]rBC\"iJzD=[5/jLAOQ~ߏ@B_Zh4J5Ϋ^RMuZ9uEJ endstream endobj 1995 0 obj << /Length 193 /Filter /FlateDecode >> stream xڕα@ .<} L &`qRG;[pqᾤ 5)+H+9s<^&|XLפ*L,r0S⺡MNMC $z11wx!"><Zi&N?>cH RaH'c ˁ:ѴmO, YK endstream endobj 1996 0 obj << /Length 201 /Filter /FlateDecode >> stream xmPE4K BBrmM>}}V́;ܹiԥS=T'u9&a+NFF⻥OK+ VZ[( f#2;܃J>PDCv@Z }•cC 7'* 4u.7mp b2rcZI_ endstream endobj 1997 0 obj << /Length 154 /Filter /FlateDecode >> stream x313T0P0asSCCB.c1s<=\ %E\N @BA,@Az H?*;&p4Aka[~ `1.WO@.^ endstream endobj 1998 0 obj << /Length 253 /Filter /FlateDecode >> stream x}J@#E`}!k.p` A+ RK E#U(y[,gǰzqꜟJz`;볟 Z.(wk~x|ws%{/xv4lnfxYDdItSn\#7@efd=`El6X4jB*`f}E_h0bj1SL̀,x>v*!*:MƢ:?-y%ۧF@-7> endstream endobj 1999 0 obj << /Length 161 /Filter /FlateDecode >> stream x313T0P0bcSCCB.1s<L =\ %E\N @B4Pe,B @d ?  B~oAd $?HzI8'W z endstream endobj 2000 0 obj << /Length 132 /Filter /FlateDecode >> stream x313T0P0bcKS#CB.cC I$r9yr+r{E=}JJS. @-\.  @x@@?C1;}pA|.WO@.O) endstream endobj 2001 0 obj << /Length 169 /Filter /FlateDecode >> stream x͏= @_#d.͟ B Fp !VbnxK q\`eW񊉣~2c!GOj .mO1dXV|-M -X endstream endobj 2002 0 obj << /Length 198 /Filter /FlateDecode >> stream xڝ;@%$p.H)L0VjiVW(x[_~0E_cƃ=2b4gA ΄Sp)-8lsQy endstream endobj 2003 0 obj << /Length 115 /Filter /FlateDecode >> stream x313T0P0b ebUel䃹 \.'O.pc.}(BIQi*Sm`Pz<7,{\W endstream endobj 2004 0 obj << /Length 171 /Filter /FlateDecode >> stream xڽ= @[&G\@7!Q1#X^,7[n8ȃW3r9Al&]'-\,cx܎` s0 n ==Cbq1 SeKvI'mr/)T8R`5zf endstream endobj 2005 0 obj << /Length 155 /Filter /FlateDecode >> stream x313T0P0bcc3CB.1s<L =\ %E\N @QhX.O$$PD2`$ȃ@H&?:7 q.WO@.ll endstream endobj 2006 0 obj << /Length 183 /Filter /FlateDecode >> stream x}=@XLvNBLH0XF[٣Q8ab^2}KJ)*%Kw4 +@@)juE]VQzB[_P :9o.A@9(dq%7@'a/=ߵG.^Tyh p A!\\[>P: endstream endobj 2007 0 obj << /Length 200 /Filter /FlateDecode >> stream xڥ= @g fI"SZYZZ(ښͣ[.(wS|7q4HRYs_8 LWCNv?$#(%p:lHj&5pGٌs V,S*7;(&A]t, -GT@8=F> $_ȥF<5ޯ endstream endobj 2008 0 obj << /Length 211 /Filter /FlateDecode >> stream xڭ= @ 4 وVVb&7J{ Lig Z 6_B޼q;QH1.#ܡ$ )ѯO-3 # ƒcM?n0O$!Wɾb|31P_6rilxz+=Տ>jO=]quBVŴ~[)D\|kse8'vG endstream endobj 2009 0 obj << /Length 158 /Filter /FlateDecode >> stream xڭ1 @ПJuj!Fp A+ RKAEh9JAqc![̃I`4-ØԈmjw쎜{Vky\Y\/|9êe_Hx+5C8#$RC\B"xo<Iw endstream endobj 2010 0 obj << /Length 185 /Filter /FlateDecode >> stream xM1 @4!s7q5@T0XErr,,2ԎgDM&rv=pr^ًYMyaoY!RrGB7 }KD#"eZSW!("PB Ca}96A=> stream x313T0P0bc 3CB.cS I$r9yr+r{E=}JJS ]  b<] @AH2`h AA~[@ Lx:B endstream endobj 2012 0 obj << /Length 148 /Filter /FlateDecode >> stream x313T0P0bcc3CB.1s<L =\ %E\N @QhX.O` $0()D? d=H2cģd> endstream endobj 2013 0 obj << /Length 186 /Filter /FlateDecode >> stream x5= 0W:oN`B`A'qRGE7^̭ ء4ؔ? ,&Q@>0[}pb*Q)QzܟvI>>yG:J^]S |-,ZHZX:^<r[C准qzb&gaQ$L endstream endobj 2014 0 obj << /Length 174 /Filter /FlateDecode >> stream x313T0P0bcc3CB.1s<L =\ %E\N @QhX.O `?aC00~ @2?Dv`N2~+ߎ #ȏߏ`` ?G#g``?A6 H@RՓ+ ɝm endstream endobj 2015 0 obj << /Length 202 /Filter /FlateDecode >> stream xE; PEoH!LUBBBN!۲t @!L@,a̻{ې lfOÄܒZrɌOp>ܘW!kJ/LnRQ;H(+p{h/ O.ok> 44W&F&R$}xY& endstream endobj 2016 0 obj << /Length 237 /Filter /FlateDecode >> stream xEαj@ dz)CB=ҩCɔdnvj:t&=$%p!:d-"zX!ZnhyxDQd}LKႲ)ֳ[{vȭ+OPy5 @U-G[;z[*lB;v\ɼHer;SHR Z88 ~Ka{ endstream endobj 2017 0 obj << /Length 176 /Filter /FlateDecode >> stream x}1 P S2Y<9*BV N⤎G(Ϥc|?!?'S3>gt#͔+^wr~ÏB.9#W!H"Px+"B I / >i`$f_$hj(D{{-ӎ~b endstream endobj 2018 0 obj << /Length 203 /Filter /FlateDecode >> stream xڝ= @_L#8MLRL!he!Vjih'({!q-6߲`}t!'<8 91 ũ piNfqJf)c2ot=̜w{@^m W÷x: dTLdO_'X`*w]!WҢqz9KU" }}d endstream endobj 2019 0 obj << /Length 141 /Filter /FlateDecode >> stream x313T0Pac S#CB.# I$r9yr+Yp{E=}JJS ]  b<] X큸7001;j?0FJ endstream endobj 2020 0 obj << /Length 222 /Filter /FlateDecode >> stream xe1N1E*i| .-V Ab $(UAݣ(>B,?kWEwk.i;O%/$=iI^>$nF6x0ڄʬ ͎X⌾T~fGvlgOȠ<|HTGǂ+ˇD5WTL3*=2,<8h endstream endobj 2021 0 obj << /Length 226 /Filter /FlateDecode >> stream xEнN0 J^ @ZHHCL @>ZlDZTe}9W|Qps}ů}PYkP|N#5[ Sj~??ScNzDDFM&4=:4WL hLVښQ5A1;,wKi sęǐ dw;-y"ͧ\ۼ>[z3Vc4 endstream endobj 2022 0 obj << /Length 181 /Filter /FlateDecode >> stream xڕ=@!$p. b&ZY+h pJLh$%^5Y (xTHN)74 U[QcL uMĄB9ƛG3a(if M( /#`cV2OZ˿Z;5t endstream endobj 2023 0 obj << /Length 207 /Filter /FlateDecode >> stream xڥ= @4{t&)!BBB,xxqFE惝}ov)ZRGk;Sʱڬ)Nюe6aܠOi(Zb>$\Cǹ.5Tº)7 P \)'ߘ'-,e$9ґ i `AY ֚ G9-c endstream endobj 2024 0 obj << /Length 241 /Filter /FlateDecode >> stream xm1N0E"4 @TE"Th+)S ͓=3uE5w|pWs/ 5gFGn{n5j+UknS=6@! `dHp糢0g0p \ύF<'"DMbLz[Zj6]*7DE??(jALP5ˠGԡ(OY*G@BR栛 5pI endstream endobj 2025 0 obj << /Length 183 /Filter /FlateDecode >> stream xڕͽ 0+- h NB`A'qRGE(}zWEq _~3#)';#I~C"cQ8|Q iT5t] '`010%p1 iBt*Rt 2;nB)4_T+~Ѭ.:\M endstream endobj 2026 0 obj << /Length 213 /Filter /FlateDecode >> stream x}O @`qM>!zI 0XɧSW؈p w3s3Y:'sÄ1P{~s8Ӵ$4'tcot=w {* (D`D:y#jAԠBQSQ]9h@9׆mƠ3/"-PIoәn ժ?|R3{6nR}Zn endstream endobj 2027 0 obj << /Length 245 /Filter /FlateDecode >> stream xm1N@ Ema|HBbE"Tj`&GkH 4أnv+4rVISJ{!Orݢ~9^ꖋknR*.PI^((`)3Sژ1+-:%8p'?, \%ᔀ^ÊH"4)MP9%7Hi/! GdL!n&{| JMc_u|_!r endstream endobj 2031 0 obj << /Length 189 /Filter /FlateDecode >> stream xڝ1 @EL70s @BBZZ( 9Z#XZ:IVt« 3Or#xjBN%7nt8SjImYǤ+]'RzΚT;l@TJ @ hxjze/ ]a;AdD/ak+?iTRS" }G@ endstream endobj 2032 0 obj << /Length 188 /Filter /FlateDecode >> stream xڝ1 @EL/ :ͮA"EVbE$Nxg1q߄l">h.!Ǧ^OXRcR 7'e|ޏՌ5ٔs@ th~//iKxO`LГtIVx?>(=Cuڕ/@RriniMoEBs endstream endobj 2033 0 obj << /Length 130 /Filter /FlateDecode >> stream x-ɱ 0 g 2'0-k3:9 TGAEfڢ|7lXU:x@='e; m;P=fpq}kw+*\ǣҟ;ZFy2ddL*R!sBY ,P# endstream endobj 2034 0 obj << /Length 131 /Filter /FlateDecode >> stream x-1 @E?^ xЙmV"RP:ٙ&Nwo\%红V\xA=y1:nwՇ Y/ t4M22DT&2+<*B# endstream endobj 2035 0 obj << /Length 94 /Filter /FlateDecode >> stream x3230W0PaCsKCB.K &r9yr+Xr{O_T.}gC.}hCX.Oz 0X [\w endstream endobj 2036 0 obj << /Length 101 /Filter /FlateDecode >> stream x3230W0PaCsc3CB.K 'r9yr+Xr{=}JJS ]  b<]d7`= 1S'W fp" endstream endobj 2037 0 obj << /Length 94 /Filter /FlateDecode >> stream xM=@PEx$^!R { T߱4J2:*54`ƴ"f@BJJ7"i endstream endobj 2038 0 obj << /Length 172 /Filter /FlateDecode >> stream x3134V0P0bSKCB.# I$r9yr+q{E=}JJS ]*c<]0A?  @CA2@5@D!dPICd \\\^ endstream endobj 2039 0 obj << /Length 175 /Filter /FlateDecode >> stream x331Q0P0bScSKCB.S1s<L =\ %E\N @QhX.O g``~?`g N}`o`F¢0?Q\\\ endstream endobj 2040 0 obj << /Length 154 /Filter /FlateDecode >> stream x3134V0P0bSKCB.# I$r9yr+q{E=}JJS ]*c<]0AI~ i"?P3@JR|Z0 @&\=) endstream endobj 2041 0 obj << /Length 330 /Filter /FlateDecode >> stream xe1K0 WbV hUw'AAAStp7?S>C>BG{I<J@MTY2Wn檜G>yv36sB<[B7^* kΛ[ojW^ar*Gɿ*ohȡYP~h)?_o``@t6J[LmS/t ]#zIm&+S %-% -3_P}Ҙw4&!YkC1R۠u㛥Ft(X@;x1lY1NN|1`'1:?%r endstream endobj 2042 0 obj << /Length 235 /Filter /FlateDecode >> stream xmj1 ^=;Od-$AhO=Xބͣ{N"Q6>fB&?N'izmf4Z||DJƠz.rM/T%V~rEP@X8 \IU{3bY1Ez$'i=Sː†LBp6Pu 8:R [49޲&&Z'XΝ_%m endstream endobj 2043 0 obj << /Length 209 /Filter /FlateDecode >> stream xڕ00#pO`Amd3ALd08Fgh< @ڴ_e4f, kӄqH2@5(xEB3 i3 5C8ZA/:L^pXpkFbIF2qUNCE>_c+vdn&~VP endstream endobj 2044 0 obj << /Length 260 /Filter /FlateDecode >> stream xڭѱJ@? LaZ 4ܪ[-'BBRP̛*y+uvg!B#n;MG4Zly\Ѣ瞚-Sӟ-5#%_v^QdRPDZTRR OԵ@*(AWE],RIR57P&?2oƐ(~#FLg5=dF#zvL;mf&,mXJ[a # }R:%e-vvS=U:霾es endstream endobj 2045 0 obj << /Length 181 /Filter /FlateDecode >> stream xuб 0  /0 D4?/iLsqINƪ&v)9 O44FQ5o3j ioKk2 DdFLƤ1(C8^QDɰ|p1۽."byҀ)gk׿R?U~ endstream endobj 2046 0 obj << /Length 254 /Filter /FlateDecode >> stream xڭѱJ@?l&yM"&`p` A+ :--7`kMg+ & XKf]{t\)pp{ =SuV=UvT]j__Z]>5(6S`-̗oնd IS03aLlB".!1Ox&pcJ&HۅrI)ܔ_,v0{ltT颧 endstream endobj 2047 0 obj << /Length 106 /Filter /FlateDecode >> stream x3ԳT0P0aKSsCB.#3 I$r9yr+q{E=}JJS ]  b<]acW3v\ endstream endobj 2048 0 obj << /Length 165 /Filter /FlateDecode >> stream x3133W0P0V5R0T05WH1*26 (ZBds<M=\ %E\N \. ц \. ?@"000=o`#?0o  0X0`ao`27Áq \\\` endstream endobj 2049 0 obj << /Length 140 /Filter /FlateDecode >> stream x35ԳT0P0bKSsCB.S I$r9yr+r{E=}JJS ]  b<]d3 eR/i& 0 d`L?`@!\=Afl endstream endobj 2050 0 obj << /Length 244 /Filter /FlateDecode >> stream xu?kP{<0p '% ur(vtـ]G|X#y=8. [~< 8:İ˵W|Ք.1wQ@jH>yo瘣1 ý 8hFx]*18yTB,a PM 2< fep\$I5+zG4VY5D NZ@fW'coQ! endstream endobj 2051 0 obj << /Length 243 /Filter /FlateDecode >> stream xUпJ@/.0fMN?Sge!VjihkR\AKT֩$EuwM1f``w%=.>jRWRkRnKO/VSYZR7T@fm큼0 {düۘ=4]L3Ȧa@bli@T|`MLjb4L1dtFW$G *.|ؙtI6Dc endstream endobj 2052 0 obj << /Length 239 /Filter /FlateDecode >> stream xڭ08#^@D'D::htGxWm~_LyxJsNgo(I5M7?/&~I#K CԼ*x1F%)dB 񑊅A8EjGU(Nk4, ~j}> stream x3535T0P0bS#SsCB.K I$r9yr+Xr{E=}JJS ]ry( , LS? 0adT Y;PCuP7 .ĵ'W K endstream endobj 2054 0 obj << /Length 221 /Filter /FlateDecode >> stream xڕѽ 0𖂁#x/i*U ~I(}JK "&HrtF*8 q0Y Ȁf4  ״ 2o@.08BDu uf,HW lf(ze~ަ_Q@6+L6elZv,XKP~EԺe֩N=v< endstream endobj 2055 0 obj << /Length 256 /Filter /FlateDecode >> stream xUϱN0 )K~h{=B @!Z̏F%Psw|J8êt0r^jE>U KWk=?ܻbuyJz_uEk?ƌ!fl#>3Z;@'7x &&ȖNm9R0!G/aEFD+E$ьMX^>a-M=:upǴ-i}GA^{sywָ+=# endstream endobj 2056 0 obj << /Length 150 /Filter /FlateDecode >> stream x3Գ4W0P0bSsJ1*2" Fr.'~1PKW4K)YKE!P E?<@0g`A bP>T*L`)`J+F Hʃr Wr endstream endobj 2057 0 obj << /Length 240 /Filter /FlateDecode >> stream xm1j0g1> stream xu1K0W v8b vtr@?')ΝCMHH^K^Y/PX.8\> stream xm; @ . Vf.1L!he!Vji(X({8Qښ}i<"Ńf{Qj{T3Qes:.{TŘ4 5E&6%/_x/PAP02g0yp&dBw:+0}ATyM6Ӣ5l.5iK|T endstream endobj 2060 0 obj << /Length 198 /Filter /FlateDecode >> stream x3134V0P0R5T01V0PH1*21PASKLr.'~PKW4K)YKE!P ETD0S$00|`A; 00* ?8Q"I&PMb`߁q ̍:]'W ckA endstream endobj 2061 0 obj << /Length 182 /Filter /FlateDecode >> stream xڍA `'?(   AZDjX.̣y҅Tcu 7f: 5P L % MBb%_/#jƒ&Ύ҄Z{Ue5TƩ-ՇW6j@-OӉ;*`{^[bTd7 wSZ= endstream endobj 2062 0 obj << /Length 198 /Filter /FlateDecode >> stream x3134V0P0V5T01Q0PH1*21PASKLr.'~PKW4K)YKE!P ETz !HԱ` |P=iu D)ph<krF=A?0`> stream x]1 @\B/ 8M(+Tr!bI q23;9nvdC)lGUgwIBf6$32d@fr@&m)2ϩ\^sϵ2HQRQO5QJrh MTrL@V@ endstream endobj 2064 0 obj << /Length 141 /Filter /FlateDecode >> stream x3236W0P0bcSKCB.# I$r9yr+Yp{E=}JJS ]*c<]70| C`003a`\=&[ endstream endobj 2065 0 obj << /Length 237 /Filter /FlateDecode >> stream xڍJ1ƿ00 v^@9Å+T[}> stream x3134V0P0bS CB.C I$r9yr+r{E=}JJS. @-\. ?&iNa`D~700n?D䇁$7 \\\y endstream endobj 2067 0 obj << /Length 122 /Filter /FlateDecode >> stream x3230W0P0aCS3CB.C I$r9yr+Zp{E=}JJS ]  b<]0@A@8~? q0\=(CE` endstream endobj 2068 0 obj << /Length 196 /Filter /FlateDecode >> stream xڵ1 @Еir3'p.#BBRPQr0E:? d37u.{ʧHrCqJzƁGz$15x2`ts [R?L3؂rkm;x3HKv@%.oԐ nn**ɍ@ÔDr endstream endobj 2069 0 obj << /Length 108 /Filter /FlateDecode >> stream x3230W0P0aCS CB.C I$r9yr+Zp{E=}JJS ]  b<]?0! ̃`qzrrƂQ. endstream endobj 2070 0 obj << /Length 177 /Filter /FlateDecode >> stream x33R0Pa3scsCB.3 I$r9yr+p{E=}JJS ]  b<]?`@=:773n? Da`N``` O7Nszrr#߈ endstream endobj 2071 0 obj << /Length 147 /Filter /FlateDecode >> stream x3134V0P0bcsCB.C I$r9yr+r{E=}JJS. @-\. ?00`D~70n?D䇁$0I.WO@.e% endstream endobj 2072 0 obj << /Length 188 /Filter /FlateDecode >> stream xڍ1@E #0e6 &naRK v9GTd)HN^f̦ǚ95(EqߜR{cRkI ? ldM*H&g8^WSQdHVR!J*- i~ nN/ookg$AH> wlzZIK endstream endobj 2073 0 obj << /Length 196 /Filter /FlateDecode >> stream xڝα @ HByuj;:9::(>Zp"]qQ |CB?2ܓ1G!#I:Ramd$V$fO"tٓH$R^K6ʯ\UW0/%>T5*4hy~> stream x3635R0PacCcsCB.# I$r9yr+Yp{E=}JJS ]  b<]3P?n3 ~o0ah`?PszrrjF endstream endobj 2075 0 obj << /Length 195 /Filter /FlateDecode >> stream x=αJ@Xf x{`TSwZ * W6`"8%Gf|q~K.4pR^j<> stream x363T0P0T5T0P05TH1*22 (Ads≮=\ %E\N \. ц \.   W  @ @,?(fQ 0pC sC3=;?f.WO@.uH endstream endobj 2077 0 obj << /Length 153 /Filter /FlateDecode >> stream x3134V0P0R5T01Q06WH1*21 ([@ds<L =\ %E\N @QhX.O `J`pB`왏I@.WO@.1c endstream endobj 2078 0 obj << /Length 183 /Filter /FlateDecode >> stream xU̱ P#k[WJ' rjj Ɔh>`Phj @ B\Q#HEldȗ$"Sg3:.{|LVkRj_ ..X ,g0i) <p&A=j|c(vk]b=(ԿOI |F? endstream endobj 2079 0 obj << /Length 233 /Filter /FlateDecode >> stream xU=KPs Xxv(zb`A' Q|A7|~Lx`7UN?8g!Aj"z$r~nhdHڙdrO/$GcHN* WUP6Aߴ45q " bx%tq_cGŲh;L t5<fOk2|+ZlECd(IBY_ endstream endobj 2080 0 obj << /Length 210 /Filter /FlateDecode >> stream xMν @ )(> stream xUj@Yi nZ$sSEGQ|x I;=F(N8^D!qiIs ǔB3I-1QYAg//74gZv* 0ÿ+]SCE@QsϰF,IqSn/'gCb^mmjg`1'>ڟK endstream endobj 2082 0 obj << /Length 183 /Filter /FlateDecode >> stream x%1 @@$|'0+AA),DQI:IUuO)Fh~!;:c̐ېዬQ֑)HpIH]RY#H[m(l2Oe-?uC endstream endobj 2083 0 obj << /Length 188 /Filter /FlateDecode >> stream xڵ1 @EH!L#d.ͺB` A+ RK EBbGRRl6Pt+ǬƬ5$Ii;Xf$#aI,Dv$f,I(K~ |[jWopG!SE /zO6x+ӸY~uд` endstream endobj 2084 0 obj << /Length 121 /Filter /FlateDecode >> stream x3135R0P0bc3SSCB.# I$r9yr+Yp{E=}JJS ]  b<]0001; aX*6T?0'W N endstream endobj 2085 0 obj << /Length 228 /Filter /FlateDecode >> stream xmαJ@o"0M^ป'pWSZY `eh>J+5E~;Yct_^iC-/+9u'Zst }{} ,, %s'l"aAZқMY'W Tc| endstream endobj 2086 0 obj << /Length 235 /Filter /FlateDecode >> stream xu1N0ЉRX`3',ZiY$R AE GQr[0"OʌǓ/^ҟ+Vɾݭ%+yxb>F:iy-29Q EPE6fLV&b&e6fՎY (y/ifU _ cBԨM>y2_ |Ǜjh endstream endobj 2087 0 obj << /Length 188 /Filter /FlateDecode >> stream xڕν @ + At-('𮶵kotrP?Q_ I+F!=ړ,o)$G$'KROt8oH&{$S^zVSBĢ iAf1h.p;`Z \2oߛy544` endstream endobj 2088 0 obj << /Length 226 /Filter /FlateDecode >> stream xڕϿjAna s=b!j WJ!`R nGG8̜EH:_1;dySpnyΟ9)_6[d?9oR&[}";YL9#;e銊Һ„pQ*+j .+xs7xĕ\ }rR /:tKuNTc'ې'jiT2Dׂ+X endstream endobj 2089 0 obj << /Length 243 /Filter /FlateDecode >> stream xmJ@O"p}dXW0 j)h()SDm>{uuVZjG+9}Mjag"VNbkx|JV+-*@ Ps&[ D>#E@rI~2> stream xڕα @ HB}Ѽ]`A'u(GQ|TZ?$w#3ihdȎhC!s8cТZp*Yz?WS2f5wHPQY 4a:B@ 8 1n -SQR-8 d_Ѯ+J_> stream xMJ@Eo[8м$AB`B]W҅E ;#Ǜ*y{wquLZZj}%OR7KmN~&wlֺ₲<>H\i%Jo*-o])L O[ `;d1a3X`LpM6{{xSHp|tO01l6 i4,e3zwgRS@v伕+c endstream endobj 2092 0 obj << /Length 237 /Filter /FlateDecode >> stream xu1N0бRD@\lBTE"T AKr!e3 gi_'aE5tB 2(_pӢ&1^_v7T]M=[b.'0S2*(ٌ`&p B!t 灼__Rc%ɞ 6{6C!Ic)A?XZ1IN+OVqY- m9 endstream endobj 2096 0 obj << /Length 134 /Filter /FlateDecode >> stream x323P0PP01P02P0TH1*24 E,!2ɹ\N\ \@a.}O_T.}gB4PO,C}?K` @cs3; CPՓ+ a'V endstream endobj 2097 0 obj << /Length 136 /Filter /FlateDecode >> stream x323P0PP5T02P04PH1*24(YBs< =\ %E\N @QhX.O9   fv6> $'W  ' endstream endobj 2098 0 obj << /Length 95 /Filter /FlateDecode >> stream x323P0PaCKCCB. \.'O.p KLz*r;8+r(D*ry(177? 'W  endstream endobj 2099 0 obj << /Length 105 /Filter /FlateDecode >> stream x323P0PaCKcCB.CrAɹ\N\ \@Q.}O_T.}gC.}hCX.O9 Ėc `3Փ+ ( endstream endobj 2100 0 obj << /Length 375 /Filter /FlateDecode >> stream xڍJ@ I];/IYHZ,]APB ɣ q4әt'Xk~=wA1թC=˸=fE9zv<> stream xuбj0d=A-pHRB;u(@19G#d`d |' 󟖋;}O5\RQ`ȻO}c~[zIc%a,D!Q$mbG2bWh*^jL/.i AjS]3}`qd;<z<ĠuH> stream xڥѽ 0ةP+AAAѹ}> stream xڽJ@ƿ%``  h`xp` A+@--|tv)-­WXZdow*;9-8㒏 >+1*R̸*gZ֔Sz-eJ~{}ty{9+Sb 頁Ș2ԠFJ -_5J5f҂Fvh4P"VeF T߄iӹ{ =#0s@7IMlVMts~!|.G9#٘0\f {qo@W5/XI endstream endobj 2104 0 obj << /Length 220 /Filter /FlateDecode >> stream xڽ=0$N`!!U'D::htq@ZmIjlB-$CϐOj^gHs`[1e ,_z?Kse0C (eml dE|QbM*mhVK;-Fi,IUAmluΧl.CNZ=xں%giz@6 7 endstream endobj 2105 0 obj << /Length 202 /Filter /FlateDecode >> stream xڅѽ 0+n/ t N&X؏+blkqRv= (+%66l8T(ԏPL]jFyb8QbL51|=3;*X(mu }6f\-~ͽ xTk݋6o]ÖoW1\9_?D߭k endstream endobj 2106 0 obj << /Length 180 /Filter /FlateDecode >> stream x3333V0P0b3 PH1*25\Dr.'~)PKW4K)YKE!P E?|@``PL1C(F*  %CA(6ŃF1dP(UPP9J>TxHJ(`\=|3 endstream endobj 2107 0 obj << /Length 296 /Filter /FlateDecode >> stream xŒj0OxܢGн@kg!M ԡm-@^[^[WI qUutqE+ z+̟00=}c~ =T`!gA@jT, 8_=eCAMF^ |:I *@=N` ڝ 垻ḱ69&>0s!of &jɤbu gϫC0IF)Zm* endstream endobj 2108 0 obj << /Length 130 /Filter /FlateDecode >> stream x373T0P0b3K3 CB.31s<L=\ %E\N \. ц \. ? Ph707000c~4ȫ_4,q.WO@. endstream endobj 2109 0 obj << /Length 112 /Filter /FlateDecode >> stream x3634W0P0a#3 CB.# I$r9yr+Yp{E=}JJS ]  b<]A~y) \=i endstream endobj 2110 0 obj << /Length 295 /Filter /FlateDecode >> stream xeұJ@ )^@>IDb<VbB29,|2IHyEqwv0Wlfl~_ڑ0)>a:q{q kY Ԥ/Mj[:cЌvt֡XQ(jE/:A f SZ xv;]EoclZP7cC+O+h8"ΰk=8Eѳ>̡(Ɲ0M4yt^i$&"귁g^* endstream endobj 2111 0 obj << /Length 152 /Filter /FlateDecode >> stream x33г4R0P0bSs3 CB.S HrW05r{*r;8+r(D*ry(*P 2C@,dBPBՃ)A0@\=R endstream endobj 2112 0 obj << /Length 258 /Filter /FlateDecode >> stream x}J1 ] {-(tdibVp> stream x]ѱJ@ Lᾁ'p<8O0)V"*+ϑ:Ygw{tx-(9bA1=3?k*hmuAoh]MN-V+rn`f \uǦxY> `=jx烷li'^ b8vUx谈 endstream endobj 2114 0 obj << /Length 184 /Filter /FlateDecode >> stream x3336Q0Pa3C3 CB.S3 I$r9yr+q{E=}JJS ]  b<]dEL0 JP 0QR(ڡfZ D}L1Nqzrr) endstream endobj 2115 0 obj << /Length 252 /Filter /FlateDecode >> stream xڥҽj0p [hd`e3$)C 2@!!G3U?& w0 ,N=j7>FTҿUx4F=E_%\ᵀ=/ɸh endstream endobj 2116 0 obj << /Length 289 /Filter /FlateDecode >> stream xe;N@rai=`;qѰR.@T@I.J|7a҈$Ci>˳؝I}^M iI/y78K6'of֘)nb-}~|bvwE)XQd9!a"[d72EW:,wX=0;rؙnW-WzUR,k9M<Iz:HxDLՐc|c=1;2؉^]Aĺ7_lo'kH;tۀ_"=\lhsoW endstream endobj 2117 0 obj << /Length 160 /Filter /FlateDecode >> stream x3731R0P0b3s3 CB.31s<̌=\ %E\N \. ц \. A70``a~@ m :y 4!B3  4'W +q endstream endobj 2118 0 obj << /Length 210 /Filter /FlateDecode >> stream xҽ 0+!vuФZj? N⤎nBh>PWj#:,{)=F(c[eZJ *8~91d plp/ër endstream endobj 2119 0 obj << /Length 280 /Filter /FlateDecode >> stream xuAK0W +<4œ`Z>J](6 SR+4)U%]\KwWfp֠zyTUsG_fk*Q$͜sP/r2 ~rFX cu jY1&ANdZ0#0@c+/=lDmGg&FK? vGcp8 h¬Xemۤ6P!!cx=K-{ endstream endobj 2120 0 obj << /Length 333 /Filter /FlateDecode >> stream xڍAK0 ^{6LSaNAODGaRAaԉ.x ?dJg9*9g9Цӯ'9+ezӉL/h[%=tvuLx0ȶ `ka5@!F֠%~ CnɱCȅsZnpIm1u2c!/Wܣ[P `Q P{&{6Gq.L!qvNCQ&yи7> stream xuϱJAba yh+RPK E;1 tƽpS|?;?xžjs3TC=-r+SrgkkrKyrM͒a{ծlB-`a:`u)xuwGW2&e˯ɦnh huaǨk} [ bԪob"EzONoɌla endstream endobj 2122 0 obj << /Length 213 /Filter /FlateDecode >> stream xѱ 0; 4X-P vtr'uTt7)7&/“ h4"rMӘzd endstream endobj 2123 0 obj << /Length 203 /Filter /FlateDecode >> stream xڝ 0OKдv vtrAPGAEA0G#8:ANȹ-Lp;"dJ Z_V[UglJ#IWc>NҽIs-0pu@܀_x vZհu/{#ҡ^EA^UzN4 E A2;Wa V4'VhLr endstream endobj 2124 0 obj << /Length 212 /Filter /FlateDecode >> stream xڽϱ0$7 x/$N$ &:9'utf,ƣ Fp $K8q b~bNe/DF4AFGi[?2%72byg6Nh:]hBQ֩L)϶?$nId[XmFiǞzՊuA63` ^j endstream endobj 2125 0 obj << /Length 210 /Filter /FlateDecode >> stream xu1j0g<7 41'z(S$ MHXGQ|JW\(T 7uN3uki1}.Gq%Cf&u#U])Yϧz\R׹fi WOp_PI! I@*#f%#~,K{ǏT#,ΰq`(nYsLޖF^V2 endstream endobj 2126 0 obj << /Length 156 /Filter /FlateDecode >> stream x33T0P0bc3sCB.cc I$r9yr+s{E=}JJS ]  b<]000```!$T ?h A30c T" h2FG&\=ak# endstream endobj 2127 0 obj << /Length 275 /Filter /FlateDecode >> stream xڝN0?+C$/~ @pKV*E"L02`ȣD`7я$7d*:}$ X endstream endobj 2128 0 obj << /Length 167 /Filter /FlateDecode >> stream xα @ ;:'zx: 7:9: *:{G;s]!3pck8YǸh PsNA^/r9E l BuL[VeTɎdÞ@`_wV| 䈚 oafaosK endstream endobj 2129 0 obj << /Length 125 /Filter /FlateDecode >> stream x323P0P0b#S3sCB.#C I$r9yr+r{E=}JJS. @-\. ? :  .WO@.P endstream endobj 2130 0 obj << /Length 220 /Filter /FlateDecode >> stream x1@`CW ,I0Q ne!V*Z'7J+)Shfe=1fOA2∇n'MxӞ#슓U|<)dg9P1csK^4Ї g Z7-Vj]p azկTP)*ܨF7́ ,a 0@ A/vP`iCiyA_ endstream endobj 2131 0 obj << /Length 110 /Filter /FlateDecode >> stream x323P0P0b#S3KCB.#C I$r9yr+r{E=}JJS. @-\. ? C 1cqzrrp^ endstream endobj 2132 0 obj << /Length 203 /Filter /FlateDecode >> stream x=@H\@ȜM B0X({+ba8垫|>2Pԏ~?Ѥ$|@jRRod5Ԍ;*gX@l$u8lSyEȞn!X#xiTCƩFHjODO'0vBJ#n $"&ݏ endstream endobj 2133 0 obj << /Length 159 /Filter /FlateDecode >> stream x3534W0P0bSCCB. HrW01r{*r;8+r(D*ry(0a@R` `$@z ɀ a/ m?C&\=?qjS endstream endobj 2134 0 obj << /Length 209 /Filter /FlateDecode >> stream xڝ= @GR2MtbSZYZZ(ډr2EH|((v̛ݝGa_ endstream endobj 2135 0 obj << /Length 218 /Filter /FlateDecode >> stream xڭн0 p[*y#4"t7p  }4бCHpH'n[~8{`zz9> stream x36׳4R0P0a3CB.c HrW06r{*r;8+r(D*ry(0`?l(g?6g u@lC{ pP endstream endobj 2137 0 obj << /Length 213 /Filter /FlateDecode >> stream xMͱN@б\DTd""R.HE) h!kfg:[\ꗺXS)Ks"Z;׌oY2=7Ro0ͬ&a8YZi4 %:1X[z83L̺E[y!8}?+O2dWtm8 \\ղuY endstream endobj 2138 0 obj << /Length 160 /Filter /FlateDecode >> stream x36׳4R0P0R5T06V03TH1*26PA3#Lr.'~PKW4K)YKE!P Ea9$luPفX$N#Ccagc{  00?r Jm endstream endobj 2139 0 obj << /Length 162 /Filter /FlateDecode >> stream x1 @ᷤL fqC@Vb--+'Gˑ<@Ⱥ!X l3pjZ>DŽm:L#c^[z?.6 6KNJV- -reByDz 7U}`(D,uxI0nҷWR hhKob endstream endobj 2140 0 obj << /Length 202 /Filter /FlateDecode >> stream x]= @Y6sݬ+0Z *ZGQr!n5|ś7ȈBR[^0$)?G19]/bLւ :c:k{-Ŭ`m88u t&p2 lB̘Ϙ> stream xeɱJ@; $p M!澀dF 1` A+ Bv362e]X'qι>giF'5Tkè;.:TQ݆UwG_oTZSE%yB7zȷ CD`Al`^Ѓ\F&i!Qڤ5#+]VڂQS"w)͊S28`?ah֌+.1%t}z= endstream endobj 2142 0 obj << /Length 236 /Filter /FlateDecode >> stream xMJ0?0> stream xeпJ@o \`^By]  @-G̣R^w]9 Opj8>xPS5ZOLIppu%?^^qDzŷ;JW\ׅˡ~ lr&Vg{'´N2;s8Gvn=ЪQob]pл ~^8:g007~ʞJT Ͼ4sM^!yJ[X' endstream endobj 2144 0 obj << /Length 207 /Filter /FlateDecode >> stream xڽ P FҡмVn?`A'qRGE7f}>BŚނ*3$|9VuQۀ}+5͞1%kTڤ|18Ux*%V738 \A&rOP deyܿ>X ?c\%#'q(IfNĴ) endstream endobj 2145 0 obj << /Length 131 /Filter /FlateDecode >> stream x337U0PbC33CB.c# I$r9yr+q{E=}JJS ]  b<] >00013 A9 CaՓ+ t^@ endstream endobj 2146 0 obj << /Length 259 /Filter /FlateDecode >> stream x]J@Of!"." E0pA.Z v |˝gH0??pNNmnҮwYUϹ勧7wk"nssa q[{_AꭅBaD4%;>#p{%*édlW]HO˷df 3ÂױtK҇FoMfl=o,"E"pLΉ~WhFF*4& !3DWZnvj endstream endobj 2147 0 obj << /Length 257 /Filter /FlateDecode >> stream xmJ0'y h[ 'i((ysƙ$;dfjj5u=5mMrPٿf~jg6wW`G*`Z@y`5@N08F  xP f͡HmVJ[\8 )qYTN KJ8L3#ęgDUk-2gB8&%1Dw>vq endstream endobj 2148 0 obj << /Length 206 /Filter /FlateDecode >> stream xڥj@@CkB  A GAẸMb/hffӱZ'd?$u{<l(潽x3\h*fTK> stream xڭбJ@?4y1[8O0Z *ښ<>=BKD+f+.yG\4P -Kn;r׼ΥLGrSm `# JT&e 0m?HMF3CP J@#J2 _.N^v2%5+w:gY9Cb)@;@f,BMB~2YGW e endstream endobj 2150 0 obj << /Length 257 /Filter /FlateDecode >> stream xuпJ@o"0y!Dr1SZ) ɣQ[X2N[3.脋%?NEav \d^j??^(]_sNs0y("=I 5poIu~ѽv ڧ5F r q/oAz Fx`cο=!)a$ܠkkR:5.̈% endstream endobj 2151 0 obj << /Length 219 /Filter /FlateDecode >> stream xڥJ@ Ci X/b ͡PB,M$G#xP .C3;&z8YQ> stream xmN0Fȃ%/~/IQ:F*E"02@bH͏GȘ!s[uY:9˅/|.|U_ݔOZ~̺1/ 2l~||}&ǹ/L'bFzNEؠtX !v$tS2WSK8Zdef-UwN: VBDXMvU=+OD6($8ㇸb+N==BZ!r5B<$gVZ}F=sӘ{~ endstream endobj 2153 0 obj << /Length 262 /Filter /FlateDecode >> stream xuJ@O};/I$7 \` A+ RKAE;Gˣ#L2&" _ɚ3.5%LErHk.2J__(\rNorn2 h!P#a]a:x-}bh~mh!?0B~! ?#;CsŨ^JԴIPG^`M !A#`xBo~^}gt tچpYzLG%NC:k>y endstream endobj 2157 0 obj << /Length 312 /Filter /FlateDecode >> stream xڽ=N04Mn@@H" * D hq)SXflЖD>=ｋ⬮1s<-:Ǘޡ*y1Ǻ;oi!{솗!kom -*A0"aD0S^hF!M$eD3IȕJ%ia:K)wr94~τ]Vz^P䞺c:2Ԛ;'RW\;vYy0tY0\qW|\[.9<4cȇyDI3r`s9PiP9v [ET endstream endobj 2158 0 obj << /Length 140 /Filter /FlateDecode >> stream x37ԳT0P0b3s CB.3C I$r9yr+r{E=}JJS. @-\. `h:> p??hE Pu>pzrr endstream endobj 2159 0 obj << /Length 271 /Filter /FlateDecode >> stream x=N@PLQm|&RheaB-MD@I@&ُ,uysoXB^P5n CGp#8jD<="jZO8blg~)zd!+`H70 5#]qkXZqB @U?6X vF>DmAD:l"7w v ;G4y(QHݔMyyW/,D endstream endobj 2160 0 obj << /Length 227 /Filter /FlateDecode >> stream xѱ@ .~Q'D::htOSGyG[I܄k-p8R@#DM&t Q@vጳ͖ҭIWtOhf9hڅ1M([xl%֢ł4qs艐Uf o 6AQ>_:nC>oEsc 1gkXKn8\ endstream endobj 2164 0 obj << /Length 96 /Filter /FlateDecode >> stream x3631R0P0F :Ő  Yɹ\N\@U()*Mw pV0wQ6T0tQg?P`Փ+ "g endstream endobj 2165 0 obj << /Length 190 /Filter /FlateDecode >> stream xڳ436W0P0bK#K CB. 3 I$r9yr+Xq{E=}JJS ]  b<]" ` )dQe21 8pfAfiA&iΔg22Ge^L0 @ Փ+  endstream endobj 2166 0 obj << /Length 293 /Filter /FlateDecode >> stream xӻJ@8Mބ VK E[7e $2E8gfA0%s' $iH&t#[eVw8134?LJ[L'd V$/%K DsХ0Gbڷ鲁fV [1>8Q.݄y4T1b<[. | ء ; \t~ۜ9AƧÇr:sLnʝrn7Іbӄ/ǵi H endstream endobj 2167 0 obj << /Length 267 /Filter /FlateDecode >> stream xڝJ1'lq0޼fpVb]hy}-86L /;q5%QwFO-kHfr;r +ZoyaC 2i寙5z>%k<&r,`vd+q3ߒ1^+ \oxE<@G*q/|Aoٸ=,8U(`ش fA-pڟڤPj"{mI倷YR endstream endobj 2168 0 obj << /Length 219 /Filter /FlateDecode >> stream x37ѳT0P0bsCCCB.33JrW03 s{*r;8+r(D*ry(00`P"0C=~d3@@C P?P 8xq83qe0w`0H+p32> f qՓ+ P endstream endobj 2169 0 obj << /Length 142 /Filter /FlateDecode >> stream x3631R0P0bcCKSCB.#1s<L=\ %E\N \. ц \.  30oAr 5 T @;af f!`` ȘՓ+ > stream x3631R0P0bc#CCB.#3JrW02 s{*r;8+r(D*ry(070o`G1 d endstream endobj 2171 0 obj << /Length 207 /Filter /FlateDecode >> stream xѡ0[*#pO@@ %0&H@! $h%#L"uDKzz٢"\1CtAݓSi֫u{СuB U|0ۀؖB%/Q@Px_Qv؁ʲ#rO ^7\gpx'A~^ɼP/nC|U endstream endobj 2172 0 obj << /Length 249 /Filter /FlateDecode >> stream xڭN@ }K!~5*1#ܣQ3T9l Iɾ5TUEš^+:pP3/F *-=UT>cKxii$@v#W@!'=r48 E\)GC B1:6b:wZK??"Xi=1wfbpY4?]e[t~x# endstream endobj 2173 0 obj << /Length 185 /Filter /FlateDecode >> stream x? P ,dМVt* ίGQzN:xȗ@ iDrj* CDJbCbqNjILjn߮#r)o̙-S/XSeFԕ+^+k۪d%A3vX}X~ö"7iӊ^Ds. endstream endobj 2174 0 obj << /Length 281 /Filter /FlateDecode >> stream xu1N0G\o$"-D $(PR[mr⛐#Lvq v '33n"O'5sj<=x/5j֝){S^˵)x|1jSn衦t8z[d yDbDΰt=ZbM΢yqPje^5X*>YY:#BIj!MlG-ƨH]$?r>Pc6A٠~I"vfD7(0l@/]3wׄ endstream endobj 2175 0 obj << /Length 268 /Filter /FlateDecode >> stream x}1K1Wn(ž/TZI* nw֯q@2P.R0:K2ONFQ]9IKTc$kmlv2JP;L5o<-Dw0ġ ;#39~cF<cpGh> stream x}?N0/`Cs' BmHJȀCH@l\+7 9 U6c#%glʮlfeח2foofךѮQSwՔk[roɴ{ q 4M@s`d<܃oh^53¼x@=tqeF3`0b)(jA>(Np5g PK>' o4s?u'4v)Jk(VλEӗM8"<¥1fdc,,@Y" -ji_[iOHw.zHhA~ ?4sxkAܴX endstream endobj 2177 0 obj << /Length 328 /Filter /FlateDecode >> stream xڍJ@'X&o h.r98'BBBQ2y{<•[?>;Yk>>+bWuóݶҖl^ߟl>ʖ;xqo4rLdX鉜dZ=5 D-FBLv.d8QOxME٠ H͐OsQ ZD3 E!Isig,ޡKo9\˜`5FUPS6#) F>hP2Ć&7t>Ǹ!|Hk`ڋ/U=š endstream endobj 2178 0 obj << /Length 312 /Filter /FlateDecode >> stream xһJ@&o\"uSne!Vj)(X9yo)SgΉ|%s~LMjf;7Ejv"7zS> 9ҋJ''f:N&Ҝf&=Ҩ#&P5AOtA`mU~ԳjʖJ.u˗o5Jj Z'@Z}A|*OMH I+kO%3?q2'X74B\G=CN{5""՜Rpf鷔'zJV_~=@}MTz? endstream endobj 2182 0 obj << /Length1 2211 /Length2 9089 /Length3 0 /Length 10268 /Filter /FlateDecode >> stream xڍvTm- J Ct CCwwtt H7Jtkݻf-f޳OsZu-)sXs9E2*rz@n77/'776n ێŠvt"x8M6YS Pry@77[oG@bP(`',# ~_f (,,G8@B*p+}E-@ `E\]]9M8a,W v;?%TMIbh[A`pWSG0` N!Ps#:@KQf;9+g"`Sfgo u@-[0@M^gB::M]L!fP7KiL ;q:Alj20;;0, ?wwkB=~@-~0wz88e7ak<Y@iiX{C,_XN.`+03%o{3;Bp~aP[h1S9M$JK<|an (=?G?ꦐRjVh07,+ `wA[Gmf;3?/v/ ~Eu*`sQEHA-m9H< lpÜ ??ځlnrP LMݱ9pqBaA] Qq׀1jz Ť~AhI]gV$A%L$c{IWokzYL -a ДXzreJҰwfo^Bu'Ή(۬ 'IN$X RHobYym`Pw%AX&Gk)q<#yAE%S;?qG?"!pX wemYVEeFfn3gTxmq[";Olw`"oևn%FV@>3z2 70Vh]soIݳG2C⢖V,S8 MPNIÐ,f"yİG^l!,S}<%5)v/kxVߺ7aZu(Ts8A4<_h,]+J-FTWܯdB4^pp^l{|lEa^TE@x*JcD;T A,Ser/T |Ѷ(6lx4豸噻/3S١ Z-G}[N]tieZo-m9¢9HP;٢:DKզน "sbt>[l+@O.ٕvs?dT\?iձyYAyӎՇX\—%ck[շ9d%Vv:%(}$+ x*^]Q0]B6C1mP,^k>цƣtR.~ Ԝ~V.616K֥/:Pm͐/N2ÁGT0_W`] $ⱥ#C卽˲LQ7,:}3TS2<`'KЫ*,/)v9܆- U/qYCH '8qHd(BVYaosREpxUm56G9nK;oz&ҨF n aKI)ge ~{X|ha#CNlSF-KlޛB审ujm~:ەaIe+d )4$y:YmAC\S08{ŘCv3{Z&2EqZlZ_Y)S~lPBZq?{GSAbw_/}d:,ˤI.6Jzl#g]"dez*] xȳo|h>nQ=N@l&OA.tlȴk1T 6߆p^acRE*A`:D7x98\DtS\qnoò63OF m; f1oIaK,uOz~?shdJ%˔nQ&2Zqk[U85!;ȩ.o5&gXc:"G[37e1&MmAP51ByKB*gc59=m_)6t4kZIV31abaD D៩=UEBSC*_~]>͐*Nf6{r}/ j_NIaͰȠڤi8õ"3E*;9֦!Tj㏅-a^U&G$I#.G]gyXFh©9֐\j{ $}K"w3cLjn>! Na1p~i懇>W3kyHC2Pyg4˶p\wD xB3.v+-/CMOqVwǏY_uvhU{VrSm!09}SdD@MsFxr~ԥ$?  U\-Y!0F'bG?>~70m+TwB90B$YXGPw:D=pO]-zO#]1AG+w5&쀙U.(EF~q\[ :0]Ъ7tm:L Ҙ)lau̕GdrQqS5k1Ԅ\tI5 f89BY(t$HUDPNŶ6[R8폌z]tZcĹ1[,[" 74sm8d;[XK )=8M[lqt:N9ex+(.XMR~mJLf*W$vK-;LR_DlMBZ p;HDwIGWM^~ɱ9d-NJO?~-YKc5ʐCư@CU$Y'oѝoha͸l&\琬mt1M=$Ttm|L K> "}>^+ **=CM 8;@z.Nt(Re; `di9BYc_}݅#"{%AVs)WA4 ,b}QjQm@ QsRSRݧhIcXAfV~ӈ 睳v 8zTHpTF q#ίZr;ۄωe[*g@rNV螘"@7*A BAOC?$?lyYMyee|vn(`M&lTqxj"|'E_|N-MFo2c}ϓB;g;M#9irrcNh@|MňNjl[Ig~K:2]н%m`1j~!fp%y`+:&gC3?jg/ ]2^TJ_בֽ2IiICX='h7(߉1HҚЄ}>@G.QeX1/h =W ,zIoFm( @zh BVTOH*iPc`T' <3w?giU\V!;Y,> #7|OPhx}SݤV{h8{szaՖi>GuHK[1M7#PPz{+4Eg3ѭ|39*|Am~Vu-V,^qguVgEjrp{'jl?Lvbb54@.3 'e5"+`pw=rs]TMG"(5?Q~EwOhek>:TJuBЂ2ȿETtGqV ^:J ´)I]RxAZpnR͌HMO\|93m[=Vt!+lH֨*cK~J2ۈ>gҵj} izYGc9b<:9PtC{>ZroS1ҵ9}ixx6•> (s$ӵ-/=gV&zAo]xhA9KE>p!S5"6B[ZFILl/ͧF2._؄rFBsd*8D2!l }]:d/C˼uі^ N-ݣFP"ͤjy+1ːm rNjرqc//N07?,"rl^p~ ]kUU9H]*4=gszO:zt~p]sE4\; gPm&wC8[9:I {^pv+5̚~ AZ`3Zmw3*]guDqUӛ$@jU7Wg%_y . )G])}w;Z *?c|epbRm'ˤlBӟV8CvK5Ij5{Oi\_/FVB> 9W|yAك:l&Sm]'$L>rŷ^0׹ß˅0hMƋV7vr\t#^N%l~0ҙiѕWgD<HhV~7 {iH<:4[0@3|l (ȩ-lKҍ8A" #?ϡIz`G*39Z ,#[;HHgnH`)%iygP¡k;4qY^cxt0t Mp*[DT(-ܹ\e#"CygCK0^'e]F8%=R-fia2LU멌6M\Y-=Q90כ ]DK  EͶ]Wn z-?J.:9tªKpuV:Or55bM㬍Y16nF DJ@(sSNd$ҍOh5/JNbt ѤsĮW\ZO;u>!zѷ1!9<&o˨SbI}łX_i#`)gYp"+XKLc`?.[Vh 3b *WLfKH#H^EToںTxH9ꇫZŀ믵d>l^ }}=W\kf޷Q;J@Q&}c |yiNs :v,Tۋ<„;eGk#R, o3w\{D҅+,lo:? ??~%f8B*^J!;Wmk<5[ޑ7&I _eF>t{%Dz##ŗ; 7j:P+V2 ǎf?P]n >ǃT98)~4J{-Rؔ'XpA5A4G8xa:5J=SO"xdŌN9pz-畛/]􁋯,tz(c*cucA6/vv}1K6pUƻ&wnQBmxC # ۶eڣif֨! 9^ZaLo^] ot Y I _r86,>]Q_{zE:tCFwzUg[/Qbh|23ШJ%TIMiƆ30zsh=<Qk5S 3)Z<,Y.c:CtrYJAnuʃO5fn 4t'G'5Σ ċ;Ñ1N9 *{K ښ9g1&ܞ<w DGUQ$x] qS Utjӗ]X|smmw[Hciщ}B \+ٝ3 ?J|# Mv+6: -B $پ(.#иlH==G;Y|:2ʧVtE8mydiv09gqoۧ2|ђ;LN4l{PZ')IV,D@и2&iZ,$f5iut"SQ eakHfEۄ;q4;B 7jA/ͫ\%CC8K'!r;Wf\kӉ(J.qDv' endstream endobj 2184 0 obj << /Length1 1936 /Length2 9581 /Length3 0 /Length 10819 /Filter /FlateDecode >> stream xڍxT[6R- )4 33tH4]- (!) ]-9kb}]}~f k#o@!^~ ?__8,@78 ĔHC-( HJB$J ;/1(Ban GϟG-@@\\v dk hY#.H`Dx#vBd8 #@yBh[`E@n@l8r;@zkt`@_ƚpN@W?r%lmk uYCA= h"k/Ck0oa [ ~"FFw|p[7 煃b%L2N 8Χr"wq!PO=bg+ ;wTW 9a~qauM1 0=2 ?[{7w￉^@ G Z##O'sdA!`祈:\RADODյ})C@fO wcoPd?3~a~[!T?O2_ Ђ"'?Mh-kC}#_8ҟb }A.u w9zȻ,o AAa7+arV^[ "[P7_@`fm Dޑ_s*RAsszـmZ!&ˇ QA>sQR )j r>C!DkLp4ΐ<@HȁA6U b #. ;v8$N tdMC y""[#d}KdM`P_R*LYbH hzm8ҭ;OtpnnȣgzmqgaNo:kiZ':JУ#2C7cKz2hvV]|>P|IȐJx)ӵ”y1d0,P PBG ՍYl҇WB}5"U$T>rS%d=n*_*3R.Y.N'rotV ~0^]>4ãlIr}ŬRcr9vT#C'`w-Ԛ^j.4` YlQ^[H4OR ?Y. f!UGfpZuf(U,^y-""8v|OIkaޛ6n|X]OJ#Ń(N,*0 $) xk6b(D>)8*{"pݲ`@,m۱}l)^j.яIlT%3+Ȣ5;[zGop̮$m,v` JbBUZA踑ƁJxbxc!3^G& yho>%:Z}lЊBTn=t>3-; 6JՊih,(4nz$DWXNpRAUkT-GktpÝʯw}C: ָI[4C5PUrlN>sEEXaP.OYZˑU[.S0rX1o&nS^DL>L9I{.T)!|`:oBZhSu&^IY8HT{"_[&Yϰ`<Om4Zjѣ-klo}C:2a%]+nR2({+g)9POYr'"t&&icޖWֵ@-O )STqf  j0d^o1ZNiMGWOfys-,:yrjIR˺l =JdeW`3aҋpuU$BDac(z1p^qDhGX^% s0Uo>$ggcDJϥ6}Rfρ ڳ)y/Z3KPRmӈYGG܌ 3{Ibbu*_lշѹgYf֘ͳSs"1pJF35,/~m͂#p]5jX*ypcdNww (yti|b:VUz\ݧWe$|y}#r'S^fݕZ _}BPY79UBJㅍe;ܸ6~I|7Mg \5OS|Y'(Ս]! J dLICMCr~h@eNjG>u׹~sE}a !?.a(1`?P䇯e6ȪI<qҿKVnz35?WuIG{] >ذ}ͨ*[37ǼQ O"mL2v-FJji%X4DÞ ϲ[*/[p-@1{7e_J]y ;a~QJܬ+|k9QYajE~#U3C1 j=׍lx wx:/DP_ pbƁ_0n3uGm ut6xh~*kɈUY۶{>7 #!@8^ }Vu1:и$rkxe#P;O=yM(I=8d3\|Ə TfzeqRM5W,a0YmC=ţ}|cT,_Sa%,S?w&:2{wW8*.x`QaߴΙR !ԽuO2E%1҅*1)O$`5*uiGn:vQȃB 4{P&]2;2$8?.k٣~|I'j$EK1 Wlыh3lDnrxAaG2Q_z ] %PY\*s`/ Q7LT&a}q=W`~?c2gzut0v'%I8t\0 Zi? Aa"̋iģԚWs|*44նhT^FZO #5U7nkƌ qAn-5DPɚnyr~y4 sY@NKa"CG ;_K+W;]|:&xvyQ#f4}Daء@=vAz}n%f K=V_p;n_Rwz]2̥ +:`$.ɶu !ҔU= rwbm{bkcg)קͪ*g@ƄK dHnL'ㄱuؔ3t}),mwٽƳ״I]*b?!$zS?]4b}Y}$s0]N@# N8\M ANR41Χ|Xe.T9ß[kRV򯽙kX|iK:Y+^ \n/W :/ 1&2%g''dAh.TN7M;©FG[#hWv IU+@tBDMiNH9PeZS|@e0(n[fQ-60<֦+ŕM!m&ͧzOC]wrxGۧkݜ\dulH6AB 5mAg|3cQ@½Sս@'YZBq/˴No$PIy<NvdM~D8^UuI\Hf8WK#DqFi!dk9׃mѺ,:4nuy`g_j&">_K6s0K~l9|B1^qh;_ o-69g 0~*ơɊqAl9Èw%^9A@a|zJʪD!3eۛC0 sc9a#axWZ1'j/Ccs݉Q;9945CyB\g.:7 [߬b, -Ĵ-d]@[IϟifΰKd/տ@8v\T-ϩ<Ϗa0~'٤fc.j)1Y=TXԙlGK Z3JQsx H-GJEh}#bǒOE(p/;FxxMK%?$}chǘT^`õݣn>&y.bF[I^'>Aܾ8}l/0`?F@fނ@^ȣND m=\o-߆Hm.3exJfF T.:O ӗj\7^ o 5r[?[VaB{[c)o20W\},)ʴ'+ et C(ScƷqTU+72 rz(+$N`i::qRiWa`xrዐsl79ݲ^9?ν Vk:c7|SK)TE~C̚$|dU)?z>#^QP%In6ۈlYH}X9BjhI[Gt1Co$Ewg)v;齃bp'#Bw{LuћZV*8r>;KS3k_g1K #‡< Q7 !&6O1X8^Vjcm*h?zۿ[va:p vK:c2]EIrnn|BVD'Sz$S]Dm(Z,靑gduv5~zư.ɷTKJI]̟mqL ab,hM8ixL}I68G! _`M}.zv˯8UI^fں w5ː܉5a;C*yyܼڄDy^FڡjweR|Une怅֑wX,ؙ M;ӑة>~ CѶCq4HK{@#yxJZX6l$"~*ц5Q^6GKbP[<{]G샪T;Ki!~&7SY 6w2/zFD9A"45l>r'ptX"/솻.5ݳuxN3VL 'qw9hsXL7$ vGD]74cU=b*TVe3ݏpDQE[Q1%%$ox ~ jL: ۦyLg!-GTr8T7Bo7䊋;qxX MDkUc ղT}͵-[s)6u ,l~wbޜt/PLi; 8eHIA-N;*E e=2Uwu$C{ZLLݷ&Λsu Ǥ1 ^LR%2.Ej#=,%s`ѻ&ڣ/4 t]?i C\w<t@QnWФ3ޖ8 QMM'WTTfȣJjZel!a`3 kB[]8:3F[CVlʸEfj&x4Qf^ͪ= BߦN)w fO&N}>\pmA ܛ)IG'd<㌺R@xؿiDALW7Nm1I蔐G? &wt}F>5ͬsdcԬ(H0( %|-`/i`тlygcbk[Ls=k#l}F~jTT2'ffcq˥Ԩٽ2$v]aia n/ta0V[%&`8^ ❍Md-С$_~WDe|پ/媙 `xʗ:vǕfR]OsMu r&brL.QGŒSU.Lhx´&ccgUx֨17RQnAXmŤ|]",}{:Ja/gq6}dN{pWvF=Ur80j7UKjj킏Uu{%\ %y'žF0+4ӄ6Ɉ~eQ-f&P-R+,%m6<>; &XM$m 8%:F(`ntd4Uo?v:nSKA%ph|@ݽ` ᷁d\y_s:.o>g$.X" 0577bWҟo| u~([PNktĞᅿ]6f`HB 0w!{%0'ܚ^Nd Bqib*ZX{_Gb[ahsH)USj)D.:1p 4shyA*UUYPSW-#̔r|I DUjp9 ΃XQ+\UѤ 1v]cLA> stream xڍtTk.( !=4HJ )C04tI )!-H*9kz};Y[WnQÐ|M=c1?? > &`3 \prP@@@HB40#@@ *! &#!PC|u8 JwB@m퐨28x~ (hv'TE0C!HC"%@>+a+"Wb EÌ oum ` =5 3Y௳ _ 0 yAa#D@0_ GW8*:P;t 蹂Pg++E4SVY+ 0+u^?u=`>}@a6HX9 .n5ſ\P0[ PTTXq@~?8U ֆ"v(P506AE u`ԭoTím(@Q"[  DP6p  Wgj vq~1 rC8B~m? B:)( uFO0^]Ajn^POz ssD茀FOTA~:#J[ʨuxBӓpKMrt߆p_w ٿMUsf]vo I{8dtҗkbӬ-BjMkU7h KzJzp)xɟbB]7Lg$̉t(cfedI44M_5F4\)k4:~7+I8qGhY9 :s@\ elz+șBqB}|L zzN~'@!4 =uW}itPW[6;M-W Z*m^o>zRmkpȘ5#\^;BւQˡ'ZWKm)1΍as&"&3zUD:ȼ/v'f 4)w<2ZrbR@~0 ҆鲦啪q@h17|j9sĪV3i|Nxfk-QJٍp2mFs87DRQ&Lo}OުY7gۻ%*B=NXd(\c0Q)ZڭLxTc'oQF*5ځ\*r|tdOT_QS eI#H~NA1j&xnlW%rzW/_Yw3\#aLz pz yh%U:ӻ}.dVvɯL1U9eRHpx#diCaȋ]g֔5T#KKO6f "xuG=+8579v[ G"\X$q2B;&kg 6irlzDl Q"/ bO/;8]Fߛ zR9"/՜p 7j奈 Y1Ӿ`3R~1HY)q;o~z?-N#77~.c=#ouǤĞֳ;_JgF5xy3F'߄;9CxESQs#UMJjXf:pl:GuHSNܲD\e~\ mpvit+?WRKVR,B-%pUsê{ ro͉J~!ǽda;>AN"|Qwf#E^LnҪ'0e^sxD.c |dšΉzK84ݫ$1 d~$vh ]Sd:IB( "ڷZj༺+E}à*O';H\h9pdKB:.K,Wl5ODVBEw-w l0KC4zs;Ã%yeT:./sD8d.F$ߤn)ؒN%@dLe3ZwM1~=}QxQ yR[-BM ['CZךfM4+Rq~x+ZL5o캺WZ#RcYEÕZR]|?ud] he_,mg7 # t~B11F~,U"Lis/Έ8JZyEՄ ڳ%VD'ߛ iӼaXQY]Ϯ}99*«7ɕV=Sz1N c@[#k*Ѫ|-|lPq[â& S3kX/Gh;Za|b-dwIZqiP}Sv-d}^&ovPm֠ ՁW{RÇ8c 'WupݱASJzyM=6|ڠ%Z){/Kop#V_v݂!#ߢ 5c<f^2Ttoyu帴1:t2~fa^g>4&Zr@KBCڍj_]m9-5͒Icș~U]Zg 5?g8%|^_$"\F0 ef2Ń>ۚF(wث(YF{g96=Ae[UuqiB1\ia1B5d 8EMK<@?Z!e\Z=W|Y~z59C`3Z.H]X8}v> c3DQ{ ʝR&E@M3zSw% dj Ny/-Gpήy@N|T$n#'eZm2E%~ziB!jho=uNxjO6noS)l WI1ސ}ʹ~Y`{q$w~a++CM:.vh`ާi o{AXm+ШC=B~G?譟.`g%)zOK|=AR]ωz4qU_PrM_fF=ZuK hdS(ڎr 70h rJHŊ,}&j?zq)F7TVi^в{SxGgnY_)2cQxqԜd(y{hi (=,ӑ)"1K.1BxflBdT|A~_y_v,ܮ ~KP؇^7{# Bkh~~:9`8߉ 7c=%)*,x>~G^]NmNQEb66ZCOG(oPp>U|TxOQ<" !֙`2+USӸ@&pc7|sֶ_oY{FDآÒ܎{=Y TK{ =lw`QY;!Ǡ'ЄcS$v2ŨXBj>Vcp/Gle~ "Қ<[wlYwuO>|$p.Xv<)UI BFf?0 )U4,UM&YFjZ8m!V))&ԁcVfH%j6S' ,v'xwJS#S5>hDu{;څN/3gxFJ;Vug QRH cv>k9j(=<3IMkjj`[mÐRq>B2|ONzjU)rWqCOm*ٴuHKg_3̢$򞓳=${%0$cɶب4[Kt4}+غ9ۊ'cFI1b_ qel{H⎊u{VK)>2&~dėPP]8qaTOtV >5Iwà\ 6۪w> Cnd5 =鵟d|SYrfYr>ƻH.?OzWU`Dgzk65w-Z{(RI]ޥ8xc+U>~Bd2WDݏuN n 3L ;Ch^mfYX(\ K,&Udne+o WO Gǰ"}w`ݞcEԕ9½^%Hә}.-W2Z/xf}r%xw:1 mfR㋭7̏R#rY74Ebd[}i+f͝t=? IjқMq!4>&hsޛh#7l$w<9M`i3cD_`O?WUM>geѽܠE_a7b+=^kZӴxu1N^G,m|#ҢRRZ37H1jTؤNw̫WS< ,} jԇN+K{1_MrF3tWI#|쮓Ln؈fb A@ByJ;a VLԛW^Ss#ygN=td5I7uaKd#۝.O'Տ?պ)ݰKqLP7Re ES>詠R E'c>ҫs_r粒䩙w!wpJĬ,FΒy?r"S /,Qj/R?j|rM|ukP)|F\Gla<Ϣhƽ9l2Zd}ގR!=A^JRtobI\@C8b&3anUŦ8 {Ehc;NZdžV#}R'oU(YF'#}#%/Ͼdl/J$v3}J"Bw®Q' '%%UXF݇|/lEf'%U4BxH Ϩ2k?~)ZA論q+.^Z-[H4E&f{^dloHeL+ErQZTX.t!9Sinr]9vk4NRzv5VADBO])zQi<<1W9W=#l| P ݨٛI]S*f ㆏>|D"e3렑5 } Vɦ Vb\Vп endstream endobj 2188 0 obj << /Length1 1648 /Length2 3429 /Length3 0 /Length 4396 /Filter /FlateDecode >> stream xڕT 8mNv%ed{fϞ%3L̘d z)eDdɒHY#dVַ;c뼞TPx#T337BU_.99g  sɹD[ D1ShVS`P(0؏DP8, @h.5;Z MZk_W ja>(` qĵ:3| ʃy.HYx.X.={x.RU|]u|iFk}Jw}EY g!ltussB:ue+"CeX0{q}^}'hHSg!hZ[<cC;|z'X, EʴMJqL(5ӟxTI/7I79!w{-MﺐWKJcxY w5-Kf1/=Vvo;S>yÝW 6^=uCQ:vb)pO02'>$u=fSrhu>~SD +k݋Wbt;5] lKK\>Q)Q/}I(EM(@ {"n5_sv$g1VbN+INɡe緣`t(mat]dwf֭x$7Ÿ^Z!P`ϸ#8G]GR:bUKA{k4% !j}yW{y/ {p7nڛZ_H_vUZ4lȏڙˋkΫ$kߡ NI\^5NNML#hڽ.P}&[)sZz469d X ۧӛjjbv~geF?T +Vģ)طrJ1mG[:h1oӼ}butU zoKHA݈M'U}Vuj";\ktSEJ YΛ]I8A>a C{4RowōC(+VG+9s1NI/ t%g,YU<'Խ:{j]д<~. QjǬ%vef @tVO,sQ:tYbQrϘ| Fv3Y3ާlxg|<@BHs)lB@G?0WHbJAQv9_R?,F'(id_Dw圝v# K%td]]۠[qem]ZYS!@,Wa hOL=0r&|f]3Duq"zFlYυǫfDsբ*oUvIǖl,zSӽ-"S1̩/?=~ԫܒIyŗ k:9Ubj{jJQ}HLK b<2 0Z$Uy%~" AݶG{L&{L/%CspM Mk۱\S&.Ci"cfj>W'tHrw鹇b}xh8Y@4;B/0-;H-r7 a'-kr/ eGrGө,Wy$ 'R R 9I{\<_i*4]0YxGx6H.0f~*(y^%sзW+ye8|9zB[q:bδ!7aoĵyexxP;\V&:'~jE6cNwmj5Re"վ^^L7](%@%z?2i|2-cF-_@t@£]z--]}?kAe.ՖN%Q<\"FVX=j1T_1< ;O]xcqq?J~(;[qK%G̭1"mB!Ym7N婄3lia}~0-ɴ_4Y,6V]s+PՆ:Д}ihQX}4Ҹ=&G♦ɢ).u7n$I 07Dڦhܵ4B6Ӏ0,oglR#{&ǎ_%-/||x-c ?=#RC9vp嵗!: Co9uqH픾i_e?^WYT$wyB j_%w:DBSuqDo!ܧcm~Kf Oٰpl<`"ܳOaik~O`uj2뎳:;v ~^WN{<5rHMl\N gQ[A+ﰓO |_ s>P(8o'a \J>:֘U.&h?dO~=?/p\Oʧ vA[!R> X5[G!t&' /4 ]^Rwi7g\m5gDʽ! &$ΗNyGemI4U}#;&G/fr˞Ptsfr/dM~ Q q)1ݺM$1]EkzCl'j I[j Qv[]eZC.tñ~FfC? z<|I]hC3Μ|=D ;|pIruZbt@6w+-8҅&GZq<_2_ZٙcBX[^Q\?33"~{ FW? yڣ{ie7]6ūDM}{!D+Dm6s"M5a]t]ْb+'=tΖ\2qr(W!UfoϊV_&r%^;]N{RUruIff;%b;Kc1B endstream endobj 2190 0 obj << /Length1 1391 /Length2 1741 /Length3 0 /Length 2602 /Filter /FlateDecode >> stream xڕT 8T&1+j"qk5rir4sp93g %עt5T&eɵD\Z=źJR"Mvyr{}=9fL5X'/g*P(tK 0ƅfqq $(bba8p "PJPQY"*Wׄ@`X8mD08"]Dv2M4M  S@02bQHpsd1`!I"+DqV( vY~3`'d `>&4ɓa;!ǃL \-*L\ (fc`#d $8GğTe03"p dW K1qԛ2/g;e7&Ń3 |I``,\={$W2 BgX qa qikr40BxrC1B†;B`6c"C0|1^Td7{窙2:!l39R4Հ% DP@b&./"A * LmA;_ؤq§ŝ5d6KEV!D `D$2_,KL3Ww3ޟ#3ILbsKř /=RU;winԲ_;qpChmC}{;#r{2mWCM72RYޱ]2aѫ[ kd V/yQԣ->?Xz0͋<F ~>A#HiYfD̑d*}zAlSQ+N,I_M ҷ*i]^kӃ?F9*)z)7QUI%p“zxPYQvN_RzIt¸YNY6{#bֶ[CaTSҤG w)UӨ6.z)\n16+IoMK_M CX{6Hy.CR)!}/y )]uu?O} zڥnŽЊѠR(Q*ALu֭oT樼/٦yvT- fvA"zKϵ!Frod1֫_ ]9]X,Gji\jփ|߅ѕyO/$,ŲulC|=[gi=)ϗ/)vT$ę~k {5NH7x6 z-F-:41I9Y{W'G7~]j .ԋ\ߔwQ_#,gsWlefW ̓1V3Zl3"{44 3@Z4{g+.px&/jYk9g])9Lq8+xwNIƌf7/ަů'׏_#^З 0yVΫl:}d'5F[һc+py&DSOwnM6ԗRr@eԁQLDUeB7˽Q!q/卉.tsu΁g#.k "Z-wmU;O(Isx^G w]^ܿ)uJؼ5 txԠ>+|m*.=PkN=U=#lB/P/ !> endstream endobj 2192 0 obj << /Length1 1317 /Length2 1506 /Length3 0 /Length 2330 /Filter /FlateDecode >> stream xڕS 8TNR M˄03g4DԺָǴRN39v4 3D6- *QPTQS*b+QU۞A}^>fv<Yfa ;[ÜF~)T*%8L~HpTY A`a之0"d 5A~A"5" c&#!PDb #0Ҕư2/  BlM`!NIW0oG$(<`!26.B>> " (p2C _|nS`cd1)_hЧrB(6 s"( PöXJ۴j~]uдK?'j(V:a(8sܣcќuUh>pgpN*AuVF^NWGZ(;Ce_vcﱶn RRu/2lycVzG5t_-8U:oӣS }rf&.MF*ġϪ)˲sNnM,7 rͺ XV}.k63!iZwƞg$W?XT>0!.izCEI3o/4-3늻׆;͙v~zgŖDݫ_ubT' 5']}UM CB#Z %1ޥDWCc4{1u֖wfVf|FxuA o4&3{1s7:YS՟v$Gk{CU#w\2MT_ly¨Is[?̪5zn0!`3Fkr}a'f}C/{եq7F%3ZzI~v ֊6M˗-Kȍ^/N~I9#fT=8~~V[v~fR?@54|A}:UWmsOl6CB8,ײǮgl.ɭe{NY52#~lү>'[V7\i<t /ȏ=?yw-Y'A\Nq{߮is#,)w;]+q)b7Oِͼvy<8d=eՅ%wFmfbsBu,)Z5U£1%Cpipn[.E7N~߼ǰd% [W?jV^*j8H}z׷w䯎PȊWt:8Cԁ(Ԓ +z_?kv[g}Fv6=t0H7ڤLt9/bjhu#8iYxz/}d `([{%ibj oZJOҧw=.q8/6Gg]מUbD=']qj>(Q%Cv7s[/ϊS&1ᇗUy̎@~GY}alernpNwRIۦ+{ѿ{nuHLvDu1ԇqxU[cS17G o3θN(w! Bv5\ϑl.!2楽9/I`M:FiwrT^;rӫ}:٨̱Q}2 Obrn\>RQRl}Jm07kj_AzDyٵ- w5! ;,gWY endstream endobj 2194 0 obj << /Length1 1425 /Length2 3010 /Length3 0 /Length 3897 /Filter /FlateDecode >> stream xڕS 8T JdVtP/ cɾDjw ("KP JKBED("BߡyΜ9rNj8ޒBjHu`bD:STt#C_z>E znpg&CiT GJo b:= f>ZM*pT\ MY]z@'9G*|Gl.8<} /~Hv%&VyCEՉp {[x9x>ݶ;E.Ļ 5a9J@=7)skǠBQ8Nj2o3DM<9}~8Z;f䑝߂>H Ѫ3B;7BEN,&fu2ME#eȧݣuGV;VsWYUdJqբYrONxy<Ӿ+͠챼lN{JqdWhx/Q[%ўٶ}βwCQd~=_v-;-c_+Q8V5I%ɵ/妭։ӫCL?9:j& H/ OыAJQ~Vdm6zsghbg{>ԔEkx0,.&1}UEN+oȦKlSz#ȕp)# 4C@nOs]vve|t2<&'G1z,{i a|`_Oדy KVEGK 5K3M#8_Ν׈7V^f}6IB,@am4rj}bZ8Êuc3 8QRAՎdQ1cQDs|P^ ڄVLB[Vxd6}vΨ̎l]uKNRt#6x2W8yQ۳B!/[iba,h\S1={PƧw=0\8ҳ>JdF}\S~E5;wgN*8^3\vlxx^wZyS}X/SpLTX SL_9iah|d(zF4@)}`}GFxw(qMn?/Җ0Ns=1ER|>L> nͭFgkdJFk)8gX(?7޷11&kiAL*}CG-daٽU UhoJg"icXrf%o+d릨^mvHSb>aIV;uWUnqOS-r٢Nrqr %\ud6cяs ,ݒ.fQЅ"O*)OwA 6_]-jkf{e.d 2"p_c}Ro~9HmKs,gZh68ɿR5֖^I1Gt25 ߚ.^❛G 'ˆ-葈W3$/EkYp<-~tWuh8!|vF@cѳfƣKdO,:<82mT,F‡fk}>ᇶʁF;_T9 1wC-yO{mm\=)860cKJV8tY0[ƱYzJumyۺE-,n-[1/p|1.\.=uMYWn7⹳kr<gSFo}D[| f|"r{gEݸ@$;15?Q0d波p+[ׄ\ }@Xt$U6Ƀ Ib;\]02{T9or}ؿ\O&l+K%lsyYk@oAG%ӵ^{z.nkpWrqw?CN1⸳/@p zY+tx!ޔ5oUSy0=ɣA*/c^M/+{No9jʿp$HW[Ree2A[V`xXq4nX}+RyWB 9)`Az32vw씙%zBGI"/%κ["yG>W8h!yFn@ d%C:|Hh|ir*bnl AujNku2Q::G¨PqNG󁻇F .}gWX@YiW+7Җ؄ zAZPY-Y6i{Ff.4a$ Sh ˌ):hIޗtf';.SEȗ*]tjQӫe endstream endobj 2196 0 obj << /Length1 1333 /Length2 2345 /Length3 0 /Length 3196 /Filter /FlateDecode >> stream xڕT 8T{/**Nv1[vd͚1sbB7ن-l7[ u/m7K nl^|!_R-a3:EP) @~^m K'bߏeY% f%G5D3e',C00Qe|b0a6p>thAlCNAChK)w7]Yi}qH:Th"ѡ0 CkkyQ?,Lv ˏ.KwX L236~Ihf<*7o97lkx)/vdp0 HDxͨ($ i."$r3i|=&K N/{LBPD>Fhg~]tng[DzƔUFg_[FoV_yaIu=+j-:farضzBRGG>^LYU>LH9Wոk<w;9R8uY~$VmZ]k64[sR|g.Aa{;8r50_v֩9WP'Y4*c-~o[WV9 Dba^7 X޷,*ޑOxE9yר8'a˟%PtHHs[1tjC9`|Ցqtܼ,{OP/Lh_)xUֱ*[E)<{n$;s3 ^|,7g)$mh׉lmojWsER&*r`ohq>XNzb]ًޑ7lC:GB%ڕ7旱>ILoltҽ_h5z݉j~}Jٛw+ Qy|OqfC}u_E\VNLN\=M[HGIg, D߽:=\BV,9JĤsjmwEmuo=~P%s8>X&h]^ɞ]*8hM {ӑ}n=Bp \H.v۔Isy(NP1g IJس{XڊOTˤ;t'y p ւh@̞nݸ4OĎ(<>StƼ]Vf񲤸7P)vZuh΂.e$x%)}ʭm!^b7yv֛Nn3 /3}V/i?;:?iVANף3رҊP;]ඹu ~J#I8)SK2E.:{ Z蹋N;LW|jӎ">dvѠUBA[f w9ݠP)#KasC';}SdMDaHO;ZBΨQ01V*ywk)@֔iXQz.-DXcYVQE!_Όx}5F~H+SoxY?Nf;DKkjDz OZhIrv`:g%J\Po4;Fmn`.汈SYOW/%'t é5Ogp8= 8Ug<}o!BP΍7Sw;l_jfuqZhǭ-[?@W z-'}F=gT 6Q> stream xڕuT .!ҡP3tJw H( 0t Ht4 "!- w18s"A?>".P{s!]MUo`4dDWp).ﰻ;ư iz1Cy撦)|+OdOJy &yNEQH.EډEFڜ,STiy{3]~b٩(GfFU!eb&Idr.1SѮ6N1hWN1X{/z,`|u41c Uu&KN:.d]>PS ԊPǵTX|.vT|?JC*/x"&k(əƇqʊqfi%ebDkL@<'$lGi >xެJޕ$XLB}*&~NQ6!N4]|՜VzԨ'"Ԫ>^ȍ?ݍ?.|fTln6ș4H,mlș=IzHkίj5T8I.3||o/T"|Bl3fw:{L|2vch3jW[o i]@7t?‚%Rߠl&]Жn-~ C$E]{rEc|J5*xԅ((͝s5!ˏ&%E*TtxR U/SL^9")/be;lwqBdϴ>xXm*8 -#lXdكUxP$s26ьY cua0"K/1:^!R̓np2ِӛ3%20^M֥ a8DLX7`kXa{Gڳ<?7M7=[Xwn6]+7Pi#қf]F[V{~RI*S%= ]<(wkvrU}S0OUI=YÒ@Tbjxv*ܳ%Z ^ap{:ia>:+8W}+Ћ(#S^L7cn D~s s( zS\-Nl krU%zjeolmn g(~*h'l0̃`0D[{a1S&'>mgFE<5Wvgr,Ha L͗3Ea ahpc]MqFO[y ]hRI*!B e W`qmuON{3o-Yz0:}r@So!tG'LNqf ]ooHJRi> MSi ]3m V3ondm03@_Q-]0rű@]"w_ըSncΚԂp\@1q+> ؠ}|5]3ȱc2m;yzefHwVW@,RuB)a( ڜ@SЎ$I÷bH@o?ލqUr49!I~<ʡĵ Dst.r>ɪ(|؃s 5yTZia.\.''nljAL@;"T|4Wx<|r \n'7JZFi$Q<4 >HV ^ ,Nf9a}z*gOx`17Y+?eRWc/;6orz<_KUEtGaT-_ɖve6M'MbC =.$$/8.|Sc9kɋ鞣XGgay@R0?AzxXxk4y[uOy  @Q8 PIt#Btounpy_.YoW[GZdy ox*Nݎ!dh I©u07{5!bTvj-|+7/NN$RxN] FQϗ*<& K}Δ:'l$,6- v :vmxS[0O=uZ<&TU""!AvvPfbHڙ)ՇͪfTb-=%35VfI cgӒ1bQFS$܅lA~tR''|>C FW䠞K1L1cJl:kUFf]2y0FAڳBnosg<&Pwk*b 藋o}f0"\VkF .Z r<d#nml5Fڊէ]q]DSN:TdTfCv[HFI6ANC7 bX(JsiX1Bǻ%t!tsK\Iʔcow悮^. LUu|:sa=2T5Bh䵮fE)Vo(H.>xI'MxS^!~dA&E-] dGkO|1e 9I2 ߞ@7}ٜ극y1 /mʷzxi8SVex_DRsA:ȉY."W *2&~$2b ,?^/eKyS: +G}~y9{SQn) NnLw3>嫐ro*cmy\4Tesxn0 ) r?rDE|n;MS/<+ZH WI㹋Z6+V>OF0P۸|'6 ȡӮf*U:#%G 4pX.?#}AxT H4TCDTNԆ"ce0,1^D,,L ڤpq Q?_Or*5 }pkK vEEҨ^YZ MLk^Kw/3R]˝n2txo JqP'"\ܢMe{{&Uz7 Ā0Vq1Yc>TP?ϟQZN_D1(bx5Q]Lk7SMHMD[VYWzv̆UnyZ-YCM;8.|xSL˛3,3 kb\؎{!Cj̧oVlna/|}iFi[쏡(/5ր/7dD^aE-o!=bJ R$#G%SYNo?I@/s2ܛ0vUN~k r+kܷ-90$6WS4|]+,|bn{k>F:o4֓df!K(9H+tGХ☪:4l;o1#qL") -3}%,/7w[&J5de7ZgB3Pi0ݯ6[{vY9-&6.&o)"xWѶ7(?pC2.QͻؙMNu~;#t.5n`/z&k:1䒃fG}}U<EO`IwJEQ)ZMq(G^>wY0U^?uXa_Dw|I!s].2޳ !2V]6萪2h8N+I(xB[ã_ϛ\#|DBHaݳ[$~Ec̜XׯĖ^.Rt^~׺#M&=ŗcl$vB <LssWj |Xը()\cKA*f|EgL4*9 TՑ>\K|S P}/l,FY)d}_hcͮ :/21g^!RZ!T/P?!!fÐ綕jb(%)zgeA-k{˫(o>~[~"nXHi%9E/^4!Sf^9瑬9"S@QjvYT-dEp'Z*o1Wc9ídl\KxA*{sa,Q5[tXf}2 h )&} :AX~Z|,Kwd$v{brUpO,d'h]V$qAx>㮟x?ɾˣ{{;eW7~ǐ,0a|]﷽y @`o6B:l}EMkxy&d{F.Py$GePۗm]Z3vdXñ06051n, x_yiBLr1(opeNt~ %8o 8,~,viZPbѮLI3NMQYs<^njqf AkwJU{/=-)B HI3Ix&Ne_DQ ѥHJxE+1 srWk"L-7*O+3l~L ,c4, 2Ldl;5r v($c^ e |L]+6uQ:'Z ~0^M<_)ˉRe4kP;=363jq=(႑Vk#mfCVSjޒ}~\hjenpyUVCJE"uDl'u07s1ݥ| +qC#lBC0!=l*Zd̙hX]ݗTMV+l+3K]Zu,qPmeXO.t?IvXc@"`CGqQ Sjחh<%g`FՓVbm(Xӄ|!"SsaCD @('4Q HZmm9dhXT B]ٺJe],~&F-k1VG^?hc,#fX[9BDlJǗ WgjgݜY#Pm:R] S5m!{>zQ!_ٙO#WhP[J}"T%r,s(۩7iM:+ӱ6~^vMw{ ASLk0r.5'HҠ4o"SUg y斍>"@-g*_G_O'*ABM.I[x;xɧ7FV A~qƦqДZps"y0f[ⳑTNB[OkݘKlXkehtIeHEݱ;iVxW1Arf_w2#DbmyK94_oQ KTVk.ܻIYQ^E1Tv ;(pq0fGN2MspЉF9ݕaAΏa0sn~'t4%PK}:!%ږҗВzJoאd9f+w-L֦C*+?+T |alQ4tMP$BMڕ׼$U[!zu`U[`ʀͽN2 n Ae H3ɧg OzAm"$oH"5N o}ūDBuqLnek!oƨuINMWS,&dM6~}'IA<j辚TiE̪U)jèqf^Dz=ٺAV>^h! eV6C]^] --B9_'eK^R'X{fQ"n0qeG7 `,>X5,ZMG޾7~ж)%N:0ތ<{i' k}:Wpc^q O.?כ'7ǺcF 3)`_w͉k]ܗM ]T:;skN/޲5ovkɢ P^xllox,@*_ ׮ak9O?\7,!)29JY¯8bJu4JXEBPeAh]Q*v!סܶi!4?9nvJzꌣe^Etm'%c; mmFS}?Ev]%G'_o=JpvZwdzqtۈ^W6 ?%]]gz5,ஏз9df!A]p%SAfk߭z579P ո9a"6"،:X#ɏXs㯘Q b!&֓@&͕엎ˮgåYuUj@Vy{J2&[9&!ri>th}4ƨ]"xQo6x+_5E j5i/V' sXELfBQyP,V?T[|EϱlN{$8;GD9{G~q# э#ϫ'[92-J,V xNrLj͇6n jFb$Rc|cÏ<%w7񨐧@ *4I׵9nW򄯿SqI\. Cώv:LIX\3ጏ;q|pHkjIt08{B~0uq8][҆fD4R>.ž=5eʹ, |RXr!E%[ ::pKAe6ɰ`9fo䏀. endstream endobj 2200 0 obj << /Length1 2036 /Length2 9571 /Length3 0 /Length 10751 /Filter /FlateDecode >> stream xڕuTk6%)) 0 1CJt )4H7Hw txuk}JMU$rq5l/:--V u a Ba2) f(N.A?NqG PBm@0o @Btrr }0X08Y`8XA udg`:B؜\EYPruY~UtM ec [d0 y-AMy%3, d7?޿قrZX89:^`k*`_@u WapuB ( +4RB`HG 5VKX8y jikmxȃ`ns?c Y8Z0/9oտNK/5_K tuP w`kX9Bdl=AjP Kڿ RsVN @tAO2F2+_veem=o_:iy]]^V/5'l,A',kv6pv\u.K7KFv%]_`::s#X /5ߐuEc ~#X>;D @?`a~CX[a_?̭>آ`Šmq2r:;> ư/?{rtjZZoNP[bpb8B./ߐWD9aq lӰC~;hCl,%(r0)r /l?y,y1)0>@^@J5-uMœ1lٹz  N׬V._/W.?ZZ¶_.JD\\[?;7[ąw#.ٯ'gu"È9*MFl9@C0RbKˠ ]ˆAA܇]I>x*ξ&Gt_֜}}-r'MSƚ"ngф(ȑN,zW>ۮez}@d)dlvmȱ{9e$|W*6v`gk: 'azi*̭Zp崓GAQ9b%cr;`$\NrׁD}DWS/Uc|U4\/ ٺ8l. L_ʳ+eu:I)/"ѲNԛ ?̦NX-G2I-50~`JAClO5(]9T3T[R&;GeTL3M,4ķRm^QC,+B\6?e*5_ {=vkp}@2Lɪo#8&rBza3ah/ n#&@}{xu7E!ŶRZ^>_‚i89t;׀ZVڧ{5_ pIiB-5 ulT"-T% W$ b)gKij.}ؠ.!D#?,#]cl>Kk!2ŝo* UcNURFcU1eGؘ2uxǢ6ck(t)КI(*ͧ}#dl,"ӧ'c5o%RlV vpoo|iU#PBؑo`ŒUpYvbҞ&2n?Ys{]/jZ5s@fe"7E $yG1nYLѼ 3}RCZ%#ʡv,Dw?_rt+RyŰsP6ݞ>%x3(> dc^E wQv/5x=.;(eSD l~~0Wb([!2r=S6śfQϑ"L|K4@c0y)i iʴ`u~Q_@ߐ#$0ӆj5"v@\A|ŠAC4EXxg}ϑ)ZaMUMDϗe4Dgq(bh|5tj @4EB25U^$h$bݗd^07n1ּN5sTa<([%GiTzPF4`id'z6g 9~Lh5:ߞD3H M+05wPY C4uFG~ ŝ=n~7H&.Gx)f@`瓽xlOkR'82\3[|6[+EZ,b/^ 1| ٬ l ҫ(Y?J}نWF`j>WQ #վL*5cFR{7@L6}IҢʾE t`날v %Y˦OPgS/ӐSī ^59YHV{dz\#(ء|s7 Aݼe%36vJ?r۸|,&JR/B]Z{f;ʺ"z[̓dIk$YÜn˸PHJ^R<$W9@k++]nI69n͵63ʜBG,@pz S;8{+Ey7ӑv©\:^qۧ(T՜uH+S281m6L/s*5ygc`r%qM'|լ,$`d uӿ+(~rAjӑk[nSW"m f=kK^V(%>X;:vэS6C!$ш4`p1kא%n7nnѓ->+$qҸң\߱O^Jgwcx]2QU̙~>{!7yj]0d6ܗ[z!ii`ACp3̇yVV"!lğ䀬²#FYfy3\Fߟ}rM|ΘXrR.@e9> 4\Pׂ]q0Ns(kګA0Tl,~@0ϵCO9c4i.7u̼ƮHV>c]QOH-".AdBS"IeµxHɈPObm)‚t3'{v}w -.i1de6Q- wP„e37rC.keNSc\8TkѺOߪ'*oV-ۺ xj(wkncPb/5ֽ6i/nQ~RF)]8pd}Β3h版?)pG}4uߘDW4.OyiSWwI-1?/P^ b ".f<y r E'{: S=[7f3󥇃BPzL)` ބAeؕ\hjn.ywڈ˺7y[q5$&әйc7wx<ɼtt=7M>/\1,h}ｉGRM2v?I?dl90GB27$YK/P,Ru =jV8N)"Xh)UoE4VBæ+4*I'](c,\r9GEU`հ}?/';8?!C/`p!Ym8^h&.9<)o@EEtH?v_fS CsUcCN;-p#'raxrABfGP*ecKBH{r=jyUOhi= ٓ- k8$)?gA^c0!Z9s҂m7D89$#LìPTsy"޾+7- Gc=_f^F6(7!w1dpU0 3-|9fc63W}vj|H13mW} ۵URמ߻|کD`~xdt g!DFe 틸9x$QC &OxOT.KD{/yh^Lp<[- XڌNߎ-cތ-{wYiŨnBg厑?c( }s'Y9\f5'yO\!&zH~r&qg!*ߤjQ`eM-[4x1\iD3-4!Fv5C@ر+UɦE -oi7f<mKԗqv{޹$uOhixF¯WkxuE _jC==of=roRSzAvk4PZF.ȫ^yn&V)u{yv7 y4reuN(Fb.Ĝ uqқ{R$⍝ّte$+mc< lKJ ?]#*AjEY"҈1@ƦPyyr Ge (%()i tp-B§m۟S N,*Nc"TdnԕPMIWSã{Mp}/VJ&&C-ص V4dO 4F=@ B;ѥg~]`o_x(=H[])ŜYs{!,qJF7:?8RoRxp)}e5p0 >^"RtY`4х}~ 5jo(-11..+Xqb-4:UQZ&fA[yTX@TԤgXKa%F)ZeUY mӗ4Ʉ6j% Vx𕋻/vjٖzh?ZS;DvcI^g! `~=e}>ܭ0HD<ߚVFD6tmIW->qYW+}</ lrR\\0x9@y1.dP:KU%[.Hr([={|nq!V(y oTg1+㽽 .?sTl{E8,_oE+,4O֓خOtAC A}p0aX a gcH"({K-o@_/0h}+?Ę;ط].KXNWeM}ƫshA)R"p5hxݤeEtT]cj18$](렱v=TIL+< gT51;,rqzRRiu^Rn\Prxx?&Y쁱Mt&_% >*ivy&Wʸi%[xWym!8*o'g8sc;28kY=~(SLE+7y񊻀Qަ-1u9 8Ii9@V+7I R3ʞ`AP6y_^/eA5Q˲# Gosy2uW%S։M%oФ"ZP,Z7~B@$ ȬP!y3hb#SWtex\畏ǁ!SkZSy_OE}%|+lp6"Ijʍ})E}zNA\*T\OKVMmv*"3ܬ7o8RL7 IoC}mˀD#I՚RSӷk7.ߩ|e!C3HfK S;^b%lC>f8FݽAj>pR.:uj?1W4I\漑7-4Zw%Rp8Kq/_,?)Y!,|<H?\ZЫ!z.:p '3E4t9EbDnSkōm[Y>BTpYk _.zԷs5/~Cm13rV^L:<5K&DwO /(|%K/(/Ŋl 5j%-N Z_~hZf_R=w(-iO7vSP ,gxe$fG16''pfEKHSz'}uzI\ޜ޼=wf巇`͏pyF <'e=^wz O=i64&! BIfF VMmN"L2Cq죬X-^)zPFU!0IΉ^kljMᒡi\1YŢz,~ǟ4&t-OC#`.:U~!*yHEzt=ڹLA}/p )M>J+/ endstream endobj 2202 0 obj << /Length1 1415 /Length2 2754 /Length3 0 /Length 3636 /Filter /FlateDecode >> stream xڕS 8WT~҂2Sm, 13fƾR=DQ)KE Y*o%*~o>}|IQKk9m< P 9{h"ۓ[VJg`^P1k7o{ۓ.;tDp9‰ӭWRaht7he_^\)[5ۄ)QgQRMilȫFHNkpG9WŘ]}>Gf92R|޹^ry.05cʳ>#[T+ ];+ '|.'S'ݱ^pwa`)1x-73ʱw~OhfpkI4:uAkF Ԕ!SKbY6 F2o%Dmݤ{À̼UbHayd{Zx~!vI@WH^ 6%N&\WU(1j<~Oyϕ^[_\!1>ѽ@'EMZ)Y [gusp2^gC7>Yb-2{ym_orKQinxlIs_|y Kn^utw}3d=mv6b6 w@ݣ 3,CbXIfGsۂc=2 '+OHfBL* ,tZAd }hHmd0]^^rmcR v;kӚ;C/B5!1QfDžGuXq1Ni,:%88vdՃl2=ekJ?ˢYX+^v괔D[Ɲ9RCj*mn`Lx4 Tܨż9\6 0"[GtR]u~~vPz29wiUr90[\ ĺdk:_ʁ' ڇ<<68e1Z#Ǧy⾟C|啜B`PkQ05mQ;xk($k9>hEHhԥX= Lg6[#J 95 剃MFR^<%k5p";sw WCRd}ݚSd_C{KEHq=-R5aC?@ ԿUyL8cDv{Q k?U:x,bڏ/=@d4/o_[T_.9FxzQL}[[!qDzT[.c"t1 CVӌtH:Otg*8&7W8UsU-3UsanZK ڽv|` 3l=)(` ߜLSE+*>>w̋r9xؒzY<@yolCNUD4qύξR}}tfuKCSͿ\!( endstream endobj 2204 0 obj << /Length1 1264 /Length2 1538 /Length3 0 /Length 2337 /Filter /FlateDecode >> stream xڕS 8T'&)ILyxL3H63IQ!JOJp)"'IJwJ.9};߷_kZ{M|,8|,b(nAhVM21Ep<&2C8Ja'|NN<0,@ lFbR6a)ƒPBXLd xGL1Iq6 xd@gEQ q\¦R,! B `i$-OH kI2BD6B(H !Bx0*#2"P>,ρ;X!FoFT xX*++HP$\οD 8:'E$R#Re;|GL,Q\FX+="d Nn|B`9\PѩFe@| E>Jzb0054x!Ŀ#OY}[H6S' Q( BK|/ A(,jb8$x,Ť$%Ӷ IRYLfiIUv!(- e*M-E`5Hڟ͌0Y @Yi0H^TJ(\UbU_mqa9#=z%?x8/~RVV\ /V%^hf@_6?6ShK nٕ.Ee5^R= δw_YVY,]ꉎ@ف%Z -yuv3=˧u)?H=f* d*x$ϴŊ%-f kT]7!rދM:S , &5Ȇl5knp,nN@Fиw{zJiިÈGꇸ ] 0ܣ|͛oj7Ϫ쩬[.~}!,lٝ-Skڼ]ZzLmNwt mr߶.SNxkvga:=zR-eaIcgWzh;KeHi{fBeylp1*نjߺ4Iyxɬ5/Ϲ[vpn nWLSX߶f,-X)#i?ZQ \G}9Zb7PX99ZՙigajDB~0Yw[;l@MRBAFlsЇ}^q2iX詔B,Z9p['>Į2(9f^HvJEAVz]txC150q{~Z0Co󅻦4/}}~cV}%kt?hK&֩/9:($R?[̷k=xE#^b)9|‹ z+|ͬ;ם>wAdiZ?B&**}jbVw %fIۗtg:h w%UOv5pdLKgUu̶JoӪ OO'hl|[iF疧WDG|Mg29J.V-XB2~ Gݭu例Q[BtW][>Ք& :NV:Z/8f.b?t7Y47v}ْ} Zo: nL'fQ}yOԻ7 m󤹵Jd?>TDV,dǺS hkjQ455첩ʸY!BfB3P-*}:4<%-ξ"VwVՈ}w:즐+uPvuy[~vziUjidVaM)-u\ l/i5DC+fߦ{pC Xi7 ~jߑɈye6F^)/ɻd$׭Ac\=~v8{LN!SKI;7CyeafحKLF H|@D.ǿ9Ì-[}Nwi O1EX[~Vgv u?j9[3ZIM3'M{`bKcQT+68}> stream xڍtTk6! % "%C %]Jt  !s|߿Yq+nQP<@ ? P#o;c $ P6E:"@ @GHAnP0@A pgO%ns @ 8V9 P_)إP(g >>www^CQv}p(AN?x XvP_ n Pk y C5M3X/7p+;dm wr<0[ QEy 䈄ƃ@PG-w t?5B"8Js;f%XS" ַss0; EgB`nMBPa ?xX*` m wҀBm HBB|?? F Pߚ!6oofq00+Ӕ3SC<- ?VgmA4_ `6p_n7?`5WІ`́@IWgGʮu AO1-ւNӫ.APXK;ەw;"$ D@Xd`omxapm #~]b""m mmG| !Hr/kWv~_m/8&X[KwѹlKOn^Dt^pTg!Rzl*>Zb>hm kKkan`aD\Lc{H;Opwgɽ {ϵfCG*ɛ6$0ҮcZ1i!: Zb+MG":3[La!uT2|0v#]]Yq?zi+ވ_B̈́HƵ*Mkxj]/ (˦mڈp>a74 }e͘~a2j{TRqcΦ#QWX8El>8חYKFРo<[d6rymZ=CO x-q|7عH 2=[QG0دȌ7$|` B}Wܢԕ CY>5hϳ>z'=jYErSRc^ p4to/SIVEǺ2Dh z(e[xk0iԈ'vf5֊ߧyccs*+k2b{t% @١a,B, EOcQv@b7U/xk_ ;D{p!FQܻk8Ԯ715 [8.PuYa͸ 'XLwUŽ{%?^']Wʚ(DLJkVv3#g҄gcX/ȉ-[E^] xvMp,z#T= 4-^ގ/&Qy![1|Kaic\%7">S+gB~Pcط|ŭGV#=i"DLI='*e|hTjénI}$BygWl1oɇ<N>f4οH}l ̉Sy2^i=:+U!R*"ѭ /u-T4|EqjK$]yn?Ĥƈf01PL!Gv:}*i*dK%"oO=K)m^_ggdmmO/,qNOH+#l0n-;%%"-~(6`ү v-JT2fjitn.Hp)mQė߄j~8MΘyV+n)#G߉"lW b؍"lu.ww%p(jm1p"18)yӗsPyV2ItwV2BT~dh&_D8gNn]0m'4Qũe-)`NV|٭d ]3e>E֛~`Uȍڙ["7%_A2v! )e?Zs-s\>!U%@AL|Yhor˩hݿ@~˅gw<g13U%9NS͙S5[ujcf7az-øw4rG !*W@~ۣ0-%OPȴilCتD OGg}]]p=yD`n4]:|c*_j20bzDZ1C+PyV| }XUc+Eu&~4uZM01lMz9kR`ݵ*әsSWNvHhT쓂0Ysm?OYՠR]A̵zOzVQXԊcuU6v _bKK0yfr}Ǎ .50/LύVb ߷gٓwte%5@$Uhi >8wFF J'珪G6)U'HKlGE$ʛ߰!`cBB1IjS!`saWi* j[~ը}P p9Qw%P) vۤdeϥ~aܫ,q> @@'Igio23QfwIMղv'>p?a1/ۺ _6gMpko>ҌuO8#z/=gw.vo7dPHSH&^SlBJG#$C$UH iS! g8c~yuVgCI]יKJҙR%1>%ٜr),WN_[ Nd67ݩ&uy2.4DYVs (Lx?[P,|*cvJGD'kF9#&YP/.ct?q7t5%޿jJœ64]&DeTIU>eeMhJHd@ۏvu )ʿyɁJ@λk>.(3E %u[c˗y)L@LXlsB=cx}^Ƨ+c5qبW)4_+Vgq#8ف$o/*0ӻ"TE9_a(g޼z[gpO4?ih.;WIXI#*_Kqցű`z3&ȣ442[a >}Y l-`;|#l~K:3':u渷"짾vgA"af[1eXτ}7;f?0PQ^ѽ‰FUpLoD~%;_*Vpg ]&,ӨGHG5  Y^Z V`Oy$vAڭ.rHy56JR^3(-C Y}3mYFkxɷ:pòXn-Q$c/_I_@N]eh91́.cԽWrT@IjGt)BCC5mV00i_݅W-ݐܞU4 %>T Ѵyx*>M%1YJlD!9cF>#oۄf#S!:aZPs:7YQZqXe\P·̕"ϔ :g{ҥ%#0@=uaowiH`/HYv0>VN )VfJ3Ewg߹p1ٞ?9qp1 nF{bҎn2XL>vA76}x+W[QO5m:eqVo-ܱJ Jm M v1.9}> UBC 89ѢtG秊^=/wG 19B*p(P˻oj25FslӅPf7w 1>xvDJ=uYFI^$/iy‡~r |jQdNknHc _c2UC5^" %E6qEP>4qŅ'S'=ϖ|,Ր{,p1v9A&lii [7,i-5nJ$bU%MlǑ^h,i w} ײ빕i'[SM01XfA]e;|Y03U*J퉭Swѭt{VA&t*>.# fs^ųmu)((J3qai(j#*1O#p%M%YpnKaZ-pͻ5phK@FYA; ꓆/S} :֊K?d~}\U4=Lj@xWUPs6m?U`;wdQYK iäP^'}A|lV 6>CsT(:ӱfga]{dxA^i$¬:7;xB#O0kیN9(9*OʹՊ8OQ*;Nf32)f&e}tc&i.gPǴv\#Ĩ">9ܛjh#RwǼ> ^5[y$;ъ肁`g5>RkjvҵJiv9zH"ps9bց!I<(&Lq^ &$&_%nܒ~)6(و ndӕ&l}F~\ cf_IJ endstream endobj 2208 0 obj << /Length1 1412 /Length2 3482 /Length3 0 /Length 4397 /Filter /FlateDecode >> stream xڍS 8זg 3=Nc([1ØcldIeɞ$Vd-DY [ ?u}5sιy}EIcIHKc3 c8 d`0i9J!BgADTB4 $Or0 )( iȾ3U}D1b\AN:R40FSA7ڍ40#a T*Y ByH(Nb7 @bmʀ IM ";=~H874O"3}#4$H?.'$1Mu>T Mn-Ў4Nh@G@灡T)GN A/"^?!smĀo0! ah{ vR7oW?x;qG ?ڞ XGhfLJX$q!?Hۉsv{yrBAB`4HӞN2,#N*E$Qi)ٓH)Pm׎u@ X%?4EdKݱjIpG 4읥A aB\JCjꔖejV׌ie }ӑ_8?Hn=p䁹ڽ3T^˄sͩSz#k"։g ++8՟NWYKHG)6ΨH2l!uu%}x,u T1x8_b~R_BynxiuY5&/!sdrגLvDYT'jK3t$$ʤqO G^讟EFBʦ'0}a̿ZKO=%f|yƸLuG>;] ڟc.\P&tĖt7O}cus+Ռ&3Kq*mBjir%эN/27cyQ|M1b@{W#m'ȜGS$@O3Z^Ei9l[ul"1M +2x-Fn m}`SΨYNdc8q"VYKʖMIF/Pm}^76?PȈ#֚i@4w_m5w"q2pns2鈈Dl[P|^njHgX|{'[" =<[9*:3ʼnqpmSnYy'z6vUqf<;`sRd +>clgP>YFHbTԇo?hbXTJq-溱Y@<C&2Dfs;"Y7c@8d: 3x((TͼUrR%Mߦ{$L-T*ac1Rݢb/DeV\ݣP.AD6pj~/)oNfo3L&Ê ms??|{i#Um㧃75W M A 4?:r DV ]MkHL(E/a资PK˵JR|UC CšjˇO;|k*o.tc1gN 'ZS [p٣ dbDe/,D0k5ʎۦS68VD;Af"[E^FuSO ·t6%DC40v(vW>\A"2(޸ҹځ55Ty.Y#)&gub,K9G˒!]`l1) FXܳuzRœ>zO`Ǧ!W* 5uMwd~oSe*ng3W=7—[᠂x (=klL\@r `c(TaR݊Oh!p$]^yp=o~U13IVAEZZ|GvqI܋J1 3$} N.C_ٽ"@{TI7n7հeH59;Sr[);׆HѰ x eT-ᦘtt*|IR%B/OӴaEcӎM(hq5XWS7Ni̴kۏ@F ީ i"aALȟ.=e¹')Haq}ͫ3\LOyfcjIh, v{>3wzu.8VA/wo +h֗p֮*JBĂ3<wEaǺB+Kz:,=ǣrLfRmN,:FB=Rm9wu?uAZ ̅?!ӴoЯCjsM\r+R~ovmf3Teni?vP$K:>΢7' m0&Ǩ,A 6\9x3z&0Vy!BVn ǡn3b$4Ylˊ^.ITKZZ_{„^&;fY#|hʜ'׷ljr X :DNsG99 4[)&N->_yo4zrդeԛP34#gO٪_tbiza${-WN0cUZmԌ`8CoKSuU|ӹozQ"|Iw0[_r70w5D`,a%^N߅X~B0%< z~/SR}S̘t }j:ۚI2Q#s0V(4؜P<4Yij<)d)V'̆BWa&o]JWwRg&M endstream endobj 1657 0 obj << /Type /ObjStm /N 100 /First 1010 /Length 8231 /Filter /FlateDecode >> stream x]Y7r~篨G;6؅PlOI(iE%=3Mvgw~* QwB(G旙(P:b<~S9;m)ug ǘD:8wN TbL\4*ǀ4Kڒr$c] ӥ*r>* JJr%n8Z)C3Q#T}|-nZtgך u ]ϩs;h8ɅDɢRKꡂQzʜ$ވ acb2arZ$bpqGcO^ѣ1mh HKh1]Fȍ!HkC _Xoq5{=~#랾dE̍{2YSX[C|. .QqKSbvKa8 (ug 1$Ww :'xblI`/A3\mXo L p T1"fQ7.$< "# 8WL eɷC9/oIXybXIyp Yc |7?p`v7)9>&e90~J*d ~=F}N' zçE8oO T`HX#XL'G>cJҼ1R֗~>f,v `gA=A+>S+l:>RzbPoYNЗJm}mݱfדWܭI6V,hQb &9͌Y >fgAʺ%кzhBlJ99kՑ@4/d B(g<7d*UR5vr+lMwk&0䇯L֦ M:Pu>uDJ 7iDQ5tJv%!`Z:_r (zօhw; l8~ʔY7`4b4F.e.k.WEr:tyugV")3eL )W܉t jcѫdeo{Ce2eC\ 95,#R)5Jӊ"GT5kh *R#T[ ]C>p#L2 nLEBpv&!fA)bDdwS5.y(M2+mM{g=.-["upTI5"}$(&ړ()%JZHC48BGW$JvlT9Wj G1 m҆H(:&)vEԢ IVN1(&[0F=b2Qt榒&Hn&Jn&Jn&ю<">=Nv¨38XJ% y I5Th5:Ar(ʘDQN(q5_uS[*%7"nqX_g&R;h ke<}ӅeW[E (9XY ^W)]pCƢl)\u.HW >$K~(! `d`Q#6}/TX/ERw`V&bIr6b&kIޣT[R>fQ"5Pjm7,]Xua˕b6z*XgXW=ɰ ȱ(ek[3C"R^Bt˰,: 8~MysѶK^$sM"a-~YadqP6VEclYYF( j$f+U *AȜZ /q@m>N@!d 45c^<`9ņW* G$ SY$у1Qh"PNH`$PT:\Iu0#PP[ $i6a\j-Cȋ=vݲ) G6fZ Ze ĪP)XQ=!@-6 :FLAA:L7 SԬHj71JH.в .'!+ `]`,6@-+g{5)flLD|ĦHr ,}fmJ_ J130(pS- :yy1)-ʳ]5C0v,3rj'/6( i8R PQ.Gg 4p@T@Oѥp{A4[d"lܲ%A+h$@H0ʚIL=<ZÁ&?4 >3un8 3t!< ǀbnPRA(jT\/LRLPS5cg evSK8ħ g0ALyg9(/8,7hTyIP РŽu:hLWCXNx=B(|S$$IzSstDD䆫~w' {g:G]K{vɝ(f$bDBg[5 -"C͞E/%!vsy[Yg^$R0{ϢnM1LTicődh03az #%)451J~qpaoza77]20;}g2?&T9LnbR`00J'JUcVK˞'JUnN<$Iᓑfd&fd&fd&fd*IӓՒ'k$MOJD<*O<c֎C  ]WD]@IO1J؃s- FTL!fTBf3UPEM*ATHUdķ6U_Ԧ*@LCB|#( !?5A?BOU')H!?UqsEaaE0ir;C/ |P*oP*{TkRhU:fHUfVkaUfVkaUTe RUqZ*HUXUQT%e R*HUNfWd R*HUH U9-UDWJU5c2J˙Y $s$s$s$s8$̪N,$ZK\Y; 3fvwd=AYOPgԙuf3AYd #7Jьd"l9V a Ȍ[&gX7Qb)P,_;PQ/WW/slAvئIzB0iEd2 h >9vf9hrEcJ[rU+egA$W@!5ߐS(W48ojTewTA8hj8B( (dk8B( (QG9&W1ybU텛{rŐA\Q1X8h,W 4+ͬ4 *hK@c?\W/rYߪͬ+߄r 2_ꡅ&WcyŒ‘o۩jr᎑bHmC2nOp[0nkm¸Z7nF6 Z((ՃA J(n`vͨqjQQ FA/pm䓫 M>yi$qaTv@T@]8K"'Pneq%A >7kFsKRc#SW/l(w,C ႛF9o,v!0(7 gĎU]\:P3i}rWK:|FOu2MFPaEREܢo%m=.J\SXgQگ?Ya*(xKfMʇ9V54/T5Pj_UėU3$n̨2rUI@P_QK>!E IZ)HZ)IZSNKIjYhn(mB |+e D5CܠQDUY 煝BPH pC!NP'QH((0L\l`F Z6(ؠ`a6Bet]4M.'" v''p" q'b I Hj't藮vT%  t ٴP $b$~g 6D;-#3 {:# p`a|'a-8fDp ՐOr|(SWƷ̒?eIC<@$94kK>uㆀ0˛4rrΩ/<8uJ}g\^GM帶К*1Ajbpj;VۡU +*\ueS\VUWUW2p%UC(v*PFʓ`ú/GrH.lXWQW/QC$xu#At#VڪMkt6 lXWmR搰mh#m5Fi1H[Fj((m5 fɆ"SPK6 !YɆ8!fBWBFfdɆ8!īYK6''(x !yBH;Eay},ۍZݤ zuC;I*!J`tϑFx(OrkUɛ)C1EFp9ra- |y$%9IIog3Oxk'n՚3J|m73j.ԣsu10B~bxwT1E{7q6nGW`5[i/u;ԪK/%XyJÔ~ A y JZJ(JJ(JJ\P٧ӛlcg/~ӓ^kدn;'Csi:׏劯z2fqQÀ]ϗ.?/jq -!lBOn槿-VkIdW2$C1t64d^XM6RuϯnVs|=-8N2j~>zh\Xt~⚷Q?V˳ſe|IM˿\{H$Fߖau+Qg7]q ε8 뛫O?NOVKdoH - O7狳໸;_-F n9t\-nne1]]˻۾q0Gzq:H?둔$ 3,ޟ/>ޝKz69}ћ9ۧCXhk%4[osa*?V\^,!\h~?=bm%.R{bc4gWd*3)kF]O]"4˳ ³|\~5f~~U>oa"U+,᳻nۻ eIJLҏ^={&_@Qzl:Sw OΉL5ӑW"x:Y2߼ՓW<04L5r0A\W~zϏwMU8$MM=gHpt9j{.N |uq_xկwb5_W_I X L!Atb[zM~o|{\L2n$Itoǿ7_' u?9=p+??ԅsˀx%a3ߗ.ci(,sՖ[f ś~óca. ŵ8ȹJţ\`&Bs1gCsϠv"O~Oh_ t8<mځJ&$o.j:V:LH+vy<߬ ɷϾ{}S:vk)d+z?xh$:C/x;7/O?.N7 mKp;2bo/$Rxx%|!p .qψVu_ujiomhm&cP;>V 5.v-6N^Q__;i~Կ&YnT>#&mԟ7w  endstream endobj 2210 0 obj << /Type /ObjStm /N 100 /First 916 /Length 2921 /Filter /FlateDecode >> stream xڥYn7}G[d" -'l|KAyP!KV/v~=ӗ9^ib]N"O;בH)w:NtI.%N:I uB^7Fg+^jK٩ׯ-{Rk.jw4Q*Ůr++v/tĬQ]d@CRzG hi͔E*jJX&K8[Q6^sVn !tl:&3iIA.fCT#GIk SЭR/\WgN. \ bӞ*7.Qd!}-3DB) Z RbBNEuOb-'gJI 'q!XR0TEE].JƆʾ+dJ˫:@HTSrN#lN U+wʆN:K]^4ik:-OE(Φkku(b镶;e(5bt٦C,Tw 'Uu-@i vi< dCP(cKl\8ջo.uwŧ_wgWwf^?/wo7>vj{5U7g//7goo~yEK_|q__<f'N7iaqiLsx.㳚ev>=k~܇~MūWX0-oۏhRwiދOOO4h9Oo~ךىG4 ?7W\m?~l^Շۛ\q*4|Wa˃[kT0qHs$>P\8a$ÓH3>igᮈ{x>>_bȧ/pT>0 =bнR,=0OnJ"Tt0}iHe@xopr FY~94qhr4 }TڅЦP!qUv!3ϙ"v0ӶjF]9gj5rzr'[uvjLLhΎ2#=S\$Dih(1aC6j,d%,D%6n6kd5C(/_0%,cff+ cCĐ`,1d)Kn`m 0-[hZ h,ER4"d)wfR4" &eE!k)xV;j4zš'@*%h2$mhKmnMl+x05:hyM6ς<x%Xz8B+8z'u`W $*@jK^o_Up,UϐTSpq)8dM cx!‹ 88j&ਙcm`lct6K F)V$߆aWtIm96~Mt[hӒ H@l:OD<Ц37oq 'm KL|L :@*KWA>ٌ~6ǒK,K,HILP`xR~6f@ endstream endobj 2248 0 obj << /Type /ObjStm /N 100 /First 1048 /Length 4172 /Filter /FlateDecode >> stream xڅ-;]" O0pp0800Gki;Z] NyXGHY2ď_/t\k…Z3Uo}_φj3jk_-oc7ca/oy xy_WxUkOx.ycnSӘ>iWhqum-55s!߾Ho]t3^s87k~+^_b:Z#Vk0Oآ%+ r < Ѷ@.`_n"W$45ܗcY֨0s_B׵/G=+\ګYxG3=}rÓܣmu;~˅TYtۣljP==Zܣ=Z5d[۶}Gy=ZkmSި|ϩ ܗ{ ݣmܣD|/6[r6ݐ܃m=Μ9myVR}W{Df=Zn{=ZL?ki'cW`:RHWG]kn6;~%ʿm]DR\N7%Wί?_ퟯo_6;iԤidҔ025t N_?z?A7ܹA !MӔiIS) an׳@W &)9(5pPjRR iB[/ͷ+(5I)@IJJMR PjRgݯJ&)(dAIdJ&CGm[:(A$J&)uPrIfgݯJ.) P%^r1 g/¯7@%J.) P IiRHJB.0? \~ J2BR7a"̱ݟۭH!!M@.!M@.BTonF]2Z`4FCjX`40z,Dy; zҐitZ4]4dB$ IDgQ{e4%%]2&(:%AIo J,vhYh JSRJP!4BҤeSJUoS~0SHSԥ4i4i=sUQlJ.eBR dksBz BR d$%j$%t*`W2Y[UA$%3;V2kJ^N^SrZג2`ė*[8*$ %,-R1YhoX$ 3,% $-^K^mܶ- xrZL\m42[H [^H'l4n&c.T L1hsmg7q [&遼ǯy琣ںMچ-ڦ%mK }+Dr7K\%.n7}oxV8V2%/w}sɋ4'\c۾mBoK`!q۷+=>9X$[H`-$0lR%} Dosknv9Jn%0n%0nֵ}rp߶Ǘ`>3;V2 K`K`TҤw lC.o)/DZP6[dfh #;jF9"]?zz nصawqN40fp-}'Sݕ44`I8L[Vj9<ӂRcvX~ya=gµK~Xyj~|B*06[vk\\?iXťKc=^>i=+xbpKYz|ab3m~ ̴)?e_/SU~ ~Of3_%)yz_S}}YϾʄer/ErfzIQz[p<`r/ɋ_R+,;7(Qv $05 V/,װJ7F q@?-7!)ݴL3)k],  y/,򒼦Ot;^v*CO~/< sLe6S?moVF[~P;5wL}8V2'o=尋%mI[*c2:|Dm~ֈVoX˼x2;_o-퍐`]c].6τJt K`{K`l\K`l\%5wL}8V2 K`TiR%rq| B%Z M=%ט6eqY> stream xڕ\]}ׯǝT _U)M^ll_RU~G~ͫo߾{ӫ/~|)uj˲2(.P\֠쯏~嫗cǿ>o'0F^EU(K.KPF.#(s}E(tΗ@K _|IG24;)NWtӕ!]teH8]%$ "(L+Cʐ/q2K r$+z@ +HXv $`|-ī;;>O EV5ϝO9W#'(s:Aʜ.P9Yfc:_st@fI/Oe|y/+˓8P|4 9.ety/ N%9J^OL+A<_˓~I/_Dir@Ky/ Yٳ~! #8C8Qjۂgө Dy`Hn4JF|pXԶ|4̆| ~s\4lWD-yI-b ~F<=Mجz6,r-9Aw@l-lϔi5eV)6%͈\o1PL[H7ѭN11NsH69wp 'Ge݁v_f he|ynՏһ/QMO 2f*52yzЇ~tx<mB7}[e⫌ьp8зUFNA %_d2bŌ'dN(*L=9|:A,7n(,_<x~䓴~>568&?{{k"NAZyoy޴\-onvś_[ݷx yr{O>ޛC~a (G7כAD w-~|}y)&-{nB˳!3`3铒 Tg nWo`|>[ YDG3G1Q<x6UcOGC\~˦Bp޿-?|~xUmVr _כ]k$ƭF:2uziu0] v75)5Pr]Ӎ햨kcW9}hl̀uOkK-kn5vԼ6浯4P7A7d:)GGŽSׯSظz WΉ-. ״rqx7Uvj]t_1E (N׎Rfu0krUWul\;}hYVwWlU~Vݹ]5ڏALs4¶[ܮ+l۔s/}_٢F1?TzR\n9ACuWM>ҮVv?O>=׭[WSw>ӾGk.FA׻^y]ׄȧ-i2Dƌm[2fzZnlwDlyޠަɰ -wA>ME#1$@ٖ5z 7oevluuAF 2Y]PNs(r̠lO$ey{H;+[w tK@yM9o9R/R@`V+na+:3Nq Z:O5zU؉HJw95ڲ Cu>sPm|\V}_~=lPТυ^o?7?=*J,( B3*OsĔnG.  (("xvti ҠSJL80q.nsIu* T$|-a$`wN T$PIÜ<0y)uԙL09Ü:)m \#rr*KUxn^vYT P%@q 3Qg T5P@U5y3n?ujvimoPT P-@pv WO6i*S L=0uր큨z8Q;^~Ա<>kR( :qNvt3,1^Bq #A7U @U TG9(_SJ;P" 3ȨJiPy:TF:HL*R*GJesy|‘92sʑR\go^f]tH́*R*GJeȶ%@EFȨ`#TF:H\S$Tʥ|-*lct%@EBH\91#rmEc@\ǯ΅G=)9R1@ٔ#r_o72(0G{D BӏV0<2nxMqp𽃩xN6pS٦xM2Ãc\jt8KxJ#J)R<x J'~œq)S> endobj 2450 0 obj << /Type /ObjStm /N 1 /First 7 /Length 102 /Filter /FlateDecode >> stream x321T0ಱ,HUwN,IOHLO-V0224V0P/-̃@sA| 7?%U?8 51$3?O Ύ u! endstream endobj 2452 0 obj << /Type /XRef /Index [0 2453] /Size 2453 /W [1 3 1] /Root 2449 0 R /Info 2451 0 R /ID [<42F317A67B2E17B9F1A9E873D8F0C663> <42F317A67B2E17B9F1A9E873D8F0C663>] /Length 6035 /Filter /FlateDecode >> stream x%y`e}vM4٤צizM$ѤIIڦIiڴI≠(8 ⁊9( 2?APQ( ?<3N*JN)Rt*S3fl,:`өt[+c,f O2zX\blP c{Y\ E32(fc],@E)X:Y-eJb,`U:r702VaX+dl562v&7tv0:vXgPX3c;XTX=aLy3cƢx cӷԍncLʢ݌cLWWwv'cmfQbl1T y/cȢYt7i3o=Z>]-2Ń k.40X7ckY<C2bLf,Ȧf01ƴϫYɘVjJa&V1u<i'X<'aNi3&*M/l~^( 쇽+T́X NC7DiRI1pH&_86!a6:ao$즄cxB %jB =H(HBs*е&3 }N(zBڞ)hHOiI Mv}C2MM,1c? gf'_pFJ8%&UQM+e!zݹgHW'&U;3kB! dfQwSIm?n:ƣu@#*L&:&S@ z,P%.p K(\tΰ|n,H2K  B,b(R(rP UV*X k հj@ I ?6Fa l:-nz  a@+A;= >5Pת>^a!#0Z0``&``f8 8 sp N<,"ЃMA@zЃ =CSr)6>S|%l\ gj>&5cpHA֤.Pߗt(C P@vXPdR=b*PJT*R T@*Ц@ߍ6P@:Сt(C O@}P'> O@}P'> O/} K@_З%%- X*U J0L&uRC4'9 hN@sМ&4 ( &4 (M@iJ}S` 2'ۖȶ`ئaض 'ö2MmLf}m`5,TsVPcR?`3lP`;h noI]zLߨa7t@'.膽z~p ! 80000aNISpa]6%x]Ǥ^c]@~_`݌I}\_G\@\@\*K.K.Kn5 n]v%n]v%n]v%n]vۥMn z#s]2w%i.Q]w %x]w %x] n?#"<; T.$;dRwG|_ .=pK\w'MTť 8#I=Q/5]4#-؉Q1ku(HDA" QD$ (HQqRJD7"эnDt#݈FD7"N 'BI!k'FD4"'ZD"N ݈8)DT%!G!"FD4"'nD"' ;" 7;KӅƴѱ"[ %P eP* VN?V^XͿ?]|˟ĨkV8Uתa4Nh&h]:@t^= A? A8a Qc0 c00 S0 3LÜ@5f?'aNiXzTa*0tJ:L%SIH9f}g~O[£Q4&gW\#0c&]X Vf˙nIo93#T=RH#UT=RH#UT=RHcxꑪ{#_|=#_|=#_|=#_|=A-GI{$푴I{D՛Y@ꑪG^Iw\Svq{@v d=zcc#W鑹NT=rzAmW4I#UT=RH;TR&`ܛ8Y8qR5XՍ<&c2<&c2<&c2<&c2<&*%OyL1 4^} dyL1dyL1dyL1dyL1\[)z1|L1$tL1I$tL1I$t̑szi\dyL1dna^ > ek >&c >&c >&c >&c >&3P  ȃ%PKA1@)A9Tr*X+a>jX 5Τ] ֛eFa l:ۡv@Fhf- m:atA7@3]kza4C\}oҷ[_w`MHOCO Y8'aNiXY} 'x} 'x} 'x} 'x} _n+ʤ>A|*S T>A-zO>O>O>O>O>O>ITdOҾ[5Yr @5}A/O>O>OQPO>O>O>gXCF Y,d @>@!,"XPPƩ=YIb%5 Xeg%P6& [`+6( as,?'ea7t@'X 几l?e@V d)/{`';ޓ1gf;{8+_ԝ3Ӡ798aEYr ^Q">G9|s&Rr_|s#\9Tg;u)«ٱ.; ^W/\Xo޸*@xz-bV㭧K+{vxG \[XZxW{#+ykE@2W}_op>> |qm1fu |p~@}''\q>](O|o[ϳB7W\!zpU—ؓ_|\*5dMLh7^/}|I&_~wHN !aۄ7squsySx?%ѵ~>}f1 GZ< 'o$aZ[?wN ׄ\(=,|?pf=aj?Îp#{F\NfES{7m<_~Mr_egA5d>GlT`yD&Y9&Y~crT^vwy.O!l<?Ӝ}@y#'r?e}g4b?>+R>翖s}o wD{qbĞΓKqE>Sݘ4o,;"Yd=GY} fs&%ϤuƝcc;l2Vvq&'_/aBcB6ȱY*-YX&әLI#&S*GcLL|d#S!JY)SN6QYY3.}4n2KMѤ\g}4e27]M}4c2>CG&|]0yuI>3ym)7IrMUM]堗J~ˮë+y#0pZ-uU V&`f@o;$HtNӠ͑z7L$C >$C >$C >$C >$C >$C >$C >$C @ yզxd X@,|(BX E J ʡC%T X `5,TZuP alMVma4Nh&h]:@t^= A? A8a Qc0 c00 S0 3pf98a`^/#Eyw?̣R endstream endobj startxref 442605 %%EOF gpml-matlab-v3.5-2014-12-08/doc/f7.gif000644 000765 000024 00000074517 12075730247 016656 0ustar00carlstaff000000 000000 GIF89a )))85,,583338882821,0 w??DDDWWWIKUfff`eovvwPjl[\[..8448Xwu.OOqqDDVV^^wwwwffai7^xj7?)EQVFfwrmqy|pXDULeiv{hC22DDUUJJeehhbgTg679(/99V7Cv5Rvv~~mmDPIUVYDLU`YdUpfpitwwryciFFGWi?_4w|qDGTYJeizP)v~mQ\Jakq{`^I̓ڰꎤ孕ѮƘōٺ͙ͬ͟Ӕѓ̪ڰ܉⫱ˈ˱!, H*\ȰÇ#JHŋ3jȱǏ CIɓ(S\ɲ˗0cʜI͛8sɳϟ@ JѣH*]ʴӧPJJիXjʵׯ`>}'سhӎEڷpPܻ֭xR[ B+ N{;<xR)Q7 A!@ˇ %hY Q >Ȝe{` '@ ztԣ[^]zۿw]<Q ??;x׿{D0t }P{,@vހWt u@\x&PA Px{3!޵X @ PGY@48FPBPg%NQHExT FQ;~Qm U-":@6<<@i%A=1!SPQ%0‹FzE rHX6TkSA\lDֲj?򄏘"9O "FddQP `P[%șEIX%X[EOcN#YGǐ8!8$ Ɠ'T k _r{  \WTE$> BrD>1aD|+YLԁ3J$ST s,^&_Lגz!3! ɇTH &@88Z E%ȅKZH+eB c!<\Jdُ&kI=b~eF" SOHH"$59QD4p$D(GR$T.)'FҎ=r&P\JQo3P[ (Ђ.3&D /0Nm %т =8&#I4X@ *?t(LX{@5aiB^r!BKd*_{XK*tG 9IВ"|B͇p!ԩDfD#`n6⇒ I8bԑ"ՠZ\H:veS?޹^{ئC F<+;G4pGF|]<2&k5H JR.Sr ]Cr֏Vťy .ڐ{\5!=g8v];e\D)Gnِj q[І&)mM* _H5'=HREҞ#ьB$Bn ihLΦ07"~$nMxّ3C'.HU]c a"0B^484oBڪ\D)iȕ(Nxk.xV+E4C"1\V&N~x+iQn`A!~#lH ߻Ӫ5P(~|\~dǓ"3b !Ë }&`H+]XΕ&frL鄧kyi@t!nA1惸#A0MY5x7!j15@ָoEbiLdi"+>gbd,ȵumk]#Tp"UW!3F3zMnHiSs/}YCZ-<3 N1|CAq{P$p+66#HNLB !'dZ{$ G4dE(~ A(fX YO*߹7OޑgI#r&4%搲A{$x:BQxF}w G#Kt[!0sr<됧59B+HSːrt 3A0?}tW 6Dz" YF]L˟=i|?컁Ǘ9c%QZC=/H8F+]gW AE#H~πk[ ڕr烴 +pRWeh E  | wp7'n@~V|^Gv%61<5B^pdMu}At`tЀ[‚'7 Xk#P _M pP{@tC]Ig~Qȧ|ߦM1Bp(te1z 1~i jC!| Hx灀{YAT!5WĀq D @t>h{DH!|@挩QHS%k9D1s7MQIWhwlz @u֓VWCx|mU 0@!)#qcggsHK)ZP^n=BiHP{ H@D(1I D [T "YJArnVWaFЅzVc]&π~ A?x 2 ߰C |{ GB)Y) Cwg~RepMф1٢)Z2,!w2J&̩P0aAhXt FgiI- I/!bOq HD6wǖ C  QZ@@Htxƶ[ʥYU\.rf΀hϐH1yI@|*H@ )! Χ앩V!1v:*@ ;;p0*:˰H}l!PAa-J󊘓vqMKR` X޲TD*I|N 8:`  < HPA E(p4.z:IAXyZIGFC n9;`;A`[ ڲ0˲25[OVqNWA n0pu}@17p  6` G{G KЏaGRQ31k{%7Z^iVMu Kۺ {DPFa6WpS~ik/ >R?#TU18aI0@F  6 A6Pq;ꉮ?i^D`{۹p6p˼Tp 9cA@ѽV;6:jWP˿-ۿ2,pv0 /P`N~yOx:q([+ 8`?;v0U(Kwh<[<D ^ `;1N{l[<'«h9TA a]py I ̺>P : 6`J,Fă3d{dZŬ,Q cL])D4v'[Г}z;~  Px<Fw[M+I]L\c ΐА~ʴ &`))&P ( =EwKW~-Ȱ MEHX\1 fe?C 0Л@% q>0 qQT| @ w /a{P;WqkOCK'`AQ Q)P !mi9^b aD J] P(V=}RehSGi 4ˣGY<0a njpP +h)~`qanAR$FNnjȺ˽`;- v}L )`IڋrWʿ=({Sp`,S$f_7It 0;!l%  <0=C q Ѩ /7 п?A~Rq`Q!+)}~ o<Pp׀ Qzz042yP=8N= E}{fM!lQ3@Z7`m& A@<3)^mN)a@C.p qJ:p]QNRXY}) +ڐt wۻ  i]qlO սk9=p G/.@tz A1p\ Dиj  |P Q &X!! "/ A3c!^ J% AAmܚg`Im,EݿԣR9KAa^t !;`^@+U> '! !?B>&LcP aL~K2/4n6`[ܝP  |=CN?v6|J>at_*Q[ {Cͻ ZQ!TR1B}|yw _A~p$B >x >vD߶ (B6]f@$J-{̆ fSΝ@0P'p)TR01UjC)2Cð ~H`! lSbp7C!j #g"{nbM ~(ϜB+@P|YRRV/A~z j`=W]d)h KU?{A|HB94i&:nK ƟfwC C3 Ak?m>| T)AJ`"Wb Ew*=ZkI.C㎧! ky3R c G!Rq]l#F?3Ck F0r .~*3Q^PA1ެ;/ pG0p;(;@0=}~s|cg8,~ }seᄕt@HPmZX (zEbc`pҪGtuZUA>vs(e j!R?{ {C<l@ D@3 y<*tqɋ45B̫aDO*")WÅ= {P€sB @`c5X {'L2ԄUB$9(}P6>Ms{~0#7(^|CDhĩ`•hh }CphKhӁF"n-9Ar <M;Fums$Dt` vwFl YDtKċe %C4$:XHH:`" wpG#y~dž0HjBwG_@f8} j% OIсy$U K(zA:yl|0PH _ɹ}Ka bx`{d X\p;@B.tLПʢP}@qWhfpN:]`H[J]KK )('~(2hg`[~X? }p?"0 mA!K3'F,\ׄM | MLpIP~ AT ?HPPPPXLh@{PH\6`}O$[$֔MMɀB̹(lc>XX} D`D%#"(ӁE2#0| Wg~ @{0]hr`Ѩ =8Ed؄|ːO@J-M4}SL(Ŗ/h8!8pz*e@,=1S6mESE8U9STE\bxHpnyLpRS !} }Fgρi08S;E?TT-U0URU]U:}Օ)<ʆ6x)ȔI`ĖUUW`FT@ǁkeHCҁ5lwQ=Us%W4Utuݕլ|0zᴐs5ZC9Bf `q@܏v؃M22$ nI-WqYrZ =:u 1XD\0҆p|hШ ځ#  PV؇|H>6H|/(P[MI [R-WR%[7R0A'dە`B[X#zeG8408E0ˁ!I04oX/ Y%ٻLѵ}EߔDՀǮ$퇆ۡ@a M" Tpn VAC؃O=X^p`vvЍ V}Ke:!@^@;mGp6Mp$Q' bh6p0"~a͛pf%n`{J>sF^qK@r0]pkZ@iceEEpJx||6Ńn\pEf=fbr04;=]g܇3PȇC8͇FY`HXR5#HƇ؇nubޏ~>5/P_"1xˆvond`h`諞d07ɂ UhF^tnBE8>sh7pROr+nzPᾎdR*чehF>8iF/PHgMڨz表jjIdS 71`KvF=np)ƹ\R y;҇EP݊2`u`Zp 𺇧rͥ^Yp#fEh-WfPcƷVb m 툠}Cp\-0dz'K`mW~n jV  Zp{Y<@/_ZEAzּ+~̙mOh?8q`^@mi"nl p0w@vb؃\JE]x )J:nxz[q47zl_p?4-O $fU l&jj8dŸf u([kg6(xul$>c'cordef %o82(_Xqi2togPb|Ou>GM&gON֫Nh'h́y0wHkw (~G,nr|L?=dzWu8ADE pPUpY]b''|M}yyOJ#8 0hͨlQ$a%du~phq`@#8]yj}@C-L'&ҘQ0D'x'˘~x hnѸıY*|n"vio󹢕H{\n;v`uY(lC „mjicˆRL"0`7rG\@VL@ Kh1KK8 9ԏBJ0(R$mT(-(:B3a;@!<(j[EqIQ#*T(6#Ȩ=Hίm1HܓY ǒ$H07sy߃w!@@dESǴ5R^o,&Eb}u}H}<(;*XH EQ B~!ׇXB^dT Ĝ շʱ֑cOX2I͝o%>MBLO%Fj R -R{]w,ArW@bOBі8H!&+ (,QV5b;xP2{v:fяaE%I|!laLf~ p`?M@-T 儛9N =i!{puP3\+@C+4IpPd(BPhSA@!ŕN2ϔHO8U@~\%5GR-OYQ9d:$;pG6wJգI*P<>%3tDzAĝyzfBoE◙]KT'O+GOBiJ?@26@\+ˊΥDeY@ #X PNц sNADRYzY -%ҝjB Fk{w #? HeԊp; 'BAfPQHsBeMװI;l-MIdqt$2uxϵ>*b +ހDe0eIF=N:{ {İ=9=oGOMJ=g+y`fIIXm0 AH<ճ l vЁ v#w'KCL1 |xAGf8p!Gw=}4~WԤRD 01# "8Ez8Ȅ M$,p}H,"Cg!p_B3)&_h هf=yi7UFL.n+EQ9 p` zPdVn!x92G ^8Br%:j(1CC`r-,/-i2t&I͑yT &Ik16x9G(#L aRRiuVe;H9˃h#aSYG L Վb`q(F1:E:^Ew#J>E1gA wV=@:bB!s_EP ~pppd-:3"C˃TJWOB\t5EɫD +܄Y5;0"A&ȜALavn==tl@o |Uf#JI9uBV?2s',毲# p\FQ{V抭BefٍpЃDtAz{RcU$#0 j*F[?t@ H3q z&-lHG6d-\4pTZ!&ӏX a[v@OhWxA Rƪ`q3miI׻"Š ycg9g~<;D}|cqYy*Є Lh"%`CQE*~ R0gĈ[D2!-IPb֔#^h _ZD)*iΛ7|6!*XV C(ՊH0L@Vr Yv?nH g?~%uƝr?XU7k wo򶴽.sJ9˫q7)~>xb?dW JO>@.ɨh|HdA,D!#}wd@ksyQ E8iW76m$ȩS*PRv.cWF{ G> qݜ Hڝņc?g1<5-L_Atg1O-nQ}d^^), W 8X)}Y1^ !AC0dB,;4AAՐuVy I zIƣ9_~ !aOT5 `~ VR8iuhQ&@aC @HD $?(H#DÒd\` ̂\-G@b@ !AL=Tm|S7A8"@& 0%5\&rbF%\,>ʵA3bSb㯠 f/СHA+^S-8->Ə$yW? ؄E@^C:<Aă `SE[&C'@@u HB_[G!X=8|>n)Ov5-0dR;_GTA0#0E*@ A\C p#E@@JRJ,&,&H4h&2L`R%QEQRJ2JQ@%/Dxx1>UD-ģG8#UD!-DP$5AxC\&';=D&B穌G~SM[5H,)A}"D҉edn}Sg{zh|(}[CAV-9B@#D"-mZjJ7XG[5(B@z Ąer_&eo50A\%yĊ(yF`}.h!1!?DDD-D/Ι;BEFr5&&llJ('fB6NCqBUB (%n&ESWq/IP4 vhARwIݠ8 耮m)"gGo?l]CI B:S?h?+BHV.kf{8qP-H:04xq<@|񫌱^1#Ex^#\ A )DjGx TY\4C+/AC D-Aeya{4d.B(2,rP2I/B-_ϐAX @,%ĕF\t0 `&ܺB Ax9,eSe2>خm.jѤ)urdqPNATd]] ,Up(C Ct% 6*_$@Gש@ 鉀 L4r:H%IV&gE'MMi5kL+G3P?Pr)w%p%@tSUmP5C@¿!B OA 'AuIu%qrzqbi>uphJ$dN:x-kAhv u?ɫP<@zHM4jKA/duC@l։>6X *s0Hs{'9 -, ''C,Æ>̱!GLIbGDame;3I @QA u7|GÀ TciPĀ:BxLS'EsFP;A:W;ā1ACI-.ñS9WaACaA||ȃ<{|ϋ<}?X=<%HcF0X zQ@QXg(R3zSzBS'Bk+)Dk m!>wC8H &*(~whīS؛ƭoDV( 0, XGo*oؤrq"$MbB^Ë' 3p? l/_G|,#EOSʾ˾>T9@]J&TaN  1fԸcG|̖kӯ 4JS M $C觃TH $FB'oz4L#0h\plVg~%m ;n]{iAXZh EFY$v 1b'Ѝ>FW*7N1LId&)~~Ə~ M6ph1FW`Ӳ+W[flWܜk֚Gv #}ĴHL'PydvȧulLȟ'i''*(IլF5؃:ZZmU.[ɔs|\M#:(FSi3[yC РQ5 }y|4I'S5SBu,%Z'7u0lluW5Yj$V Yقu:HG'yBa1ڶtÅL$Z4+y`S\z2"}@֞g~rE!)p }[!3X`Vf% Wɦ׭m*I?IPdX |•P!2h7Å@~Hh<E:$~~ACZư[R^jYub,a%]!u̙bFd(GL֫p<p2)n}"jkt#Lm.Ch$C[PV_תtէ:#' 3})bt8[6ƿwCW&KԚ: :J<z֓#Mm,Wֶm{D1(|4hmbVBU=0= ¸fn%D40SC+4WFJ+_7!㗲>T-S-a[+]Ko{րJVcd9_nc=Aj[9GMUB06F1ڠL ;eߞ* ]CtLL#`p0(O2m#aL $b(2Яf6B.l!*σ* e]܁A!OLTG!N#T!0Y$b@ka #Ta0Laol2/2 4tP̳k<`f#"CI㈔۞00 ]os#a$%<#J b8D#$՘ d-H! %γ#\q@%cnA d  % )QN/! $]C!(@ 3b!#(oO F 0A4KpTV('VHDDD!w1"!Ax>bT #=0JNNh# 0d=#a6%ʢOt{KQJXhG>Mb*_#[ @?X@7R.4.J" @A'C@#POL#PC N#EQpME1JJcF?ZnVi@bДT! m87"r.8rJY"t2&T+?bvh4t!rR#ڑ +DʯM\(R#V!2A<lTt4- ##9N+2`U2@YT!IuZR2B"O#BB91Bx|:U@S#sU[H6 5a `XX{bYIEA+f&7"l#v6BA9O;1 T  S V_ie` a;A85jbC U5! . !c6. @ -MڂAS=>BL ,#%582#9DMM'/!" *&h4>A (ru !Pm_zpmi f-]Q5"Z2lUludc,`Z226ʄ6eΕTR`g"-SpA2N P"*RWbiGx6~a%C{{` kXWc{8<mb4B 0DԄ-3 %j6z4Mt7BHtz≌#sug%,BbIlQ8vl<$`,>%)0t#Fo!#HA4N#;O8E&OHab,Q/Ƙ{Em=}ɶVl،AiW#XJ6 V!D_'ށF`HF#HD2~`ohK Z/!?);W#gom65L؞S͸]C%?v_AYj6~.#3]SBk5r`0/5csG¸JX2bsji#'b[;*&SYu'+Xb7R#X*ox*+[L;@7#;S!6eqv)9Sֈ5b.aaR'^ ԡH1NY+ֵ\Ӌt=}# wMQEJ8;V2Əxboutv٠*Zl Lt*]yk v!$A]2uԗ"3b[!T1"y2"VX#(27:(Z(7e v<u!g~ x>2dt37Da#|F#I{!:!VUU[!E&dx|*S`smhAmFn}U zJ gMB\MG,uBbBu2EEA"OIrABqdfP.xBG~hIqB ( Q$F+cDG^FsEIEB'C^A}WAqZAߐ*RdP.`2(Ej ;BE=!-MQmJ5%IAd ^!d>3ݨYAx9U*EL*ŲRBFgTVJ9) ^ϕ@I(>xYQ7Ty sQ'Mx{бM)e(§p@=:i&R2 MGSAk9=. ;/Z;!OA˛ZբuCcAǭ:4|e b^h'>RXr?;RD# R7MqQU42!V#@BGt0`eFahAuD3VᲺ#F^ڗA7_?2\"ggTd||KG6)QDLLr`PDVt$ G%U)HJDF~GvXsv:qAnEM 5E!D~Y} FրLf*ԙv$*YȄ ABBHEN,h,er'ɯUA3* @ga)Rq9@4PB͡h?PkbD< J6F@B AHH: ]$ 8?rӶ 3"O'(5QkhҌJ.dFW)U,$;5)8z?2~ 6#ZCD'xͼ80i>v*x.K,P@mf"] JHuwEBy^O AC,., Zn{׊՜bFzG._A;@y"Dzp"vG}]7ǢYgh)Q4M߫Yhna|3|XP Q W1x@)Ta$PC7R96+R}Plq fm'A KT_p|2:(s'a 1ENu54gaevx.Ni~0xXF$3Ȇ87h @s=ȃ@xp` `q%pn&GNR4eTxPL0Ximȉga-!998txs O@ A @a}( FC-#PKg+?@XdE;o<3Z8@ 0 p D^!=Fbش}a('Ǒr~ vE2/&cpV70 c HfwU0*(@/9-2-R `2i\@Y;YFf?! %UDB7FEYC@a! ?V9 g ILpɕNٔJtyv9zyrٗxo ɗFIY~i)u uy9` ' Psx|qЙ<q1 ȍG@Dˠ)D_4!6)0JnFW>9 nXr.haGifi_ɝ "Yy\쨚Ɋp@ (y7yO[~Fa% hFy^)l+#r?cB|hfxՃI!ڝwXx9` py1*7gZ(D%Gz%` v19buAU:`٩Iɑ胿|0$FXA>6MB!MqWܠTkZbZuf: %*)oƏG}*F ?sb $5/zU :e MJ=B}CBN1 :PFu~픛[@ Q;lډly_YJA3:ɑAڭ! 6kSilqoњV!8f ʝ ;aAFa$`NC@YZPe2Hiʲzj ӧМgbAA "h` [D0NU@: QۉRf:zpG?ob{QAA:J 8Ѩt YY |бUK;ٯ;zCиA 1 t;0*tQ q Ԯr9G!0-g#"z@_ hޠ8P#GnRK Ѵq  Ae 4;+ k9BA@˭J` 7`:H[H5ozgT+[w]F1?B DkU6P;ř%|; /rNf3xХ5u=|@B\ͰzAk[, A]us34Dphu/r < I X.K-FQA`&gJ+(!k%Dn Dž E%!O!T=[z(IlzJҨFAÐliuYͿ ,gA׷3Lˆ JL(h0aOli`<ο|\`@N|`J"t ʲ; cV ) mpr+\"Gţ>p 6r6e{h\^O[Q+0-s}C-0HP D@EK`G~* ?h@!҉X -]QF1 1!D}D р֐li22,~{PM_%F@F gmjr+o_:,a$ lG9-(L H".^UB/QA ?~(zap!@QAo3,uj{΍Ywݟ^\; hA.UGQ@>r3' -~3؍! 1b# ϾRPQHJ-rnb@DnN000^ ? NN5{RA,k'.^y.A pg1p+ @ ~f͜W}^h1B_ n kFw#}P@nBq!kE_ ۀ 1 %f ;>\;ñ☆ ka㭅_앏. .WB ^Wa 5 P=,+? `.o0_ 0DohYg/%/rEpg /OVAoކZ>OWA >kDPCÅ֪+]ѣg>tvhdcNF$J1eΤY'lRW`BhD-:gOԠ;En=H a3'8'Η\re6ZhΥifAPEe)ݕQCĕߣG|g#H+Z LY(6^y'gP|y&P'\akJG a gle-#h0[,V)atfZg9@_^vB /{D{f, 0:_=(w&Hy9oQ ged˽˸24&Nt^kpjJ(g*@ilsK0cfyƾgqG@t(0&UʂqJb ;,t2Y2q. J<9=iü̚)HKD"ljC}J4 #"ԋL.*p"y`=Qb}TzFeO1)&A2G(h)Qg!حPGR1TvYfJ 2Px!VEYe.Tv)׹ֻ( ^AàI=:Y~ ,82bGwm !G꣝eu w)w.Z)<uY͂I*4}jvHid%m$b)6BÖz̡BIh宻k#"Gahy&ܥ'~ܚ>rh 'xb )hŗ)ͦ&Ӿm~ߖ.n)dNhF`]fFD Fֶ&ù_#'f<0"|G 7 Ԇh5+~EwTIƦ7ez+WCrs#^|OFH H1eZ6.n1<(Dq^wn)4j&z/h5|B*l'nL>Prs?j.&0CB$P{ !C#dˌ;!/׸}*H5b0DRx&>b'E=?yJW)/yÓ|1Q&d (A= 4&C1GFH+?1jt t35lCR>`RdPEd&JN,fM)MawJUE^J@2uMyDZvSEB8lU'G2&Sz%PҜ&iNMDt{kc3Ls~UfײFӝnIk 4R0WjZCG PeNz,in beNnMζccV.E+50V!duV6t.8>;|:VQj#r׃ղ~]e/{vlg:|XWJ_XAA ^vo|3_oOЁXGp[Ađ epX^:0M;=̋c#sAv _15[U1] Lt}- gd9˶:Dc\ j&U~цHrQL@stR;LFbdaVQgÿUMfڽLh2 h4L\BȲFSG4;j'jjü#aCӍQ;U50"64cb֩4Wlh?$Åټdw!ua2ho73Q39mO#xlyr&dv |w j-8PDaX&F^odz\i @7[5Q3q}CJ3ޏ\h(Yo<;(UM1G ͛nL+M0@uip0@^9cBX9cw֛Kgs7׆$];%(!žw6xRg~J0^7ۙ{_Wbdx@A@oDŤgxJ| {M#NHSp)@@`oW?ٟ_G?_?? +P?C@?@K@C9d9CKC=ÑK;>,DCL3>DDDWWWSMMfffkilwwvXccZYY55666Vvr-EEUUSSvvffSNEEUUMMvvww}}ffff-b|q6?)DJQNVEfw~qmPqxzpsaDJTZSfvypcZ..DDUUJJffde66737Vv9JwwDPIUVYCKVafphrwwqxchWGV3èe?pvV0w|lCKSYIfkS)mvwtQ[MbroWfʓڳƾƙɎٺͭ蚙Ӊͫڋ㫱щDZ!, H*\ȰÇ#JHŋ3jȱǏ CIɓ(S\ɲ˗0cʜI͛8sɳϟ@ JѣH*]ʴӧPJJիXjʵׯ`ÊNhӪmNkʝ3YtݫrKo LX#?.̸1WjP#HUŎCݻ?XHW^ ϒp\zx y:֫ONһ[;޷[OA߷' u xӱg{7`vQ7z|P~ȟxww j`yX!7`d68(`PT@TP @z5Gvax]F. A`^Vg@P=OXBPs &Wu`Ep8Y9kHL)H t@OAdR&!hJi6'&w*DZꪬw<8` joX\"AvH?HJG҄??(`SPo@s%|F[@K` ;.{>N$ a6LO Jڈ@?d&m7Щ@I?d_US1,P8+#6NxO #J B<{ne:xӋH.=Y#PC4< E+@t[A%|R<$@t?7P0a 1@D3 ~EaіOG #xʣt@.xbP'%,?7KRč?D鷿zDgfK/|"4PRxZ=<$0nTptuCaHq#Q(FHB3w$E'DEF8=t A2AC%h@CȌ ^vЍz4F@)A,#lx@z 5*P@t+K`3h ?,"tG,$G&0phaL*>XC 2:ۻ}|E }ŷXar),&x# \bt:H 'm^a|vf|͗W|uftޔ GVP&dE-`{og' K0 9 @Zh 1qaQɗV %hpp % 0B vAvo" 0pvp Zs  m0[bQ ͷ|PPQ`ZS rV"0{[:? M0 2tͰX?8fWWqj$x8?b@ P@) D$3]={J4  i |.>x|؄0J73'xh+dhp ZW(@@ꈎχB(`&iPNk6jlDD9`p?G fӰL^ l |K71i Z0?&"dF0 $0Kb @qA(N GPE }px[Ѵxi;1!}WyIc:&CqrgPP O hF? >:x@m k`bÇv uC9J|!5}9C38 H@^@ dpU! ' 0 v|pSH嘎xyGIz!)G``& A RIEȴ Йq2!CsgFu ( CFעh~[uw7p :CPPg xj^JtAa戠< rEc`刐"Yl4pWwgD e1 Ѡp t>? 0_[a Q2hn i \] OQ x Rg ppڣȦnN[Cjz |*(cx=dX(p16`I@H4X W ҶXpP 8ѠGZQ%x\f7R  i@f*aFlnbԠ4VY"(`<BV+Kc0TJ7"  Tx$mƪ hbvt[H<*g;P PV ӕ@(9K@ :Y@bW Z@+Ѿ 3Hs&I˵+?(`<O zdoZ# @doc) DO1X}%G4E? o 0]Ѫ,/^\!3!͈?+GN&A a{L\b14 (u[Fi=ю;Q} lA `rt 7I0=)j񫻗m}Jp@g\plԡ]aVa`pPS'Lֺ۽mY˄K L-*VJT15ol10, \r&_2۾=]2IpY P@ ~-)ȹ}(xhkqJ5!M]tjX k-"U9x:/I0o P! k@m6qPm[ ׍J+P@K ~e܍Z|4:q>fng~iΦj=Q!q(!h^h\d ;̥VE?Mi 6\h梺kN-=Ѱ npd*=Σb0v]r Cb=8\ `3͛柾n>@T~s=}(qg(CsߴCQT(Yl^&hm.;q-GmM}]&;ⵞ1Pf}8Mh0 ^~߬ >N81:ƩiK +pJ/hƉ B@9iQه@i_.BN} rS|]S*o{c]<\7e1f_q߱ z=~OPkp1X=(^>)_bQt+Aܳw:`3A ԰vd*pg{3Z_D ! bĝ QL2%Z멳:SsaDf[{lz/.-^ĘQF-"h%Mܰ!]!L5męSN=5r"69:ۊAU -:VU\4bBSH /3AT\E]Y*OkR@^2 FXb ئQ9b״򸊆M9vlꦱ)U*~0khD>lUr.\99iV%+ENgW*i-ϟ*$Hp')Ad1d×̙H;iNZp9  Тw*İ %n >RǛ%|''!QEa a$}ya8.2 ALjAL #2J)+., Pӟ37$`R@PBEP׎WfY'.pYI⧔+E={Hwa9R8")j[=R0]H^WUk7fYC~LA! od1"oYŸ٣Gr k(djt֎k'-'5 q#t̜Zig~P ?I$ŷ(\ uݫ!n:m' q*G.LBG"Y"Z(pm}"FbǍ74\quoH iC]_O< s Ĉ2l"MMPOO4}GH0*2U\TC֥BqD\ e!ql2 G B@X )446[|&D$D[UD;5 " Qυ-\ Cxiьn8>"DPa L4`E&$b6X_fFΥB22 EbxFN2R7< A@z@DBg49X7G$B`р#4$VÜEј_L$ٓwp, wu#EbYvy:FJZ9 *25B>hDrs !'/.\&O&\"2 Mfӡ2>Ԩ;B.$J]&ÔL}̩Kp8Z {C]8ц^ "sL1R̜0oU&c,<ÒiQai2HNSLUHVpu| W)hj^0;֋Դ8M#Yj$t@jT(3Qپ>!X1ra3v" %~Ԓav?0r<%ݩhF>x2m$@%#`ؐ[@GVfQs Ll@YH>AeƇ6 XCO9} `b>yy`';kQ1i],c1i|c:\?2 yHϐaS?#Ї^lx Ѐ- q9{kwL H^j3sk.:ǃ# +Vu<~m9A o6IΊ2 Vdx xbÚ,*jAl:>gD"vo܊ ՙȖQ e@3Q]lo y@ykJ `Q5n͑k\hL͐p\T?)TuW@ ]3 fsׅCQ S4݈mԈVh'! I`2wA_ u,JJ׵1=G^ʅ&ܨ H!nm3^bΛTS~zݓExpVؚ XV*)DŦCѻ?Y?[p#wXRԩ}(D'5A`T:B]Tfqgoz] l,R&HG _zEĊU Mɛ;x"`36R0ar50qЫGS[? ~C=NjAX1!fC_K6r89h3@P T?##۫qY*)#T/ '8"4"8[Ox-,ۋB)D@C-ix/k9*(`C*⪫($`JXL0)/d;+B@ C5P BA5錋@Cy,XAdQY?ZdWXĐ{%Ar(ZDPG#Р8􇰪eC! n<3 HL?dYɹp$񇵨+!*&ċPL%!f7A9=0h5ȅ30j4k3H3ĄԝwȂ`?Zl:bGlW8C{Hp~t*P.4$>4țT ʖq|08n6= )0D4pLKxB"kH YA¿[#XkFLHÈb,>* 39\+ ȰD(CQ/++!b@Ƞ]*MxĪHwɊrz8x LMOL Lq9=o*8aWNN2{L>+ LLV4Ono8<~1G飌)(BT z܇?HiK ݝt5O}PD QF ?ɊJ咖hNjȆX(!H٠(eJ((̈́8Ȅ 1(CB j,OuPF'j4B}06(Y8AȌohgp%#t ҝS-.sQ%SR5X#>]bZ`ˈM@J7|gpX |5`"5Q|lռa[3 dE97yPO蓬0Qh"0lLgwTeŋ?v  f%"YHG#%OV)NPh:%4hI)}%VWyxאW{; 򣉅dI>a1 Ą*T~@>e<= էbYاX̦&d| } q,ʪԛa Y9θ: YsX؋Z- ˨X XZڐXWwiD 5ZrXÍȆ!!R@`[mѨ})܏(Vey%ĭԽ]a G5hV <ڈUx|9"LIͳ,@EeE^feWKS>H9Bːȹl;zXouA c*| ސ~W݉9ͦLU 4^(+3?<8~? [uҳXNxhx']A;Mx}aE`^ h61 7Pˢ` ,8G#s^'87<6h66Ή V%,cAЀ-BY/@u.נd=Sc.`Ze%H:;%H7ȓ#xXΧTB|;XxAz{,h:Mfca`cZkbd 17C?0D8U,u!˲Y9%-8f3(`՟m` !ʘg51J\ J,м,e˳@G--d[- 6ב6a 9&U~< q#AIRM=FSE48,N#>be6iޑVi`i5i >^n'$1#=B Ji 09_^֝Kjk{E|d0-|h)*5Ýkepk%@֨ kkjl&&Etd~ #yzVh3hC!tHm8mи6pmnn}iV)Q@8ސpCU~@ )+ʝn%6Eo(o8o輅ledT=p*ߌ]E?ba.^HFpn8bg;iVz%ۦJ-W qK cxRR+R)}ȱ5PE~^gqj9pﲸTÎ0NV0CHLk*Y1C dad#ĈwMljzIm( 'up4p h(Ng)Qo؅%ˌ =ψq@pa@^h3b'b uX(i0Po2sQF@ݱ]ʘz;2V2dtl0SPde]FV=^gaMl6/!xoO˺xQx׈`xBj=}H-,v.o%.6ΈHp/^8b6ktN gGw8KxzWB«geq/myGBp+Pʦ0 sv5`* 1z :ԚNb` 3@̤f؏ T'!n @ ,)4ȇ\ܺ8{ppi NG ؉3.iN 8H3nFk@,a@ ;pͳF1:B?l4Oab,DK@岽wQGr㓮!#I3lS8٠  ?~D@avz|@{`Lڶy ցC'14YPdضWFnR\d0bːw.EyDTb ߔ˨!+I*'D*=)8E.ʀB>1F50 z0^Hjp̥*S8>̇rIyǬB[Q1+G$anb=4<*r KIY:΁(Q 0x]ﹽ촧*J@&!vTGMȺԄ&Әn@!Os#! S8ւ\ / 6ex N"$J=1SBhD؄#ֱZHq@| l 0D1L DV լ%A1\r3$&lwqQI0E82 $:V$'<zyӒD4~0?eTdRƥ% ,AQ; il#PE?ZZ: N.'c|Ё$CD$"A %04U;.P:{$Pj",|& j\@zF=Xe nP|ӰS!FgAbSI&Mf?n"Z2&sX3&S\ӢYY`AׁNo8=G\!NC@BiL(OF[oP==7'EQ"нvin]8! "";fu#pM@`MP@`JH=h !DJ rT'}' zo7@8fI> I2Gn ? rye/}@opKӠ@vm_{3y~ ={dVnJ 9C 0ڛĈ,L@ y [xB6- Z~?%x@_J^ tE܌E^=G٠ZdC@{5Ë`ݰD_Yh t h&h&Bux@xł'^ 6E E1ćO}9DP2` X39胮i<`X_D`xyqxBy-O A)C'!p]KDFE2/,D ?̝!x-_=(^HB$4=( &@7L& hN*L2"@.hh =s2&<$DFDFd AcDEВ!@/X,#DCZD ,ZA4]1p礄XD̯+/ SJ)DC8 V,<;\L$D~ȁ@Uъ`#$Y(i7| D<=E AB(meB+! (Jl"S1֥Aҽ:l?)(P.C DC8"D L;J@Do ^?d?lV/bvoRd*C 2 >@C$; ̳N2jlԭ%sKT+&\+d+A9dAMhI>(%uc\((Z9+OCX##(r[2S!~KuئAEܴ)D*PƭEp%n< 'Dh4F f]ٙ5Cd8MŰtAq?-22HPtKFqE4c˳ ?h(.54𰛅 83ؙ,G-@6 *4J D78CC ,rqˁ|fA"b;#B^ b d8k8* _yz{7jόqw>H,HPjB:ne64AP@ y6HVX)/5Q8t]-9?TCv86JTCAD4 Ŀ݃/=( (E @[ YFD.ޔKt o4.A ͑rD$:9[ۊr@CI ظ1?B' d}7Y!AD9-CH$蠂f0y?~K|}z0qVå; pz;_Ya_)'E3B#>P/tk{= a5-0R]cD't6;e~YӃAzHt L?BM$0 #oph[W~c~=~YN :#&^_jkCDȧEe 9h2 J3PP)๳<2D@'@5D@-@40l,g;/rrcG}FdI'Ezr8b`QJ(qF0>hQG*8ZOS;Myӧf5!V_˥E5|-nTF Do9ttN%DdV$NPtcl\;+Z$QQ#GL))ɏb:+뚵HB]w&:TV`r`mɅQk6C#x7Nw2^$@ N`FZP‰v<&.G-LÉz1 #7 3Cր3#8Râ[EtBh#hXFGR"Nab!Z@ <1iLG(YA0e ~DABjѧ?:6*C5Q{q$,8M9(ĢFģ%A UϒjQ9 A dfR`z>@9A:mP$lHϓ:ZngzHAbN(aG?'K'r'0|Fz a=H,D =4rKQx`0j|ɱT%hy`lPPL0(a0dՓT8a=@82&A  ~@GYfߑ7-I<j[wL"5*uMQwş}; YC4`(e (FS5?@hokJH8LZej) Hg PwNW˓wirVACv9~rZYaZ#~ ʣZfIè0cd2`c7а0̩ 1Xi%5԰Ɲ}(.oP I7G?+]18|$10AkP7z޵@=%)HrW#G1z>BpfH0?xc k NFWԗKPb0?A`^7,=4TJD2e`Oy0%;rdC1)cb|KiA0o,p!4$6ŘV\8#5S$DCD&$F|C4INuȓhDp|P=vHB)Єzi>1LL[d; b*+pEmI<щ50Zy!1C75Cݔ%˕Ilz 9 tbK~4ZkDhّ峖HD@]#i@gj292pE~7 GE@^)=e (D@؁y聂l#$lώhCmz ֫cL*F6 H({G 0,!EMbS(&H)E=cG6hG$T# 0/OFTV#؏TL+A ؘɆ1PLaJ@1( bE\9tV:"[%hZԨ"Q`9v8S|lJ~ȣa"G f)]M \\y$E5*6P Mf |mPFpyl3>'FmD$ ^\>ӑ1Quv@,bxeЄ8AR;"VmD^4%o X Tm ZB8S4cV|_.xq/D5`7I9a#Xr %B XFHR<=}; -GUüKA!Rd-p BdD 2>2 덏%'l&g#e=I¡򋾌`d_P.BPK $L*)y\g(xAE E$#q+LiwjAz~"1aߙ< `drEDVi{/ڀlE91=8X\(IԢ~0]"\%EbGB1C}JY(x6IBO A؄?jYeeB"햕'BObk8=\]'3mn7|j^aV#R!8wĎ°42M6G:А,oyO`a{D> 䫛>˵ |ݣ h :aS!  `VYWHCF"&@D4|#ZXMn%&(T~톯~@ݒo+8G'R&~ G`# K>N?rl4#r'ހ;"LĄIP 'l$+@"ī|@d`AP+m1&)$ $<ѭN$"0LPsR #z @iNg~$ !I~+,䁧ЏUF@ #j d.dLa" #taɪ(14$# 'M2%0f>̠p/XA#UOC $!@2Wv@PDd%;ɶQqF#$+ r ?$ER8ؠpF!.๾a.(\7l:Z0 jA`r>.2Md =32 5 2$1B-U3%Y3: ,1`RaF rr(lPL,Ni"R40JHa*3 5'xPgJB%&vj#PNR->AE>V6Ev$/`/iax^#ܣr^:=!l?`Ќ`*E䶠A A((lFMj>$UR-Y$##@ۊOl A@ةT <TZe0_wp,s=A3!!j%tt8+X#GMS$5-*H#" $TBOe !/kDRw!9y s1OFJ)g Ѐ\fOsU/EFu963KU5qPE)H5HUH`@`>D]dA@fG ~A-b;qbJ"xX fQM.B Y=zn+3R4zjZ?BZͪZ= UJR@8¸ABU@6azRLb(B_?=:`0:_LA*Ϯ.EhOHftab5#xG?Z_#L9PV+Xh/aJ`ZZ~ ,x(I*tJDA!& =Y4gH$$7ycԓ(5$k-=lk!A'NPokCb_#TmM`в # j{OYâ#c3j(:s8.C`tp~$8`$ #^7o4(L=B|a N!.aA(SuE\y=qt+zK5k<t%Dtb,8BҀ`ܬ[`b jco< qRSA*%Yp#Cqz+6M %$)8':W8%5OV;G #  @ؗpITL,^oӦwRp3WW3*"Yz9&v"%67PD8k~+r . TbT`'~a9dX$a \yȕ31JdA[7<x?djّkc8'S\;gy `|"`av[#P(MY1ȕbYmpäVoAPYNUO)q#,8Y%ي5X YC#%tS]İdUC_Q!BoEat2X`aiـYiF7 #8$c7<#XQ_θVa(TvU9waIzbao@v5sȚjD::{cW( aAh1.@ tmNBdK4#7!/.!Ȉ!ғO*w];yïqդzs" jꡔ1v$ l# *ᇃزɴUɊa3e6(iRRmQ bAb0wfK1EQa\a Ltsg.6[ő[:¯8YCtj*TB鐔Wnw2C6 LI `ͤ{ck_)e! >,") `v =(MUjz"2?2 il%1!ZmK @ݠ i4!\K;B.@C.H(#Hs$#>h#AV"ٍ݆9wf G]%OV無x]!O vj R̐'no*W|\!s*ig@!fG fRRB0Xaa~! 0"Rj O(RjgxT~MΙË+a}oATal9o;Bp7@a A~QB]DGwb1b!R-]XDu$z!@3ߑv~Ocis^~$|ҕ!v7UC'r0A ^!.TAA$m4Ÿ <0… :|Dz*ZxF=@1B+?B$ 4}J`& @$'HvG,WSI)nxE֭\ q0d˚= RC3r6[hkK0ehx $N#Eʱ Qn f,PEU?-1D#vO\ʬzPlٴkTЀo-ʵH׶iɥnMh)"sJ/S R&[-!=G=&y]mr;fa;AHB Lo%hQ \EЅ:(A+:lM (O'" ?qNBD $ t=KR#:!SB#Vj4)9 #%(iL` ^`C?|1RlC6*hdP> e^m\ꑇ_yR0 2NQa |rBN L@.!O (A,$@roA)A 2E:*r*_NA҄̃h5RLвdH<00 M8Bh AvAY "K\v ɝ #YIÕ!\y {<]ʈ;b!SdҐQKEK\$/I 3dpq<8@R `![92DPgB B@Q8٢y A:ِ|Scw!߁E "4FE@=I,`C-Dt:A|`Fq@:dyGJ `\};UQXBVh=ܚVU5N`#Q{*W+bXp,e/+YfֱMd7kYbֳ%-g?kЦv%"U6d/P2Ԗ|o B 8 yy.DH zLp B(JYYx]sR,oy%[68P@j­};i@~ BfE~ (>.ln+[w~zJ5F,Ko,X_#Օ@amH:̇Q2jE {ƭ:Cx. '.ܐ aP/x3``t :$xމtxf0iҕ"/}@4nOp(AeH8Oꬳ/ Ib^}G':`8PXB I#h9l XR!op :׼&x ?8Pz !o:a50ʰ$P= &C9]'=mlr*O5ЊAC?3LX%b]{N[ir2O Y#jV/ D5 H`@[f@#pPnpF 0 4ako50b-8QJ0O,T/MyJ4&'u`X0F$D>Qtn;g)Ḝu !8) D1`-挷~=g ir;7Ļ5ا//y:|e.Hdk:X L~|,*0iwsL~f=bK8 |hlתꇏwKYQ[T $߂o0]B~"yJFpLf3g4!nt" oS%h18[$]H&h ,Ep0`pFuq7GsYB(B'ͧG*w 0pw崅$w !|×Ih%X;>D' :}I ;h7eMAdPh6t0p# rAV#Tx)eh؇a(ń "+ 1 ; lo4(khGGc+ zT ͆] aHfB}CxhH|g: 7xR kXPh똂Xjp@}=& hUs&h툁Y yٰ Ph Ph@y)AoA4'`x's7#MQJhj,FBCu<* YdC=2̀ !"e+ [`qQ|>(t[̯,W`˹-1tQP0XkY4"T" |Yrδ Wyc@8YJ[JHH:,Ξg[R:`Ljo <*/ j, 9 JW }`' -1m3 4*nLa( D]YH]K;<%,=kp\HlEnEh <,M+˵>\(hSv@0K~\,\CW'9 jj@ yAlW,=ڢ Ӹ<6V; JP}ٝ aڦڹ]ڧ Б`BHp!?q C\Mڠ?͇#P=}k MGmEmE-p]VڏnzEK@դ۷=׽wpzjWJҬ;@>)*`}X#?m0pmE<B>BrۢFѬfk(@ mpH@xIPZ@W~7m[i4?nQv p怜B~],LZA7쇠jܵ鱳鱓g p)N%iDebe> :]?"fo&`îclPnbA-ru5nNj%bLT~jfYg \]P. !b-UgSiz~+pŐ)OTh OTv$@ p8<=0&U0@@@9>WUoZoYRS`_Obdfjihs/uwo@xv_~|7_Y5oM42_>p}/J+ROG ou__*fl/>Ԡfn.6F>fS3\/4oVׯ͕/FD>+vODU \D ]OB2tm ? .dC%B\oG!E`I)ZpQK)'RKK1uHp?3 xY襋9S,]B:)U$&[~BeBӅH &}{_0.,qݴ5;;(\w9w\ i*jĦfA@ʴaΔ[XS{QS-{re>E,|;yBB}޻zMz5TNz'K PpA8:>jp: QүÔ0!H Q$.!:ҺE)F:hFIDoI\2( Uh# P!!=2G'ݔ. T0AQфƶM#(>E!R ;gpml-matlab-v3.5-2014-12-08/doc/f4.gif000644 000765 000024 00000031525 12075730266 016644 0ustar00carlstaff000000 000000 GIF89a """***---...///000111333555888:::;;;<<<>>>???@@@AAADDDHHHIIIJJJMMMNNNOOOPPPQQQRRRSSSUUU[[[\\\]]]^^^```aaabbbcccdddfffggnhhhiiijjjkkklllmmmnnnpppssstttuuuvvvwwwxxxzzz~~~tteell --!!%% ""((//22114477==>>ccBBFFIIKKRRUU\\jjaacc``jjkkppppzz񃃃파읝롡뫫鹹!,A H*\ȰÇ#JHŋ3jȱǏ CIɓ(S\ɲ˗0cʜI͛8sɳϟ@ JѣH*]ʴӧPJJիXjʵW/HKٳhӪ]˶[ʝK] D˷߿fŚOÈ+^̸ǐ#7ec5_g}S^"E,/LSho? b.N8cNJi CW{:Cw~Ν5"Q81wn=u_\ ov#ki!`twg`|vE^F e<ߎWoK܊rM,krwqϾLx03ng,U#"l' k wHXO{E(\'Uṃ@V'p`qpMٜ<]p5<%1V7Y< R14q\ HA%͹dps@qwn{Hc 5 DϑX& uJX=t+ϘEJn}cg/D_?*0 CgN K#(II2@ %bF~Ͳcq.A$D&06q-<gV3g(1OyBK3九I"Ĉ'z/T:JtT+we`tz[ʄI#[%V6% UN+4p B-ښo&"W5H%rnJ|Mpcܭ)jJ٪ȩT:K d"cmh"i&=*$`5PQVy:{- ^@(lemMo0lU|[FvR _nC&5_ȎAc8K,zLSBZM\7tzD. \`EItH9D6ҦuM(qQE1 ȭt11&#Pڜ]ƫ* 4O !!$Fn6KTs;$=)cp@H*Mx  %(ۜv~;$yd&qF,ܑlwdM4t:pI&~JX3ԺǦ31/PдNq!1JvE Y=ۊ:QJsfo)vP'z4U֝Tq741I l>?>c71{wmc[! ͑kG&9iANBwBnm|o737}!|u95U 9D!w6qo:q5l'tǂ$bzqSO6qp" 1hģvۑ.!m13k8|*xAӄTH#Eb+!r`傋&Va?L ˜a &;0o/==LU9Wq^1!MbG))qrnAk"w lCaj`Q.fZU Uhk'+t !A3wU? V&bXʣ"!Lá烤GevQsX_Q jlӑ.qNzPSzea7# *'s0a!zs D#!/ fDէ wvQdfQH媿 Qqw73,~izb=H=U{&ecɚ*G)TFb; YէdgkI/ [Ikv\ =%Y0sPf2+A.vЬ팀jC.q?=n|QiM=uщܐ9k>A 4W"y]jR֑bfgχqKq7*<3nw{=4{E)R;]/ɺyyBՇ-ՊpǐaJ1 ?Q۷\0f<ЊŽ Լ:ι#3Oq9{6}f3Cń$3$lL5߽/<8F A2YR|#.V +؅mՓWEpQsǍ Aw}*ͽ0rѼU ƌ؇1'y߹Kaw (pܐ!0,~`Gaw atnE?Л ',e6M7]'*Dt~$7RoNmd_qCM; ?x2n,eo<ڝȊ 8p1a&QQ﷍szm݊!AP/sOPtmRab`!K!t%\7Q0bs4O>~/q$SꎁaK?>{\@ZmAUQLп 6a<mqEualv8o`Ѵ(_~ Pv7`#!^|}k)lh)}|qN9#C&ƕtar'&0S(}Vi!zț!`'POQLwf1Z%yN=ﶯoB O\"ZQL'CEݹDVpI@ DPB I (pL/#S"5~r MDRJXjA'P,:}TPE4pS"f"è4>2RG!QIX2M@sZ ]\uV^}QQ(R^\*re-̐)foSoOΥj6h j>BbjFcƄ/Cm =Grj??x+EEM;G!`ZqVޑ ]qտi&BJ?>r66$P tj N(<Њ4L!r/B 7롁+*).aj@B;ˌ&lB Cdǡ295b+8&1VF^M8CB,D. " Ŀ*B74̏"lʃҮOA=$J;X-<|;:Ǻ-JOl+CNGN84ЏZ,N$4Kˊ1R;]IuWa[ D(`mPjֱ(OU 6۠PVh".!< V1h6(S^kA$p> t_6X "r'Z`m)01w\ ߃L2Aa]@"XEرNbPҌ5ďL^c*ԡH:N=GD1@ޝ?a[*J~=zǶ`ZN.) :[JQ% x*AXB rĄB;@q6 ޹;o.2#)b`k}c:"enjέu"0*K ȏ(t!BQ >.oʓOP_.߾GOR [,?oN$ Ч!ܽNHu0@F*#L<0}栓 /!H[It 0> .!7^3 UsƐER~d;M`rLĕӐ'A$;Q %|y=,!D38p _JqE|` I(q sF6YnCn B<1+z0ɹpq*"t'sl"CDrC΁\ d3Y>$/ <#iI&B͉2$HX9hO\?$ !&كz”&(Ak4Љ9p~3 <שNy g;y3d;)za|4}CiІ&ed/c2?)ёZtH!zRT —J!L\S>U}=4$cB %S)ط` r$ I'Zq3LFy۩ W$+ *BX1P*H^Xd*^BYOd !J d QJ(e ~HՖFA} >bvwq4*B׆կA. oXW gNk(ҙ!؇ed .4SRի&$[-r\0Iu˩+3!~u=p+)n';F7! PY,ɄeEN'cMUlBїB>8I34<%SEV0L6=nƁLb>ֳW:%rRA2 ́VoʵF !2@r%?zX+9yÆiX432kM  A,ALPY6H&Zd1fu'zpllH;. Ѡf$2iI_X ʏLr\z6v,LNFF#ȋ ̨'Aٺ m :V.oHtJv[)& ?\2Qsݐ1#9'.@z ʲWOX=?F\!B-/3^ȩ#skZ Gm"Dk1$GZ-Ϝ]Lb*=#hw7V`XPtg ?-6 u4gO*74*}Y&^uW9'`*H.< JdE2{Ud:94ͳ̑\:9C u[d\ ~^#> g]_@l({9>CĊ;:? TZs*(5  4k 81#IA!2` 082<<d %.O.ɻ S K&\!N;d=Xr@%830r58F`sr$<2=48RC˘c#?̘qcrD c5  uBM,PGPL0 2|XDŹP ON{6E9DxJ׹ƂE9Ө 5[ 8|*P/ a /kY M{n@x%*kFЙB{|m<t8T9YD+ dGT.` '[H|T} M0g 4P9 Qŕ۱e%[Ʊh|#gGFC"CHEtXQJػ$/0ā0G xID="H (Nh4h+ʐ GܓĐ>N ZlP$HTEGS:U:)!96(Ӛ{8֒1V[IJ?ǰ<-\% @ 9`ԆϽpW(K&h-7:`{NVN5eK;$֩RaPk)8t]gkAWZN3<Čݞ]mU̘ EٝSA_}X5e޴OtGY٭NkO="@YXe%x,zIN8%Pԁ\<8}9W^KϮX)4Xۛ yN=x/Yΐ֣}ٍݙ̝Q d J+[hY}MM {u[ m;gE6@H40S\U՝ `MK ٚJXH!,3pE dYYW\t͈GH4O/˲(45¿Tq֭ԀA_ތqݺh%^PWΩ88 d{y7-VĥhAeP`ȁ#8vݚ`a_]ˠ7j:p0Y#>;ߧ3=^[){ik;8`a$6_[,K%T:(Ҳ<؁MS[kiS( Z=]9)1֋|Es֕^ЉdLx _[`pUt<бж=8. J dPMR P}Jϩ?l%YAIZ ,N>)1\)ϲhf/>TM qOP @X\~ЃЃHI(g}m[v>:;( ;reVaT44`+.в!n `g"><#F^!a\IP&.´U+Ѻ&Z.~xV  G |GrjCC0jBA#dh)+^M7"0hkR)huϨF)fZ)Rƞl6)~lͦNhmֈ@6D MDxEmG/Xk+ظbg` kgLf(Eo ,1 =G4gnQmAIpB汱ffP(^dЮ`F%ֻ"ｰeee2m!1&hA(ӭ $3{Kv( ?oYniF7 sЙᆸ6F`#W _ 7;0s?^˨ >c&^ gL__-qO6V ~#L0;gs`_ ]'@9Pi}qu.2ӱeF۝-8 :G7oߵDvP ?Iu H|!vwpǭ&2g[/A xPsX5f" pضtawa%vS泖j#V7'O.߈ҾIzP0X[7}ߋ" ةqHߘ{z wgt׈GFcxzx7(㋦'|7|S1ȇɇuHHg୮7/z`Gr&?$ ];oΠ{)H}(10πS.~}ťC]/ ?~d4=IߗϖgY){0~T@,h „ P'8"ƌ7r#Ȑ"G,iB (K3Q"^^Ҭi#=](d3B-j(ҤGU*ؒ&"E ݬZP=8qG9̤lQhעT-Fp >ȉ"hC܄dDЂ31䋏#˸D4Ep僙(.m4-~R#<Vm6ܺOSf"u7ʗ 3͛aQw%lk?˽Jo]G$ uo=IlƼj_ZOE7}UA *n$XU &aHrȐgGpMhS&8$7ه"G/ˆPZЅt't$3#(R)M~|$M:WdP!6$Y>9R0tY4wZ.d $IZ]BHuڹ`HI䝁 ^GקDs (jo}Vj@>ZTaNbr2)j)QV G*du(5z+*H+N8%hq.edN!-zmĮDkfhB&m.2 oޢnzл[ hFa$\ ;0K+lqC\O$k1(g1-31:L'r) sLt1\PKM4KA_œ 5a=6e}6i6m6q=7u}7y  b{ۉ@wV"80fz¸iVR/נ hyY9끮A_i v>{A;X"&Y |MRg}?rd}0o+BϞ~Լ5"z R}xu1\#3 q*g&3G8Á-!s>!(!F@=?<;.>@=@B?15DFCFHE;;IKH0AOM0KMKNNFNOMDA;GPROQPSURLHUVTCLWXVINQMZ\YKR]_\YS\ac_a^SXac`ceb^YefdfheX^il^ikhd__dlnkjnqnpmiekfdjrtqtvsolvxuipxzwsq{|y|~{vtnw~}~xxn{섅|~톈뉋쏑ꔖ헙陛뜞蜡럡磥槩骬歯豰嬰泵䷹滶⹻ν!Created with GIMP, H*\ȰÇ#JHŋ3jȱǏ CIɓ(S\ɲ˗0cʜI͛8sɳϟ@ JѣH*]ʴӧPJJիX,ׯ`ÊKٳhўzu*۷p}jݻxU+} LÈ+^̘08o5gE˘3k̹Ϟ[ xӨSNHװcc>12* Aͻ NqߙEs=أ pԲkνO{I˧/tbV%[>={F$@H2~ 6}4C @!DoLsC:A8(bxTx4]bC"C $E'ՌLS.(ӊ3i$F!N,A'TV Q@ : %R"?i eㅖ\QhG5u ?  ω64#9}29H㝐1/0Rs*p'59$nx>4+::j70B('jH6^`HV+hw҈ .[QJ^ݫ4&ނU^BO:EE,Q`߼KM5tp3p rh5)܈-]#O48ZXdƤ41Ś 0Ì-H1KBm#4,9-PÇ BSݦ4أ\v\5|*׉F3hـjF)swq4w{ gBQ ^aw Ҥ8҈w{P7:ӘI(CxsdzV ,:;`s5/BpxZ99Ǔ~M46c^'>[5.g4pi$ ~$~+ȾǺ5ֿ>@U ; hL9\#.U"P6,xA!(QTFQ =p%P s N!UfTPԈ^6H#JFD<1+8T6A%cjUαj&]Blڌꆦ1H0a sU08/FE f8DidKU`$5DZ#kh0 X)CZ%xL /IL;H2$ t#AL* %%6#F?VJ8,0e.FeMqΐn \^z;?ΩC<?!t>=ϩ0PxΌP uDX0F-# '@y5riH@,G}#!iKHqژ7GҍT>$Gi|@/̱Azss3B#c,S X"wX\H$TT#? / PGf$r_*t̘_1ئP #8P:.7^L٥"xct}P^Mh9-uXVdZ&2aY7I `6!.p" ($qQyLppsUجVI `#21i`c=^Փἒ0!`p0G1ġt}F4*Z5{-y+&}ApvuVKS밙2tڿ D<③b ^mںҡ4t@fq QDn_"Cra'DU! 9PHhʨ(M #GLލL@5ƒ1W9P13 D!*D dbh>H+!8^i b ixFAB1lb V:B!1!@x=_ D@b;i@0;{>bm; dv@h{i/"niIrBUpHLJY 1c G`$|i?.r0 8B؁px=rr$q| @Ks\=Hc.)r 2t=Hat|Ugd+T4fڄaM <8o%eƆ34q@P ޷u<>ą[yx @vKR :p`.$aZH?d509wYX=e[Ld*ٞzDX4L (c͞/Cl8CP%;UԘ}򕏒8_^*@ 疮QT5KM%Ґ b:(A>Q}7IP\ h6}d 9WHĀ=s @ (x P@]H#ҰP=!w$(w#*HXExV' q A&I0 J'$0&6"2W6(& 0o G K(]` = :a D! ߐŔ |)~K#4p IpCqŰ#Ҁx8@-؇.|'pAq2c 2 w*t1T=#`QP,H:Ap uhvOn?a*`! aBgȋi=q1ڠ\%`w8' x ~t҅5!P"" -_ U6De1 6$ Xx pn)HMm:F -*_y&yj1 :/P yG]ׄ @qGu2g sy;K1a R9 Rst(gU Q0 r " *?)[&] )ИՐb9=f:7blYB)(@>P9YiU1 l"5Ydt ‰ ȗ G2R 2հG a?3q0Cv::2 *t >R 3'P 1 " " fym S(B 8%G䞄>, h0 t ܒؠXzq ܑ ͢+ r 4p!=ubb&ę4ܹ=]FE0(Y 4z(`" X`i3q>u( **й#؜4? ^H Wv$Be  o8 2I _W3r hB s L`T'r-{@5H x'a*pF*y) @2zT0 H^ƫ$!@ P ,4 P2QVK( Ƣ _y K`ڰUzt p*@(0uL&O1 PLQ Xq,~pc y.MCjtNddQ* v }g*hF2 q<`FLЬ Q;+*E*OPAa o אh+g0{Je) T6ި0@j/ũ + 2!l_=@foT C^J<0)H8 ΀*P R{!&ް hEא."p q !Q ~Ѝ>@ɑln x@2[DEz;+ Tٔ[E+jm) ݡY*${h i4 ހ ͒,2Ut*P{ h&{ --Y';"P@ph&"C{&s@ $rV bVL1 Ā*>9 a'V/PL @1 py|DIpW@4YaPkl@ ( 3` \` J~k)Lc0AݰF*")' p ,G/B W~*pX)'1_8;`p~ Q͕Ta0 Icexj`R! \  R X0L*߸/:P@0@PP*@QAĐK'Ēہ0b *TFe)4q(MJ*(M/2gyي 9LJI9'*I rK,] U'@DQh= -0@n%&-O`w a"&pj l%$M@Y1 ;@a "?PpP )P߀;G@ q@9;ͧHkp^9Pj!<֪*\$7ف `ִ$X#gE2圀P p uu 1 gu2X3cB3 esMu*@kv{'@8\d 9݀ 000Ӏ=~U+hw^ Τp^=ه^* ˣ㑉1 2JP!]g <TxX 74bA&{9^y0`bB>" 5t )E3 @!ax@@'—!]φ<5~^hd! '8}&0*:"|0`፨N j `d`i: XV^el.<'"y}$2`b@!R̀90"])TN - Pvm\2Ur +Ґ)P@?p>ѐPn1l JӠ# n'䀕 DX/zfR ǸǸF)' &ǘmþ ]I VG?U0,hDQ o0/oP0q H?/U ܓT * 1Sґ /'P7p Y Tޑo?P7u1ӯP7Y2{ȀJ DP 6tD V8Qŋ9vLH1F"?fD5ԤU05 cYMԦ-Pˆ>x$yr EŚUVPQEAؤ'I"Vm-jpZuś^}")1A '%P_NsQӚI4\4Ehҥ{NOfвNVùm߮nu=xNន<kjRČƜy-lc)6LuZ8O60  ءW;|]V^kVsp䚞h$+BAh8%C ,H%" C+ƭ>D_> k z:k lpǦaGXȮh <4h)`MϭZL13MkʚTk)%$`qF.&z2Ѭ)34РRJ@L5trSO9:~y. lzd"F*-+9J|bZJE= r8#MH{-gm/O=}gZk Ub@PԉJZbRӪ ovD%6$ tYޠ!`&BkFlj%iZkZx`fWRx" D _ba{ @Yx 3N:ѕjt ahBR/4؀CL`NV&sC\@:KH@GKgʑv/L?$u5ceħVtwI8K#M Ad% R.#V66 P&`{~>yBN#ԎEF!N#~h [:ֵ~@\YG7?]6pDT' BHi܃VKp4׹%K_KpL.c&a: 0Nŗ2bsqW LyEҵ.Y._d-F#b dH:(ᒻ7bI[5ZW&.ſA€y2 <89D*nU s#& !ڗ֖)Zҍ"T3ķC:W* d>܊>dX;ńx7YgX",$ލld!&Xh1hX8F$=,*֘w[BFP/iC28qckK8XWUȎE8Ec?D9oаU4kxY 4>7"(TX :_0(9"˔h j꺤=-?crܸr(FzǔxY&Lru4t=Ag |zlHyqM0}v1sI"%s< t0QFr* 4z?#y:*k0 ~>ssYRkځ^}MҠk@#9Nsh9KH/h߸sh%Rp Usl׹G`0膡[: 06;+{"ʑ5"C[a#aӡZOoSlH  h=t+56؁U )&QN H 2?&L?C{P]Hø#– HEpijȂw+GhP@Hp1';Hl+5H(ƀ J@>3xizOp ](g\(F]j {{Rcbl(+hcZ,-v@Z-ظxJƙH{(B*qQ 8rrzV8CwCȀȋȌ,CGGh7a R8szs mk#):Ǚl[(ᮚ hs+PW) q`JqNhJieKkQOh"C0Ml p`# Ik ˺ i =#sʴ'L@jPKBf:(qp!-x>ɨu(rOl/FR/-E^&#b+`ҁ+Ԃk6,6G06J}EjL 0qI9ĠS@#VgͽJ|Hi X*l)ΐ N! p0]4i@rM&,#H(&H5s{TP?cqQ42O8Hia ؃:vG[- ZS^0 i83)"Cᬕ1rr0|!T0: B Ȃ@jQ:jقQ,X4'ɸN iclKPZh( eL3xSpK\b'e-1GomMHn1SpKjP|cPi];sRPj( U[u.7 it Vt>5EڸW}XK HgMzXΪ" vؾD &7<2^%+PWFkSEm`OT|-b9זh`MKlsӇ`DCGCGC7`@e[XO%iP׿7xsFj sK ]k 9}7|= vb?PI^Y\X\:܄Xi7^yDF)d'ވ77/8NpTe*Ȁx(\PUhvhENH870hӨr E_Uxo޴D~{9j]wڽ8FhN4X-HZr fXMzv (M f$J0̓6 }8֞`X)܊eҋx~ۋ{10ℹؑ uD@Ip^J %״zɄd~)L3h%84PR“0u\ͽ8Y`t$ #+:ЃFJȇD 2.0je-8@-$0XOqSH8䴥+pT gE\ ,C6\x&Ǹ+o19m*#ɨ p9[PS ]7dVp<.fSX`ُU }xR5^1lȁ$vXNhx=p J @zƋ{(LЂ`<``V$h0hix% v.Df"DDhGD0jP r(..h̀a,Vpvl l?(-ekh 8،kl=dXjႸ1m*bXЃl`nXw>dn~ЋeKboYwA.Յ9.K4h`w uNŦF&5]b8@ƾfPl9Ї$F\E 8PjHOOqZ " o H~~e/)n,r`VsU•l xn0jF `ljsƋ=O{ 2\Xa` 7h#X8FBqj8IN>6ph3n / E9<@Pl" י\^$>gp)DB,ƵW p })%Ӎzà 'AKgCp%8>0`ݻrvw8 (@0bx *WA}&305jAj7$v$ʅPtJhxh njN]5a'қЮI7Kiu) G+X$ Ӂ( Y gj)ӴzROCvf*`< +g5(df&@~H\ %0a+,@3t؍J&M4v(w|& $G{abϧY(m܅ta5i"OuUV7^g"ndĎ~jGJ;78z?5#2$)Z љJ]_zӜQM)H[1nTBc!:ar4Ȑ䄲Q$X|?[^_f2, XsOLi"liJ{:*@(N[-Y+ )}8(DȲ!5U8&^WRF^yϜNS5X2_ޅ'2[{u~ז[8cuq !Ǒ҄-KR,MbxJ@Ӑ=?">*8\sn47 $#W5*Hx@Mn4PLpfM<<+;HKO^sD< TYлRArULqtD&B+\f7C28S'?Jя!^D#oa8 30b3`t O؋8Dp e)d8N!xz%\@ Pg]`>B*Kn@k @&P" pİ4M7Сn&*:1-&Z%S[/ d~DZ +%vH0J=P!e :,bì`jc#T;+āo4RɐAv "jD|A LihcDWAR_$Xd{f`PNR#-5ЏTI݋4^)BxEF2K\jb}%ũJ̵s`;,%i\łz Xb``$1 4 (5Pf8i!V/&_/m ~)s_p A~DCRe!`qt o@:%l4q-lT#@q^q>r.>@O"8"KKј!eMvaTO 6" s`A>F``'cN).H:pD۪4XPVzi# 2;R^3&!!K0m|!zڴ68O@Ҙgk,({f8zd^DuzgePPf m!ym# =& bAM?Ptbd:c/!O:<7{ cpĜ 0d2agM#N4 )퓇ޛU0, yh ChQxe T>׭  5I>_(ݞt _q2MxdU: H8y߀ $  5F"ְL%f\L6pN NC7pM@I7 |堡AŞRG%^"-v4|B]9'^?< "J: zB L@ OCxyE&$,Ң%J;0\9c9:LipM )`4cP]#d6-/Ԛ-,p[BjM89 6<`) =f>!$68iB)$[fFPC52͞}B.d=&ma5EBcedj9\LA7h 9|&hJhiEU'La>9-[dN&5D7p.>,>T'Aojo2Fpuyyޥa<UE4$efO`ǽSE8pa/EWcXr]xx_5|&(&'LQN@>h'|B!&|(&4B(&T:4$ HI!2$4xB|=h]$(sT)(|f6(&pH%_g5#,#0#4Nqi%E1 XR ۭ\{28`(\480N 'i@7䘩2> ~4[)14A*TH7|F 8V4i4)ipMH3$~&|X̂C#%X#%g*F~Й9-M^\]ሩB^5d@;a$J)*]bꛕG5jHkH3D$ iqBr_l+C '-5`C.-߾_>;ZY*% >^DX.<@L(! ܃[ťȉ-D>A&5~lib,rX96+WR^^\"CdzQS.4$k@&E6 >m4\gD3xk%1CC@n5(HxA5CFFA#v]%k$6 ՝Fr-H"PО|ø@6x5`5qP-W.,傰ah0hjHxnJ8%d%Fz64dDK7-B9|~^mz@8W(BetBPBoiЀ4(ÿb oRpaf cf8Ec 4`o|\h]Bp`:@ː0]DB6(q\)hX*]%(Rn ?C.v É4P`4ht'@*"E>AT_}M4^ԙo0u$D zNeI~M"|Bʃ<-4_$pt09Ha1|'D$:i.M!^9/ q30_4 F3is=EM/(Q XgDC耋]x@˃ 00Cw{I1K\CHG\&CVk 'ژI VM74躷14k:z]MusZED2,!N0@]ȣPK";+H[j$ká',_5NC{~5 fw 0vzJT:g0^$q4D UN\6?Tĉ'<.h2`.C:h(KC1pBT+hpöLB+$?.?::B@C<H|dB{\*xB^?w_ kW/;@X4j&T!i8bE1fԸcljy9dIϜTeG\ZxZC )0G4i7_f( %dF:jUWfRHLmJ[7l`т$қh:Օn]wUůpD6ͺoNq$-/|oe˗1g7"p:Pi(kMQOٷ-m3fπiIikt,cYqӮ5WtfիĞhB֔Av ~}Yvl5%Ai(f CˠeF!P *o)il!K,T**lKʙUiaܺ$F_-ǁ !^t` O4L ӈ5b$$3 #≻S0DgLi~eh)8x0TYw9+ AVyb5Zh6x}i2vdy}wn-3eǑ뫾 hqioQbLi9m}U Vip4̐ǫJ_F'3턑ZɼX[6z=i,]Wة!Xdmʦl_hDDaOCasːX塽F74s$o`7TO3N #þw}E޲- `8nL W>@ Cu"'s!_4xƒ, ]~W6Tm:d/m[:#.d'Fq?9a‘7A h  =}l,HE.t#CB%1IM6'AJQf4)Qt+aKYΒ-qK]/LaM)N,eBFM NwAڤfjM"H D&t1{BS=i q"If橁^ @($Q z1oR]4'1Rd@BS)IeRaH Ftf4M]S.$EZB0UB"EQ+T-AjF +@K*ԮV`hU/zl~L"M̸V&QM c@=@%Z^&B? M5Z,TV}Q6(4?d@ 'գ =X[ Vmq[G[V "bqV^e@sN /˅ue]nZwr8L h@FЁ"ȀЁ 8O"~(@6;@B-%80G.$=R6qZH8j-(+D19xD D_x$IHH9gWఇ EP"pG8T,vG\,lr`"eű$D><NFDDȡ +D6c0%Q$@_5"liTaQdg)wv" n=F?sI)rh4E<}cp)6SZ"g5j8˹9C6'#4I &%A0G  DܙՌmhAx7_,!H$M P;3X4z |s"D*:3Xa?`c I R5(8ΠWAo?.c *, O]UQԱ^SE2=uA9rs/ˆw& ;gpml-matlab-v3.5-2014-12-08/doc/f1.gif000644 000765 000024 00000011576 12075730310 016633 0ustar00carlstaff000000 000000 GIF89a? """***---111333555888AAADDDFFFKKKSSSUUUYYY___```dddfffhhhwww~~~88HHXXllhhppttwwڊ磣!,pH,Ȥrl:ШtJZجvzxL.zn|N~  uIT..SFDNOtBIKsK.+JrzS"pl<X-ŋ&:a!Ə ilj8@ĸH&%ݜ IKbiO9nA^ػO< w@~\߁ڇ FXZ@xah Gb'fa(>R"'"+^43*"`;;&7VcD$CGNPѤOa;W% :3:_~xΔQTo hOe鄜MI_ʄ&K U[ө6yXA٦})X ì Q*g=jYfpvn0:Ĭ 4(}%j),X m0-ET0DZüBK < SpOw\1{|l ,r*ղ 4P>+*PV-mIn0+yڍ`}pR@uм邰cc )V x'zAJƊ4`iD#nސ{14mGNGTn9N;zU{x5X cyn_$-Ͷ.e !Ժ'b7gz`>_h;QףN/}M>/ SADPF)Q AuUoLGxoB<*ئj D?kaאr(|ab(üaXvcnF 8%n~yh2ZN]q9D71FeBbB4F]d#E9JWcH )N #eb.!e0@ Є")baRj`P l^±KH C U`L Ί/e Ph\ `+_(NnI8m$Y%rh5 -^j *ЇV#lN\fQ@.mJWI?*`A[KPv;;lg Dk*o%B{ WP`#bHq)`%$K[%,oYׯAq&S$V:7|{|)y0>Έ!abƈP{I.Z承a C[WW ߌq<+ T1C @[e ,bsOB*v^LJ_,$-=BrD|8A _!B7 =8{vS сߪ)^!QMupI/B׎fbLfC4#R V jWr|l$ˮP{Y0BܐhCh>k'1gJ/TmѶ¹mI'9lwi$ۺY63o2X?܊u$~784hYl>{~x0g7&'xw`p8m;,Ѱ\ƶ~AGH\NDYGIι4 np]AOur1RF6CuK?[ʰt^1Q?Ws,W k.9~Xjoj`Og{j֫(t-sIGNÿ+\A@orV .3[ׯ=k* M  6t8*_Wxq+J]֯Uku\~/ߺb1PD`q@!7~eu88Oz}FB OQfEkzZyħ3wIV0,mVvRq z[ #1}O@a'GP:ǃFqH+ HχmgG}(f}IFLQ8C| td8fVw!-y%넂2xivx?HE.`|zQh<ЇFĈE`wE8kHJh${XJfZCaqBj`P0z g<68hH(E8@!Do@eʨ#twg@͈8hMD؃Ƙb@騆@n*;k:qpv8@8۸l}~0V j {6Xd=#э~YzDmYB<}'2KGx3-^>0  < dqd%IsJv-[U9Seyg{bm]F1(VY ҭ7PM/}B8\T-}DLm5]s ]ce=϶Q)^mSʢ`=zP&=4%,$'E \T[,*PO@EMF}_sHEچȬȱ =gSۜpͺ} ^bۍͺ9EA;gpml-matlab-v3.5-2014-12-08/doc/f2.gif000644 000765 000024 00000035352 12075730302 016633 0ustar00carlstaff000000 000000 GIF89a !!!"""***---///000111333555888;;;<<<>>>???@@@AAADDDIIIKKKLLLMMMNNNOOOPPPQQQRRRSSSUUUYYYZZZ[[[\\\]]]^^^___```aaabbbcccdddeeefffhhhiiijjjkkklllmmmooohhxpppssstttuuuvvvwwwxxxzzz~~~kkQQUU !!'' ""//2211447744>>??BBFFHHIIMMRRUU\\aacc``jjkkppvvzz}}Á파헗읝뙙뫫鹹!,E H*\ȰÇ#JHŋ3jȱǏ CIɓ(S\ɲ˗0cʜI͛8sɳϟ@ JѣH*]ʴӧPJJիXjʵW3LKٳhӪ]˶[ "Kݻo2`0V.޿ AnŸǐ#K0$G9cׄA>SLH$CiXLא;!AH3gϦ 4oC>qɞ 2CW{:Cg:ν{ٙmlu_h^9/ ɜ6 =P 1D|%(YfG~3'G *cnR}9_$H҅aH7ϙ(e⁊8r ReB2Q&mk$!TVP&-"H\ڏgd Zr׉lrmv(RY3bf ɗ1B X(Fg"Ӕh$RJnlv)fa#H(Q0A:֩ff 뮿@ ~+oF,FD!dlw $~ǯ&m}jٖkU(spۭ|,zba+ULݹDljj$ܐ{lnV'4 '(>kBǫ/՛yIܑqkR,9w YXF7Kvn{v#~Ý>%XZ4y*ʹm  cT#z#ƙ}/?Y*]5νqr!seO0 c4 R yoBz 5;A d@1`W'⃠kG"8‚q"̷d`V!_J D1| AP|ā(їsB.2Dstx)ð8Ch ` MT 7 8df?"~^Ip<̍ ɠJnCi36$U(G .BV.QٛA!)C$ Ich-Buͤ&dUąa%I/ǝ89d`DžcLL遖| \'RHϕi ́2Cܬ[E931%g{dA`B+ (P (Pt<7 1A1*`+'S hXM(. SSD.2uXfCzd@BV1M%N,.#L$U"LX(# Qhid7#$؁aX$"CM2NƲAJE9Z("$))eWʮĩ0@!6Y(]2{DT{঵ %`f;PH޵`w;^񮕼5(k7]/z^ͽo~_w kdw5p_‡zse@–n zN! kUJ(\N撩}P*9Rw!O&%ȏcDt$Wr2|2I-$ʢ2-PzWb}uCR1g,9 ?fv%ik *fdeK n&zd"k (gD/0 @ fMG!,$▮!uilat2lCAzj mSײ]S281?dA$mudwUqrI*uMr|0tMV;#kAٞag_"A-l`R 9tE:N;q"O ]{M~&Y6rQ\UC7vDDsW\SψAo 'l^[aH@-oy[3:L%0k>GIxkؘM]&g`;>u!Sv^<{DO]o)"-N ,] !sm_~MnؾZ.OAa߷INyOt/d[AQz ޹ЅM7Z9d'.ĉMHYy2 -Glq{v0\>}5D }`RuQ n0HS!YU|A`h 2XwOzZ1pS0Y]z' HHV #'l79 )|#x|f1D !w33}sWFE0xQ%@:I~űBXnlx)rg0s:mqWLYqQ-qaH:}Ew炓>na XIU3ExQֱR/AH'}rC@rjzAsHyZ~wuw1qKaF,HS-NG$vmׇln8JfCz`*qmIօp6Y7Bt9 F'Q1'|88m !+ уĸ/ 4&5B ZD Y o5h'R|Qp@S8}-H#LOdhaa 2>I8OHw4 s!,WtwS`B]A,rZ s` %,!x+qAq~@W~a82:UO9٩QtD80V2KS j0%&)PB&ѳq[4Oǁ ixP L 92?VᵍQ tdBYMK0SwRJ* >ƫKb Њ>(,4'lYIAd`"b [֥ٺj, 0e0hֻb#~6Yj뼮1KM,K vUK}F c"˶fx1kM:(*9¾0K|emyS"Vvfy{a6"$ȢJ?6%, ZҎǡb5fU0ĊSV~&řRŝň!o. \Kr r؋k-T`Cq,Ǚiz;3hZ%CdA1d+#aPk*kSQlK  EX)(CiRU0Zt)l'@L C,9 'qZ%Ubc q #9g(l*YZ:Dh0kk'ṭͫ(#1X8#&a.|KOa m-_M?WhY}ֿʴbӐn5M'~֭asD¢|bW ar=R8 DiYP1"Ϗ-B2"qi߄ٷ948$-+Y ,nm;)"dv}0IOŴ A}#-$֠Ő1aɁB6IG,*|^m\&,LL7 3M9kT R$-Y; ,Ps W W 2 p0Y0eld: EZD~ϡԭ1 "d+OϩmN[]Xq qẠ--ZfQf AZ; Il+=AK?f]cdH 3-aV3#S*Lۋ`i>IOމP:-RSؘ1򞀤 U}}uD0oԁr#nm) ˍAQ=?b7bP"09R1xhxHL3.QNѮ"= =3m L\ A$#|.vM3*;M@-W|S<1]n]j G؄J \2 d)L鐄Aa+- 23 c}-CYFcez N5'90GڒqGD#k4u/A*41SOdIpvz\:eMsT0zH#>MQ&톝6B 4/$&s-C :"  ne6Cf~s4H(-O\6MQ @ DPaCAD=QD'T@F `IqF%M0J-]S̕38 JI4Yd' $1ȡ'>U) JM^'O~8X% Z`StBm!s 7葨`RFZ^xpǕ-_~,x $h)dqb&Ғ]~"쁔 eGF vBL*DG2TkPHPRr62h⌠Ö=1 )$@Ǐl!7+9/I43), 6Q0JsP2;/PԋB)d~j#LH02?$N@LkC{h!H|p؈$,5\ $*|oD$ zRŊ@Բ-=J*j 5Zh) d@6$+)tI(+8>tKG- @+N61[)HD ŃE%&hDSuŵ9RCh1;|$<=ܬT=%]u"YCV%sPlB(Vex:@H\tZjMq A`P nkFÍI;dTi-.0Ҳ VD?{ٽ%?p8,JQ' 譍Pcxc )e͍@9d#$5-4А@+jEnZSK^I#0>us&#z%SAT83dOzdcAVNhJ??N$s ^2_yRb*QC3JpC0 2H87)%ȶZNJO3@ZIP FC)ؠSF."Xӌ娞d{!LX#bBB\hm=D I"H7Dqi~!<pfb s  `$vVƒi\#GGӍ[ZB2Ad'''29,, L",aQC9yC3SUCS+R9LA;ZuzAE$(K"5e~#KrȻ}%A 2=v]W94 $w'I'+q|6ӶpTӒJL&y >ڤI *\ByvDc% ˀ&dStU$0 ZBR L0p l3e. I/3h9$uRRshdI$Ҧ0?,^.V<Ip*ȓP:q$d(HTB6dq$mS.ID9QVӒ`V׃8y ɺ) Y|bSe-x +P`q4Q,^Ϣp/1;؁neRX,"b!/B[|xJX7&%iM[ɢWqD2 qA!hAR H-/} U#c~$]dLGN pA@r5H˕o6-);&p3U;lЦ{2X0IiOaHhkXD@$$NΆ c@R&ky&CPoXHe" գIbM- z ܈"pCqAe g8H[K$@iX$D2M5"Ie`*Si3a8CWR#ǁ4W7U&8:l[HLRv,YS\ML`i0,^{ `=qY  rϗ$цx%I@i!< g`ƛo}JMD˧d &58D( q0'&UqCbݢ8~),*.yBYso2rLZ.:]<'+$sCҦXY$}eR:EiKM&[H V]n CvVY7ʋ"X>)E rLu &M_鳗};O·byZݐ*pO^=m?y<3^I&42sY#tⷳxBӝ&)>2m&܁4C g]*O;[V;p;+1(p[ #s r KJ?Q5- c72 K 68rc7389+Zg:YÏ8 +H3-@*2 6 Lkp* + ;  8d )$XzA/< AA4C;!Zy4-D; *yz"+lc7 P QPB4DyC(A@P\z# lꔫHB83@>DAPMȸ#zʠCN0X>>{]|)EГ N4 CʵG)AH =TF7؋ op|9G  ,ĈO" @`#P4P,|2rNFc `HEU<  =#F3S[Is|{60,A GPA $ Ès:#ȕDK7ěD%~!@bԫ%"II]ˮHLXǀңKĘڵ)$k Nh͔ !l6Nh?pM (M=QJX: \ ̀JƇK; d >H ,@-! t J" "Ő8y+ قHpNk ˕وχh ߑtMeLM}JDIJ5c.4ϐNuMG9Y"АpKˆU~NNG)iH`4]s=jU@ eS2 v^`+{V19iq\nhH$hpxGGf(l(LXBzwzvu@r@Jpt'WsoQ<7Un+pG2g⫐r x1DDnkwm]4FM(#% ,*CJ9[raXppwdZ>g?xb/7[NO"wXf\,J`zGIo&rz`Z?=ǃĕS!|f(wϴJ_M0km>zCuM"ZM =` u q‰뚯XO# 'ʐ4 ,z]7`O9xn(A|׫tϕ#J7՝B5X[  aDoy7L˭\'PN ?Ar|tM2;Wo-@AD,h „ 2la&Rh"ƌ7rh "GB@UhW!DHI(yz,@/M$3ȄWm'#=Y4ᛕh.4!ׄL/aJhj:s\:MOɧ1>44H]QTEBQBL)MHͪQHv4|HPD] *!Rn*1G%Fn' 9|hG(yjs@g1;J#)grخxH { (U1 A7P9l5 YX6+f$UA.A.кsCS!I /t/#]!(AFԠPq-fOrE/cQ"h\K¿ixEbNoYt$ޢdaA垰ěqf9,U$ hI; '"*ں̄6_įїAhcjM̰yp Np+A UFE*_??gH*-?ora|`Ӂ{_iԓ`)`1`w A``[JRMhViM ĝ Z >} .D%Nm'Y!B|[ ։K9ҷ`O^!@IQaXa/E_>1 _",_A N1 " a!!*!&" #b 2!."R%Zb&n&*b$!&&b$j$"""*rb)z"&^*R+"b%")"$,6"'2b,b/0"A#2&2.#363>#4F4N#5V5^#6f6n#7v7~#88cZ9njji;#eB 멆1?#Ņ$!A(UIBAaTB` СAEDN]d!&dFdN&eVe^&f6af&gRfgvhZhi YijNH(D#(B ?TRܤ&qDkV#؁hD%%P@xaz wZ%AZeZX#wN'dOjIw'J穄B sngYV}XDJ@m(u (Jc*@w"h}'~c̀ GZ hUpAxB愒GhqN'h{(I0 (2(>h H HhtO''ezFza'hBǨ $vhth>mJ(At6P AEH2 8JtD!\kNZȢ KjGF*:DjV^Ǫf*Dp֪)i *+RH;gpml-matlab-v3.5-2014-12-08/doc/usageSampling.m000644 000765 000024 00000005114 12075753717 020622 0ustar00carlstaff000000 000000 clear all, close all n1=10; n2=5; % number of data points from each class n1/n2 sc=1; % change the scale of the initial problem % 1) draw data from two Gaussians S1 = eye(2); S2 = [1 0.95; 0.95 1]; m1 = [0.75; 0]; m2 = [-0.75; 0]; randn('seed',17); rand('seed',17) % coordinates x1 = chol(S1)'*randn(2,n1)+repmat(m1,1,n1); x2 = chol(S2)'*randn(2,n2)+repmat(m2,1,n2); x1(:,5) = x1(:,2); x1(:,1) = x1(:,4); x2(:,1) = x2(:,2); % rank deficiency in K xtr = [x1 x2]'; % class labels ytr = [repmat(-1,[1,n1]) repmat(1,[1,n2])]'; tt = (-5:.3:5)*sc; [t1 t2] = meshgrid(tt,tt); x1 = x1*sc; x2=x2*sc; xtr=xtr*sc; m1=m1*sc; m2=m2*sc; t = [t1(:) t2(:)]; z1 = exp(-sum((t-repmat(m1',length(t),1))*inv(S1).*(t-repmat(m1',length(t),1)),2)/2); z2 = exp(-sum((t-repmat(m2',length(t),1))*inv(S2).*(t-repmat(m2',length(t),1)),2)/2); xte = t; clear t xtr(end,:)=[2.9,2.3]; ytr(end)=-1; % modif % 2) set GP parameters cov = {@covSum,{@covSEiso,@covNoise}}; hyp.cov = [0; 2; -Inf]; % ell,sf,sn lik = {'likLogistic'}; hyp.lik = []; % likLogistic or likErf mn = {'meanZero'}; hyp.mean = []; inf = 'infEP'; % 3) predict using approximated inference [nlZ,dnlZ,post] = gp(hyp, inf, mn, cov, lik, xtr, ytr); [ymu,ys2,fmu,fs2,junk,post] = gp(hyp,inf,mn,cov,lik,xtr,post,xte); % 4) prediction using MCMC sampling % set MCMC parameters, see some more details in inf/infMCMC.m % We have two samplers implemented, namely % hmc - Hybrid Monte Carlo, and % ess - Elliptical Slice Sampling. par.sampler = 'hmc'; par.Nsample = 20; % par.sampler = 'ess'; % par.Nais = 5; par.Nsample = 200; par.Nskip = 100; tic [posts,nlZs,dnlZs] = infMCMC(hyp,mn,cov,lik,xtr,ytr,par); toc [ymus,ys2s,fmus,fs2s,junk,posts] = gp(hyp,@infMCMC,mn,cov,lik,xtr,posts,xte); % 5a) echo results fprintf('nlZ-EP=%f, nlZ-AIS=%f\n', nlZ, nlZs) fprintf('acceptance rate (MCMC) = %1.2f%%\n',100*posts.acceptance_rate_MCMC) for r=1:length(posts.acceptance_rate_AIS) fprintf('acceptance rate (AIS) = %1.2f%%\n',100*posts.acceptance_rate_AIS(r)) end % 5b) print results figure subplot(221) plot([-12,12],[-12,12],'r'), hold on, plot(fmus,fmu,'k.'), title('\mu_f') xlabel('MCMC'), ylabel(inf) subplot(222) plot([0,10],[0,10],'r'), hold on, plot(sqrt(fs2s),sqrt(fs2),'k.') title('\sigma_f') xlabel('MCMC'), ylabel(inf) subplot(223) plot([-1,1],[-1,1],'r'), hold on, plot(ymus,ymu,'k.'), title('\mu_y') xlabel('MCMC'), ylabel(inf) subplot(224) plot([0,1],[0,1],'r'), hold on, plot(sqrt(ys2s),sqrt(ys2),'k.') title('\sigma_y') xlabel('MCMC'), ylabel(inf)gpml-matlab-v3.5-2014-12-08/doc/demoClassification.m000644 000765 000024 00000011117 12075734611 021613 0ustar00carlstaff000000 000000 disp('See http://www.gaussianprocess.org/gpml/code/matlab/doc/ for details.') disp('Hit any key to continue...'); pause disp(' '); disp('clear all, close all') clear all, close all write_fig = 0; disp(' ') disp('n1 = 80; n2 = 40; % number of data points from each class') n1 = 80; n2 = 40; disp('S1 = eye(2); S2 = [1 0.95; 0.95 1]; % the two covariance matrices') S1 = eye(2); S2 = [1 0.95; 0.95 1]; disp('m1 = [0.75; 0]; m2 = [-0.75; 0]; % the two means') m1 = [0.75; 0]; m2 = [-0.75; 0]; disp(' ') disp('x1 = bsxfun(@plus, chol(S1)''*gpml_randn(0.2, 2, n1), m1);') x1 = bsxfun(@plus, chol(S1)'*gpml_randn(0.2, 2, n1), m1); disp('x2 = bsxfun(@plus, chol(S2)''*gpml_randn(0.3, 2, n2), m2);') x2 = bsxfun(@plus, chol(S2)'*gpml_randn(0.3, 2, n2), m2); disp(' ') disp('x = [x1 x2]''; y = [-ones(1,n1) ones(1,n2)]'';') x = [x1 x2]'; y = [-ones(1,n1) ones(1,n2)]'; figure(6) disp('plot(x1(1,:), x1(2,:), ''b+''); hold on;'); plot(x1(1,:), x1(2,:), 'b+', 'MarkerSize', 12); hold on disp('plot(x2(1,:), x2(2,:), ''r+'');'); plot(x2(1,:), x2(2,:), 'r+', 'MarkerSize', 12); disp(' ') disp('[t1 t2] = meshgrid(-4:0.1:4,-4:0.1:4);') [t1 t2] = meshgrid(-4:0.1:4,-4:0.1:4); disp('t = [t1(:) t2(:)]; n = length(t); % these are the test inputs') t = [t1(:) t2(:)]; n = length(t); % these are the test inputs disp('tmm = bsxfun(@minus, t, m1'');') tmm = bsxfun(@minus, t, m1'); disp('p1 = n1*exp(-sum(tmm*inv(S1).*tmm/2,2))/sqrt(det(S1));') p1 = n1*exp(-sum(tmm*inv(S1).*tmm/2,2))/sqrt(det(S1)); disp('tmm = bsxfun(@minus, t, m2'');') tmm = bsxfun(@minus, t, m2'); disp('p2 = n2*exp(-sum(tmm*inv(S2).*tmm/2,2))/sqrt(det(S2));') p2 = n2*exp(-sum(tmm*inv(S2).*tmm/2,2))/sqrt(det(S2)); set(gca, 'FontSize', 24) disp('contour(t1, t2, reshape(p2./(p1+p2), size(t1)), [0.1:0.1:0.9])') contour(t1, t2, reshape(p2./(p1+p2), size(t1)), [0.1:0.1:0.9]) [c h] = contour(t1, t2, reshape(p2./(p1+p2), size(t1)), [0.5 0.5]); set(h, 'LineWidth', 2) colorbar grid axis([-4 4 -4 4]) if write_fig, print -depsc f6.eps; end disp(' '); disp('Hit any key to continue...'); pause disp(' ') disp('meanfunc = @meanConst; hyp.mean = 0;') meanfunc = @meanConst; hyp.mean = 0; disp('covfunc = @covSEard; hyp.cov = log([1 1 1]);') covfunc = @covSEard; hyp.cov = log([1 1 1]); disp('likfunc = @likErf;') likfunc = @likErf; disp(' ') disp('hyp = minimize(hyp, @gp, -40, @infEP, meanfunc, covfunc, likfunc, x, y);') hyp = minimize(hyp, @gp, -40, @infEP, meanfunc, covfunc, likfunc, x, y); disp('[a b c d lp] = gp(hyp, @infEP, meanfunc, covfunc, likfunc, x, y, t, ones(n, 1));') [a b c d lp] = gp(hyp, @infEP, meanfunc, covfunc, likfunc, x, y, t, ones(n,1)); disp(' ') figure(7) set(gca, 'FontSize', 24) disp('plot(x1(1,:), x1(2,:), ''b+''); hold on') plot(x1(1,:), x1(2,:), 'b+', 'MarkerSize', 12); hold on disp('plot(x2(1,:), x2(2,:), ''r+'')') plot(x2(1,:), x2(2,:), 'r+', 'MarkerSize', 12) disp('contour(t1, t2, reshape(exp(lp), size(t1)), 0.1:0.1:0.9)') contour(t1, t2, reshape(exp(lp), size(t1)), 0.1:0.1:0.9) [c h] = contour(t1, t2, reshape(exp(lp), size(t1)), [0.5 0.5]); set(h, 'LineWidth', 2) colorbar grid axis([-4 4 -4 4]) if write_fig, print -depsc f7.eps; end disp(' '); disp('Hit any key to continue...'); pause disp('large scale classification using the FITC approximation') disp('[u1,u2] = meshgrid(linspace(-2,2,5)); u = [u1(:),u2(:)];') [u1,u2] = meshgrid(linspace(-2,2,5)); u = [u1(:),u2(:)]; clear u1; clear u2 disp('nu = size(u,1);') nu = size(u,1); disp('covfuncF = {@covFITC, {covfunc}, u};') covfuncF = {@covFITC, {covfunc}, u}; disp('inffunc = @infFITC_Laplace;') inffunc = @infFITC_EP; % one could also use @infFITC_Laplace disp('hyp = minimize(hyp, @gp, -40, inffunc, meanfunc, covfuncF, likfunc, x, y);') hyp = minimize(hyp, @gp, -40, inffunc, meanfunc, covfuncF, likfunc, x, y); disp('[a b c d lp] = gp(hyp, inffunc, meanfunc, covfuncF, likfunc, x, y, t, ones(n,1));') [a b c d lp] = gp(hyp, inffunc, meanfunc, covfuncF, likfunc, x, y, t, ones(n,1)); disp(' ') figure(8) set(gca, 'FontSize', 24) disp('plot(x1(1,:), x1(2,:), ''b+''); hold on') plot(x1(1,:), x1(2,:), 'b+', 'MarkerSize', 12); hold on disp('plot(x2(1,:), x2(2,:), ''r+'')') plot(x2(1,:), x2(2,:), 'r+', 'MarkerSize', 12) disp('contour(t1, t2, reshape(exp(lp), size(t1)), 0.1:0.1:0.9)') contour(t1, t2, reshape(exp(lp), size(t1)), 0.1:0.1:0.9) [c h] = contour(t1, t2, reshape(exp(lp), size(t1)), [0.5 0.5]); set(h, 'LineWidth', 2) disp('plot(u(:,1),u(:,2),''ko'', ''MarkerSize'', 12)') plot(u(:,1),u(:,2),'ko', 'MarkerSize', 12) colorbar grid axis([-4 4 -4 4]) if write_fig, print -depsc f8.eps; end disp(' ')gpml-matlab-v3.5-2014-12-08/doc/f3.gif000644 000765 000024 00000033052 12075730274 016637 0ustar00carlstaff000000 000000 GIF89a !!!"""&&&(((***---...///000111222333555888:::<<<>>>???@@@AAADDDKKKMMMNNNOOOPPPQQQRRRSSSUUUYYY[[[\\\]]]^^^___```aaabbbcccdddfffggnhhhiiijjjkkklllmmmnnnooopppqqqrrrssstttuuuwwwxxxzzz}}}~~~^^WW99 >> !!%%'' //2211447766==BBFFHHMMOORRUUVV\\aacc``jjkkppppttvv񀀀Ӂ파읝롡뫫鹹!,k H*\ȰÇ#JHŋ3jȱǏ CIɓ(S\ɲ˗0cʜI͛8sɳϟ@ JѣH*]ʴӧPJJիXjʵW'PKٳhӪ]˶[&Kݻt(߿ˊ-+ԪW+^̸ǐ#Kf< ǫ&|aW>>SDI/L^ͺkȞ qYB@.(dwDJ}?D-ȓ+oȫ6D#oN}!z 3mla蔆47DH>AϿql$a&zrB)bC .Pr\H`!z3 (-8(I(׆H*䡁hipX<Ht6BCLKL6$yg"YKGfib`JsRXuh|cKW&;ۚxJ oJlhbcLaenH}V%HL!)hFebFVY#XZ-JF"lhWvU덷V&E[&VG&ud'Iv.{f;':fsB&DT+ &fãڶ޺a@ʝ嚩X^{$RrB`"']P1͸FTw$&Bv bw "E=AAqbH@raXu0W;mb'P.͜"۪f"֑nacuKRi3h+(z(c46Ke#vAxlSkc+ y'"uMҍ-t{V=8^7v ă 銧Rvcw*~Q,*D{J{ĸAcrEbzRԘT%N*ĊvʊG ή!D@NQZ!UCMZdMFG.pkiRc慞Ft "J3TbɘWRp&bnyMbIt $HJƲ1+L4 |GGItMY}T N2t%D)n"dVI|/1Otc LgZ57~9Lk1{q3Y'9NyΓ9?1:T:"/ayP*NL;*cR#BQg1Iu<"Fe,d&Ǭt83qg H-IibN J$>@p*yiίO̩bӓ"kYChdV}X4Ҩeje_@^xA!QŜB5 `  puHA7xA6dYOdjJۛ֨ UzJ MTfZiblXRDk@ՠ+<ʢ&i4Éxw"lAJPɔE߁wpf\ p[R%aOvp! kp3b "NUa?5m8 bU"g9^F@UercJyF2M{gT*F} pɫA3UQr1ucjI"l cŐ_1K#k[T$n0d"= a~Wù䡹$2&깾s.1Mv2IJX;o ȞRQY@!\5^:1@-e[B/R-4 +N[FuA&˄M)ZN 9Ff]H3 gDǼ۠ MH&$nm?En&]MtLX%ce#ܺ,$Tj'CWdc|yt:e|Dvh[#FI }<#tL4E<:D]&(W%#xR'B4HU$)X_&V|5h!#զ8 C.3CX2zE|@)1;KdT>F~Io^"rdXyݓUu!2<5=Eb19NJ5c'#м+3#{ w-r$}ʗ"Ο+BihW#B g b([`(/!Pvnv"W&HrW|dC{t'* q\$5eT2\n"yxDz4@aq/?X6:w DaD|8gT}}ڷǷ%?4SJ3`%gq,q}g{8pTW}B}\9.Y2gNa@QLY؄QA`*=6A񀿔p[({㑁j]H0>dD'|s&pt0Z$]r&H7IX+m+asFcQq(H!665:ACF4~9wLW{P!{0эq4M;#8Dk.7w2Ň/QoFl!w#498Wx5yLXΈ^ *)^q%7 ;0N 9d 2 2. HzOs uHCExKY _Ï%uomxW*KД ,B/D7Q玉<H"%xx2!F F' M8i G(hC?%zOP>/hjG3ԧw q2 2$Oi1AvHqbAN Uɜ,Y!x{3 e)9,qMcihI!etgI"z9 C<)KI9*< :x~WGhSS*QS2xH*zT,щ6'QoI|\K  (G#ኃ)~Zz?7irafJq,cT5z)]aViNo*p)=*tyV1?,"4 1p'ERxg9T*3^?ĚPGu^Dv a>t1|AB5L.q}CíԅxVwd")jRm,288<Q!! Pe)كjrQMH%!Z6ۄ/j5lljkYFtB~m HޙLf D 5Fpð;Ts:Y#4 VXVmh0CJF^x -])whYM}|~Kwp֖Vݓg?#BuZ"cCx @z}@e,M㓣 >Jr:p^$phNK[,8> *A$DWǻHV͋55%0=BC7b[bb0bb6k_nzv@ _cg'qE<bTqpcB< ALhM$i:~Q!Įik0| 5dH"1/Vۋ}ས Ŧaqu<I&5xgSU"?}ƪ b裱}PɀT k(z  ?>~l0l$><@w?\ܧIjuᴖQ,:$>ߴzxItZN|!ZGsEZ&գ8 A $Khq#i+k$EL  cuB̤w1Mc]`@ FV 5(: A1&HÓa]qargAwQ4!aز7MFFb*ȽJu V͙ͯͩ`M>RG 9][}sNag ^9Ӊq˝d8aK]1lW؆M[;|cx'pBL&ƞ$C ћMƝ;L׎18Y+۝޽QTaD!Ğӯ -MP;*HI3)m-MAh8B$--\W+nF⨴2˚LqPeZYU qՊ CdwyสiԜN֨&Q˪ڕBl8׼&E-M>ϱ3v 6쯑惘}G1=qz _J-wX&JLKT'#q銁!$:0g)ɡn5b~3 1)%-U $펤!N 1k;T^p{XgxJ.2P&/P O눱U4>^JDE <-&&&ލ1=K OL [ i$)/qwGUO!AEpcmgwzzF O2.lncrLL, j8+  d-s6=`1ONy % |v'^nB )qWwYQK@ $HA@$B[zQF=~H%2RJ-k KEt)HڀTPEAhP=8+*,y p*R@iVXe X NO0K,qU^EYmWaa_u:-_lNk9KhFF+R+><2Y^Pn޽MR'PcSp?/Ԇ"Y]QJ{;PI5^%̍HnĠTp7y <^?W֩'O$@x1Rԃ0¬@h^yd"Hc0ĠP03+bA1Oč&{MBw$L+}$:Cmd#S2c +2˖jl2x3L㈸B4z1Te#R/erH44W685/5?.5@-EVIVs1ΒfC 1>uW42k-US>rŴTW}6pVSU6nx_9pԍHy}UNb !&wT1#7u7 XO=BU#'l&,3DS);H0&dUfFϴt9Ervf2}/K=:lJk4nިh'7#hx(i\-:;p䨌T [ǤNLV-Rƻ)!_PQ`ҠیO<ч=9rrՆC81ߛxh ϭjQO\ I# r;o`knA߷:0a'/ГDmbŴr[q*!_HZXG°j  HĢ %J6h:AWx,o?`pr _Q ;ؒqeDk`Z-t#JR+ R8]g~Yіm(%җ*J(ZdD( 2Df.)е' *`tLzm'$VQlHO0!媇2ʹ!Zw;BqB܎Ȓ a w$U&9cA2)pt#r!DRfBw@{I-a""( ]B)i)u[UGSdL"AODyGeG$RL{fJ\`"@X(U̓H,VD1'#(<)Fihf!49g! -u45bЁGuxK3>ONmZI8e0.kD*|P%H@B\;7"ρBH's*Vl \lڥ"h { eH =H m@0;&znlf/ ъx=i7+rM-lW٦u| D[٢vekZXŬN( *ԏ+:>/du-;t,(x$ tFv6V]sP bɾH,hn_< dyA׽Ɛ]A4`-HDi;`F[^<1 (_!IuRX ua D0Mxj)E1 )QE⋟8\ h DҬXBc^AˆʄH0a)at0;9^@C&Rʰ A e``. t&v:h|37#n>)yquݜ@# ĺ)(-C! FFrckԌlBxMܡe+NЇnي nN4pH PX ];nÇ}jLƉwFHDrD _J+"u!B숭.NaY#U[N+Dmn xI du? ^AQZj_Ʃ+PXڞa.8[)X<M%Id`P  %`y\Nʺ*EnN{!bݎzԫŠ@"(\,4Ң }t}[*#'i{dT7p R \@MqFw0Nvzzh ]lJWK(/r$R@Z#4]_s)%ħ7~QND!>^0{H̦cC6hF2{)?3{-s#E}^}&BY,8?h1 p>aU [R<7o@K,@[ - TU|[ Zs = @q/ 89ŠP; eb6$1M;"̕lP >:>.ڑ 6$H0W8i+d>3C" 0q;𪱠N2@{ =C&Y*Rxåı:{rJވ) LD, ; 1A*d?/:9YL 99cġp>3 0 9% Z -z9-̡?L1lO$2 :ؿkCP2GZ %c^ TLR ǮW+ $xbD HmWpy,X8Wl +d 0d1KXcI0\Z䍿tư@Z)A 1C \J n\QD 60X2"6ȏUpωŵ GJy s&z M0sQ̒AP NȏxR̓گɘހ@ΩXA܍PK%au)hH qNGjѹ b@G=UK#)3HR@JU`4 SviƒFRjdl+!KMOx#SQӈGr><T&Yw)ɳB䯌8EtK /}bح-b׎xb&FLXJ64f$[  <D]bCA.gd`eFե 8R0{l~cNf0e AQ;e批d&:HZa~\ؠ(DHN`&nr孻a] \hM \!N@3lgGfL,02,di3GS׌j^ ImUЀ RxlZ AuCkX'%OXa8V­*& :433/8h¯U6\@NX5 iW5M,Vb\ܾXfr1瘨ֈ-1X)`.D쥄`~ {952hm$AލUx6 hދ% XX֌a.֕;ꮶRh(!UpoB䡈 l–FXeo crY!@yHY8b)Bselzc2halq0= > Q$T8 [!s$Z Sx_8U8z4!R(xt!0PgMHgBn]KT5g ;B`d d!H8umGI0ti݈8D qrsH'*H[N.[$xL=[[ kO 82.-;7Gy?yO".RDyS  hN04}2F| 51P>@:o dWl EW('@k7ZȘq+,ZJi؎'r<Nh2R]OnJQR/Sw 0sfWA[93wv|{nXhϗ8*;q ,Jwhs{5K NӑR0s va'&rd2w}7$ W@ԉp@nڗhW,h „ &Ĉ'Rhb- 7rE^Ô 貖>*g|Ec˜:%jU;-j4J2-PG5Jdә F!kSP-Z]tdײeaظr %ջC_ֵ}Aޡu|, Jq:},l9+ Ȓ)ewp*t§=QfڵXƈ4זعwE_v5OΧ>:3nxnnlADJ)V=| (A^է^\`Yn)" rk)tu_h'h&Xr`PGJV)!&GI!=lgv…'kÁr`B'#bËA5ZAEB^B4gg%MLda4($j\ 'D(mYd } 9G#u6ki))% Ș؆UL{vAwNG\yVyQuUJZ6)u:\u Z埝hũWq\{)*Vd_(s6e"rYGۂeH ܳYf|b.!\},|]V!^V0%KRa"^J{a]q%o-Mi)])`& \|zRuR&|3WG#] "lY]bJBt\[O4qm>K\W {=7k(=,'_d TJ}Z=}wBho\@-Ax/❳3%=O  hIuGR)9X~WJ~^z; |V#?=y.PRέ!QR=^Po\k84ʽM4.c?&t~ @ p1޷7-&JP., +6 !`mjt.i!.&0a, sOQ߲r鰈r o:Kg)6I|Es?)f*Fu`EHNt&6G.Yρ4!nH *bQOtq)>@!#9٢gW"ᎊDEGD0.$*CQN*RY4~0F?R%0{C ƌo#K3J@&rJ2,y2Eh)3x '?=)I@v\^bBEG mM RG`P41fN\32l ԥ2N}*T*թRU*VխrMW*ֱfU)"D'\*׹ҵv+^׽~+`+=,bXSHHFM ],Rl`Xd+ ft8 ִjs[D"/ֶ|P}A.׸ԫn%Օ("v{Zyŋ*м !{;Ͻ}|}g_8n".uKSOwkׂ͝s%Yx"cblx IF>2%3N~2,)S,C2;wel9^>q,uyffri[IֻrA[ O-.AO%}45V"yΡU4~_nQngAsz(IDZ/0N,RiRGz(Ep&ʭӶ X,nY`W"@~2|eZ 6![w&PR+@Em^c%X~#B[ "s]l(ބ7 MorNZK"NX9":9~_^{ҏo#e%\/B4AT<KH@;gpml-matlab-v3.5-2014-12-08/lik/likSech2.m000644 000765 000024 00000020502 12513017551 017461 0ustar00carlstaff000000 000000 function [varargout] = likSech2(hyp, y, mu, s2, inf, i) % likSech2 - sech-square likelihood function for regression. Often, the sech- % square distribution is also referred to as the logistic distribution not to be % confused with the logistic function for classification. The expression for the % likelihood is % likSech2(t) = Z / cosh(tau*(y-t))^2 where % tau = pi/(2*sqrt(3)*sn) and Z = tau/2 % and y is the mean and sn^2 is the variance. % % hyp = [ log(sn) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. The moments % \int f^k likSech2(y,f) N(f|mu,var) df are calculated via a Gaussian % scale mixture approximation. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-09-02. % % See also LIKFUNCTIONS.M, LIKLOGISTIC.M. if nargin<3, varargout = {'1'}; return; end % report number of hyperparameters sn = exp(hyp); tau = pi/(2*sqrt(3)*sn); lZ = log(pi) - log(sn) - log(4*sqrt(3)); if nargin<5 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>3&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability evaluation lp = lZ - 2*logcosh(tau*(y-mu)); s2 = 0; else % prediction lp = likSech2(hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 ymu = mu; % first y moment if nargout>2 ys2 = s2 + sn.^2; % second y moment end end varargout = {lp,ymu,ys2}; else % inference mode switch inf case 'infLaplace' r = y-mu; [g,dg,d2g,d3g] = logcosh(tau.*r); % precompute derivatives if nargin<6 % no derivative mode dlp = {}; d2lp = {}; d3lp = {}; lp = lZ - 2*g; if nargout>1 % first derivatives dlp = 2*tau.*dg; if nargout>2 % 2nd derivative of log likelihood d2lp = -2*tau.^2.*d2g; if nargout>3 % 3rd derivative of log likelihood d3lp = 2*tau.^3.*d3g; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode lp_dhyp = 2*tau.*r.*dg - 1; % derivative w.r.t. sn dlp_dhyp = -2*tau.*(dg+tau.*r.*d2g); d2lp_dhyp = 2*tau.^2.*(2*d2g + tau.*r.*d3g); varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' n = max([length(y),length(mu),length(s2),length(sn)]); on = ones(n,1); y = y.*on; mu = mu.*on; s2 = s2.*on; sn = sn.*on; % vectors only fac = 1e3; % factor between the widths of the two distributions ... % ... from when one considered a delta peak, we use 3 orders of magnitude idlik = fac*sn1) and nu*sn^2/(nu-2) is the variance (for nu>2). % % The hyperparameters are: % % hyp = [ log(nu-1) % log(sn) ] % % Note that the parametrisation guarantees nu>1, thus the mean always exists. % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-09-02. % % See also LIKFUNCTIONS.M. if nargin<3, varargout = {'2'}; return; end % report number of hyperparameters numin = 1; % minimum value of nu nu = exp(hyp(1))+numin; sn2 = exp(2*hyp(2)); % extract hyperparameters lZ = gammaln(nu/2+1/2) - gammaln(nu/2) - log(nu*pi*sn2)/2; if nargin<5 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>3&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability evaluation lp = lZ - (nu+1)*log( 1+(y-mu).^2./(nu.*sn2) )/2; s2 = 0; else % prediction lp = likT(hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 ymu = mu; % first y moment; for nu<=1 this is the mode if nargout>2 if nu<=2 ys2 = Inf(size(mu)); % variance does not always exist else ys2 = (s2 + nu*sn2/(nu-2)).*ones(size(mu)); % second y moment end end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' r = y-mu; r2 = r.*r; if nargin<6 % no derivative mode dlp = {}; d2lp = {}; d3lp = {}; lp = lZ - (nu+1)*log( 1+r2./(nu.*sn2) )/2; if nargout>1 a = r2+nu*sn2; dlp = (nu+1)*r./a; % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = (nu+1)*(r2-nu*sn2)./a.^2; if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = (nu+1)*2*r.*(r2-3*nu*sn2)./a.^3; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode a = r2+nu*sn2; a2 = a.*a; a3 = a2.*a; if i==1 % derivative w.r.t. nu lp_dhyp = nu*( psi(nu/2+1/2)-psi(nu/2) )/2 - 1/2 ... -nu*log(1+r2/(nu*sn2))/2 +(nu/2+1/2)*r2./(nu*sn2+r2); lp_dhyp = (1-numin/nu)*lp_dhyp; % correct for lower bound on nu dlp_dhyp = nu*r.*( a - sn2*(nu+1) )./a2; dlp_dhyp = (1-numin/nu)*dlp_dhyp; % correct for lower bound on nu d2lp_dhyp = nu*( r2.*(r2-3*sn2*(1+nu)) + nu*sn2^2 )./a3; d2lp_dhyp = (1-numin/nu)*d2lp_dhyp; % correct for lower bound on nu else % derivative w.r.t. sn lp_dhyp = (nu+1)*r2./a - 1; dlp_dhyp = -(nu+1)*2*nu*sn2*r./a2; d2lp_dhyp = (nu+1)*2*nu*sn2*(a-4*r2)./a3; end varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargout>1 error('infEP not supported since likT is not log-concave') end n = max([length(y),length(mu),length(s2)]); on = ones(n,1); y = y(:).*on; mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only % since we are not aware of an analytical expression of the integral, % we use Gaussian-Hermite quadrature N = 20; [t,w] = gauher(N); oN = ones(1,N); lZ = likT(hyp, y*oN, sig*t'+mu*oN, []); lZ = log_expA_x(lZ,w); % log( exp(lZ)*w ) varargout = {lZ}; case 'infVB' % variational lower site bound % t(s) \propto (1+(s-y)^2/(nu*s2))^(-nu/2+1/2) % the bound has the form: (b+z/ga)*f - f.^2/(2*ga) - h(ga)/2 n = numel(s2); b = zeros(n,1); y = y.*ones(n,1); z = y; varargout = {b,z}; end end % computes y = log( exp(A)*x ) in a numerically safe way by subtracting the % maximal value in each row to avoid cancelation after taking the exp function y = log_expA_x(A,x) N = size(A,2); maxA = max(A,[],2); % number of columns, max over columns y = log(exp(A-maxA*ones(1,N))*x) + maxA; % exp(A) = exp(A-max(A))*exp(max(A))gpml-matlab-v3.5-2014-12-08/lik/likInvGauss.m000644 000765 000024 00000011107 12513017653 020260 0ustar00carlstaff000000 000000 function [varargout] = likInvGauss(link, hyp, y, mu, s2, inf, i) % likInvGauss - Inverse Gaussian likelihood function for strictly positive data % y. The expression for the likelihood is % likInvGauss(f) = sqrt(lam/(2*pi*y^3))*exp(-lam*(mu-y)^2/(2*mu^2*y)) with % mean=mu and variance=mu^3/lam where mu = g(f) is the Inverse Gaussian % intensity, f is a Gaussian process, y is the strictly positive data. % Hence, we have -- with log(Zy) = -(log(lam)-log(2*pi*y^3))/2 % llik(f) = log(likInvGauss(f)) = -lam*(y-mu)^2/(2*mu^2*y) - log(Zy). % The larger one chooses lam, the stronger the likelihood resembles a Gaussian % since skewness = 3*sqrt(mu/lam) and kurtosis = 15*mu/lam. % % We provide two inverse link functions 'exp' and 'logistic': % g(f) = exp(f) and g(f) = log(1+exp(f))). % The link functions are located at util/glm_invlink_*.m. % % Note that for neither link function the likelihood lik(f) is log concave. % % The hyperparameters are: % % hyp = [ log(lam) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % See also LIKFUNCTIONS.M. % % Copyright (c) by Hannes Nickisch, 2013-10-16. if nargin<4, varargout = {'1'}; return; end % report number of hyperparameters lam = exp(hyp); if nargin<6 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>3&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability lg = g(mu,link); elg = exp(lg); lZy = -(log(lam)-log(2*pi*y.^3))/2; % normalisation constant lp = -lam*(y./elg-1).^2 ./(2*y) - lZy; else lp = likInvGauss(link, hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 % compute y moments by quadrature n = max([length(y),length(mu),length(s2)]); on = ones(n,1); N = 20; [t,w] = gauher(N); oN = ones(1,N); lw = ones(n,1)*log(w'); mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only lg = g(sig*t'+mu*oN,link); ymu = exp(logsumexp2(lg+lw)); % first moment using Gaussian-Hermite quad if nargout>2 elg = exp(lg); yv = elg.^3/lam; % second y moment from inverse Gaussian distribution ys2 = (yv+(elg-ymu*oN).^2)*w; end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' [lg,dlg,d2lg,d3lg] = g(mu,link); elg = exp(lg); if nargin<7 % no derivative mode lZy = -(log(lam)-log(2*pi*y.^3))/2; % normalisation constant lp = -lam*(y./elg-1).^2 ./(2*y) - lZy; dlp = {}; d2lp = {}; d3lp = {}; % return arguments if nargout>1 dlp = lam*(y./elg-1).*dlg./elg; % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = lam*( (y./elg-1).*(d2lg-dlg.^2) - y.*dlg.^2./elg )./elg; if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = lam*( (y./elg-1) .* (4*dlg.^3-6*dlg.*d2lg+d3lg) ... + (3*dlg.^3 - 3*dlg.*d2lg) )./ elg; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode lp_dhyp = 1/2-lam*(y./elg-1).^2 ./(2*y); % deriv. of log lik w.r.t. lam dlp_dhyp = lam*(y./elg-1).*dlg./elg; % first derivative d2lp_dhyp = lam*( (y./elg-1).*(d2lg-dlg.^2) - y.*dlg.^2./elg )./elg; % 2nd varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargin<7 % no derivative mode % Since we are not aware of an analytical expression of the integral, % we use quadrature. varargout = cell(1,nargout); [varargout{:}] = lik_epquad({@likInvGauss,link},hyp,y,mu,s2); else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' error('infVB not supported') end end % compute the log intensity using the inverse link function function varargout = g(f,link) varargout = cell(nargout, 1); % allocate the right number of output arguments if strcmp(link,'exp') [varargout{:}] = glm_invlink_exp(f); else [varargout{:}] = glm_invlink_logistic(f); endgpml-matlab-v3.5-2014-12-08/lik/likMix.m000644 000765 000024 00000020074 12231677545 017272 0ustar00carlstaff000000 000000 function [varargout] = likMix(lik, hyp, varargin) % likMix - Mixture of likelihoods for regression/classification. % The expression for the likelihood is % likMix(t) = sum_i=1..m w_i * lik_i(t), % where lik_i are the m individual likelihood functions combined by a weighted % sum with weights wi. % % The hyperparameters are: % % hyp = [ log(w_1) % log(w_2) % ... % log(w_m) % hyp_1 % hyp_2 % ... % hyp_m ] % % Note that the mixture weights are normalised using the softmax function, so % that 1 = sum w_i. Here, hyp_i are the hyperparameters for the individual % likelihoods. % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Hannes Nickisch and Rowan McAllister, 2013-10-23. % % See also LIKFUNCTIONS.M. m = numel(lik); % number of mixture components if m==0, error('We require at least one mixture component.'), end for i = 1:m % iterate over mixture components f = lik(i); if iscell(f{:}), f = f{:}; end % expand cell array if necessary j(i) = cellstr(feval(f{:})); % collect number hypers end s = @(x) logsumexp2(x); % define a shortcut to util/logsumexp2.m if nargin<4 % report number of parameters L = char(j(1)); for i=2:length(lik), L = [L, '+', char(j(i))]; end varargout = {['(',num2str(m),'+',L,')']}; return end mu = varargin{2}; n = length(mu); % number of query points if numel(varargin)>3 inf = varargin{4}; % varargin = {y, mu, s2, inf, i} else inf = ''; end v = []; % v vector indicates to which likelihood parameters belong nhyp = m; % number of hyperparameters; the first m are the mixture weights for i = 1:m ni = eval(char(j(i))); nhyp = nhyp + ni; v = [v, repmat(i,1,ni)]; end if nhyp>length(hyp), error('not enough hyperparameters'), end if strcmp(inf,'infVB'), error('infVB not supported'); end na = nargout; if nargout>1 && strcmp(inf,'infLaplace') && nargin>6, na = max(na,3); end argout = zeros(n,m,na); % collect output arguments from individual lik fun for i=1:m % iteration over mixture components f = lik(i); if iscell(f{:}), f = f{:}; end % expand cell array if necessary id = find(v==i)+m; % hyperparameter indices out = cell(size(argout,3),1); nn = min(nargin,6)-2; % do not ask for derivs [out{:}] = feval(f{:}, hyp(id), varargin{1:nn}); % ask individual likelihoods for j=1:size(argout,3), argout(:,i,j) = out{j}; end end lw = reshape(hyp(1:m),1,[]); lwn = lw-s(lw); % w = exp(lwn), sum(w)=1, weights one_lwn = ones(n,1)*lwn; % the same as above only that the weights are repeated varargout = cell(nargout,1); % allocate memory for output arguments if nargin<6 % prediction mode if inf is not present % lp and ymu outputs are of the weighted sum of integral structure if nargout>0, varargout{1} = s(one_lwn + argout(:,:,1)); end if nargout>1, varargout{2} = sum(exp(one_lwn).*argout(:,:,2), 2); end if nargout>2 % ys2 output computed using the parallel axis theorem argout3corr = argout(:,:,3) + (argout(:,:,2)-varargout{2}*ones(1,m)).^2; varargout{3} = sum(exp(one_lwn).*argout3corr, 2); end else if nargin>6, i = varargin{5}; end % make arguments available switch inf case {'infEP','infLaplace'} % the two cases are structurally similar if nargin<7 % no derivative mode if nargout>0 % lf={lZ,lp} for inf={'infEP','infLaplace'} lfi = argout(:,:,1); % partition functions of individual likelihoods lf = s(lfi + one_lwn); % weighted sum in the exp domain varargout{1} = lf; if nargout>1 % dlf dlfi = argout(:,:,2); % derivatives of individual terms % Using f*dlf=df, f=exp(lf), f=sum_i wi*fi, df=sum_i wi*dfi we obtain % dlf = sum_i exp(lfi-lf+lwi)*dlfi = sum_i ai*dlfi. a = exp(lfi - lf*ones(1,m) + one_lwn); dlf = sum(a.*dlfi,2); varargout{2} = dlf; if nargout>2 % d2lf d2lfi = argout(:,:,3); % 2nd derivatives of individual terms % Using d2lf=d(df/f)=(d2f*f-df^2)/f^2 <=> d2f=f*(d2lf+dlf^2) and % d2f = sum_i wi*d2fi, we get d2lf = sum_i ai*(d2lfi+dlfi^2)-dlf^2. d2lf = sum(a.*(d2lfi+dlfi.*dlfi),2) - dlf.*dlf; varargout{3} = d2lf; if nargout>3 && strcmp(inf,'infLaplace') % d3lf % Using d3f = sum_i wi*d3fi, d3lf=d((d2f*f-df^2)/f^2), we obtain % d3f = f*(d3lf+dlf*(3*d2lf+dlf^2)) <=> % d3lf = d3f/f - dlf*(3*d2lf+dlf^2). d3lfi = argout(:,:,4); % 3rd derivatives of individual terms d3lf = sum(a.*(d3lfi+dlfi.*(3*d2lfi+dlfi.*dlfi)),2); d3lf = d3lf - dlf.*(3*d2lf+dlf.*dlf); varargout{4} = d3lf; end end end end else % d(lf)/dhyp and d(d2lf)/dhyp if nargout>0 lfi = argout(:,:,1); % partition functions of individual likelihoods lf = s(lfi + one_lwn); % weighted sum in the exp domain a = exp(lfi - lf*ones(1,m) + one_lwn); if i<=m % the first m hyperparameters are mixture weights lf_dhyp = a(:,i) - exp(lwn(i)); varargout{1} = lf_dhyp; if nargout>1 && strcmp(inf,'infLaplace') % d(dlf)/dhyp dlfi = argout(:,:,2); dlwn = zeros(size(lw)); dlwn(i) = 1; dlwn = dlwn - exp(lw(i)-s(lw)); dla = ones(n,1)*dlwn - lf_dhyp*ones(1,m); % d log(v) / dhyp da = a.*dla; dlf_dhyp = sum(da.*dlfi,2); varargout{2} = dlf_dhyp; if nargout>2 % d(d2lf)/dhyp d2lfi = argout(:,:,3); dlf = sum(a.*dlfi,2); d2lf_dhyp = sum(da.*(d2lfi+dlfi.*dlfi),2) - 2*dlf.*dlf_dhyp; varargout{3} = d2lf_dhyp; end end else j = v(i-m); % index of mixture component under consideration id = find(v==j)+m; % hyper parameter indices i = i-min(id)+1; % index of likelihood parameter of mixture j f = lik(j); if iscell(f{:}), f = f{:}; end % expand cell if necessary if nargout==1 lfi_dhyp = feval(f{:}, hyp(id), varargin{1:end-1}, i); elseif nargout==2 [lfi_dhyp,dlfi_dhyp] = feval(f{:}, hyp(id), varargin{1:end-1}, i); else [lfi_dhyp,dlfi_dhyp,d2lfi_dhyp] = ... feval(f{:}, hyp(id), varargin{1:end-1}, i); end lf_dhyp = exp(lwn(j)+lfi(:,j)-lf) .* lfi_dhyp(:); % d(lf)/dhyp varargout{1} = lf_dhyp; if nargout>1 % d(dlf)/dhyp dlfi = argout(:,:,2); d2lfi = argout(:,:,3); dlf = sum(a.*dlfi,2); dla = -lf_dhyp*ones(1,m); dla(:,j) = dla(:,j)+lfi_dhyp(:); da = a.*dla; dlf_dhyp = sum(da.*dlfi,2) + a(:,j).*dlfi_dhyp(:); varargout{2} = dlf_dhyp; if nargout>2 % d(d2lf)/dhyp p = d2lfi+dlfi.*dlfi; dpj = 2*dlfi(:,j).*dlfi_dhyp(:) + d2lfi_dhyp(:); d2lf_dhyp = a(:,j).*dpj + sum(da.*p,2) - 2*dlf.*dlf_dhyp; varargout{3} = d2lf_dhyp; end end end end end case 'infVB' error('infVB not supported') end endgpml-matlab-v3.5-2014-12-08/lik/likWeibull.m000644 000765 000024 00000010704 12513017443 020123 0ustar00carlstaff000000 000000 function [varargout] = likWeibull(link, hyp, y, mu, s2, inf, i) % likWeibull - Weibull likelihood function for strictly positive data y. The % expression for the likelihood is % likWeibull(f) = g1*ka/mu * (g1*y/mu)^(ka-1) * exp(-(g1*y/mu)^ka) with % gj = gamma(1+j/ka), mean=mu and variance=mu^2*(g2/g1^2-1) where mu = g(f) is % the Weibull intensity, f is a Gaussian process, y is the positive data. % Hence, we have llik(f) = log(likWeibull(f)) = % log(g1*ka/mu) + (ka-1)*log(g1*y/mu) - (g1*y/mu)^ka. % % We provide two inverse link functions 'exp' and 'logistic': % g(f) = exp(f) and g(f) = log(1+exp(f))). % The link functions are located at util/glm_invlink_*.m. % % Note that for neither link function the likelihood lik(f) is log concave. % % The hyperparameters are: % % hyp = [ log(ka) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % See also LIKFUNCTIONS.M. % % Copyright (c) by Hannes Nickisch, 2013-10-30. if nargin<4, varargout = {'1'}; return; end % report number of hyperparameters ka = exp(hyp); lg1 = gammaln(1+1/ka); g1 = exp(lg1); dlg1 = -psi(1+1/ka)/ka; if nargin<6 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>4&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability lg = g(mu,link); lp = lg1 + log(ka) + (ka-1)*(lg1+log(y)) - ka*lg - exp(ka*(lg1+log(y)-lg)); else lp = likWeibull(link, hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 % compute y moments by quadrature n = max([length(y),length(mu),length(s2)]); on = ones(n,1); N = 20; [t,w] = gauher(N); oN = ones(1,N); lw = ones(n,1)*log(w'); mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only lg = g(sig*t'+mu*oN,link); ymu = exp(logsumexp2(lg+lw)); % first moment using Gaussian-Hermite quad if nargout>2 elg = exp(lg); g2 = gamma(1+2/ka); yv = elg.^2*(g2/g1^2-1); % second y moment from Weibull distribution ys2 = (yv+(elg-ymu*oN).^2)*w; end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' [lg,dlg,d2lg,d3lg] = g(mu,link); elg = exp(-ka*lg); if nargin<7 % no derivative mode lp = lg1 + log(ka) + (ka-1)*(lg1+log(y)) -ka*lg - exp(ka*(lg1+log(y)-lg)); dlp = {}; d2lp = {}; d3lp = {}; % return arguments if nargout>1 dlp = -ka*dlg + ka*(g1*y).^ka .* elg.*dlg; % dlp, deriv of log lik if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = -ka*d2lg + ka*(g1*y).^ka .* ( -ka*elg.*dlg.^2 + elg.*d2lg ); if nargout>3 % d3lp, 3rd derivative of log likelihood a = ka^2*dlg.^3 -3*ka*dlg.*d2lg + d3lg; d3lp = - ka*d3lg + ka*(g1*y).^ka .* a .*elg; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode v = ka*(lg1+log(y)-lg); ev = exp(v); % derivative of log lik w.r.t. ka w = v+ka*dlg1; dw = -ka*dlg; d2w = -ka*d2lg; lp_dhyp = 1 + w - ev.*w; dlp_dhyp = dw.*(1-ev.*(1+w)); % first derivative d2lp_dhyp = d2w.*(1-ev.*(1+w)) - dw.^2.*(ev.*(2+w)); % and also second varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargin<7 % no derivative mode % Since we are not aware of an analytical expression of the integral, % we use quadrature. varargout = cell(1,nargout); [varargout{:}] = lik_epquad({@likWeibull,link},hyp,y,mu,s2); else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' error('infVB not supported') end end % compute the log intensity using the inverse link function function varargout = g(f,link) varargout = cell(nargout, 1); % allocate the right number of output arguments if strcmp(link,'exp') [varargout{:}] = glm_invlink_exp(f); else [varargout{:}] = glm_invlink_logistic(f); endgpml-matlab-v3.5-2014-12-08/lik/likBeta.m000644 000765 000024 00000011336 12513020000 017354 0ustar00carlstaff000000 000000 function [varargout] = likBeta(link, hyp, y, mu, s2, inf, i) % likBeta - Beta likelihood function for interval data y from [0,1]. % The expression for the likelihood is % likBeta(f) = 1/Z * y^(mu*phi-1) * (1-y)^((1-mu)*phi-1) with % mean=mu and variance=mu*(1-mu)/(1+phi) where mu = g(f) is the Beta intensity, % f is a Gaussian process, y is the interval data and % Z = Gamma(phi)/Gamma(phi*mu)/Gamma(phi*(1-mu)). % Hence, we have % llik(f) = log(likBeta(f)) = -lam*(y-mu)^2/(2*mu^2*y) - log(Zy). % % We provide two inverse link functions 'logit' and 'expexp': % g(f) = 1/(1+exp(-f)) and g(f) = exp(-exp(-f))). % The link functions are located at util/glm_invlink_*.m. % % Note that for neither link function the likelihood lik(f) is log concave. % % The hyperparameters are: % % hyp = [ log(phi) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % See also LIKFUNCTIONS.M. % % Copyright (c) by Hannes Nickisch, 2014-03-04. if nargin<4, varargout = {'1'}; return; end % report number of hyperparameters phi = exp(hyp); if nargin<6 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>4&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability lg = g(mu,link); elg = exp(lg); v = phi*elg; w = phi-v; a0 = gammaln(w)-gammaln(phi); lp = (v-1).*log(y) + (w-1).*log(1-y) - gammaln(v) - a0; else lp = likBeta(link, hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 % compute y moments by quadrature n = max([length(y),length(mu),length(s2)]); on = ones(n,1); N = 20; [t,w] = gauher(N); oN = ones(1,N); lw = ones(n,1)*log(w'); mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only lg = g(sig*t'+mu*oN,link); ymu = exp(logsumexp2(lg+lw)); % first moment using Gaussian-Hermite quad if nargout>2 elg = exp(lg); yv = elg.*(1-elg)/(1+phi); % second y moment from Beta distribution ys2 = (yv+(elg-ymu*oN).^2)*w; end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' [lg,dlg,d2lg,d3lg] = g(mu,link); elg = exp(lg); v = phi*elg; w = phi-v; if nargin<7 % no derivative mode a0 = gammaln(phi-v)-gammaln(phi); lp = (v-1).*log(y) + (w-1).*log(1-y) - gammaln(v) - a0; dlp = {}; d2lp = {}; d3lp = {}; % return arguments if nargout>1 % dlp, derivative of log likelihood a1 = v.*(log(y)-log(1-y) + psi(0,w)-psi(0,v)); dlp = dlg.*a1; if nargout>2 % d2lp, 2nd derivative of log likelihood a2 = v.^2.*(psi(1,w)+psi(1,v)); z = dlg.^2+d2lg; d2lp = z.*a1 - dlg.^2.*a2; if nargout>3 % d3lp, 3rd derivative of log likelihood a3 = v.^3.*(psi(2,w)-psi(2,v)); d3lp = (dlg.*z+2*dlg.*d2lg+d3lg).*a1 - 3*dlg.*z.*a2 + dlg.^3.*a3; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode % deriv. of log lik w.r.t. phi lp_dhyp = v.*log(y)+w.*log(1-y)-v.*psi(0,v)-w.*psi(0,w)+phi*psi(0,phi); a1 = v.*(log(y)-log(1-y) + psi(0,w)-psi(0,v)); da1 = a1 + v.*(w.*psi(1,w)-v.*psi(1,v)); dlp_dhyp = dlg.*da1; % first derivative a2 = v.^2.*(psi(1,w)+psi(1,v)); z = dlg.^2+d2lg; da2 = v.^2.*(w.*psi(2,w)+v.*psi(2,v)) + 2*a2; d2lp_dhyp = z.*da1 - dlg.^2.*da2; % second derivative varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargin<7 % no derivative mode % Since we are not aware of an analytical expression of the integral, % we use quadrature. varargout = cell(1,nargout); [varargout{:}] = lik_epquad({@likBeta,link},hyp,y,mu,s2); else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' error('infVB not supported') end end % compute the log intensity using the inverse link function function varargout = g(f,link) varargout = cell(nargout, 1); % allocate the right number of output arguments if strcmp(link,'expexp') [varargout{:}] = glm_invlink_expexp(f); else [varargout{:}] = glm_invlink_logit(f); end gpml-matlab-v3.5-2014-12-08/lik/likUni.m000644 000765 000024 00000005066 12513017473 017263 0ustar00carlstaff000000 000000 function [varargout] = likUni(hyp, y, mu, s2, inf, i) % likUni - Uniform likelihood function for classification. The expression for % the likelihood is % likUni(t) = 1/2. % % There are no hyperparameters: % % hyp = [ ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Hannes Nickisch, 2013-09-02. % % See also LIKFUNCTIONS.M. if nargin<3, varargout = {'0'}; return; end % report number of hyperparameters if nargin<5 % prediction mode if inf is not present lp = -log(2)*ones(size(mu)); ymu = {}; ys2 = {}; if nargout>1 p = exp(lp); ymu = 2*p-1; % first y moment if nargout>2 ys2 = 4*p.*(1-p); % second y moment end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' if nargin<6 % no derivative mode lp = -log(2)*ones(size(mu)); dlp = {}; d2lp = {}; d3lp = {}; if nargout>1 dlp = zeros(size(mu)); % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = zeros(size(mu)); if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = zeros(size(mu)); end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode varargout = {[],[],[]}; % derivative w.r.t. hypers end case 'infEP' if nargin<6 % no derivative mode lZ = -log(2)*ones(size(mu)); % log part function dlZ = {}; d2lZ = {}; if nargout>1 dlZ = zeros(size(mu)); % 1st derivative w.r.t. mean if nargout>2 d2lZ = zeros(size(mu)); % 2nd derivative w.r.t. mean end end varargout = {lZ,dlZ,d2lZ}; else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' % variational lower site bound % t(s) = 1/2 % the bound has the form: (b+z/ga)*f - f.^2/(2*ga) - h(ga)/2 n = numel(s2); b = zeros(n,1); y = y.*ones(n,1); z = y; varargout = {b,z}; end endgpml-matlab-v3.5-2014-12-08/lik/likPoisson.m000644 000765 000024 00000010122 12513017567 020153 0ustar00carlstaff000000 000000 function [varargout] = likPoisson(link, hyp, y, mu, s2, inf, i) % likPoisson - Poisson likelihood function for count data y. The expression for % the likelihood is % likPoisson(f) = mu^y * exp(-mu) / y! with mean=variance=mu % where mu = g(f) is the Poisson intensity, f is a % Gaussian process, y is the non-negative integer count data and % y! = gamma(y+1) its factorial. Hence, we have -- with Zy = gamma(y+1) = y! -- % llik(f) = log(likPoisson(f)) = log(g(f))*y - g(f) - log(Zy). % The larger the intensity mu, the stronger the likelihood resembles a Gaussian % since skewness = 1/sqrt(mu) and kurtosis = 1/mu. % % We provide two inverse link functions 'exp' and 'logistic': % For g(f) = exp(f), we have lik(f) = exp(f*y-exp(f)) / Zy. % For g(f) = log(1+exp(f))), we have lik(f) = log^y(1+exp(f)))(1+exp(f)) / Zy. % The link functions are located at util/glm_invlink_*.m. % % Note that for both intensities g(f) the likelihood lik(f) is log concave. % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % See also LIKFUNCTIONS.M. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-10-16. if nargin<4, varargout = {'0'}; return; end % report number of hyperparameters if nargin<6 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>4&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability lg = g(mu,link); lp = lg.*y - exp(lg) - gammaln(y+1); else lp = likPoisson(link, hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 % compute y moments by quadrature n = max([length(y),length(mu),length(s2)]); on = ones(n,1); N = 20; [t,w] = gauher(N); oN = ones(1,N); lw = ones(n,1)*log(w'); mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only lg = g(sig*t'+mu*oN,link); ymu = exp(logsumexp2(lg+lw)); % first moment using Gaussian-Hermite quad if nargout>2 elg = exp(lg); yv = elg; % second y moment from Poisson distribution ys2 = (yv+(elg-ymu*oN).^2)*w; end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' if nargin<7 % no derivative mode [lg,dlg,d2lg,d3lg] = g(mu,link); elg = exp(lg); lp = lg.*y - elg - gammaln(y+1); dlp = {}; d2lp = {}; d3lp = {}; % return arguments if nargout>1 dlp = dlg.*(y-elg); % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = d2lg.*(y-elg) - dlg.*dlg.*elg; if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = d3lg.*(y-elg) - dlg.*(dlg.*dlg+3*d2lg).*elg; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode varargout = {[],[],[]}; % derivative w.r.t. hypers end case 'infEP' if nargin<7 % no derivative mode % Since we are not aware of an analytical expression of the integral, % hence we use quadrature. varargout = cell(1,nargout); [varargout{:}] = lik_epquad({@likPoisson,link},hyp,y,mu,s2); else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' error('infVB not supported') end end % compute the log intensity using the inverse link function function varargout = g(f,link) varargout = cell(nargout, 1); % allocate the right number of output arguments if strcmp(link,'exp') [varargout{:}] = glm_invlink_exp(f); else [varargout{:}] = glm_invlink_logistic(f); endgpml-matlab-v3.5-2014-12-08/lik/likLaplace.m000644 000765 000024 00000015412 12513017640 020061 0ustar00carlstaff000000 000000 function [varargout] = likLaplace(hyp, y, mu, s2, inf, i) % likLaplace - Laplacian likelihood function for regression. % The expression for the likelihood is % likLaplace(t) = exp(-|t-y|/b)/(2*b) with b = sn/sqrt(2), % where y is the mean and sn^2 is the variance. % % The hyperparameters are: % % hyp = [ log(sn) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-10-16. % % See also LIKFUNCTIONS.M. if nargin<3, varargout = {'1'}; return; end % report number of hyperparameters sn = exp(hyp); b = sn/sqrt(2); if nargin<5 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>3&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability evaluation lp = -abs(y-mu)./b -log(2*b); s2 = 0; else % prediction lp = likLaplace(hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 ymu = mu; % first y moment if nargout>2 ys2 = s2 + sn.^2; % second y moment end end varargout = {lp,ymu,ys2}; else % inference mode switch inf case 'infLaplace' if nargin<6 % no derivative mode if numel(y)==0, y=0; end ymmu = y-mu; dlp = {}; d2lp = {}; d3lp = {}; lp = -abs(ymmu)/b -log(2*b); if nargout>1 dlp = sign(ymmu)/b; % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = zeros(size(ymmu)); if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = zeros(size(ymmu)); end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode lp_dhyp = abs(y-mu)/b - 1; % derivative of log likelihood w.r.t. hypers dlp_dhyp = sign(mu-y)/b; % first derivative, d2lp_dhyp = zeros(size(mu)); % and also of the second mu derivative varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' n = max([numel(y),numel(mu),numel(s2),numel(sn)]); on = ones(n,1); y = y(:).*on; mu = mu(:).*on; s2 = s2(:).*on; sn = sn(:).*on; % vectors only fac = 1e3; % factor between the widths of the two distributions ... % ... from when one considered a delta peak, we use 3 orders of magnitude idlik = fac*sn1 lqp = -zp.^2/2 - log(2*pi)/2 - logphi(-zp); % log( N(z)/Phi(z) ) lqm = -zm.^2/2 - log(2*pi)/2 - logphi( zm); dap = -exp(lqp-log(s2(id))/2) + sqrt(2)./sn(id); dam = exp(lqm-log(s2(id))/2) - sqrt(2)./sn(id); % ( exp(ap).*dap + exp(am).*dam )./( exp(ap) + exp(am) ) dlZ(id) = expABz_expAx([ap,am],[1;1],[dap,dam],[1;1]); if nargout>2 a = sqrt(8)./sn(id)./sqrt(s2(id)); bp = 2./sn(id).^2 - (a - zp./s2(id)).*exp(lqp); bm = 2./sn(id).^2 - (a + zm./s2(id)).*exp(lqm); % d2lZ(id) = ( exp(ap).*bp + exp(am).*bm )./( exp(ap) + exp(am) )... % - dlZ(id).^2; d2lZ(id) = expABz_expAx([ap,am],[1;1],[bp,bm],[1;1]) - dlZ(id).^2; end end end varargout = {lZ,dlZ,d2lZ}; else % derivative mode dlZhyp = zeros(n,1); if any(idlik) dlZhyp(idlik) = 0; end if any(idgau) dlZhyp(idgau) = ... likLaplace(log(sn(idgau)), mu(idgau), y(idgau), 'infLaplace', 1); end if any(id) % substitution to obtain unit variance, zero mean Laplacian tmu = (mu(id)-y(id))./sn(id); tvar = s2(id)./sn(id).^2; zp = (tvar+tmu/sqrt(2))./sqrt(tvar); vp = tvar+sqrt(2)*tmu; zm = (tvar-tmu/sqrt(2))./sqrt(tvar); vm = tvar-sqrt(2)*tmu; dzp = (-s2(id)./sn(id)+tmu.*sn(id)/sqrt(2)) ./ sqrt(s2(id)); dvp = -2*tvar - sqrt(2)*tmu; dzm = (-s2(id)./sn(id)-tmu.*sn(id)/sqrt(2)) ./ sqrt(s2(id)); dvm = -2*tvar + sqrt(2)*tmu; lezp = logerfc(zp); % ap = exp(vp).*ezp lezm = logerfc(zm); % am = exp(vm).*ezm vmax = max([vp+lezp,vm+lezm],[],2); % subtract max to avoid numerical pb ep = exp(vp+lezp-vmax); em = exp(vm+lezm-vmax); dap = ep.*(dvp - 2/sqrt(pi)*exp(-zp.^2-lezp).*dzp); dam = em.*(dvm - 2/sqrt(pi)*exp(-zm.^2-lezm).*dzm); dlZhyp(id) = (dap+dam)./(ep+em) - 1; end varargout = {dlZhyp}; % deriv. wrt hypers end case 'infVB' % variational lower site bound % t(s) = exp(-sqrt(2)|y-s|/sn) / sqrt(2*sn²) % the bound has the form: (b+z/ga)*f - f.^2/(2*ga) - h(ga)/2 n = numel(s2); b = zeros(n,1); y = y.*ones(n,1); z = y; varargout = {b,z}; end end % logerfc(z) = log(1-erf(z)) function lc = logerfc(z) lc = logphi(-z*sqrt(2)) + log(2); function y = expABz_expAx(A,x,B,z) N = size(A,2); maxA = max(A,[],2); % number of columns, max over columns A = A-maxA*ones(1,N); % subtract maximum value y = ( (exp(A).*B)*z ) ./ ( exp(A)*x );gpml-matlab-v3.5-2014-12-08/lik/likGaussWarp.m000644 000765 000024 00000021632 12513020134 020425 0ustar00carlstaff000000 000000 function [varargout] = likGaussWarp(warp, hyp, y, mu, varargin) % likGaussWarp - Warped Gaussian likelihood for regression. % The expression for the likelihood is % likGaussWarp( y | t ) = likGauss( g(y) | t ) * g'(y), % where likGauss is the Gaussian likelihood and g is the warping function. % % The hyperparameters are: % % hyp = [ theta_1 % theta_2 % .. % theta_ng % log(sn) ] % % Here, sn is the standard deviation of the underlying Gaussian and theta_i for % i=1..ng are the ng hyperparameters of the warping function g. % % At the moment, likGaussWarp offers 3 different warping functions: % id yields g(y) = y => likGaussWarp = likGauss % poly e.g. 'poly1' yields g(y) = y => likGaussWarp = likGauss % 'poly3' yields g(y) = y + c1*sy*ay^2 + c2*sy*ay^3 % where sy = sign(y), ay = abs(y), cj = exp(theta_j) % tanh e.g. 'tanh0' yields g(y) = y => likGaussWarp = likGauss % 'tanh2' yields g(y) = y + a1*tanh(b1*(y+c1)) + a2*tanh(b2*(y+c2)) % where aj = exp(theta_j), bj = exp(theta_j+m), bj = theta_j+2*m % % The code is based on the exposition in the paper Warped Gaussian Processes, % NIPS, 2003 by Edward Snelson, Carl Rasmussen and Zoubin Ghahramani. % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Hannes Nickisch, 2013-10-24. % % See also LIKFUNCTIONS.M. lik = {@likGauss}; % in principle any likelihood function can be warped but only % for homoscedastic likelihoods, in particular Gaussian has feasible integrals if numel(warp)==0, warp = 'id'; end % set default warping function ng = g(warp); % number of hyperparameters for the warping function nhyp = ['(',num2str(ng),'+',feval(lik{:}),')']; % number of hyperparameters if nargin<4, varargout = {nhyp}; return, end % report number of parameters nhyp = eval(nhyp); if nhyp>length(hyp), error('not enough hyperparameters'), end [gy,lgpy] = g(warp,y,hyp(1:ng)); % evaluate warping function i = 0; if nargin>6, i = varargin{3}; varargin{3} = varargin{3}-ng; end varargout = cell(nargout,1); % allocate memory for output arguments if i==0 || ng4, s2 = varargin{1}; end % s2==0 ? if nargin>4&&numel(s2)>0&&norm(s2)>eps>0, s2zero = 0; end, end if s2zero % log probability lp = likGaussWarp(warp, hyp, y, mu, [], 'infLaplace'); s2 = 0*mu; else lp = likGaussWarp(warp, hyp, y, mu, s2, 'infEP'); % prediction end if nargout>0, varargout{1} = lp; end % works for any lik % the predictive moments are very hard to compute for lik not being likGauss if nargout>1 ymu = mu; % first g(y) moment sn2 = exp(2*hyp(ng+1)); % Gaussian noise variance ys2 = s2 + sn2; % second g(y) moment % ymuM = ig(warp,ymu,hyp(1:ng)); % median % yupp = ig(warp,ymu+2*sqrt(ys2),hyp(1:ng)); % 95% confidence interval % ylow = ig(warp,ymu-2*sqrt(ys2),hyp(1:ng)); % ys2C = (yupp-ylow).^2/16; N = 20; [t,w] = gauher(N); oN = ones(1,N); % Gaussian-Hermite quadrature Z = sqrt(ys2(:))*t'+ymu(:)*oN; Y = ig(warp,Z,hyp(1:ng)); ymu = Y*w; ys2 = (Y-ymu*oN).^2*w; % first and second y moment varargout{2} = reshape(ymu,size(mu)); if nargout>2 varargout{3} = ys2; end end else inf = varargin{2}; % obtain input variables switch inf case {'infLaplace','infEP'} % they have the same structure if nargin<7 % no derivative mode if nargout>0, varargout{1} = varargout{1} + lgpy; end else % derivative mode if i<=ng % derivatives w.r.t. warping function parameters n = max([numel(y),numel(mu)]); for j=2:nargout, varargout{j} = zeros(n,1); end [dgy,dlgpy] = g(warp,y,hyp(1:ng),i); % warping function derivative out = cell(nargout+1,1); % allocate memory [out{:}] = likGaussWarp(warp, hyp, y, mu, varargin{1:2}); % query lik % works only for homoscedastic likelihoods where y and mu can be swapped if nargout>0, varargout{1} = dlgpy - out{2}.*dgy; end % apply chain rule if nargout>1, varargout{2} = - out{3}.*dgy; end if nargout>2, varargout{3} = - out{4}.*dgy; end end end case 'infVB' % output does not depend on mu and following parameters end end % marshalling of parameters and available warping functions function varargout = g(warp,varargin) varargout = cell(nargout, 1); % allocate the right number of output arguments if strcmp(warp,'id') % indentity warping likGaussWarp = likGauss if nargin<2 if nargout>0, varargout{1} = 0; end elseif nargin<4 if nargout>0, varargout{1} = varargin{1}; end if nargout>1, varargout{2} = 0*varargin{1}; end end elseif numel(strfind(warp,'poly'))>0 m = str2double(warp(5:end)); if nargin<2 && nargout>0, varargout{1} = m-1; return, end [varargout{:}] = g_poly(varargin{:}); elseif numel(strfind(warp,'tanh'))>0 m = str2double(warp(5:end)); if nargin<2 && nargout>0, varargout{1} = 3*m; return, end [varargout{:}] = g_tanh(varargin{:}); end % invert g(y) = z <=> ig(z) = y via bisection search + Newton iterations function [y,n,d] = ig(warp,z,hyp) y = z; gy = g(warp,z,hyp)-z; dz = max(abs(z(:))); % lower bound search ylow while any(0gy(:)), y(0>gy) = y(0>gy)+dz; gy = g(warp,y,hyp)-z; end, yupp = y; for n=1:12 % bisection search ylow<=y<=yupp d = max(abs(gy(:))); if d0) = y(gy>0); end for n=1:12 % Newton iterations [gy,lgpy] = g(warp,y,hyp); gpy = exp(lgpy); y = y - (gy-z)./gpy; y(yyupp) = yupp(y>yupp); % keep brackets d = max( abs(gy(:)-z(:)) ); if dsqrt(eps), fprintf('Not converged: res=%1.4e\n',d), end % poly warping function g(y) and log of the derivative log(g'(y))>0 % or derivatives of the latter w.r.t. ith hyperparameter function [gy,lgpy] = g_poly(y,hyp,i) m = numel(hyp)+1; c = exp(hyp); if nargin==2 % function values gy = y; gpy = 1; ay = abs(y); for j=2:m gy = gy + c(j-1)*ay.^j; gpy = gpy + c(j-1)*j*ay.^(j-1); end gy = sign(y).*gy; lgpy = log(gpy); else % derivatives gpy = 1; ay = abs(y); for j=2:m gpy = gpy + c(j-1)*j*ay.^(j-1); end gy = c(i)*ay.^j; lgpy = c(i)*j*ay.^(j-1)./gpy; end % tanh warping function g(y) and log of the derivative log(g'(y))>0 % or derivatives of the latter w.r.t. ith hyperparameter function [gy,lgpy] = g_tanh(y,hyp,i) m = numel(hyp)/3; a = exp(hyp(1:m)); b = exp(hyp(m+(1:m))); c = hyp(2*m+(1:m)); if nargin==2 % function values gy = y; gpy = 1; for j=1:m ai = a(j); bi = b(j); ci = c(j); ti = tanh(bi*(y+ci)); dti = 1-ti.^2; gy = gy + ai *ti; gpy = gpy + ai*bi*dti; end lgpy = log(gpy); else % derivatives gpy = 1; for j=1:m ai = a(j); bi = b(j); ci = c(j); ti = tanh(bi*(y+ci)); dti = 1-ti.^2; gpy = gpy + ai*bi*dti; end if i<=m j = i; ai = a(j); bi = b(j); ci = c(j); ti = tanh(bi*(y+ci)); dti = 1-ti.^2; gy = ai*ti; lgpy = ai*bi*dti./gpy; elseif i<=2*m j = i-m; ai = a(j); bi = b(j); ci = c(j); ti = tanh(bi*(y+ci)); dti = 1-ti.^2; gy = ai*bi*dti.*(y+ci); d2ti = -2*ti.*dti; lgpy = ai*bi*(dti+bi*d2ti.*(y+ci))./gpy; else j = i-2*m; ai = a(j); bi = b(j); ci = c(j); ti = tanh(bi*(y+ci)); dti = 1-ti.^2; gy = ai*bi*dti; d2ti = -2*ti.*dti; lgpy = ai*bi^2*d2ti./gpy; end endgpml-matlab-v3.5-2014-12-08/lik/likGumbel.m000644 000765 000024 00000007610 12513017671 017740 0ustar00carlstaff000000 000000 function [varargout] = likGumbel(sign, hyp, y, mu, s2, inf, i) % likGumbel - Gumbel likelihood function for extremal value regression. % The expression for the likelihood is % likGumbel(t) = exp(-z-exp(-z))/be, z = ga+s*(y-t)/be, be = sn*sqrt(6)/pi % where s={+1,-1} is a sign switching between left and right skewed, ga is the % Euler-Mascheroni constant, y is the mean, sn^2 is the variance. % The skewness and kurtosis of likGumbel are 1.14*s and 2.4, respectively. % % The hyperparameters are: % % hyp = [ log(sn) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Hannes Nickisch, 2013-11-01. % % See also LIKFUNCTIONS.M. if nargin<4, varargout = {'1'}; return; end % report number of hyperparameters if sign=='-', s = -1; else s = 1; end % extract sign of skewness sn2 = exp(2*hyp); % extract hyperparameters ga = 0.5772156649; % Euler-Mascheroni constant be = sqrt(6*sn2)/pi; lZ = -log(be); if nargin<6 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>4&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability evaluation lp = likGumbel(sign, hyp, y, mu, [], 'infLaplace'); s2 = 0; else % prediction lp = likGumbel(sign, hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 ymu = mu; % first y moment if nargout>2 ys2 = s2 + sn2; % second y moment end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' z = ga+s*(y-mu)/be; emz = exp(-z); if nargin<7 % no derivative mode dlp = {}; d2lp = {}; d3lp = {}; lp = lZ -z -emz; if nargout>1 dz = -s/be; % dz/dmu dlp = dz*(emz-1); % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = -dz^2*emz; if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = dz^3*emz; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative w.r.t. log(sn) dz = -s/be; % dz/dmu dzs = -s*(y-mu)/be; % dz/dlog(sn) lp_dhyp = dzs.*(emz-1) -1; dlp_dhyp = dz*(1-emz.*(1+dzs)); d2lp_dhyp = dz^2*emz.*(2+dzs); varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargout>1 error('infEP not supported since likT is not log-concave') end n = max([length(y),length(mu),length(s2)]); on = ones(n,1); y = y(:).*on; mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only % since we are not aware of an analytical expression of the integral, % we use Gaussian-Hermite quadrature N = 20; [t,w] = gauher(N); oN = ones(1,N); lZ = likGumbel(sign, hyp, y*oN, sig*t'+mu*oN, []); lZ = log_expA_x(lZ,w); % log( exp(lZ)*w ) varargout = {lZ}; case 'infVB' error('infVB not supported') end end % computes y = log( exp(A)*x ) in a numerically safe way by subtracting the % maximal value in each row to avoid cancelation after taking the exp function y = log_expA_x(A,x) N = size(A,2); maxA = max(A,[],2); % number of columns, max over columns y = log(exp(A-maxA*ones(1,N))*x) + maxA; % exp(A) = exp(A-max(A))*exp(max(A))gpml-matlab-v3.5-2014-12-08/lik/likNegBinom.m000664 000765 000024 00000011145 12513017605 020220 0ustar00carlstaff000000 000000 function [varargout] = likNegBinom(link, hyp, y, mu, s2, inf, i) % likNegBinom - Negative binomial likelihood function for count data y. % The expression for the likelihood is % likNegBinom(f) = 1/Z * mu^y / (r+mu)^(r+y), Z = r^r*G(y+r)/(G(y+1)*G(r)) % with G(t)=gamma(t)=(t-1)!, mean=mu and variance=mu*(mu+r)/r, where r is the % number of failures parameters, mu = g(f) is the negative binomial intensity, % f is a Gaussian process and y is the non-negative integer count data. % Hence, we have -- with log(Z)=r*log(r)+L(y+r)-L(y+1)-L(r), L(t)=gammaln(t) -- % llik(f) = log(likNegBinom(f)) = y*log(mu)-(r+y)*log(r+mu)-log(Z). % % We provide two inverse link functions 'exp' and 'logistic': % g(f) = exp(f) and g(f) = log(1+exp(f))). % The link functions are located at util/glm_invlink_*.m. % % Note that for neither link function the likelihood lik(f) is log concave. % % The hyperparameters are: % % hyp = [ log(r) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % See also LIKFUNCTIONS.M. % % Copyright (c) by Hannes Nickisch, 2015-01-21. if nargin<4, varargout = {'1'}; return; end % report number of hyperparameters lr = hyp; r = exp(lr); if nargin<6 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>4&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability lZ = gammaln(y+1) + gammaln(r) - gammaln(y+r) - r*lr; lg = g(mu,link); mx = max(lg,lr); lgr = log(exp(lg-mx)+exp(lr-mx))+mx; % log(exp(lg)+r) lp = y.*lg - (y+r).*lgr - lZ; else lp = likNegBinom(link, hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 % compute y moments by quadrature n = max([length(y),length(mu),length(s2)]); on = ones(n,1); N = 20; [t,w] = gauher(N); oN = ones(1,N); lw = ones(n,1)*log(w'); mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only lg = g(sig*t'+mu*oN,link); ymu = exp(logsumexp2(lg+lw)); % first moment using Gaussian-Hermite quad if nargout>2 elg = exp(lg); yv = elg.*(elg/r+1); % second y moment from negative binomial distribution ys2 = (yv+(elg-ymu*oN).^2)*w; end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' [lg,dlg,d2lg,d3lg] = g(mu,link); elg = exp(lg); lZ = gammaln(y+1) + gammaln(r) - gammaln(y+r) - r*lr; mx = max(lg,lr); lgr = log(exp(lg-mx)+exp(lr-mx))+mx; % log(exp(lg)+r) a = 1./(1+r./elg); da = a.*(1-a).*dlg; % auxiliary quantities if nargin<7 % no derivative mode lp = y.*lg - (y+r).*lgr - lZ; dlp = {}; d2lp = {}; d3lp = {}; % return arguments if nargout>1 dlp = y.*dlg - (y+r).*a.*dlg; % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = y.*d2lg - (y+r).*(a.*d2lg + da.*dlg); if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = y.*d3lg - (y+r).*(a.*d3lg + da.*(3*d2lg +(1-2*a).*dlg.*dlg)); end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode b = (y+r)./(elg+r); lp_dhyp = r*(1+log(r)-lgr-b-psi(r)+psi(y+r)); dlp_dhyp = r*dlg.*a.*(b-1); % first derivative d2lp_dhyp = r*((d2lg.*a+dlg.*da).*(b-1)-(dlg.*a).^2.*b); % and also second varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargin<7 % no derivative mode % Since we are not aware of an analytical expression of the integral, % hence we use quadrature. varargout = cell(1,nargout); [varargout{:}] = lik_epquad({@likNegBinom,link},hyp,y,mu,s2); else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' error('infVB not supported') end end % compute the log intensity using the inverse link function function varargout = g(f,link) varargout = cell(nargout, 1); % allocate the right number of output arguments if strcmp(link,'exp') [varargout{:}] = glm_invlink_exp(f); else [varargout{:}] = glm_invlink_logistic(f); endgpml-matlab-v3.5-2014-12-08/lik/likGauss.m000664 000765 000024 00000006631 12526350575 017622 0ustar00carlstaff000000 000000 function [varargout] = likGauss(hyp, y, mu, s2, inf, i) % likGauss - Gaussian likelihood function for regression. The expression for the % likelihood is % likGauss(t) = exp(-(t-y)^2/2*sn^2) / sqrt(2*pi*sn^2), % where y is the mean and sn is the standard deviation. % % The hyperparameters are: % % hyp = [ log(sn) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2015-05-18. % File automatically generated using noweb. % % See also LIKFUNCTIONS.M. if nargin<3, varargout = {'1'}; return; end % report number of hyperparameters sn2 = exp(2*hyp); if nargin<5 % prediction mode if inf is not present if isempty(y), y = zeros(size(mu)); end s2zero = 1; if nargin>3&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability lp = -(y-mu).^2./sn2/2-log(2*pi*sn2)/2; s2 = 0; else lp = likGauss(hyp, y, mu, s2, 'infEP'); % prediction end ymu = {}; ys2 = {}; if nargout>1 ymu = mu; % first y moment if nargout>2 ys2 = s2 + sn2; % second y moment end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' if nargin<6 % no derivative mode if isempty(y), y=0; end ymmu = y-mu; dlp = {}; d2lp = {}; d3lp = {}; lp = -ymmu.^2/(2*sn2) - log(2*pi*sn2)/2; if nargout>1 dlp = ymmu/sn2; % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = -ones(size(ymmu))/sn2; if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = zeros(size(ymmu)); end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode lp_dhyp = (y-mu).^2/sn2 - 1; % derivative of log likelihood w.r.t. hypers dlp_dhyp = 2*(mu-y)/sn2; % first derivative, d2lp_dhyp = 2*ones(size(mu))/sn2; % and also of the second mu derivative varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargin<6 % no derivative mode lZ = -(y-mu).^2./(sn2+s2)/2 - log(2*pi*(sn2+s2))/2; % log part function dlZ = {}; d2lZ = {}; if nargout>1 dlZ = (y-mu)./(sn2+s2); % 1st derivative w.r.t. mean if nargout>2 d2lZ = -1./(sn2+s2); % 2nd derivative w.r.t. mean end end varargout = {lZ,dlZ,d2lZ}; else % derivative mode dlZhyp = ((y-mu).^2./(sn2+s2)-1) ./ (1+s2./sn2); % deriv. w.r.t. hyp.lik varargout = {dlZhyp}; end case 'infVB' % variational lower site bound % t(s) = exp(-(y-s)^2/2sn2)/sqrt(2*pi*sn2) % the bound has the form: (b+z/ga)*f - f.^2/(2*ga) - h(ga)/2 n = numel(s2); b = zeros(n,1); y = y.*ones(n,1); z = y; varargout = {b,z}; end end gpml-matlab-v3.5-2014-12-08/lik/likGamma.m000644 000765 000024 00000010735 12513017737 017554 0ustar00carlstaff000000 000000 function [varargout] = likGamma(link, hyp, y, mu, s2, inf, i) % likGamma - Gamma likelihood function for strictly positive data y. The % expression for the likelihood is % likGamma(f) = al^al*y^(al-1)/gamma(al) * exp(-y*al/mu) / mu^al with % mean=mu and variance=mu^2/al where mu = g(f) is the Gamma intensity, f is a % Gaussian process, y is the strictly positive data. Hence, we have -- with % log(Zy) = log(gamma(al)) - al*log(al) + (1-al)*log(y) % llik(f) = log(likGamma(f)) = -al*( log(g(f)) + y/g(f) ) - log(Zy). % The larger one chooses al, the stronger the likelihood resembles a Gaussian % since skewness = 2/sqrt(al) and kurtosis = 6/al. % % We provide two inverse link functions 'exp' and 'logistic': % g(f) = exp(f) and g(f) = log(1+exp(f))). % The link functions are located at util/glm_invlink_*.m. % % Note that for neither link function the likelihood lik(f) is log concave. % % The hyperparameters are: % % hyp = [ log(al) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % See also LIKFUNCTIONS.M. % % Copyright (c) by Hannes Nickisch, 2013-10-16. if nargin<4, varargout = {'1'}; return; end % report number of hyperparameters al = exp(hyp); if nargin<6 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>4&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability lg = g(mu,link); lZy = gammaln(al) - al*log(al) + (1-al)*log(y); % normalisation constant lp = -al*(lg+y./exp(lg)) - lZy; else lp = likGamma(link, hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 % compute y moments by quadrature n = max([length(y),length(mu),length(s2)]); on = ones(n,1); N = 20; [t,w] = gauher(N); oN = ones(1,N); lw = ones(n,1)*log(w'); mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only lg = g(sig*t'+mu*oN,link); ymu = exp(logsumexp2(lg+lw)); % first moment using Gaussian-Hermite quad if nargout>2 elg = exp(lg); yv = elg.^2/al; % second y moment from Gamma distribution ys2 = (yv+(elg-ymu*oN).^2)*w; end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' [lg,dlg,d2lg,d3lg] = g(mu,link); elg = exp(lg); if nargin<7 % no derivative mode lZy = gammaln(al) - al*log(al) + (1-al)*log(y); % normalisation constant lp = -al*(lg+y./elg) - lZy; dlp = {}; d2lp = {}; d3lp = {}; % return arguments if nargout>1 dlp = -al*dlg.*(1-y./elg); % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = -al*d2lg.*(1-y./elg) - al*dlg.*dlg.*y./elg; if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = -al*d3lg.*(1-y./elg) + al*dlg.*(dlg.*dlg-3*d2lg).*y./elg; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode dlZy = al*psi(0,al) - al*(log(al) + 1 + log(y)); lp_dhyp = -al*(lg+y./elg) - dlZy; % derivative of log likelihood w.r.t. al dlp_dhyp = -al*dlg.*(1-y./elg); % first derivative d2lp_dhyp = -al*d2lg.*(1-y./elg) - al*dlg.*dlg.*y./elg; % and also second varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargin<7 % no derivative mode % Since we are not aware of an analytical expression of the integral, % we use quadrature. varargout = cell(1,nargout); [varargout{:}] = lik_epquad({@likGamma,link},hyp,y,mu,s2); else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' error('infVB not supported') end end % compute the log intensity using the inverse link function function varargout = g(f,link) varargout = cell(nargout, 1); % allocate the right number of output arguments if strcmp(link,'exp') [varargout{:}] = glm_invlink_exp(f); else [varargout{:}] = glm_invlink_logistic(f); endgpml-matlab-v3.5-2014-12-08/lik/likErf.m000644 000765 000024 00000005637 12513017763 017252 0ustar00carlstaff000000 000000 function [varargout] = likErf(hyp, y, mu, s2, inf, i) % likErf - Error function or cumulative Gaussian likelihood function for binary % classification or probit regression. The expression for the likelihood is % likErf(t) = (1+erf(t/sqrt(2)))/2 = normcdf(t). % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2014-03-19. % % See also LIKFUNCTIONS.M. if nargin<3, varargout = {'0'}; return; end % report number of hyperparameters if nargin>1, y = sign(y); y(y==0) = 1; else y = 1; end % allow only +/- 1 values if numel(y)==0, y = 1; end if nargin<5 % prediction mode if inf is not present y = y.*ones(size(mu)); % make y a vector s2zero = 1; if nargin>3&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability evaluation lp = logphi(y.*mu); else % prediction lp = likErf(hyp, y, mu, s2, 'infEP'); end p = exp(lp); ymu = {}; ys2 = {}; if nargout>1 ymu = 2*p-1; % first y moment if nargout>2 ys2 = 4*p.*(1-p); % second y moment end end varargout = {lp,ymu,ys2}; else % inference mode switch inf case 'infLaplace' if nargin<6 % no derivative mode f = mu; yf = y.*f; % product latents and labels varargout = cell(nargout,1); [varargout{:}] = logphi(yf); % query logphi if nargout>1 varargout{2} = y.*varargout{2}; if nargout>3, varargout{4} = y.*varargout{4}; end end else % derivative mode varargout = {[],[],[]}; % derivative w.r.t. hypers end case 'infEP' if nargin<6 % no derivative mode z = mu./sqrt(1+s2); dlZ = {}; d2lZ = {}; if numel(y)>0, z = z.*y; end if nargout<=1, lZ = logphi(z); % log part function else [lZ,n_p] = logphi(z); end if nargout>1 if numel(y)==0, y=1; end dlZ = y.*n_p./sqrt(1+s2); % 1st derivative wrt mean if nargout>2, d2lZ = -n_p.*(z+n_p)./(1+s2); end % 2nd derivative end varargout = {lZ,dlZ,d2lZ}; else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' error('infVB not supported') end endgpml-matlab-v3.5-2014-12-08/lik/likLogistic.m000644 000765 000024 00000013771 12513017627 020310 0ustar00carlstaff000000 000000 function [varargout] = likLogistic(hyp, y, mu, s2, inf, i) % likLogistic - logistic function for binary classification or logit regression. % The expression for the likelihood is % likLogistic(t) = 1./(1+exp(-t)). % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. The moments % \int f^k likLogistic(y,f) N(f|mu,var) df are calculated via a cumulative % Gaussian scale mixture approximation. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-09-02. % % See also LIKFUNCTIONS.M. if nargin<3, varargout = {'0'}; return; end % report number of hyperparameters if nargin>1, y = sign(y); y(y==0) = 1; else y = 1; end % allow only +/- 1 values if numel(y)==0, y = 1; end if nargin<5 % prediction mode if inf is not present y = y.*ones(size(mu)); % make y a vector s2zero = 1; if nargin>3&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability evaluation yf = y.*mu; % product latents and labels lp = yf; ok = -351 p = exp(lp); ymu = 2*p-1; % first y moment if nargout>2 ys2 = 4*p.*(1-p); % second y moment end end varargout = {lp,ymu,ys2}; else % inference mode switch inf case 'infLaplace' if nargin<6 % no derivative mode f = mu; yf = y.*f; s = -yf; % product latents and labels dlp = {}; d2lp = {}; d3lp = {}; % return arguments ps = max(0,s); lp = -(ps+log(exp(-ps)+exp(s-ps))); % lp = -(log(1+exp(s))) if nargout>1 % first derivatives s = min(0,f); p = exp(s)./(exp(s)+exp(s-f)); % p = 1./(1+exp(-f)) dlp = (y+1)/2-p; % derivative of log likelihood if nargout>2 % 2nd derivative of log likelihood d2lp = -exp(2*s-f)./(exp(s)+exp(s-f)).^2; if nargout>3 % 3rd derivative of log likelihood d3lp = 2*d2lp.*(0.5-p); end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode varargout = {[],[],[]}; % derivative w.r.t. hypers end case 'infEP' if nargin<6 % no derivative mode y = y.*ones(size(mu)); % make y a vector % likLogistic(t) \approx 1/2 + \sum_{i=1}^5 (c_i/2) erf(lam_i/sqrt(2)t) lam = sqrt(2)*[0.44 0.41 0.40 0.39 0.36]; % approx coeffs lam_i and c_i c = [1.146480988574439e+02; -1.508871030070582e+03; 2.676085036831241e+03; -1.356294962039222e+03; 7.543285642111850e+01 ]; [lZc,dlZc,d2lZc] = likErf([], y*ones(1,5), mu*lam, s2*(lam.^2), inf); lZ = log_expA_x(lZc,c); % A=lZc, B=dlZc, d=c.*lam', lZ=log(exp(A)*c) dlZ = expABz_expAx(lZc, c, dlZc, c.*lam'); % ((exp(A).*B)*d)./(exp(A)*c) % d2lZ = ((exp(A).*Z)*e)./(exp(A)*c) - dlZ.^2 where e = c.*(lam.^2)' d2lZ = expABz_expAx(lZc, c, dlZc.^2+d2lZc, c.*(lam.^2)') - dlZ.^2; % The scale mixture approximation does not capture the correct asymptotic % behavior; we have linear decay instead of quadratic decay as suggested % by the scale mixture approximation. By observing that for large values % of -f*y ln(p(y|f)) for likLogistic is linear in f with slope y, we are % able to analytically integrate the tail region. val = abs(mu)-196/200*s2-4; % empirically determined bound at val==0 lam = 1./(1+exp(-10*val)); % interpolation weights lZtail = min(s2/2-abs(mu),-0.1); % apply the same to p(y|f) = 1 - p(-y|f) dlZtail = -sign(mu); d2lZtail = zeros(size(mu)); id = y.*mu>0; lZtail(id) = log(1-exp(lZtail(id))); % label and mean agree dlZtail(id) = 0; lZ = (1-lam).* lZ + lam.* lZtail; % interpolate between scale .. dlZ = (1-lam).* dlZ + lam.* dlZtail; % .. mixture and .. d2lZ = (1-lam).*d2lZ + lam.*d2lZtail; % .. tail approximation varargout = {lZ,dlZ,d2lZ}; else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' % variational lower site bound % using -log(1+exp(-s)) = s/2 -log( 2*cosh(s/2) ); % the bound has the form: (b+z/ga)*f - f.^2/(2*ga) - h(ga)/2 n = numel(s2); b = (y/2).*ones(n,1); z = zeros(size(b)); varargout = {b,z}; end end % computes y = log( exp(A)*x ) in a numerically safe way by subtracting the % maximal value in each row to avoid cancelation after taking the exp function y = log_expA_x(A,x) N = size(A,2); maxA = max(A,[],2); % number of columns, max over columns y = log(exp(A-maxA*ones(1,N))*x) + maxA; % exp(A) = exp(A-max(A))*exp(max(A)) % computes y = ( (exp(A).*B)*z ) ./ ( exp(A)*x ) in a numerically safe way % The function is not general in the sense that it yields correct values for % all types of inputs. We assume that the values are close together. function y = expABz_expAx(A,x,B,z) N = size(A,2); maxA = max(A,[],2); % number of columns, max over columns A = A-maxA*ones(1,N); % subtract maximum value y = ( (exp(A).*B)*z ) ./ ( exp(A)*x ); gpml-matlab-v3.5-2014-12-08/lik/likExp.m000644 000765 000024 00000002600 12233776331 017257 0ustar00carlstaff000000 000000 function [varargout] = likExp(link, hyp, varargin) % likExp - Exponential likelihood function for strictly positive data y. The % expression for the likelihood is % likExp(f) = exp(-y/mu) / mu with % mean=mu and variance=mu^2 where mu = g(f) is the intensity, f is a % Gaussian process, y is the strictly positive data. Hence, we have % llik(f) = log(likExp(f)) = -y/g(f) - log(g(f)). % % Internally, the function is in fact a wrapper around likGamma with hyper % parameter al=1. % % We provide two inverse link functions 'exp' and 'logistic': % g(f) = exp(f) and g(f) = log(1+exp(f))). % The link functions are located at util/glm_invlink_*.m. % Note that for the 'exp' intensity the likelihood lik(f) is log concave. % % The hyperparameters are: % % hyp = [ ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % See also LIKFUNCTIONS.M, LIKGAMMA.M. % % Copyright (c) by Hannes Nickisch, 2013-10-29. if nargin<4, varargout = {'0'}; return; end % report number of hyperparameters varargout = cell(nargout, 1); % allocate the right number of output arguments for j=1:nargout, varargout{j} = []; end % derivative output if nargin<=6, [varargout{:}] = likGamma(link,0,varargin{:}); end % log(al) = 0gpml-matlab-v3.5-2014-12-08/prior/priorLogNormal.m000664 000765 000024 00000001666 12403654516 021361 0ustar00carlstaff000000 000000 function [lp,dlp]=priorLogNormal(mu,s2,x) % Univariate Log-normal hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = exp(-(log(x)-mu)^2/(2*s2)) / (sqrt(2*pi*s2)*x) % % where mu(1x1) is the log scale parameter, s2(1x1) is the shape parameter % and x(1xN) contains query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-08. % % See also PRIORDISTRIBUTIONS.M. if nargin<2, error('mu and s2 parameters need to be provided'), end if ~(isscalar(mu)&&isscalar(s2)) error('mu and s2 parameters need to be scalars'), end if nargin<3, lp = exp(sqrt(s2)*randn+mu); return, end % return a sample lx = log(x); lp = -(lx-mu).^2/(2*s2) - log(2*pi*s2)/2 - lx; dlp = -(lx-mu+s2)./(x*s2); lp(x<0) = -inf; dlp(x<0) = 0;gpml-matlab-v3.5-2014-12-08/prior/priorSmoothBox1.m000664 000765 000024 00000003353 12505360730 021460 0ustar00carlstaff000000 000000 function [lp,dlp] = priorSmoothBox1(a,b,eta,x) % Univariate smoothed box prior distribution with linear decay in the log domain % and infinite support over the whole real axis. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = 1/w*sigmoid(eta*(x-a))*(1-sigmoid(eta*(x-b))), % where w = abs(b-a) and sigmoid(z) = 1/(1+exp(-z)) % % a(1x1) is the lower bound parameter, b(1x1) is the upper bound parameter, % eta(1x1)>0 is the slope parameter and x(1xN) contains query hyperparameters % for prior evaluation. Larger values of eta make the distribution more % box-like. % % The distribution p(x) has mean and variance given by: % mu = (a+b)/2 and s2 = w^2/(1-exp(-2*g))*(1+(pi/g)^2)/12, g = eta*w/2. % % /------------\ % / \ % -------- | | --------> x % a b % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Jose Vallet and Hannes Nickisch, 2015-03-27. % % See also PRIORDISTRIBUTIONS.M. if nargin<3, error('a, b and eta parameters need to be provided'), end if b<=a, error('b must be greater than a.'), end if ~(isscalar(a)&&isscalar(b)&&isscalar(eta)) error('a, b and eta parameters need to be scalar values') end if nargin<4 % inverse sampling u = exp((b-a)*eta*rand()); lp = log((u-1)/(exp(-eta*a)-u*exp(-eta*b)))/eta; return end [lpa,dlpa] = logr(eta*(x-a)); [lpb,dlpb] = logr(-eta*(x-b)); lp = lpa + lpb - log(b-a) + log(1-exp((a-b)*eta)); dlp = eta*(dlpa - dlpb); % r(z) = 1/(1+exp(-z)), log(r(z)) = -log(1+exp(-z)) function [lr,dlr] = logr(z) lr = z; ok = -351 % both mu/s2 scalar => inflate D = size(x,1); mu = mu*ones(D,1); s = s*ones(D,1); lds = D*lds; end if ~(ndims(x)==2 && size(x,1)==D), error('x needs to be (Dxn)'), end oN = ones(1,size(x,2)); if size(s,2)==D xm = x-mu*oN; xs = s\xm; else xm = x-mu*oN; xs = xm./(s*oN); end dlp = -sqrt(2)*sign(xs); lp = -sqrt(2)*sum(abs(xs),1) - D*log(2)/2 - lds; if size(s,2)==D, dlp = s'\dlp; else dlp = dlp./(s*oN); endgpml-matlab-v3.5-2014-12-08/prior/priorLaplace.m000664 000765 000024 00000001476 12403653021 021015 0ustar00carlstaff000000 000000 function [lp,dlp] = priorLaplace(mu,s2,x) % Univariate Laplacian hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = exp(-abs(x-mu)/b)/(2*b), where b = sqrt(s2/2), % % mu(1x1) is the mean parameter, s2(1x1) is the variance parameter and % x(1xN) contains query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-09. % % See also PRIORDISTRIBUTIONS.M. if nargin<3 % return a sample u = rand-1/2; lp = mu - sqrt(s2/2)*sign(u)*log(1-2*abs(u)); return end b = sqrt(s2/2); lp = -abs(x-mu)/b - log(2*b); dlp = -sign(x-mu)/b;gpml-matlab-v3.5-2014-12-08/prior/priorWeibull.m000664 000765 000024 00000001654 12417540104 021057 0ustar00carlstaff000000 000000 function [lp,dlp] = priorWeibull(lam,k,x) % Univariate Weibull hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = (k/lam) * (x/lam)^(k-1) * exp(-(x/lam)^k) % % where lam(1x1) is the scale parameter, k(1x1) is the scale parameter % and x(1xN) contains query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-08. % % See also PRIORDISTRIBUTIONS.M. if nargin<2, error('lam and k parameters need to be provided'), end if ~(isscalar(lam)&&isscalar(k)) error('lam and k parameters need to be scalars'), end if nargin<3,lp = lam*(-log(rand))^(1/k); return, end % return a sample lp = log(k/lam) + (k-1)*log(x/lam) - (x/lam).^k; dlp = (k-1)./x - (k/lam)*(x/lam).^(k-1); lp(x<0) = -inf; dlp(x<0) = 0;gpml-matlab-v3.5-2014-12-08/prior/priorSmoothBox2.m000664 000765 000024 00000003762 12505273642 021472 0ustar00carlstaff000000 000000 function [lp,dlp] = priorSmoothBox2(a,b,eta,x) % Univariate smoothed box prior distribution with quadratic decay in the log % domain and infinite support over the whole real axis. % The plateau pdf is built by cutting a Gaussian into two parts and % inserting a uniform distribution from a to b. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % / G(x|a,s^2), x<=a, % p(x) = 1/(w*(1/eta+1)) * | 1 , a0 is the slope parameter and x(1xN) contains query hyperparameters % for prior evaluation. Larger values of eta make the distribution more % box-like. % % The distribution p(x) has mean and variance given by: % mu = (a+b)/2 and s2 = (w/2)^2*(eta^3/3+eta^2+4*eta/pi+2/pi)/(eta^3+eta^2). % % /------------\ % / \ % -------- | | --------> x % a b % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Jose Vallet and Hannes Nickisch, 2015-03-27. % % See also PRIORDISTRIBUTIONS.M. if nargin<3, error('a, b and eta parameters need to be provided'), end if b<=a, error('b must be greater than a.'), end if ~(isscalar(a)&&isscalar(b)&&isscalar(eta)) error('a, b and eta parameters need to be scalar values') end w = abs(b-a); sab = w/(eta*sqrt(2*pi)); % box width and boundary slope if nargin<4 % return a sample if randb; lp(i) = lp(i) - (x(i)-b).^2/(2*sab^2); dlp(i) = (b-x(i))/sab^2;gpml-matlab-v3.5-2014-12-08/prior/priorDeltaMulti.m000664 000765 000024 00000001032 12440604653 021514 0ustar00carlstaff000000 000000 function [lp,dlp] = priorDeltaMulti(x) % Dummy hyperparameter prior distribution to fix the value of a hyperparameter. % The function is not intended to be evaluated but exists merely to make % the user aware of the possibility to use it. The function is equivalent % to priorClampedMulti. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-12-06. % % See also PRIORDISTRIBUTIONS.M. error('The function is not intended to be called directly.')gpml-matlab-v3.5-2014-12-08/prior/priorInvGauss.m000664 000765 000024 00000002174 12417540127 021216 0ustar00carlstaff000000 000000 function [lp,dlp] = priorInvGauss(mu,lam,x) % Univariate Inverse Gaussian hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = exp(-lam*(x-mu)^2/(2*mu^2*x)) / sqrt(2*pi*x^3/lam) % % where mu(1x1) is the mean parameter, lam(1x1) is the scale parameter % and x(1xN) contains query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-08. % % See also PRIORDISTRIBUTIONS.M. if nargin<2, error('mu and lam parameters need to be provided'), end if ~(isscalar(mu)&&isscalar(lam)) error('mu and lam parameters need to be scalars'),end if nargin<3 % return a sample n = randn; y = n*n; r = mu + mu*(mu*y-sqrt(4*mu*lam*y+mu^2*y^2))/(2*lam); z = rand; if z<=mu/(mu+r) lp = r; else lp = mu^2/r; end return end lp = -lam*(x-mu).^2./(2*mu^2*x) - log(2*pi*x.^3/lam)/2; q = (x-mu)./x; dlp = -lam*q.*(2-q)/(2*mu^2) - 3./(2*x); lp(x<0) = -inf; dlp(x<0) = 0;gpml-matlab-v3.5-2014-12-08/prior/priorDelta.m000664 000765 000024 00000001020 12440604620 020470 0ustar00carlstaff000000 000000 function [lp,dlp] = priorDelta(x) % Dummy hyperparameter prior distribution to fix the value of a hyperparameter. % The function is not intended to be evaluated but exists merely to make % the user aware of the possibility to use it. The function is equivalent % to priorClamped. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-12-06. % % See also PRIORDISTRIBUTIONS.M. error('The function is not intended to be called directly.')gpml-matlab-v3.5-2014-12-08/prior/priorGaussMulti.m000664 000765 000024 00000003466 12417541265 021565 0ustar00carlstaff000000 000000 function [lp,dlp] = priorGaussMulti(mu,s2,x) % Multivariate Gaussian hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = exp(-r2/2) / sqrt(det(2*pi*s2)) where r2(x) = (x-mu)'*inv(s2)*(x-mu), % % mu(Dx1) is the mean parameter, s2(Dx1) or s2(DxD) is the variance parameter % and x(DxN) contains query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-12. % % See also PRIORDISTRIBUTIONS.M, PRIORGAUSS.M. if nargin<2, error('mu and s2 parameters need to be provided'), end if ndims(mu)~=2 || size(mu,2)~=1, error('mu needs to be (Dx1)'), end D = size(mu,1); s2_ok = ndims(s2)==2 && all(size(s2)==[D,1] | size(s2)==[D,D]); if ~s2_ok, error('s2 needs to be (DxD) or (Dx1)'), end if size(s2,2)==D % full multivariate case s = chol(s2)'; lds = sum(log(diag(s))); % lds = log(det(s2))/2 else % diagonal covariance s = sqrt(s2); lds = sum(log(s)); end if nargin<3 % return a random sample lp = randn(D,1); % unit sample if size(s,2)==D, lp = s*lp+mu; else lp = s.*lp+mu; end % affine transformation return end if D==1 && size(x,1)>1 % both mu/s2 scalar => inflate D = size(x,1); mu = mu*ones(D,1); s = s*ones(D,1); lds = D*lds; end if ~(ndims(x)==2 && size(x,1)==D), error('x needs to be (Dxn)'), end oN = ones(1,size(x,2)); if size(s,2)==D xs = s\(x-mu*oN); dlp = -s'\xs; else xs = (x-mu*oN)./(s*oN); dlp = -xs./(s*oN); end lp = -sum(xs.^2,1)/2 - D*log(2*pi)/2 - lds;gpml-matlab-v3.5-2014-12-08/prior/priorGamma.m000664 000765 000024 00000002515 12417540212 020473 0ustar00carlstaff000000 000000 function [lp,dlp] = priorGamma(k,t,x) % Univariate Gamma hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = exp(x/t)/gamma(k)*x^(k-1)/t^k % % where k(1x1) is the shape parameter, t(1x1) is the scale parameter % and x(1xN) contains query hyperparameters for prior evaluation. % % Sampling is done using the algorithm from p. 53 of the paper Generating gamma % variates by a modified rejection technique by J.H. Ahrens and U. Dieter, 1982, % ACM, 25, 47–54. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-08. % % See also PRIORDISTRIBUTIONS.M. if nargin<2, error('k and t parameters need to be provided'), end if ~(isscalar(k)&&isscalar(t)) error('k and t parameters need to be scalars'), end if nargin<3 m = 1; d = k-floor(k); % fractional part v0 = exp(1)/(exp(1)+d); while true v = rand(3,1); if v(1)<=v0 r = v(2)^(1/d); s = v(3)*r^(d-1); else r = 1-log(v(2)); s = v(3)*exp(-r); end if s<=r^(d-1)*exp(-r), break, end m = m+1; end u = rand(floor(k),1); lp = t*(r-sum(log(u))); return end lx = log(x); lp = -gammaln(k) - k*log(t) + (k-1)*lx - x/t; dlp = (k-1)./x - 1/t; lp(x<0) = -inf; dlp(x<0) = 0;gpml-matlab-v3.5-2014-12-08/prior/priorGauss.m000664 000765 000024 00000001565 12440605070 020537 0ustar00carlstaff000000 000000 function [lp,dlp] = priorGauss(mu,s2,x) % Univariate Gaussian hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = exp(-(x-mu)^2/(2*s2)) / sqrt(2*pi*s2), where % % mu(1x1) is the mean parameter, s2(1x1) is the variance parameter and % x(1xN) contains query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-09. % % See also PRIORDISTRIBUTIONS.M. if nargin<2, error('mu and s2 parameters need to be provided'), end if ~(isscalar(mu)&&isscalar(s2)) error('mu and s2 parameters need to be scalars') end if nargin<3, lp = sqrt(s2)*randn+mu; return, end % return a sample lp = -(x-mu).^2/(2*s2) - log(2*pi*s2)/2; dlp = -(x-mu)/s2;gpml-matlab-v3.5-2014-12-08/prior/priorClamped.m000664 000765 000024 00000001020 12440604215 021004 0ustar00carlstaff000000 000000 function [lp,dlp] = priorClamped(x) % Dummy hyperparameter prior distribution to fix the value of a hyperparameter. % The function is not intended to be evaluated but exists merely to make % the user aware of the possibility to use it. The function is equivalent % to priorDelta. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-12-06. % % See also PRIORDISTRIBUTIONS.M. error('The function is not intended to be called directly.')gpml-matlab-v3.5-2014-12-08/prior/priorMix.m000664 000765 000024 00000002362 12403653021 020204 0ustar00carlstaff000000 000000 function [lp,dlp] = priorMix(w,priors,x) % Mixture of Univariate or Multivariate hyperparameter prior distributions. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = sum_i w_i p_i(x), % % w(bx1) is the mixture weight vector (nonnegative, normalised to add up to 1), % priors(bx1) is a cell array containing the priors to be mixed and x(1xN) % contains query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-08. % % See also PRIORDISTRIBUTIONS.M. if nargin<2, error('specification of w and priors required'), end n = numel(w); w = w(:); if abs(sum(w)-1)>1e-9, error('w needs to sum up to 1'), end if any(w<0), error('w needs to be nonnegative'), end for i=1:n, if ~iscell(priors{i}), priors{i} = {priors{i}}; end, end % sample if nargin<3, i = 1+nnz(rand>cumsum(w)); lp = feval(priors{i}{:}); return, end lpi = zeros(numel(x),n); dlpi = zeros(numel(x),n); for i=1:n, [lpi(:,i),dlpi(:,i)] = feval(priors{i}{:},x(:)); end mx = max(lpi,[],2); lp = log(exp(lpi-mx*ones(1,n))*w) + mx; pi = exp(lpi); dlp = ((pi.*dlpi)*w)./(pi*w);gpml-matlab-v3.5-2014-12-08/prior/priorTransform.m000664 000765 000024 00000002256 12403654460 021434 0ustar00carlstaff000000 000000 function [lp,dlp] = priorTransform(g,dg,ig,prior,x) % Transformed univariate hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % prior(y), where y = g(x), % % g is the transformation function (monotonically increasing and invertable), % dg is its derivative, ig is its inverse, prior is the distribution to be % transformed and x(1xN) contains query hyperparameters for prior evaluation. % Note that p(x) = prior(g(x)) is not necessarily normalised w.r.t. x. % % For example, to put a nonnegative gamma prior on the lenthscale % parameter ell of covSEiso, one can use % {@priorTransform,@exp,@exp,@log,{@priorGamma,k,t}} % since ell is represented in the log domain. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-08. % % See also PRIORDISTRIBUTIONS.M. if nargin<4, error('g, dg, ig and prior parameters need to be provided'), end if nargin<5, lp = ig(feval(prior{:})); return, end % apply inverse sampling if ~iscell(prior), prior = {prior}; end [lp,dlp] = feval(prior{:},g(x)); dlp = dlp.*dg(x);gpml-matlab-v3.5-2014-12-08/prior/priorT.m000664 000765 000024 00000002170 12417736504 017664 0ustar00carlstaff000000 000000 function [lp,dlp] = priorT(mu,s2,nu,x) % Univariate Student's t hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = ( 1 + (x-mu)^2/(s2*(nu-2)) )^(-(nu+1)/2) / Z, where % Z = gamma(nu/2) * sqrt(pi*(nu-2)*s2) / gamma((nu+1)/2), % % mu(1x1) is the mean parameter, s2(1x1) is the variance parameter, % nu(1x1) > 2 is the degrees of freedom parameter and x(1xN) contains query % hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-10-16. % % See also PRIORDISTRIBUTIONS.M. if nargin<3, error('mu, s2 and nu parameters need to be provided'), end if ~(isscalar(mu)&&isscalar(s2)&&isscalar(nu)) error('mu, s2 and nu parameters need to be scalars') end if nargin<4, lp = mu + sqrt(s2*(nu-2)/priorGamma(nu/2,2))*randn; return, end lZ = gammaln((nu+1)/2)-gammaln(nu/2)-log(pi*(nu-2)*s2)/2; lp = -(nu+1)/2*log(1+(x-mu).^2/(s2*(nu-2))) + lZ; dlp = (nu+1)*(mu-x)./(mu^2-2*mu*x+s2*(nu-2)+x.^2);gpml-matlab-v3.5-2014-12-08/prior/priorTMulti.m000664 000765 000024 00000004270 12417736501 020677 0ustar00carlstaff000000 000000 function [lp,dlp] = priorTMulti(mu,s2,nu,x) % Multivariate Laplace hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = ( 1 + r2/(nu-2) )^(-(nu+D)/2) / Z, where % Z = gamma(nu/2) * sqrt(det(pi*(nu-2)*s2)) / gamma((nu+D)/2), and % r2(x) = (x-mu)'*inv(s2)*(x-mu), % % further mu(Dx1) is the mean parameter, s2(Dx1) or s2(DxD) is the variance % parameter, nu(1x1) > 2 is the degrees of freedom parameter and x(DxN) contains % query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Hannes Nickisch, 2014-10-16. % % See also PRIORDISTRIBUTIONS.M, PRIORLAPLACE.M. if nargin<3, error('mu, s2 and nu parameters need to be provided'), end if ndims(mu)~=2 || size(mu,2)~=1, error('mu needs to be (Dx1)'), end D = size(mu,1); s2_ok = ndims(s2)==2 && all(size(s2)==[D,1] | size(s2)==[D,D]); if ~s2_ok, error('s2 needs to be (DxD) or (Dx1)'), end if size(s2,2)==D % full multivariate case s = chol(s2)'; lds = sum(log(diag(s))); % lds = log(det(s2))/2 else % diagonal covariance s = sqrt(s2); lds = sum(log(s)); end if ~isscalar(nu), error('nu parameter needs to be a scalar'), end if nargin<4 % return a random sample lp = randn(D,1); for d=1:D, lp(d) = sqrt((nu-2)/priorGamma(nu/2,2)) * lp(d); end % unit sample if size(s,2)==D, lp = s*lp+mu; else lp = s.*lp+mu; end % affine transformation return end if D==1 && size(x,1)>1 % both mu/s2 scalar => inflate D = size(x,1); mu = mu*ones(D,1); s = s*ones(D,1); lds = D*lds; end if ~(ndims(x)==2 && size(x,1)==D), error('x needs to be (Dxn)'), end oN = ones(1,size(x,2)); oD = ones(D,1); if size(s,2)==D xm = x-mu*oN; xs = s\xm; else xm = x-mu*oN; xs = xm./(s*oN); end lZ = gammaln((nu+D)/2) - gammaln(nu/2) - D*log(pi*(nu-2))/2 - lds; r2 = sum(xs.^2,1); lp = -(nu+D)/2*log(1+r2/(nu-2)) + lZ; dlp = -(nu+D)/(nu-2)*xs./(oD*(1+r2/(nu-2))); if size(s,2)==D, dlp = s'\dlp; else dlp = dlp./(s*oN); endgpml-matlab-v3.5-2014-12-08/prior/priorClampedMulti.m000664 000765 000024 00000001032 12440604340 022021 0ustar00carlstaff000000 000000 function [lp,dlp] = priorClampedMulti(x) % Dummy hyperparameter prior distribution to fix the value of a hyperparameter. % The function is not intended to be evaluated but exists merely to make % the user aware of the possibility to use it. The function is equivalent % to priorDeltaMulti. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-12-06. % % See also PRIORDISTRIBUTIONS.M. error('The function is not intended to be called directly.')gpml-matlab-v3.5-2014-12-08/inf/infEP.m000644 000765 000024 00000013640 12443115761 017024 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infEP(hyp, mean, cov, lik, x, y) % Expectation Propagation approximation to the posterior Gaussian Process. % The function takes a specified covariance function (see covFunctions.m) and % likelihood function (see likFunctions.m), and is designed to be used with % gp.m. See also infMethods.m. In the EP algorithm, the sites are % updated in random order, for better performance when cases are ordered % according to the targets. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2013-09-13. % % See also INFMETHODS.M. persistent last_ttau last_tnu % keep tilde parameters between calls tol = 1e-4; max_sweep = 10; min_sweep = 2; % tolerance to stop EP iterations inf = 'infEP'; n = size(x,1); if isnumeric(cov), K = cov; % use provided covariance matrix else K = feval(cov{:}, hyp.cov, x); end % evaluate the covariance matrix if isnumeric(mean), m = mean; % use provided mean vector else m = feval(mean{:}, hyp.mean, x); end % evaluate the mean vector % A note on naming: variables are given short but descriptive names in % accordance with Rasmussen & Williams "GPs for Machine Learning" (2006): mu % and s2 are mean and variance, nu and tau are natural parameters. A leading t % means tilde, a subscript _ni means "not i" (for cavity parameters), or _n % for a vector of cavity parameters. N(f|mu,Sigma) is the posterior. % marginal likelihood for ttau = tnu = zeros(n,1); equals n*log(2) for likCum* nlZ0 = -sum(feval(lik{:}, hyp.lik, y, m, diag(K), inf)); if any(size(last_ttau) ~= [n 1]) % find starting point for tilde parameters ttau = zeros(n,1); tnu = zeros(n,1); % init to zero if no better guess Sigma = K; % initialize Sigma and mu, the parameters of .. mu = m; nlZ = nlZ0; % .. the Gaussian posterior approximation else ttau = last_ttau; tnu = last_tnu; % try the tilde values from previous call [Sigma,mu,L,alpha,nlZ] = epComputeParams(K, y, ttau, tnu, lik, hyp, m, inf); if nlZ > nlZ0 % if zero is better .. ttau = zeros(n,1); tnu = zeros(n,1); % .. then init with zero instead Sigma = K; % initialize Sigma and mu, the parameters of .. mu = m; nlZ = nlZ0; % .. the Gaussian posterior approximation end end nlZ_old = Inf; sweep = 0; % converged, max. sweeps or min. sweeps? while (abs(nlZ-nlZ_old) > tol && sweep < max_sweep) || sweep tol error('maximum number of sweeps exceeded in function infEP') end last_ttau = ttau; last_tnu = tnu; % remember for next call post.alpha = alpha; post.sW = sqrt(ttau); post.L = L; % return posterior params if nargout>2 % do we want derivatives? dnlZ = hyp; % allocate space for derivatives tau_n = 1./diag(Sigma)-ttau; % compute the log marginal likelihood nu_n = mu./diag(Sigma)-tnu; % vectors of cavity parameters sW = sqrt(ttau); F = alpha*alpha'-repmat(sW,1,n).*solve_chol(L,diag(sW)); % covariance hypers for i=1:length(hyp.cov) dK = feval(cov{:}, hyp.cov, x, [], i); dnlZ.cov(i) = -sum(sum(F.*dK))/2; end for i = 1:numel(hyp.lik) % likelihood hypers dlik = feval(lik{:}, hyp.lik, y, nu_n./tau_n, 1./tau_n, inf, i); dnlZ.lik(i) = -sum(dlik); end [junk,dlZ] = feval(lik{:}, hyp.lik, y, nu_n./tau_n, 1./tau_n, inf);% mean hyps for i = 1:numel(hyp.mean) dm = feval(mean{:}, hyp.mean, x, i); dnlZ.mean(i) = -dlZ'*dm; end end % function to compute the parameters of the Gaussian approximation, Sigma and % mu, and the negative log marginal likelihood, nlZ, from the current site % parameters, ttau and tnu. Also returns L (useful for predictions). function [Sigma,mu,L,alpha,nlZ] = epComputeParams(K,y,ttau,tnu,lik,hyp,m,inf) n = length(y); % number of training cases sW = sqrt(ttau); % compute Sigma and mu L = chol(eye(n)+sW*sW'.*K); % L'*L=B=eye(n)+sW*K*sW V = L'\(repmat(sW,1,n).*K); Sigma = K - V'*V; alpha = tnu-sW.*solve_chol(L,sW.*(K*tnu+m)); mu = K*alpha+m; v = diag(Sigma); tau_n = 1./diag(Sigma)-ttau; % compute the log marginal likelihood nu_n = mu./diag(Sigma)-tnu; % vectors of cavity parameters lZ = feval(lik{:}, hyp.lik, y, nu_n./tau_n, 1./tau_n, inf); p = tnu-m.*ttau; q = nu_n-m.*tau_n; % auxiliary vectors nlZ = sum(log(diag(L))) - sum(lZ) - p'*Sigma*p/2 + (v'*p.^2)/2 ... - q'*((ttau./tau_n.*q-2*p).*v)/2 - sum(log(1+ttau./tau_n))/2; gpml-matlab-v3.5-2014-12-08/inf/infFITC.m000644 000765 000024 00000013077 12460506237 017252 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infFITC(hyp, mean, cov, lik, x, y) % FITC approximation to the posterior Gaussian process. The function is % equivalent to infExact with the covariance function: % Kt = Q + G; G = diag(g); g = diag(K-Q); Q = Ku'*inv(Quu)*Ku; % where Ku and Kuu are covariances w.r.t. to inducing inputs xu, snu2 = sn2/1e6 % is the noise of the inducing inputs and Quu = Kuu + snu2*eye(nu). % We fixed the standard deviation of the inducing inputs snu to be a one per mil % of the measurement noise's standard deviation sn. % The implementation exploits the Woodbury matrix identity % inv(Kt) = inv(G) - inv(G)*V'*inv(eye(nu)+V*inv(G)*V')*V*inv(G) % in order to be applicable to large datasets. The computational complexity % is O(n nu^2) where n is the number of data points x and nu the number of % inducing inputs in xu. % % The function takes a specified covariance function (see covFunctions.m) and % likelihood function (see likFunctions.m), and is designed to be used with % gp.m and in conjunction with covFITC and likGauss. % % The inducing points can be specified through 1) the 2nd covFITC parameter or % by 2) providing a hyp.xu hyperparameters. Note that 2) has priority over 1). % In case 2) is provided and derivatives dnlZ are requested, there will also be % a dnlZ.xu field allowing to optimise w.r.t. to the inducing points xu. However % the derivatives dnlZ.xu can only be computed for one of the following eight % covariance functions: cov{Matern|PP|RQ|SE}{iso|ard}. % % Copyright (c) by Ed Snelson, Carl Edward Rasmussen % and Hannes Nickisch, 2013-10-28. % % See also INFMETHODS.M, COVFITC.M. if iscell(lik), likstr = lik{1}; else likstr = lik; end if ~ischar(likstr), likstr = func2str(likstr); end if ~strcmp(likstr,'likGauss') % NOTE: no explicit call to likGauss error('Inference with inFITC only possible with Gaussian likelihood.'); end cov1 = cov{1}; if isa(cov1, 'function_handle'), cov1 = func2str(cov1); end if ~strcmp(cov1,'covFITC'); error('Only covFITC supported.'), end % check cov if isfield(hyp,'xu'), cov{3} = hyp.xu; end % hyp.xu is provided, replace cov{3} [diagK,Kuu,Ku] = feval(cov{:}, hyp.cov, x); % evaluate covariance matrix m = feval(mean{:}, hyp.mean, x); % evaluate mean vector [n, D] = size(x); nu = size(Kuu,1); cov{3} = reshape(cov{3},[],D); sn2 = exp(2*hyp.lik); % noise variance of likGauss snu2 = 1e-6*sn2; % hard coded inducing inputs noise Luu = chol(Kuu+snu2*eye(nu)); % Kuu + snu2*I = Luu'*Luu V = Luu'\Ku; % V = inv(Luu')*Ku => V'*V = Q g_sn2 = diagK + sn2 - sum(V.*V,1)'; % g + sn2 = diag(K) + sn2 - diag(Q) Lu = chol(eye(nu) + (V./repmat(g_sn2',nu,1))*V'); % Lu'*Lu=I+V*diag(1/g_sn2)*V' r = (y-m)./sqrt(g_sn2); be = Lu'\(V*(r./sqrt(g_sn2))); iKuu = solve_chol(Luu,eye(nu)); % inv(Kuu + snu2*I) = iKuu post.alpha = Luu\(Lu\be); % return the posterior parameters post.L = solve_chol(Lu*Luu,eye(nu)) - iKuu; % Sigma-inv(Kuu) post.sW = ones(n,1)/sqrt(sn2); % unused for FITC prediction with gp.m if nargout>1 % do we want the marginal likelihood nlZ = sum(log(diag(Lu))) + (sum(log(g_sn2)) + n*log(2*pi) + r'*r - be'*be)/2; if nargout>2 % do we want derivatives? dnlZ = hyp; % allocate space for derivatives al = r./sqrt(g_sn2) - (V'*(Lu\be))./g_sn2; % al = (Kt+sn2*eye(n))\(y-m) B = iKuu*Ku; w = B*al; W = Lu'\(V./repmat(g_sn2',nu,1)); for i = 1:numel(hyp.cov) [ddiagKi,dKuui,dKui] = feval(cov{:}, hyp.cov, x, [], i); % eval cov deriv R = 2*dKui-dKuui*B; v = ddiagKi - sum(R.*B,1)'; % diag part of cov deriv dnlZ.cov(i) = (ddiagKi'*(1./g_sn2)+w'*(dKuui*w-2*(dKui*al))-al'*(v.*al)... - sum(W.*W,1)*v - sum(sum((R*W').*(B*W'))) )/2; end diag_dK = 1./g_sn2 - sum(W.*W,1)' - al.*al; % diag(dnlZ/dK) dnlZ.lik = sn2*sum(diag_dK); % since snu2 is a fixed fraction of sn2, there is a covariance-like term in BWt = B*W'; % the derivative as well dKuui = 2*snu2; R = -dKuui*B; v = -sum(R.*B,1)'; % diag part of cov deriv dnlZ.lik = dnlZ.lik + (w'*dKuui*w -al'*(v.*al)... - sum(W.*W,1)*v - sum(sum((R*W').*BWt)) )/2; for i = 1:numel(hyp.mean) dnlZ.mean(i) = -feval(mean{:}, hyp.mean, x, i)'*al; end if isfield(hyp,'xu') % derivatives w.r.t. inducing points xu xu = cov{3}; cov = cov{2}; % get the non FITC part of the covariance function Kpu = cov_deriv_sq_dist(cov,hyp.cov,xu,x); % d K(xu,x ) / d D^2 Kpuu = cov_deriv_sq_dist(cov,hyp.cov,xu); % d K(xu,xu) / d D^2 if iscell(cov), covstr = cov{1}; else covstr = cov; end if ~ischar(covstr), covstr = func2str(covstr); end if numel(strfind(covstr,'iso'))>0 % characteristic length scale e = 2*exp(-2*hyp.cov(1)); else e = 2*exp(-2*hyp.cov(1:D)); end v = diag_dK-1./g_sn2; % BdK = B * ( diag(diag(dnlZ/dK)) - dnlZ/dK ) BdK = B.*repmat(v',nu,1) + BWt*W + (B*al)*al'; A = Kpu.*BdK; C = Kpuu.*(BdK*B'); C = diag(sum(C,2)-sum(A,2)) - C; dnlZ.xu = A*x*diag(e) + C*xu*diag(e); % bring in data and inducing points end end endgpml-matlab-v3.5-2014-12-08/inf/infVB.m000644 000765 000024 00000014052 12443116022 017014 0ustar00carlstaff000000 000000 function [post, nlZ, dnlZ] = infVB(hyp, mean, cov, lik, x, y, opt) % Variational approximation to the posterior Gaussian process. % The function takes a specified covariance function (see covFunctions.m) and % likelihood function (see likFunctions.m), and is designed to be used with % gp.m. See also infMethods.m. % % Minimisation of an upper bound on the negative marginal likelihood using a % sequence of infLaplace calls where the smoothed likelihood % likVB(f) = lik(..,g,..) * exp(b*(f-g)), g = sign(f-z)*sqrt((f-z)^2+v)+z, where % v .. marginal variance = (positive) smoothing width, and % lik .. lik function such that p(y|f)=lik(..,f,..). % % The problem is convex whenever the likelihood is log-concave. At the end, the % optimal width W is obtained analytically. % % Copyright (c) by Hannes Nickisch 2014-03-20. % % See also INFMETHODS.M. n = size(x,1); if isnumeric(cov), K = cov; % use provided covariance matrix else K = feval(cov{:}, hyp.cov, x); end % evaluate the covariance matrix if isnumeric(mean), m = mean; % use provided mean vector else m = feval(mean{:}, hyp.mean, x); end % evaluate the mean vector if iscell(lik), likstr = lik{1}; else likstr = lik; end if ~ischar(likstr), likstr = func2str(likstr); end if nargin<=6, opt = []; end % make opt variable available if ~isfield(opt,'postL'), opt.postL = false; end % not compute L in infLaplace if isfield(opt,'out_nmax'), out_nmax = opt.out_nmax; % maximal no of outer loops else out_nmax = 15; end % default value if isfield(opt,'out_tol'), out_tol = opt.out_tol; % outer loop convergence else out_tol = 1e-5; end % default value sW = ones(n,1); % init with some reasonable value opt.postL = false; % avoid computation of L inside infLaplace for i=1:out_nmax U = chol(eye(n)+sW*sW'.*K)'\(repmat(sW,1,n).*K); v = diag(K) - sum(U.*U,1)'; % v = diag( inv(inv(K)+diag(sW.*sW)) ) post = infLaplace(hyp, m, K, {@likVB,v,lik}, x, y, opt); % post.sW is very different from the optimal sW for non Gaussian likelihoods sW_old = sW; f = m+K*post.alpha; % posterior mean [lp,dlp,d2lp,sW,b,z] = feval(@likVB,v,lik,hyp.lik,y,f); if max(abs(sW-sW_old))2 % do we want derivatives? iKtil = repmat(sW,1,n).*solve_chol(post.L,diag(sW));% sW*B^-1*sW=inv(K+inv(W)) dnlZ = hyp; % allocate space for derivatives for j=1:length(hyp.cov) % covariance hypers dK = feval(cov{:}, hyp.cov, x, [], j); if j==1, w = iKtil*(c.*ga); end dnlZ.cov(j) = sum(sum(iKtil.*dK))/2 - (w'*dK*w)/2; end if ~strcmp(likstr,'likGauss') % likelihood hypers for j=1:length(hyp.lik) sign_fmz = 2*(f-z>=0)-1; % strict sign mapping; sign(0) = 1 g = sign_fmz.*sqrt((f-z).^2 + v) + z; dhhyp = -2*feval(lik{:},hyp.lik,y,g,[],'infLaplace',j); dnlZ.lik(j) = sum(dhhyp)/2; end else % special treatment for the Gaussian case dnlZ.lik = sum(sum( (post.L'\eye(n)).^2 )) - exp(2*hyp.lik)*(alpha'*alpha); end for j=1:length(hyp.mean) % mean hypers dm = feval(mean{:}, hyp.mean, x, j); dnlZ.mean(j) = -alpha'*dm; end end % Smoothed likelihood function; instead of p(y|f)=lik(..,f,..) compute % likVB(f) = lik(..,g,..)*exp(b*(f-g)), g = sign(f-z)*sqrt((f-z)^2+v)+z, where % v .. marginal variance = (positive) smoothing width, and % lik .. lik function such that feval(lik{:},varargin{:}) yields a result. % The smoothing results from a lower bound on the likelihood: % p(y|f) \ge exp( (b+z/ga)*f - f.^2/(2*ga) - h(ga)/2 ) function [varargout] = likVB(v, lik, varargin) [b,z] = feval(lik{:},varargin{1:2},[],zeros(size(v)),'infVB'); f = varargin{3}; % obtain location of evaluation sign_fmz = 2*(f-z>=0)-1; % strict sign mapping; sign(0) = 1 g = sign_fmz.*sqrt((f-z).^2 + v) + z; varargin{3} = g; id = v==0 | abs(f./sqrt(v+eps))>1e10; % correct asymptotics of f -> +/-Inf varargout = cell(nargout,1); % allocate output, eval lik(..,g,..) [varargout{1:min(nargout,3)}] = feval(lik{:},varargin{1:3},[],'infLaplace'); if nargout>0 lp = varargout{1}; varargout{1} = lp + b.*(f-g); varargout{1}(id) = lp(id); % correct asymptotics of f -> +/-Inf if nargout>1 % first derivative dg_df = (abs(f-z)+eps)./(abs(g-z)+eps); % stabilised dg/df for v=0, f=0 dlp = varargout{2}; varargout{2} = dlp.*dg_df + b.*(1-dg_df); varargout{2}(id) = dlp(id); % correct asymptotics of f -> +/-Inf if nargout>2 % second derivative d2lp = varargout{3}; g_e = g-z + sign_fmz*eps; v_g3 = v./(g_e.*g_e.*g_e); % stabilised v./g.^3 to cover v=0 and f=0 varargout{3} = (dlp-b).*v_g3 + d2lp.*dg_df.*dg_df; varargout{3}(id) = d2lp(id); % correct asymptotics of f -> +/-Inf if nargout>3 W = abs( (b-dlp)./(g-z+sign_fmz/1.5e8) ); % optimal sW value varargout{4} = sqrt(W); if nargout>4 varargout{5} = b; if nargout>5 varargout{6} = z; end end end end end end gpml-matlab-v3.5-2014-12-08/inf/infKL.m000644 000765 000024 00000024061 12504556214 017025 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infKL(hyp, mean, cov, lik, x, y) % Approximation to the posterior Gaussian Process by minimization of the % KL-divergence. The function is structurally very similar to infEP; the % only difference being the local divergence measure minimised. % In infEP, one minimises KL(p,q) whereas in infKL one minimises KL(q,p) % locally be iterating over the sites. For log-concave likelihoods, the % latter minimisation constitutes a 2d joint convex problem, so convergence % is guaranteed. % The function takes a specified covariance function (see covFunctions.m) and % likelihood function (see likFunctions.m), and is designed to be used with % gp.m. See also infMethods.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2013-09-13. % % See also INFMETHODS.M. out = false; persistent last_ttau last_tnu % keep tilde parameters between calls tol = 1e-3; max_sweep = 15; min_sweep = 2; % tolerance to stop KL iterations n = size(x,1); K = feval(cov{:}, hyp.cov, x); % evaluate the covariance matrix m = feval(mean{:}, hyp.mean, x); % evaluate the mean vector % A note on naming: variables are given short but descriptive names in % accordance with Rasmussen & Williams "GPs for Machine Learning" (2006): mu % and s2 are mean and variance, nu and tau are natural parameters. A leading t % means tilde, a subscript _ni means "not i" (for cavity parameters), or _n % for a vector of cavity parameters. N(f|mu,Sigma) represents the posterior. % marginal likelihood for ttau = tnu = zeros(n,1); equals n*log(2) for likCum* nlZ0 = -sum(likKL(diag(K),lik,hyp.lik,y,m)); if any(size(last_ttau) ~= [n 1]) % find starting point for tilde parameters ttau = zeros(n,1); % initialize to zero if we have no better guess tnu = zeros(n,1); Sigma = K; % initialize Sigma and mu, the parameters of .. mu = m; % .. the Gaussian posterior approximation nlZ = nlZ0; else ttau = last_ttau; % try the tilde values from previous call tnu = last_tnu; [Sigma,mu,L,alpha,nlZ] = klComputeParams(K,y,ttau,tnu,lik,hyp,m); if nlZ > nlZ0 % if zero is better .. ttau = zeros(n,1); % .. then initialize with zero instead tnu = zeros(n,1); Sigma = K; % initialize Sigma and mu, the parameters of .. mu = m; % .. the Gaussian posterior approximation nlZ = nlZ0; end end nlZ_old = Inf; sweep = 0; % converged, max. sweeps or min. sweeps? while (nlZ_old-nlZ > tol && sweep < max_sweep) || sweep tol error('maximum number of sweeps exceeded in function infKL') end last_ttau = ttau; last_tnu = tnu; % remember for next call post.alpha = alpha; post.sW = sqrt(ttau); post.L = L; % return posterior params if nargout>2 % do we want derivatives? v = diag(Sigma); [lp,df,d2f,dv] = likKL(v,lik,hyp.lik,y,mu); dnlZ = hyp; % allocate space for derivatives for j=1:length(hyp.cov) % covariance hypers dK = feval(cov{:},hyp.cov,x,[],j); AdK = A*dK; z = diag(AdK) + sum(A.*AdK,2) - sum(A'.*AdK,1)'; dnlZ.cov(j) = alpha'*dK*(alpha/2-df) - z'*dv; end for j=1:length(hyp.lik) % likelihood hypers lp_dhyp = likKL(v,lik,hyp.lik,y,K*post.alpha+m,[],[],j); dnlZ.lik(j) = -sum(lp_dhyp); end for j=1:length(hyp.mean) % mean hypers dm = feval(mean{:}, hyp.mean, x, j); dnlZ.mean(j) = -alpha'*dm; end end % function to compute the parameters of the Gaussian approximation, Sigma and % mu, from the current site parameters, ttau and tnu. Also returns L and upper % bound on negative marginal likelihood. function [Sigma,mu,L,alpha,nlZ,A] = klComputeParams(K,y,ttau,tnu,lik,hyp,m) n = length(tnu); % number of training cases sW = sqrt(ttau); % compute Sigma and mu L = chol(eye(n)+sW*sW'.*K); % L'*L=B=eye(n)+sW*K*sW V = L'\(repmat(sW,1,n).*K); Sigma = K - V'*V; alpha = tnu-sW.*solve_chol(L,sW.*(K*tnu+m)); mu = K*alpha+m; v = diag(Sigma); A = (eye(n)+K*diag(ttau))\eye(n); % A = Sigma*inv(K) lp = likKL(v,lik,hyp.lik,y,mu); % evaluate likelihood nlZ = -sum(lp) - (logdet(A)-alpha'*(mu-m)-trace(A)+n)/2; % upper bound on -lZ % We compute the Gaussian Q(f)=N(f|m,s^2) minimising the KL divergence % KL(Q||P) where P is the product of the cavity distribution q_n(f) and the % likelihood p(y|f) such that P(f) = 1/Z * q_n(f)*p(y|f). % The cavity distribution q_n(f) is an unnormalised Gaussian with precision % parameter tau_n and location parameter nu_n, hence the cavity can be written % as q_n(f) = exp(nu_n*f-tau_n/2*f^2). % The minimisation is convex iff. the likelihood p(y|f) is log-concave. The % optimisation is performed using Newton descent with backtracking line search. function [m,s,kl] = klmin(lik, hyp, y, nu_n, tau_n) ep = 1e-9; % tiny Hessian ridge for stability gthresh = 1e-8; % gradient convergence threshold lik_str = lik{1}; if ~ischar(lik_str), lik_str = func2str(lik_str); end if strcmp(lik_str,'likGauss') % likGauss can be done analytically sn2 = exp(2*hyp); s = 1/sqrt(1/sn2+tau_n); m = s^2*(nu_n+y/sn2); % init variables to minimum else s = 1/sqrt(tau_n); m = nu_n/tau_n; % init variables to cavity distribution end ll = likKL(s^2,lik,hyp,y,m); % evaluate likelihood kl = (s^2+m^2)*tau_n/2 - log(s) - nu_n*m - ll; % init the KL div up to a const for i=1:20 [ll,dm,d2m,dv,d2v,dmdv] = likKL(s^2,lik,hyp,y,m); % evaluate likelihood klold = kl; mold = m; sold = s; % remember last value kl = (s^2+m^2)*tau_n/2 - log(s) - nu_n*m - ll; % KL-divergence up to const dm = tau_n*m-nu_n-dm; d2m = tau_n-d2m; % compute kl derivatives ds = s*tau_n-1/s-2*s*dv; d2s = tau_n+1/s^2-2*dv-(2*s)^2*d2v; dmds=-2*s*dmdv; detH = ((d2m+ep)*(d2s+ep)-dmds^2); % Hessian determinant m = m-(dm*(d2s+ep)-ds*dmds)/detH; s = s-(ds*(d2m+ep)-dm*dmds)/detH; % Newton for j=1:10 % backtracking line search if klold>kl, break, end % we did descend so no need to step back m = (m+mold)/2; s = (s+sold)/2; kl = (s^2+m^2)*tau_n/2 - log(s) - nu_n*m - likKL(s^2,lik,hyp,y,m); end d = abs(dm)+abs(dv); % overall gradient if j==10, m = mold; s = sold; d = 0; end if d0 using the LU decomposition of A function y = logdet(A) [L,U] = lu(A); u = diag(U); if prod(sign(u))~=det(L), error('det(A)<=0'), end y = sum(log(abs(u))); % Gaussian smoothed likelihood function; instead of p(y|f)=lik(..,f,..) compute % log likKL(f) = int log lik(..,t,..) N(f|t,v) dt, where % v .. marginal variance = (positive) smoothing width, and % lik .. lik function such that feval(lik{:},varargin{:}) yields a result. % All return values are separately integrated using Gaussian-Hermite quadrature. function [ll,df,d2f,dv,d2v,dfdv] = likKL(v, lik, varargin) f = varargin{3}; % obtain location of evaluation sv = sqrt(v); % smoothing width lik_str = lik{1}; if ~ischar(lik_str), lik_str = func2str(lik_str); end if strcmp(lik_str,'likLaplace') % likLaplace can be done analytically b = exp(varargin{1})/sqrt(2); y = varargin{2}; mu = (f-y)/b; z = (f-y)./sv; Nz = exp(-z.^2/2)/sqrt(2*pi); Cz = (1+erf(z/sqrt(2)))/2; ll = (1-2*Cz).*mu - 2/b*sv.*Nz - log(2*b); df = (1-2*Cz)/b; d2f = -2*Nz./(b*(sv+eps)); dv = d2f/2; d2v = (z.*z-1)./(v+eps).*d2f/4; dfdv = -z.*d2f./(2*sv+eps); else N = 20; % number of quadrature points [t,w] = gauher(N); % location and weights for Gaussian-Hermite quadrature ll = 0; df = 0; d2f = 0; dv = 0; d2v = 0; dfdv = 0; % init return arguments for i=1:N % use Gaussian quadrature varargin{3} = f + sv*t(i); % coordinate transform of the quadrature points [lp,dlp,d2lp]=feval(lik{:},varargin{1:3},[],'infLaplace',varargin{6:end}); ll = ll + w(i)*lp; % value of the integral df = df + w(i)*dlp; % derivative wrt. mean d2f = d2f + w(i)*d2lp; % 2nd derivative wrt. mean ai = t(i)./(2*sv+eps); dvi = dlp.*ai; dv = dv+w(i)*dvi; % deriv wrt. var d2v = d2v + w(i)*(d2lp.*(t(i)^2/2)-dvi)./(v+eps)/2; % 2nd deriv wrt. var dfdv = dfdv + w(i)*(ai.*d2lp); % mixed second derivatives end endgpml-matlab-v3.5-2014-12-08/inf/infLaplace.m000644 000765 000024 00000017414 12443115530 020056 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infLaplace(hyp, mean, cov, lik, x, y, opt) % Laplace approximation to the posterior Gaussian process. % The function takes a specified covariance function (see covFunctions.m) and % likelihood function (see likFunctions.m), and is designed to be used with % gp.m. See also infMethods.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2013-05-02. % % See also INFMETHODS.M. persistent last_alpha % copy of the last alpha if any(isnan(last_alpha)), last_alpha = zeros(size(last_alpha)); end % prevent if nargin<=6, opt = []; end % make opt variable available if isfield(opt,'postL'), postL = opt.postL; % recompute matrix L for post else postL = true; end % default value inf = 'infLaplace'; n = size(x,1); if isnumeric(cov), K = cov; % use provided covariance matrix else K = feval(cov{:}, hyp.cov, x); end % evaluate the covariance matrix if isnumeric(mean), m = mean; % use provided mean vector else m = feval(mean{:}, hyp.mean, x); end % evaluate the mean vector likfun = @(f) feval(lik{:},hyp.lik,y,f,[],inf); % log likelihood function if any(size(last_alpha)~=[n,1]) % find a good starting point for alpha and f alpha = zeros(n,1); % start at mean if sizes do not match else alpha = last_alpha; % try last one if Psi(alpha,m,K,likfun) > -sum(likfun(m)) % default f==m better => use it alpha = zeros(n,1); end end % switch between optimisation methods alpha = irls(alpha, m,K,likfun, opt); % run optimisation f = K*alpha+m; % compute latent function values last_alpha = alpha; % remember for next call [lp,dlp,d2lp,d3lp] = likfun(f); W = -d2lp; isWneg = any(W<0); post.alpha = alpha; % return the posterior parameters post.sW = sqrt(abs(W)).*sign(W); % preserve sign in case of negative % diagnose optimality err = @(x,y) norm(x-y)/max([norm(x),norm(y),1]); % we need to have alpha = dlp % dev = err(alpha,dlp); if dev>1e-4, warning('Not at optimum %1.2e.',dev), end if postL || nargout>1 if isWneg % switch between Cholesky and LU decomposition mode % For post.L = -inv(K+diag(1./W)), we us the non-default parametrisation. [ldA, iA, post.L] = logdetA(K,W); % A=eye(n)+K*W is as safe as symmetric B nlZ = alpha'*(f-m)/2 - sum(lp) + ldA/2; else sW = post.sW; post.L = chol(eye(n)+sW*sW'.*K); % recompute nlZ = alpha'*(f-m)/2 + sum(log(diag(post.L))-lp); % ..(f-m)/2 -lp +ln|B|/2 end end if nargout>2 % do we want derivatives? dnlZ = hyp; % allocate space for derivatives if isWneg % switch between Cholesky and LU decomposition mode Z = -post.L; % inv(K+inv(W)) g = sum(iA.*K,2)/2; % deriv. of ln|B| wrt W; g = diag(inv(inv(K)+diag(W)))/2 else Z = repmat(sW,1,n).*solve_chol(post.L,diag(sW)); %sW*inv(B)*sW=inv(K+inv(W)) C = post.L'\(repmat(sW,1,n).*K); % deriv. of ln|B| wrt W g = (diag(K)-sum(C.^2,1)')/2; % g = diag(inv(inv(K)+W))/2 end dfhat = g.*d3lp; % deriv. of nlZ wrt. fhat: dfhat=diag(inv(inv(K)+W)).*d3lp/2 for i=1:length(hyp.cov) % covariance hypers dK = feval(cov{:}, hyp.cov, x, [], i); dnlZ.cov(i) = sum(sum(Z.*dK))/2 - alpha'*dK*alpha/2; % explicit part b = dK*dlp; % b-K*(Z*b) = inv(eye(n)+K*diag(W))*b dnlZ.cov(i) = dnlZ.cov(i) - dfhat'*( b-K*(Z*b) ); % implicit part end for i=1:length(hyp.lik) % likelihood hypers [lp_dhyp,dlp_dhyp,d2lp_dhyp] = feval(lik{:},hyp.lik,y,f,[],inf,i); dnlZ.lik(i) = -g'*d2lp_dhyp - sum(lp_dhyp); % explicit part b = K*dlp_dhyp; % b-K*(Z*b) = inv(eye(n)+K*diag(W))*b dnlZ.lik(i) = dnlZ.lik(i) - dfhat'*( b-K*(Z*b) ); % implicit part end for i=1:length(hyp.mean) % mean hypers dm = feval(mean{:}, hyp.mean, x, i); dnlZ.mean(i) = -alpha'*dm; % explicit part dnlZ.mean(i) = dnlZ.mean(i) - dfhat'*(dm-K*(Z*dm)); % implicit part end end % Evaluate criterion Psi(alpha) = alpha'*K*alpha + likfun(f), where % f = K*alpha+m, and likfun(f) = feval(lik{:},hyp.lik,y, f, [],inf). function [psi,dpsi,f,alpha,dlp,W] = Psi(alpha,m,K,likfun) f = K*alpha+m; [lp,dlp,d2lp] = likfun(f); W = -d2lp; psi = alpha'*(f-m)/2 - sum(lp); if nargout>1, dpsi = K*(alpha-dlp); end % Run IRLS Newton algorithm to optimise Psi(alpha). function alpha = irls(alpha, m,K,likfun, opt) if isfield(opt,'irls_maxit'), maxit = opt.irls_maxit; % max no of Newton steps else maxit = 20; end % default value if isfield(opt,'irls_Wmin'), Wmin = opt.irls_Wmin; % min likelihood curvature else Wmin = 0.0; end % default value if isfield(opt,'irls_tol'), tol = opt.irls_tol; % stop Newton iterations else tol = 1e-6; end % default value smin_line = 0; smax_line = 2; % min/max line search steps size range nmax_line = 10; % maximum number of line search steps thr_line = 1e-4; % line search threshold Psi_line = @(s,alpha,dalpha) Psi(alpha+s*dalpha, m,K,likfun); % line search pars_line = {smin_line,smax_line,nmax_line,thr_line}; % line seach parameters search_line = @(alpha,dalpha) brentmin(pars_line{:},Psi_line,5,alpha,dalpha); f = K*alpha+m; [lp,dlp,d2lp] = likfun(f); W = -d2lp; n = size(K,1); Psi_new = Psi(alpha,m,K,likfun); Psi_old = Inf; % make sure while loop starts by the largest old objective val it = 0; % this happens for the Student's t likelihood while Psi_old - Psi_new > tol && it det(A) = det(P)*prod(diag(U)) ldA = sum(log(abs(u))); end if nargout>1, iA = U\(L\P); end % return the inverse if required if nargout>2, mwiA = -repmat(w,1,n).*iA; endgpml-matlab-v3.5-2014-12-08/inf/infFITC_Laplace.m000644 000765 000024 00000026153 12466200266 020671 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infFITC_Laplace(hyp, mean, cov, lik, x, y) % FITC-Laplace approximation to the posterior Gaussian process. The function is % equivalent to infLaplace with the covariance function: % Kt = Q + G; G = diag(g); g = diag(K-Q); Q = Ku'*inv(Kuu + snu2*eye(nu))*Ku; % where Ku and Kuu are covariances w.r.t. to inducing inputs xu and % snu2 = sn2/1e6 is the noise of the inducing inputs. We fixed the standard % deviation of the inducing inputs snu to be a one per mil of the measurement % noise's standard deviation sn. In case of a likelihood without noise % parameter sn2, we simply use snu2 = 1e-6. % % The implementation exploits the Woodbury matrix identity % inv(Kt) = inv(G) - inv(G)*Ku'*inv(Kuu+Ku*inv(G)*Ku')*Ku*inv(G) % in order to be applicable to large datasets. The computational complexity % is O(n nu^2) where n is the number of data points x and nu the number of % inducing inputs in xu. % The posterior N(f|h,Sigma) is given by h = m+mu with mu = nn + P'*gg and % Sigma = inv(inv(K)+diag(W)) = diag(d) + P'*R0'*R'*R*R0*P. % % The function takes a specified covariance function (see covFunctions.m) and % likelihood function (see likFunctions.m), and is designed to be used with % gp.m and in conjunction with covFITC. % % The inducing points can be specified through 1) the 2nd covFITC parameter or % by 2) providing a hyp.xu hyperparameters. Note that 2) has priority over 1). % In case 2) is provided and derivatives dnlZ are requested, there will also be % a dnlZ.xu field allowing to optimise w.r.t. to the inducing points xu. However % the derivatives dnlZ.xu can only be computed for one of the following eight % covariance functions: cov{Matern|PP|RQ|SE}{iso|ard}. % % Copyright (c) by Hannes Nickisch, 2015-02-09. % % See also INFMETHODS.M, COVFITC.M. persistent last_alpha % copy of the last alpha if any(isnan(last_alpha)), last_alpha = zeros(size(last_alpha)); end % prevent tol = 1e-6; % tolerance for when to stop the Newton iterations smax = 2; Nline = 10; thr = 1e-4; % line search parameters maxit = 20; % max number of Newton steps in f inf = 'infLaplace'; cov1 = cov{1}; if isa(cov1, 'function_handle'), cov1 = func2str(cov1); end if ~strcmp(cov1,'covFITC'); error('Only covFITC supported.'), end % check cov if isfield(hyp,'xu'), cov{3} = hyp.xu; end % hyp.xu is provided, replace cov{3} [diagK,Kuu,Ku] = feval(cov{:}, hyp.cov, x); % evaluate covariance matrix if ~isempty(hyp.lik) % hard coded inducing inputs noise sn2 = exp(2*hyp.lik(end)); snu2 = 1e-6*sn2; % similar to infFITC else snu2 = 1e-6; end [n, D] = size(x); nu = size(Kuu,1); m = feval(mean{:}, hyp.mean, x); % evaluate the mean vector rot180 = @(A) rot90(rot90(A)); % little helper functions chol_inv = @(A) rot180(chol(rot180(A))')\eye(nu); % chol(inv(A)) R0 = chol_inv(Kuu+snu2*eye(nu)); % initial R, used for refresh O(nu^3) V = R0*Ku; d0 = diagK-sum(V.*V,1)'; % initial d, needed for refresh O(n*nu^2) Psi_old = Inf; % make sure while loop starts by the largest old objective val if any(size(last_alpha)~=[n,1]) % find a good starting point for alpha and f alpha = zeros(n,1); f = mvmK(alpha,V,d0)+m; % start at mean if sizes not match [lp,dlp,d2lp] = feval(lik{:},hyp.lik,y,f,[],inf); W=-d2lp; Psi_new = -sum(lp); else alpha = last_alpha; f = mvmK(alpha,V,d0)+m; % try last one [lp,dlp,d2lp] = feval(lik{:},hyp.lik,y,f,[],inf); W=-d2lp; Psi_new = alpha'*(f-m)/2 - sum(lp); % objective for last alpha Psi_def = -feval(lik{:},hyp.lik,y,m,[],inf); % objective for default init f==m if Psi_def < Psi_new % if default is better, we use it alpha = zeros(n,1); f = mvmK(alpha,V,d0)+m; [lp,dlp,d2lp] = feval(lik{:},hyp.lik,y,f,[],inf); W=-d2lp; Psi_new=-sum(lp); end end isWneg = any(W<0); % flag indicating whether we found negative values of W it = 0; % this happens for the Student's t likelihood while Psi_old - Psi_new > tol && it2 % do we want derivatives? dnlZ = hyp; % allocate space for derivatives [d,P,R] = fitcRefresh(d0,Ku,R0,V,W); % g = diag(inv(inv(K)+W))/2 g = d/2 + sum(((R*R0)*P).^2,1)'/2; t = W./(1+W.*d0); dfhat = g.*d3lp; % deriv. of nlZ wrt. fhat: dfhat=diag(inv(inv(K)+W)).*d3lp/2 for i=1:length(hyp.cov) % covariance hypers [ddiagK,dKuu,dKu] = feval(cov{:}, hyp.cov, x, [], i); % eval cov derivatives dA = 2*dKu'-R0tV'*dKuu; % dQ = dA*R0tV w = sum(dA.*R0tV',2); v = ddiagK-w; % w = diag(dQ); v = diag(dK)-diag(dQ); dnlZ.cov(i) = ddiagK'*t - sum(RVdd.*RVdd,1)*v; % explicit part dnlZ.cov(i) = dnlZ.cov(i)-sum(sum((RVdd*dA).*(RVdd*R0tV'))); % explicit part dnlZ.cov(i) = dnlZ.cov(i)/2-alpha'*(dA*(R0tV*alpha)+v.*alpha)/2; % explicit b = dA*(R0tV*dlp) + v.*dlp; % b-K*(Z*b) = inv(eye(n)+K*diag(W))*b KZb = mvmK(mvmZ(b,RVdd,t),V,d0); dnlZ.cov(i) = dnlZ.cov(i) - dfhat'*( b-KZb ); % implicit part end for i=1:length(hyp.lik) % likelihood hypers [lp_dhyp,dlp_dhyp,d2lp_dhyp] = feval(lik{:},hyp.lik,y,f,[],inf,i); dnlZ.lik(i) = -g'*d2lp_dhyp - sum(lp_dhyp); % explicit part b = mvmK(dlp_dhyp,V,d0); % implicit part dnlZ.lik(i) = dnlZ.lik(i) - dfhat'*(b-mvmK(mvmZ(b,RVdd,t),V,d0)); if i==numel(hyp.lik) % since snu2 is a fixed fraction of sn2, there is a covariance-like term % in the derivative as well snu = sqrt(snu2); T = chol_inv(Kuu + snu2*eye(nu)); T = T'*(T*(snu*Ku)); st2 = sum(T.*T,1)'; z = alpha'*(T'*(T*alpha)-st2.*alpha) - sum(RVdd.*RVdd,1)*st2; z = z + sum(sum( (RVdd*T').^2 )); b = (st2.*dlp-T'*(T*dlp))*2; KZb = mvmK(mvmZ(b,RVdd,t),V,d0); z = z - dfhat'*( b-KZb ); dnlZ.lik(i) = dnlZ.lik(i) + z; end end for i=1:length(hyp.mean) % mean hypers dm = feval(mean{:}, hyp.mean, x, i); dnlZ.mean(i) = -alpha'*dm; % explicit part Zdm = mvmZ(dm,RVdd,t); dnlZ.mean(i) = dnlZ.mean(i) - dfhat'*(dm-mvmK(Zdm,V,d0)); % implicit part end if isfield(hyp,'xu') % derivatives w.r.t. inducing points xu xu = cov{3}; cov = cov{2}; % get the non FITC part of the covariance function Kpu = cov_deriv_sq_dist(cov,hyp.cov,xu,x); % d K(xu,x ) / d D^2 Kpuu = cov_deriv_sq_dist(cov,hyp.cov,xu); % d K(xu,xu) / d D^2 if iscell(cov), covstr = cov{1}; else covstr = cov; end if ~ischar(covstr), covstr = func2str(covstr); end if numel(strfind(covstr,'iso'))>0 % characteristic length scale e = 2*exp(-2*hyp.cov(1)); else e = 2*exp(-2*hyp.cov(1:D)); end q = dfhat - mvmZ(mvmK(dfhat,V,d0),RVdd,t); % implicit part B = (R0'*R0)*Ku; diag_dK = alpha.*alpha + sum(RVdd.*RVdd,1)' - t + 2*dlp.*q; v = diag_dK+t; % BdK = B * ( dnlZ/dK - diag(diag(dnlZ/dK)) ) BdK = (B*alpha)*alpha' - B.*repmat(v',nu,1) + (B*dlp)*q' + (B*q)*dlp'; BdK = BdK + (B*RVdd')*RVdd; A = Kpu.*BdK; C = Kpuu.*(BdK*B'); C = diag(sum(C,2)-sum(A,2)) - C; dnlZ.xu = A*x*diag(e) + C*xu*diag(e); % bring in data and inducing points end end % matrix vector multiplication with Z=inv(K+inv(W)) function Zx = mvmZ(x,RVdd,t) Zx = t.*x - RVdd'*(RVdd*x); % matrix vector multiplication with approximate covariance matrix function Kal = mvmK(al,V,d0) Kal = V'*(V*al) + d0.*al; % criterion Psi at alpha + s*dalpha for line search function [Psi,alpha,f,dlp,W] = Psi_line(s,dalpha,alpha,hyp,V,d0,m,lik,y,inf) alpha = alpha + s*dalpha; f = mvmK(alpha,V,d0)+m; [lp,dlp,d2lp] = feval(lik{:},hyp.lik,y,f,[],inf); W = -d2lp; Psi = alpha'*(f-m)/2 - sum(lp); % refresh the representation of the posterior from initial and site parameters % to prevent possible loss of numerical precision after many epfitcUpdates % effort is O(n*nu^2) provided that nu1 % do we want the marginal likelihood? nlZ = (y-m)'*alpha/2 + sum(log(diag(L))) + n*log(2*pi*sl)/2; % -log marg lik if nargout>2 % do we want derivatives? dnlZ = hyp; % allocate space for derivatives Q = solve_chol(L,eye(n))/sl - alpha*alpha'; % precompute for convenience for i = 1:numel(hyp.cov) dnlZ.cov(i) = sum(sum(Q.*feval(cov{:}, hyp.cov, x, [], i)))/2; end dnlZ.lik = sn2*trace(Q); for i = 1:numel(hyp.mean) dnlZ.mean(i) = -feval(mean{:}, hyp.mean, x, i)'*alpha; end end end gpml-matlab-v3.5-2014-12-08/inf/infGrid.m000664 000765 000024 00000017155 12445165123 017413 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infGrid(hyp, mean, cov, lik, x, y, opt) % Inference for a GP with Gaussian likelihood and covGrid covariance. % The (Kronecker) covariance matrix used is given by: % K = kron( kron(...,K{2}), K{1} ) = K_p x .. x K_2 x K_1. % % Compute a parametrization of the posterior, the negative log marginal % likelihood and its derivatives w.r.t. the hyperparameters. % The result is exact for complete grids, otherwise results are approximate. % See also "help infMethods". % % The function takes a specified covariance function (see covFunctions.m) and % likelihood function (see likFunctions.m), and is designed to be used with % gp.m and in conjunction with covFITC and likGauss. % % Copyright (c) by Hannes Nickisch and Andrew Wilson, 2014-12-03. % % See also INFMETHODS.M, COVGRID.M. if iscell(lik), likstr = lik{1}; else likstr = lik; end if ~ischar(likstr), likstr = func2str(likstr); end if ~strcmp(likstr,'likGauss') % NOTE: no explicit call to likGauss error('Inference with infGrid only possible with Gaussian likelihood.'); end cov1 = cov{1}; if isa(cov1, 'function_handle'), cov1 = func2str(cov1); end if ~strcmp(cov1,'covGrid'); error('Only covGrid supported.'), end % check cov xg = cov{3}; p = numel(xg); % underlying grid N = 1; D = 0; for i=1:p, N = N*size(xg{i},1); D = D+size(xg{i},2); end % dims [K,M,xe] = feval(cov{:}, hyp.cov, x); % evaluate covariance mat constituents xe = M*xe; n = size(xe,1); m = feval(mean{:}, hyp.mean, xe); % evaluate mean vector if nargin<=6, opt = []; end % make opt variable available if isfield(opt,'cg_tol'), cgtol = opt.cg_tol; % stop conjugate gradients else cgtol = 1e-5; end if isfield(opt,'cg_maxit'), cgmit = opt.cg_maxit; % number of cg iterations else cgmit = min(n,20); end for j=1:numel(K) if iscell(K{j}) && strcmp(K{j}{1},'toep'), K{j} = toeplitz(K{j}{2}); end end sn2 = exp(2*hyp.lik); % noise variance of likGauss V = cell(size(K)); E = cell(size(K)); % eigenvalue decomposition for j=1:numel(K) if iscell(K{j}) && strcmp(K{j}{1},'toep') [V{j},E{j}] = eigr(toeplitz(K{j}{2})); else [V{j},E{j}] = eigr(K{j}); end end e = 1; for i=1:numel(E), e = kron(diag(E{i}),e); end % eigenvalue diagonal if numel(m)==N s = 1./(e+sn2); ord = 1:N; % V*diag(s)*V' = inv(K+sn2*eye(N)) else [eord,ord] = sort(e,'descend'); % sort long vector of eigenvalues if n>N, eord = [eord;zeros(n-N,1)]; ord = [ord;(N+1:n)']; end % special case s = 1./((n/N)*eord(1:n) + sn2); % approx using top n eigenvalues end if numel(m)==N % decide for Kronecker magic or conjugate gradients L = @(k) kronmvm(V,repmat(-s,1,size(k,2)).*kronmvm(V,k,1)); % mvm callback else % go for conjugate gradients mvm = @(t) mvmK(t,K,sn2,M); % single parameter mvm L = @(k) -solveMVM(k,mvm,cgtol,cgmit); end alpha = -L(y-m); post.alpha = alpha; % return the posterior parameters post.sW = ones(n,1)/sqrt(sn2); % sqrt of noise precision post.L = L; % function to compute inv(K+sn2*eye(N))*Ks if nargout>1 % do we want the marginal likelihood? lda = -sum(log(s)); % exact kron % exact: lda = 2*sum(log(diag(chol(M*kronmvm(K,eye(N))*M'+sn2*eye(n))))); nlZ = (y-m)'*alpha/2 + n*log(2*pi)/2 + lda/2; if nargout>2 % do we want derivatives? dnlZ = hyp; % allocate space for derivatives, define Q=inv(M*K*M'+sn2*I) Mtal = M'*alpha; % blow up alpha vector from n to N for i = 1:numel(hyp.cov) dK = feval(cov{:}, hyp.cov, x, [], i); for j=1:numel(dK) % expand Toeplitz if iscell(dK{j}) && strcmp(dK{j}{1},'toep') dK{j} = toeplitz(dK{j}{2}); end end P = cell(size(dK)); % auxiliary Kronecker matrix P = V'*dK*V for j = 1:numel(dK), P{j} = sum((V{j}'*dK{j}).*V{j}',2); end p = 1; for j=1:numel(P), p = kron(P{j},p); end % p = diag(P) if n>N, p = [p;zeros(n-N,1)]; end % special case p = (n/N)*p(ord(1:n)); % approximate if incomplete grid observation dnlZ.cov(i) = (p'*s - Mtal'*kronmvm(dK,Mtal))/2; end dnlZ.lik = sn2*(sum(s) - alpha'*alpha); % sum(s) = trace(Q) for i = 1:numel(hyp.mean) dnlZ.mean(i) = -feval(mean{:}, hyp.mean, xe, i)'*alpha; end end end % Solve x=A*b with symmetric A(n,n), b(n,m), x(n,m) using conjugate gradients. % The method is along the lines of PCG but suited for matrix inputs b. function [x,flag,relres,iter,r] = conjgrad(A,b,tol,maxit) if nargin<3, tol = 1e-10; end if nargin<4, maxit = min(size(b,1),20); end x0 = zeros(size(b)); x = x0; if isnumeric(A), r = b-A*x; else r = b-A(x); end, r2 = sum(r.*r,1); r2new = r2; nb = sqrt(sum(b.*b,1)); flag = 0; iter = 1; relres = sqrt(r2)./nb; todo = relres>=tol; if ~any(todo), flag = 1; return, end on = ones(size(b,1),1); r = r(:,todo); d = r; for iter = 2:maxit if isnumeric(A), z = A*d; else z = A(d); end a = r2(todo)./sum(d.*z,1); a = on*a; x(:,todo) = x(:,todo) + a.*d; r = r - a.*z; r2new(todo) = sum(r.*r,1); relres = sqrt(r2new)./nb; cnv = relres(todo)=tol; d = d(:,~cnv); r = r(:,~cnv); % get rid of converged if ~any(todo), flag = 1; return, end b = r2new./r2; % Fletcher-Reeves d = r + (on*b(todo)).*d; r2 = r2new; end % solve q = mvm(p) via conjugate gradients function q = solveMVM(p,mvm,varargin) [q,flag,relres,iter] = conjgrad(mvm,p,varargin{:}); % like pcg if ~flag,error('Not converged after %d iterations, r=%1.2e\n',iter,relres),end % mvm so that q = M*K*M'*p + sn2*p using the Kronecker representation function q = mvmK(p,K,sn2,M) q = M*kronmvm(K,M'*p) + sn2*p; % Perform a matrix vector multiplication b = A*x with a matrix A being a % Kronecker product given by A = kron( kron(...,As{2}), As{1} ). function b = kronmvm(As,x,transp) if nargin>2 && ~isempty(transp) && transp % transposition by transposing parts for i=1:numel(As), As{i} = As{i}'; end end m = zeros(numel(As),1); n = zeros(numel(As),1); % extract sizes for i=1:numel(n) if iscell(As{i}) && strcmp(As{i}{1},'toep') m(i) = size(As{i}{2},1); n(i) = size(As{i}{2},1); else [m(i),n(i)] = size(As{i}); end end d = size(x,2); b = x; for i=1:numel(n) a = reshape(b,[prod(m(1:i-1)), n(i), prod(n(i+1:end))*d]); % prepare input tmp = reshape(permute(a,[1,3,2]),[],n(i))*As{i}'; b = permute(reshape(tmp,[size(a,1),size(a,3),m(i)]),[1,3,2]); end b = reshape(b,prod(m),d); % bring result in correct shape % Real eigenvalues and eigenvectors up to the rank of a real symmetric matrix. % Decompose A into V*D*V' with orthonormal matrix V and diagonal matrix D. % Diagonal entries of D obave the rank r of the matrix A as returned by % the call rank(A,tol) are zero. function [V,D] = eigr(A,tol) [V,D] = eig((A+A')/2); n = size(A,1); % decomposition of strictly symmetric A d = max(real(diag(D)),0); [d,ord] = sort(d,'descend'); % tidy up and sort if nargin<2, tol = size(A,1)*eps(max(d)); end, r = sum(d>tol); % get rank(A) d(r+1:n) = 0; D = diag(d); % set junk eigenvalues to strict zeros V(:,1:r) = real(V(:,ord(1:r))); V(:,r+1:n) = null(V(:,1:r)'); % ortho completion gpml-matlab-v3.5-2014-12-08/inf/infFITC_EP.m000644 000765 000024 00000027525 12437655340 017645 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infFITC_EP(hyp, mean, cov, lik, x, y) % FITC-EP approximation to the posterior Gaussian process. The function is % equivalent to infEP with the covariance function: % Kt = Q + G; G = diag(g); g = diag(K-Q); Q = Ku'*inv(Kuu + snu2*eye(nu))*Ku; % where Ku and Kuu are covariances w.r.t. to inducing inputs xu and % snu2 = sn2/1e6 is the noise of the inducing inputs. We fixed the standard % deviation of the inducing inputs snu to be a one per mil of the measurement % noise's standard deviation sn. In case of a likelihood without noise % parameter sn2, we simply use snu2 = 1e-6. % For details, see The Generalized FITC Approximation, Andrew Naish-Guzman and % Sean Holden, NIPS, 2007. % % The implementation exploits the Woodbury matrix identity % inv(Kt) = inv(G) - inv(G)*Ku'*inv(Kuu+Ku*inv(G)*Ku')*Ku*inv(G) % in order to be applicable to large datasets. The computational complexity % is O(n nu^2) where n is the number of data points x and nu the number of % inducing inputs in xu. % The posterior N(f|h,Sigma) is given by h = m+mu with mu = nn + P'*gg and % Sigma = inv(inv(K)+diag(W)) = diag(d) + P'*R0'*R'*R*R0*P. Here, we use the % site parameters: b,w=$b,\pi$=tnu,ttau, P=$P'$, nn=$\nu$, gg=$\gamma$ % % The function takes a specified covariance function (see covFunctions.m) and % likelihood function (see likFunctions.m), and is designed to be used with % gp.m and in conjunction with covFITC. % % The inducing points can be specified through 1) the 2nd covFITC parameter or % by 2) providing a hyp.xu hyperparameters. Note that 2) has priority over 1). % In case 2) is provided and derivatives dnlZ are requested, there will also be % a dnlZ.xu field allowing to optimise w.r.t. to the inducing points xu. However % the derivatives dnlZ.xu can only be computed for one of the following eight % covariance functions: cov{Matern|PP|RQ|SE}{iso|ard}. % % Copyright (c) by Hannes Nickisch, 2013-10-29. % % See also INFMETHODS.M, COVFITC.M. persistent last_ttau last_tnu % keep tilde parameters between calls tol = 1e-4; max_sweep = 20; min_sweep = 2; % tolerance to stop EP iterations inf = 'infEP'; cov1 = cov{1}; if isa(cov1, 'function_handle'), cov1 = func2str(cov1); end if ~strcmp(cov1,'covFITC'); error('Only covFITC supported.'), end % check cov if isfield(hyp,'xu'), cov{3} = hyp.xu; end % hyp.xu is provided, replace cov{3} [diagK,Kuu,Ku] = feval(cov{:}, hyp.cov, x); % evaluate covariance matrix if ~isempty(hyp.lik) % hard coded inducing inputs noise sn2 = exp(2*hyp.lik(end)); snu2 = 1e-6*sn2; % similar to infFITC else snu2 = 1e-6; end [n, D] = size(x); nu = size(Kuu,1); m = feval(mean{:}, hyp.mean, x); % evaluate the mean vector rot180 = @(A) rot90(rot90(A)); % little helper functions chol_inv = @(A) rot180(chol(rot180(A))')\eye(nu); % chol(inv(A)) R0 = chol_inv(Kuu+snu2*eye(nu)); % initial R, used for refresh O(nu^3) V = R0*Ku; d0 = diagK-sum(V.*V,1)'; % initial d, needed for refresh O(n*nu^2) % A note on naming: variables are given short but descriptive names in % accordance with Rasmussen & Williams "GPs for Machine Learning" (2006): mu % and s2 are mean and variance, nu and tau are natural parameters. A leading t % means tilde, a subscript _ni means "not i" (for cavity parameters), or _n % for a vector of cavity parameters. % marginal likelihood for ttau = tnu = zeros(n,1); equals n*log(2) for likCum* nlZ0 = -sum(feval(lik{:}, hyp.lik, y, m, diagK, inf)); if any(size(last_ttau) ~= [n 1]) % find starting point for tilde parameters ttau = zeros(n,1); % initialize to zero if we have no better guess tnu = zeros(n,1); [d,P,R,nn,gg] = epfitcRefresh(d0,Ku,R0,V, ttau,tnu); % compute initial repres. nlZ = nlZ0; else ttau = last_ttau; % try the tilde values from previous call tnu = last_tnu; [d,P,R,nn,gg] = epfitcRefresh(d0,Ku,R0,V, ttau,tnu); % compute initial repres. nlZ = epfitcZ(d,P,R,nn,gg,ttau,tnu,d0,R0,Ku,y,lik,hyp,m,inf); if nlZ > nlZ0 % if zero is better .. ttau = zeros(n,1); % .. then initialize with zero instead tnu = zeros(n,1); [d,P,R,nn,gg] = epfitcRefresh(d0,Ku,R0,V, ttau,tnu); % initial repres. nlZ = nlZ0; end end nlZ_old = Inf; sweep = 0; % converged, max. sweeps or min. sweeps? while (abs(nlZ-nlZ_old) > tol && sweep < max_sweep) || sweep2 % do we want derivatives? dnlZ = hyp; % allocate space for derivatives RVdd = RV.*repmat(dd',nu,1); for i=1:length(hyp.cov) [ddiagK,dKuu,dKu] = feval(cov{:}, hyp.cov, x, [], i); % eval cov derivatives dA = 2*dKu'-R0tV'*dKuu; % dQ = dA*R0tV w = sum(dA.*R0tV',2); v = ddiagK-w; % w = diag(dQ); v = diag(dK)-diag(dQ); z = dd'*(v+w) - sum(RVdd.*RVdd,1)*v - sum(sum( (RVdd*dA)'.*(R0tV*RVdd') )); dnlZ.cov(i) = (z - alpha'*(alpha.*v) - (alpha'*dA)*(R0tV*alpha))/2; end for i = 1:numel(hyp.lik) % likelihood hypers dlik = feval(lik{:}, hyp.lik, y, nu_n./tau_n, 1./tau_n, inf, i); dnlZ.lik(i) = -sum(dlik); if i==numel(hyp.lik) % since snu2 is a fixed fraction of sn2, there is a covariance-like term % in the derivative as well v = sum(R0tV.*R0tV,1)'; z = sum(sum( (RVdd*R0tV').^2 )) - sum(RVdd.*RVdd,1)*v; z = z + post.alpha'*post.alpha - alpha'*(v.*alpha); dnlZ.lik(i) = dnlZ.lik(i) + snu2*z; end end [junk,dlZ] = feval(lik{:}, hyp.lik, y, nu_n./tau_n, 1./tau_n, inf);% mean hyps for i = 1:numel(hyp.mean) dm = feval(mean{:}, hyp.mean, x, i); dnlZ.mean(i) = -dlZ'*dm; end if isfield(hyp,'xu') % derivatives w.r.t. inducing points xu xu = cov{3}; cov = cov{2}; % get the non FITC part of the covariance function Kpu = cov_deriv_sq_dist(cov,hyp.cov,xu,x); % d K(xu,x ) / d D^2 Kpuu = cov_deriv_sq_dist(cov,hyp.cov,xu); % d K(xu,xu) / d D^2 if iscell(cov), covstr = cov{1}; else covstr = cov; end if ~ischar(covstr), covstr = func2str(covstr); end if numel(strfind(covstr,'iso'))>0 % characteristic length scale e = 2*exp(-2*hyp.cov(1)); else e = 2*exp(-2*hyp.cov(1:D)); end B = (R0'*R0)*Ku; W = ttau; t = W./(1+W.*d0); diag_dK = alpha.*alpha + sum(RVdd.*RVdd,1)' - t; v = diag_dK+t; % BdK = B * ( dnlZ/dK - diag(diag(dnlZ/dK)) ) BdK = (B*alpha)*alpha' - B.*repmat(v',nu,1); BdK = BdK + (B*RVdd')*RVdd; A = Kpu.*BdK; C = Kpuu.*(BdK*B'); C = diag(sum(C,2)-sum(A,2)) - C; dnlZ.xu = A*x*diag(e) + C*xu*diag(e); % bring in data and inducing points end end % refresh the representation of the posterior from initial and site parameters % to prevent possible loss of numerical precision after many epfitcUpdates % effort is O(n*nu^2) provided that nu0 R = cholupdate(R,sqrt( r)*R'*v,'-'); else R = cholupdate(R,sqrt(-r)*R'*v,'+'); end gg = gg + ((dbi-dwi*(hi-m(i)))/t)*(R0'*(R'*(R*(R0*Pi)))); % O(nu^2) w(i) = wi; b(i) = bi; % update site parameters O(1) Pi = Pi/t; % O(nu) gpml-matlab-v3.5-2014-12-08/inf/infMCMC.m000644 000765 000024 00000024661 12075753276 017256 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infMCMC(hyp, mean, cov, lik, x, y, par) % Markov Chain Monte Carlo (MCMC) sampling from posterior and % Annealed Importance Sampling (AIS) for marginal likelihood estimation. % % The algorithms are not to be used as a black box, since the acceptance rate % of the samplers need to be carefully monitored. Also, there are no derivatives % of the marginal likelihood available. % % There are additional parameters: % - par.sampler switch between the samplers 'hmc' or 'ess' % - par.Nsample num of samples % - par.Nskip num of steps out of which one sample kept % - par.Nburnin num of burn in samples (corresponds to Nskip*Nburning steps) % - par.Nais num of AIS runs to remove finite temperature bias % Default values are 'sampler=hmc', Nsample=200, Nskip=40, Nburnin=10, Nais=3. % % The Hybrid Monte Carlo Sampler (HMC) is implemented as described in the % technical report: Probabilistic Inference using MCMC Methods by Radford Neal, % CRG-TR-93-1, 1993. % % Instead of sampling from f ~ 1/Zf * N(f|m,K) P(y|f), we use a % parametrisation in terms of alpha = inv(K)*(f-m) and sample from % alpha ~ P(a) = 1/Za * N(a|0,inv(K)) P(y|K*a+m) to increase numerical % stability since log P(a) = -(a'*K*a)/2 + log P(y|K*a+m) + C and its % gradient can be computed safely. % % The leapfrog stepsize as a time discretisation stepsize comes with a % tradeoff: % - too small: frequently accept, slow exploration, accurate dynamics % - too large: seldomly reject, fast exploration, inaccurate dynamics % In order to balance between the two extremes, we adaptively adjust the % stepsize in order to keep the acceptance rate close to a target acceptance % rate. Taken from http://deeplearning.net/tutorial/hmc.html. % % The code issues a warning in case the overall acceptance rate did deviate % strongly from the target. This can indicate problems with the sampler. % % The Elliptical Slice Sampler (ESS) is a straight implementation that is % inspired by the code shipped with the paper: Elliptical slice sampling by % Iain Murray, Ryan Prescott Adams and David J.C. MacKay, AISTATS 2010. % % Annealed Importance Sampling (AIS) to determine the marginal likelihood is % described in the technical report: AIS, Radford Neal, 1998. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2012-11-07. % % See also INFMETHODS.M. if nargin<7, par = []; end % analyse parameter structure if isfield(par,'sampler'), alg=par.sampler; else alg = 'hmc'; end if isfield(par,'Nsample'), N =par.Nsample; else N = 200; end if isfield(par,'Nskip'), Ns =par.Nskip; else Ns = 40; end if isfield(par,'Nburnin'), Nb =par.Nburnin; else Nb = 10; end if isfield(par,'Nais'), R =par.Nais; else R = 3; end K = feval(cov{:}, hyp.cov, x); % evaluate the covariance matrix m = feval(mean{:}, hyp.mean, x); % evaluate the mean vector n = size(K,1); [cK,fail] = chol(K); % try an ordinary Cholesky decomposition if fail, sr2 = 1e-8*sum(diag(K))/n; cK = chol(K+sr2*eye(n)); end % regularise T = (N+Nb)*Ns; % overall number of steps [alpha,Na] = sample(K,cK,m,y,lik,hyp.lik, N,Nb,Ns, alg); % sample w/o annealing post.alpha = alpha; al = sum(alpha,2)/N; post.L = -(cK\(cK'\eye(n)) + al*al' - alpha*alpha'/N); % inv(K) - cov(alpha) post.sW = []; post.acceptance_rate_MCMC = Na/T; % additional output parameter if nargout>1 % annealed importance sampling % discrete time t from 1 to T and temperature tau from tau(1)=0 to tau(T)=1 taus = [zeros(1,Nb),linspace(0,1,N)].^4; % annealing schedule, effort at start % the fourth power idea is taken from the Kuss&Rasmussen paper, 2005 JMLR % Z(t) := \int N(f|m,K) lik(f)^taus(t) df hence Z(1) = 1 and Z(T) = Z % Z = Z(T)/Z(1) = prod_t Z(t)/Z(t-1); % ln Z(t)/Z(t-1) = ( tau(t)-tau(t-1) ) * loglik(f_t) lZ = zeros(R,1); dtaus = diff(taus(Nb+(1:N))); % we have: sum(dtaus)==1 for r=1:R [A,Na] = sample(K,cK,m,y,lik,hyp.lik, N,Nb,Ns, alg, taus); % AIS for t=2:N % evaluate the likelihood sample-wise lp = feval(lik{:},hyp.lik,y,K*A(:,t)+m,[],'infLaplace'); lZ(r) = lZ(r)+dtaus(t-1)*sum(lp); end post.acceptance_rate_AIS(r) = Na/T; % additional output parameters end nlZ = log(R)-logsumexp(lZ); % remove finite temperature bias, softmax average if nargout>2 % marginal likelihood derivatives are not computed dnlZ = struct('cov',0*hyp.cov, 'mean',0*hyp.mean, 'lik',0*hyp.lik); end end %% choose between HMC and ESS depending on the alg string function [alpha,Na] = sample(K,cK,m,y,lik,hyp, N,Nb,Ns, alg, varargin) if strcmpi(alg,'hmc') [alpha,Na] = sample_hmc(K, m,y,lik,hyp, N,Nb,Ns, varargin{:}); else [alpha,Na] = sample_ess(cK,m,y,lik,hyp, N,Nb,Ns, varargin{:}); end %% sample using elliptical slices function [alpha,T] = sample_ess(cK,m,y,lik,hyp, N,Nb,Ns, taus) if nargin>=9, tau = taus(1); else tau = 1; end % default is no annealing T = (N+Nb)*Ns; % overall number of steps F = zeros(size(m,1),N); for t=1:T if nargin>=9, tau = taus(1+floor((t-1)/Ns)); end % parameter from schedule if t==1, f=0*m; l=sum(feval(lik{:},hyp,y,f+m)); end % init sample f & lik l r = cK'*randn(size(m)); % random sample from N(0,K) [f,l] = sample_ess_step(f,l,r,m,y,lik,hyp,tau); if mod(t,Ns)==0 % keep one state out of Ns if t/Ns>Nb, F(:,t/Ns-Nb) = f; end % wait for Nb burn-in steps end end alpha = cK\(cK'\F); %% elliptical slice sampling: one step function [f,l] = sample_ess_step(f,l,r,m,y,lik,hyp,tau) if nargin<7, tau=1; end, if tau>1, tau=1; end, if tau<0, tau=0; end h = log(rand) + tau*l; % acceptance threshold a = rand*2*pi; amin = a-2*pi; amax = a; % bracket whole ellipse k = 0; % emergency break while true % slice sampling loop; f for proposed angle diff; check if on slice fp = f*cos(a) + r*sin(a); % move on ellipsis defined by r l = sum(feval(lik{:},hyp,y,fp+m)); if tau*l>h || k>20, break, end % exit if new point is on slice or exhausted if a>0, amax=a; elseif a<0, amin=a; end % shrink slice to rejected point a = rand*(amax-amin) + amin; k = k+1; % propose new angle difference; break end f = fp; % accept %% sample using Hamiltonian dynamics as proposal algorithm function [alpha,Na] = sample_hmc(K,m,y,lik,hyp, N,Nb,Ns, taus) % use adaptive stepsize rule to enforce a specific acceptance rate epmin = 1e-6; % minimum leapfrog stepsize epmax = 9e-1; % maximum leapfrog stepsize ep = 1e-2; % initial leapfrog stepsize acc_t = 0.9; % target acceptance rate acc = 0; % current acceptance rate epinc = 1.02; % increase factor of stepsize if acceptance rate is below target epdec = 0.98; % decrease factor of stepsize if acceptance rate is above target lam = 0.01; % exponential moving average computation of the acceptance rate % 2/(3*lam) steps half height; lam/nstep = 0.02/33, 0.01/66, 0.005/133 l = 20; % number of leapfrog steps to perform for one step n = size(K,1); T = (N+Nb)*Ns; % overall number of steps alpha = zeros(n,N); % sample points al = zeros(n,1); % current position if nargin>=9, tau = taus(1); else tau = 1; end % default is no annealing [gold,eold] = E(al,K,m,y,lik,hyp,tau); % initial energy, gradient Na = 0; % number of accepted points for t=1:T if nargin>=9, tau = taus(1+floor((t-1)/Ns)); end % parameter from schedule p = randn(n,1); % random initial momentum q = al; g = gold; Hold = (p'*p)/2 + eold; % H = Ekin + Epot => Hamiltonian for ll=1:l % leapfrog discretization steps, Euler like p = p - (ep/2)*g; % half step in momentum p q = q + ep*p; % full step in position q g = E(q,K,m,y,lik,hyp,tau); % compute new gradient g p = p - (ep/2)*g; % half step in momentum p end [g,e] = E(q,K,m,y,lik,hyp,tau); H = (p'*p)/2 + e; % recompute Hamiltonian acc = (1-lam)*acc; % decay current acceptance rate if log(rand) < Hold-H % accept with p = min(1,exp(Hold-H)) al = q; % keep new state, gold = g; % gradient eold = e; % and potential energy acc = acc + lam; % increase rate due to acceptance Na = Na+1; % increase number accepted steps end if acc>acc_t % too large acceptance rate => increase leapfrog stepsize ep = epinc*ep; else % too small acceptance rate => decrease leapfrog stepsize ep = epdec*ep; end if epepmax, ep = epmax; end if mod(t,Ns)==0 % keep one state out of Ns if t/Ns>Nb, alpha(:,t/Ns-Nb) = al; end % wait for Nb burn-in steps end end if Na/T dE/dal = K*( al-dloglik(f) ) if nargin<7, tau=1; end, if tau>1, tau=1; end, if tau<0, tau=0; end Kal = K*al; [lp,dlp] = feval(lik{:},hyp,y,Kal+m,[],'infLaplace'); g = Kal-K*(tau*dlp); if nargout>1, e = (al'*Kal)/2 - tau*sum(lp); end %% y = logsumexp(x) = log(sum(exp(x(:))) avoiding overflow function y=logsumexp(x) mx = max(x(:)); y = log(sum(exp(x(:)-mx)))+mx; gpml-matlab-v3.5-2014-12-08/inf/infLOO.m000644 000765 000024 00000010234 12064603752 017146 0ustar00carlstaff000000 000000 function [post nloo dnloo loo] = infLOO(hyp, mean, cov, lik, x, y) % Leave-One-Out - Perform Least-Squares GP predictions in the style of the % Pseudo Likelihood in §5.4.2 p. 117 and the Probabilistic Least-squares % Classifier of §6.5 p. 146 of the GPML book. % For Gaussian likelihood, the prediction is the same as with infExact, for % lik{Erf,Logistic} the result corresponds to probabilistic least-squares % classification. In any case, the objective returned is not the marginal % likelihood but the predictive log probability. % The standard deviation of the noise either comes from the sn parameter of the % likelihood function lik{Gauss,Laplace,Sech2} or from the covariance function % by means of using covSum and covNoise i.e. for lik{Uni,Erf,Logistic}. % We then compute the negative leave-one-out predictive probability and its % derivatives w.r.t. the hyperparameters. See also "help infMethods". % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2012-11-07 % % See also INFMETHODS.M. [n, D] = size(x); K = feval(cov{:}, hyp.cov, x); % evaluate covariance matrix m = feval(mean{:}, hyp.mean, x); % evaluate mean vector if numel(hyp.lik)==0 cov1 = cov{1}; if isa(cov1, 'function_handle'), cov1 = func2str(cov1); end if ~strcmp(cov1,'covSum'); error('Only covSum supported.'), end % check cov cov2 = cov{2}; npar = 0; for i=1:length(cov2) cov2i = cov2{i}; if numel(cov2i)==1 if isa(cov2i, 'function_handle'), cov2i = func2str(cov2i); end if strcmp(cov2i,'covNoise'), isn = npar+1; sn2 = exp(2*hyp.cov(isn)); end end npar = npar + eval(feval(cov2i)); end if ~exist('sn2','var'), error('We need covNoise in the covSum.'), end else % Gauss, Laplace, Sech2 sn2 = exp(2*hyp.lik(end)); isn = 0; K = K + sn2*eye(n); end L = chol(K)/sqrt(sn2); % Cholesky factor of covariance with noise alpha = solve_chol(L,y-m)/sn2; post.alpha = alpha; % return the posterior parameters post.sW = ones(n,1)/sqrt(sn2); % sqrt of noise precision vector post.L = L; % L = chol(eye(n)+sW*sW'.*K) if nargout>1 % do we want the marginal likelihood? sdeff = 1./post.sW; % effective Gaussian width mneff = K*alpha + m; % alpha = inv(K)*(mneff-m), effective Gaussian center % all loo marginal means and variances at once V = post.L'\diag(1./sdeff); r = sum(V.*V,1)'; % r=diag( inv(K) ) loo.fmu = mneff - alpha./r; % GPML book, eq. 5.12 loo.fs2 = 1./r - sn2; % GPML book, eq. 5.12 [loo.lp, loo.ymu, loo.ys2] = feval(lik{:},hyp.lik,y,loo.fmu,loo.fs2); nloo = -sum(loo.lp); % negative leave-one-out predictive probability if nargout>2 % do we want derivatives? [lZ,dlZmn,d2lZmn] = feval(lik{:},hyp.lik,y,loo.fmu,loo.fs2,'infEP'); dlZva = (d2lZmn+dlZmn.*dlZmn)/2; % derivative w.r.t. variance dnloo = hyp; iK = solve_chol(L,eye(n))/sn2; for j=1:numel(hyp.cov) dKj = feval(cov{:}, hyp.cov, x, [], j); Zj = solve_chol(L,dKj)/sn2; % GPML book, eq. 5.13 dva = sum(Zj.*iK,2)./(r.*r); % GPML book, eq. 5.13 dmn = (Zj*alpha)./r - alpha.*dva; % GPML book, eq. 5.13 if j==isn, dva = dva - 2*sn2; end % additional part for covNoise dnloo.cov(j) = -dlZmn'*dmn -dlZva'*dva; end for j=1:numel(hyp.lik) dnloo.lik(j) = -sum( feval(lik{:},hyp.lik,y,loo.fmu,loo.fs2,'infEP',j) ); if j==numel(hyp.lik) dva = 2*sn2*sum(iK.*iK,2)./(r.*r); % GPML book, eq. 5.13 dmn = 2*solve_chol(L,alpha)./r - alpha.*dva; % GPML book, eq. 5.13 dva = dva - 2*sn2; dnloo.lik(j) = dnloo.lik(j) -dlZmn'*dmn -dlZva'*dva; end end for j=1:numel(hyp.mean) dmj = feval(mean{:}, hyp.mean, x, j); dmn = solve_chol(L,dmj)./(sn2*r); dnloo.mean(j) = -dlZmn'*dmn; end end end gpml-matlab-v3.5-2014-12-08/inf/infPrior.m000664 000765 000024 00000013227 12440106624 017611 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infPrior(inf, prior, hyp, varargin) % Inference with hyperparameter prior. Any inference method can be used. % The prior specification is done by means of the prior parameter that has % the same structure as hyp. Examples: % 1) >> prior.cov{2} = {@priorLaplace,mu,s2}; % put a Laplace prior on the second covariance function hyperparameter % i.e. hyp.cov(2) ~ Laplace(mu,s2) % 2) >> prior.mean{3} = {@priorDelta} or % >> prior.mean{3} = @priorDelta or % >> prior.mean{3} = 'priorClamped' % fix the third mean function hyperparameter to its current value and % exclude it from optimisation % i.e. hyp.mean(3) ~ delta(hyp.mean(3)) % 3) >> prior.xu = repmat({{@priorGauss,0,1}},D*nu,1) % put an individual standard normal prior on every inducing input coordinate % for FITC inference % i.e. hyp.xu(i) ~ N(0,1) for all i = 1..D*nu % 4) >> prior.multi{1} = {@priorGaussMulti,mu,s2, struct('xu',1:D*nu)} % put a joint Gaussian prior with mean vector mu and covariance matrix % s2 on the inducing input coordinates in FITC inference % i.e. hyp.xu ~ N(mu,s2) % 5) >> prior.multi{1} = {@priorTMulti,mu,s2,nu, off+(1:D)} % put a joint Student's t prior with mean vector mu, covariance matrix s2 % and nu degrees of freedom on the weights of the mean function e.g. % hyp.mean ~ T(mu,s2,nu) for meanLinear % Here, we do not have a struct as in 4) but the index in unwrap(hyp). % 6) >> p = {@priorWeibull,lam,k}; g = @exp; dg = @exp; ig = @log; % >> prior.cov{2} = {@priorTransform,g,dg,ig,p} % put a univariate Weibull distribution on the exponentiated second % covariance function hyperparameter % i.e. exp(hyp.cov(2)) ~ Weibull(lam,k) % % Return a parametrization of the posterior, the negative log marginal % likelihood and its derivatives w.r.t. the hyperparameters. % % Note that at most 1 prior is allowed per hyperparameter. We consider the % specification of more than 1 prior for a hyperparameter as an error. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Hannes Nickisch and Roman Garnett, 2014-12-04. % % See also INFMETHODS.M, USAGEPRIOR.M, PRIORDISTRIBUTIONS.M. [post nlZ dnlZ] = inf(hyp, varargin{:}); % perform inference if ~isempty(prior) % add hyperprior contributions to nlZ and dnlZ nam = fieldnames(prior); num = zeros(numel(unwrap(hyp)),1); % number of priors per hyperparameter for i=1:numel(nam) % iterate over kinds of hyperparameters cov/lik/mean/xu ni = nam{i}; % name of the ith field if strcmp(ni,'multi') % first catch multivariate priors p = prior.(ni); for j=1:numel(p) % iterate over individual muti-hyperparameter components pj = p{j}; if ~isempty(pj) % only proceed if a nonempty prior is specified idx = pj{end}; pj = pj(1:end-1); % grab index from cell end pjstr = pj{1}; if ~ischar(pjstr), pjstr = func2str(pjstr); end if numel(pjstr)<5 || ~strcmp(pjstr(end-4:end),'Multi') error('multivariate hyperpriors are called Multi') end if isstruct(idx) % massage idx into a vector idxs = rewrap(hyp,zeros(size(num))); % structured index for nj = fieldnames(idx), idxs.(nj{1})( idx.(nj{1}) ) = 1; end idx = unwrap(idxs)>0; % linearise structured index else idxz = zeros(size(num)); idxz(idx) = 1; idx = idxz>0; % binary index end if sum(idx)<=1, error('multivariate priors need >1 hyperparam'), end num(idx) = num(idx)+1; % inc prior counter hypu = unwrap(hyp); dnlZ = unwrap(dnlZ); if strncmp(pjstr,'priorClamped',12) || ... % clamp derivative to zero strncmp(pjstr,'priorDelta',10), dnlZ(idx) = 0; else [lp,dlp] = feval(pj{:}, hypu(idx)); % evaluate prior distribution nlZ = nlZ-lp; dnlZ(idx) = dnlZ(idx)-dlp(:); % update nlZ and dnlZ end dnlZ = rewrap(hyp,dnlZ); else error('multivariate priors should be non empty') end end continue % jump to univariate priors end if ~isfield(hyp,ni), error(['unknown prior field ',ni,' in hyp']), end p = prior.(ni); if numel(p)~=numel(hyp.(ni)), error(['bad hyp/prior field length ',ni]), end for j=1:numel(p) % iterate over individual hyperparameter components pj = p{j}; if ~iscell(pj) && ~isempty(pj), pj = {pj}; end % enforce cell if ~isempty(pj) % only proceed if a nonempty prior is specified num = rewrap(hyp,num); num.(ni)(j) = num.(ni)(j)+1; % inc prior counter num = unwrap(num); pj1str = pj{1}; if ~ischar(pj1str), pj1str = func2str(pj1str); end if strncmp(pj1str,'priorClamped',12) || strncmp(pj1str,'priorDelta',10) dnlZ.(ni)(j) = 0; % clamp derivative to zero else [lp,dlp] = feval(pj{:}, hyp.(ni)(j)); % evaluate prior distribution nlZ = nlZ-lp; dnlZ.(ni)(j) = dnlZ.(ni)(j)-dlp; % update nlZ and dnlZ end end end end if any(num>1) % check for hypers with more than a single prior num = rewrap(hyp,num); nam = fieldnames(num); s = ''; for i=1:numel(nam) idx = find(num.(nam{i})>1); for j=1:numel(idx) s = [s,sprintf('hyp.%s(%d) ',nam{i},idx(j))]; end end error(['More than 1 prior specified for ',s(1:end-1),'.']) end endgpml-matlab-v3.5-2014-12-08/util/binaryEPGP.m000664 000765 000024 00000005753 12425212207 020165 0ustar00carlstaff000000 000000 function varargout = binaryEPGP(hyper, covfunc, x, y, xstar) % binaryEPGP - The Expectation Propagation approximation for binary Gaussian % process classification. Two modes are possible: training or testing: if no % test cases are supplied, then the approximate negative log marginal % likelihood and its partial derivatives wrt the hyperparameters is computed; % this mode is used to fit the hyperparameters. If test cases are given, then % the test set predictive probabilities are returned. The program is flexible % in allowing a multitude of covariance functions. % % usage: [nlZ, dnlZ ] = binaryEPGP(hyper, covfunc, x, y); % or: [p, mu, s2, nlZ] = binaryEPGP(hyper, covfunc, x, y, xstar); % % where: % % hyper is a (column) vector of hyperparameters % covfunc is the name of the covariance function (see below) % lik is the name of the likelihood function (see below) % x is a n by D matrix of training inputs % y is a (column) vector (of size n) of binary +1/-1 targets % xstar is a nn by D matrix of test inputs % nlZ is the returned value of the negative log marginal likelihood % dnlZ is a (column) vector of partial derivatives of the negative % log marginal likelihood wrt each log hyperparameter % p is a (column) vector (of length nn) of predictive probabilities % mu is a (column) vector (of length nn) of predictive latent means % s2 is a (column) vector (of length nn) of predictive latent variances % % The length of the vector of hyperparameters depends on the covariance % function, as specified by the "covfunc" input to the function, specifying the % name of a covariance function. A number of different covariance function are % implemented, and it is not difficult to add new ones. See "help covFunctions" % for the details % % The function can conveniently be used with the "minimize" function to train % a Gaussian process, eg: % % [hyper, fX, i] = minimize(hyper, 'binaryEPGP', length, 'covSEiso', % 'logistic', x, y); % % Copyright (c) 2004, 2005, 2006, 2007 Carl Edward Rasmussen, 2010-07-21. if nargin<4 || nargin>5 disp('Usage: [nlZ, dnlZ ] = binaryEPGP(hyper, covfunc, x, y);') disp(' or: [p, mu, s2, nlZ] = binaryEPGP(hyper, covfunc, x, y, xstar);') return end % Note, this function is just a wrapper provided for backward compatibility, % the functionality is now provided by the more general gp function. lik = @likErf; inf = @infEP; mean = @meanZero; varargout = cell(nargout, 1); % allocate the right number of output arguments hyp.cov = hyper; if nargin==4 [varargout{:}] = gp(hyp,inf,mean,covfunc,lik,x,y); if nargout>1, varargout{2} = varargout{2}.cov; end else [ymu,ys2,fmu,fs2] = gp(hyp,inf,mean,covfunc,lik,x,y,xstar); if nargout>0, varargout{1} = (1+ymu)/2; end if nargout>1, varargout{2} = fmu; end if nargout>2, varargout{3} = fs2; end if nargout>3, varargout{4} = gp(hyp,inf,mean,covfunc,lik,x,y); end % nlZ end gpml-matlab-v3.5-2014-12-08/util/binaryLaplaceGP.m000664 000765 000024 00000007030 12425212207 021210 0ustar00carlstaff000000 000000 function varargout = binaryLaplaceGP(hyper, covfunc, lik, x, y, xstar) % binaryLaplaceGP - Laplace's approximation for binary Gaussian process % classification. Two modes are possible: training or testing: if no test % cases are supplied, then the approximate negative log marginal likelihood % and its partial derivatives wrt the hyperparameters is computed; this mode is % used to fit the hyperparameters. If test cases are given, then the test set % predictive probabilities are returned. The program is flexible in allowing % several different likelihood functions and a multitude of covariance % functions. % % usage: [nlZ, dnlZ ] = binaryLaplaceGP(hyper, covfunc, lik, x, y); % or: [p, mu, s2, nlZ] = binaryLaplaceGP(hyper, covfunc, lik, x, y, xstar); % % where: % % hyper is a (column) vector of hyperparameters % covfunc is the name of the covariance function (see below) % lik is the name of the likelihood function (see below) % x is a n by D matrix of training inputs % y is a (column) vector (of size n) of binary +1/-1 targets % xstar is a nn by D matrix of test inputs % nlZ is the returned value of the negative log marginal likelihood % dnlZ is a (column) vector of partial derivatives of the negative % log marginal likelihood wrt each log hyperparameter % p is a (column) vector (of length nn) of predictive probabilities % mu is a (column) vector (of length nn) of predictive latent means % s2 is a (column) vector (of length nn) of predictive latent variances % % The length of the vector of log hyperparameters depends on the covariance % function, as specified by the "covfunc" input to the function, specifying the % name of a covariance function. A number of different covariance function are % implemented, and it is not difficult to add new ones. See "help covFunctions" % for the details. % % The shape of the likelihood function is given by the "lik" input to the % function, specifying the name of the likelihood function. The two implemented % likelihood functions are: % % logistic the logistic function: 1/(1+exp(-x)) % cumGauss the cumulative Gaussian (error function) % % The function can conveniently be used with the "minimize" function to train % a Gaussian process, eg: % % [hyper, fX, i] = minimize(hyper, 'binaryLaplaceGP', length, 'covSEiso', % 'logistic', x, y); % % Copyright (c) 2004, 2005, 2006, 2007 Carl Edward Rasmussen, 2010-07-21. if nargin<5 || nargin>6 disp('Usage: [nlZ, dnlZ ] = binaryLaplaceGP(hyper, covfunc, lik, x, y);') disp(' or: [p, mu, s2, nlZ] = binaryLaplaceGP(hyper, covfunc, lik, x, y, xstar);') return end % Note, this function is just a wrapper provided for backward compatibility, % the functionality is now provided by the more general gp function. if strcmp(lik,'logistic') lik = @likLogistic; elseif strcmpi(lik,'cumgauss') lik = @likErf; else error('Allowable likelihoods: logistic and cumGauss.'); end inf = @infLaplace; mean = @meanZero; varargout = cell(nargout, 1); % allocate the right number of output arguments hyp.cov = hyper; if nargin==5 [varargout{:}] = gp(hyp,inf,mean,covfunc,lik,x,y); if nargout>1, varargout{2} = varargout{2}.cov; end else [ymu,ys2,fmu,fs2] = gp(hyp,inf,mean,covfunc,lik,x,y,xstar); if nargout>0, varargout{1} = (1+ymu)/2; end if nargout>1, varargout{2} = fmu; end if nargout>2, varargout{3} = fs2; end if nargout>3, varargout{4} = gp(hyp,inf,mean,covfunc,lik,x,y); end % nlZ end gpml-matlab-v3.5-2014-12-08/util/logsumexp2.m000664 000765 000024 00000000706 12425212207 020323 0ustar00carlstaff000000 000000 % Compute y = log( sum(exp(x),2) ), the softmax in a numerically safe way by % subtracting the row maximum to avoid cancelation after taking the exp % the sum is done along the rows. % % Copyright (c) by Hannes Nickisch, 2013-10-16. function [y,x] = logsumexp2(logx) N = size(logx,2); max_logx = max(logx,[],2); % we have all values in the log domain, and want to calculate a sum x = exp(logx-max_logx*ones(1,N)); y = log(sum(x,2)) + max_logx;gpml-matlab-v3.5-2014-12-08/util/solve_chol.c000664 000765 000024 00000004371 12425212207 020343 0ustar00carlstaff000000 000000 /* solve_chol - solve a linear system A*X = B using the Cholesky factorization * of A (square, symmetric and positive definite) using LAPACK/DPOTRS. * * Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2014-02-13. * * Modifications from 2014-02-09 as suggested by Todd Small: * - types of q,m,n changed from long to integer, q error check had wrong sign */ #include "mex.h" #include #ifdef MEX_INFORMATION_VERSION /* now we are compiling for Matlab */ #if defined(_WIN64) #define integer long long #else #define integer long #endif #define doublereal double #else /* now we are compiling for Octave */ #ifdef __APPLE__ #include typedef __CLPK_integer integer; typedef __CLPK_doublereal doublereal; #else typedef int integer; typedef double doublereal; #endif #endif #if !defined(_WIN32) || !defined(MEX_INFORMATION_VERSION) /* not Win32/Matlab */ #define dpotrs dpotrs_ #endif extern integer dpotrs_(char *, integer *, integer *, doublereal *, integer *, doublereal *, integer *, integer *); void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) { /* Matlab call: X = solve_chol(A, B) */ doublereal *C; integer n, m, q; if (nrhs != 2 || nlhs > 1) /* check input */ mexErrMsgTxt("Usage: X = solve_chol(A, B)"); n = mxGetM(prhs[0]); /* number of rows in inputs A and B */ if (n != mxGetN(prhs[0])) mexErrMsgTxt("First argument matrix must be square."); if (n != mxGetM(prhs[1])) mexErrMsgTxt("Both argument matrices must have the same number of rows."); m = mxGetN(prhs[1]); /* number of colums in second input B */ plhs[0] = mxCreateDoubleMatrix(n, m, mxREAL); /* space for output X */ C = mxGetPr(plhs[0]); if (n == 0) return; /* if argument was empty matrix, do no more */ memcpy( C, mxGetPr(prhs[1]), m*n*mxGetElementSize(plhs[0]) ); /* copy data */ dpotrs("U", &n, &m, mxGetPr(prhs[0]), &n, C, &n, &q); /* solve system */ if (q < 0) mexErrMsgTxt("Error: illegal input to solve_chol"); } gpml-matlab-v3.5-2014-12-08/util/solve_chol.m000664 000765 000024 00000001742 12425212207 020354 0ustar00carlstaff000000 000000 % solve_chol - solve linear equations from the Cholesky factorization. % Solve A*X = B for X, where A is square, symmetric, positive definite. The % input to the function is R the Cholesky decomposition of A and the matrix B. % Example: X = solve_chol(chol(A),B); % % NOTE: The program code is written in the C language for efficiency and is % contained in the file solve_chol.c, and should be compiled using matlabs mex % facility. However, this file also contains a (less efficient) matlab % implementation, supplied only as a help to people unfamiliar with mex. If % the C code has been properly compiled and is available, it automatically % takes precendence over the matlab code in this file. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2010-09-18. function X = solve_chol(L, B) if nargin ~= 2 || nargout > 1 error('Wrong number of arguments.'); end if size(L,1) ~= size(L,2) || size(L,1) ~= size(B,1) error('Wrong sizes of matrix arguments.'); end X = L\(L'\B);gpml-matlab-v3.5-2014-12-08/util/lbfgsb.m000664 000765 000024 00000005563 12425212207 017463 0ustar00carlstaff000000 000000 % LBFGSB Call the nonlinear bound-constrained solver that uses % limited-memory BFGS quasi-Newton updates. % % The basic function call is % % LBFGSB(x0,lb,ub,objfunc,gradfunc) % % The first input argument x0 is either a matrix or a cell array of % matrices. It declares the starting point for the solver. % % The second and third input arguments lb and ub must be of the exact same % structure as x0. They declare the lower and upper bounds on the % variables, respectively. Set an entry to Inf to declare no bound. % % The next two input arguments must be the names of MATLAB routines % (M-files): % % objfunc Calculates the objective function at the current % point. The routine must accept as many inputs as cell % entry in x0 (or one input if x0 is a matrix). The % output must be a scalar representing the objective % evaluated at the current point. % % gradfunc Computes the gradient of the objective at the current % point. The input is the same as objfunc, but it must % return as many outputs as there are inputs, and each % of the outputs have the same matrix structure as its % corresponding input. % % Optionally, one may choose to pass additional auxiliary data to the % MATLAB callback routines listed above through the function call % LBFGSB(...,auxdata). If auxdata is the empty matrix, no extra % information is passed. It is important to observe that the auxiliary % data MAY NOT CHANGE through the course of the L-BFGS optimization! The % auxiliary data keep the same values as they possessed in the initial % call. If you need variables that change over time, you may want to % consider global variables (type HELP GLOBAL). % % LBFGSB(...,auxdata,iterfunc) specifies an additional callback routine % which is called once per algorithm iteration. The callback routine must % take the form ITERFUNC(T,F,X1,X2,...,AUXDATA). T is the current % iteration of the algorithm. F is the current value of the objective. The % inputs X1, X2,... are exactly the same as the ones passed to objfunc and % gradfunc. Finally, extra information may be passed through the input % AUXDATA. No outputs are expected from iterfunc. If iterfunc is the empty % string, no routine is called. % % These inputs may be followed by parameter/value pairs to modify the % default algorithm options. The algorithm option 'maxiter' sets the % maximum number of iterations. The rest of the options 'm', 'factr' and % 'pgtol' are described in the L-BFGS-B documentation. Note that in this % implementation we divide factr by the machine precision, so that the % convergence condition is satisfied when it is less than factr, not less % than factr times the machine precision. gpml-matlab-v3.5-2014-12-08/util/lbfgsb/000775 000765 000024 00000000000 12526350576 017312 5ustar00carlstaff000000 000000 gpml-matlab-v3.5-2014-12-08/util/minimize_lbfgsb_gradfun.m000664 000765 000024 00000004526 12425212207 023070 0ustar00carlstaff000000 000000 function G = minimize_lbfgsb_gradfun(X,varargin) % extract input arguments varargin = varargin{1}; strctX = varargin{2}; f = varargin{1}; % global variables serve as communication interface between calls global minimize_lbfgsb_iteration_number global minimize_lbfgsb_objective global minimize_lbfgsb_gradient global minimize_lbfgsb_X if norm(unwrap(X)-unwrap(minimize_lbfgsb_X))>1e-10 [y,G] = feval(f,rewrap(strctX,X),varargin{3:end}); else y = minimize_lbfgsb_objective(minimize_lbfgsb_iteration_number); G = minimize_lbfgsb_gradient; end % memorise gradient and position minimize_lbfgsb_gradient = G; minimize_lbfgsb_X = X; G = unwrap(G); % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse rewrap.m. function v = unwrap(s) v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s)));% alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; unwrap(s{i})]; end end % other types are ignored % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also unwrap.m. function [s v] = rewrap(s, v) if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed gpml-matlab-v3.5-2014-12-08/util/glm_invlink_exp.m000664 000765 000024 00000000673 12425212207 021406 0ustar00carlstaff000000 000000 % Compute the log intensity for the inverse link function g(f) = exp(f). % % The function is used in GLM likelihoods such as likPoisson, likGamma, likBeta % and likInvGauss. % % Copyright (c) by Hannes Nickisch, 2013-10-16. function [lg,dlg,d2lg,d3lg] = glm_invlink_exp(f) lg = f; if nargout>1 dlg = ones(size(f)); if nargout>2 d2lg = zeros(size(f)); if nargout>2 d3lg = zeros(size(f)); end end endgpml-matlab-v3.5-2014-12-08/util/minimize_lbfgsb.m000664 000765 000024 00000010574 12425212207 021362 0ustar00carlstaff000000 000000 function [X, fX, i] = minimize_lbfgsb(X, f, length, varargin) % Minimize a differentiable multivariate function using quasi Newton. % % Usage: [X, fX, i] = minimize_lbfgsb(X, f, length, P1, P2, P3, ... ) % % X initial guess; may be of any type, including struct and cell array % f the name or pointer to the function to be minimized. The function % f must return two arguments, the value of the function, and it's % partial derivatives wrt the elements of X. The partial derivative % must have the same type as X. % length length of the run; if it is positive, it gives the maximum number of % line searches, if negative its absolute gives the maximum allowed % number of function evaluations. Optionally, length can have a second % component, which will indicate the reduction in function value to be % expected in the first line-search (defaults to 1.0). % P1, P2 ... parameters are passed to the function f. % % X the returned solution % fX vector of function values indicating progress made % i number of iterations (line searches or function evaluations, % depending on the sign of "length") used at termination. % % The function returns when either its length is up, or if no further progress % can be made (ie, we are at a (local) minimum, or so close that due to % numerical problems, we cannot get any closer). NOTE: If the function % terminates within a few iterations, it could be an indication that the % function values and derivatives are not consistent (ie, there may be a bug in % the implementation of your "f" function). % % Copyright (C) 2010 by Hannes Nickisch, 2010-02-05 % global variables serve as communication interface between calls global minimize_lbfgsb_iteration_number global minimize_lbfgsb_objective global minimize_lbfgsb_gradient global minimize_lbfgsb_X % init global variables minimize_lbfgsb_iteration_number = 0; minimize_lbfgsb_objective = Inf; minimize_lbfgsb_gradient = 0*unwrap(X); minimize_lbfgsb_X = X; X0 = X; lb = -Inf*ones(size(unwrap(X0))); ub = Inf*ones(size(unwrap(X0))); maxiter = abs(length); % max number of iterations % no callback routine used so far % m is the number of saved vectors used to estimate the Hessian % factr is the precision 1e-12 X = lbfgsb( unwrap(X0), lb, ub, 'minimize_lbfgsb_objfun', ... 'minimize_lbfgsb_gradfun', ... {f,X0,varargin{:}}, [], ... 'maxiter',maxiter, 'm',4, 'factr',1e-12, 'pgtol',1e-5); i = minimize_lbfgsb_iteration_number; fX = minimize_lbfgsb_objective; X = rewrap(X0,X); % clear global variables clear minimize_lbfgsb_iteration_number clear minimize_lbfgsb_objective clear minimize_lbfgsb_gradient clear minimize_lbfgsb_X % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse rewrap.m. function v = unwrap(s) v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s)));% alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; unwrap(s{i})]; end end % other types are ignored % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also unwrap.m. function [s v] = rewrap(s, v) if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed gpml-matlab-v3.5-2014-12-08/util/gpr.m000664 000765 000024 00000004667 12425212207 017020 0ustar00carlstaff000000 000000 function varargout = gpr(hyper, cov, x, y, xs) % gpr - Gaussian process regression, with a named covariance function. Two % modes are possible: training and prediction: if no test data are given, the % function returns minus the log likelihood and its partial derivatives with % respect to the hyperparameters; this mode is used to fit the hyperparameters. % If test data are given, then (marginal) Gaussian predictions are computed, % whose mean and variance are returned. Note that in cases where the covariance % function has noise contributions, the variance returned in s2 is for noisy % test targets; if you want the variance of the noise-free latent function, you % must substract the noise variance. % % usage: [nlZ dnlZ] = gpr(hyp, cov, x, y) % or: [mu s2] = gpr(hyp, cov, x, y, xs) % % where: % % hyp is a (column) vector of log hyperparameters % cov is the covariance function % x is a n by D matrix of training inputs % y is a (column) vector (of size n) of targets % xs is a ns by D matrix of test inputs % nlZ is the returned value of the negative log marginal likelihood % dnlZ is a (column) vector of partial derivatives of the negative % log marginal likelihood wrt each log hyperparameter % mu is a (column) vector (of size nn) of prediced means % s2 is a (column) vector (of size nn) of predicted variances % % For more help on covariance functions, see "help covFunctions". % % Copyright (c) 2010 Carl Edward Rasmussen and Hannes Nickisch 2010-06-18. err = 'we need cov = {''covSum'', {cov1, ..,''covNoise'', .., covP} }; to map to gp.m'; if ~strcmp(cov{1},'covSum'), error(err), end id = 0; nhyp = zeros(length(cov{2}),1); for i=1:length(cov{2}) if strcmp(cov{2}{i},'covNoise'), id = i; break, end nhyp(i) = eval(feval(cov{2}{i})); end if id==0, error(err), else nhyp = sum(nhyp(1:id-1)); end cov{2} = cov{2}([1:id-1,id+1:end]); hyp.lik = hyper(nhyp+1); hyp.cov = hyper([1:nhyp,nhyp+2:end]); % Note, this function is just a wrapper provided for backward compatibility, % the functionality is now provided by the more general gp function. lik = @likGauss; inf = @infExact; mean = @meanZero; varargout = cell(nargout, 1); % allocate the right number of output arguments if nargin==4 [varargout{:}] = gp(hyp,inf,mean,cov,lik,x,y); if nargout>1, varargout{2} = [varargout{2}.lik; varargout{2}.cov]; end else [varargout{:}] = gp(hyp,inf,mean,cov,lik,x,y,xs); end gpml-matlab-v3.5-2014-12-08/util/glm_invlink_logit.m000664 000765 000024 00000001422 12425212207 021721 0ustar00carlstaff000000 000000 % Compute the log intensity for the inverse link function g(f) = 1/(1+exp(-f)). % % The function is used in GLM likelihoods such as likPoisson, likGamma, likBeta % and likInvGauss. % % Copyright (c) by Hannes Nickisch, 2013-10-16. function varargout = glm_invlink_logit(f) varargout = cell(nargout, 1); % allocate the right number of output arguments [varargout{:}] = glm_invlink_logistic(f); if nargout>0 elg = exp(varargout{1}); varargout{1} = f - elg; if nargout>1 dlg = varargout{2}; varargout{2} = 1 - elg.*dlg; if nargout>2 d2lg = varargout{3}; varargout{3} = -elg.*(dlg.^2+d2lg); if nargout>3 d3lg = varargout{4}; varargout{4} = -elg.*(dlg.^3+3*d2lg.*dlg+d3lg); end end end endgpml-matlab-v3.5-2014-12-08/util/glm_invlink_expexp.m000664 000765 000024 00000000653 12425212207 022121 0ustar00carlstaff000000 000000 % Compute the log intensity for the inverse link function g(f) = exp(-exp(-f)). % % The function is used in GLM likelihoods such as likPoisson, likGamma, likBeta % and likInvGauss. % % Copyright (c) by Hannes Nickisch, 2013-10-16. function [lg,dlg,d2lg,d3lg] = glm_invlink_expexp(f) lg = -exp(-f); if nargout>1 dlg = -lg; if nargout>2 d2lg = lg; if nargout>2 d3lg = -lg; end end endgpml-matlab-v3.5-2014-12-08/util/sq_dist.m000664 000765 000024 00000003657 12425212207 017674 0ustar00carlstaff000000 000000 % sq_dist - a function to compute a matrix of all pairwise squared distances % between two sets of vectors, stored in the columns of the two matrices, a % (of size D by n) and b (of size D by m). If only a single argument is given % or the second matrix is empty, the missing matrix is taken to be identical % to the first. % % Usage: C = sq_dist(a, b) % or: C = sq_dist(a) or equiv.: C = sq_dist(a, []) % % Where a is of size Dxn, b is of size Dxm (or empty), C is of size nxm. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-12-13. function C = sq_dist(a, b) if nargin<1 || nargin>3 || nargout>1, error('Wrong number of arguments.'); end bsx = exist('bsxfun','builtin'); % since Matlab R2007a 7.4.0 and Octave 3.0 if ~bsx, bsx = exist('bsxfun'); end % bsxfun is not yet "builtin" in Octave [D, n] = size(a); % Computation of a^2 - 2*a*b + b^2 is less stable than (a-b)^2 because numerical % precision can be lost when both a and b have very large absolute value and the % same sign. For that reason, we subtract the mean from the data beforehand to % stabilise the computations. This is OK because the squared error is % independent of the mean. if nargin==1 % subtract mean mu = mean(a,2); if bsx a = bsxfun(@minus,a,mu); else a = a - repmat(mu,1,size(a,2)); end b = a; m = n; else [d, m] = size(b); if d ~= D, error('Error: column lengths must agree.'); end mu = (m/(n+m))*mean(b,2) + (n/(n+m))*mean(a,2); if bsx a = bsxfun(@minus,a,mu); b = bsxfun(@minus,b,mu); else a = a - repmat(mu,1,n); b = b - repmat(mu,1,m); end end if bsx % compute squared distances C = bsxfun(@plus,sum(a.*a,1)',bsxfun(@minus,sum(b.*b,1),2*a'*b)); else C = repmat(sum(a.*a,1)',1,m) + repmat(sum(b.*b,1),n,1) - 2*a'*b; end C = max(C,0); % numerical noise can cause C to negative i.e. C > -1e-14 gpml-matlab-v3.5-2014-12-08/util/cov_deriv_sq_dist.m000664 000765 000024 00000003562 12425212207 021727 0ustar00carlstaff000000 000000 % Compute derivative k'(x^p,x^q) of a stationary covariance k(d2) (ard or iso) % w.r.t. to squared distance d2 = (x^p - x^q)'*inv(P)*(x^p - x^q) measure. Here % P is either diagonal with ARD parameters ell_1^2,...,ell_D^2 where D is the % dimension of the input space or ell^2 times the unit matrix for isotropic % covariance. % The derivatives can only be computed for one of the following eight % covariance functions: cov{Matern|PP|RQ|SE}{iso|ard}. % % Copyright (c) by Hannes Nickisch, 2013-10-28. % % See also INFFITC.M, COVFITC.M. function Kp = cov_deriv_sq_dist(cov,hyp,x,z) if nargin<4, z = []; end % make sure, z exists xeqz = numel(z)==0; dg = strcmp(z,'diag') && numel(z)>0; % determine mode if iscell(cov), covstr = cov{1}; else covstr = cov; end if ~ischar(covstr), covstr = func2str(covstr); end if numel([strfind(covstr,'iso'),strfind(covstr,'ard')])==0 error('Only iso|ard covariances allowed for derivatives w.r.t. xu.') elseif numel([strfind(covstr,'covLIN'); strfind(covstr,'covGabor'); strfind(covstr,'covPER')])>0 error('Gabor|LIN|PER covariances not allowed for derivatives w.r.t. xu.') end [n,D] = size(x); if numel(strfind(covstr,'iso')), id = 1:D; else id = 1; end % *iso covariance ell1 = exp(hyp(1)); % first characteristic length scale Kp = feval(cov{:},hyp,x,z,1); % use derivative w.r.t. log(ell(1)) % precompute squared distances if dg % vector kxx d2 = zeros(n,1); else if xeqz % symmetric matrix Kxx d2 = sq_dist(x(:,id)'/ell1); else % cross covariances Kxz d2 = sq_dist(x(:,id)'/ell1,z(:,id)'/ell1); end end Kp = -1/2*Kp./d2; Kp(d2==0) = 0;gpml-matlab-v3.5-2014-12-08/util/make.m000664 000765 000024 00000004447 12425212207 017141 0ustar00carlstaff000000 000000 % For compilation under Windows/Octave you need to install the package % octavex.x-headers. % Under Linux/Octave, compilation works out of the box. % % There is a big variety of platforms running Matlab/Octave around and the % script is tested for: % % Linux 32bit / Matlab R2008b, 7.7 % Linux 64bit / Matlab R2010b, 7.11 % Windows 7 32bit / Matlab R14, 2004, 7.0 % Windows 7 64bit / Matlab R2011b, 7.13 % Windows XP 32bit / Matlab R2010b, 7.11 % OS X 10.7.5 / Matlab R2013b % OS X 10.9 / Matlab R2013b % % Linux 32bit / Octave 3.2.4 % OS X 10.9.1 / Octave 3.8.0 built with Homebrew % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2014-02-13. OCTAVE = exist('OCTAVE_VERSION') ~= 0; % check if we run Matlab or Octave me = mfilename; % what is my filename mydir = which(me); mydir = mydir(1:end-2-numel(me)); % where am I located cd(mydir) fprintf('Compiling solve_chol.c ...\n') if OCTAVE % Octave if ismac % Accelerate framework needed on OS X fprintf('mkoctfile --mex solve_chol.c "-Wl,-framework,Accelerate"\n') mkoctfile --mex solve_chol.c '-Wl,-framework,Accelerate' else fprintf('mkoctfile --mex solve_chol.c\n') mkoctfile --mex solve_chol.c end delete solve_chol.o else % Matlab if ispc % Windows try % take care of old Matlab versions cc = mex.getCompilerConfigurations; cc = lower(cc.Manufacturer);% compiler catch cc = 'lcc'; end if numel(strfind(computer,'64')) % 64 bit ospath = ['extern/lib/win64/',cc]; else % 32 bit ospath = ['extern/lib/win32/',cc]; end comp_cmd = 'mex -O solve_chol.c -output solve_chol'; fprintf('%s "../%s"\n',comp_cmd,[ospath,'/libmwlapack.lib']) eval([comp_cmd,' ''',matlabroot,'/',ospath,'/libmwlapack.lib','''']) else fprintf('mex -O -lmwlapack solve_chol.c\n') eval('mex -O -lmwlapack solve_chol.c') end end fprintf('Done!\n')gpml-matlab-v3.5-2014-12-08/util/logphi.m000664 000765 000024 00000004325 12425212207 017501 0ustar00carlstaff000000 000000 % Safe computation of logphi(z) = log(normcdf(z)) and its derivatives % dlogphi(z) = normpdf(x)/normcdf(x). % The function is based on index 5725 in Hart et al. and gsl_sf_log_erfc_e. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-11-13. function [lp,dlp,d2lp,d3lp] = logphi(z) z = real(z); % support for real arguments only lp = zeros(size(z)); % allocate memory id1 = z.*z<0.0492; % first case: close to zero lp0 = -z(id1)/sqrt(2*pi); c = [ 0.00048204; -0.00142906; 0.0013200243174; 0.0009461589032; -0.0045563339802; 0.00556964649138; 0.00125993961762116; -0.01621575378835404; 0.02629651521057465; -0.001829764677455021; 2*(1-pi/3); (4-pi)/3; 1; 1]; f = 0; for i=1:14, f = lp0.*(c(i)+f); end, lp(id1) = -2*f-log(2); id2 = z<-11.3137; % second case: very small r = [ 1.2753666447299659525; 5.019049726784267463450; 6.1602098531096305441; 7.409740605964741794425; 2.9788656263939928886 ]; q = [ 2.260528520767326969592; 9.3960340162350541504; 12.048951927855129036034; 17.081440747466004316; 9.608965327192787870698; 3.3690752069827527677 ]; num = 0.5641895835477550741; for i=1:5, num = -z(id2).*num/sqrt(2) + r(i); end den = 1.0; for i=1:6, den = -z(id2).*den/sqrt(2) + q(i); end e = num./den; lp(id2) = log(e/2) - z(id2).^2/2; id3 = ~id2 & ~id1; lp(id3) = log(erfc(-z(id3)/sqrt(2))/2); % third case: rest if nargout>1 % compute first derivative dlp = zeros(size(z)); % allocate memory dlp( id2) = abs(den./num) * sqrt(2/pi); % strictly positive first derivative dlp(~id2) = exp(-z(~id2).*z(~id2)/2-lp(~id2))/sqrt(2*pi); % safe computation if nargout>2 % compute second derivative d2lp = -dlp.*abs(z+dlp); % strictly negative second derivative if nargout>3 % compute third derivative d3lp = -d2lp.*abs(z+2*dlp)-dlp; % strictly positive third derivative end end endgpml-matlab-v3.5-2014-12-08/util/unwrap.m000664 000765 000024 00000001213 12425212207 017524 0ustar00carlstaff000000 000000 % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse rewrap.m. function v = unwrap(s) v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s))); % alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; unwrap(s{i})]; end end % other types are ignored gpml-matlab-v3.5-2014-12-08/util/minimize_v2.m000664 000765 000024 00000027260 12425212207 020452 0ustar00carlstaff000000 000000 % minimize.m - minimize a smooth differentiable multivariate function using % LBFGS (Limited memory LBFGS) or CG (Conjugate Gradients) % Usage: [X, fX, i] = minimize(X, F, p, other, ... ) % where % X is an initial guess (any type: vector, matrix, cell array, struct) % F is the objective function (function pointer or name) % p parameters - if p is a number, it corresponds to p.length below % p.length allowed 1) # linesearches or 2) if -ve minus # func evals % p.method minimization method, 'BFGS', 'LBFGS' or 'CG' % p.verbosity 0 quiet, 1 line, 2 line + warnings (default), 3 graphical % p.mem number of directions used in LBFGS (default 100) % other, ... other parameters, passed to the function F % X returned minimizer % fX vector of function values showing minimization progress % i final number of linesearches or function evaluations % The function F must take the following syntax [f, df] = F(X, other, ...) % where f is the function value and df its partial derivatives. The types of X % and df must be identical (vector, matrix, cell array, struct, etc). % % Copyright (C) 1996 - 2011 by Carl Edward Rasmussen, 2011-10-13. % Permission is hereby granted, free of charge, to any person OBTAINING A COPY % OF THIS SOFTWARE AND ASSOCIATED DOCUMENTATION FILES (THE "SOFTWARE"), TO DEAL % IN THE SOFTWARE WITHOUT RESTRICTION, INCLUDING WITHOUT LIMITATION THE RIGHTS % to use, copy, modify, merge, publish, distribute, sublicense, and/or sell % copies of the Software, and to permit persons to whom the Software is % furnished to do so, subject to the following conditions: % % The above copyright notice and this permission notice shall be included in % all copies or substantial portions of the Software. % % THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR % IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, % FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE % AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER % LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, % OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE % SOFTWARE. function [X, fX, i] = minimize(X, F, p, varargin) if isnumeric(p), p = struct('length', p); end % convert p to struct if p.length > 0, p.S = 'linesearch #'; else p.S = 'function evaluation #'; end; x = unwrap(X); % convert initial guess to vector if ~isfield(p,'method'), if length(x) > 1000, p.method = @LBFGS; else p.method = @BFGS; end; end % set default method if ~isfield(p,'verbosity'), p.verbosity = 2; end % default 1 line text output if ~isfield(p,'MFEPLS'), p.MFEPLS = 10; end % Max Func Evals Per Line Search if ~isfield(p,'MSR'), p.MSR = 100; end % Max Slope Ratio default f(F, X, varargin{:}); % set up the function f [fx, dfx] = f(x); % initial function value and derivatives if p.verbosity, printf('Initial Function Value %4.6e\r', fx); end if p.verbosity > 2, clf; subplot(211); hold on; xlabel(p.S); ylabel('function value'); plot(p.length < 0, fx, '+'); drawnow; end [x, fX, i] = feval(p.method, x, fx, dfx, p); % minimize using direction method X = rewrap(X, x); % convert answer to original representation if p.verbosity, printf('\n'); end function [x, fx, i] = CG(x0, fx0, dfx0, p) if ~isfield(p, 'SIG'), p.SIG = 0.1; end % default for line search quality i = p.length < 0; ok = 0; % initialize resource counter r = -dfx0; s = -r'*r; b = -1/(s-1); bs = -1; fx = fx0; % steepest descent while i < abs(p.length) b = b*bs/min(b*s,bs/p.MSR); % suitable initial step size using slope ratio [x, b, fx0, dfx, i] = lineSearch(x0, fx0, dfx0, r, s, b, i, p); if i < 0 % if line search failed i = -i; if ok, ok = 0; r = -dfx; else break; end % try steepest or stop else ok = 1; bs = b*s; % record step times slope (for slope ratio method) r = (dfx'*(dfx-dfx0))/(dfx0'*dfx0)*r - dfx; % Polack-Ribiere CG end s = r'*dfx; if s >= 0, r = -dfx; s = r'*dfx; ok = 0; end % slope must be -ve x0 = x; dfx0 = dfx; fx = [fx; fx0]; % replace old values with new ones end function [x, fx, i] = BFGS(x0, fx0, dfx0, p) if ~isfield(p, 'SIG'), p.SIG = 0.5; end % default for line search quality i = p.length < 0; ok = 0; % initialize resource counter x = x0; fx = fx0; r = -dfx0; s = -r'*r; b = -1/(s-1); H = eye(length(x0)); while i < abs(p.length) [x, b, fx0, dfx, i] = lineSearch(x0, fx0, dfx0, r, s, b, i, p); if i < 0 i = -i; if ok, ok = 0; else break; end; % try steepest or stop else ok = 1; t = x - x0; y = dfx - dfx0; ty = t'*y; Hy = H*y; H = H + (ty+y'*Hy)/ty^2*t*t' - 1/ty*Hy*t' - 1/ty*t*Hy'; % BFGS update end r = -H*dfx; s = r'*dfx; x0 = x; dfx0 = dfx; fx = [fx; fx0]; end function [x, fx, i] = LBFGS(x0, fx0, dfx0, p) if ~isfield(p, 'SIG'), p.SIG = 0.5; end % default for line search quality n = length(x0); k = 0; ok = 0; x = x0; fx = fx0; bs = -1/p.MSR; if isfield(p, 'mem'), m = p.mem; else m = min(100, n); end % set memory size a = zeros(1, m); t = zeros(n, m); y = zeros(n, m); % allocate memory i = p.length < 0; % initialize resource counter while i < abs(p.length) q = dfx0; for j = rem(k-1:-1:max(0,k-m),m)+1 a(j) = t(:,j)'*q/rho(j); q = q-a(j)*y(:,j); end if k == 0, r = -q/(q'*q); else r = -t(:,j)'*y(:,j)/(y(:,j)'*y(:,j))*q; end for j = rem(max(0,k-m):k-1,m)+1 r = r-t(:,j)*(a(j)+y(:,j)'*r/rho(j)); end s = r'*dfx0; if s >= 0, r = -dfx0; s = r'*dfx0; k = 0; ok = 0; end b = bs/min(bs,s/p.MSR); % suitable initial step size (usually 1) if isnan(r) | isinf(r) % if nonsense direction i = -i; % try steepest or stop else [x, b, fx0, dfx, i] = lineSearch(x0, fx0, dfx0, r, s, b, i, p); end if i < 0 % if line search failed i = -i; if ok, ok = 0; k = 0; else break; end % try steepest or stop else j = rem(k,m)+1; t(:,j) = x-x0; y(:,j) = dfx-dfx0; rho(j) = t(:,j)'*y(:,j); ok = 1; k = k+1; bs = b*s; end x0 = x; dfx0 = dfx; fx = [fx; fx0]; % replace and add values end function [x, a, fx, df, i] = lineSearch(x0, f0, df0, d, s, a, i, p) if p.length < 0, LIMIT = min(p.MFEPLS, -i-p.length); else LIMIT = p.MFEPLS; end p0.x = 0.0; p0.f = f0; p0.df = df0; p0.s = s; p1 = p0; % init p0 and p1 j = 0; p3.x = a; wp(p0, p.SIG, 0); % set step & Wolfe-Powell conditions if p.verbosity > 2 A = [-a a]/5; nd = norm(d); subplot(212); hold off; plot(0, f0, 'k+'); hold on; plot(nd*A, f0+s*A, 'k-'); xlabel('distance in line search direction'); ylabel('function value'); end while 1 % keep extrapolating as long as necessary ok = 0; while ~ok & j < LIMIT try % try, catch and bisect to safeguard extrapolation evaluation j = j+1; [p3.f p3.df] = f(x0+p3.x*d); p3.s = p3.df'*d; ok = 1; if isnan(p3.f+p3.s) | isinf(p3.f+p3.s) error('Objective function returned Inf or NaN',''); end; catch if p.verbosity > 1, printf('\n'); warning(lasterr); end % warn or silence p3.x = (p1.x+p3.x)/2; ok = 0; p3.f = NaN; % bisect, and retry end end if p.verbosity > 2 plot(nd*p3.x, p3.f, 'b+'); plot(nd*(p3.x+A), p3.f+p3.s*A, 'b-'); drawnow end if wp(p3) | j >= LIMIT, break; end % done? p0 = p1; p1 = p3; % move points back one unit p3.x = p0.x + minCubic(p1.x-p0.x, p1.f-p0.f, p0.s, p1.s, 1); % cubic extrap end while 1 % keep interpolating as long as necessary if p1.f > p3.f, p2 = p3; else p2 = p1; end % make p2 the best so far if wp(p2) > 1 | j >= LIMIT, break; end % done? p2.x = p1.x + minCubic(p3.x-p1.x, p3.f-p1.f, p1.s, p3.s, 0); % cubic interp j = j+1; [p2.f p2.df] = f(x0+p2.x*d); p2.s = p2.df'*d; if p.verbosity > 2 plot(nd*p2.x, p2.f, 'r+'); plot(nd*(p2.x+A), p2.f+p2.s*A, 'r'); drawnow end if wp(p2) > -1 & p2.s > 0 | wp(p2) < -1, p3 = p2; else p1 = p2; end % bracket end x = x0+p2.x*d; fx = p2.f; df = p2.df; a = p2.x; % return the value found if p.length < 0, i = i+j; else i = i+1; end % count func evals or line searches if p.verbosity, printf('%s %6i; value %4.6e\r', p.S, i, fx); end if wp(p2) < 2, i = -i; end % indicate faliure if p.verbosity > 2 if i>0, plot(norm(d)*p2.x, fx, 'go'); end subplot(211); plot(abs(i), fx, '+'); drawnow; end function z = minCubic(x, df, s0, s1, extr) % minimizer of approximating cubic INT = 0.1; EXT = 5.0; % interpolate and extrapolation limits A = -6*df+3*(s0+s1)*x; B = 3*df-(2*s0+s1)*x; if B<0, z = s0*x/(s0-s1); else z = -s0*x*x/(B+sqrt(B*B-A*s0*x)); end if extr % are we extrapolating? if ~isreal(z) | ~isfinite(z) | z < x | z > x*EXT, z = EXT*x; end % fix bad z z = max(z, (1+INT)*x); % extrapolate by at least INT else % else, we are interpolating if ~isreal(z) | ~isfinite(z) | z < 0 | z > x, z = x/2; end; % fix bad z z = min(max(z, INT*x), (1-INT)*x); % at least INT away from the boundaries end function y = wp(p, SIG, RHO) persistent a b c sig rho; if nargin == 3 % if three arguments, then set up the Wolfe-Powell conditions a = RHO*p.s; b = p.f; c = -SIG*p.s; sig = SIG; rho = RHO; y= 0; else if p.f > a*p.x+b % function value too large? if a > 0, y = -1; else y = -2; end else if p.s < -c, y = 0; elseif p.s > c, y = 1; else y = 2; end % if sig*abs(p.s) > c, a = rho*p.s; b = p.f-a*p.x; c = sig*abs(p.s); end end end function [fx, dfx] = f(varargin) persistent F p; if nargout == 0 p = varargin; if ischar(p{1}), F = str2func(p{1}); else F = p{1}; end else [fx, dfx] = F(rewrap(p{2}, varargin{1}), p{3:end}); dfx = unwrap(dfx); end function v = unwrap(s) % extract num elements of s (any type) into v (vector) v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s))); % alphabetize, conv to cell, recurse elseif iscell(s) % cell array elements are for i = 1:numel(s), v = [v; unwrap(s{i})]; end % handled sequentially end % other types are ignored function [s v] = rewrap(s, v) % map elements of v (vector) onto s (any type) if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed function printf(varargin) fprintf(varargin{:}); if exist('fflush','builtin'), fflush(stdout); end gpml-matlab-v3.5-2014-12-08/util/brentmin.m000664 000765 000024 00000007574 12425212207 020046 0ustar00carlstaff000000 000000 %% BRENTMIN: Brent's minimization method in one dimension function [xmin,fmin,funccount,varargout] = ... brentmin(xlow,xupp,Nitmax,tol,f,nout,varargin) % code taken from % § 10.2 Parabolic Interpolation and Brent's Method in One Dimension % Press, Teukolsky, Vetterling & Flannery % Numerical Recipes in C, Cambridge University Press, 2002 % % [xmin,fmin,funccout,varargout] = BRENTMIN(xlow,xupp,Nit,tol,f,nout,varargin) % Given a function f, and given a search interval this routine isolates % the minimum of fractional precision of about tol using Brent's method. % % INPUT % ----- % xlow,xupp: search interval such that xlow<=xmin<=xupp % Nitmax: maximum number of function evaluations made by the routine % tol: fractional precision % f: [y,varargout{:}] = f(x,varargin{:}) is the function % nout: no. of outputs of f (in varargout) in addition to the y value % % OUTPUT % ------ % fmin: minimal function value % xmin: corresponding abscissa-value % funccount: number of function evaluations made % varargout: additional outputs of f at optimum % % Copyright (c) by Hannes Nickisch 2010-01-10. if nargin<6, nout = 0; end varargout = cell(nout,1); % tolerance is no smaller than machine's floating point precision tol = max(tol,eps); % Evaluate endpoints fa = f(xlow,varargin{:}); fb = f(xupp,varargin{:}); funccount = 2; % number of function evaluations % Compute the start point seps = sqrt(eps); c = 0.5*(3.0 - sqrt(5.0));% golden ratio a = xlow; b = xupp; v = a + c*(b-a); w = v; xf = v; d = 0.0; e = 0.0; x = xf; [fx,varargout{:}] = f(x,varargin{:}); funccount = funccount + 1; fv = fx; fw = fx; xm = 0.5*(a+b); tol1 = seps*abs(xf) + tol/3.0; tol2 = 2.0*tol1; % Main loop while ( abs(xf-xm) > (tol2 - 0.5*(b-a)) ) gs = 1; % Is a parabolic fit possible if abs(e) > tol1 % Yes, so fit parabola gs = 0; r = (xf-w)*(fx-fv); q = (xf-v)*(fx-fw); p = (xf-v)*q-(xf-w)*r; q = 2.0*(q-r); if q > 0.0, p = -p; end q = abs(q); r = e; e = d; % Is the parabola acceptable if ( (abs(p)q*(a-xf)) && (p= xm, e = a-xf; else e = b-xf; end d = c*e; end % The function must not be evaluated too close to xf si = sign(d) + (d == 0); x = xf + si * max( abs(d), tol1 ); [fu,varargout{:}] = f(x,varargin{:}); funccount = funccount + 1; % Update a, b, v, w, x, xm, tol1, tol2 if fu <= fx if x >= xf, a = xf; else b = xf; end v = w; fv = fw; w = xf; fw = fx; xf = x; fx = fu; else % fu > fx if x < xf, a = x; else b = x; end if ( (fu <= fw) || (w == xf) ) v = w; fv = fw; w = x; fw = fu; elseif ( (fu <= fv) || (v == xf) || (v == w) ) v = x; fv = fu; end end xm = 0.5*(a+b); tol1 = seps*abs(xf) + tol/3.0; tol2 = 2.0*tol1; if funccount >= Nitmax % typically we should not get here % warning(sprintf(['Maximum number of iterations (%d) exceeded:', ... % 'precision is not guaranteed'],Nitmax)) % fprintf('[%1.3f,%1.3f,%1.3f]\n',xlow,xf,xupp) break end end % while % check that endpoints are less than the minimum found if ( (fa < fx) && (fa <= fb) ) xf = xlow; fx = fa; elseif fb < fx xf = xupp; fx = fb; end fmin = fx; xmin = xf; gpml-matlab-v3.5-2014-12-08/util/minimize.m000664 000765 000024 00000025667 12425212207 020054 0ustar00carlstaff000000 000000 function [X, fX, i] = minimize(X, f, length, varargin) % Minimize a differentiable multivariate function using conjugate gradients. % % Usage: [X, fX, i] = minimize(X, f, length, P1, P2, P3, ... ) % % X initial guess; may be of any type, including struct and cell array % f the name or pointer to the function to be minimized. The function % f must return two arguments, the value of the function, and it's % partial derivatives wrt the elements of X. The partial derivative % must have the same type as X. % length length of the run; if it is positive, it gives the maximum number of % line searches, if negative its absolute gives the maximum allowed % number of function evaluations. Optionally, length can have a second % component, which will indicate the reduction in function value to be % expected in the first line-search (defaults to 1.0). % P1, P2, ... parameters are passed to the function f. % % X the returned solution % fX vector of function values indicating progress made % i number of iterations (line searches or function evaluations, % depending on the sign of "length") used at termination. % % The function returns when either its length is up, or if no further progress % can be made (ie, we are at a (local) minimum, or so close that due to % numerical problems, we cannot get any closer). NOTE: If the function % terminates within a few iterations, it could be an indication that the % function values and derivatives are not consistent (ie, there may be a bug in % the implementation of your "f" function). % % The Polack-Ribiere flavour of conjugate gradients is used to compute search % directions, and a line search using quadratic and cubic polynomial % approximations and the Wolfe-Powell stopping criteria is used together with % the slope ratio method for guessing initial step sizes. Additionally a bunch % of checks are made to make sure that exploration is taking place and that % extrapolation will not be unboundedly large. % % See also: checkgrad % % Copyright (C) 2001 - 2010 by Carl Edward Rasmussen, 2010-01-03 INT = 0.1; % don't reevaluate within 0.1 of the limit of the current bracket EXT = 3.0; % extrapolate maximum 3 times the current step-size MAX = 20; % max 20 function evaluations per line search RATIO = 10; % maximum allowed slope ratio SIG = 0.1; RHO = SIG/2; % SIG and RHO are the constants controlling the Wolfe- % Powell conditions. SIG is the maximum allowed absolute ratio between % previous and new slopes (derivatives in the search direction), thus setting % SIG to low (positive) values forces higher precision in the line-searches. % RHO is the minimum allowed fraction of the expected (from the slope at the % initial point in the linesearch). Constants must satisfy 0 < RHO < SIG < 1. % Tuning of SIG (depending on the nature of the function to be optimized) may % speed up the minimization; it is probably not worth playing much with RHO. % The code falls naturally into 3 parts, after the initial line search is % started in the direction of steepest descent. 1) we first enter a while loop % which uses point 1 (p1) and (p2) to compute an extrapolation (p3), until we % have extrapolated far enough (Wolfe-Powell conditions). 2) if necessary, we % enter the second loop which takes p2, p3 and p4 chooses the subinterval % containing a (local) minimum, and interpolates it, unil an acceptable point % is found (Wolfe-Powell conditions). Note, that points are always maintained % in order p0 <= p1 <= p2 < p3 < p4. 3) compute a new search direction using % conjugate gradients (Polack-Ribiere flavour), or revert to steepest if there % was a problem in the previous line-search. Return the best value so far, if % two consecutive line-searches fail, or whenever we run out of function % evaluations or line-searches. During extrapolation, the "f" function may fail % either with an error or returning Nan or Inf, and minimize should handle this % gracefully. if max(size(length)) == 2, red=length(2); length=length(1); else red=1; end if length>0, S='Linesearch'; else S='Function evaluation'; end i = 0; % zero the run length counter ls_failed = 0; % no previous line search has failed [f0 df0] = feval(f, X, varargin{:}); % get function value and gradient Z = X; X = unwrap(X); df0 = unwrap(df0); fprintf('%s %6i; Value %4.6e\r', S, i, f0); if exist('fflush','builtin') fflush(stdout); end fX = f0; i = i + (length<0); % count epochs?! s = -df0; d0 = -s'*s; % initial search direction (steepest) and slope x3 = red/(1-d0); % initial step is red/(|s|+1) while i < abs(length) % while not finished i = i + (length>0); % count iterations?! X0 = X; F0 = f0; dF0 = df0; % make a copy of current values if length>0, M = MAX; else M = min(MAX, -length-i); end while 1 % keep extrapolating as long as necessary x2 = 0; f2 = f0; d2 = d0; f3 = f0; df3 = df0; success = 0; while ~success && M > 0 try M = M - 1; i = i + (length<0); % count epochs?! [f3 df3] = feval(f, rewrap(Z,X+x3*s), varargin{:}); df3 = unwrap(df3); if isnan(f3) || isinf(f3) || any(isnan(df3)+isinf(df3)), error(' '),end success = 1; catch % catch any error which occured in f x3 = (x2+x3)/2; % bisect and try again end end if f3 < F0, X0 = X+x3*s; F0 = f3; dF0 = df3; end % keep best values d3 = df3'*s; % new slope if d3 > SIG*d0 || f3 > f0+x3*RHO*d0 || M == 0 % are we done extrapolating? break end x1 = x2; f1 = f2; d1 = d2; % move point 2 to point 1 x2 = x3; f2 = f3; d2 = d3; % move point 3 to point 2 A = 6*(f1-f2)+3*(d2+d1)*(x2-x1); % make cubic extrapolation B = 3*(f2-f1)-(2*d1+d2)*(x2-x1); x3 = x1-d1*(x2-x1)^2/(B+sqrt(B*B-A*d1*(x2-x1))); % num. error possible, ok! if ~isreal(x3) || isnan(x3) || isinf(x3) || x3 < 0 % num prob | wrong sign? x3 = x2*EXT; % extrapolate maximum amount elseif x3 > x2*EXT % new point beyond extrapolation limit? x3 = x2*EXT; % extrapolate maximum amount elseif x3 < x2+INT*(x2-x1) % new point too close to previous point? x3 = x2+INT*(x2-x1); end end % end extrapolation while (abs(d3) > -SIG*d0 || f3 > f0+x3*RHO*d0) && M > 0 % keep interpolating if d3 > 0 || f3 > f0+x3*RHO*d0 % choose subinterval x4 = x3; f4 = f3; d4 = d3; % move point 3 to point 4 else x2 = x3; f2 = f3; d2 = d3; % move point 3 to point 2 end if f4 > f0 x3 = x2-(0.5*d2*(x4-x2)^2)/(f4-f2-d2*(x4-x2)); % quadratic interpolation else A = 6*(f2-f4)/(x4-x2)+3*(d4+d2); % cubic interpolation B = 3*(f4-f2)-(2*d2+d4)*(x4-x2); x3 = x2+(sqrt(B*B-A*d2*(x4-x2)^2)-B)/A; % num. error possible, ok! end if isnan(x3) || isinf(x3) x3 = (x2+x4)/2; % if we had a numerical problem then bisect end x3 = max(min(x3, x4-INT*(x4-x2)),x2+INT*(x4-x2)); % don't accept too close [f3 df3] = feval(f, rewrap(Z,X+x3*s), varargin{:}); df3 = unwrap(df3); if f3 < F0, X0 = X+x3*s; F0 = f3; dF0 = df3; end % keep best values M = M - 1; i = i + (length<0); % count epochs?! d3 = df3'*s; % new slope end % end interpolation if abs(d3) < -SIG*d0 && f3 < f0+x3*RHO*d0 % if line search succeeded X = X+x3*s; f0 = f3; fX = [fX' f0]'; % update variables fprintf('%s %6i; Value %4.6e\r', S, i, f0); if exist('fflush','builtin') fflush(stdout); end s = (df3'*df3-df0'*df3)/(df0'*df0)*s - df3; % Polack-Ribiere CG direction df0 = df3; % swap derivatives d3 = d0; d0 = df0'*s; if d0 > 0 % new slope must be negative s = -df0; d0 = -s'*s; % otherwise use steepest direction end x3 = x3 * min(RATIO, d3/(d0-realmin)); % slope ratio but max RATIO ls_failed = 0; % this line search did not fail else X = X0; f0 = F0; df0 = dF0; % restore best point so far if ls_failed || i > abs(length) % line search failed twice in a row break; % or we ran out of time, so we give up end s = -df0; d0 = -s'*s; % try steepest x3 = 1/(1-d0); ls_failed = 1; % this line search failed end end X = rewrap(Z,X); fprintf('\n'); if exist('fflush','builtin') fflush(stdout); end function v = unwrap(s) % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse rewrap.m. v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s))); % alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; unwrap(s{i})]; end end % other types are ignored function [s v] = rewrap(s, v) % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also unwrap.m. if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed gpml-matlab-v3.5-2014-12-08/util/minimize_lbfgsb_objfun.m000664 000765 000024 00000005207 12425212207 022722 0ustar00carlstaff000000 000000 function y = minimize_lbfgsb_objfun(X,varargin) % extract input arguments varargin = varargin{1}; strctX = varargin{2}; f = varargin{1}; % global variables serve as communication interface between calls global minimize_lbfgsb_iteration_number global minimize_lbfgsb_objective global minimize_lbfgsb_gradient global minimize_lbfgsb_X recompute = 0; if minimize_lbfgsb_iteration_number==0 recompute = 1; else if norm(unwrap(X)-unwrap(minimize_lbfgsb_X))>1e-10 recompute = 1; end end if recompute [y,G] = feval(f,rewrap(strctX,X),varargin{3:end}); else y = minimize_lbfgsb_objective(minimize_lbfgsb_iteration_number); G = minimize_lbfgsb_gradient; end % increase global counter, memorise objective function, gradient and position minimize_lbfgsb_iteration_number = minimize_lbfgsb_iteration_number+1; minimize_lbfgsb_objective(minimize_lbfgsb_iteration_number,1) = y; minimize_lbfgsb_gradient = G; minimize_lbfgsb_X = X; % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse rewrap.m. function v = unwrap(s) v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s)));% alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; unwrap(s{i})]; end end % other types are ignored % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also unwrap.m. function [s v] = rewrap(s, v) if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed gpml-matlab-v3.5-2014-12-08/util/gauher.m000664 000765 000024 00000004305 12425212207 017470 0ustar00carlstaff000000 000000 % compute abscissas and weight factors for Gaussian-Hermite quadrature % % CALL: [x,w] = gauher(N) % % x = base points (abscissas) % w = weight factors % N = number of base points (abscissas) (integrates an up to (2N-1)th order % polynomial exactly) % % p(x)=exp(-x^2/2)/sqrt(2*pi), a =-Inf, b = Inf % % The Gaussian Quadrature integrates a (2n-1)th order % polynomial exactly and the integral is of the form % b N % Int ( p(x)* F(x) ) dx = Sum ( w_j* F( x_j ) ) % a j=1 % % this procedure uses the coefficients a(j), b(j) of the % recurrence relation % % b p (x) = (x - a ) p (x) - b p (x) % j j j j-1 j-1 j-2 % % for the various classical (normalized) orthogonal polynomials, % and the zero-th moment % % 1 = integral w(x) dx % % of the given polynomial's weight function w(x). Since the % polynomials are orthonormalized, the tridiagonal matrix is % guaranteed to be symmetric. function [x,w]=gauher(N) if N==20 % return precalculated values x=[ -7.619048541679757;-6.510590157013656;-5.578738805893203; -4.734581334046057;-3.943967350657318;-3.18901481655339 ; -2.458663611172367;-1.745247320814127;-1.042945348802751; -0.346964157081356; 0.346964157081356; 1.042945348802751; 1.745247320814127; 2.458663611172367; 3.18901481655339 ; 3.943967350657316; 4.734581334046057; 5.578738805893202; 6.510590157013653; 7.619048541679757]; w=[ 0.000000000000126; 0.000000000248206; 0.000000061274903; 0.00000440212109 ; 0.000128826279962; 0.00183010313108 ; 0.013997837447101; 0.061506372063977; 0.161739333984 ; 0.260793063449555; 0.260793063449555; 0.161739333984 ; 0.061506372063977; 0.013997837447101; 0.00183010313108 ; 0.000128826279962; 0.00000440212109 ; 0.000000061274903; 0.000000000248206; 0.000000000000126 ]; else b = sqrt( (1:N-1)/2 )'; [V,D] = eig( diag(b,1) + diag(b,-1) ); w = V(1,:)'.^2; x = sqrt(2)*diag(D); end gpml-matlab-v3.5-2014-12-08/util/minimize_v1.m000664 000765 000024 00000025702 12425212207 020450 0ustar00carlstaff000000 000000 function [X, fX, i] = minimize_old(X, f, length, varargin) % Minimize a differentiable multivariate function using conjugate gradients. % % Usage: [X, fX, i] = minimize(X, f, length, P1, P2, P3, ... ) % % X initial guess; may be of any type, including struct and cell array % f the name or pointer to the function to be minimized. The function % f must return two arguments, the value of the function, and it's % partial derivatives wrt the elements of X. The partial derivative % must have the same type as X. % length length of the run; if it is positive, it gives the maximum number of % line searches, if negative its absolute gives the maximum allowed % number of function evaluations. Optionally, length can have a second % component, which will indicate the reduction in function value to be % expected in the first line-search (defaults to 1.0). % P1, P2, ... parameters are passed to the function f. % % X the returned solution % fX vector of function values indicating progress made % i number of iterations (line searches or function evaluations, % depending on the sign of "length") used at termination. % % The function returns when either its length is up, or if no further progress % can be made (ie, we are at a (local) minimum, or so close that due to % numerical problems, we cannot get any closer). NOTE: If the function % terminates within a few iterations, it could be an indication that the % function values and derivatives are not consistent (ie, there may be a bug in % the implementation of your "f" function). % % The Polack-Ribiere flavour of conjugate gradients is used to compute search % directions, and a line search using quadratic and cubic polynomial % approximations and the Wolfe-Powell stopping criteria is used together with % the slope ratio method for guessing initial step sizes. Additionally a bunch % of checks are made to make sure that exploration is taking place and that % extrapolation will not be unboundedly large. % % See also: checkgrad % % Copyright (C) 2001 - 2010 by Carl Edward Rasmussen, 2010-01-03 INT = 0.1; % don't reevaluate within 0.1 of the limit of the current bracket EXT = 3.0; % extrapolate maximum 3 times the current step-size MAX = 20; % max 20 function evaluations per line search RATIO = 10; % maximum allowed slope ratio SIG = 0.1; RHO = SIG/2; % SIG and RHO are the constants controlling the Wolfe- % Powell conditions. SIG is the maximum allowed absolute ratio between % previous and new slopes (derivatives in the search direction), thus setting % SIG to low (positive) values forces higher precision in the line-searches. % RHO is the minimum allowed fraction of the expected (from the slope at the % initial point in the linesearch). Constants must satisfy 0 < RHO < SIG < 1. % Tuning of SIG (depending on the nature of the function to be optimized) may % speed up the minimization; it is probably not worth playing much with RHO. % The code falls naturally into 3 parts, after the initial line search is % started in the direction of steepest descent. 1) we first enter a while loop % which uses point 1 (p1) and (p2) to compute an extrapolation (p3), until we % have extrapolated far enough (Wolfe-Powell conditions). 2) if necessary, we % enter the second loop which takes p2, p3 and p4 chooses the subinterval % containing a (local) minimum, and interpolates it, unil an acceptable point % is found (Wolfe-Powell conditions). Note, that points are always maintained % in order p0 <= p1 <= p2 < p3 < p4. 3) compute a new search direction using % conjugate gradients (Polack-Ribiere flavour), or revert to steepest if there % was a problem in the previous line-search. Return the best value so far, if % two consecutive line-searches fail, or whenever we run out of function % evaluations or line-searches. During extrapolation, the "f" function may fail % either with an error or returning Nan or Inf, and minimize should handle this % gracefully. if max(size(length)) == 2, red=length(2); length=length(1); else red=1; end if length>0, S='Linesearch'; else S='Function evaluation'; end i = 0; % zero the run length counter ls_failed = 0; % no previous line search has failed [f0 df0] = feval(f, X, varargin{:}); % get function value and gradient Z = X; X = unwrap(X); df0 = unwrap(df0); fprintf('%s %6i; Value %4.6e\r', S, i, f0); if exist('fflush','builtin') fflush(stdout); end fX = f0; i = i + (length<0); % count epochs?! s = -df0; d0 = -s'*s; % initial search direction (steepest) and slope x3 = red/(1-d0); % initial step is red/(|s|+1) while i < abs(length) % while not finished i = i + (length>0); % count iterations?! X0 = X; F0 = f0; dF0 = df0; % make a copy of current values if length>0, M = MAX; else M = min(MAX, -length-i); end while 1 % keep extrapolating as long as necessary x2 = 0; f2 = f0; d2 = d0; f3 = f0; df3 = df0; success = 0; while ~success && M > 0 try M = M - 1; i = i + (length<0); % count epochs?! [f3 df3] = feval(f, rewrap(Z,X+x3*s), varargin{:}); df3 = unwrap(df3); if isnan(f3) || isinf(f3) || any(isnan(df3)+isinf(df3)), error(''), end success = 1; catch % catch any error which occured in f x3 = (x2+x3)/2; % bisect and try again end end if f3 < F0, X0 = X+x3*s; F0 = f3; dF0 = df3; end % keep best values d3 = df3'*s; % new slope if d3 > SIG*d0 || f3 > f0+x3*RHO*d0 || M == 0 % are we done extrapolating? break end x1 = x2; f1 = f2; d1 = d2; % move point 2 to point 1 x2 = x3; f2 = f3; d2 = d3; % move point 3 to point 2 A = 6*(f1-f2)+3*(d2+d1)*(x2-x1); % make cubic extrapolation B = 3*(f2-f1)-(2*d1+d2)*(x2-x1); x3 = x1-d1*(x2-x1)^2/(B+sqrt(B*B-A*d1*(x2-x1))); % num. error possible, ok! if ~isreal(x3) || isnan(x3) || isinf(x3) || x3 < 0 % num prob | wrong sign? x3 = x2*EXT; % extrapolate maximum amount elseif x3 > x2*EXT % new point beyond extrapolation limit? x3 = x2*EXT; % extrapolate maximum amount elseif x3 < x2+INT*(x2-x1) % new point too close to previous point? x3 = x2+INT*(x2-x1); end end % end extrapolation while (abs(d3) > -SIG*d0 || f3 > f0+x3*RHO*d0) && M > 0 % keep interpolating if d3 > 0 || f3 > f0+x3*RHO*d0 % choose subinterval x4 = x3; f4 = f3; d4 = d3; % move point 3 to point 4 else x2 = x3; f2 = f3; d2 = d3; % move point 3 to point 2 end if f4 > f0 x3 = x2-(0.5*d2*(x4-x2)^2)/(f4-f2-d2*(x4-x2)); % quadratic interpolation else A = 6*(f2-f4)/(x4-x2)+3*(d4+d2); % cubic interpolation B = 3*(f4-f2)-(2*d2+d4)*(x4-x2); x3 = x2+(sqrt(B*B-A*d2*(x4-x2)^2)-B)/A; % num. error possible, ok! end if isnan(x3) || isinf(x3) x3 = (x2+x4)/2; % if we had a numerical problem then bisect end x3 = max(min(x3, x4-INT*(x4-x2)),x2+INT*(x4-x2)); % don't accept too close [f3 df3] = feval(f, rewrap(Z,X+x3*s), varargin{:}); df3 = unwrap(df3); if f3 < F0, X0 = X+x3*s; F0 = f3; dF0 = df3; end % keep best values M = M - 1; i = i + (length<0); % count epochs?! d3 = df3'*s; % new slope end % end interpolation if abs(d3) < -SIG*d0 && f3 < f0+x3*RHO*d0 % if line search succeeded X = X+x3*s; f0 = f3; fX = [fX' f0]'; % update variables fprintf('%s %6i; Value %4.6e\r', S, i, f0); if exist('fflush','builtin') fflush(stdout); end s = (df3'*df3-df0'*df3)/(df0'*df0)*s - df3; % Polack-Ribiere CG direction df0 = df3; % swap derivatives d3 = d0; d0 = df0'*s; if d0 > 0 % new slope must be negative s = -df0; d0 = -s'*s; % otherwise use steepest direction end x3 = x3 * min(RATIO, d3/(d0-realmin)); % slope ratio but max RATIO ls_failed = 0; % this line search did not fail else X = X0; f0 = F0; df0 = dF0; % restore best point so far if ls_failed || i > abs(length) % line search failed twice in a row break; % or we ran out of time, so we give up end s = -df0; d0 = -s'*s; % try steepest x3 = 1/(1-d0); ls_failed = 1; % this line search failed end end X = rewrap(Z,X); fprintf('\n'); if exist('fflush','builtin') fflush(stdout); end % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse rewrap.m. function v = unwrap(s) v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s))); % alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; unwrap(s{i})]; end end % other types are ignored % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also unwrap.m. function [s v] = rewrap(s, v) if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed gpml-matlab-v3.5-2014-12-08/util/rewrap.m000664 000765 000024 00000001766 12425212207 017525 0ustar00carlstaff000000 000000 % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also unwrap.m. function [s v] = rewrap(s, v) if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed gpml-matlab-v3.5-2014-12-08/util/lik_epquad.m000664 000765 000024 00000003126 12425212207 020333 0ustar00carlstaff000000 000000 % Compute infEP part of a likelihood function based on the infLaplace part using % Gaussian-Hermite quadrature. % % The function is used in GLM likelihoods such as likPoisson, likGamma, likBeta % and likInvGauss. % % Copyright (c) by Hannes Nickisch, 2013-10-16. function varargout = lik_epquad(lik,hyp,y,mu,s2) n = max([length(y),length(mu),length(s2)]); on = ones(n,1); N = 20; [t,w] = gauher(N); oN = ones(1,N); lw = ones(n,1)*log(w'); y = y(:).*on; mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only [lpi,dlpi,d2lpi] = feval(lik{:},hyp,y*oN,sig*t'+mu*oN,[],'infLaplace'); lZ = s(lpi+lw); dlZ = {}; d2lZ = {}; if nargout>1 % 1st derivative wrt mean % Using p*dlp=dp, p=exp(lp), Z=sum_i wi*pi, dZ = sum_i wi*dpi we obtain % dlZ = sum_i exp(lpi-lZ+lwi)*dlpi = sum_i ai*dlpi. a = exp(lpi - lZ*oN + lw); dlZ = sum(a.*dlpi,2); if nargout>2 % 2nd derivative wrt mean % Using d2lZ=(d2Z*Z-dZ^2)/Z^2 <=> d2Z=Z*(d2lZ+dlZ^2) and % d2Z = sum_i wi*d2Zi, we get d2lZ = sum_i ai*(d2lpi+dlpi^2)-dlZ^2. d2lZ = sum(a.*(d2lpi+dlpi.*dlpi),2) - dlZ.*dlZ; end end varargout = {lZ,dlZ,d2lZ}; % computes y = log( sum(exp(x),2) ), the softmax in a numerically safe way by % subtracting the row maximum to avoid cancelation after taking the exp % the sum is done along the rows function [y,x] = s(logx) N = size(logx,2); max_logx = max(logx,[],2); % we have all values in the log domain, and want to calculate a sum x = exp(logx-max_logx*ones(1,N)); y = log(sum(x,2)) + max_logx;gpml-matlab-v3.5-2014-12-08/util/binaryGP.m000664 000765 000024 00000011547 12425212207 017736 0ustar00carlstaff000000 000000 function varargout = binaryGP(hyper, approx, covfunc, lik, x, y, xstar) % Approximate binary Gaussian Process classification. Two modes are possible: % training or testing: if no test cases are supplied, then the approximate % negative log marginal likelihood and its partial derivatives wrt the % hyperparameters is computed; this mode is used to fit the hyperparameters. If % test cases are given, then the test set predictive probabilities are % returned. Exact inference is intractible, the function uses a specified % approximation method (see approximations.m), flexible covariance functions % (see covFunctions.m) and likelihood functions (see likelihoods.m). % % usage: [nlZ, dnlZ ] = binaryGP(hyper, approx, covfunc, lik, x, y); % or: [p,mu,s2,nlZ] = binaryGP(hyper, approx, covfunc, lik, x, y, xstar); % % where: % % hyper is a column vector of hyperparameters % approx is a function specifying an approximation method for inference % covfunc is the name of the covariance function (see below) % lik is the name of the likelihood function % x is a n by D matrix of training inputs % y is a (column) vector (of size n) of binary +1/-1 targets % xstar is a nn by D matrix of test inputs % nlZ is the returned value of the negative log marginal likelihood % dnlZ is a (column) vector of partial derivatives of the negative % log marginal likelihood wrt each hyperparameter % p is a (column) vector (of length nn) of predictive probabilities % mu is a (column) vector (of length nn) of predictive latent means % s2 is a (column) vector (of length nn) of predictive latent variances % % The length of the vector of hyperparameters depends on the covariance % function, as specified by the "covfunc" input to the function, specifying the % name of a covariance function. A number of different covariance function are % implemented, and it is not difficult to add new ones. See covFunctions.m for % the details. % % The "lik" input argument specifies the name of the likelihood function (see % likelihoods.m). % % The "approx" input argument to the function specifies an approximation method % (see approximations.m). An approximation method returns a representation of % the approximate Gaussian posterior. Usually, the approximate posterior admits % the form N(m=K*alpha, V=inv(inv(K)+W)), where alpha is a vector and W is % diagonal. The approximation method returns: % % alpha is a (sparse or full column vector) containing inv(K)*m, where K % is the prior covariance matrix and m the approx posterior mean % sW is a (sparse or full column) vector containing diagonal of sqrt(W) % the approximate posterior covariance matrix is inv(inv(K)+W) % L is a (sparse or full) matrix, L = chol(sW*K*sW+eye(n)) % % In cases where the approximate posterior variance does not admit the form % V=inv(inv(K)+W) with diagonal W, L contains instead -inv(K+inv(W)), and sW % is unused. % % The alpha parameter may be sparse. In that case sW and L can either be sparse % or full (retaining only the non-zero rows and columns, as indicated by the % sparsity structure of alpha). The L paramter is allowed to be empty, in % which case it will be computed. % % The function can conveniently be used with the "minimize" function to train % a Gaussian Process, eg: % % [hyper, fX, i] = minimize(hyper, 'binaryGP', length, 'approxEP', 'covSEiso', 'logistic', x, y); % % where "length" gives the length of the run: if it is positive, it gives the % maximum number of line searches, if negative its absolute gives the maximum % allowed number of function evaluations. % % Copyright (c) 2007 Carl Edward Rasmussen and Hannes Nickisch, 2010-07-21. if nargin<6 || nargin>7 disp('Usage: [nlZ, dnlZ ] = binaryGP(hyper,approx,covfunc,lik,x,y);') disp(' or: [p,mu,s2,nlZ] = binaryGP(hyper,approx,covfunc,lik,x,y,xstar);') return end % Note, this function is just a wrapper provided for backward compatibility, % the functionality is now provided by the more general gp function. if strcmp(lik,'logistic') lik = @likLogistic; elseif strcmpi(lik,'cumgauss') lik = @likErf; else error('Allowable likelihoods: logistic and cumGauss.'); end if strcmp(approx,'approxLA') inf = @infLaplace; elseif strcmp(approx,'approxEP') inf = @infEP; else error('Allowable approximations: approxLA and approxEP.'); end mean = @meanZero; varargout = cell(nargout, 1); % allocate the right number of output arguments hyp.cov = hyper; if nargin==6 [varargout{:}] = gp(hyp,inf,mean,covfunc,lik,x,y); if nargout>1, varargout{2} = varargout{2}.cov; end else [ymu,ys2,fmu,fs2] = gp(hyp,inf,mean,covfunc,lik,x,y,xstar); if nargout>0, varargout{1} = (1+ymu)/2; end if nargout>1, varargout{2} = fmu; end if nargout>2, varargout{3} = fs2; end if nargout>3, varargout{4} = gp(hyp,inf,mean,covfunc,lik,x,y); end % nlZ endgpml-matlab-v3.5-2014-12-08/util/elsympol.m000664 000765 000024 00000001273 12425212207 020062 0ustar00carlstaff000000 000000 % Evaluate the order R elementary symmetric polynomial Newton's identity aka % the Newton–Girard formulae: http://en.wikipedia.org/wiki/Newton's_identities % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-01-10. function E = elsympol(Z,R) % evaluate 'power sums' of the individual terms in Z sz = size(Z); P = zeros([sz(1:2),R]); for r=1:R, P(:,:,r) = sum(Z.^r,3); end E = zeros([sz(1:2),R+1]); % E(:,:,r+1) yields polynomial r E(:,:,1) = ones(sz(1:2)); if R==0, return, end % init recursion E(:,:,2) = P(:,:,1); if R==1, return, end % init recursion for r=2:R for i=1:r E(:,:,r+1) = E(:,:,r+1) + P(:,:,i).*E(:,:,r+1-i)*(-1)^(i-1)/r; end end gpml-matlab-v3.5-2014-12-08/util/glm_invlink_logistic.m000664 000765 000024 00000001256 12425212207 022425 0ustar00carlstaff000000 000000 % Compute the log intensity for the inverse link function g(f) = log(1+exp(f))). % % The function is used in GLM likelihoods such as likPoisson, likGamma, likBeta % and likInvGauss. % % Copyright (c) by Hannes Nickisch, 2013-10-16. function [lg,dlg,d2lg,d3lg] = glm_invlink_logistic(f) l1pef = max(0,f) + log(1+exp(-abs(f))); % safely compute log(1+exp(f)) lg = log(l1pef); id = f<-15; lg(id) = f(id); % fix log(log(1+exp(f))) limits if nargout>1 sm = 1./(1+exp(-f)); dlg = sm./l1pef; dlg(f<-15) = 1; if nargout>2 sp = 1./(1+exp(f)); d2lg = dlg.*(sp-dlg); if nargout>2 d3lg = d2lg.*(sp-2*dlg) - dlg.*sp.*sm; end end endgpml-matlab-v3.5-2014-12-08/util/lbfgsb/matlabscalar.cpp000664 000765 000024 00000001730 12425212207 022427 0ustar00carlstaff000000 000000 #include "matlabscalar.h" #include "matlabexception.h" // Function definitions. // ----------------------------------------------------------------- double& getMatlabScalar (const mxArray* ptr) { if (!mxIsDouble(ptr)) throw MatlabException("Matlab array must be of type double"); if (mxGetNumberOfElements(ptr) != 1) throw MatlabException("The Matlab array must be a scalar"); return *mxGetPr(ptr); } double& createMatlabScalar (mxArray*& ptr) { ptr = mxCreateDoubleScalar(0); return *mxGetPr(ptr); } // Function definitions for class MatlabScalar. // --------------------------------------------------------------- MatlabScalar::MatlabScalar (const mxArray* ptr) : x(getMatlabScalar(ptr)) { } MatlabScalar::MatlabScalar (mxArray*& ptr, double value) : x(createMatlabScalar(ptr)) { x = value; } MatlabScalar::MatlabScalar (MatlabScalar& source) : x(source.x) { } MatlabScalar& MatlabScalar::operator= (double value) { x = value; return *this; } gpml-matlab-v3.5-2014-12-08/util/lbfgsb/program.cpp000664 000765 000024 00000007675 12425212207 021466 0ustar00carlstaff000000 000000 #include "program.h" #include // Function definitions. // ----------------------------------------------------------------- // Copy a C-style string (a null-terminated character array) to a // non-C-style string (a simple character array). The length of the // destination character array is given by "ndest". If the source is // shorter than the destination, the destination is padded with blank // characters. void copyCStrToCharArray (const char* source, char* dest, int ndest) { // Get the length of the source C string. int nsource = strlen(source); // Only perform the copy if the source can fit into the destination. if (nsource < ndest) { // Copy the string. strcpy(dest,source); // Fill in the rest of the string with blanks. for (int i = nsource; i < ndest; i++) dest[i] = ' '; } } // Return true if the two strings are the same. The second input // argument must be a C-style string (a null-terminated character // array), but the first input argument need not be one. We only // compare the two strings up to the length of "cstr". bool strIsEqualToCStr (const char* str, const char* cstr) { // Get the length of the C string. int n = strlen(cstr); return !strncmp(str,cstr,n); } // Function definitions for class Program. // ----------------------------------------------------------------- Program::Program (int n, double* x, double* lb, double* ub, int* btype, int m, int maxiter, double factr, double pgtol) { this->n = n; this->x = x; this->lb = lb; this->ub = ub; this->btype = btype; this->m = m; this->maxiter = maxiter; this->factr = factr; this->pgtol = pgtol; this->iprint = defaultprintlevel; this->owner = false; initStructures(); } Program::Program (int n, int m, int maxiter, double factr, double pgtol) { this->n = n; this->x = new double[n]; this->lb = new double[n]; this->ub = new double[n]; this->btype = new int[n]; this->m = m; this->maxiter = maxiter; this->factr = factr; this->pgtol = pgtol; this->iprint = defaultprintlevel; this->owner = true; initStructures(); } Program::~Program() { if (owner) { delete[] x; delete[] lb; delete[] ub; delete[] btype; } delete[] g; delete[] wa; delete[] iwa; } SolverExitStatus Program::runSolver() { SolverExitStatus status = success; // The return value. // Initialize the objective function and gradient to zero. f = 0; for (int i = 0; i < n; i++) g[i] = 0; // This initial call sets up the structures for L-BFGS. callLBFGS("START"); // Repeat until we've reached the maximum number of iterations. int t = 0; while (true) { // Do something according to the "task" from the previous call to // L-BFGS. if (strIsEqualToCStr(task,"FG")) { // Evaluate the objective function and the gradient of the // objective at the current point. f = computeObjective(n,x); computeGradient(n,x,g); } else if (strIsEqualToCStr(task,"NEW_X")) { // Go to the next iteration and call the iterative callback // routine. t++; iterCallback(t,x,f); // If we've reached the maximum number of iterations, terminate // the optimization. if (t == maxiter) { callLBFGS("STOP"); break; } } else if (strIsEqualToCStr(task,"CONV")) break; else if (strIsEqualToCStr(task,"ABNO")) { status = abnormalTermination; break; } else if (strIsEqualToCStr(task,"ERROR")) { status = errorOnInput; break; } // Call L-BFGS again. callLBFGS(); } return status; } void Program::callLBFGS (const char* cmd) { if (cmd) copyCStrToCharArray(cmd,task,60); setulb_(&n,&m,x,lb,ub,btype,&f,g,&factr,&pgtol,wa,iwa,task,&iprint, csave,lsave,isave,dsave); } void Program::initStructures() { f = 0; g = new double[n]; wa = new double[(2*m + 5)*n + 12*m*(m + 1)]; iwa = new int[3*n]; } gpml-matlab-v3.5-2014-12-08/util/lbfgsb/matlabstring.cpp000664 000765 000024 00000002032 12425212207 022464 0ustar00carlstaff000000 000000 #include #include "matlabstring.h" #include "matlabexception.h" // Function definitions. // ----------------------------------------------------------------- char* copystring (const char* source) { int n = strlen(source); // The length of the string. char* dest = new char[n+1]; // The return value. strcpy(dest,source); return dest; } // Function definitions for class MatlabString. // ----------------------------------------------------------------- MatlabString::MatlabString (const mxArray* ptr) { s = 0; // Check to make sure the Matlab array is a string. if (!mxIsChar(ptr)) throw MatlabException("Matlab array must be a string (of type CHAR)"); // Get the string passed as a Matlab array. s = mxArrayToString(ptr); if (s == 0) throw MatlabException("Unable to obtain string from Matlab array"); } MatlabString::MatlabString (const MatlabString& source) { s = 0; // Copy the source string. s = copystring(source.s); } MatlabString::~MatlabString() { if (s) mxFree(s); } gpml-matlab-v3.5-2014-12-08/util/lbfgsb/LICENSE000664 000765 000024 00000077330 12425212207 020313 0ustar00carlstaff000000 000000 GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007 Copyright (C) 2007 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The GNU General Public License is a free, copyleft license for software and other kinds of works. The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. We, the Free Software Foundation, use the GNU General Public License for most of our software; it applies also to any other work released this way by its authors. You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others. For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. Developers that use the GNU GPL protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License giving you legal permission to copy, distribute and/or modify it. For the developers' and authors' protection, the GPL clearly explains that there is no warranty for this free software. For both users' and authors' sake, the GPL requires that modified versions be marked as changed, so that their problems will not be attributed erroneously to authors of previous versions. Some devices are designed to deny users access to install or run modified versions of the software inside them, although the manufacturer can do so. This is fundamentally incompatible with the aim of protecting users' freedom to change the software. The systematic pattern of such abuse occurs in the area of products for individuals to use, which is precisely where it is most unacceptable. Therefore, we have designed this version of the GPL to prohibit the practice for those products. If such problems arise substantially in other domains, we stand ready to extend this provision to those domains in future versions of the GPL, as needed to protect the freedom of users. Finally, every program is threatened constantly by software patents. States should not allow patents to restrict development and use of software on general-purpose computers, but in those that do, we wish to avoid the special danger that patents applied to a free program could make it effectively proprietary. To prevent this, the GPL assures that patents cannot be used to render the program non-free. The precise terms and conditions for copying, distribution and modification follow. TERMS AND CONDITIONS 0. Definitions. "This License" refers to version 3 of the GNU General Public License. "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. A "covered work" means either the unmodified Program or a work based on the Program. To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 1. Source Code. The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. The Corresponding Source for a work in source code form is that same work. 2. Basic Permissions. All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 3. Protecting Users' Legal Rights From Anti-Circumvention Law. No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 4. Conveying Verbatim Copies. You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 5. Conveying Modified Source Versions. You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 6. Conveying Non-Source Forms. You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. "Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. 7. Additional Terms. "Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or d) Limiting the use for publicity purposes of names of licensors or authors of the material; or e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. 8. Termination. You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. 9. Acceptance Not Required for Having Copies. You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. 10. Automatic Licensing of Downstream Recipients. Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. 11. Patents. A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. 12. No Surrender of Others' Freedom. If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. 13. Use with the GNU Affero General Public License. Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU Affero General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such. 14. Revised Versions of this License. The Free Software Foundation may publish revised and/or new versions of the GNU General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU General Public License, you may choose any version ever published by the Free Software Foundation. If the Program specifies that a proxy can decide which future versions of the GNU General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. 15. Disclaimer of Warranty. THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. Limitation of Liability. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 17. Interpretation of Sections 15 and 16. If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. END OF TERMS AND CONDITIONS gpml-matlab-v3.5-2014-12-08/util/lbfgsb/matlabexception.h000664 000765 000024 00000001545 12425212207 022631 0ustar00carlstaff000000 000000 #ifndef INCLUDE_MATLABEXCEPTION #define INCLUDE_MATLABEXCEPTION #include // Class MatlabException // ----------------------------------------------------------------- // This class just makes it easier for me to throw exceptions. Its // functionality really has nothing to do with MATLAB. class MatlabException : public std::exception { public: MatlabException (const char* message) throw(); ~MatlabException() throw() { }; // The copy constructor makes a shallow copy. MatlabException (const MatlabException& source) throw(); // The copy assignment operator makes a shallow copy as well. MatlabException& operator= (const MatlabException& source); // Return the message string. virtual const char* what () const throw() { return message; }; private: const char* message; // The error message. }; #endif gpml-matlab-v3.5-2014-12-08/util/lbfgsb/Makefile000664 000765 000024 00000006734 12440131251 020742 0ustar00carlstaff000000 000000 # for 'make nomex' you need to provide the variables MEX_SUFFIX and MATLAB_LIB # for 'make mex' you need to provide the variable MEX and have mex set up # for 'make oct' you need to provide the variable OCT and OCTAVE_INCLUDE ### 1) Choose Matlab configuration ############################################ ### path to your Matlab installation ### you can find it by the commands 'locate matlab' or 'find / -name "matlab"' MATLAB_HOME=/usr/local/bin/matlab ### For 32 bit Linux, uncomment the following three lines. -------------------- MEX = $(MATLAB_HOME)/bin/mex MEX_SUFFIX = mexglx MATLAB_LIB = -L$(MATLAB_HOME)/bin/glnx86 -lmex ### For 64 bit Linux, uncomment the following three lines. -------------------- #MEX = $(MATLAB_HOME)/bin/mex #MEX_SUFFIX = mexa64 #MATLAB_LIB = -L$(MATLAB_HOME)/bin/glnxa64 -lmex ### For Mac OS X, uncomment the following three lines. ------------------------ #MEX = /Applications/MATLAB7/bin/mex #MEX_SUFFIX = mexmac #MATLAB_LIB = -L$(MATLAB_HOME)/bin/maci -lmex ### 2) Choose Octave configuration ############################################ OCTAVE_INCLUDE = /usr/local/include/octave-3.2.4/octave OCT = mkoctfile ### 3) Choose L-BFGS-B configuration ########################################## ### Uncomment the following 2 lines for LBFGSB 2.4. --------------------------- LBFGSB_OBJS = solver_2_4.o LIBS = ### Uncomment the following 2 lines for LFBGSB 3.0 with blas libs provided. --- #LBFGSB_OBJS = solver_3_0.o linpack.o timer.o blas.o #LIBS = ### Uncomment the following 2 lines for LFBGSB 3.0 own blas libs. ------------- #LBFGSB_OBJS = solver_3_0.o linpack.o timer.o #LIBS = -lblas ### 4) Choose FORTRAN compiler ################################################ FFLAGS = -O3 -fPIC -fexceptions -Wall -g -Wno-uninitialized ### To work with f77, uncomment the following two lines. ---------------------- #F77 = f77 #FORTRAN_LIB = g2c ### To use gfortran, uncomment the following two lines. ----------------------- F77 = gfortran FORTRAN_LIB = gfortran ### 5) Choose C++ compiler #################################################### CXX = g++ CFLAGS = -O3 -fPIC -pthread -Wall -Werror -ansi -ffast-math -fomit-frame-pointer ### 6) If you need to specify extra directories where to look for libraries, ### write the paths of the directories, separated by a white space LIBDIRS = #LIBDIRS = ./ /usr/lib/gcc/x86_64-linux-gnu/4.8/ ############################################################################### ### Do not edit below ######################################################### ############################################################################### TARGET = lbfgsb OBJS = $(LBFGSB_OBJS) matlabexception.o matlabscalar.o matlabstring.o \ matlabmatrix.o arrayofmatrices.o program.o matlabprogram.o \ lbfgsb.o %.o: %.cpp $(CXX) $(CFLAGS) -I$(OCTAVE_INCLUDE) -I$(MATLAB_HOME)/extern/include -o $@ -c $^ %.o: %.f $(F77) $(FFLAGS) -o $@ -c $^ all: nomex mex: $(TARGET)_mex copy tidy nomex: $(TARGET)_nomex copy tidy octave: $(TARGET)_octave copy tidy oct: octave $(TARGET)_mex: $(OBJS) $(MEX) -cxx CXX=$(CXX) CC=$(CXX) FC=$(FCC) LD=$(CXX) -l$(FORTRAN_LIB) -lm \ -O -output $(TARGET) $^ $(TARGET)_nomex: $(OBJS) $(CXX) $^ -shared -o $(TARGET).$(MEX_SUFFIX) $(MATLAB_LIB) $(addprefix -L,$(LIBDIRS)) -l$(FORTRAN_LIB) -lm $(TARGET)_octave: $(OBJS) $(OCT) --mex -o $(TARGET).mex $^ $(LIBS)$ copy: mv $(TARGET).mex* .. tidy: rm -f *.o clean: tidy rm -f ../$(TARGET).mex* gpml-matlab-v3.5-2014-12-08/util/lbfgsb/lbfgsb.cpp000664 000765 000024 00000010024 12425212207 021234 0ustar00carlstaff000000 000000 #include "mex.h" #ifndef __linux__ #include "matrix.h" #endif #include "arrayofmatrices.h" #include "matlabexception.h" #include "matlabscalar.h" #include "matlabstring.h" #include "matlabmatrix.h" #include "arrayofmatrices.h" #include "program.h" #include "matlabprogram.h" #include #include extern void _main(); // Constants. // ----------------------------------------------------------------- const int minNumInputArgs = 5; // Function definitions. // ----------------------------------------------------------------- void mexFunction (int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) try { // Check to see if we have the correct number of input and output // arguments. if (nrhs < minNumInputArgs) throw MatlabException("Incorrect number of input arguments"); // Get the starting point for the variables. This is specified in // the first input argument. The variables must be either a single // matrix or a cell array of matrices. int k = 0; // The index of the current input argument. ArrayOfMatrices x0(prhs[k++]); // Create the output, which stores the solution obtained from // running IPOPT. There should be as many output arguments as cell // entries in X. if (nlhs != x0.length()) throw MatlabException("Incorrect number of output arguments"); ArrayOfMatrices x(plhs,x0); // Load the lower and upper bounds on the variables as // ArrayOfMatrices objects. They should have the same structure as // the ArrayOfMatrices object "x". ArrayOfMatrices lb(prhs[k++]); ArrayOfMatrices ub(prhs[k++]); // Check to make sure the bounds make sense. if (lb != x || ub != x) throw MatlabException("Input arguments LB and UB must have the same \ structure as X"); // Get the Matlab callback functions. MatlabString objFunc(prhs[k++]); MatlabString gradFunc(prhs[k++]); // Get the auxiliary data. const mxArray* auxData; const mxArray* ptr = prhs[k++]; if (nrhs > 5) { if (mxIsEmpty(ptr)) auxData = 0; else auxData = ptr; } else auxData = 0; // Get the intermediate callback function. MatlabString* iterFunc; ptr = prhs[k++]; if (nrhs > 6) { if (mxIsEmpty(ptr)) iterFunc = 0; else iterFunc = new MatlabString(ptr); } else iterFunc = 0; // Set the options for the L-BFGS algorithm to their defaults. int maxiter = defaultmaxiter; int m = defaultm; double factr = defaultfactr; double pgtol = defaultpgtol; // Process the remaining input arguments, which set options for // the IPOPT algorithm. while (k < nrhs) { // Get the option label from the Matlab input argument. MatlabString optionLabel(prhs[k++]); if (k < nrhs) { // Get the option value from the Matlab input argument. MatlabScalar optionValue(prhs[k++]); double value = optionValue; if (!strcmp(optionLabel,"maxiter")) maxiter = (int) value; else if (!strcmp(optionLabel,"m")) m = (int) value; else if (!strcmp(optionLabel,"factr")) factr = value / mxGetEps(); else if (!strcmp(optionLabel,"pgtol")) pgtol = value; else { if (iterFunc) delete iterFunc; throw MatlabException("Nonexistent option"); } } } // Create a new instance of the optimization problem. x = x0; MatlabProgram program(x,lb,ub,&objFunc,&gradFunc,iterFunc, (mxArray*) auxData,m,maxiter,factr,pgtol); // Run the L-BFGS-B solver. SolverExitStatus exitStatus = program.runSolver(); if (exitStatus == abnormalTermination) { if (iterFunc) delete iterFunc; throw MatlabException("Solver unable to satisfy convergence \ criteria due to abnormal termination"); } else if (exitStatus == errorOnInput) { if (iterFunc) delete iterFunc; throw MatlabException("Invalid inputs to L-BFGS routine"); } // Free the dynamically allocated memory. if (iterFunc) delete iterFunc; } catch (std::exception& error) { mexErrMsgTxt(error.what()); } gpml-matlab-v3.5-2014-12-08/util/lbfgsb/matlabstring.h000664 000765 000024 00000002200 12425212207 022126 0ustar00carlstaff000000 000000 #ifndef INCLUDE_MATLABSTRING #define INCLUDE_MATLABSTRING #include "mex.h" #include // Function declarations. // ----------------------------------------------------------------- // Copy a C-style string (i.e. a null-terminated character array). char* copystring (const char* source); // Class MatlabString. // ----------------------------------------------------------------- // This class encapsulates read-only access to a MATLAB character // array. class MatlabString { public: // The constructor accepts as input a pointer to a Matlab array, // which must be a valid string (array of type CHAR). explicit MatlabString (const mxArray* ptr); // The copy constructor makes a full copy of the source string. MatlabString (const MatlabString& source); // The destructor. ~MatlabString(); // Conversion operator for null-terminated string. operator const char* () const { return s; }; protected: char* s; // The null-terminated string. // The copy assignment operator is not proper, thus remains // protected. MatlabString& operator= (const MatlabString& source) { return *this; }; }; #endif gpml-matlab-v3.5-2014-12-08/util/lbfgsb/matlabprogram.h000664 000765 000024 00000005501 12425212207 022276 0ustar00carlstaff000000 000000 #ifndef INCLUDE_MATLABPROGRAM #define INCLUDE_MATLABPROGRAM #include "mex.h" #include "program.h" #include "matlabscalar.h" #include "matlabstring.h" #include "arrayofmatrices.h" // Class MatlabProgram. // ----------------------------------------------------------------- // This is an implementation of the abstract class Program. class MatlabProgram: public Program { public: // On input, "variables" should contain the initial point for the // optimization routine. If no iterative callback function is // specified, "iterFunc" may be set to 0. Also, if no auxiliary data // is needed, it may also be set to 0. MatlabProgram (ArrayOfMatrices& variables, const ArrayOfMatrices& lowerbounds, const ArrayOfMatrices& upperbounds, const MatlabString* objFunc, const MatlabString* gradFunc, const MatlabString* iterFunc, mxArray* auxData, int m = defaultm, int maxiter = defaultmaxiter, double factr = defaultfactr, double pgtol = defaultpgtol); // The destructor. virtual ~MatlabProgram(); // These provide definitions for the pure virtual functions of the // abstract parent class. virtual double computeObjective (int n, double* x); virtual void computeGradient (int n, double* x, double* g); virtual void iterCallback (int t, double* x, double f); // Run the solver. Upon completion, the solution is stored in // "variables". SolverExitStatus runSolver(); protected: ArrayOfMatrices& variables; // Storage for the initial value and // solution. const MatlabString* objFunc; // The objective callback function. const MatlabString* gradFunc; // The gradient callback function. const MatlabString* iterFunc; // The iterative callback function. ArrayOfMatrices* varMatlab; // Inputs to the Matlab callback mxArray** varInputs; // functions representing the // current values of the variables. MatlabScalar* fMatlab; // Input to the Matlab callback // functions representing the // current value of the objective. MatlabScalar* tMatlab; // Input to the Matlab callback // functions representing the // current iteration. int numInputsObjFunc; // The number of inputs passed to the // objective callback function. int numInputsGradFunc; // The number of inputs passed to the // gradient callback function. int numInputsIterFunc; // The number of inputs passed to the // iterative callback function. mxArray** inputsObjFunc; // The inputs passed to the objective // callback function. mxArray** inputsGradFunc; // The inputs passed to the gradient // callback function. mxArray** inputsIterFunc; // The inputs passed to the iterative // callback function. }; #endif gpml-matlab-v3.5-2014-12-08/util/lbfgsb/blas.f000664 000765 000024 00000014162 12425212207 020370 0ustar00carlstaff000000 000000 c c L-BFGS-B is released under the “New BSD License” (aka “Modified BSD c License” or “3-clause license”) c Please read attached file License.txt c double precision function dnrm2(n,x,incx) integer n,incx double precision x(n) c ********** c c Function dnrm2 c c Given a vector x of length n, this function calculates the c Euclidean norm of x with stride incx. c c The function statement is c c double precision function dnrm2(n,x,incx) c c where c c n is a positive integer input variable. c c x is an input array of length n. c c incx is a positive integer variable that specifies the c stride of the vector. c c Subprograms called c c FORTRAN-supplied ... abs, max, sqrt c c MINPACK-2 Project. February 1991. c Argonne National Laboratory. c Brett M. Averick. c c ********** integer i double precision scale dnrm2 = 0.0d0 scale = 0.0d0 do 10 i = 1, n, incx scale = max(scale, abs(x(i))) 10 continue if (scale .eq. 0.0d0) return do 20 i = 1, n, incx dnrm2 = dnrm2 + (x(i)/scale)**2 20 continue dnrm2 = scale*sqrt(dnrm2) return end c====================== The end of dnrm2 =============================== subroutine daxpy(n,da,dx,incx,dy,incy) c c constant times a vector plus a vector. c uses unrolled loops for increments equal to one. c jack dongarra, linpack, 3/11/78. c double precision dx(*),dy(*),da integer i,incx,incy,ix,iy,m,mp1,n c if(n.le.0)return if (da .eq. 0.0d0) return if(incx.eq.1.and.incy.eq.1)go to 20 c c code for unequal increments or equal increments c not equal to 1 c ix = 1 iy = 1 if(incx.lt.0)ix = (-n+1)*incx + 1 if(incy.lt.0)iy = (-n+1)*incy + 1 do 10 i = 1,n dy(iy) = dy(iy) + da*dx(ix) ix = ix + incx iy = iy + incy 10 continue return c c code for both increments equal to 1 c c c clean-up loop c 20 m = mod(n,4) if( m .eq. 0 ) go to 40 do 30 i = 1,m dy(i) = dy(i) + da*dx(i) 30 continue if( n .lt. 4 ) return 40 mp1 = m + 1 do 50 i = mp1,n,4 dy(i) = dy(i) + da*dx(i) dy(i + 1) = dy(i + 1) + da*dx(i + 1) dy(i + 2) = dy(i + 2) + da*dx(i + 2) dy(i + 3) = dy(i + 3) + da*dx(i + 3) 50 continue return end c====================== The end of daxpy =============================== subroutine dcopy(n,dx,incx,dy,incy) c c copies a vector, x, to a vector, y. c uses unrolled loops for increments equal to one. c jack dongarra, linpack, 3/11/78. c double precision dx(*),dy(*) integer i,incx,incy,ix,iy,m,mp1,n c if(n.le.0)return if(incx.eq.1.and.incy.eq.1)go to 20 c c code for unequal increments or equal increments c not equal to 1 c ix = 1 iy = 1 if(incx.lt.0)ix = (-n+1)*incx + 1 if(incy.lt.0)iy = (-n+1)*incy + 1 do 10 i = 1,n dy(iy) = dx(ix) ix = ix + incx iy = iy + incy 10 continue return c c code for both increments equal to 1 c c c clean-up loop c 20 m = mod(n,7) if( m .eq. 0 ) go to 40 do 30 i = 1,m dy(i) = dx(i) 30 continue if( n .lt. 7 ) return 40 mp1 = m + 1 do 50 i = mp1,n,7 dy(i) = dx(i) dy(i + 1) = dx(i + 1) dy(i + 2) = dx(i + 2) dy(i + 3) = dx(i + 3) dy(i + 4) = dx(i + 4) dy(i + 5) = dx(i + 5) dy(i + 6) = dx(i + 6) 50 continue return end c====================== The end of dcopy =============================== double precision function ddot(n,dx,incx,dy,incy) c c forms the dot product of two vectors. c uses unrolled loops for increments equal to one. c jack dongarra, linpack, 3/11/78. c double precision dx(*),dy(*),dtemp integer i,incx,incy,ix,iy,m,mp1,n c ddot = 0.0d0 dtemp = 0.0d0 if(n.le.0)return if(incx.eq.1.and.incy.eq.1)go to 20 c c code for unequal increments or equal increments c not equal to 1 c ix = 1 iy = 1 if(incx.lt.0)ix = (-n+1)*incx + 1 if(incy.lt.0)iy = (-n+1)*incy + 1 do 10 i = 1,n dtemp = dtemp + dx(ix)*dy(iy) ix = ix + incx iy = iy + incy 10 continue ddot = dtemp return c c code for both increments equal to 1 c c c clean-up loop c 20 m = mod(n,5) if( m .eq. 0 ) go to 40 do 30 i = 1,m dtemp = dtemp + dx(i)*dy(i) 30 continue if( n .lt. 5 ) go to 60 40 mp1 = m + 1 do 50 i = mp1,n,5 dtemp = dtemp + dx(i)*dy(i) + dx(i + 1)*dy(i + 1) + * dx(i + 2)*dy(i + 2) + dx(i + 3)*dy(i + 3) + dx(i + 4)*dy(i + 4) 50 continue 60 ddot = dtemp return end c====================== The end of ddot ================================ subroutine dscal(n,da,dx,incx) c c scales a vector by a constant. c uses unrolled loops for increment equal to one. c jack dongarra, linpack, 3/11/78. c modified 3/93 to return if incx .le. 0. c double precision da,dx(*) integer i,incx,m,mp1,n,nincx c if( n.le.0 .or. incx.le.0 )return if(incx.eq.1)go to 20 c c code for increment not equal to 1 c nincx = n*incx do 10 i = 1,nincx,incx dx(i) = da*dx(i) 10 continue return c c code for increment equal to 1 c c c clean-up loop c 20 m = mod(n,5) if( m .eq. 0 ) go to 40 do 30 i = 1,m dx(i) = da*dx(i) 30 continue if( n .lt. 5 ) return 40 mp1 = m + 1 do 50 i = mp1,n,5 dx(i) = da*dx(i) dx(i + 1) = da*dx(i + 1) dx(i + 2) = da*dx(i + 2) dx(i + 3) = da*dx(i + 3) dx(i + 4) = da*dx(i + 4) 50 continue return end c====================== The end of dscal =============================== gpml-matlab-v3.5-2014-12-08/util/lbfgsb/README000664 000765 000024 00000051032 12425212207 020155 0ustar00carlstaff000000 000000 # A MATLAB interface for L-BFGS-B ###Update As of March 24, 2014, the MATLAB code supports the latest version of the L-BFGS-B solver (version 3.0), and is compatible with [GNU Octave](https://www.gnu.org/software/octave). Thank you to [Jose Vallet](http://github.com/josombio) for providing these updates. ###Overview L-BFGS-B is a collection of Fortran 77 routines for solving large-scale nonlinear optimization problems with bound constraints on the variables. One of the key features of the nonlinear solver is that knowledge of the Hessian is not required; the solver computes search directions by keeping track of a quadratic model of the objective function with a limited-memory BFGS (Broyden-Fletcher-Goldfarb-Shanno) approximation to the Hessian *(see Note #1)*. The algorithm was developed by Ciyou Zhu, Richard Byrd and Jorge Nocedal. For more information, go to the [original distribution site](http://www.ece.northwestern.edu/~nocedal/lbfgsb.html) for the L-BFGS-B software package. I've designed an interface to the L-BFGS-B solver so that it can be called like any other function in MATLAB *(see Note #2)*. See the text below for more information on installing and calling this function in MATLAB. Along the way, I've also developed a C++ class that encapsulates all the "messy" details in executing the L-BFGS-B code. See below for instructions on how to incorporate this C++ class info your own code. This code has been tested in MATLAB and Octave on Linux, and is integrated into [GPML](http://www.gaussianprocess.org/gpml/code). It has also been executed successfully in Windows; see below. ###License Copyright (c) 2013, 2014 Peter Carbonetto The lbfgsb-matlab project repository is free software: you can redistribute it and/or modify it under the terms of the [GNU General Public License](http://www.gnu.org/licenses/gpl.html) as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but **without any warranty**; without even the implied warranty of **merchantability** or **fitness for a particular purpose**. See [LICENSE](LICENSE) for more details. ###Installation Follow these steps to compile and install **lbfgsb-matlab** for MATLAB and Octave. Some of these steps are applicable to MATLAB or Octave only, and you can skip them if you want to install the interface for only one of them. These installation instructions assume you have Linux, although similar steps should also work for other UNIX-based operating systems, such as Mac OS X. These instructions also assume you have [GNU Make](http://www.gnu.org/software/make) installed. (We have successfully compiled this source code with gcc versions 4.6.3 and 4.6.4, under Ubuntu Linux 12.10 and 13.10, and have successfully run the MATLAB scripts in Octave 3.6.2, 3.6.4, 3.8.0 and 3.8.1.) **For Windows users:** If you have a Windows operating system, Guillaume Jacquenot has generously provided [instructions](INSTALL.WINDOWS.txt) for compiling the software on Windows with [gnumex](http://gnumex.sourceforge.net) and Mingw. These instructions are slightly out of date, and suggestions for improving these installation instructions are welcome. We will create a MEX file, which is basically a file that contains a routine that can be called from MATLAB as if it were a built-it function. To learn about MEX files, I refer you to [this document](http://www.mathworks.com/support/tech-notes/1600/1605.html) on the MathWorks website. **Download the source code.** Clone or fork the this repository, or download this repository as a [ZIP file](http://github.com/pcarbo/lbfgsb-matlab/archive/master.zip) and unpack the ZIP file. The [src](src) subdirectory contains some MATLAB files (ending in .m), some C++ source and header files (ending in .h and .cpp), a single Fortran 77 source file, [solver.f](src/solver.f), containing the L-BFGS-B routines, and a [Makefile](src/Makefile). I've included a version of the Fortran routines that is more recent than what is available for download at the [distribution site](http://www.ece.northwestern.edu/~nocedal/lbfgsb.html) at Northwestern University. In the following, we will refer to the directory containing your local copy of the repository as LBFGSB_HOME. Wherever you find LBFGSB_HOME, you should substitute it with your own directory. **Install the C++ and Fortran 77 compilers.** In order to build the MEX file, you will need both a C++ compiler and a Fortran 77 compiler. Unfortunately, you can't just use any compiler. You have to use the precise one supported by MATLAB. For instance, if you are running Mac OS X 7.2 on a Linux operating system, you will need to install [GNU compiler collection](http://www.gnu.org/software/gcc) (GCC) version 3.4.4. Even if you already have a compiler installed on Linux, it may be the wrong version, and if you use the wrong version things could go horribly wrong. It is important that you use the correct version of the compiler, otherwise you will encounter linking errors. Refer to [this document](http://www.mathworks.com/support/tech-notes/1600/1601.html) to find out which compiler is supported by your version of MATLAB. You may also be able to use the **apt-get** program to install gfortran, as in: sudo apt-get install build-essential gfortran **Install Octave.** You will of course need a working installation of Matlab or Octave. In the case of Octave, if you don't have it the easiest way to install it is from the repositories using **apt-get**: sudo apt-get install octave If you want to use a more recent version of Octave, you will have to [install Octave from the source code](http://www.gnu.org/software/octave/doc/interpreter/Installation.html). After that, find the path to the mkoctfile compiler (in the bin directory) and the include directory. You will need those paths for steps below. Additionally you will need the Octave development tools, libs and header files to compile MEX files: sudo apt-get install liboctave-dev **Configure MATLAB.** Next, you need to set up and configure MATLAB to build MEX Files. This is explained quite nicely in [this MathWorks support document](http://www.mathworks.com/support/tech-notes/1600/1605.html). **Simple compilation procedure for MATLAB.** (Thanks to Stefan Harmeling.) You might be able to build and compile the MEX File from source code simply by calling MATLAB's mex program with the C++ and Fortran source files as inputs, like so: mex -output lbfgsb *.cpp solver.f If that doesn't work (e.g. you get linking errors, or it makes MATLAB crash when you try to call it), then you may have to follow the more complicated installation instructions below. **Modify the Makefile.** You are almost ready to build the MEX file. But before you do so, you need to edit the Makefile to coincide with your system setup. Edit the Makefile located inside the LBFGSB_HOME/src directory, following the instructions provided in this file. With this Makefile you may create: - A mex file for Matlab using the mex compiler (mex taget). - A mex file for Matlab using your C compiler (nomex target). - A mex file for Octave using mkoctfile (oct target). Regarding the compilation for Octave, the variable **OCTAVE_INCLUDE** is the directory where the Octave header files required for development are installed. Make sure that it points to the correct directory corresponding to your Octave installation. **OCT** is the command to call the mkoctfile compiler. If mkoctfile is in the path, simply leave it as it is. If not, include in the variable the full path to it. If you have several versions of Octave installed, make sure that OCT points to the mkoctfile program that corresponds to the Octave version you would like to use. For MATLAB, the variable **MEX** is the executable called to build the MEX file. The variables **CXX** and **F77** must be the names of your C++ and Fortran 77 compilers. **MEXSUFFIX** is the MEX file extension specific to your platform. (For instance, the extension for Linux is **mexglx**.) The variable **MATLAB_HOME** must be the base directory of your MATLAB installation. Finally, **CFLAGS** and **FFLAGS** are options passed to the C++ and Fortran compilers, respectively. Often these flags coincide with your MEX options file (see [here](http://www.mathworks.com/support/tech-notes/1600/1605.html) for more information on the options file). But often the settings in the MEX option file are incorrect, and so the options must be set manually. MATLAB requires some special compilation flags for various reasons, one being that it requires position-independent code. These instructions are vague (I apologize), and this step may require a bit of trial and error until before you get it right. **INSTALLDIR** is the installation directory relative to the LBFGSB_HOME/src directory (see Installation below). It may be helpful to look at the GCC documentation in order to understand what these various compiler flags mean. **Build the MEX file.** If you are in the directory containing all the source files, calling **make** at the command prompt will first compile the Fortran and C++ source files into object code (.o files). After that, the make program calls the MEX script, which links all the object files together into a single MEX file. Call **make** from the LBFGSB_HOME/src directory using the appropriate target, one of: make oct make nomex make mex If you didn't get any errors, then you are ready to try out the bound-constrained solver in MATLAB or Octave. Note that even if you didn't get any errors, there's still a possibility that you didn't link the MEX file properly, in which case executing the MEX file will cause MATLAB to crash. But there's only one way to find out: the hard way. **Final installation steps.** Installing the mex file is as simple as placing it wherever MATLAB or Octave can find it; that is, in its "path". There are many ways to do this. Whatever approach you take, note that the mex file does not contain the usage documentation, which is provided in lbfgsb.m. So wherever you place lbfgsb.mex, make sure that you include lbfgsb.m in the same directory. Perhaps the simplest approach to copy lbfgsb.mex and lbfgsb.m to a a specific folder. Alternatively, make install will create the .mex and .m files in the INSTALLDIR directory, and this directory will be automatically created if it does not exist. Once you are in MATLAB or Octave, add the installation directory to the MATLAB or Octave path can be done using the command **addpath**. Note that once you exit MATLAB or Octave the updated path is not preserved, and you will need to use addpath in every new session. You can add the addpath command to ~/.octaverc or ~/matlab/startup. In this way, the path will automatically be updated every time you start a new session. ###A brief tutorial I've written a short script [examplehs038.m](src/examplehs038.m) which demonstrates how to call **lbfgsb** for solving a very small optimization problem, the Hock & Schittkowski test problem #38 **(Note #3)**. The optimization problem has 4 variables. They are bounded from below by -10 and from above by +10. We've set the starting point to (-3, -1, -3, -1). We've written two MATLAB functions for computing the objective function and the gradient of the objective function at a given point. These functions can be found in the files [computeObjectiveHS038.m](src/computeObjectiveHS038.m) and [computeGradientHS038.m](src/computeGradientHS038.m). If you run the script, the solver should very quickly progress toward the point (1, 1, 1, 1). The objective is 0 at this point. The basic MATLAB function call is lbfgsb(x0,lb,ub,objfunc,gradfunc) The first input argument **x0** declares the starting point for the solver. The second and third input arguments define the lower and upper bounds on the variables, respectively. If an entry of **lb** is set to **-Inf**, then this is equivalent to no lower bound for that entry (the same applies to upper bounds, except that it must be **+Inf** instead). The next two input arguments must be the names of MATLAB routines (M-files). The first routine must calculate the objective function at the current point. The output must be a scalar representing the objective evaluated at the current point. The second function must compute the gradient of the objective at the current point. The input is the same as **objfunc**, but it must return as many outputs as there are inputs. For a complete guide on using the MATLAB interface, type **help lbfgsb** in MATLAB. The script [exampleldaimages.m](src/exampleldaimages.m) is a more complicated example. It uses the L-BFGS-B solver to compute some statistics of a posterior distribution for a text analysis model called latent Dirichlet allication (LDA). The model is used to estimate the topics for a collection of documents. The optimization problem is to compute an approximation to the posterior, since it is not possible to compute the posterior exactly. In this case, the optimization problem corresponds to minimizing the distance between the true posterior distribution and an approximate distribution. For more information, see the paper of Blei, Ng and Jordan, 2003 **(Note #4)**. In order to run this example, you will need to install the [lightspeed](http://research.microsoft.com/~minka/software/lightspeed) for MATLAB developed by Tom Minka at Microsoft Research. The script starts by generating a bunch of canonical images that represent topics; if a pixel is white then a word at that position is more likely to appear in that topic. Note that the order of the pixels in the image is unimportant, since LDA is a "bag of words" model. In another figure we show a small sample of the data generated from the topics. Each image represents a document, and the pixels in the image depict the word proportions in that document **(Note #5)**. Next, the script runs the L-BFGS-B solver to find a local minimum to the variational objective. Buried in the M-file [mflda.m](src/mflda.m) is the call to the solver: [xi gamma Phi] = lbfgsb({xi gamma Phi},... {repmat(lb,W,K) repmat(lb,K,D) repmat(lb,K,M)},... {repmat(inf,W,K) repmat(inf,K,D) repmat(inf,K,M)},... 'computeObjectiveMFLDA','computeGradientMFLDA',... {nu eta L w lnZconst},'callbackMFLDA',... 'm',m_lbfgs,'factr',1e-12,'pgtol',... tolerance,'maxiter',maxiter); This example will give us the opportunity to demonstrate some of the more complicated aspects of the our MATLAB interface. The first input to **lbfgsb** sets the initial iterate of the optimization algorithm. Notice that we've passed a cell array with entries that are matrices. Indeed, it is possible to pass either a matrix or a cell array. Whatever structure is passed, the other input arguments (and outputs) must also abide by this structure. The lower and upper bounds on variables are also cell arrays. All the upper bounds are set to infinity, which means that the variables are only bounded from below. It is instructive to examine the callback routines. They look like: f = computeObjectiveMFLDA(xi,gamma,Phi,auxdata) [dxi, dgamma, dPhi] = ... computeGradientMFLDA(xi,gamma,Phi,auxdata) Notice that each entry to the cell array is its own input argument to the callback routines. The same applies for the output arguments of the gradient callback function. The sixth input argument is a cell array that contains auxiliary data. This data will be passed to the MATLAB functions that evaluate the objective and gradient. The seventh input argument specifies a callback routine that is called exactly once for every iteration. This callback routine is most useful for examining the progress of the solver. The remaining input arguments are label/value pairs that override some of the default settings of L-BFGS-B. After converging to a solution, we display topic samples drawn from the variational approximation. A good solution should come close to recovering the true topics from which the data was generated (although they might not be in the same order). Don't be surprised if it takes many iterations before the optimization algorithm converges on a local minimum. (In repeated trials, I found it as many as 2000 iterations to converge to a stationary point of the objective.) This particular problem demonstrates the limitations of limited-memory quasi-Newton approximations to the Hessian; the low storage requirements can come at the cost of slow convergence to the solution. ###The C++ interface One of the nice byproducts of writing a MATLAB interface for L-BFGS-B is that I've ended up with a neat little C++ class that encapsulates the nuts and bolts of executing the solver. A brief description of the Program class can be found in the header file [program.h](src/program.h). "Program" is an abstract class, which means that its impossible to instantiate a Program object. This is because the class has member functions that aren't yet implemented. These are called pure virtual functions. In order to use the Program class, one needs to define another class that inherits the Program class and that implements the pure virtual functions. The class MatlabProgram (in [matlabprogram.h](src/matlabprogram.h)) is an example of such a class. The new child class must declare and implement these two functions: virtual double computeObjective (int n, double* x); virtual void computeGradient (int n, double* x, double* g); See the header file for a detailed description of these functions. The only remaining detail is calling the Program constructor. After that, it is just a matter of declaring a new object of type MyProgram then calling the function runSolver. ###Overview of source files solver.f
Fortran 77 source for the L-BFGS-B solver routines. program.h
program.cpp
 Header and source file for the C++ interface. array.h
arrayofmatrices.h
matlabexception.h
matlabmatrix.h
matlabprogram.h
matlabscalar.h
matlabstring.h
arrayofmatrices.cpp
matlabexception.cpp
matlabmatrix.cpp
matlabprogram.cpp
matlabscalar.cpp
matlabstring.cpp
lbfgsb.cpp
 Header and source files for the MATLAB interface. lbfgsb.m
Usage instructions for the MEX file. examplehs038.m
MATLAB script for solving the H&S test example #38. computeObjectiveHS038.m
computeGradientHS038.m
genericcallback.m
 MATLAB functions used by the script [examplehs038.m](src/examplehs038.m). exampleldaimages.m
MATLAB script that generates synthetic documents and topics, then computes a mean field variational approximation to the posterior distribution of the latent Dirichlet allocation model, then displays the result. mflda.m
This function computes a mean field variational approximation to the posterior distribution of the latent Dirichlet allocation model by minimizing the distance between the variational distribution and the true distribution. It uses the L-BFGS-B to minimize the objective function subject to the bound constraints. The objective function also acts as a lower bound on the logarithm of the denominator that appears from the application of Bayes rule **(Note #4)**. callbackMFLDA.m
computeObjectiveMFLDA.m
computeGradientMFLDA.m
computelnZ.m
computelnZconst.m
computem.m
dirichletrnd.m
gammarnd.m
createsyntheticdata.m
mfldainit.m
 Some functions used by [exampleldaimages.m](src/exampleldaimages.m) and [mflda.m](src/mflda.m). ###Credits MATLAB interface Developed by
[Peter Carbonetto](http://www.cs.ubc.ca/spider/pcarbo)
Dept. of Human Genetics
University of Chicago Support for Octave contributed by
[Jose Vallet](http://github.com/josombio)
School of Electrical Engineering
Aalto University Thank you to Ciyou Zhu, Richard Byrd, Jorge Nocedal and Jose Luis for making their L-BFGS-B Fortran code available, and to Hannes Nickisch for ... ###Notes 1. Ciyou Zhu, Richard H. Byrd, Peihuang Lu and Jorge Nocedal (1997). Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization. *ACM Transactions on Mathematical Software* 23(4): 550-560. 2. [Another MATLAB interface](http://www.cs.toronto.edu/~liam/software.shtml) for the L-BFGS-B routines has been developed by Liam Stewart at the University of Toronto. 3. Willi Hock and Klaus Schittkowski (1981). *Test Examples for Nonlinear Programming Codes.* Lecture Notes in Economics and Mathematical Systems, Vol. 187, Springer-Verlag. 4. David M. Blei, Andrew Y. Ng, Michael I. Jordan (2003). Latent Dirichlet allocation. *Journal of Machine Learning Research* Vol3: 993-1022. 5. Tom L. Griffiths and Mark Steyvers (2004). Finding scientific topics. *Proceedings of the National Academy of Sciences* 101: 5228-5235. gpml-matlab-v3.5-2014-12-08/util/lbfgsb/timer.f000664 000765 000024 00000002130 12425212207 020557 0ustar00carlstaff000000 000000 c c L-BFGS-B is released under the “New BSD License” (aka “Modified BSD c License” or “3-clause license”) c Please read attached file License.txt c subroutine timer(ttime) double precision ttime c real temp c c This routine computes cpu time in double precision; it makes use of c the intrinsic f90 cpu_time therefore a conversion type is c needed. c c J.L Morales Departamento de Matematicas, c Instituto Tecnologico Autonomo de Mexico c Mexico D.F. c c J.L Nocedal Department of Electrical Engineering and c Computer Science. c Northwestern University. Evanston, IL. USA c c January 21, 2011 c temp = sngl(ttime) call cpu_time(temp) ttime = dble(temp) return end gpml-matlab-v3.5-2014-12-08/util/lbfgsb/linpack.f000664 000765 000024 00000014274 12425212207 021074 0ustar00carlstaff000000 000000 c c L-BFGS-B is released under the “New BSD License” (aka “Modified BSD c License” or “3-clause license”) c Please read attached file License.txt c subroutine dpofa(a,lda,n,info) integer lda,n,info double precision a(lda,*) c c dpofa factors a double precision symmetric positive definite c matrix. c c dpofa is usually called by dpoco, but it can be called c directly with a saving in time if rcond is not needed. c (time for dpoco) = (1 + 18/n)*(time for dpofa) . c c on entry c c a double precision(lda, n) c the symmetric matrix to be factored. only the c diagonal and upper triangle are used. c c lda integer c the leading dimension of the array a . c c n integer c the order of the matrix a . c c on return c c a an upper triangular matrix r so that a = trans(r)*r c where trans(r) is the transpose. c the strict lower triangle is unaltered. c if info .ne. 0 , the factorization is not complete. c c info integer c = 0 for normal return. c = k signals an error condition. the leading minor c of order k is not positive definite. c c linpack. this version dated 08/14/78 . c cleve moler, university of new mexico, argonne national lab. c c subroutines and functions c c blas ddot c fortran sqrt c c internal variables c double precision ddot,t double precision s integer j,jm1,k c begin block with ...exits to 40 c c do 30 j = 1, n info = j s = 0.0d0 jm1 = j - 1 if (jm1 .lt. 1) go to 20 do 10 k = 1, jm1 t = a(k,j) - ddot(k-1,a(1,k),1,a(1,j),1) t = t/a(k,k) a(k,j) = t s = s + t*t 10 continue 20 continue s = a(j,j) - s c ......exit if (s .le. 0.0d0) go to 40 a(j,j) = sqrt(s) 30 continue info = 0 40 continue return end c====================== The end of dpofa =============================== subroutine dtrsl(t,ldt,n,b,job,info) integer ldt,n,job,info double precision t(ldt,*),b(*) c c c dtrsl solves systems of the form c c t * x = b c or c trans(t) * x = b c c where t is a triangular matrix of order n. here trans(t) c denotes the transpose of the matrix t. c c on entry c c t double precision(ldt,n) c t contains the matrix of the system. the zero c elements of the matrix are not referenced, and c the corresponding elements of the array can be c used to store other information. c c ldt integer c ldt is the leading dimension of the array t. c c n integer c n is the order of the system. c c b double precision(n). c b contains the right hand side of the system. c c job integer c job specifies what kind of system is to be solved. c if job is c c 00 solve t*x=b, t lower triangular, c 01 solve t*x=b, t upper triangular, c 10 solve trans(t)*x=b, t lower triangular, c 11 solve trans(t)*x=b, t upper triangular. c c on return c c b b contains the solution, if info .eq. 0. c otherwise b is unaltered. c c info integer c info contains zero if the system is nonsingular. c otherwise info contains the index of c the first zero diagonal element of t. c c linpack. this version dated 08/14/78 . c g. w. stewart, university of maryland, argonne national lab. c c subroutines and functions c c blas daxpy,ddot c fortran mod c c internal variables c double precision ddot,temp integer case,j,jj c c begin block permitting ...exits to 150 c c check for zero diagonal elements. c do 10 info = 1, n c ......exit if (t(info,info) .eq. 0.0d0) go to 150 10 continue info = 0 c c determine the task and go to it. c case = 1 if (mod(job,10) .ne. 0) case = 2 if (mod(job,100)/10 .ne. 0) case = case + 2 go to (20,50,80,110), case c c solve t*x=b for t lower triangular c 20 continue b(1) = b(1)/t(1,1) if (n .lt. 2) go to 40 do 30 j = 2, n temp = -b(j-1) call daxpy(n-j+1,temp,t(j,j-1),1,b(j),1) b(j) = b(j)/t(j,j) 30 continue 40 continue go to 140 c c solve t*x=b for t upper triangular. c 50 continue b(n) = b(n)/t(n,n) if (n .lt. 2) go to 70 do 60 jj = 2, n j = n - jj + 1 temp = -b(j+1) call daxpy(j,temp,t(1,j+1),1,b(1),1) b(j) = b(j)/t(j,j) 60 continue 70 continue go to 140 c c solve trans(t)*x=b for t lower triangular. c 80 continue b(n) = b(n)/t(n,n) if (n .lt. 2) go to 100 do 90 jj = 2, n j = n - jj + 1 b(j) = b(j) - ddot(jj-1,t(j+1,j),1,b(j+1),1) b(j) = b(j)/t(j,j) 90 continue 100 continue go to 140 c c solve trans(t)*x=b for t upper triangular. c 110 continue b(1) = b(1)/t(1,1) if (n .lt. 2) go to 130 do 120 j = 2, n b(j) = b(j) - ddot(j-1,t(1,j),1,b(1),1) b(j) = b(j)/t(j,j) 120 continue 130 continue 140 continue 150 continue return end c====================== The end of dtrsl =============================== gpml-matlab-v3.5-2014-12-08/util/lbfgsb/arrayofmatrices.h000664 000765 000024 00000004144 12425212207 022643 0ustar00carlstaff000000 000000 #ifndef INCLUDE_ARRAYOFMATRICES #define INCLUDE_ARRAYOFMATRICES #include "array.h" #include "matlabmatrix.h" #include "mex.h" // Class ArrayOfMatrices. // ----------------------------------------------------------------- class ArrayOfMatrices : public Array { public: // This version of the constructor behaves just like its parent. explicit ArrayOfMatrices (int length) : Array(length) { }; // This constructor creates an array of matrices from a Matlab // array. It accepts either a matrix in double precision, or a cell // array with entries that are matrices. explicit ArrayOfMatrices (const mxArray* ptr); // This constructor creates an array of matrices from a collection // of Matlab arrays. The Matlab arrays must be matrices. ArrayOfMatrices (const mxArray* ptrs[], int numptrs); // This constructor creates an array of matrices and the // associated Matlab structures. The Matlab structures are // matrices. The second input argument acts as a template for the // creation of the matrices, but the data from "model" is not // actually copied into the new ArrayOfMatrices object. It is up // to the user to make sure that the array of mxArray pointers has // enough room for the pointers to the Matlab arrays. ArrayOfMatrices (mxArray* ptrs[], const ArrayOfMatrices& model); // This constructor creates an array of matrices using the second // input argument as a model. The input argument "data" contains // the element data. Note that the information is NOT copied from // the model! ArrayOfMatrices (double* data, const ArrayOfMatrices& model); // The copy constructor makes a shallow copy of the source object. ArrayOfMatrices (const ArrayOfMatrices& source); // The destructor. ~ArrayOfMatrices(); // Copy assignment operator that observes the same behaviour as // the Array copy assignment operator. ArrayOfMatrices& operator= (const ArrayOfMatrices& source); // Return the total number of elements in all the matrices. int numelems() const; protected: static int getnummatlabmatrices (const mxArray* ptr); }; #endif gpml-matlab-v3.5-2014-12-08/util/lbfgsb/array.h000664 000765 000024 00000010714 12425212207 020566 0ustar00carlstaff000000 000000 #ifndef INCLUDE_ARRAY #define INCLUDE_ARRAY #include "matlabexception.h" #include // Function definitions. // ----------------------------------------------------------------- template void copymemory (const Type* source, Type* dest, int length) { memcpy(dest,source,sizeof(Type)*length); } // Class Array // ----------------------------------------------------------------- // An Array object stores a collection of ordered elements. The key // aspect of Array objects is that they do not necessarily achieve // encapsulation; they do not necessarily retain independent // ownership of their data. That is, the data could be modified // externally by another agent. This behaviour is determined by the // data member "owner". If "owner" is false, the array does not take // care of allocation and deallocation of the data in memory. // // Copy assignment behaves different than usual---it copies the // data, but requires that the destination already have the proper // resources allocated in memory. // // The copy constructor performs a shallow copy. template class Array { public: // This constructor allocates memory for an array of elements of // the specificed length. explicit Array (int length); // This constructor is set to point to an already existing // array. As such, it does not gain ownership of the elements. Array (Type* data, int length); // The copy constructor performs a shallow copy, so that both // arrays share the same copy of the data. Array (const Array& source); // The destructor. ~Array(); // Set all the elements of the array to the specified value. void setvalue (const Type& value); // Copy the elements to from the source array. Both the source and // destination array must have the same length. void inject (const Type* source); void inject (const Array& source); Array& operator= (const Type* source); Array& operator= (const Array& source); // Get the number of elements in the array. int length() const { return n; }; // Elements of an array A can be accessed the subscript operator; // e.g. A[i]. For the sake of efficiency, no checks are made to // ensure that the subscripting is legal. It is up to the user to // ensure that i is non-negative and less than A.length(). Note // that subscripts, as per C++ convention, start at 0. Type& operator[] (int i); const Type& operator[] (int i) const; // Returns true if the two arrays have the same dimensions // (i.e. the same lengths). bool operator== (const Array& a) const { return n == a.n; }; bool operator!= (const Array& a) const { return !(*this == a); }; protected: Type* elems; int n; // The number of elements in the array. bool owner; // Whether the object has ownership of the data. }; // Function definitions for class Array. // ----------------------------------------------------------------- template Array::Array (int length) { n = length; owner = true; elems = 0; elems = new Type[length]; } template Array::Array (Type* data, int length) { n = length; owner = false; elems = data; } template Array::Array (const Array& source) { n = source.n; owner = false; elems = source.elems; } template Array::~Array () { if (owner && elems) delete[] elems; } template void Array::setvalue (const Type& value) { for (int i = 0; i < n; i++) elems[i] = value; } template void Array::inject (const Type* source) { // Check for self-assignment. If there is self-assignment, there // is no need to copy because the objects share the same data! if (elems != source) copymemory(source,elems,n); } template void Array::inject (const Array& source) { if (n != source.n) throw MatlabException("Unable to perform copy; arrays have \ different lengths"); inject(source.elems); } template Array& Array::operator= (const Type* source) { inject(source); return *this; } template Array& Array::operator= (const Array& source) { inject(source); return *this; } template Type& Array::operator[] (int i) { return elems[i]; } template const Type& Array::operator[] (int i) const { return elems[i]; } #endif gpml-matlab-v3.5-2014-12-08/util/lbfgsb/solver_3_0.f000664 000765 000024 00000376670 12425212207 021441 0ustar00carlstaff000000 000000 c c L-BFGS-B is released under the “New BSD License” (aka “Modified BSD License” c or “3-clause license”) c Please read attached file License.txt c c=========== L-BFGS-B (version 3.0. April 25, 2011 =================== c c This is a modified version of L-BFGS-B. Minor changes in the updated c code appear preceded by a line comment as follows c c c-jlm-jn c c Major changes are described in the accompanying paper: c c Jorge Nocedal and Jose Luis Morales, Remark on "Algorithm 778: c L-BFGS-B: Fortran Subroutines for Large-Scale Bound Constrained c Optimization" (2011). To appear in ACM Transactions on c Mathematical Software, c c The paper describes an improvement and a correction to Algorithm 778. c It is shown that the performance of the algorithm can be improved c significantly by making a relatively simple modication to the subspace c minimization phase. The correction concerns an error caused by the use c of routine dpmeps to estimate machine precision. c c The total work space **wa** required by the new version is c c 2*m*n + 11m*m + 5*n + 8*m c c the old version required c c 2*m*n + 12m*m + 4*n + 12*m c c c J. Nocedal Department of Electrical Engineering and c Computer Science. c Northwestern University. Evanston, IL. USA c c c J.L Morales Departamento de Matematicas, c Instituto Tecnologico Autonomo de Mexico c Mexico D.F. Mexico. c c March 2011 c c============================================================================= subroutine setulb(n, m, x, l, u, nbd, f, g, factr, pgtol, wa, iwa, + task, iprint, csave, lsave, isave, dsave) character*60 task, csave logical lsave(4) integer n, m, iprint, + nbd(n), iwa(3*n), isave(44) double precision f, factr, pgtol, x(n), l(n), u(n), g(n), c c-jlm-jn + wa(2*m*n + 5*n + 11*m*m + 8*m), dsave(29) c ************ c c Subroutine setulb c c This subroutine partitions the working arrays wa and iwa, and c then uses the limited memory BFGS method to solve the bound c constrained optimization problem by calling mainlb. c (The direct method will be used in the subspace minimization.) c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c x is a double precision array of dimension n. c On entry x is an approximation to the solution. c On exit x is the current approximation. c c l is a double precision array of dimension n. c On entry l is the lower bound on x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound on x. c On exit u is unchanged. c c nbd is an integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, and c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c f is a double precision variable. c On first entry f is unspecified. c On final exit f is the value of the function at x. c c g is a double precision array of dimension n. c On first entry g is unspecified. c On final exit g is the value of the gradient at x. c c factr is a double precision variable. c On entry factr >= 0 is specified by the user. The iteration c will stop when c c (f^k - f^{k+1})/max{|f^k|,|f^{k+1}|,1} <= factr*epsmch c c where epsmch is the machine precision, which is automatically c generated by the code. Typical values for factr: 1.d+12 for c low accuracy; 1.d+7 for moderate accuracy; 1.d+1 for extremely c high accuracy. c On exit factr is unchanged. c c pgtol is a double precision variable. c On entry pgtol >= 0 is specified by the user. The iteration c will stop when c c max{|proj g_i | i = 1, ..., n} <= pgtol c c where pg_i is the ith component of the projected gradient. c On exit pgtol is unchanged. c c wa is a double precision working array of length c (2mmax + 5)nmax + 12mmax^2 + 12mmax. c c iwa is an integer working array of length 3nmax. c c task is a working string of characters of length 60 indicating c the current job when entering and quitting this subroutine. c c iprint is an integer variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c csave is a working string of characters of length 60. c c lsave is a logical working array of dimension 4. c On exit with 'task' = NEW_X, the following information is c available: c If lsave(1) = .true. then the initial X has been replaced by c its projection in the feasible set; c If lsave(2) = .true. then the problem is constrained; c If lsave(3) = .true. then each variable has upper and lower c bounds; c c isave is an integer working array of dimension 44. c On exit with 'task' = NEW_X, the following information is c available: c isave(22) = the total number of intervals explored in the c search of Cauchy points; c isave(26) = the total number of skipped BFGS updates before c the current iteration; c isave(30) = the number of current iteration; c isave(31) = the total number of BFGS updates prior the current c iteration; c isave(33) = the number of intervals explored in the search of c Cauchy point in the current iteration; c isave(34) = the total number of function and gradient c evaluations; c isave(36) = the number of function value or gradient c evaluations in the current iteration; c if isave(37) = 0 then the subspace argmin is within the box; c if isave(37) = 1 then the subspace argmin is beyond the box; c isave(38) = the number of free variables in the current c iteration; c isave(39) = the number of active constraints in the current c iteration; c n + 1 - isave(40) = the number of variables leaving the set of c active constraints in the current iteration; c isave(41) = the number of variables entering the set of active c constraints in the current iteration. c c dsave is a double precision working array of dimension 29. c On exit with 'task' = NEW_X, the following information is c available: c dsave(1) = current 'theta' in the BFGS matrix; c dsave(2) = f(x) in the previous iteration; c dsave(3) = factr*epsmch; c dsave(4) = 2-norm of the line search direction vector; c dsave(5) = the machine precision epsmch generated by the code; c dsave(7) = the accumulated time spent on searching for c Cauchy points; c dsave(8) = the accumulated time spent on c subspace minimization; c dsave(9) = the accumulated time spent on line search; c dsave(11) = the slope of the line search function at c the current point of line search; c dsave(12) = the maximum relative step length imposed in c line search; c dsave(13) = the infinity norm of the projected gradient; c dsave(14) = the relative step length in the line search; c dsave(15) = the slope of the line search function at c the starting point of the line search; c dsave(16) = the square of the 2-norm of the line search c direction vector. c c Subprograms called: c c L-BFGS-B Library ... mainlb. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: a c limited memory FORTRAN code for solving bound constrained c optimization problems'', Tech. Report, NAM-11, EECS Department, c Northwestern University, 1994. c c (Postscript files of these papers are available via anonymous c ftp to eecs.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ c-jlm-jn integer lws,lr,lz,lt,ld,lxp,lwa, + lwy,lsy,lss,lwt,lwn,lsnd if (task .eq. 'START') then isave(1) = m*n isave(2) = m**2 isave(3) = 4*m**2 isave(4) = 1 ! ws m*n isave(5) = isave(4) + isave(1) ! wy m*n isave(6) = isave(5) + isave(1) ! wsy m**2 isave(7) = isave(6) + isave(2) ! wss m**2 isave(8) = isave(7) + isave(2) ! wt m**2 isave(9) = isave(8) + isave(2) ! wn 4*m**2 isave(10) = isave(9) + isave(3) ! wsnd 4*m**2 isave(11) = isave(10) + isave(3) ! wz n isave(12) = isave(11) + n ! wr n isave(13) = isave(12) + n ! wd n isave(14) = isave(13) + n ! wt n isave(15) = isave(14) + n ! wxp n isave(16) = isave(15) + n ! wa 8*m endif lws = isave(4) lwy = isave(5) lsy = isave(6) lss = isave(7) lwt = isave(8) lwn = isave(9) lsnd = isave(10) lz = isave(11) lr = isave(12) ld = isave(13) lt = isave(14) lxp = isave(15) lwa = isave(16) call mainlb(n,m,x,l,u,nbd,f,g,factr,pgtol, + wa(lws),wa(lwy),wa(lsy),wa(lss), wa(lwt), + wa(lwn),wa(lsnd),wa(lz),wa(lr),wa(ld),wa(lt),wa(lxp), + wa(lwa), + iwa(1),iwa(n+1),iwa(2*n+1),task,iprint, + csave,lsave,isave(22),dsave) return end c======================= The end of setulb ============================= subroutine mainlb(n, m, x, l, u, nbd, f, g, factr, pgtol, ws, wy, + sy, ss, wt, wn, snd, z, r, d, t, xp, wa, + index, iwhere, indx2, task, + iprint, csave, lsave, isave, dsave) implicit none character*60 task, csave logical lsave(4) integer n, m, iprint, nbd(n), index(n), + iwhere(n), indx2(n), isave(23) double precision f, factr, pgtol, + x(n), l(n), u(n), g(n), z(n), r(n), d(n), t(n), c-jlm-jn + xp(n), + wa(8*m), + ws(n, m), wy(n, m), sy(m, m), ss(m, m), + wt(m, m), wn(2*m, 2*m), snd(2*m, 2*m), dsave(29) c ************ c c Subroutine mainlb c c This subroutine solves bound constrained optimization problems by c using the compact formula of the limited memory BFGS updates. c c n is an integer variable. c On entry n is the number of variables. c On exit n is unchanged. c c m is an integer variable. c On entry m is the maximum number of variable metric c corrections allowed in the limited memory matrix. c On exit m is unchanged. c c x is a double precision array of dimension n. c On entry x is an approximation to the solution. c On exit x is the current approximation. c c l is a double precision array of dimension n. c On entry l is the lower bound of x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound of x. c On exit u is unchanged. c c nbd is an integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c f is a double precision variable. c On first entry f is unspecified. c On final exit f is the value of the function at x. c c g is a double precision array of dimension n. c On first entry g is unspecified. c On final exit g is the value of the gradient at x. c c factr is a double precision variable. c On entry factr >= 0 is specified by the user. The iteration c will stop when c c (f^k - f^{k+1})/max{|f^k|,|f^{k+1}|,1} <= factr*epsmch c c where epsmch is the machine precision, which is automatically c generated by the code. c On exit factr is unchanged. c c pgtol is a double precision variable. c On entry pgtol >= 0 is specified by the user. The iteration c will stop when c c max{|proj g_i | i = 1, ..., n} <= pgtol c c where pg_i is the ith component of the projected gradient. c On exit pgtol is unchanged. c c ws, wy, sy, and wt are double precision working arrays used to c store the following information defining the limited memory c BFGS matrix: c ws, of dimension n x m, stores S, the matrix of s-vectors; c wy, of dimension n x m, stores Y, the matrix of y-vectors; c sy, of dimension m x m, stores S'Y; c ss, of dimension m x m, stores S'S; c yy, of dimension m x m, stores Y'Y; c wt, of dimension m x m, stores the Cholesky factorization c of (theta*S'S+LD^(-1)L'); see eq. c (2.26) in [3]. c c wn is a double precision working array of dimension 2m x 2m c used to store the LEL^T factorization of the indefinite matrix c K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c c where E = [-I 0] c [ 0 I] c c snd is a double precision working array of dimension 2m x 2m c used to store the lower triangular part of c N = [Y' ZZ'Y L_a'+R_z'] c [L_a +R_z S'AA'S ] c c z(n),r(n),d(n),t(n), xp(n),wa(8*m) are double precision working arrays. c z is used at different times to store the Cauchy point and c the Newton point. c xp is used to safeguard the projected Newton direction c c sg(m),sgo(m),yg(m),ygo(m) are double precision working arrays. c c index is an integer working array of dimension n. c In subroutine freev, index is used to store the free and fixed c variables at the Generalized Cauchy Point (GCP). c c iwhere is an integer working array of dimension n used to record c the status of the vector x for GCP computation. c iwhere(i)=0 or -3 if x(i) is free and has bounds, c 1 if x(i) is fixed at l(i), and l(i) .ne. u(i) c 2 if x(i) is fixed at u(i), and u(i) .ne. l(i) c 3 if x(i) is always fixed, i.e., u(i)=x(i)=l(i) c -1 if x(i) is always free, i.e., no bounds on it. c c indx2 is an integer working array of dimension n. c Within subroutine cauchy, indx2 corresponds to the array iorder. c In subroutine freev, a list of variables entering and leaving c the free set is stored in indx2, and it is passed on to c subroutine formk with this information. c c task is a working string of characters of length 60 indicating c the current job when entering and leaving this subroutine. c c iprint is an INTEGER variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c csave is a working string of characters of length 60. c c lsave is a logical working array of dimension 4. c c isave is an integer working array of dimension 23. c c dsave is a double precision working array of dimension 29. c c c Subprograms called c c L-BFGS-B Library ... cauchy, subsm, lnsrlb, formk, c c errclb, prn1lb, prn2lb, prn3lb, active, projgr, c c freev, cmprlb, matupd, formt. c c Minpack2 Library ... timer c c Linpack Library ... dcopy, ddot. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: FORTRAN c Subroutines for Large Scale Bound Constrained Optimization'' c Tech. Report, NAM-11, EECS Department, Northwestern University, c 1994. c c [3] R. Byrd, J. Nocedal and R. Schnabel "Representations of c Quasi-Newton Matrices and their use in Limited Memory Methods'', c Mathematical Programming 63 (1994), no. 4, pp. 129-156. c c (Postscript files of these papers are available via anonymous c ftp to eecs.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ logical prjctd,cnstnd,boxed,updatd,wrk character*3 word integer i,k,nintol,itfile,iback,nskip, + head,col,iter,itail,iupdat, + nseg,nfgv,info,ifun, + iword,nfree,nact,ileave,nenter double precision theta,fold,ddot,dr,rr,tol, + xstep,sbgnrm,ddum,dnorm,dtd,epsmch, + cpu1,cpu2,cachyt,sbtime,lnscht,time1,time2, + gd,gdold,stp,stpmx,time double precision one,zero parameter (one=1.0d0,zero=0.0d0) if (task .eq. 'START') then epsmch = epsilon(one) call timer(time1) c Initialize counters and scalars when task='START'. c for the limited memory BFGS matrices: col = 0 head = 1 theta = one iupdat = 0 updatd = .false. iback = 0 itail = 0 iword = 0 nact = 0 ileave = 0 nenter = 0 fold = zero dnorm = zero cpu1 = zero gd = zero stpmx = zero sbgnrm = zero stp = zero gdold = zero dtd = zero c for operation counts: iter = 0 nfgv = 0 nseg = 0 nintol = 0 nskip = 0 nfree = n ifun = 0 c for stopping tolerance: tol = factr*epsmch c for measuring running time: cachyt = 0 sbtime = 0 lnscht = 0 c 'word' records the status of subspace solutions. word = '---' c 'info' records the termination information. info = 0 itfile = 8 if (iprint .ge. 1) then c open a summary file 'iterate.dat' open (8, file = 'iterate.dat', status = 'unknown') endif c Check the input arguments for errors. call errclb(n,m,factr,l,u,nbd,task,info,k) if (task(1:5) .eq. 'ERROR') then call prn3lb(n,x,f,task,iprint,info,itfile, + iter,nfgv,nintol,nskip,nact,sbgnrm, + zero,nseg,word,iback,stp,xstep,k, + cachyt,sbtime,lnscht) return endif call prn1lb(n,m,l,u,x,iprint,itfile,epsmch) c Initialize iwhere & project x onto the feasible set. call active(n,l,u,nbd,x,iwhere,iprint,prjctd,cnstnd,boxed) c The end of the initialization. else c restore local variables. prjctd = lsave(1) cnstnd = lsave(2) boxed = lsave(3) updatd = lsave(4) nintol = isave(1) itfile = isave(3) iback = isave(4) nskip = isave(5) head = isave(6) col = isave(7) itail = isave(8) iter = isave(9) iupdat = isave(10) nseg = isave(12) nfgv = isave(13) info = isave(14) ifun = isave(15) iword = isave(16) nfree = isave(17) nact = isave(18) ileave = isave(19) nenter = isave(20) theta = dsave(1) fold = dsave(2) tol = dsave(3) dnorm = dsave(4) epsmch = dsave(5) cpu1 = dsave(6) cachyt = dsave(7) sbtime = dsave(8) lnscht = dsave(9) time1 = dsave(10) gd = dsave(11) stpmx = dsave(12) sbgnrm = dsave(13) stp = dsave(14) gdold = dsave(15) dtd = dsave(16) c After returning from the driver go to the point where execution c is to resume. if (task(1:5) .eq. 'FG_LN') goto 666 if (task(1:5) .eq. 'NEW_X') goto 777 if (task(1:5) .eq. 'FG_ST') goto 111 if (task(1:4) .eq. 'STOP') then if (task(7:9) .eq. 'CPU') then c restore the previous iterate. call dcopy(n,t,1,x,1) call dcopy(n,r,1,g,1) f = fold endif goto 999 endif endif c Compute f0 and g0. task = 'FG_START' c return to the driver to calculate f and g; reenter at 111. goto 1000 111 continue nfgv = 1 c Compute the infinity norm of the (-) projected gradient. call projgr(n,l,u,nbd,x,g,sbgnrm) if (iprint .ge. 1) then write (6,1002) iter,f,sbgnrm write (itfile,1003) iter,nfgv,sbgnrm,f endif if (sbgnrm .le. pgtol) then c terminate the algorithm. task = 'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL' goto 999 endif c ----------------- the beginning of the loop -------------------------- 222 continue if (iprint .ge. 99) write (6,1001) iter + 1 iword = -1 c if (.not. cnstnd .and. col .gt. 0) then c skip the search for GCP. call dcopy(n,x,1,z,1) wrk = updatd nseg = 0 goto 333 endif cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Compute the Generalized Cauchy Point (GCP). c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc call timer(cpu1) call cauchy(n,x,l,u,nbd,g,indx2,iwhere,t,d,z, + m,wy,ws,sy,wt,theta,col,head, + wa(1),wa(2*m+1),wa(4*m+1),wa(6*m+1),nseg, + iprint, sbgnrm, info, epsmch) if (info .ne. 0) then c singular triangular system detected; refresh the lbfgs memory. if(iprint .ge. 1) write (6, 1005) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. call timer(cpu2) cachyt = cachyt + cpu2 - cpu1 goto 222 endif call timer(cpu2) cachyt = cachyt + cpu2 - cpu1 nintol = nintol + nseg c Count the entering and leaving variables for iter > 0; c find the index set of free and active variables at the GCP. call freev(n,nfree,index,nenter,ileave,indx2, + iwhere,wrk,updatd,cnstnd,iprint,iter) nact = n - nfree 333 continue c If there are no free variables or B=theta*I, then c skip the subspace minimization. if (nfree .eq. 0 .or. col .eq. 0) goto 555 cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Subspace minimization. c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc call timer(cpu1) c Form the LEL^T factorization of the indefinite c matrix K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c where E = [-I 0] c [ 0 I] if (wrk) call formk(n,nfree,index,nenter,ileave,indx2,iupdat, + updatd,wn,snd,m,ws,wy,sy,theta,col,head,info) if (info .ne. 0) then c nonpositive definiteness in Cholesky factorization; c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1006) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. call timer(cpu2) sbtime = sbtime + cpu2 - cpu1 goto 222 endif c compute r=-Z'B(xcp-xk)-Z'g (using wa(2m+1)=W'(xcp-x) c from 'cauchy'). call cmprlb(n,m,x,g,ws,wy,sy,wt,z,r,wa,index, + theta,col,head,nfree,cnstnd,info) if (info .ne. 0) goto 444 c-jlm-jn call the direct method. call subsm( n, m, nfree, index, l, u, nbd, z, r, xp, ws, wy, + theta, x, g, col, head, iword, wa, wn, iprint, info) 444 continue if (info .ne. 0) then c singular triangular system detected; c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1005) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. call timer(cpu2) sbtime = sbtime + cpu2 - cpu1 goto 222 endif call timer(cpu2) sbtime = sbtime + cpu2 - cpu1 555 continue cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Line search and optimality tests. c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c Generate the search direction d:=z-x. do 40 i = 1, n d(i) = z(i) - x(i) 40 continue call timer(cpu1) 666 continue call lnsrlb(n,l,u,nbd,x,f,fold,gd,gdold,g,d,r,t,z,stp,dnorm, + dtd,xstep,stpmx,iter,ifun,iback,nfgv,info,task, + boxed,cnstnd,csave,isave(22),dsave(17)) if (info .ne. 0 .or. iback .ge. 20) then c restore the previous iterate. call dcopy(n,t,1,x,1) call dcopy(n,r,1,g,1) f = fold if (col .eq. 0) then c abnormal termination. if (info .eq. 0) then info = -9 c restore the actual number of f and g evaluations etc. nfgv = nfgv - 1 ifun = ifun - 1 iback = iback - 1 endif task = 'ABNORMAL_TERMINATION_IN_LNSRCH' iter = iter + 1 goto 999 else c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1008) if (info .eq. 0) nfgv = nfgv - 1 info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. task = 'RESTART_FROM_LNSRCH' call timer(cpu2) lnscht = lnscht + cpu2 - cpu1 goto 222 endif else if (task(1:5) .eq. 'FG_LN') then c return to the driver for calculating f and g; reenter at 666. goto 1000 else c calculate and print out the quantities related to the new X. call timer(cpu2) lnscht = lnscht + cpu2 - cpu1 iter = iter + 1 c Compute the infinity norm of the projected (-)gradient. call projgr(n,l,u,nbd,x,g,sbgnrm) c Print iteration information. call prn2lb(n,x,f,g,iprint,itfile,iter,nfgv,nact, + sbgnrm,nseg,word,iword,iback,stp,xstep) goto 1000 endif 777 continue c Test for termination. if (sbgnrm .le. pgtol) then c terminate the algorithm. task = 'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL' goto 999 endif ddum = max(abs(fold), abs(f), one) if ((fold - f) .le. tol*ddum) then c terminate the algorithm. task = 'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH' if (iback .ge. 10) info = -5 c i.e., to issue a warning if iback>10 in the line search. goto 999 endif c Compute d=newx-oldx, r=newg-oldg, rr=y'y and dr=y's. do 42 i = 1, n r(i) = g(i) - r(i) 42 continue rr = ddot(n,r,1,r,1) if (stp .eq. one) then dr = gd - gdold ddum = -gdold else dr = (gd - gdold)*stp call dscal(n,stp,d,1) ddum = -gdold*stp endif if (dr .le. epsmch*ddum) then c skip the L-BFGS update. nskip = nskip + 1 updatd = .false. if (iprint .ge. 1) write (6,1004) dr, ddum goto 888 endif cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Update the L-BFGS matrix. c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc updatd = .true. iupdat = iupdat + 1 c Update matrices WS and WY and form the middle matrix in B. call matupd(n,m,ws,wy,sy,ss,d,r,itail, + iupdat,col,head,theta,rr,dr,stp,dtd) c Form the upper half of the pds T = theta*SS + L*D^(-1)*L'; c Store T in the upper triangular of the array wt; c Cholesky factorize T to J*J' with c J' stored in the upper triangular of wt. call formt(m,wt,sy,ss,col,theta,info) if (info .ne. 0) then c nonpositive definiteness in Cholesky factorization; c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1007) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. goto 222 endif c Now the inverse of the middle matrix in B is c [ D^(1/2) O ] [ -D^(1/2) D^(-1/2)*L' ] c [ -L*D^(-1/2) J ] [ 0 J' ] 888 continue c -------------------- the end of the loop ----------------------------- goto 222 999 continue call timer(time2) time = time2 - time1 call prn3lb(n,x,f,task,iprint,info,itfile, + iter,nfgv,nintol,nskip,nact,sbgnrm, + time,nseg,word,iback,stp,xstep,k, + cachyt,sbtime,lnscht) 1000 continue c Save local variables. lsave(1) = prjctd lsave(2) = cnstnd lsave(3) = boxed lsave(4) = updatd isave(1) = nintol isave(3) = itfile isave(4) = iback isave(5) = nskip isave(6) = head isave(7) = col isave(8) = itail isave(9) = iter isave(10) = iupdat isave(12) = nseg isave(13) = nfgv isave(14) = info isave(15) = ifun isave(16) = iword isave(17) = nfree isave(18) = nact isave(19) = ileave isave(20) = nenter dsave(1) = theta dsave(2) = fold dsave(3) = tol dsave(4) = dnorm dsave(5) = epsmch dsave(6) = cpu1 dsave(7) = cachyt dsave(8) = sbtime dsave(9) = lnscht dsave(10) = time1 dsave(11) = gd dsave(12) = stpmx dsave(13) = sbgnrm dsave(14) = stp dsave(15) = gdold dsave(16) = dtd 1001 format (//,'ITERATION ',i5) 1002 format + (/,'At iterate',i5,4x,'f= ',1p,d12.5,4x,'|proj g|= ',1p,d12.5) 1003 format (2(1x,i4),5x,'-',5x,'-',3x,'-',5x,'-',5x,'-',8x,'-',3x, + 1p,2(1x,d10.3)) 1004 format (' ys=',1p,e10.3,' -gs=',1p,e10.3,' BFGS update SKIPPED') 1005 format (/, +' Singular triangular system detected;',/, +' refresh the lbfgs memory and restart the iteration.') 1006 format (/, +' Nonpositive definiteness in Cholesky factorization in formk;',/, +' refresh the lbfgs memory and restart the iteration.') 1007 format (/, +' Nonpositive definiteness in Cholesky factorization in formt;',/, +' refresh the lbfgs memory and restart the iteration.') 1008 format (/, +' Bad direction in the line search;',/, +' refresh the lbfgs memory and restart the iteration.') return end c======================= The end of mainlb ============================= subroutine active(n, l, u, nbd, x, iwhere, iprint, + prjctd, cnstnd, boxed) logical prjctd, cnstnd, boxed integer n, iprint, nbd(n), iwhere(n) double precision x(n), l(n), u(n) c ************ c c Subroutine active c c This subroutine initializes iwhere and projects the initial x to c the feasible set if necessary. c c iwhere is an integer array of dimension n. c On entry iwhere is unspecified. c On exit iwhere(i)=-1 if x(i) has no bounds c 3 if l(i)=u(i) c 0 otherwise. c In cauchy, iwhere is given finer gradations. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer nbdd,i double precision zero parameter (zero=0.0d0) c Initialize nbdd, prjctd, cnstnd and boxed. nbdd = 0 prjctd = .false. cnstnd = .false. boxed = .true. c Project the initial x to the easible set if necessary. do 10 i = 1, n if (nbd(i) .gt. 0) then if (nbd(i) .le. 2 .and. x(i) .le. l(i)) then if (x(i) .lt. l(i)) then prjctd = .true. x(i) = l(i) endif nbdd = nbdd + 1 else if (nbd(i) .ge. 2 .and. x(i) .ge. u(i)) then if (x(i) .gt. u(i)) then prjctd = .true. x(i) = u(i) endif nbdd = nbdd + 1 endif endif 10 continue c Initialize iwhere and assign values to cnstnd and boxed. do 20 i = 1, n if (nbd(i) .ne. 2) boxed = .false. if (nbd(i) .eq. 0) then c this variable is always free iwhere(i) = -1 c otherwise set x(i)=mid(x(i), u(i), l(i)). else cnstnd = .true. if (nbd(i) .eq. 2 .and. u(i) - l(i) .le. zero) then c this variable is always fixed iwhere(i) = 3 else iwhere(i) = 0 endif endif 20 continue if (iprint .ge. 0) then if (prjctd) write (6,*) + 'The initial X is infeasible. Restart with its projection.' if (.not. cnstnd) + write (6,*) 'This problem is unconstrained.' endif if (iprint .gt. 0) write (6,1001) nbdd 1001 format (/,'At X0 ',i9,' variables are exactly at the bounds') return end c======================= The end of active ============================= subroutine bmv(m, sy, wt, col, v, p, info) integer m, col, info double precision sy(m, m), wt(m, m), v(2*col), p(2*col) c ************ c c Subroutine bmv c c This subroutine computes the product of the 2m x 2m middle matrix c in the compact L-BFGS formula of B and a 2m vector v; c it returns the product in p. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c sy is a double precision array of dimension m x m. c On entry sy specifies the matrix S'Y. c On exit sy is unchanged. c c wt is a double precision array of dimension m x m. c On entry wt specifies the upper triangular matrix J' which is c the Cholesky factor of (thetaS'S+LD^(-1)L'). c On exit wt is unchanged. c c col is an integer variable. c On entry col specifies the number of s-vectors (or y-vectors) c stored in the compact L-BFGS formula. c On exit col is unchanged. c c v is a double precision array of dimension 2col. c On entry v specifies vector v. c On exit v is unchanged. c c p is a double precision array of dimension 2col. c On entry p is unspecified. c On exit p is the product Mv. c c info is an integer variable. c On entry info is unspecified. c On exit info = 0 for normal return, c = nonzero for abnormal return when the system c to be solved by dtrsl is singular. c c Subprograms called: c c Linpack ... dtrsl. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,k,i2 double precision sum if (col .eq. 0) return c PART I: solve [ D^(1/2) O ] [ p1 ] = [ v1 ] c [ -L*D^(-1/2) J ] [ p2 ] [ v2 ]. c solve Jp2=v2+LD^(-1)v1. p(col + 1) = v(col + 1) do 20 i = 2, col i2 = col + i sum = 0.0d0 do 10 k = 1, i - 1 sum = sum + sy(i,k)*v(k)/sy(k,k) 10 continue p(i2) = v(i2) + sum 20 continue c Solve the triangular system call dtrsl(wt,m,col,p(col+1),11,info) if (info .ne. 0) return c solve D^(1/2)p1=v1. do 30 i = 1, col p(i) = v(i)/sqrt(sy(i,i)) 30 continue c PART II: solve [ -D^(1/2) D^(-1/2)*L' ] [ p1 ] = [ p1 ] c [ 0 J' ] [ p2 ] [ p2 ]. c solve J^Tp2=p2. call dtrsl(wt,m,col,p(col+1),01,info) if (info .ne. 0) return c compute p1=-D^(-1/2)(p1-D^(-1/2)L'p2) c =-D^(-1/2)p1+D^(-1)L'p2. do 40 i = 1, col p(i) = -p(i)/sqrt(sy(i,i)) 40 continue do 60 i = 1, col sum = 0.d0 do 50 k = i + 1, col sum = sum + sy(k,i)*p(col+k)/sy(i,i) 50 continue p(i) = p(i) + sum 60 continue return end c======================== The end of bmv =============================== subroutine cauchy(n, x, l, u, nbd, g, iorder, iwhere, t, d, xcp, + m, wy, ws, sy, wt, theta, col, head, p, c, wbp, + v, nseg, iprint, sbgnrm, info, epsmch) implicit none integer n, m, head, col, nseg, iprint, info, + nbd(n), iorder(n), iwhere(n) double precision theta, epsmch, + x(n), l(n), u(n), g(n), t(n), d(n), xcp(n), + wy(n, col), ws(n, col), sy(m, m), + wt(m, m), p(2*m), c(2*m), wbp(2*m), v(2*m) c ************ c c Subroutine cauchy c c For given x, l, u, g (with sbgnrm > 0), and a limited memory c BFGS matrix B defined in terms of matrices WY, WS, WT, and c scalars head, col, and theta, this subroutine computes the c generalized Cauchy point (GCP), defined as the first local c minimizer of the quadratic c c Q(x + s) = g's + 1/2 s'Bs c c along the projected gradient direction P(x-tg,l,u). c The routine returns the GCP in xcp. c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c x is a double precision array of dimension n. c On entry x is the starting point for the GCP computation. c On exit x is unchanged. c c l is a double precision array of dimension n. c On entry l is the lower bound of x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound of x. c On exit u is unchanged. c c nbd is an integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, and c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c g is a double precision array of dimension n. c On entry g is the gradient of f(x). g must be a nonzero vector. c On exit g is unchanged. c c iorder is an integer working array of dimension n. c iorder will be used to store the breakpoints in the piecewise c linear path and free variables encountered. On exit, c iorder(1),...,iorder(nleft) are indices of breakpoints c which have not been encountered; c iorder(nleft+1),...,iorder(nbreak) are indices of c encountered breakpoints; and c iorder(nfree),...,iorder(n) are indices of variables which c have no bound constraits along the search direction. c c iwhere is an integer array of dimension n. c On entry iwhere indicates only the permanently fixed (iwhere=3) c or free (iwhere= -1) components of x. c On exit iwhere records the status of the current x variables. c iwhere(i)=-3 if x(i) is free and has bounds, but is not moved c 0 if x(i) is free and has bounds, and is moved c 1 if x(i) is fixed at l(i), and l(i) .ne. u(i) c 2 if x(i) is fixed at u(i), and u(i) .ne. l(i) c 3 if x(i) is always fixed, i.e., u(i)=x(i)=l(i) c -1 if x(i) is always free, i.e., it has no bounds. c c t is a double precision working array of dimension n. c t will be used to store the break points. c c d is a double precision array of dimension n used to store c the Cauchy direction P(x-tg)-x. c c xcp is a double precision array of dimension n used to return the c GCP on exit. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c ws, wy, sy, and wt are double precision arrays. c On entry they store information that defines the c limited memory BFGS matrix: c ws(n,m) stores S, a set of s-vectors; c wy(n,m) stores Y, a set of y-vectors; c sy(m,m) stores S'Y; c wt(m,m) stores the c Cholesky factorization of (theta*S'S+LD^(-1)L'). c On exit these arrays are unchanged. c c theta is a double precision variable. c On entry theta is the scaling factor specifying B_0 = theta I. c On exit theta is unchanged. c c col is an integer variable. c On entry col is the actual number of variable metric c corrections stored so far. c On exit col is unchanged. c c head is an integer variable. c On entry head is the location of the first s-vector (or y-vector) c in S (or Y). c On exit col is unchanged. c c p is a double precision working array of dimension 2m. c p will be used to store the vector p = W^(T)d. c c c is a double precision working array of dimension 2m. c c will be used to store the vector c = W^(T)(xcp-x). c c wbp is a double precision working array of dimension 2m. c wbp will be used to store the row of W corresponding c to a breakpoint. c c v is a double precision working array of dimension 2m. c c nseg is an integer variable. c On exit nseg records the number of quadratic segments explored c in searching for the GCP. c c sg and yg are double precision arrays of dimension m. c On entry sg and yg store S'g and Y'g correspondingly. c On exit they are unchanged. c c iprint is an INTEGER variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c sbgnrm is a double precision variable. c On entry sbgnrm is the norm of the projected gradient at x. c On exit sbgnrm is unchanged. c c info is an integer variable. c On entry info is 0. c On exit info = 0 for normal return, c = nonzero for abnormal return when the the system c used in routine bmv is singular. c c Subprograms called: c c L-BFGS-B Library ... hpsolb, bmv. c c Linpack ... dscal dcopy, daxpy. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: FORTRAN c Subroutines for Large Scale Bound Constrained Optimization'' c Tech. Report, NAM-11, EECS Department, Northwestern University, c 1994. c c (Postscript files of these papers are available via anonymous c ftp to eecs.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ logical xlower,xupper,bnded integer i,j,col2,nfree,nbreak,pointr, + ibp,nleft,ibkmin,iter double precision f1,f2,dt,dtm,tsum,dibp,zibp,dibp2,bkmin, + tu,tl,wmc,wmp,wmw,ddot,tj,tj0,neggi,sbgnrm, + f2_org double precision one,zero parameter (one=1.0d0,zero=0.0d0) c Check the status of the variables, reset iwhere(i) if necessary; c compute the Cauchy direction d and the breakpoints t; initialize c the derivative f1 and the vector p = W'd (for theta = 1). if (sbgnrm .le. zero) then if (iprint .ge. 0) write (6,*) 'Subgnorm = 0. GCP = X.' call dcopy(n,x,1,xcp,1) return endif bnded = .true. nfree = n + 1 nbreak = 0 ibkmin = 0 bkmin = zero col2 = 2*col f1 = zero if (iprint .ge. 99) write (6,3010) c We set p to zero and build it up as we determine d. do 20 i = 1, col2 p(i) = zero 20 continue c In the following loop we determine for each variable its bound c status and its breakpoint, and update p accordingly. c Smallest breakpoint is identified. do 50 i = 1, n neggi = -g(i) if (iwhere(i) .ne. 3 .and. iwhere(i) .ne. -1) then c if x(i) is not a constant and has bounds, c compute the difference between x(i) and its bounds. if (nbd(i) .le. 2) tl = x(i) - l(i) if (nbd(i) .ge. 2) tu = u(i) - x(i) c If a variable is close enough to a bound c we treat it as at bound. xlower = nbd(i) .le. 2 .and. tl .le. zero xupper = nbd(i) .ge. 2 .and. tu .le. zero c reset iwhere(i). iwhere(i) = 0 if (xlower) then if (neggi .le. zero) iwhere(i) = 1 else if (xupper) then if (neggi .ge. zero) iwhere(i) = 2 else if (abs(neggi) .le. zero) iwhere(i) = -3 endif endif pointr = head if (iwhere(i) .ne. 0 .and. iwhere(i) .ne. -1) then d(i) = zero else d(i) = neggi f1 = f1 - neggi*neggi c calculate p := p - W'e_i* (g_i). do 40 j = 1, col p(j) = p(j) + wy(i,pointr)* neggi p(col + j) = p(col + j) + ws(i,pointr)*neggi pointr = mod(pointr,m) + 1 40 continue if (nbd(i) .le. 2 .and. nbd(i) .ne. 0 + .and. neggi .lt. zero) then c x(i) + d(i) is bounded; compute t(i). nbreak = nbreak + 1 iorder(nbreak) = i t(nbreak) = tl/(-neggi) if (nbreak .eq. 1 .or. t(nbreak) .lt. bkmin) then bkmin = t(nbreak) ibkmin = nbreak endif else if (nbd(i) .ge. 2 .and. neggi .gt. zero) then c x(i) + d(i) is bounded; compute t(i). nbreak = nbreak + 1 iorder(nbreak) = i t(nbreak) = tu/neggi if (nbreak .eq. 1 .or. t(nbreak) .lt. bkmin) then bkmin = t(nbreak) ibkmin = nbreak endif else c x(i) + d(i) is not bounded. nfree = nfree - 1 iorder(nfree) = i if (abs(neggi) .gt. zero) bnded = .false. endif endif 50 continue c The indices of the nonzero components of d are now stored c in iorder(1),...,iorder(nbreak) and iorder(nfree),...,iorder(n). c The smallest of the nbreak breakpoints is in t(ibkmin)=bkmin. if (theta .ne. one) then c complete the initialization of p for theta not= one. call dscal(col,theta,p(col+1),1) endif c Initialize GCP xcp = x. call dcopy(n,x,1,xcp,1) if (nbreak .eq. 0 .and. nfree .eq. n + 1) then c is a zero vector, return with the initial xcp as GCP. if (iprint .gt. 100) write (6,1010) (xcp(i), i = 1, n) return endif c Initialize c = W'(xcp - x) = 0. do 60 j = 1, col2 c(j) = zero 60 continue c Initialize derivative f2. f2 = -theta*f1 f2_org = f2 if (col .gt. 0) then call bmv(m,sy,wt,col,p,v,info) if (info .ne. 0) return f2 = f2 - ddot(col2,v,1,p,1) endif dtm = -f1/f2 tsum = zero nseg = 1 if (iprint .ge. 99) + write (6,*) 'There are ',nbreak,' breakpoints ' c If there are no breakpoints, locate the GCP and return. if (nbreak .eq. 0) goto 888 nleft = nbreak iter = 1 tj = zero c------------------- the beginning of the loop ------------------------- 777 continue c Find the next smallest breakpoint; c compute dt = t(nleft) - t(nleft + 1). tj0 = tj if (iter .eq. 1) then c Since we already have the smallest breakpoint we need not do c heapsort yet. Often only one breakpoint is used and the c cost of heapsort is avoided. tj = bkmin ibp = iorder(ibkmin) else if (iter .eq. 2) then c Replace the already used smallest breakpoint with the c breakpoint numbered nbreak > nlast, before heapsort call. if (ibkmin .ne. nbreak) then t(ibkmin) = t(nbreak) iorder(ibkmin) = iorder(nbreak) endif c Update heap structure of breakpoints c (if iter=2, initialize heap). endif call hpsolb(nleft,t,iorder,iter-2) tj = t(nleft) ibp = iorder(nleft) endif dt = tj - tj0 if (dt .ne. zero .and. iprint .ge. 100) then write (6,4011) nseg,f1,f2 write (6,5010) dt write (6,6010) dtm endif c If a minimizer is within this interval, locate the GCP and return. if (dtm .lt. dt) goto 888 c Otherwise fix one variable and c reset the corresponding component of d to zero. tsum = tsum + dt nleft = nleft - 1 iter = iter + 1 dibp = d(ibp) d(ibp) = zero if (dibp .gt. zero) then zibp = u(ibp) - x(ibp) xcp(ibp) = u(ibp) iwhere(ibp) = 2 else zibp = l(ibp) - x(ibp) xcp(ibp) = l(ibp) iwhere(ibp) = 1 endif if (iprint .ge. 100) write (6,*) 'Variable ',ibp,' is fixed.' if (nleft .eq. 0 .and. nbreak .eq. n) then c all n variables are fixed, c return with xcp as GCP. dtm = dt goto 999 endif c Update the derivative information. nseg = nseg + 1 dibp2 = dibp**2 c Update f1 and f2. c temporarily set f1 and f2 for col=0. f1 = f1 + dt*f2 + dibp2 - theta*dibp*zibp f2 = f2 - theta*dibp2 if (col .gt. 0) then c update c = c + dt*p. call daxpy(col2,dt,p,1,c,1) c choose wbp, c the row of W corresponding to the breakpoint encountered. pointr = head do 70 j = 1,col wbp(j) = wy(ibp,pointr) wbp(col + j) = theta*ws(ibp,pointr) pointr = mod(pointr,m) + 1 70 continue c compute (wbp)Mc, (wbp)Mp, and (wbp)M(wbp)'. call bmv(m,sy,wt,col,wbp,v,info) if (info .ne. 0) return wmc = ddot(col2,c,1,v,1) wmp = ddot(col2,p,1,v,1) wmw = ddot(col2,wbp,1,v,1) c update p = p - dibp*wbp. call daxpy(col2,-dibp,wbp,1,p,1) c complete updating f1 and f2 while col > 0. f1 = f1 + dibp*wmc f2 = f2 + 2.0d0*dibp*wmp - dibp2*wmw endif f2 = max(epsmch*f2_org,f2) if (nleft .gt. 0) then dtm = -f1/f2 goto 777 c to repeat the loop for unsearched intervals. else if(bnded) then f1 = zero f2 = zero dtm = zero else dtm = -f1/f2 endif c------------------- the end of the loop ------------------------------- 888 continue if (iprint .ge. 99) then write (6,*) write (6,*) 'GCP found in this segment' write (6,4010) nseg,f1,f2 write (6,6010) dtm endif if (dtm .le. zero) dtm = zero tsum = tsum + dtm c Move free variables (i.e., the ones w/o breakpoints) and c the variables whose breakpoints haven't been reached. call daxpy(n,tsum,d,1,xcp,1) 999 continue c Update c = c + dtm*p = W'(x^c - x) c which will be used in computing r = Z'(B(x^c - x) + g). if (col .gt. 0) call daxpy(col2,dtm,p,1,c,1) if (iprint .gt. 100) write (6,1010) (xcp(i),i = 1,n) if (iprint .ge. 99) write (6,2010) 1010 format ('Cauchy X = ',/,(4x,1p,6(1x,d11.4))) 2010 format (/,'---------------- exit CAUCHY----------------------',/) 3010 format (/,'---------------- CAUCHY entered-------------------') 4010 format ('Piece ',i3,' --f1, f2 at start point ',1p,2(1x,d11.4)) 4011 format (/,'Piece ',i3,' --f1, f2 at start point ', + 1p,2(1x,d11.4)) 5010 format ('Distance to the next break point = ',1p,d11.4) 6010 format ('Distance to the stationary point = ',1p,d11.4) return end c====================== The end of cauchy ============================== subroutine cmprlb(n, m, x, g, ws, wy, sy, wt, z, r, wa, index, + theta, col, head, nfree, cnstnd, info) logical cnstnd integer n, m, col, head, nfree, info, index(n) double precision theta, + x(n), g(n), z(n), r(n), wa(4*m), + ws(n, m), wy(n, m), sy(m, m), wt(m, m) c ************ c c Subroutine cmprlb c c This subroutine computes r=-Z'B(xcp-xk)-Z'g by using c wa(2m+1)=W'(xcp-x) from subroutine cauchy. c c Subprograms called: c c L-BFGS-B Library ... bmv. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,j,k,pointr double precision a1,a2 if (.not. cnstnd .and. col .gt. 0) then do 26 i = 1, n r(i) = -g(i) 26 continue else do 30 i = 1, nfree k = index(i) r(i) = -theta*(z(k) - x(k)) - g(k) 30 continue call bmv(m,sy,wt,col,wa(2*m+1),wa(1),info) if (info .ne. 0) then info = -8 return endif pointr = head do 34 j = 1, col a1 = wa(j) a2 = theta*wa(col + j) do 32 i = 1, nfree k = index(i) r(i) = r(i) + wy(k,pointr)*a1 + ws(k,pointr)*a2 32 continue pointr = mod(pointr,m) + 1 34 continue endif return end c======================= The end of cmprlb ============================= subroutine errclb(n, m, factr, l, u, nbd, task, info, k) character*60 task integer n, m, info, k, nbd(n) double precision factr, l(n), u(n) c ************ c c Subroutine errclb c c This subroutine checks the validity of the input data. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i double precision one,zero parameter (one=1.0d0,zero=0.0d0) c Check the input arguments for errors. if (n .le. 0) task = 'ERROR: N .LE. 0' if (m .le. 0) task = 'ERROR: M .LE. 0' if (factr .lt. zero) task = 'ERROR: FACTR .LT. 0' c Check the validity of the arrays nbd(i), u(i), and l(i). do 10 i = 1, n if (nbd(i) .lt. 0 .or. nbd(i) .gt. 3) then c return task = 'ERROR: INVALID NBD' info = -6 k = i endif if (nbd(i) .eq. 2) then if (l(i) .gt. u(i)) then c return task = 'ERROR: NO FEASIBLE SOLUTION' info = -7 k = i endif endif 10 continue return end c======================= The end of errclb ============================= subroutine formk(n, nsub, ind, nenter, ileave, indx2, iupdat, + updatd, wn, wn1, m, ws, wy, sy, theta, col, + head, info) integer n, nsub, m, col, head, nenter, ileave, iupdat, + info, ind(n), indx2(n) double precision theta, wn(2*m, 2*m), wn1(2*m, 2*m), + ws(n, m), wy(n, m), sy(m, m) logical updatd c ************ c c Subroutine formk c c This subroutine forms the LEL^T factorization of the indefinite c c matrix K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c where E = [-I 0] c [ 0 I] c The matrix K can be shown to be equal to the matrix M^[-1]N c occurring in section 5.1 of [1], as well as to the matrix c Mbar^[-1] Nbar in section 5.3. c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c nsub is an integer variable c On entry nsub is the number of subspace variables in free set. c On exit nsub is not changed. c c ind is an integer array of dimension nsub. c On entry ind specifies the indices of subspace variables. c On exit ind is unchanged. c c nenter is an integer variable. c On entry nenter is the number of variables entering the c free set. c On exit nenter is unchanged. c c ileave is an integer variable. c On entry indx2(ileave),...,indx2(n) are the variables leaving c the free set. c On exit ileave is unchanged. c c indx2 is an integer array of dimension n. c On entry indx2(1),...,indx2(nenter) are the variables entering c the free set, while indx2(ileave),...,indx2(n) are the c variables leaving the free set. c On exit indx2 is unchanged. c c iupdat is an integer variable. c On entry iupdat is the total number of BFGS updates made so far. c On exit iupdat is unchanged. c c updatd is a logical variable. c On entry 'updatd' is true if the L-BFGS matrix is updatd. c On exit 'updatd' is unchanged. c c wn is a double precision array of dimension 2m x 2m. c On entry wn is unspecified. c On exit the upper triangle of wn stores the LEL^T factorization c of the 2*col x 2*col indefinite matrix c [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c c wn1 is a double precision array of dimension 2m x 2m. c On entry wn1 stores the lower triangular part of c [Y' ZZ'Y L_a'+R_z'] c [L_a+R_z S'AA'S ] c in the previous iteration. c On exit wn1 stores the corresponding updated matrices. c The purpose of wn1 is just to store these inner products c so they can be easily updated and inserted into wn. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c ws, wy, sy, and wtyy are double precision arrays; c theta is a double precision variable; c col is an integer variable; c head is an integer variable. c On entry they store the information defining the c limited memory BFGS matrix: c ws(n,m) stores S, a set of s-vectors; c wy(n,m) stores Y, a set of y-vectors; c sy(m,m) stores S'Y; c wtyy(m,m) stores the Cholesky factorization c of (theta*S'S+LD^(-1)L') c theta is the scaling factor specifying B_0 = theta I; c col is the number of variable metric corrections stored; c head is the location of the 1st s- (or y-) vector in S (or Y). c On exit they are unchanged. c c info is an integer variable. c On entry info is unspecified. c On exit info = 0 for normal return; c = -1 when the 1st Cholesky factorization failed; c = -2 when the 2st Cholesky factorization failed. c c Subprograms called: c c Linpack ... dcopy, dpofa, dtrsl. c c c References: c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: a c limited memory FORTRAN code for solving bound constrained c optimization problems'', Tech. Report, NAM-11, EECS Department, c Northwestern University, 1994. c c (Postscript files of these papers are available via anonymous c ftp to eecs.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer m2,ipntr,jpntr,iy,is,jy,js,is1,js1,k1,i,k, + col2,pbegin,pend,dbegin,dend,upcl double precision ddot,temp1,temp2,temp3,temp4 double precision one,zero parameter (one=1.0d0,zero=0.0d0) c Form the lower triangular part of c WN1 = [Y' ZZ'Y L_a'+R_z'] c [L_a+R_z S'AA'S ] c where L_a is the strictly lower triangular part of S'AA'Y c R_z is the upper triangular part of S'ZZ'Y. if (updatd) then if (iupdat .gt. m) then c shift old part of WN1. do 10 jy = 1, m - 1 js = m + jy call dcopy(m-jy,wn1(jy+1,jy+1),1,wn1(jy,jy),1) call dcopy(m-jy,wn1(js+1,js+1),1,wn1(js,js),1) call dcopy(m-1,wn1(m+2,jy+1),1,wn1(m+1,jy),1) 10 continue endif c put new rows in blocks (1,1), (2,1) and (2,2). pbegin = 1 pend = nsub dbegin = nsub + 1 dend = n iy = col is = m + col ipntr = head + col - 1 if (ipntr .gt. m) ipntr = ipntr - m jpntr = head do 20 jy = 1, col js = m + jy temp1 = zero temp2 = zero temp3 = zero c compute element jy of row 'col' of Y'ZZ'Y do 15 k = pbegin, pend k1 = ind(k) temp1 = temp1 + wy(k1,ipntr)*wy(k1,jpntr) 15 continue c compute elements jy of row 'col' of L_a and S'AA'S do 16 k = dbegin, dend k1 = ind(k) temp2 = temp2 + ws(k1,ipntr)*ws(k1,jpntr) temp3 = temp3 + ws(k1,ipntr)*wy(k1,jpntr) 16 continue wn1(iy,jy) = temp1 wn1(is,js) = temp2 wn1(is,jy) = temp3 jpntr = mod(jpntr,m) + 1 20 continue c put new column in block (2,1). jy = col jpntr = head + col - 1 if (jpntr .gt. m) jpntr = jpntr - m ipntr = head do 30 i = 1, col is = m + i temp3 = zero c compute element i of column 'col' of R_z do 25 k = pbegin, pend k1 = ind(k) temp3 = temp3 + ws(k1,ipntr)*wy(k1,jpntr) 25 continue ipntr = mod(ipntr,m) + 1 wn1(is,jy) = temp3 30 continue upcl = col - 1 else upcl = col endif c modify the old parts in blocks (1,1) and (2,2) due to changes c in the set of free variables. ipntr = head do 45 iy = 1, upcl is = m + iy jpntr = head do 40 jy = 1, iy js = m + jy temp1 = zero temp2 = zero temp3 = zero temp4 = zero do 35 k = 1, nenter k1 = indx2(k) temp1 = temp1 + wy(k1,ipntr)*wy(k1,jpntr) temp2 = temp2 + ws(k1,ipntr)*ws(k1,jpntr) 35 continue do 36 k = ileave, n k1 = indx2(k) temp3 = temp3 + wy(k1,ipntr)*wy(k1,jpntr) temp4 = temp4 + ws(k1,ipntr)*ws(k1,jpntr) 36 continue wn1(iy,jy) = wn1(iy,jy) + temp1 - temp3 wn1(is,js) = wn1(is,js) - temp2 + temp4 jpntr = mod(jpntr,m) + 1 40 continue ipntr = mod(ipntr,m) + 1 45 continue c modify the old parts in block (2,1). ipntr = head do 60 is = m + 1, m + upcl jpntr = head do 55 jy = 1, upcl temp1 = zero temp3 = zero do 50 k = 1, nenter k1 = indx2(k) temp1 = temp1 + ws(k1,ipntr)*wy(k1,jpntr) 50 continue do 51 k = ileave, n k1 = indx2(k) temp3 = temp3 + ws(k1,ipntr)*wy(k1,jpntr) 51 continue if (is .le. jy + m) then wn1(is,jy) = wn1(is,jy) + temp1 - temp3 else wn1(is,jy) = wn1(is,jy) - temp1 + temp3 endif jpntr = mod(jpntr,m) + 1 55 continue ipntr = mod(ipntr,m) + 1 60 continue c Form the upper triangle of WN = [D+Y' ZZ'Y/theta -L_a'+R_z' ] c [-L_a +R_z S'AA'S*theta] m2 = 2*m do 70 iy = 1, col is = col + iy is1 = m + iy do 65 jy = 1, iy js = col + jy js1 = m + jy wn(jy,iy) = wn1(iy,jy)/theta wn(js,is) = wn1(is1,js1)*theta 65 continue do 66 jy = 1, iy - 1 wn(jy,is) = -wn1(is1,jy) 66 continue do 67 jy = iy, col wn(jy,is) = wn1(is1,jy) 67 continue wn(iy,iy) = wn(iy,iy) + sy(iy,iy) 70 continue c Form the upper triangle of WN= [ LL' L^-1(-L_a'+R_z')] c [(-L_a +R_z)L'^-1 S'AA'S*theta ] c first Cholesky factor (1,1) block of wn to get LL' c with L' stored in the upper triangle of wn. call dpofa(wn,m2,col,info) if (info .ne. 0) then info = -1 return endif c then form L^-1(-L_a'+R_z') in the (1,2) block. col2 = 2*col do 71 js = col+1 ,col2 call dtrsl(wn,m2,col,wn(1,js),11,info) 71 continue c Form S'AA'S*theta + (L^-1(-L_a'+R_z'))'L^-1(-L_a'+R_z') in the c upper triangle of (2,2) block of wn. do 72 is = col+1, col2 do 74 js = is, col2 wn(is,js) = wn(is,js) + ddot(col,wn(1,is),1,wn(1,js),1) 74 continue 72 continue c Cholesky factorization of (2,2) block of wn. call dpofa(wn(col+1,col+1),m2,col,info) if (info .ne. 0) then info = -2 return endif return end c======================= The end of formk ============================== subroutine formt(m, wt, sy, ss, col, theta, info) integer m, col, info double precision theta, wt(m, m), sy(m, m), ss(m, m) c ************ c c Subroutine formt c c This subroutine forms the upper half of the pos. def. and symm. c T = theta*SS + L*D^(-1)*L', stores T in the upper triangle c of the array wt, and performs the Cholesky factorization of T c to produce J*J', with J' stored in the upper triangle of wt. c c Subprograms called: c c Linpack ... dpofa. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,j,k,k1 double precision ddum double precision zero parameter (zero=0.0d0) c Form the upper half of T = theta*SS + L*D^(-1)*L', c store T in the upper triangle of the array wt. do 52 j = 1, col wt(1,j) = theta*ss(1,j) 52 continue do 55 i = 2, col do 54 j = i, col k1 = min(i,j) - 1 ddum = zero do 53 k = 1, k1 ddum = ddum + sy(i,k)*sy(j,k)/sy(k,k) 53 continue wt(i,j) = ddum + theta*ss(i,j) 54 continue 55 continue c Cholesky factorize T to J*J' with c J' stored in the upper triangle of wt. call dpofa(wt,m,col,info) if (info .ne. 0) then info = -3 endif return end c======================= The end of formt ============================== subroutine freev(n, nfree, index, nenter, ileave, indx2, + iwhere, wrk, updatd, cnstnd, iprint, iter) integer n, nfree, nenter, ileave, iprint, iter, + index(n), indx2(n), iwhere(n) logical wrk, updatd, cnstnd c ************ c c Subroutine freev c c This subroutine counts the entering and leaving variables when c iter > 0, and finds the index set of free and active variables c at the GCP. c c cnstnd is a logical variable indicating whether bounds are present c c index is an integer array of dimension n c for i=1,...,nfree, index(i) are the indices of free variables c for i=nfree+1,...,n, index(i) are the indices of bound variables c On entry after the first iteration, index gives c the free variables at the previous iteration. c On exit it gives the free variables based on the determination c in cauchy using the array iwhere. c c indx2 is an integer array of dimension n c On entry indx2 is unspecified. c On exit with iter>0, indx2 indicates which variables c have changed status since the previous iteration. c For i= 1,...,nenter, indx2(i) have changed from bound to free. c For i= ileave+1,...,n, indx2(i) have changed from free to bound. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer iact,i,k nenter = 0 ileave = n + 1 if (iter .gt. 0 .and. cnstnd) then c count the entering and leaving variables. do 20 i = 1, nfree k = index(i) c write(6,*) ' k = index(i) ', k c write(6,*) ' index = ', i if (iwhere(k) .gt. 0) then ileave = ileave - 1 indx2(ileave) = k if (iprint .ge. 100) write (6,*) + 'Variable ',k,' leaves the set of free variables' endif 20 continue do 22 i = 1 + nfree, n k = index(i) if (iwhere(k) .le. 0) then nenter = nenter + 1 indx2(nenter) = k if (iprint .ge. 100) write (6,*) + 'Variable ',k,' enters the set of free variables' endif 22 continue if (iprint .ge. 99) write (6,*) + n+1-ileave,' variables leave; ',nenter,' variables enter' endif wrk = (ileave .lt. n+1) .or. (nenter .gt. 0) .or. updatd c Find the index set of free and active variables at the GCP. nfree = 0 iact = n + 1 do 24 i = 1, n if (iwhere(i) .le. 0) then nfree = nfree + 1 index(nfree) = i else iact = iact - 1 index(iact) = i endif 24 continue if (iprint .ge. 99) write (6,*) + nfree,' variables are free at GCP ',iter + 1 return end c======================= The end of freev ============================== subroutine hpsolb(n, t, iorder, iheap) integer iheap, n, iorder(n) double precision t(n) c ************ c c Subroutine hpsolb c c This subroutine sorts out the least element of t, and puts the c remaining elements of t in a heap. c c n is an integer variable. c On entry n is the dimension of the arrays t and iorder. c On exit n is unchanged. c c t is a double precision array of dimension n. c On entry t stores the elements to be sorted, c On exit t(n) stores the least elements of t, and t(1) to t(n-1) c stores the remaining elements in the form of a heap. c c iorder is an integer array of dimension n. c On entry iorder(i) is the index of t(i). c On exit iorder(i) is still the index of t(i), but iorder may be c permuted in accordance with t. c c iheap is an integer variable specifying the task. c On entry iheap should be set as follows: c iheap .eq. 0 if t(1) to t(n) is not in the form of a heap, c iheap .ne. 0 if otherwise. c On exit iheap is unchanged. c c c References: c Algorithm 232 of CACM (J. W. J. Williams): HEAPSORT. c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c ************ integer i,j,k,indxin,indxou double precision ddum,out if (iheap .eq. 0) then c Rearrange the elements t(1) to t(n) to form a heap. do 20 k = 2, n ddum = t(k) indxin = iorder(k) c Add ddum to the heap. i = k 10 continue if (i.gt.1) then j = i/2 if (ddum .lt. t(j)) then t(i) = t(j) iorder(i) = iorder(j) i = j goto 10 endif endif t(i) = ddum iorder(i) = indxin 20 continue endif c Assign to 'out' the value of t(1), the least member of the heap, c and rearrange the remaining members to form a heap as c elements 1 to n-1 of t. if (n .gt. 1) then i = 1 out = t(1) indxou = iorder(1) ddum = t(n) indxin = iorder(n) c Restore the heap 30 continue j = i+i if (j .le. n-1) then if (t(j+1) .lt. t(j)) j = j+1 if (t(j) .lt. ddum ) then t(i) = t(j) iorder(i) = iorder(j) i = j goto 30 endif endif t(i) = ddum iorder(i) = indxin c Put the least member in t(n). t(n) = out iorder(n) = indxou endif return end c====================== The end of hpsolb ============================== subroutine lnsrlb(n, l, u, nbd, x, f, fold, gd, gdold, g, d, r, t, + z, stp, dnorm, dtd, xstep, stpmx, iter, ifun, + iback, nfgv, info, task, boxed, cnstnd, csave, + isave, dsave) character*60 task, csave logical boxed, cnstnd integer n, iter, ifun, iback, nfgv, info, + nbd(n), isave(2) double precision f, fold, gd, gdold, stp, dnorm, dtd, xstep, + stpmx, x(n), l(n), u(n), g(n), d(n), r(n), t(n), + z(n), dsave(13) c ********** c c Subroutine lnsrlb c c This subroutine calls subroutine dcsrch from the Minpack2 library c to perform the line search. Subroutine dscrch is safeguarded so c that all trial points lie within the feasible region. c c Subprograms called: c c Minpack2 Library ... dcsrch. c c Linpack ... dtrsl, ddot. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ********** integer i double precision ddot,a1,a2 double precision one,zero,big parameter (one=1.0d0,zero=0.0d0,big=1.0d+10) double precision ftol,gtol,xtol parameter (ftol=1.0d-3,gtol=0.9d0,xtol=0.1d0) if (task(1:5) .eq. 'FG_LN') goto 556 dtd = ddot(n,d,1,d,1) dnorm = sqrt(dtd) c Determine the maximum step length. stpmx = big if (cnstnd) then if (iter .eq. 0) then stpmx = one else do 43 i = 1, n a1 = d(i) if (nbd(i) .ne. 0) then if (a1 .lt. zero .and. nbd(i) .le. 2) then a2 = l(i) - x(i) if (a2 .ge. zero) then stpmx = zero else if (a1*stpmx .lt. a2) then stpmx = a2/a1 endif else if (a1 .gt. zero .and. nbd(i) .ge. 2) then a2 = u(i) - x(i) if (a2 .le. zero) then stpmx = zero else if (a1*stpmx .gt. a2) then stpmx = a2/a1 endif endif endif 43 continue endif endif if (iter .eq. 0 .and. .not. boxed) then stp = min(one/dnorm, stpmx) else stp = one endif call dcopy(n,x,1,t,1) call dcopy(n,g,1,r,1) fold = f ifun = 0 iback = 0 csave = 'START' 556 continue gd = ddot(n,g,1,d,1) if (ifun .eq. 0) then gdold=gd if (gd .ge. zero) then c the directional derivative >=0. c Line search is impossible. write(6,*)' ascent direction in projection gd = ', gd info = -4 return endif endif call dcsrch(f,gd,stp,ftol,gtol,xtol,zero,stpmx,csave,isave,dsave) xstep = stp*dnorm if (csave(1:4) .ne. 'CONV' .and. csave(1:4) .ne. 'WARN') then task = 'FG_LNSRCH' ifun = ifun + 1 nfgv = nfgv + 1 iback = ifun - 1 if (stp .eq. one) then call dcopy(n,z,1,x,1) else do 41 i = 1, n x(i) = stp*d(i) + t(i) 41 continue endif else task = 'NEW_X' endif return end c======================= The end of lnsrlb ============================= subroutine matupd(n, m, ws, wy, sy, ss, d, r, itail, + iupdat, col, head, theta, rr, dr, stp, dtd) integer n, m, itail, iupdat, col, head double precision theta, rr, dr, stp, dtd, d(n), r(n), + ws(n, m), wy(n, m), sy(m, m), ss(m, m) c ************ c c Subroutine matupd c c This subroutine updates matrices WS and WY, and forms the c middle matrix in B. c c Subprograms called: c c Linpack ... dcopy, ddot. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer j,pointr double precision ddot double precision one parameter (one=1.0d0) c Set pointers for matrices WS and WY. if (iupdat .le. m) then col = iupdat itail = mod(head+iupdat-2,m) + 1 else itail = mod(itail,m) + 1 head = mod(head,m) + 1 endif c Update matrices WS and WY. call dcopy(n,d,1,ws(1,itail),1) call dcopy(n,r,1,wy(1,itail),1) c Set theta=yy/ys. theta = rr/dr c Form the middle matrix in B. c update the upper triangle of SS, c and the lower triangle of SY: if (iupdat .gt. m) then c move old information do 50 j = 1, col - 1 call dcopy(j,ss(2,j+1),1,ss(1,j),1) call dcopy(col-j,sy(j+1,j+1),1,sy(j,j),1) 50 continue endif c add new information: the last row of SY c and the last column of SS: pointr = head do 51 j = 1, col - 1 sy(col,j) = ddot(n,d,1,wy(1,pointr),1) ss(j,col) = ddot(n,ws(1,pointr),1,d,1) pointr = mod(pointr,m) + 1 51 continue if (stp .eq. one) then ss(col,col) = dtd else ss(col,col) = stp*stp*dtd endif sy(col,col) = dr return end c======================= The end of matupd ============================= subroutine prn1lb(n, m, l, u, x, iprint, itfile, epsmch) integer n, m, iprint, itfile double precision epsmch, x(n), l(n), u(n) c ************ c c Subroutine prn1lb c c This subroutine prints the input data, initial point, upper and c lower bounds of each variable, machine precision, as well as c the headings of the output. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i if (iprint .ge. 0) then write (6,7001) epsmch write (6,*) 'N = ',n,' M = ',m if (iprint .ge. 1) then write (itfile,2001) epsmch write (itfile,*)'N = ',n,' M = ',m write (itfile,9001) if (iprint .gt. 100) then write (6,1004) 'L =',(l(i),i = 1,n) write (6,1004) 'X0 =',(x(i),i = 1,n) write (6,1004) 'U =',(u(i),i = 1,n) endif endif endif 1004 format (/,a4, 1p, 6(1x,d11.4),/,(4x,1p,6(1x,d11.4))) 2001 format ('RUNNING THE L-BFGS-B CODE',/,/, + 'it = iteration number',/, + 'nf = number of function evaluations',/, + 'nseg = number of segments explored during the Cauchy search',/, + 'nact = number of active bounds at the generalized Cauchy point' + ,/, + 'sub = manner in which the subspace minimization terminated:' + ,/,' con = converged, bnd = a bound was reached',/, + 'itls = number of iterations performed in the line search',/, + 'stepl = step length used',/, + 'tstep = norm of the displacement (total step)',/, + 'projg = norm of the projected gradient',/, + 'f = function value',/,/, + ' * * *',/,/, + 'Machine precision =',1p,d10.3) 7001 format ('RUNNING THE L-BFGS-B CODE',/,/, + ' * * *',/,/, + 'Machine precision =',1p,d10.3) 9001 format (/,3x,'it',3x,'nf',2x,'nseg',2x,'nact',2x,'sub',2x,'itls', + 2x,'stepl',4x,'tstep',5x,'projg',8x,'f') return end c======================= The end of prn1lb ============================= subroutine prn2lb(n, x, f, g, iprint, itfile, iter, nfgv, nact, + sbgnrm, nseg, word, iword, iback, stp, xstep) character*3 word integer n, iprint, itfile, iter, nfgv, nact, nseg, + iword, iback double precision f, sbgnrm, stp, xstep, x(n), g(n) c ************ c c Subroutine prn2lb c c This subroutine prints out new information after a successful c line search. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,imod c 'word' records the status of subspace solutions. if (iword .eq. 0) then c the subspace minimization converged. word = 'con' else if (iword .eq. 1) then c the subspace minimization stopped at a bound. word = 'bnd' else if (iword .eq. 5) then c the truncated Newton step has been used. word = 'TNT' else word = '---' endif if (iprint .ge. 99) then write (6,*) 'LINE SEARCH',iback,' times; norm of step = ',xstep write (6,2001) iter,f,sbgnrm if (iprint .gt. 100) then write (6,1004) 'X =',(x(i), i = 1, n) write (6,1004) 'G =',(g(i), i = 1, n) endif else if (iprint .gt. 0) then imod = mod(iter,iprint) if (imod .eq. 0) write (6,2001) iter,f,sbgnrm endif if (iprint .ge. 1) write (itfile,3001) + iter,nfgv,nseg,nact,word,iback,stp,xstep,sbgnrm,f 1004 format (/,a4, 1p, 6(1x,d11.4),/,(4x,1p,6(1x,d11.4))) 2001 format + (/,'At iterate',i5,4x,'f= ',1p,d12.5,4x,'|proj g|= ',1p,d12.5) 3001 format(2(1x,i4),2(1x,i5),2x,a3,1x,i4,1p,2(2x,d7.1),1p,2(1x,d10.3)) return end c======================= The end of prn2lb ============================= subroutine prn3lb(n, x, f, task, iprint, info, itfile, + iter, nfgv, nintol, nskip, nact, sbgnrm, + time, nseg, word, iback, stp, xstep, k, + cachyt, sbtime, lnscht) character*60 task character*3 word integer n, iprint, info, itfile, iter, nfgv, nintol, + nskip, nact, nseg, iback, k double precision f, sbgnrm, time, stp, xstep, cachyt, sbtime, + lnscht, x(n) c ************ c c Subroutine prn3lb c c This subroutine prints out information when either a built-in c convergence test is satisfied or when an error message is c generated. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i if (task(1:5) .eq. 'ERROR') goto 999 if (iprint .ge. 0) then write (6,3003) write (6,3004) write(6,3005) n,iter,nfgv,nintol,nskip,nact,sbgnrm,f if (iprint .ge. 100) then write (6,1004) 'X =',(x(i),i = 1,n) endif if (iprint .ge. 1) write (6,*) ' F =',f endif 999 continue if (iprint .ge. 0) then write (6,3009) task if (info .ne. 0) then if (info .eq. -1) write (6,9011) if (info .eq. -2) write (6,9012) if (info .eq. -3) write (6,9013) if (info .eq. -4) write (6,9014) if (info .eq. -5) write (6,9015) if (info .eq. -6) write (6,*)' Input nbd(',k,') is invalid.' if (info .eq. -7) + write (6,*)' l(',k,') > u(',k,'). No feasible solution.' if (info .eq. -8) write (6,9018) if (info .eq. -9) write (6,9019) endif if (iprint .ge. 1) write (6,3007) cachyt,sbtime,lnscht write (6,3008) time if (iprint .ge. 1) then if (info .eq. -4 .or. info .eq. -9) then write (itfile,3002) + iter,nfgv,nseg,nact,word,iback,stp,xstep endif write (itfile,3009) task if (info .ne. 0) then if (info .eq. -1) write (itfile,9011) if (info .eq. -2) write (itfile,9012) if (info .eq. -3) write (itfile,9013) if (info .eq. -4) write (itfile,9014) if (info .eq. -5) write (itfile,9015) if (info .eq. -8) write (itfile,9018) if (info .eq. -9) write (itfile,9019) endif write (itfile,3008) time endif endif 1004 format (/,a4, 1p, 6(1x,d11.4),/,(4x,1p,6(1x,d11.4))) 3002 format(2(1x,i4),2(1x,i5),2x,a3,1x,i4,1p,2(2x,d7.1),6x,'-',10x,'-') 3003 format (/, + ' * * *',/,/, + 'Tit = total number of iterations',/, + 'Tnf = total number of function evaluations',/, + 'Tnint = total number of segments explored during', + ' Cauchy searches',/, + 'Skip = number of BFGS updates skipped',/, + 'Nact = number of active bounds at final generalized', + ' Cauchy point',/, + 'Projg = norm of the final projected gradient',/, + 'F = final function value',/,/, + ' * * *') 3004 format (/,3x,'N',4x,'Tit',5x,'Tnf',2x,'Tnint',2x, + 'Skip',2x,'Nact',5x,'Projg',8x,'F') 3005 format (i5,2(1x,i6),(1x,i6),(2x,i4),(1x,i5),1p,2(2x,d10.3)) 3007 format (/,' Cauchy time',1p,e10.3,' seconds.',/ + ' Subspace minimization time',1p,e10.3,' seconds.',/ + ' Line search time',1p,e10.3,' seconds.') 3008 format (/,' Total User time',1p,e10.3,' seconds.',/) 3009 format (/,a60) 9011 format (/, +' Matrix in 1st Cholesky factorization in formk is not Pos. Def.') 9012 format (/, +' Matrix in 2st Cholesky factorization in formk is not Pos. Def.') 9013 format (/, +' Matrix in the Cholesky factorization in formt is not Pos. Def.') 9014 format (/, +' Derivative >= 0, backtracking line search impossible.',/, +' Previous x, f and g restored.',/, +' Possible causes: 1 error in function or gradient evaluation;',/, +' 2 rounding errors dominate computation.') 9015 format (/, +' Warning: more than 10 function and gradient',/, +' evaluations in the last line search. Termination',/, +' may possibly be caused by a bad search direction.') 9018 format (/,' The triangular system is singular.') 9019 format (/, +' Line search cannot locate an adequate point after 20 function',/ +,' and gradient evaluations. Previous x, f and g restored.',/, +' Possible causes: 1 error in function or gradient evaluation;',/, +' 2 rounding error dominate computation.') return end c======================= The end of prn3lb ============================= subroutine projgr(n, l, u, nbd, x, g, sbgnrm) integer n, nbd(n) double precision sbgnrm, x(n), l(n), u(n), g(n) c ************ c c Subroutine projgr c c This subroutine computes the infinity norm of the projected c gradient. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i double precision gi double precision one,zero parameter (one=1.0d0,zero=0.0d0) sbgnrm = zero do 15 i = 1, n gi = g(i) if (nbd(i) .ne. 0) then if (gi .lt. zero) then if (nbd(i) .ge. 2) gi = max((x(i)-u(i)),gi) else if (nbd(i) .le. 2) gi = min((x(i)-l(i)),gi) endif endif sbgnrm = max(sbgnrm,abs(gi)) 15 continue return end c======================= The end of projgr ============================= subroutine subsm ( n, m, nsub, ind, l, u, nbd, x, d, xp, ws, wy, + theta, xx, gg, + col, head, iword, wv, wn, iprint, info ) implicit none integer n, m, nsub, col, head, iword, iprint, info, + ind(nsub), nbd(n) double precision theta, + l(n), u(n), x(n), d(n), xp(n), xx(n), gg(n), + ws(n, m), wy(n, m), + wv(2*m), wn(2*m, 2*m) c ********************************************************************** c c This routine contains the major changes in the updated version. c The changes are described in the accompanying paper c c Jose Luis Morales, Jorge Nocedal c "Remark On Algorithm 788: L-BFGS-B: Fortran Subroutines for Large-Scale c Bound Constrained Optimization". Decemmber 27, 2010. c c J.L. Morales Departamento de Matematicas, c Instituto Tecnologico Autonomo de Mexico c Mexico D.F. c c J, Nocedal Department of Electrical Engineering and c Computer Science. c Northwestern University. Evanston, IL. USA c c January 17, 2011 c c ********************************************************************** c c c Subroutine subsm c c Given xcp, l, u, r, an index set that specifies c the active set at xcp, and an l-BFGS matrix B c (in terms of WY, WS, SY, WT, head, col, and theta), c this subroutine computes an approximate solution c of the subspace problem c c (P) min Q(x) = r'(x-xcp) + 1/2 (x-xcp)' B (x-xcp) c c subject to l<=x<=u c x_i=xcp_i for all i in A(xcp) c c along the subspace unconstrained Newton direction c c d = -(Z'BZ)^(-1) r. c c The formula for the Newton direction, given the L-BFGS matrix c and the Sherman-Morrison formula, is c c d = (1/theta)r + (1/theta*2) Z'WK^(-1)W'Z r. c c where c K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c c Note that this procedure for computing d differs c from that described in [1]. One can show that the matrix K is c equal to the matrix M^[-1]N in that paper. c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c nsub is an integer variable. c On entry nsub is the number of free variables. c On exit nsub is unchanged. c c ind is an integer array of dimension nsub. c On entry ind specifies the coordinate indices of free variables. c On exit ind is unchanged. c c l is a double precision array of dimension n. c On entry l is the lower bound of x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound of x. c On exit u is unchanged. c c nbd is a integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, and c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c x is a double precision array of dimension n. c On entry x specifies the Cauchy point xcp. c On exit x(i) is the minimizer of Q over the subspace of c free variables. c c d is a double precision array of dimension n. c On entry d is the reduced gradient of Q at xcp. c On exit d is the Newton direction of Q. c c xp is a double precision array of dimension n. c used to safeguard the projected Newton direction c c xx is a double precision array of dimension n c On entry it holds the current iterate c On output it is unchanged c gg is a double precision array of dimension n c On entry it holds the gradient at the current iterate c On output it is unchanged c c ws and wy are double precision arrays; c theta is a double precision variable; c col is an integer variable; c head is an integer variable. c On entry they store the information defining the c limited memory BFGS matrix: c ws(n,m) stores S, a set of s-vectors; c wy(n,m) stores Y, a set of y-vectors; c theta is the scaling factor specifying B_0 = theta I; c col is the number of variable metric corrections stored; c head is the location of the 1st s- (or y-) vector in S (or Y). c On exit they are unchanged. c c iword is an integer variable. c On entry iword is unspecified. c On exit iword specifies the status of the subspace solution. c iword = 0 if the solution is in the box, c 1 if some bound is encountered. c c wv is a double precision working array of dimension 2m. c c wn is a double precision array of dimension 2m x 2m. c On entry the upper triangle of wn stores the LEL^T factorization c of the indefinite matrix c c K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c where E = [-I 0] c [ 0 I] c On exit wn is unchanged. c c iprint is an INTEGER variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c info is an integer variable. c On entry info is unspecified. c On exit info = 0 for normal return, c = nonzero for abnormal return c when the matrix K is ill-conditioned. c c Subprograms called: c c Linpack dtrsl. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer pointr,m2,col2,ibd,jy,js,i,j,k double precision alpha, xk, dk, temp1, temp2 double precision one,zero parameter (one=1.0d0,zero=0.0d0) c double precision dd_p if (nsub .le. 0) return if (iprint .ge. 99) write (6,1001) c Compute wv = W'Zd. pointr = head do 20 i = 1, col temp1 = zero temp2 = zero do 10 j = 1, nsub k = ind(j) temp1 = temp1 + wy(k,pointr)*d(j) temp2 = temp2 + ws(k,pointr)*d(j) 10 continue wv(i) = temp1 wv(col + i) = theta*temp2 pointr = mod(pointr,m) + 1 20 continue c Compute wv:=K^(-1)wv. m2 = 2*m col2 = 2*col call dtrsl(wn,m2,col2,wv,11,info) if (info .ne. 0) return do 25 i = 1, col wv(i) = -wv(i) 25 continue call dtrsl(wn,m2,col2,wv,01,info) if (info .ne. 0) return c Compute d = (1/theta)d + (1/theta**2)Z'W wv. pointr = head do 40 jy = 1, col js = col + jy do 30 i = 1, nsub k = ind(i) d(i) = d(i) + wy(k,pointr)*wv(jy)/theta + + ws(k,pointr)*wv(js) 30 continue pointr = mod(pointr,m) + 1 40 continue call dscal( nsub, one/theta, d, 1 ) c c----------------------------------------------------------------- c Let us try the projection, d is the Newton direction iword = 0 call dcopy ( n, x, 1, xp, 1 ) c do 50 i=1, nsub k = ind(i) dk = d(i) xk = x(k) if ( nbd(k) .ne. 0 ) then c if ( nbd(k).eq.1 ) then ! lower bounds only x(k) = max( l(k), xk + dk ) if ( x(k).eq.l(k) ) iword = 1 else c if ( nbd(k).eq.2 ) then ! upper and lower bounds xk = max( l(k), xk + dk ) x(k) = min( u(k), xk ) if ( x(k).eq.l(k) .or. x(k).eq.u(k) ) iword = 1 else c if ( nbd(k).eq.3 ) then ! upper bounds only x(k) = min( u(k), xk + dk ) if ( x(k).eq.u(k) ) iword = 1 end if end if end if c else ! free variables x(k) = xk + dk end if 50 continue c if ( iword.eq.0 ) then go to 911 end if c c check sign of the directional derivative c dd_p = zero do 55 i=1, n dd_p = dd_p + (x(i) - xx(i))*gg(i) 55 continue if ( dd_p .gt.zero ) then call dcopy( n, xp, 1, x, 1 ) write(6,*) ' Positive dir derivative in projection ' write(6,*) ' Using the backtracking step ' else go to 911 endif c c----------------------------------------------------------------- c alpha = one temp1 = alpha ibd = 0 do 60 i = 1, nsub k = ind(i) dk = d(i) if (nbd(k) .ne. 0) then if (dk .lt. zero .and. nbd(k) .le. 2) then temp2 = l(k) - x(k) if (temp2 .ge. zero) then temp1 = zero else if (dk*alpha .lt. temp2) then temp1 = temp2/dk endif else if (dk .gt. zero .and. nbd(k) .ge. 2) then temp2 = u(k) - x(k) if (temp2 .le. zero) then temp1 = zero else if (dk*alpha .gt. temp2) then temp1 = temp2/dk endif endif if (temp1 .lt. alpha) then alpha = temp1 ibd = i endif endif 60 continue if (alpha .lt. one) then dk = d(ibd) k = ind(ibd) if (dk .gt. zero) then x(k) = u(k) d(ibd) = zero else if (dk .lt. zero) then x(k) = l(k) d(ibd) = zero endif endif do 70 i = 1, nsub k = ind(i) x(k) = x(k) + alpha*d(i) 70 continue cccccc 911 continue if (iprint .ge. 99) write (6,1004) 1001 format (/,'----------------SUBSM entered-----------------',/) 1004 format (/,'----------------exit SUBSM --------------------',/) return end c====================== The end of subsm =============================== subroutine dcsrch(f,g,stp,ftol,gtol,xtol,stpmin,stpmax, + task,isave,dsave) character*(*) task integer isave(2) double precision f,g,stp,ftol,gtol,xtol,stpmin,stpmax double precision dsave(13) c ********** c c Subroutine dcsrch c c This subroutine finds a step that satisfies a sufficient c decrease condition and a curvature condition. c c Each call of the subroutine updates an interval with c endpoints stx and sty. The interval is initially chosen c so that it contains a minimizer of the modified function c c psi(stp) = f(stp) - f(0) - ftol*stp*f'(0). c c If psi(stp) <= 0 and f'(stp) >= 0 for some step, then the c interval is chosen so that it contains a minimizer of f. c c The algorithm is designed to find a step that satisfies c the sufficient decrease condition c c f(stp) <= f(0) + ftol*stp*f'(0), c c and the curvature condition c c abs(f'(stp)) <= gtol*abs(f'(0)). c c If ftol is less than gtol and if, for example, the function c is bounded below, then there is always a step which satisfies c both conditions. c c If no step can be found that satisfies both conditions, then c the algorithm stops with a warning. In this case stp only c satisfies the sufficient decrease condition. c c A typical invocation of dcsrch has the following outline: c c task = 'START' c 10 continue c call dcsrch( ... ) c if (task .eq. 'FG') then c Evaluate the function and the gradient at stp c goto 10 c end if c c NOTE: The user must no alter work arrays between calls. c c The subroutine statement is c c subroutine dcsrch(f,g,stp,ftol,gtol,xtol,stpmin,stpmax, c task,isave,dsave) c where c c f is a double precision variable. c On initial entry f is the value of the function at 0. c On subsequent entries f is the value of the c function at stp. c On exit f is the value of the function at stp. c c g is a double precision variable. c On initial entry g is the derivative of the function at 0. c On subsequent entries g is the derivative of the c function at stp. c On exit g is the derivative of the function at stp. c c stp is a double precision variable. c On entry stp is the current estimate of a satisfactory c step. On initial entry, a positive initial estimate c must be provided. c On exit stp is the current estimate of a satisfactory step c if task = 'FG'. If task = 'CONV' then stp satisfies c the sufficient decrease and curvature condition. c c ftol is a double precision variable. c On entry ftol specifies a nonnegative tolerance for the c sufficient decrease condition. c On exit ftol is unchanged. c c gtol is a double precision variable. c On entry gtol specifies a nonnegative tolerance for the c curvature condition. c On exit gtol is unchanged. c c xtol is a double precision variable. c On entry xtol specifies a nonnegative relative tolerance c for an acceptable step. The subroutine exits with a c warning if the relative difference between sty and stx c is less than xtol. c On exit xtol is unchanged. c c stpmin is a double precision variable. c On entry stpmin is a nonnegative lower bound for the step. c On exit stpmin is unchanged. c c stpmax is a double precision variable. c On entry stpmax is a nonnegative upper bound for the step. c On exit stpmax is unchanged. c c task is a character variable of length at least 60. c On initial entry task must be set to 'START'. c On exit task indicates the required action: c c If task(1:2) = 'FG' then evaluate the function and c derivative at stp and call dcsrch again. c c If task(1:4) = 'CONV' then the search is successful. c c If task(1:4) = 'WARN' then the subroutine is not able c to satisfy the convergence conditions. The exit value of c stp contains the best point found during the search. c c If task(1:5) = 'ERROR' then there is an error in the c input arguments. c c On exit with convergence, a warning or an error, the c variable task contains additional information. c c isave is an integer work array of dimension 2. c c dsave is a double precision work array of dimension 13. c c Subprograms called c c MINPACK-2 ... dcstep c c MINPACK-1 Project. June 1983. c Argonne National Laboratory. c Jorge J. More' and David J. Thuente. c c MINPACK-2 Project. October 1993. c Argonne National Laboratory and University of Minnesota. c Brett M. Averick, Richard G. Carter, and Jorge J. More'. c c ********** double precision zero,p5,p66 parameter(zero=0.0d0,p5=0.5d0,p66=0.66d0) double precision xtrapl,xtrapu parameter(xtrapl=1.1d0,xtrapu=4.0d0) logical brackt integer stage double precision finit,ftest,fm,fx,fxm,fy,fym,ginit,gtest, + gm,gx,gxm,gy,gym,stx,sty,stmin,stmax,width,width1 c Initialization block. if (task(1:5) .eq. 'START') then c Check the input arguments for errors. if (stp .lt. stpmin) task = 'ERROR: STP .LT. STPMIN' if (stp .gt. stpmax) task = 'ERROR: STP .GT. STPMAX' if (g .ge. zero) task = 'ERROR: INITIAL G .GE. ZERO' if (ftol .lt. zero) task = 'ERROR: FTOL .LT. ZERO' if (gtol .lt. zero) task = 'ERROR: GTOL .LT. ZERO' if (xtol .lt. zero) task = 'ERROR: XTOL .LT. ZERO' if (stpmin .lt. zero) task = 'ERROR: STPMIN .LT. ZERO' if (stpmax .lt. stpmin) task = 'ERROR: STPMAX .LT. STPMIN' c Exit if there are errors on input. if (task(1:5) .eq. 'ERROR') return c Initialize local variables. brackt = .false. stage = 1 finit = f ginit = g gtest = ftol*ginit width = stpmax - stpmin width1 = width/p5 c The variables stx, fx, gx contain the values of the step, c function, and derivative at the best step. c The variables sty, fy, gy contain the value of the step, c function, and derivative at sty. c The variables stp, f, g contain the values of the step, c function, and derivative at stp. stx = zero fx = finit gx = ginit sty = zero fy = finit gy = ginit stmin = zero stmax = stp + xtrapu*stp task = 'FG' goto 1000 else c Restore local variables. if (isave(1) .eq. 1) then brackt = .true. else brackt = .false. endif stage = isave(2) ginit = dsave(1) gtest = dsave(2) gx = dsave(3) gy = dsave(4) finit = dsave(5) fx = dsave(6) fy = dsave(7) stx = dsave(8) sty = dsave(9) stmin = dsave(10) stmax = dsave(11) width = dsave(12) width1 = dsave(13) endif c If psi(stp) <= 0 and f'(stp) >= 0 for some step, then the c algorithm enters the second stage. ftest = finit + stp*gtest if (stage .eq. 1 .and. f .le. ftest .and. g .ge. zero) + stage = 2 c Test for warnings. if (brackt .and. (stp .le. stmin .or. stp .ge. stmax)) + task = 'WARNING: ROUNDING ERRORS PREVENT PROGRESS' if (brackt .and. stmax - stmin .le. xtol*stmax) + task = 'WARNING: XTOL TEST SATISFIED' if (stp .eq. stpmax .and. f .le. ftest .and. g .le. gtest) + task = 'WARNING: STP = STPMAX' if (stp .eq. stpmin .and. (f .gt. ftest .or. g .ge. gtest)) + task = 'WARNING: STP = STPMIN' c Test for convergence. if (f .le. ftest .and. abs(g) .le. gtol*(-ginit)) + task = 'CONVERGENCE' c Test for termination. if (task(1:4) .eq. 'WARN' .or. task(1:4) .eq. 'CONV') goto 1000 c A modified function is used to predict the step during the c first stage if a lower function value has been obtained but c the decrease is not sufficient. if (stage .eq. 1 .and. f .le. fx .and. f .gt. ftest) then c Define the modified function and derivative values. fm = f - stp*gtest fxm = fx - stx*gtest fym = fy - sty*gtest gm = g - gtest gxm = gx - gtest gym = gy - gtest c Call dcstep to update stx, sty, and to compute the new step. call dcstep(stx,fxm,gxm,sty,fym,gym,stp,fm,gm, + brackt,stmin,stmax) c Reset the function and derivative values for f. fx = fxm + stx*gtest fy = fym + sty*gtest gx = gxm + gtest gy = gym + gtest else c Call dcstep to update stx, sty, and to compute the new step. call dcstep(stx,fx,gx,sty,fy,gy,stp,f,g, + brackt,stmin,stmax) endif c Decide if a bisection step is needed. if (brackt) then if (abs(sty-stx) .ge. p66*width1) stp = stx + p5*(sty - stx) width1 = width width = abs(sty-stx) endif c Set the minimum and maximum steps allowed for stp. if (brackt) then stmin = min(stx,sty) stmax = max(stx,sty) else stmin = stp + xtrapl*(stp - stx) stmax = stp + xtrapu*(stp - stx) endif c Force the step to be within the bounds stpmax and stpmin. stp = max(stp,stpmin) stp = min(stp,stpmax) c If further progress is not possible, let stp be the best c point obtained during the search. if (brackt .and. (stp .le. stmin .or. stp .ge. stmax) + .or. (brackt .and. stmax-stmin .le. xtol*stmax)) stp = stx c Obtain another function and derivative. task = 'FG' 1000 continue c Save local variables. if (brackt) then isave(1) = 1 else isave(1) = 0 endif isave(2) = stage dsave(1) = ginit dsave(2) = gtest dsave(3) = gx dsave(4) = gy dsave(5) = finit dsave(6) = fx dsave(7) = fy dsave(8) = stx dsave(9) = sty dsave(10) = stmin dsave(11) = stmax dsave(12) = width dsave(13) = width1 return end c====================== The end of dcsrch ============================== subroutine dcstep(stx,fx,dx,sty,fy,dy,stp,fp,dp,brackt, + stpmin,stpmax) logical brackt double precision stx,fx,dx,sty,fy,dy,stp,fp,dp,stpmin,stpmax c ********** c c Subroutine dcstep c c This subroutine computes a safeguarded step for a search c procedure and updates an interval that contains a step that c satisfies a sufficient decrease and a curvature condition. c c The parameter stx contains the step with the least function c value. If brackt is set to .true. then a minimizer has c been bracketed in an interval with endpoints stx and sty. c The parameter stp contains the current step. c The subroutine assumes that if brackt is set to .true. then c c min(stx,sty) < stp < max(stx,sty), c c and that the derivative at stx is negative in the direction c of the step. c c The subroutine statement is c c subroutine dcstep(stx,fx,dx,sty,fy,dy,stp,fp,dp,brackt, c stpmin,stpmax) c c where c c stx is a double precision variable. c On entry stx is the best step obtained so far and is an c endpoint of the interval that contains the minimizer. c On exit stx is the updated best step. c c fx is a double precision variable. c On entry fx is the function at stx. c On exit fx is the function at stx. c c dx is a double precision variable. c On entry dx is the derivative of the function at c stx. The derivative must be negative in the direction of c the step, that is, dx and stp - stx must have opposite c signs. c On exit dx is the derivative of the function at stx. c c sty is a double precision variable. c On entry sty is the second endpoint of the interval that c contains the minimizer. c On exit sty is the updated endpoint of the interval that c contains the minimizer. c c fy is a double precision variable. c On entry fy is the function at sty. c On exit fy is the function at sty. c c dy is a double precision variable. c On entry dy is the derivative of the function at sty. c On exit dy is the derivative of the function at the exit sty. c c stp is a double precision variable. c On entry stp is the current step. If brackt is set to .true. c then on input stp must be between stx and sty. c On exit stp is a new trial step. c c fp is a double precision variable. c On entry fp is the function at stp c On exit fp is unchanged. c c dp is a double precision variable. c On entry dp is the the derivative of the function at stp. c On exit dp is unchanged. c c brackt is an logical variable. c On entry brackt specifies if a minimizer has been bracketed. c Initially brackt must be set to .false. c On exit brackt specifies if a minimizer has been bracketed. c When a minimizer is bracketed brackt is set to .true. c c stpmin is a double precision variable. c On entry stpmin is a lower bound for the step. c On exit stpmin is unchanged. c c stpmax is a double precision variable. c On entry stpmax is an upper bound for the step. c On exit stpmax is unchanged. c c MINPACK-1 Project. June 1983 c Argonne National Laboratory. c Jorge J. More' and David J. Thuente. c c MINPACK-2 Project. October 1993. c Argonne National Laboratory and University of Minnesota. c Brett M. Averick and Jorge J. More'. c c ********** double precision zero,p66,two,three parameter(zero=0.0d0,p66=0.66d0,two=2.0d0,three=3.0d0) double precision gamma,p,q,r,s,sgnd,stpc,stpf,stpq,theta sgnd = dp*(dx/abs(dx)) c First case: A higher function value. The minimum is bracketed. c If the cubic step is closer to stx than the quadratic step, the c cubic step is taken, otherwise the average of the cubic and c quadratic steps is taken. if (fp .gt. fx) then theta = three*(fx - fp)/(stp - stx) + dx + dp s = max(abs(theta),abs(dx),abs(dp)) gamma = s*sqrt((theta/s)**2 - (dx/s)*(dp/s)) if (stp .lt. stx) gamma = -gamma p = (gamma - dx) + theta q = ((gamma - dx) + gamma) + dp r = p/q stpc = stx + r*(stp - stx) stpq = stx + ((dx/((fx - fp)/(stp - stx) + dx))/two)* + (stp - stx) if (abs(stpc-stx) .lt. abs(stpq-stx)) then stpf = stpc else stpf = stpc + (stpq - stpc)/two endif brackt = .true. c Second case: A lower function value and derivatives of opposite c sign. The minimum is bracketed. If the cubic step is farther from c stp than the secant step, the cubic step is taken, otherwise the c secant step is taken. else if (sgnd .lt. zero) then theta = three*(fx - fp)/(stp - stx) + dx + dp s = max(abs(theta),abs(dx),abs(dp)) gamma = s*sqrt((theta/s)**2 - (dx/s)*(dp/s)) if (stp .gt. stx) gamma = -gamma p = (gamma - dp) + theta q = ((gamma - dp) + gamma) + dx r = p/q stpc = stp + r*(stx - stp) stpq = stp + (dp/(dp - dx))*(stx - stp) if (abs(stpc-stp) .gt. abs(stpq-stp)) then stpf = stpc else stpf = stpq endif brackt = .true. c Third case: A lower function value, derivatives of the same sign, c and the magnitude of the derivative decreases. else if (abs(dp) .lt. abs(dx)) then c The cubic step is computed only if the cubic tends to infinity c in the direction of the step or if the minimum of the cubic c is beyond stp. Otherwise the cubic step is defined to be the c secant step. theta = three*(fx - fp)/(stp - stx) + dx + dp s = max(abs(theta),abs(dx),abs(dp)) c The case gamma = 0 only arises if the cubic does not tend c to infinity in the direction of the step. gamma = s*sqrt(max(zero,(theta/s)**2-(dx/s)*(dp/s))) if (stp .gt. stx) gamma = -gamma p = (gamma - dp) + theta q = (gamma + (dx - dp)) + gamma r = p/q if (r .lt. zero .and. gamma .ne. zero) then stpc = stp + r*(stx - stp) else if (stp .gt. stx) then stpc = stpmax else stpc = stpmin endif stpq = stp + (dp/(dp - dx))*(stx - stp) if (brackt) then c A minimizer has been bracketed. If the cubic step is c closer to stp than the secant step, the cubic step is c taken, otherwise the secant step is taken. if (abs(stpc-stp) .lt. abs(stpq-stp)) then stpf = stpc else stpf = stpq endif if (stp .gt. stx) then stpf = min(stp+p66*(sty-stp),stpf) else stpf = max(stp+p66*(sty-stp),stpf) endif else c A minimizer has not been bracketed. If the cubic step is c farther from stp than the secant step, the cubic step is c taken, otherwise the secant step is taken. if (abs(stpc-stp) .gt. abs(stpq-stp)) then stpf = stpc else stpf = stpq endif stpf = min(stpmax,stpf) stpf = max(stpmin,stpf) endif c Fourth case: A lower function value, derivatives of the same sign, c and the magnitude of the derivative does not decrease. If the c minimum is not bracketed, the step is either stpmin or stpmax, c otherwise the cubic step is taken. else if (brackt) then theta = three*(fp - fy)/(sty - stp) + dy + dp s = max(abs(theta),abs(dy),abs(dp)) gamma = s*sqrt((theta/s)**2 - (dy/s)*(dp/s)) if (stp .gt. sty) gamma = -gamma p = (gamma - dp) + theta q = ((gamma - dp) + gamma) + dy r = p/q stpc = stp + r*(sty - stp) stpf = stpc else if (stp .gt. stx) then stpf = stpmax else stpf = stpmin endif endif c Update the interval which contains a minimizer. if (fp .gt. fx) then sty = stp fy = fp dy = dp else if (sgnd .lt. zero) then sty = stx fy = fx dy = dx endif stx = stp fx = fp dx = dp endif c Compute the new step. stp = stpf return end gpml-matlab-v3.5-2014-12-08/util/lbfgsb/solver_2_4.f000664 000765 000024 00000423410 12425212207 021425 0ustar00carlstaff000000 000000 c================ L-BFGS-B (version 2.4) ========================== subroutine setulb(n, m, x, l, u, nbd, f, g, factr, pgtol, wa, iwa, + task, iprint, csave, lsave, isave, dsave) character*60 task, csave logical lsave(4) integer n, m, iprint, + nbd(n), iwa(3*n), isave(44) double precision f, factr, pgtol, x(n), l(n), u(n), g(n), + wa(2*m*n+4*n+11*m*m+8*m), dsave(29) c ************ c c Subroutine setulb c c This subroutine partitions the working arrays wa and iwa, and c then uses the limited memory BFGS method to solve the bound c constrained optimization problem by calling mainlb. c (The direct method will be used in the subspace minimization.) c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c x is a double precision array of dimension n. c On entry x is an approximation to the solution. c On exit x is the current approximation. c c l is a double precision array of dimension n. c On entry l is the lower bound on x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound on x. c On exit u is unchanged. c c nbd is an integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, and c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c f is a double precision variable. c On first entry f is unspecified. c On final exit f is the value of the function at x. c c g is a double precision array of dimension n. c On first entry g is unspecified. c On final exit g is the value of the gradient at x. c c factr is a double precision variable. c On entry factr >= 0 is specified by the user. The iteration c will stop when c c (f^k - f^{k+1})/max{|f^k|,|f^{k+1}|,1} <= factr*epsmch c c where epsmch is the machine precision, which is automatically c generated by the code. Typical values for factr: 1.d+12 for c low accuracy; 1.d+7 for moderate accuracy; 1.d+1 for extremely c high accuracy. c On exit factr is unchanged. c c pgtol is a double precision variable. c On entry pgtol >= 0 is specified by the user. The iteration c will stop when c c max{|proj g_i | i = 1, ..., n} <= pgtol c c where pg_i is the ith component of the projected gradient. c On exit pgtol is unchanged. c c wa is a double precision working array of length c (2mmax + 4)nmax + 11mmax^2 + 8mmax. c c iwa is an integer working array of length 3nmax. c c task is a working string of characters of length 60 indicating c the current job when entering and quitting this subroutine. c c iprint is an integer variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c csave is a working string of characters of length 60. c c lsave is a logical working array of dimension 4. c On exit with 'task' = NEW_X, the following information is c available: c If lsave(1) = .true. then the initial X has been replaced by c its projection in the feasible set; c If lsave(2) = .true. then the problem is constrained; c If lsave(3) = .true. then each variable has upper and lower c bounds; c c isave is an integer working array of dimension 44. c On exit with 'task' = NEW_X, the following information is c available: c isave(22) = the total number of intervals explored in the c search of Cauchy points; c isave(26) = the total number of skipped BFGS updates before c the current iteration; c isave(30) = the number of current iteration; c isave(31) = the total number of BFGS updates prior the current c iteration; c isave(33) = the number of intervals explored in the search of c Cauchy point in the current iteration; c isave(34) = the total number of function and gradient c evaluations; c isave(36) = the number of function value or gradient c evaluations in the current iteration; c if isave(37) = 0 then the subspace argmin is within the box; c if isave(37) = 1 then the subspace argmin is beyond the box; c isave(38) = the number of free variables in the current c iteration; c isave(39) = the number of active constraints in the current c iteration; c n + 1 - isave(40) = the number of variables leaving the set of c active constraints in the current iteration; c isave(41) = the number of variables entering the set of active c constraints in the current iteration. c c dsave is a double precision working array of dimension 29. c On exit with 'task' = NEW_X, the following information is c available: c dsave(1) = current 'theta' in the BFGS matrix; c dsave(2) = f(x) in the previous iteration; c dsave(3) = factr*epsmch; c dsave(4) = 2-norm of the line search direction vector; c dsave(5) = the machine precision epsmch generated by the code; c dsave(7) = the accumulated time spent on searching for c Cauchy points; c dsave(8) = the accumulated time spent on c subspace minimization; c dsave(9) = the accumulated time spent on line search; c dsave(11) = the slope of the line search function at c the current point of line search; c dsave(12) = the maximum relative step length imposed in c line search; c dsave(13) = the infinity norm of the projected gradient; c dsave(14) = the relative step length in the line search; c dsave(15) = the slope of the line search function at c the starting point of the line search; c dsave(16) = the square of the 2-norm of the line search c direction vector. c c Subprograms called: c c L-BFGS-B Library ... mainlb. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: a c limited memory FORTRAN code for solving bound constrained c optimization problems'', Tech. Report, NAM-11, EECS Department, c Northwestern University, 1994. c c (Postscript files of these papers are available via anonymous c ftp to ece.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer l1,l2,l3,lws,lr,lz,lt,ld,lwa,lwy,lsy,lss,lwt,lwn,lsnd if (task .eq. 'START') then isave(1) = m*n isave(2) = m**2 isave(3) = 4*m**2 isave(4) = 1 isave(5) = isave(4) + isave(1) isave(6) = isave(5) + isave(1) isave(7) = isave(6) + isave(2) isave(8) = isave(7) + isave(2) isave(9) = isave(8) isave(10) = isave(9) + isave(2) isave(11) = isave(10) + isave(3) isave(12) = isave(11) + isave(3) isave(13) = isave(12) + n isave(14) = isave(13) + n isave(15) = isave(14) + n isave(16) = isave(15) + n endif l1 = isave(1) l2 = isave(2) l3 = isave(3) lws = isave(4) lwy = isave(5) lsy = isave(6) lss = isave(7) lwt = isave(9) lwn = isave(10) lsnd = isave(11) lz = isave(12) lr = isave(13) ld = isave(14) lt = isave(15) lwa = isave(16) call mainlb(n,m,x,l,u,nbd,f,g,factr,pgtol, + wa(lws),wa(lwy),wa(lsy),wa(lss),wa(lwt), + wa(lwn),wa(lsnd),wa(lz),wa(lr),wa(ld),wa(lt), + wa(lwa),iwa(1),iwa(n+1),iwa(2*n+1),task,iprint, + csave,lsave,isave(22),dsave) return end c======================= The end of setulb ============================= subroutine mainlb(n, m, x, l, u, nbd, f, g, factr, pgtol, ws, wy, + sy, ss, wt, wn, snd, z, r, d, t, wa, + index, iwhere, indx2, task, iprint, + csave, lsave, isave, dsave) character*60 task, csave logical lsave(4) integer n, m, iprint, nbd(n), index(n), + iwhere(n), indx2(n), isave(23) double precision f, factr, pgtol, + x(n), l(n), u(n), g(n), z(n), r(n), d(n), t(n), + wa(8*m), ws(n, m), wy(n, m), sy(m, m), ss(m, m), + wt(m, m), wn(2*m, 2*m), snd(2*m, 2*m), dsave(29) c ************ c c Subroutine mainlb c c This subroutine solves bound constrained optimization problems by c using the compact formula of the limited memory BFGS updates. c c n is an integer variable. c On entry n is the number of variables. c On exit n is unchanged. c c m is an integer variable. c On entry m is the maximum number of variable metric c corrections allowed in the limited memory matrix. c On exit m is unchanged. c c x is a double precision array of dimension n. c On entry x is an approximation to the solution. c On exit x is the current approximation. c c l is a double precision array of dimension n. c On entry l is the lower bound of x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound of x. c On exit u is unchanged. c c nbd is an integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c f is a double precision variable. c On first entry f is unspecified. c On final exit f is the value of the function at x. c c g is a double precision array of dimension n. c On first entry g is unspecified. c On final exit g is the value of the gradient at x. c c factr is a double precision variable. c On entry factr >= 0 is specified by the user. The iteration c will stop when c c (f^k - f^{k+1})/max{|f^k|,|f^{k+1}|,1} <= factr*epsmch c c where epsmch is the machine precision, which is automatically c generated by the code. c On exit factr is unchanged. c c pgtol is a double precision variable. c On entry pgtol >= 0 is specified by the user. The iteration c will stop when c c max{|proj g_i | i = 1, ..., n} <= pgtol c c where pg_i is the ith component of the projected gradient. c On exit pgtol is unchanged. c c ws, wy, sy, and wt are double precision working arrays used to c store the following information defining the limited memory c BFGS matrix: c ws, of dimension n x m, stores S, the matrix of s-vectors; c wy, of dimension n x m, stores Y, the matrix of y-vectors; c sy, of dimension m x m, stores S'Y; c ss, of dimension m x m, stores S'S; c wt, of dimension m x m, stores the Cholesky factorization c of (theta*S'S+LD^(-1)L'); see eq. c (2.26) in [3]. c c wn is a double precision working array of dimension 2m x 2m c used to store the LEL^T factorization of the indefinite matrix c K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c c where E = [-I 0] c [ 0 I] c c snd is a double precision working array of dimension 2m x 2m c used to store the lower triangular part of c N = [Y' ZZ'Y L_a'+R_z'] c [L_a +R_z S'AA'S ] c c z(n),r(n),d(n),t(n),wa(8*m) are double precision working arrays. c z is used at different times to store the Cauchy point and c the Newton point. c c c index is an integer working array of dimension n. c In subroutine freev, index is used to store the free and fixed c variables at the Generalized Cauchy Point (GCP). c c iwhere is an integer working array of dimension n used to record c the status of the vector x for GCP computation. c iwhere(i)=0 or -3 if x(i) is free and has bounds, c 1 if x(i) is fixed at l(i), and l(i) .ne. u(i) c 2 if x(i) is fixed at u(i), and u(i) .ne. l(i) c 3 if x(i) is always fixed, i.e., u(i)=x(i)=l(i) c -1 if x(i) is always free, i.e., no bounds on it. c c indx2 is an integer working array of dimension n. c Within subroutine cauchy, indx2 corresponds to the array iorder. c In subroutine freev, a list of variables entering and leaving c the free set is stored in indx2, and it is passed on to c subroutine formk with this information. c c task is a working string of characters of length 60 indicating c the current job when entering and leaving this subroutine. c c iprint is an INTEGER variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c csave is a working string of characters of length 60. c c lsave is a logical working array of dimension 4. c c isave is an integer working array of dimension 23. c c dsave is a double precision working array of dimension 29. c c c Subprograms called c c L-BFGS-B Library ... cauchy, subsm, lnsrlb, formk, c c errclb, prn1lb, prn2lb, prn3lb, active, projgr, c c freev, cmprlb, matupd, formt. c c Minpack2 Library ... timer, dpmeps. c c Linpack Library ... dcopy, ddot. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: FORTRAN c Subroutines for Large Scale Bound Constrained Optimization'' c Tech. Report, NAM-11, EECS Department, Northwestern University, c 1994. c c [3] R. Byrd, J. Nocedal and R. Schnabel "Representations of c Quasi-Newton Matrices and their use in Limited Memory Methods'', c Mathematical Programming 63 (1994), no. 4, pp. 129-156. c c (Postscript files of these papers are available via anonymous c ftp to ece.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ logical prjctd,cnstnd,boxed,updatd,wrk character*3 word integer i,k,nintol,itfile,iback,nskip, + head,col,iter,itail,iupdat, + nint,nfgv,info,ifun, + iword,nfree,nact,ileave,nenter double precision theta,fold,ddot,dr,rr,tol,dpmeps, + xstep,sbgnrm,ddum,dnorm,dtd,epsmch, + cpu1,cpu2,cachyt,sbtime,lnscht,time1,time2, + gd,gdold,stp,stpmx,time double precision one,zero parameter (one=1.0d0,zero=0.0d0) if (task .eq. 'START') then call timer(time1) c Generate the current machine precision. epsmch = dpmeps() fold = 0.0d0 dnorm = 0.0d0 cpu1 = 0.0d0 gd = 0.0d0 sbgnrm = 0.0d0 stp = 0.0d0 stpmx = 0.0d0 gdold = 0.0d0 dtd = 0.0d0 c Initialize counters and scalars when task='START'. c for the limited memory BFGS matrices: col = 0 head = 1 theta = one iupdat = 0 updatd = .false. iback = 0 itail = 0 ifun = 0 iword = 0 nact = 0 ileave = 0 nenter = 0 c for operation counts: iter = 0 nfgv = 0 nint = 0 nintol = 0 nskip = 0 nfree = n c for stopping tolerance: tol = factr*epsmch c for measuring running time: cachyt = 0 sbtime = 0 lnscht = 0 c 'word' records the status of subspace solutions. word = '---' c 'info' records the termination information. info = 0 itfile = 0 if (iprint .ge. 1) then c open a summary file 'iterate.dat' open (8, file = 'iterate.dat', status = 'unknown') itfile = 8 endif c Check the input arguments for errors. call errclb(n,m,factr,l,u,nbd,task,info,k) if (task(1:5) .eq. 'ERROR') then call prn3lb(n,x,f,task,iprint,info,itfile, + iter,nfgv,nintol,nskip,nact,sbgnrm, + zero,nint,word,iback,stp,xstep,k, + cachyt,sbtime,lnscht) return endif call prn1lb(n,m,l,u,x,iprint,itfile,epsmch) c Initialize iwhere & project x onto the feasible set. call active(n,l,u,nbd,x,iwhere,iprint,prjctd,cnstnd,boxed) c The end of the initialization. else c restore local variables. prjctd = lsave(1) cnstnd = lsave(2) boxed = lsave(3) updatd = lsave(4) nintol = isave(1) itfile = isave(3) iback = isave(4) nskip = isave(5) head = isave(6) col = isave(7) itail = isave(8) iter = isave(9) iupdat = isave(10) nint = isave(12) nfgv = isave(13) info = isave(14) ifun = isave(15) iword = isave(16) nfree = isave(17) nact = isave(18) ileave = isave(19) nenter = isave(20) theta = dsave(1) fold = dsave(2) tol = dsave(3) dnorm = dsave(4) epsmch = dsave(5) cpu1 = dsave(6) cachyt = dsave(7) sbtime = dsave(8) lnscht = dsave(9) time1 = dsave(10) gd = dsave(11) stpmx = dsave(12) sbgnrm = dsave(13) stp = dsave(14) gdold = dsave(15) dtd = dsave(16) c After returning from the driver go to the point where execution c is to resume. if (task(1:5) .eq. 'FG_LN') goto 666 if (task(1:5) .eq. 'NEW_X') goto 777 if (task(1:5) .eq. 'FG_ST') goto 111 if (task(1:4) .eq. 'STOP') then if (task(7:9) .eq. 'CPU') then c restore the previous iterate. call dcopy(n,t,1,x,1) call dcopy(n,r,1,g,1) f = fold endif goto 999 endif endif c Compute f0 and g0. task = 'FG_START' c return to the driver to calculate f and g; reenter at 111. goto 1000 111 continue nfgv = 1 c Compute the infinity norm of the (-) projected gradient. call projgr(n,l,u,nbd,x,g,sbgnrm) if (iprint .ge. 1) then write (6,1002) iter,f,sbgnrm write (itfile,1003) iter,nfgv,sbgnrm,f endif if (sbgnrm .le. pgtol) then c terminate the algorithm. task = 'CONVERGENCE: NORM OF PROJECTED GRADIENT <= PGTOL' goto 999 endif c ----------------- the beginning of the loop -------------------------- 222 continue if (iprint .ge. 99) write (6,1001) iter + 1 iword = -1 c if (.not. cnstnd .and. col .gt. 0) then c skip the search for GCP. call dcopy(n,x,1,z,1) wrk = updatd nint = 0 goto 333 endif cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Compute the Generalized Cauchy Point (GCP). c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc call timer(cpu1) call cauchy(n,x,l,u,nbd,g,indx2,iwhere,t,d,z, + m,wy,ws,sy,wt,theta,col,head, + wa(1),wa(2*m+1),wa(4*m+1),wa(6*m+1),nint, + iprint,sbgnrm,info,epsmch) if (info .ne. 0) then c singular triangular system detected; refresh the lbfgs memory. if(iprint .ge. 1) write (6, 1005) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. call timer(cpu2) cachyt = cachyt + cpu2 - cpu1 goto 222 endif call timer(cpu2) cachyt = cachyt + cpu2 - cpu1 nintol = nintol + nint c Count the entering and leaving variables for iter > 0; c find the index set of free and active variables at the GCP. call freev(n,nfree,index,nenter,ileave,indx2, + iwhere,wrk,updatd,cnstnd,iprint,iter) nact = n - nfree 333 continue c If there are no free variables or B=theta*I, then c skip the subspace minimization. if (nfree .eq. 0 .or. col .eq. 0) goto 555 cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Subspace minimization. c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc call timer(cpu1) c Form the LEL^T factorization of the indefinite c matrix K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c where E = [-I 0] c [ 0 I] if (wrk) call formk(n,nfree,index,nenter,ileave,indx2,iupdat, + updatd,wn,snd,m,ws,wy,sy,theta,col,head,info) if (info .ne. 0) then c nonpositive definiteness in Cholesky factorization; c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1006) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. call timer(cpu2) sbtime = sbtime + cpu2 - cpu1 goto 222 endif c compute r=-Z'B(xcp-xk)-Z'g (using wa(2m+1)=W'(xcp-x) c from 'cauchy'). call cmprlb(n,m,x,g,ws,wy,sy,wt,z,r,wa,index, + theta,col,head,nfree,cnstnd,info) if (info .ne. 0) goto 444 c call the direct method. call subsm(n,m,nfree,index,l,u,nbd,z,r,ws,wy,theta, + col,head,iword,wa,wn,iprint,info) 444 continue if (info .ne. 0) then c singular triangular system detected; c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1005) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. call timer(cpu2) sbtime = sbtime + cpu2 - cpu1 goto 222 endif call timer(cpu2) sbtime = sbtime + cpu2 - cpu1 555 continue cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Line search and optimality tests. c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c Generate the search direction d:=z-x. do 40 i = 1, n d(i) = z(i) - x(i) 40 continue call timer(cpu1) 666 continue call lnsrlb(n,l,u,nbd,x,f,fold,gd,gdold,g,d,r,t,z,stp,dnorm, + dtd,xstep,stpmx,iter,ifun,iback,nfgv,info,task, + boxed,cnstnd,csave,isave(22),dsave(17)) if (info .ne. 0 .or. iback .ge. 20) then c restore the previous iterate. call dcopy(n,t,1,x,1) call dcopy(n,r,1,g,1) f = fold if (col .eq. 0) then c abnormal termination. if (info .eq. 0) then info = -9 c restore the actual number of f and g evaluations etc. nfgv = nfgv - 1 ifun = ifun - 1 iback = iback - 1 endif task = 'ABNORMAL_TERMINATION_IN_LNSRCH' iter = iter + 1 goto 999 else c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1008) if (info .eq. 0) nfgv = nfgv - 1 info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. task = 'RESTART_FROM_LNSRCH' call timer(cpu2) lnscht = lnscht + cpu2 - cpu1 goto 222 endif else if (task(1:5) .eq. 'FG_LN') then c return to the driver for calculating f and g; reenter at 666. goto 1000 else c calculate and print out the quantities related to the new X. call timer(cpu2) lnscht = lnscht + cpu2 - cpu1 iter = iter + 1 c Compute the infinity norm of the projected (-)gradient. call projgr(n,l,u,nbd,x,g,sbgnrm) c Print iteration information. call prn2lb(n,x,f,g,iprint,itfile,iter,nfgv,nact, + sbgnrm,nint,word,iword,iback,stp,xstep) goto 1000 endif 777 continue c Test for termination. if (sbgnrm .le. pgtol) then c terminate the algorithm. task = 'CONVERGENCE: NORM OF PROJECTED GRADIENT <= PGTOL' goto 999 endif ddum = max(abs(fold), abs(f), one) if ((fold - f) .le. tol*ddum) then c terminate the algorithm. task = 'CONVERGENCE: REL_REDUCTION_OF_F <= FACTR*EPSMCH' if (iback .ge. 10) info = -5 c i.e., to issue a warning if iback>10 in the line search. goto 999 endif c Compute d=newx-oldx, r=newg-oldg, rr=y'y and dr=y's. do 42 i = 1, n r(i) = g(i) - r(i) 42 continue rr = ddot(n,r,1,r,1) if (stp .eq. one) then dr = gd - gdold ddum = -gdold else dr = (gd - gdold)*stp call dscal(n,stp,d,1) ddum = -gdold*stp endif if (dr .le. epsmch*ddum) then c skip the L-BFGS update. nskip = nskip + 1 updatd = .false. if (iprint .ge. 1) write (6,1004) dr, ddum goto 888 endif cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Update the L-BFGS matrix. c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc updatd = .true. iupdat = iupdat + 1 c Update matrices WS and WY and form the middle matrix in B. call matupd(n,m,ws,wy,sy,ss,d,r,itail, + iupdat,col,head,theta,rr,dr,stp,dtd) c Form the upper half of the pds T = theta*SS + L*D^(-1)*L'; c Store T in the upper triangular of the array wt; c Cholesky factorize T to J*J' with c J' stored in the upper triangular of wt. call formt(m,wt,sy,ss,col,theta,info) if (info .ne. 0) then c nonpositive definiteness in Cholesky factorization; c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1007) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. goto 222 endif c Now the inverse of the middle matrix in B is c [ D^(1/2) O ] [ -D^(1/2) D^(-1/2)*L' ] c [ -L*D^(-1/2) J ] [ 0 J' ] 888 continue c -------------------- the end of the loop ----------------------------- goto 222 999 continue call timer(time2) time = time2 - time1 call prn3lb(n,x,f,task,iprint,info,itfile, + iter,nfgv,nintol,nskip,nact,sbgnrm, + time,nint,word,iback,stp,xstep,k, + cachyt,sbtime,lnscht) 1000 continue c Save local variables. lsave(1) = prjctd lsave(2) = cnstnd lsave(3) = boxed lsave(4) = updatd isave(1) = nintol isave(3) = itfile isave(4) = iback isave(5) = nskip isave(6) = head isave(7) = col isave(8) = itail isave(9) = iter isave(10) = iupdat isave(12) = nint isave(13) = nfgv isave(14) = info isave(15) = ifun isave(16) = iword isave(17) = nfree isave(18) = nact isave(19) = ileave isave(20) = nenter dsave(1) = theta dsave(2) = fold dsave(3) = tol dsave(4) = dnorm dsave(5) = epsmch dsave(6) = cpu1 dsave(7) = cachyt dsave(8) = sbtime dsave(9) = lnscht dsave(10) = time1 dsave(11) = gd dsave(12) = stpmx dsave(13) = sbgnrm dsave(14) = stp dsave(15) = gdold dsave(16) = dtd 1001 format (//,'ITERATION ',i5) 1002 format + (/,'At iterate',i5,4x,'f= ',1p,d12.5,4x,'|proj g|= ',1p,d12.5) 1003 format (2(1x,i4),5x,'-',5x,'-',3x,'-',5x,'-',5x,'-',8x,'-',3x, + 1p,2(1x,d10.3)) 1004 format (' ys=',1p,e10.3,' -gs=',1p,e10.3,' BFGS update SKIPPED') 1005 format (/, +' Singular triangular system detected;',/, +' refresh the lbfgs memory and restart the iteration.') 1006 format (/, +' Nonpositive definiteness in Cholesky factorization in formk;',/, +' refresh the lbfgs memory and restart the iteration.') 1007 format (/, +' Nonpositive definiteness in Cholesky factorization in formt;',/, +' refresh the lbfgs memory and restart the iteration.') 1008 format (/, +' Bad direction in the line search;',/, +' refresh the lbfgs memory and restart the iteration.') return end c======================= The end of mainlb ============================= subroutine active(n, l, u, nbd, x, iwhere, iprint, + prjctd, cnstnd, boxed) logical prjctd, cnstnd, boxed integer n, iprint, nbd(n), iwhere(n) double precision x(n), l(n), u(n) c ************ c c Subroutine active c c This subroutine initializes iwhere and projects the initial x to c the feasible set if necessary. c c iwhere is an integer array of dimension n. c On entry iwhere is unspecified. c On exit iwhere(i)=-1 if x(i) has no bounds c 3 if l(i)=u(i) c 0 otherwise. c In cauchy, iwhere is given finer gradations. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer nbdd,i double precision zero parameter (zero=0.0d0) c Initialize nbdd, prjctd, cnstnd and boxed. nbdd = 0 prjctd = .false. cnstnd = .false. boxed = .true. c Project the initial x to the easible set if necessary. do 10 i = 1, n if (nbd(i) .gt. 0) then if (nbd(i) .le. 2 .and. x(i) .le. l(i)) then if (x(i) .lt. l(i)) then prjctd = .true. x(i) = l(i) endif nbdd = nbdd + 1 else if (nbd(i) .ge. 2 .and. x(i) .ge. u(i)) then if (x(i) .gt. u(i)) then prjctd = .true. x(i) = u(i) endif nbdd = nbdd + 1 endif endif 10 continue c Initialize iwhere and assign values to cnstnd and boxed. do 20 i = 1, n if (nbd(i) .ne. 2) boxed = .false. if (nbd(i) .eq. 0) then c this variable is always free iwhere(i) = -1 c otherwise set x(i)=mid(x(i), u(i), l(i)). else cnstnd = .true. if (nbd(i) .eq. 2 .and. u(i) - l(i) .le. zero) then c this variable is always fixed iwhere(i) = 3 else iwhere(i) = 0 endif endif 20 continue if (iprint .ge. 0) then if (prjctd) write (6,*) + 'The initial X is infeasible. Restart with its projection.' if (.not. cnstnd) + write (6,*) 'This problem is unconstrained.' endif if (iprint .gt. 0) write (6,1001) nbdd 1001 format (/,'At X0 ',i9,' variables are exactly at the bounds') return end c======================= The end of active ============================= subroutine bmv(m, sy, wt, col, v, p, info) integer m, col, info double precision sy(m, m), wt(m, m), v(2*col), p(2*col) c ************ c c Subroutine bmv c c This subroutine computes the product of the 2m x 2m middle matrix c in the compact L-BFGS formula of B and a 2m vector v; c it returns the product in p. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c sy is a double precision array of dimension m x m. c On entry sy specifies the matrix S'Y. c On exit sy is unchanged. c c wt is a double precision array of dimension m x m. c On entry wt specifies the upper triangular matrix J' which is c the Cholesky factor of (thetaS'S+LD^(-1)L'). c On exit wt is unchanged. c c col is an integer variable. c On entry col specifies the number of s-vectors (or y-vectors) c stored in the compact L-BFGS formula. c On exit col is unchanged. c c v is a double precision array of dimension 2col. c On entry v specifies vector v. c On exit v is unchanged. c c p is a double precision array of dimension 2col. c On entry p is unspecified. c On exit p is the product Mv. c c info is an integer variable. c On entry info is unspecified. c On exit info = 0 for normal return, c = nonzero for abnormal return when the system c to be solved by dtrsl is singular. c c Subprograms called: c c Linpack ... dtrsl. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,k,i2 double precision sum if (col .eq. 0) return c PART I: solve [ D^(1/2) O ] [ p1 ] = [ v1 ] c [ -L*D^(-1/2) J ] [ p2 ] [ v2 ]. c solve Jp2=v2+LD^(-1)v1. p(col + 1) = v(col + 1) do 20 i = 2, col i2 = col + i sum = 0.0d0 do 10 k = 1, i - 1 sum = sum + sy(i,k)*v(k)/sy(k,k) 10 continue p(i2) = v(i2) + sum 20 continue c Solve the triangular system call dtrsl(wt,m,col,p(col+1),11,info) if (info .ne. 0) return c solve D^(1/2)p1=v1. cc do 30 i = 1, col cc p(i) = v(i)/sqrt(sy(i,i)) cc30 continue c PART II: solve [ -D^(1/2) D^(-1/2)*L' ] [ p1 ] = [ p1 ] c [ 0 J' ] [ p2 ] [ p2 ]. c solve J^Tp2=p2. call dtrsl(wt,m,col,p(col+1),01,info) if (info .ne. 0) return c compute p1=-D^(-1/2)(p1-D^(-1/2)L'p2) c =-D^(-1/2)p1+D^(-1)L'p2. do 40 i = 1, col cc p(i) = -p(i)/sqrt(sy(i,i)) combined with do 30 loop cc into the next line p(i) = -v(i)/sy(i,i) 40 continue do 60 i = 1, col sum = 0.d0 do 50 k = i + 1, col sum = sum + sy(k,i)*p(col+k)/sy(i,i) 50 continue p(i) = p(i) + sum 60 continue return end c======================== The end of bmv =============================== subroutine cauchy(n, x, l, u, nbd, g, iorder, iwhere, t, d, xcp, + m, wy, ws, sy, wt, theta, col, head, p, c, wbp, + v, nint, iprint, sbgnrm, info, epsmch) integer n, m, head, col, nint, iprint, info, + nbd(n), iorder(n), iwhere(n) double precision theta, epsmch, + x(n), l(n), u(n), g(n), t(n), d(n), xcp(n), + wy(n, col), ws(n, col), sy(m, m), + wt(m, m), p(2*m), c(2*m), wbp(2*m), v(2*m) c ************ c c Subroutine cauchy c c For given x, l, u, g (with sbgnrm > 0), and a limited memory c BFGS matrix B defined in terms of matrices WY, WS, WT, and c scalars head, col, and theta, this subroutine computes the c generalized Cauchy point (GCP), defined as the first local c minimizer of the quadratic c c Q(x + s) = g's + 1/2 s'Bs c c along the projected gradient direction P(x-tg,l,u). c The routine returns the GCP in xcp. c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c x is a double precision array of dimension n. c On entry x is the starting point for the GCP computation. c On exit x is unchanged. c c l is a double precision array of dimension n. c On entry l is the lower bound of x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound of x. c On exit u is unchanged. c c nbd is an integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, and c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c g is a double precision array of dimension n. c On entry g is the gradient of f(x). g must be a nonzero vector. c On exit g is unchanged. c c iorder is an integer working array of dimension n. c iorder will be used to store the breakpoints in the piecewise c linear path and free variables encountered. On exit, c iorder(1),...,iorder(nleft) are indices of breakpoints c which have not been encountered; c iorder(nleft+1),...,iorder(nbreak) are indices of c encountered breakpoints; and c iorder(nfree),...,iorder(n) are indices of variables which c have no bound constraits along the search direction. c c iwhere is an integer array of dimension n. c On entry iwhere indicates only the permanently fixed (iwhere=3) c or free (iwhere= -1) components of x. c On exit iwhere records the status of the current x variables. c iwhere(i)=-3 if x(i) is free and has bounds, but is not moved c 0 if x(i) is free and has bounds, and is moved c 1 if x(i) is fixed at l(i), and l(i) .ne. u(i) c 2 if x(i) is fixed at u(i), and u(i) .ne. l(i) c 3 if x(i) is always fixed, i.e., u(i)=x(i)=l(i) c -1 if x(i) is always free, i.e., it has no bounds. c c t is a double precision working array of dimension n. c t will be used to store the break points. c c d is a double precision array of dimension n used to store c the Cauchy direction P(x-tg)-x. c c xcp is a double precision array of dimension n used to return the c GCP on exit. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c ws, wy, sy, and wt are double precision arrays. c On entry they store information that defines the c limited memory BFGS matrix: c ws(n,m) stores S, a set of s-vectors; c wy(n,m) stores Y, a set of y-vectors; c sy(m,m) stores S'Y; c wt(m,m) stores the c Cholesky factorization of (theta*S'S+LD^(-1)L'). c On exit these arrays are unchanged. c c theta is a double precision variable. c On entry theta is the scaling factor specifying B_0 = theta I. c On exit theta is unchanged. c c col is an integer variable. c On entry col is the actual number of variable metric c corrections stored so far. c On exit col is unchanged. c c head is an integer variable. c On entry head is the location of the first s-vector c (or y-vector) in S (or Y). c On exit col is unchanged. c c p is a double precision working array of dimension 2m. c p will be used to store the vector p = W^(T)d. c c c is a double precision working array of dimension 2m. c c will be used to store the vector c = W^(T)(xcp-x). c c wbp is a double precision working array of dimension 2m. c wbp will be used to store the row of W corresponding c to a breakpoint. c c v is a double precision working array of dimension 2m. c c nint is an integer variable. c On exit nint records the number of quadratic segments explored c in searching for the GCP. c c iprint is an INTEGER variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c sbgnrm is a double precision variable. c On entry sbgnrm is the norm of the projected gradient at x. c On exit sbgnrm is unchanged. c c info is an integer variable. c On entry info is 0. c On exit info = 0 for normal return, c = nonzero for abnormal return when the the system c used in routine bmv is singular. c c Subprograms called: c c L-BFGS-B Library ... hpsolb, bmv. c c Linpack ... dscal dcopy, daxpy. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: FORTRAN c Subroutines for Large Scale Bound Constrained Optimization'' c Tech. Report, NAM-11, EECS Department, Northwestern University, c 1994. c c (Postscript files of these papers are available via anonymous c ftp to ece.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ logical xlower,xupper,bnded integer i,j,col2,nfree,nbreak,pointr, + ibp,nleft,ibkmin,iter double precision f1,f2,dt,dtm,tsum,dibp,zibp,dibp2,bkmin, + tu,tl,wmc,wmp,wmw,ddot,tj,tj0,neggi,sbgnrm, + f2_org double precision one,zero parameter (one=1.0d0,zero=0.0d0) c Check the status of the variables, reset iwhere(i) if necessary; c compute the Cauchy direction d and the breakpoints t; initialize c the derivative f1 and the vector p = W'd (for theta = 1). if (sbgnrm .le. zero) then if (iprint .ge. 0) write (6,*) 'Subgnorm = 0. GCP = X.' call dcopy(n,x,1,xcp,1) return endif bnded = .true. nfree = n + 1 nbreak = 0 ibkmin = 0 bkmin = zero col2 = 2*col f1 = zero if (iprint .ge. 99) write (6,3010) c We set p to zero and build it up as we determine d. do 20 i = 1, col2 p(i) = zero 20 continue c In the following loop we determine for each variable its bound c status and its breakpoint, and update p accordingly. c Smallest breakpoint is identified. do 50 i = 1, n neggi = -g(i) if (iwhere(i) .ne. 3 .and. iwhere(i) .ne. -1) then c if x(i) is not a constant and has bounds, c compute the difference between x(i) and its bounds. if (nbd(i) .le. 2) tl = x(i) - l(i) if (nbd(i) .ge. 2) tu = u(i) - x(i) c If a variable is close enough to a bound c we treat it as at bound. xlower = nbd(i) .le. 2 .and. tl .le. zero xupper = nbd(i) .ge. 2 .and. tu .le. zero c reset iwhere(i). iwhere(i) = 0 if (xlower) then if (neggi .le. zero) iwhere(i) = 1 else if (xupper) then if (neggi .ge. zero) iwhere(i) = 2 else if (abs(neggi) .le. zero) iwhere(i) = -3 endif endif pointr = head if (iwhere(i) .ne. 0 .and. iwhere(i) .ne. -1) then d(i) = zero else d(i) = neggi f1 = f1 - neggi*neggi c calculate p := p - W'e_i* (g_i). do 40 j = 1, col p(j) = p(j) + wy(i,pointr)* neggi p(col + j) = p(col + j) + ws(i,pointr)*neggi pointr = mod(pointr,m) + 1 40 continue if (nbd(i) .le. 2 .and. nbd(i) .ne. 0 + .and. neggi .lt. zero) then c x(i) + d(i) is bounded; compute t(i). nbreak = nbreak + 1 iorder(nbreak) = i t(nbreak) = tl/(-neggi) if (nbreak .eq. 1 .or. t(nbreak) .lt. bkmin) then bkmin = t(nbreak) ibkmin = nbreak endif else if (nbd(i) .ge. 2 .and. neggi .gt. zero) then c x(i) + d(i) is bounded; compute t(i). nbreak = nbreak + 1 iorder(nbreak) = i t(nbreak) = tu/neggi if (nbreak .eq. 1 .or. t(nbreak) .lt. bkmin) then bkmin = t(nbreak) ibkmin = nbreak endif else c x(i) + d(i) is not bounded. nfree = nfree - 1 iorder(nfree) = i if (abs(neggi) .gt. zero) bnded = .false. endif endif 50 continue c The indices of the nonzero components of d are now stored c in iorder(1),...,iorder(nbreak) and iorder(nfree),...,iorder(n). c The smallest of the nbreak breakpoints is in t(ibkmin)=bkmin. if (theta .ne. one) then c complete the initialization of p for theta not= one. call dscal(col,theta,p(col+1),1) endif c Initialize GCP xcp = x. call dcopy(n,x,1,xcp,1) if (nbreak .eq. 0 .and. nfree .eq. n + 1) then c is a zero vector, return with the initial xcp as GCP. if (iprint .gt. 100) write (6,1010) (xcp(i), i = 1, n) return endif c Initialize c = W'(xcp - x) = 0. do 60 j = 1, col2 c(j) = zero 60 continue c Initialize derivative f2. f2 = -theta*f1 f2_org = f2 if (col .gt. 0) then call bmv(m,sy,wt,col,p,v,info) if (info .ne. 0) return f2 = f2 - ddot(col2,v,1,p,1) endif dtm = -f1/f2 tsum = zero nint = 1 if (iprint .ge. 99) + write (6,*) 'There are ',nbreak,' breakpoints ' c If there are no breakpoints, locate the GCP and return. if (nbreak .eq. 0) goto 888 nleft = nbreak iter = 1 tj = zero c------------------- the beginning of the loop ------------------------- 777 continue c Find the next smallest breakpoint; c compute dt = t(nleft) - t(nleft + 1). tj0 = tj if (iter .eq. 1) then c Since we already have the smallest breakpoint we need not do c heapsort yet. Often only one breakpoint is used and the c cost of heapsort is avoided. tj = bkmin ibp = iorder(ibkmin) else if (iter .eq. 2) then c Replace the already used smallest breakpoint with the c breakpoint numbered nbreak > nlast, before heapsort call. if (ibkmin .ne. nbreak) then t(ibkmin) = t(nbreak) iorder(ibkmin) = iorder(nbreak) endif c Update heap structure of breakpoints c (if iter=2, initialize heap). endif call hpsolb(nleft,t,iorder,iter-2) tj = t(nleft) ibp = iorder(nleft) endif dt = tj - tj0 if (dt .ne. zero .and. iprint .ge. 100) then write (6,4011) nint,f1,f2 write (6,5010) dt write (6,6010) dtm endif c If a minimizer is within this interval, c locate the GCP and return. if (dtm .lt. dt) goto 888 c Otherwise fix one variable and c reset the corresponding component of d to zero. tsum = tsum + dt nleft = nleft - 1 iter = iter + 1 dibp = d(ibp) d(ibp) = zero if (dibp .gt. zero) then zibp = u(ibp) - x(ibp) xcp(ibp) = u(ibp) iwhere(ibp) = 2 else zibp = l(ibp) - x(ibp) xcp(ibp) = l(ibp) iwhere(ibp) = 1 endif if (iprint .ge. 100) write (6,*) 'Variable ',ibp,' is fixed.' if (nleft .eq. 0 .and. nbreak .eq. n) then c all n variables are fixed, c return with xcp as GCP. dtm = dt goto 999 endif c Update the derivative information. nint = nint + 1 dibp2 = dibp**2 c Update f1 and f2. c temporarily set f1 and f2 for col=0. f1 = f1 + dt*f2 + dibp2 - theta*dibp*zibp f2 = f2 - theta*dibp2 if (col .gt. 0) then c update c = c + dt*p. call daxpy(col2,dt,p,1,c,1) c choose wbp, c the row of W corresponding to the breakpoint encountered. pointr = head do 70 j = 1,col wbp(j) = wy(ibp,pointr) wbp(col + j) = theta*ws(ibp,pointr) pointr = mod(pointr,m) + 1 70 continue c compute (wbp)Mc, (wbp)Mp, and (wbp)M(wbp)'. call bmv(m,sy,wt,col,wbp,v,info) if (info .ne. 0) return wmc = ddot(col2,c,1,v,1) wmp = ddot(col2,p,1,v,1) wmw = ddot(col2,wbp,1,v,1) c update p = p - dibp*wbp. call daxpy(col2,-dibp,wbp,1,p,1) c complete updating f1 and f2 while col > 0. f1 = f1 + dibp*wmc f2 = f2 + 2.0d0*dibp*wmp - dibp2*wmw endif f2 = max(epsmch*f2_org,f2) if (nleft .gt. 0) then dtm = -f1/f2 goto 777 c to repeat the loop for unsearched intervals. else if(bnded) then f1 = zero f2 = zero dtm = zero else dtm = -f1/f2 endif c------------------- the end of the loop ------------------------------- 888 continue if (iprint .ge. 99) then write (6,*) write (6,*) 'GCP found in this segment' write (6,4010) nint,f1,f2 write (6,6010) dtm endif if (dtm .le. zero) dtm = zero tsum = tsum + dtm c Move free variables (i.e., the ones w/o breakpoints) and c the variables whose breakpoints haven't been reached. call daxpy(n,tsum,d,1,xcp,1) 999 continue c Update c = c + dtm*p = W'(x^c - x) c which will be used in computing r = Z'(B(x^c - x) + g). if (col .gt. 0) call daxpy(col2,dtm,p,1,c,1) if (iprint .gt. 100) write (6,1010) (xcp(i),i = 1,n) if (iprint .ge. 99) write (6,2010) 1010 format ('Cauchy X = ',/,(4x,1p,6(1x,d11.4))) 2010 format (/,'---------------- exit CAUCHY----------------------',/) 3010 format (/,'---------------- CAUCHY entered-------------------') 4010 format ('Piece ',i3,' --f1, f2 at start point ',1p,2(1x,d11.4)) 4011 format (/,'Piece ',i3,' --f1, f2 at start point ', + 1p,2(1x,d11.4)) 5010 format ('Distance to the next break point = ',1p,d11.4) 6010 format ('Distance to the stationary point = ',1p,d11.4) return end c====================== The end of cauchy ============================== subroutine cmprlb(n, m, x, g, ws, wy, sy, wt, z, r, wa, index, + theta, col, head, nfree, cnstnd, info) logical cnstnd integer n, m, col, head, nfree, info, index(n) double precision theta, + x(n), g(n), z(n), r(n), wa(4*m), + ws(n, m), wy(n, m), sy(m, m), wt(m, m) c ************ c c Subroutine cmprlb c c This subroutine computes r=-Z'B(xcp-xk)-Z'g by using c wa(2m+1)=W'(xcp-x) from subroutine cauchy. c c Subprograms called: c c L-BFGS-B Library ... bmv. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,j,k,pointr double precision a1,a2 if (.not. cnstnd .and. col .gt. 0) then do 26 i = 1, n r(i) = -g(i) 26 continue else do 30 i = 1, nfree k = index(i) r(i) = -theta*(z(k) - x(k)) - g(k) 30 continue call bmv(m,sy,wt,col,wa(2*m+1),wa(1),info) if (info .ne. 0) then info = -8 return endif pointr = head do 34 j = 1, col a1 = wa(j) a2 = theta*wa(col + j) do 32 i = 1, nfree k = index(i) r(i) = r(i) + wy(k,pointr)*a1 + ws(k,pointr)*a2 32 continue pointr = mod(pointr,m) + 1 34 continue endif return end c======================= The end of cmprlb ============================= subroutine errclb(n, m, factr, l, u, nbd, task, info, k) character*60 task integer n, m, info, k, nbd(n) double precision factr, l(n), u(n) c ************ c c Subroutine errclb c c This subroutine checks the validity of the input data. c c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i double precision zero parameter (zero=0.0d0) c Check the input arguments for errors. if (n .le. 0) task = 'ERROR: N .LE. 0' if (m .le. 0) task = 'ERROR: M .LE. 0' if (factr .lt. zero) task = 'ERROR: FACTR .LT. 0' c Check the validity of the arrays nbd(i), u(i), and l(i). do 10 i = 1, n if (nbd(i) .lt. 0 .or. nbd(i) .gt. 3) then c return task = 'ERROR: INVALID NBD' info = -6 k = i endif if (nbd(i) .eq. 2) then if (l(i) .gt. u(i)) then c return task = 'ERROR: NO FEASIBLE SOLUTION' info = -7 k = i endif endif 10 continue return end c======================= The end of errclb ============================= subroutine formk(n, nsub, ind, nenter, ileave, indx2, iupdat, + updatd, wn, wn1, m, ws, wy, sy, theta, col, + head, info) integer n, nsub, m, col, head, nenter, ileave, iupdat, + info, ind(n), indx2(n) double precision theta, wn(2*m, 2*m), wn1(2*m, 2*m), + ws(n, m), wy(n, m), sy(m, m) logical updatd c ************ c c Subroutine formk c c This subroutine forms the LEL^T factorization of the indefinite c c matrix K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c where E = [-I 0] c [ 0 I] c The matrix K can be shown to be equal to the matrix M^[-1]N c occurring in section 5.1 of [1], as well as to the matrix c Mbar^[-1] Nbar in section 5.3. c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c nsub is an integer variable c On entry nsub is the number of subspace variables in free set. c On exit nsub is not changed. c c ind is an integer array of dimension nsub. c On entry ind specifies the indices of subspace variables. c On exit ind is unchanged. c c nenter is an integer variable. c On entry nenter is the number of variables entering the c free set. c On exit nenter is unchanged. c c ileave is an integer variable. c On entry indx2(ileave),...,indx2(n) are the variables leaving c the free set. c On exit ileave is unchanged. c c indx2 is an integer array of dimension n. c On entry indx2(1),...,indx2(nenter) are the variables entering c the free set, while indx2(ileave),...,indx2(n) are the c variables leaving the free set. c On exit indx2 is unchanged. c c iupdat is an integer variable. c On entry iupdat is the total number of BFGS updates made so far. c On exit iupdat is unchanged. c c updatd is a logical variable. c On entry 'updatd' is true if the L-BFGS matrix is updatd. c On exit 'updatd' is unchanged. c c wn is a double precision array of dimension 2m x 2m. c On entry wn is unspecified. c On exit the upper triangle of wn stores the LEL^T factorization c of the 2*col x 2*col indefinite matrix c [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c c wn1 is a double precision array of dimension 2m x 2m. c On entry wn1 stores the lower triangular part of c [Y' ZZ'Y L_a'+R_z'] c [L_a+R_z S'AA'S ] c in the previous iteration. c On exit wn1 stores the corresponding updated matrices. c The purpose of wn1 is just to store these inner products c so they can be easily updated and inserted into wn. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c ws, wy, sy, and wtyy are double precision arrays; c theta is a double precision variable; c col is an integer variable; c head is an integer variable. c On entry they store the information defining the c limited memory BFGS matrix: c ws(n,m) stores S, a set of s-vectors; c wy(n,m) stores Y, a set of y-vectors; c sy(m,m) stores S'Y; c wtyy(m,m) stores the Cholesky factorization c of (theta*S'S+LD^(-1)L') c theta is the scaling factor specifying B_0 = theta I; c col is the number of variable metric corrections stored; c head is the location of the 1st s- (or y-) vector in S (or Y). c On exit they are unchanged. c c info is an integer variable. c On entry info is unspecified. c On exit info = 0 for normal return; c = -1 when the 1st Cholesky factorization failed; c = -2 when the 2st Cholesky factorization failed. c c Subprograms called: c c Linpack ... dcopy, dpofa, dtrsl. c c c References: c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: a c limited memory FORTRAN code for solving bound constrained c optimization problems'', Tech. Report, NAM-11, EECS Department, c Northwestern University, 1994. c c (Postscript files of these papers are available via anonymous c ftp to ece.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer m2,ipntr,jpntr,iy,is,jy,js,is1,js1,k1,i,k, + col2,pbegin,pend,dbegin,dend,upcl double precision ddot,temp1,temp2,temp3,temp4 double precision zero parameter (zero=0.0d0) c Form the lower triangular part of c WN1 = [Y' ZZ'Y L_a'+R_z'] c [L_a+R_z S'AA'S ] c where L_a is the strictly lower triangular part of S'AA'Y c R_z is the upper triangular part of S'ZZ'Y. if (updatd) then if (iupdat .gt. m) then c shift old part of WN1. do 10 jy = 1, m - 1 js = m + jy call dcopy(m-jy,wn1(jy+1,jy+1),1,wn1(jy,jy),1) call dcopy(m-jy,wn1(js+1,js+1),1,wn1(js,js),1) call dcopy(m-1,wn1(m+2,jy+1),1,wn1(m+1,jy),1) 10 continue endif c put new rows in blocks (1,1), (2,1) and (2,2). pbegin = 1 pend = nsub dbegin = nsub + 1 dend = n iy = col is = m + col ipntr = head + col - 1 if (ipntr .gt. m) ipntr = ipntr - m jpntr = head do 20 jy = 1, col js = m + jy temp1 = zero temp2 = zero temp3 = zero c compute element jy of row 'col' of Y'ZZ'Y do 15 k = pbegin, pend k1 = ind(k) temp1 = temp1 + wy(k1,ipntr)*wy(k1,jpntr) 15 continue c compute elements jy of row 'col' of L_a and S'AA'S do 16 k = dbegin, dend k1 = ind(k) temp2 = temp2 + ws(k1,ipntr)*ws(k1,jpntr) temp3 = temp3 + ws(k1,ipntr)*wy(k1,jpntr) 16 continue wn1(iy,jy) = temp1 wn1(is,js) = temp2 wn1(is,jy) = temp3 jpntr = mod(jpntr,m) + 1 20 continue c put new column in block (2,1). jy = col jpntr = head + col - 1 if (jpntr .gt. m) jpntr = jpntr - m ipntr = head do 30 i = 1, col is = m + i temp3 = zero c compute element i of column 'col' of R_z do 25 k = pbegin, pend k1 = ind(k) temp3 = temp3 + ws(k1,ipntr)*wy(k1,jpntr) 25 continue ipntr = mod(ipntr,m) + 1 wn1(is,jy) = temp3 30 continue upcl = col - 1 else upcl = col endif c modify the old parts in blocks (1,1) and (2,2) due to changes c in the set of free variables. ipntr = head do 45 iy = 1, upcl is = m + iy jpntr = head do 40 jy = 1, iy js = m + jy temp1 = zero temp2 = zero temp3 = zero temp4 = zero do 35 k = 1, nenter k1 = indx2(k) temp1 = temp1 + wy(k1,ipntr)*wy(k1,jpntr) temp2 = temp2 + ws(k1,ipntr)*ws(k1,jpntr) 35 continue do 36 k = ileave, n k1 = indx2(k) temp3 = temp3 + wy(k1,ipntr)*wy(k1,jpntr) temp4 = temp4 + ws(k1,ipntr)*ws(k1,jpntr) 36 continue wn1(iy,jy) = wn1(iy,jy) + temp1 - temp3 wn1(is,js) = wn1(is,js) - temp2 + temp4 jpntr = mod(jpntr,m) + 1 40 continue ipntr = mod(ipntr,m) + 1 45 continue c modify the old parts in block (2,1). ipntr = head do 60 is = m + 1, m + upcl jpntr = head do 55 jy = 1, upcl temp1 = zero temp3 = zero do 50 k = 1, nenter k1 = indx2(k) temp1 = temp1 + ws(k1,ipntr)*wy(k1,jpntr) 50 continue do 51 k = ileave, n k1 = indx2(k) temp3 = temp3 + ws(k1,ipntr)*wy(k1,jpntr) 51 continue if (is .le. jy + m) then wn1(is,jy) = wn1(is,jy) + temp1 - temp3 else wn1(is,jy) = wn1(is,jy) - temp1 + temp3 endif jpntr = mod(jpntr,m) + 1 55 continue ipntr = mod(ipntr,m) + 1 60 continue c Form the upper triangle of WN = [D+Y' ZZ'Y/theta -L_a'+R_z' ] c [-L_a +R_z S'AA'S*theta] m2 = 2*m do 70 iy = 1, col is = col + iy is1 = m + iy do 65 jy = 1, iy js = col + jy js1 = m + jy wn(jy,iy) = wn1(iy,jy)/theta wn(js,is) = wn1(is1,js1)*theta 65 continue do 66 jy = 1, iy - 1 wn(jy,is) = -wn1(is1,jy) 66 continue do 67 jy = iy, col wn(jy,is) = wn1(is1,jy) 67 continue wn(iy,iy) = wn(iy,iy) + sy(iy,iy) 70 continue c Form the upper triangle of c WN= [ LL' L^-1(-L_a'+R_z')] c [(-L_a +R_z)L'^-1 S'AA'S*theta ] c first Cholesky factor (1,1) block of wn to get LL' c with L' stored in the upper triangle of wn. call dpofa(wn,m2,col,info) if (info .ne. 0) then info = -1 return endif c then form L^-1(-L_a'+R_z') in the (1,2) block. col2 = 2*col do 71 js = col+1 ,col2 call dtrsl(wn,m2,col,wn(1,js),11,info) 71 continue c Form S'AA'S*theta + (L^-1(-L_a'+R_z'))'L^-1(-L_a'+R_z') in the c upper triangle of (2,2) block of wn. do 72 is = col+1, col2 do 74 js = is, col2 wn(is,js) = wn(is,js) + ddot(col,wn(1,is),1,wn(1,js),1) 74 continue 72 continue c Cholesky factorization of (2,2) block of wn. call dpofa(wn(col+1,col+1),m2,col,info) if (info .ne. 0) then info = -2 return endif return end c======================= The end of formk ============================== subroutine formt(m, wt, sy, ss, col, theta, info) integer m, col, info double precision theta, wt(m, m), sy(m, m), ss(m, m) c ************ c c Subroutine formt c c This subroutine forms the upper half of the pos. def. and symm. c T = theta*SS + L*D^(-1)*L', stores T in the upper triangle c of the array wt, and performs the Cholesky factorization of T c to produce J*J', with J' stored in the upper triangle of wt. c c Subprograms called: c c Linpack ... dpofa. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,j,k,k1 double precision ddum double precision zero parameter (zero=0.0d0) c Form the upper half of T = theta*SS + L*D^(-1)*L', c store T in the upper triangle of the array wt. do 52 j = 1, col wt(1,j) = theta*ss(1,j) 52 continue do 55 i = 2, col do 54 j = i, col k1 = min(i,j) - 1 ddum = zero do 53 k = 1, k1 ddum = ddum + sy(i,k)*sy(j,k)/sy(k,k) 53 continue wt(i,j) = ddum + theta*ss(i,j) 54 continue 55 continue c Cholesky factorize T to J*J' with c J' stored in the upper triangle of wt. call dpofa(wt,m,col,info) if (info .ne. 0) then info = -3 endif return end c======================= The end of formt ============================== subroutine freev(n, nfree, index, nenter, ileave, indx2, + iwhere, wrk, updatd, cnstnd, iprint, iter) integer n, nfree, nenter, ileave, iprint, iter, + index(n), indx2(n), iwhere(n) logical wrk, updatd, cnstnd c ************ c c Subroutine freev c c This subroutine counts the entering and leaving variables when c iter > 0, and finds the index set of free and active variables c at the GCP. c c cnstnd is a logical variable indicating whether bounds are present c c index is an integer array of dimension n c for i=1,...,nfree, index(i) are the indices of free variables c for i=nfree+1,...,n, index(i) are the indices of bound variables c On entry after the first iteration, index gives c the free variables at the previous iteration. c On exit it gives the free variables based on the determination c in cauchy using the array iwhere. c c indx2 is an integer array of dimension n c On entry indx2 is unspecified. c On exit with iter>0, indx2 indicates which variables c have changed status since the previous iteration. c For i= 1,...,nenter, indx2(i) have changed from bound to free. c For i= ileave+1,...,n, indx2(i) have changed from free to bound. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer iact,i,k nenter = 0 ileave = n + 1 if (iter .gt. 0 .and. cnstnd) then c count the entering and leaving variables. do 20 i = 1, nfree k = index(i) if (iwhere(k) .gt. 0) then ileave = ileave - 1 indx2(ileave) = k if (iprint .ge. 100) write (6,*) + 'Variable ',k,' leaves the set of free variables' endif 20 continue do 22 i = 1 + nfree, n k = index(i) if (iwhere(k) .le. 0) then nenter = nenter + 1 indx2(nenter) = k if (iprint .ge. 100) write (6,*) + 'Variable ',k,' enters the set of free variables' endif 22 continue if (iprint .ge. 99) write (6,*) + n+1-ileave,' variables leave; ',nenter,' variables enter' endif wrk = (ileave .lt. n+1) .or. (nenter .gt. 0) .or. updatd c Find the index set of free and active variables at the GCP. nfree = 0 iact = n + 1 do 24 i = 1, n if (iwhere(i) .le. 0) then nfree = nfree + 1 index(nfree) = i else iact = iact - 1 index(iact) = i endif 24 continue if (iprint .ge. 99) write (6,*) + nfree,' variables are free at GCP ',iter + 1 return end c======================= The end of freev ============================== subroutine hpsolb(n, t, iorder, iheap) integer iheap, n, iorder(n) double precision t(n) c ************ c c Subroutine hpsolb c c This subroutine sorts out the least element of t, and puts the c remaining elements of t in a heap. c c n is an integer variable. c On entry n is the dimension of the arrays t and iorder. c On exit n is unchanged. c c t is a double precision array of dimension n. c On entry t stores the elements to be sorted, c On exit t(n) stores the least elements of t, and t(1) to t(n-1) c stores the remaining elements in the form of a heap. c c iorder is an integer array of dimension n. c On entry iorder(i) is the index of t(i). c On exit iorder(i) is still the index of t(i), but iorder may be c permuted in accordance with t. c c iheap is an integer variable specifying the task. c On entry iheap should be set as follows: c iheap .eq. 0 if t(1) to t(n) is not in the form of a heap, c iheap .ne. 0 if otherwise. c On exit iheap is unchanged. c c c References: c Algorithm 232 of CACM (J. W. J. Williams): HEAPSORT. c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c ************ integer i,j,k,indxin,indxou double precision ddum,out if (iheap .eq. 0) then c Rearrange the elements t(1) to t(n) to form a heap. do 20 k = 2, n ddum = t(k) indxin = iorder(k) c Add ddum to the heap. i = k 10 continue if (i.gt.1) then j = i/2 if (ddum .lt. t(j)) then t(i) = t(j) iorder(i) = iorder(j) i = j goto 10 endif endif t(i) = ddum iorder(i) = indxin 20 continue endif c Assign to 'out' the value of t(1), the least member of the heap, c and rearrange the remaining members to form a heap as c elements 1 to n-1 of t. if (n .gt. 1) then i = 1 out = t(1) indxou = iorder(1) ddum = t(n) indxin = iorder(n) c Restore the heap 30 continue j = i+i if (j .le. n-1) then if (t(j+1) .lt. t(j)) j = j+1 if (t(j) .lt. ddum ) then t(i) = t(j) iorder(i) = iorder(j) i = j goto 30 endif endif t(i) = ddum iorder(i) = indxin c Put the least member in t(n). t(n) = out iorder(n) = indxou endif return end c====================== The end of hpsolb ============================== subroutine lnsrlb(n, l, u, nbd, x, f, fold, gd, gdold, g, d, r, t, + z, stp, dnorm, dtd, xstep, stpmx, iter, ifun, + iback, nfgv, info, task, boxed, cnstnd, csave, + isave, dsave) character*60 task, csave logical boxed, cnstnd integer n, iter, ifun, iback, nfgv, info, + nbd(n), isave(2) double precision f, fold, gd, gdold, stp, dnorm, dtd, xstep, + stpmx, x(n), l(n), u(n), g(n), d(n), r(n), t(n), + z(n), dsave(13) c ********** c c Subroutine lnsrlb c c This subroutine calls subroutine dcsrch from the Minpack2 library c to perform the line search. Subroutine dscrch is safeguarded so c that all trial points lie within the feasible region. c c Subprograms called: c c Minpack2 Library ... dcsrch. c c Linpack ... dtrsl, ddot. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ********** integer i double precision ddot,a1,a2 double precision one,zero,big parameter (one=1.0d0,zero=0.0d0,big=1.0d+10) double precision ftol,gtol,xtol parameter (ftol=1.0d-3,gtol=0.9d0,xtol=0.1d0) if (task(1:5) .eq. 'FG_LN') goto 556 dtd = ddot(n,d,1,d,1) dnorm = sqrt(dtd) c Determine the maximum step length. stpmx = big if (cnstnd) then if (iter .eq. 0) then stpmx = one else do 43 i = 1, n a1 = d(i) if (nbd(i) .ne. 0) then if (a1 .lt. zero .and. nbd(i) .le. 2) then a2 = l(i) - x(i) if (a2 .ge. zero) then stpmx = zero else if (a1*stpmx .lt. a2) then stpmx = a2/a1 endif else if (a1 .gt. zero .and. nbd(i) .ge. 2) then a2 = u(i) - x(i) if (a2 .le. zero) then stpmx = zero else if (a1*stpmx .gt. a2) then stpmx = a2/a1 endif endif endif 43 continue endif endif if (iter .eq. 0 .and. .not. boxed) then stp = min(one/dnorm, stpmx) else stp = one endif call dcopy(n,x,1,t,1) call dcopy(n,g,1,r,1) fold = f ifun = 0 iback = 0 csave = 'START' 556 continue gd = ddot(n,g,1,d,1) if (ifun .eq. 0) then gdold=gd if (gd .ge. zero) then c the directional derivative >=0. c Line search is impossible. info = -4 return endif endif call dcsrch(f,gd,stp,ftol,gtol,xtol,zero,stpmx,csave,isave,dsave) xstep = stp*dnorm if (csave(1:4) .ne. 'CONV' .and. csave(1:4) .ne. 'WARN') then task = 'FG_LNSRCH' ifun = ifun + 1 nfgv = nfgv + 1 iback = ifun - 1 if (stp .eq. one) then call dcopy(n,z,1,x,1) else do 41 i = 1, n x(i) = stp*d(i) + t(i) 41 continue endif else task = 'NEW_X' endif return end c======================= The end of lnsrlb ============================= subroutine matupd(n, m, ws, wy, sy, ss, d, r, itail, + iupdat, col, head, theta, rr, dr, stp, dtd) integer n, m, itail, iupdat, col, head double precision theta, rr, dr, stp, dtd, d(n), r(n), + ws(n, m), wy(n, m), sy(m, m), ss(m, m) c ************ c c Subroutine matupd c c This subroutine updates matrices WS and WY, and forms the c middle matrix in B. c c Subprograms called: c c Linpack ... dcopy, ddot. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer j,pointr double precision ddot double precision one parameter (one=1.0d0) c Set pointers for matrices WS and WY. if (iupdat .le. m) then col = iupdat itail = mod(head+iupdat-2,m) + 1 else itail = mod(itail,m) + 1 head = mod(head,m) + 1 endif c Update matrices WS and WY. call dcopy(n,d,1,ws(1,itail),1) call dcopy(n,r,1,wy(1,itail),1) c Set theta=yy/ys. theta = rr/dr c Form the middle matrix in B. c update the upper triangle of SS, c and the lower triangle of SY: if (iupdat .gt. m) then c move old information do 50 j = 1, col - 1 call dcopy(j,ss(2,j+1),1,ss(1,j),1) call dcopy(col-j,sy(j+1,j+1),1,sy(j,j),1) 50 continue endif c add new information: the last row of SY c and the last column of SS: pointr = head do 51 j = 1, col - 1 sy(col,j) = ddot(n,d,1,wy(1,pointr),1) ss(j,col) = ddot(n,ws(1,pointr),1,d,1) pointr = mod(pointr,m) + 1 51 continue if (stp .eq. one) then ss(col,col) = dtd else ss(col,col) = stp*stp*dtd endif sy(col,col) = dr return end c======================= The end of matupd ============================= subroutine prn1lb(n, m, l, u, x, iprint, itfile, epsmch) integer n, m, iprint, itfile double precision epsmch, x(n), l(n), u(n) c ************ c c Subroutine prn1lb c c This subroutine prints the input data, initial point, upper and c lower bounds of each variable, machine precision, as well as c the headings of the output. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i if (iprint .ge. 0) then write (6,7001) epsmch write (6,*) 'N = ',n,' M = ',m if (iprint .ge. 1) then write (itfile,2001) epsmch write (itfile,*)'N = ',n,' M = ',m write (itfile,9001) if (iprint .gt. 100) then write (6,1004) 'L =',(l(i),i = 1,n) write (6,1004) 'X0 =',(x(i),i = 1,n) write (6,1004) 'U =',(u(i),i = 1,n) endif endif endif 1004 format (/,a4, 1p, 6(1x,d11.4),/,(4x,1p,6(1x,d11.4))) 2001 format ('RUNNING THE L-BFGS-B CODE',/,/, + 'it = iteration number',/, + 'nf = number of function evaluations',/, + 'nint = number of segments explored during the Cauchy search',/, + 'nact = number of active bounds at the generalized Cauchy point' + ,/, + 'sub = manner in which the subspace minimization terminated:' + ,/,' con = converged, bnd = a bound was reached',/, + 'itls = number of iterations performed in the line search',/, + 'stepl = step length used',/, + 'tstep = norm of the displacement (total step)',/, + 'projg = norm of the projected gradient',/, + 'f = function value',/,/, + ' * * *',/,/, + 'Machine precision =',1p,d10.3) 7001 format ('RUNNING THE L-BFGS-B CODE',/,/, + ' * * *',/,/, + 'Machine precision =',1p,d10.3) 9001 format (/,3x,'it',3x,'nf',2x,'nint',2x,'nact',2x,'sub',2x,'itls', + 2x,'stepl',4x,'tstep',5x,'projg',8x,'f') return end c======================= The end of prn1lb ============================= subroutine prn2lb(n, x, f, g, iprint, itfile, iter, nfgv, nact, + sbgnrm, nint, word, iword, iback, stp, xstep) character*3 word integer n, iprint, itfile, iter, nfgv, nact, nint, + iword, iback double precision f, sbgnrm, stp, xstep, x(n), g(n) c ************ c c Subroutine prn2lb c c This subroutine prints out new information after a successful c line search. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,imod c 'word' records the status of subspace solutions. if (iword .eq. 0) then c the subspace minimization converged. word = 'con' else if (iword .eq. 1) then c the subspace minimization stopped at a bound. word = 'bnd' else if (iword .eq. 5) then c the truncated Newton step has been used. word = 'TNT' else word = '---' endif if (iprint .ge. 99) then write (6,*) 'LINE SEARCH',iback,' times; norm of step = ',xstep write (6,2001) iter,f,sbgnrm if (iprint .gt. 100) then write (6,1004) 'X =',(x(i), i = 1, n) write (6,1004) 'G =',(g(i), i = 1, n) endif else if (iprint .gt. 0) then imod = mod(iter,iprint) if (imod .eq. 0) write (6,2001) iter,f,sbgnrm endif if (iprint .ge. 1) write (itfile,3001) + iter,nfgv,nint,nact,word,iback,stp,xstep,sbgnrm,f 1004 format (/,a4, 1p, 6(1x,d11.4),/,(4x,1p,6(1x,d11.4))) 2001 format + (/,'At iterate',i5,4x,'f= ',1p,d12.5,4x,'|proj g|= ',1p,d12.5) 3001 format(2(1x,i4),2(1x,i5),2x,a3,1x,i4,1p,2(2x,d7.1),1p,2(1x,d10.3)) return end c======================= The end of prn2lb ============================= subroutine prn3lb(n, x, f, task, iprint, info, itfile, + iter, nfgv, nintol, nskip, nact, sbgnrm, + time, nint, word, iback, stp, xstep, k, + cachyt, sbtime, lnscht) character*60 task character*3 word integer n, iprint, info, itfile, iter, nfgv, nintol, + nskip, nact, nint, iback, k double precision f, sbgnrm, time, stp, xstep, cachyt, sbtime, + lnscht, x(n) c ************ c c Subroutine prn3lb c c This subroutine prints out information when either a built-in c convergence test is satisfied or when an error message is c generated. c c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i if (task(1:5) .eq. 'ERROR') goto 999 if (iprint .ge. 0) then write (6,3003) write (6,3004) write(6,3005) n,iter,nfgv,nintol,nskip,nact,sbgnrm,f if (iprint .ge. 100) then write (6,1004) 'X =',(x(i),i = 1,n) endif if (iprint .ge. 1) write (6,*) ' F =',f endif 999 continue if (iprint .ge. 0) then write (6,3009) task if (info .ne. 0) then if (info .eq. -1) write (6,9011) if (info .eq. -2) write (6,9012) if (info .eq. -3) write (6,9013) if (info .eq. -4) write (6,9014) if (info .eq. -5) write (6,9015) if (info .eq. -6) write (6,*)' Input nbd(',k,') is invalid.' if (info .eq. -7) + write (6,*)' l(',k,') > u(',k,'). No feasible solution.' if (info .eq. -8) write (6,9018) if (info .eq. -9) write (6,9019) endif if (iprint .ge. 1) write (6,3007) cachyt,sbtime,lnscht write (6,3008) time if (iprint .ge. 1) then if (info .eq. -4 .or. info .eq. -9) then write (itfile,3002) + iter,nfgv,nint,nact,word,iback,stp,xstep endif write (itfile,3009) task if (info .ne. 0) then if (info .eq. -1) write (itfile,9011) if (info .eq. -2) write (itfile,9012) if (info .eq. -3) write (itfile,9013) if (info .eq. -4) write (itfile,9014) if (info .eq. -5) write (itfile,9015) if (info .eq. -8) write (itfile,9018) if (info .eq. -9) write (itfile,9019) endif write (itfile,3008) time endif endif 1004 format (/,a4, 1p, 6(1x,d11.4),/,(4x,1p,6(1x,d11.4))) 3002 format(2(1x,i4),2(1x,i5),2x,a3,1x,i4,1p,2(2x,d7.1),6x,'-',10x,'-') 3003 format (/, + ' * * *',/,/, + 'Tit = total number of iterations',/, + 'Tnf = total number of function evaluations',/, + 'Tnint = total number of segments explored during', + ' Cauchy searches',/, + 'Skip = number of BFGS updates skipped',/, + 'Nact = number of active bounds at final generalized', + ' Cauchy point',/, + 'Projg = norm of the final projected gradient',/, + 'F = final function value',/,/, + ' * * *') 3004 format (/,3x,'N',3x,'Tit',2x,'Tnf',2x,'Tnint',2x, + 'Skip',2x,'Nact',5x,'Projg',8x,'F') 3005 format (i5,2(1x,i4),(1x,i6),(2x,i4),(1x,i5),1p,2(2x,d10.3)) 3007 format (/,' Cauchy time',1p,e10.3,' seconds.',/ + ' Subspace minimization time',1p,e10.3,' seconds.',/ + ' Line search time',1p,e10.3,' seconds.') 3008 format (/,' Total User time',1p,e10.3,' seconds.',/) 3009 format (/,a60) 9011 format (/, +' Matrix in 1st Cholesky factorization in formk is not Pos. Def.') 9012 format (/, +' Matrix in 2st Cholesky factorization in formk is not Pos. Def.') 9013 format (/, +' Matrix in the Cholesky factorization in formt is not Pos. Def.') 9014 format (/, +' Derivative >= 0, backtracking line search impossible.',/, +' Previous x, f and g restored.',/, +' Possible causes: 1 error in function or gradient evaluation;',/, +' 2 rounding errors dominate computation.') 9015 format (/, +' Warning: more than 10 function and gradient',/, +' evaluations in the last line search. Termination',/, +' may possibly be caused by a bad search direction.') 9018 format (/,' The triangular system is singular.') 9019 format (/, +' Line search cannot locate an adequate point after 20 function',/ +,' and gradient evaluations. Previous x, f and g restored.',/, +' Possible causes: 1 error in function or gradient evaluation;',/, +' 2 rounding error dominate computation.') return end c======================= The end of prn3lb ============================= subroutine projgr(n, l, u, nbd, x, g, sbgnrm) integer n, nbd(n) double precision sbgnrm, x(n), l(n), u(n), g(n) c ************ c c Subroutine projgr c c This subroutine computes the infinity norm of the projected c gradient. c c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i double precision gi double precision zero parameter (zero=0.0d0) sbgnrm = zero do 15 i = 1, n gi = g(i) if (nbd(i) .ne. 0) then if (gi .lt. zero) then if (nbd(i) .ge. 2) gi = max((x(i)-u(i)),gi) else if (nbd(i) .le. 2) gi = min((x(i)-l(i)),gi) endif endif sbgnrm = max(sbgnrm,abs(gi)) 15 continue return end c======================= The end of projgr ============================= subroutine subsm(n, m, nsub, ind, l, u, nbd, x, d, ws, wy, theta, + col, head, iword, wv, wn, iprint, info) integer n, m, nsub, col, head, iword, iprint, info, + ind(nsub), nbd(n) double precision theta, + l(n), u(n), x(n), d(n), + ws(n, m), wy(n, m), + wv(2*m), wn(2*m, 2*m) c ************ c c Subroutine subsm c c Given xcp, l, u, r, an index set that specifies c the active set at xcp, and an l-BFGS matrix B c (in terms of WY, WS, SY, WT, head, col, and theta), c this subroutine computes an approximate solution c of the subspace problem c c (P) min Q(x) = r'(x-xcp) + 1/2 (x-xcp)' B (x-xcp) c c subject to l<=x<=u c x_i=xcp_i for all i in A(xcp) c c along the subspace unconstrained Newton direction c c d = -(Z'BZ)^(-1) r. c c The formula for the Newton direction, given the L-BFGS matrix c and the Sherman-Morrison formula, is c c d = (1/theta)r + (1/theta*2) Z'WK^(-1)W'Z r. c c where c K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c c Note that this procedure for computing d differs c from that described in [1]. One can show that the matrix K is c equal to the matrix M^[-1]N in that paper. c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c nsub is an integer variable. c On entry nsub is the number of free variables. c On exit nsub is unchanged. c c ind is an integer array of dimension nsub. c On entry ind specifies the coordinate indices of free variables. c On exit ind is unchanged. c c l is a double precision array of dimension n. c On entry l is the lower bound of x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound of x. c On exit u is unchanged. c c nbd is a integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, and c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c x is a double precision array of dimension n. c On entry x specifies the Cauchy point xcp. c On exit x(i) is the minimizer of Q over the subspace of c free variables. c c d is a double precision array of dimension n. c On entry d is the reduced gradient of Q at xcp. c On exit d is the Newton direction of Q. c c ws and wy are double precision arrays; c theta is a double precision variable; c col is an integer variable; c head is an integer variable. c On entry they store the information defining the c limited memory BFGS matrix: c ws(n,m) stores S, a set of s-vectors; c wy(n,m) stores Y, a set of y-vectors; c theta is the scaling factor specifying B_0 = theta I; c col is the number of variable metric corrections stored; c head is the location of the 1st s- (or y-) vector in S (or Y). c On exit they are unchanged. c c iword is an integer variable. c On entry iword is unspecified. c On exit iword specifies the status of the subspace solution. c iword = 0 if the solution is in the box, c 1 if some bound is encountered. c c wv is a double precision working array of dimension 2m. c c wn is a double precision array of dimension 2m x 2m. c On entry the upper triangle of wn stores the LEL^T factorization c of the indefinite matrix c c K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c where E = [-I 0] c [ 0 I] c On exit wn is unchanged. c c iprint is an INTEGER variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c info is an integer variable. c On entry info is unspecified. c On exit info = 0 for normal return, c = nonzero for abnormal return c when the matrix K is ill-conditioned. c c Subprograms called: c c Linpack dtrsl. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ logical temp1_updated integer pointr,m2,col2,ibd,jy,js,i,j,k double precision alpha,dk,temp1,temp2 double precision one,zero parameter (one=1.0d0,zero=0.0d0) if (nsub .le. 0) return if (iprint .ge. 99) write (6,1001) c Compute wv = W'Zd. pointr = head do 20 i = 1, col temp1 = zero temp2 = zero do 10 j = 1, nsub k = ind(j) temp1 = temp1 + wy(k,pointr)*d(j) temp2 = temp2 + ws(k,pointr)*d(j) 10 continue wv(i) = temp1 wv(col + i) = theta*temp2 pointr = mod(pointr,m) + 1 20 continue c Compute wv:=K^(-1)wv. m2 = 2*m col2 = 2*col call dtrsl(wn,m2,col2,wv,11,info) if (info .ne. 0) return do 25 i = 1, col wv(i) = -wv(i) 25 continue call dtrsl(wn,m2,col2,wv,01,info) if (info .ne. 0) return c Compute d = (1/theta)d + (1/theta**2)Z'W wv. pointr = head do 40 jy = 1, col js = col + jy do 30 i = 1, nsub k = ind(i) d(i) = d(i) + wy(k,pointr)*wv(jy)/theta + + ws(k,pointr)*wv(js) 30 continue pointr = mod(pointr,m) + 1 40 continue do 50 i = 1, nsub d(i) = d(i)/theta 50 continue c Backtrack to the feasible region. alpha = one temp1 = alpha do 60 i = 1, nsub k = ind(i) dk = d(i) if (nbd(k) .ne. 0) then temp1_updated = .false. if (dk .lt. zero .and. nbd(k) .le. 2) then temp2 = l(k) - x(k) if (temp2 .ge. zero) then temp1 = zero temp1_updated = .true. else if (dk*alpha .lt. temp2) then temp1 = temp2/dk temp1_updated = .true. endif else if (dk .gt. zero .and. nbd(k) .ge. 2) then temp2 = u(k) - x(k) if (temp2 .le. zero) then temp1 = zero temp1_updated = .true. else if (dk*alpha .gt. temp2) then temp1 = temp2/dk temp1_updated = .true. endif endif cc logical variable temp1_updated added to eliminate unexpected cc trigger of the if statement due to possible difference between cc hardware precision and double precision. if (temp1_updated .and. temp1 .lt. alpha) then alpha = temp1 ibd = i endif endif 60 continue if (alpha .lt. one) then dk = d(ibd) k = ind(ibd) if (dk .gt. zero) then x(k) = u(k) d(ibd) = zero else if (dk .lt. zero) then x(k) = l(k) d(ibd) = zero endif endif do 70 i = 1, nsub k = ind(i) x(k) = x(k) + alpha*d(i) 70 continue if (iprint .ge. 99) then if (alpha .lt. one) then write (6,1002) alpha else write (6,*) 'SM solution inside the box' end if if (iprint .gt.100) write (6,1003) (x(i),i=1,n) endif if (alpha .lt. one) then iword = 1 else iword = 0 endif if (iprint .ge. 99) write (6,1004) 1001 format (/,'----------------SUBSM entered-----------------',/) 1002 format ( 'ALPHA = ',f7.5,' backtrack to the BOX') 1003 format ('Subspace solution X = ',/,(4x,1p,6(1x,d11.4))) 1004 format (/,'----------------exit SUBSM --------------------',/) return end c====================== The end of subsm =============================== subroutine dcsrch(f,g,stp,ftol,gtol,xtol,stpmin,stpmax, + task,isave,dsave) character*(*) task integer isave(2) double precision f,g,stp,ftol,gtol,xtol,stpmin,stpmax double precision dsave(13) c ********** c c Subroutine dcsrch c c This subroutine finds a step that satisfies a sufficient c decrease condition and a curvature condition. c c Each call of the subroutine updates an interval with c endpoints stx and sty. The interval is initially chosen c so that it contains a minimizer of the modified function c c psi(stp) = f(stp) - f(0) - ftol*stp*f'(0). c c If psi(stp) <= 0 and f'(stp) >= 0 for some step, then the c interval is chosen so that it contains a minimizer of f. c c The algorithm is designed to find a step that satisfies c the sufficient decrease condition c c f(stp) <= f(0) + ftol*stp*f'(0), c c and the curvature condition c c abs(f'(stp)) <= gtol*abs(f'(0)). c c If ftol is less than gtol and if, for example, the function c is bounded below, then there is always a step which satisfies c both conditions. c c If no step can be found that satisfies both conditions, then c the algorithm stops with a warning. In this case stp only c satisfies the sufficient decrease condition. c c A typical invocation of dcsrch has the following outline: c c task = 'START' c 10 continue c call dcsrch( ... ) c if (task .eq. 'FG') then c Evaluate the function and the gradient at stp c goto 10 c end if c c NOTE: The user must no alter work arrays between calls. c c The subroutine statement is c c subroutine dcsrch(f,g,stp,ftol,gtol,xtol,stpmin,stpmax, c task,isave,dsave) c where c c f is a double precision variable. c On initial entry f is the value of the function at 0. c On subsequent entries f is the value of the c function at stp. c On exit f is the value of the function at stp. c c g is a double precision variable. c On initial entry g is the derivative of the function at 0. c On subsequent entries g is the derivative of the c function at stp. c On exit g is the derivative of the function at stp. c c stp is a double precision variable. c On entry stp is the current estimate of a satisfactory c step. On initial entry, a positive initial estimate c must be provided. c On exit stp is the current estimate of a satisfactory step c if task = 'FG'. If task = 'CONV' then stp satisfies c the sufficient decrease and curvature condition. c c ftol is a double precision variable. c On entry ftol specifies a nonnegative tolerance for the c sufficient decrease condition. c On exit ftol is unchanged. c c gtol is a double precision variable. c On entry gtol specifies a nonnegative tolerance for the c curvature condition. c On exit gtol is unchanged. c c xtol is a double precision variable. c On entry xtol specifies a nonnegative relative tolerance c for an acceptable step. The subroutine exits with a c warning if the relative difference between sty and stx c is less than xtol. c On exit xtol is unchanged. c c stpmin is a double precision variable. c On entry stpmin is a nonnegative lower bound for the step. c On exit stpmin is unchanged. c c stpmax is a double precision variable. c On entry stpmax is a nonnegative upper bound for the step. c On exit stpmax is unchanged. c c task is a character variable of length at least 60. c On initial entry task must be set to 'START'. c On exit task indicates the required action: c c If task(1:2) = 'FG' then evaluate the function and c derivative at stp and call dcsrch again. c c If task(1:4) = 'CONV' then the search is successful. c c If task(1:4) = 'WARN' then the subroutine is not able c to satisfy the convergence conditions. The exit value of c stp contains the best point found during the search. c c If task(1:5) = 'ERROR' then there is an error in the c input arguments. c c On exit with convergence, a warning or an error, the c variable task contains additional information. c c isave is an integer work array of dimension 2. c c dsave is a double precision work array of dimension 13. c c Subprograms called c c MINPACK-2 ... dcstep c c MINPACK-1 Project. June 1983. c Argonne National Laboratory. c Jorge J. More' and David J. Thuente. c c MINPACK-2 Project. October 1993. c Argonne National Laboratory and University of Minnesota. c Brett M. Averick, Richard G. Carter, and Jorge J. More'. c c ********** double precision zero,p5,p66 parameter(zero=0.0d0,p5=0.5d0,p66=0.66d0) double precision xtrapl,xtrapu parameter(xtrapl=1.1d0,xtrapu=4.0d0) logical brackt integer stage double precision finit,ftest,fm,fx,fxm,fy,fym,ginit,gtest, + gm,gx,gxm,gy,gym,stx,sty,stmin,stmax,width,width1 c Initialization block. if (task(1:5) .eq. 'START') then c Check the input arguments for errors. if (stp .lt. stpmin) task = 'ERROR: STP .LT. STPMIN' if (stp .gt. stpmax) task = 'ERROR: STP .GT. STPMAX' if (g .ge. zero) task = 'ERROR: INITIAL G .GE. ZERO' if (ftol .lt. zero) task = 'ERROR: FTOL .LT. ZERO' if (gtol .lt. zero) task = 'ERROR: GTOL .LT. ZERO' if (xtol .lt. zero) task = 'ERROR: XTOL .LT. ZERO' if (stpmin .lt. zero) task = 'ERROR: STPMIN .LT. ZERO' if (stpmax .lt. stpmin) task = 'ERROR: STPMAX .LT. STPMIN' c Exit if there are errors on input. if (task(1:5) .eq. 'ERROR') return c Initialize local variables. brackt = .false. stage = 1 finit = f ginit = g gtest = ftol*ginit width = stpmax - stpmin width1 = width/p5 c The variables stx, fx, gx contain the values of the step, c function, and derivative at the best step. c The variables sty, fy, gy contain the value of the step, c function, and derivative at sty. c The variables stp, f, g contain the values of the step, c function, and derivative at stp. stx = zero fx = finit gx = ginit sty = zero fy = finit gy = ginit stmin = zero stmax = stp + xtrapu*stp task = 'FG' goto 1000 else c Restore local variables. if (isave(1) .eq. 1) then brackt = .true. else brackt = .false. endif stage = isave(2) ginit = dsave(1) gtest = dsave(2) gx = dsave(3) gy = dsave(4) finit = dsave(5) fx = dsave(6) fy = dsave(7) stx = dsave(8) sty = dsave(9) stmin = dsave(10) stmax = dsave(11) width = dsave(12) width1 = dsave(13) endif c If psi(stp) <= 0 and f'(stp) >= 0 for some step, then the c algorithm enters the second stage. ftest = finit + stp*gtest if (stage .eq. 1 .and. f .le. ftest .and. g .ge. zero) + stage = 2 c Test for warnings. if (brackt .and. (stp .le. stmin .or. stp .ge. stmax)) + task = 'WARNING: ROUNDING ERRORS PREVENT PROGRESS' if (brackt .and. stmax - stmin .le. xtol*stmax) + task = 'WARNING: XTOL TEST SATISFIED' if (stp .eq. stpmax .and. f .le. ftest .and. g .le. gtest) + task = 'WARNING: STP = STPMAX' if (stp .eq. stpmin .and. (f .gt. ftest .or. g .ge. gtest)) + task = 'WARNING: STP = STPMIN' cc New warning statement added to eliminate the unexpected case cc of stp=stx due to possible difference between hardware precision cc and double precision. if (stp .eq. stx) + task = 'WARNING: STP = STX' c Test for convergence. if (f .le. ftest .and. abs(g) .le. gtol*(-ginit)) + task = 'CONVERGENCE' c Test for termination. if (task(1:4) .eq. 'WARN' .or. task(1:4) .eq. 'CONV') goto 1000 c A modified function is used to predict the step during the c first stage if a lower function value has been obtained but c the decrease is not sufficient. if (stage .eq. 1 .and. f .le. fx .and. f .gt. ftest) then c Define the modified function and derivative values. fm = f - stp*gtest fxm = fx - stx*gtest fym = fy - sty*gtest gm = g - gtest gxm = gx - gtest gym = gy - gtest c Call dcstep to update stx, sty, and to compute the new step. call dcstep(stx,fxm,gxm,sty,fym,gym,stp,fm,gm, + brackt,stmin,stmax) c Reset the function and derivative values for f. fx = fxm + stx*gtest fy = fym + sty*gtest gx = gxm + gtest gy = gym + gtest else c Call dcstep to update stx, sty, and to compute the new step. call dcstep(stx,fx,gx,sty,fy,gy,stp,f,g, + brackt,stmin,stmax) endif c Decide if a bisection step is needed. if (brackt) then if (abs(sty-stx) .ge. p66*width1) stp = stx + p5*(sty - stx) width1 = width width = abs(sty-stx) endif c Set the minimum and maximum steps allowed for stp. if (brackt) then stmin = min(stx,sty) stmax = max(stx,sty) else stmin = stp + xtrapl*(stp - stx) stmax = stp + xtrapu*(stp - stx) endif c Force the step to be within the bounds stpmax and stpmin. stp = max(stp,stpmin) stp = min(stp,stpmax) c If further progress is not possible, let stp be the best c point obtained during the search. if (brackt .and. (stp .le. stmin .or. stp .ge. stmax) + .or. (brackt .and. stmax-stmin .le. xtol*stmax)) stp = stx c Obtain another function and derivative. task = 'FG' 1000 continue c Save local variables. if (brackt) then isave(1) = 1 else isave(1) = 0 endif isave(2) = stage dsave(1) = ginit dsave(2) = gtest dsave(3) = gx dsave(4) = gy dsave(5) = finit dsave(6) = fx dsave(7) = fy dsave(8) = stx dsave(9) = sty dsave(10) = stmin dsave(11) = stmax dsave(12) = width dsave(13) = width1 end c====================== The end of dcsrch ============================== subroutine dcstep(stx,fx,dx,sty,fy,dy,stp,fp,dp,brackt, + stpmin,stpmax) logical brackt double precision stx,fx,dx,sty,fy,dy,stp,fp,dp,stpmin,stpmax c ********** c c Subroutine dcstep c c This subroutine computes a safeguarded step for a search c procedure and updates an interval that contains a step that c satisfies a sufficient decrease and a curvature condition. c c The parameter stx contains the step with the least function c value. If brackt is set to .true. then a minimizer has c been bracketed in an interval with endpoints stx and sty. c The parameter stp contains the current step. c The subroutine assumes that if brackt is set to .true. then c c min(stx,sty) < stp < max(stx,sty), c c and that the derivative at stx is negative in the direction c of the step. c c The subroutine statement is c c subroutine dcstep(stx,fx,dx,sty,fy,dy,stp,fp,dp,brackt, c stpmin,stpmax) c c where c c stx is a double precision variable. c On entry stx is the best step obtained so far and is an c endpoint of the interval that contains the minimizer. c On exit stx is the updated best step. c c fx is a double precision variable. c On entry fx is the function at stx. c On exit fx is the function at stx. c c dx is a double precision variable. c On entry dx is the derivative of the function at c stx. The derivative must be negative in the direction of c the step, that is, dx and stp - stx must have opposite c signs. c On exit dx is the derivative of the function at stx. c c sty is a double precision variable. c On entry sty is the second endpoint of the interval that c contains the minimizer. c On exit sty is the updated endpoint of the interval that c contains the minimizer. c c fy is a double precision variable. c On entry fy is the function at sty. c On exit fy is the function at sty. c c dy is a double precision variable. c On entry dy is the derivative of the function at sty. c On exit dy is the derivative of the function at the exit sty. c c stp is a double precision variable. c On entry stp is the current step. If brackt is set to .true. c then on input stp must be between stx and sty. c On exit stp is a new trial step. c c fp is a double precision variable. c On entry fp is the function at stp c On exit fp is unchanged. c c dp is a double precision variable. c On entry dp is the the derivative of the function at stp. c On exit dp is unchanged. c c brackt is an logical variable. c On entry brackt specifies if a minimizer has been bracketed. c Initially brackt must be set to .false. c On exit brackt specifies if a minimizer has been bracketed. c When a minimizer is bracketed brackt is set to .true. c c stpmin is a double precision variable. c On entry stpmin is a lower bound for the step. c On exit stpmin is unchanged. c c stpmax is a double precision variable. c On entry stpmax is an upper bound for the step. c On exit stpmax is unchanged. c c MINPACK-1 Project. June 1983 c Argonne National Laboratory. c Jorge J. More' and David J. Thuente. c c MINPACK-2 Project. October 1993. c Argonne National Laboratory and University of Minnesota. c Brett M. Averick and Jorge J. More'. c c ********** double precision zero,p66,two,three parameter(zero=0.0d0,p66=0.66d0,two=2.0d0,three=3.0d0) double precision gamma,p,q,r,s,sgnd,stpc,stpf,stpq,theta sgnd = dp*(dx/abs(dx)) c First case: A higher function value. The minimum is bracketed. c If the cubic step is closer to stx than the quadratic step, the c cubic step is taken, otherwise the average of the cubic and c quadratic steps is taken. if (fp .gt. fx) then theta = three*(fx - fp)/(stp - stx) + dx + dp s = max(abs(theta),abs(dx),abs(dp)) gamma = s*sqrt((theta/s)**2 - (dx/s)*(dp/s)) if (stp .lt. stx) gamma = -gamma p = (gamma - dx) + theta q = ((gamma - dx) + gamma) + dp r = p/q stpc = stx + r*(stp - stx) stpq = stx + ((dx/((fx - fp)/(stp - stx) + dx))/two)* + (stp - stx) if (abs(stpc-stx) .lt. abs(stpq-stx)) then stpf = stpc else stpf = stpc + (stpq - stpc)/two endif brackt = .true. c Second case: A lower function value and derivatives of opposite c sign. The minimum is bracketed. If the cubic step is farther from c stp than the secant step, the cubic step is taken, otherwise the c secant step is taken. else if (sgnd .lt. zero) then theta = three*(fx - fp)/(stp - stx) + dx + dp s = max(abs(theta),abs(dx),abs(dp)) gamma = s*sqrt((theta/s)**2 - (dx/s)*(dp/s)) if (stp .gt. stx) gamma = -gamma p = (gamma - dp) + theta q = ((gamma - dp) + gamma) + dx r = p/q stpc = stp + r*(stx - stp) stpq = stp + (dp/(dp - dx))*(stx - stp) if (abs(stpc-stp) .gt. abs(stpq-stp)) then stpf = stpc else stpf = stpq endif brackt = .true. c Third case: A lower function value, derivatives of the same sign, c and the magnitude of the derivative decreases. else if (abs(dp) .lt. abs(dx)) then c The cubic step is computed only if the cubic tends to infinity c in the direction of the step or if the minimum of the cubic c is beyond stp. Otherwise the cubic step is defined to be the c secant step. theta = three*(fx - fp)/(stp - stx) + dx + dp s = max(abs(theta),abs(dx),abs(dp)) c The case gamma = 0 only arises if the cubic does not tend c to infinity in the direction of the step. gamma = s*sqrt(max(zero,(theta/s)**2-(dx/s)*(dp/s))) if (stp .gt. stx) gamma = -gamma p = (gamma - dp) + theta q = (gamma + (dx - dp)) + gamma r = p/q if (r .lt. zero .and. gamma .ne. zero) then stpc = stp + r*(stx - stp) else if (stp .gt. stx) then stpc = stpmax else stpc = stpmin endif stpq = stp + (dp/(dp - dx))*(stx - stp) if (brackt) then c A minimizer has been bracketed. If the cubic step is c closer to stp than the secant step, the cubic step is c taken, otherwise the secant step is taken. if (abs(stpc-stp) .lt. abs(stpq-stp)) then stpf = stpc else stpf = stpq endif if (stp .gt. stx) then stpf = min(stp+p66*(sty-stp),stpf) else stpf = max(stp+p66*(sty-stp),stpf) endif else c A minimizer has not been bracketed. If the cubic step is c farther from stp than the secant step, the cubic step is c taken, otherwise the secant step is taken. if (abs(stpc-stp) .gt. abs(stpq-stp)) then stpf = stpc else stpf = stpq endif stpf = min(stpmax,stpf) stpf = max(stpmin,stpf) endif c Fourth case: A lower function value, derivatives of the c same sign, and the magnitude of the derivative does not c decrease. If the minimum is not bracketed, the step is either c stpmin or stpmax, otherwise the cubic step is taken. else if (brackt) then theta = three*(fp - fy)/(sty - stp) + dy + dp s = max(abs(theta),abs(dy),abs(dp)) gamma = s*sqrt((theta/s)**2 - (dy/s)*(dp/s)) if (stp .gt. sty) gamma = -gamma p = (gamma - dp) + theta q = ((gamma - dp) + gamma) + dy r = p/q stpc = stp + r*(sty - stp) stpf = stpc else if (stp .gt. stx) then stpf = stpmax else stpf = stpmin endif endif c Update the interval which contains a minimizer. if (fp .gt. fx) then sty = stp fy = fp dy = dp else if (sgnd .lt. zero) then sty = stx fy = fx dy = dx endif stx = stp fx = fp dx = dp endif c Compute the new step. stp = stpf end c====================== The end of dcstep ============================== subroutine timer(ttime) double precision ttime c ********* c c Subroutine timer c c This subroutine is used to determine user time. In a typical c application, the user time for a code segment requires calls c to subroutine timer to determine the initial and final time. c c The subroutine statement is c c subroutine timer(ttime) c c where c c ttime is an output variable which specifies the user time. c c Argonne National Laboratory and University of Minnesota. c MINPACK-2 Project. c c Modified October 1990 by Brett M. Averick. c c ********** real temp real tarray(2) real etime c The first element of the array tarray specifies user time temp = etime(tarray) ttime = dble(tarray(1)) return end c====================== The end of timer =============================== double precision function dnrm2(n,x,incx) integer n,incx double precision x(n) c ********** c c Function dnrm2 c c Given a vector x of length n, this function calculates the c Euclidean norm of x with stride incx. c c The function statement is c c double precision function dnrm2(n,x,incx) c c where c c n is a positive integer input variable. c c x is an input array of length n. c c incx is a positive integer variable that specifies the c stride of the vector. c c Subprograms called c c FORTRAN-supplied ... abs, max, sqrt c c MINPACK-2 Project. February 1991. c Argonne National Laboratory. c Brett M. Averick. c c ********** integer i double precision scale dnrm2 = 0.0d0 scale = 0.0d0 do 10 i = 1, n, incx scale = max(scale, abs(x(i))) 10 continue if (scale .eq. 0.0d0) return do 20 i = 1, n, incx dnrm2 = dnrm2 + (x(i)/scale)**2 20 continue dnrm2 = scale*sqrt(dnrm2) return end c====================== The end of dnrm2 =============================== double precision function dpmeps() c ********** c c Subroutine dpeps c c This subroutine computes the machine precision parameter c dpmeps as the smallest floating point number such that c 1 + dpmeps differs from 1. c c This subroutine is based on the subroutine machar described in c c W. J. Cody, c MACHAR: A subroutine to dynamically determine machine parameters, c ACM Trans. Math. Soft., 14, 1988, pages 303-311. c c The subroutine statement is: c c subroutine dpeps(dpmeps) c c where c c dpmeps is a double precision variable. c On entry dpmeps need not be specified. c On exit dpmeps is the machine precision. c c MINPACK-2 Project. February 1991. c Argonne National Laboratory and University of Minnesota. c Brett M. Averick. c c ******* integer i,ibeta,irnd,it,itemp,negep double precision a,b,beta,betain,betah,temp,tempa,temp1, + zero,one,two data zero,one,two /0.0d0,1.0d0,2.0d0/ c determine ibeta, beta ala malcolm. a = one b = one 10 continue a = a + a temp = a + one temp1 = temp - a if (temp1 - one .eq. zero) go to 10 20 continue b = b + b temp = a + b itemp = int(temp - a) if (itemp .eq. 0) go to 20 ibeta = itemp beta = dble(ibeta) c determine it, irnd. it = 0 b = one 30 continue it = it + 1 b = b * beta temp = b + one temp1 = temp - b if (temp1 - one .eq. zero) go to 30 irnd = 0 betah = beta/two temp = a + betah if (temp - a .ne. zero) irnd = 1 tempa = a + beta temp = tempa + betah if ((irnd .eq. 0) .and. (temp - tempa .ne. zero)) irnd = 2 c determine dpmeps. negep = it + 3 betain = one/beta a = one do 40 i = 1, negep a = a*betain 40 continue 50 continue temp = one + a if (temp - one .ne. zero) go to 60 a = a*beta go to 50 60 continue dpmeps = a if ((ibeta .eq. 2) .or. (irnd .eq. 0)) go to 70 a = (a*(one + a))/two temp = one + a if (temp - one .ne. zero) dpmeps = a 70 return end c====================== The end of dpmeps ============================== subroutine daxpy(n,da,dx,incx,dy,incy) c c constant times a vector plus a vector. c uses unrolled loops for increments equal to one. c jack dongarra, linpack, 3/11/78. c double precision dx(*),dy(*),da integer i,incx,incy,ix,iy,m,mp1,n c if(n.le.0)return if (da .eq. 0.0d0) return if(incx.eq.1.and.incy.eq.1)go to 20 c c code for unequal increments or equal increments c not equal to 1 c ix = 1 iy = 1 if(incx.lt.0)ix = (-n+1)*incx + 1 if(incy.lt.0)iy = (-n+1)*incy + 1 do 10 i = 1,n dy(iy) = dy(iy) + da*dx(ix) ix = ix + incx iy = iy + incy 10 continue return c c code for both increments equal to 1 c c c clean-up loop c 20 m = mod(n,4) if( m .eq. 0 ) go to 40 do 30 i = 1,m dy(i) = dy(i) + da*dx(i) 30 continue if( n .lt. 4 ) return 40 mp1 = m + 1 do 50 i = mp1,n,4 dy(i) = dy(i) + da*dx(i) dy(i + 1) = dy(i + 1) + da*dx(i + 1) dy(i + 2) = dy(i + 2) + da*dx(i + 2) dy(i + 3) = dy(i + 3) + da*dx(i + 3) 50 continue return end c====================== The end of daxpy =============================== subroutine dcopy(n,dx,incx,dy,incy) c c copies a vector, x, to a vector, y. c uses unrolled loops for increments equal to one. c jack dongarra, linpack, 3/11/78. c double precision dx(*),dy(*) integer i,incx,incy,ix,iy,m,mp1,n c if(n.le.0)return if(incx.eq.1.and.incy.eq.1)go to 20 c c code for unequal increments or equal increments c not equal to 1 c ix = 1 iy = 1 if(incx.lt.0)ix = (-n+1)*incx + 1 if(incy.lt.0)iy = (-n+1)*incy + 1 do 10 i = 1,n dy(iy) = dx(ix) ix = ix + incx iy = iy + incy 10 continue return c c code for both increments equal to 1 c c c clean-up loop c 20 m = mod(n,7) if( m .eq. 0 ) go to 40 do 30 i = 1,m dy(i) = dx(i) 30 continue if( n .lt. 7 ) return 40 mp1 = m + 1 do 50 i = mp1,n,7 dy(i) = dx(i) dy(i + 1) = dx(i + 1) dy(i + 2) = dx(i + 2) dy(i + 3) = dx(i + 3) dy(i + 4) = dx(i + 4) dy(i + 5) = dx(i + 5) dy(i + 6) = dx(i + 6) 50 continue return end c====================== The end of dcopy =============================== double precision function ddot(n,dx,incx,dy,incy) c c forms the dot product of two vectors. c uses unrolled loops for increments equal to one. c jack dongarra, linpack, 3/11/78. c double precision dx(*),dy(*),dtemp integer i,incx,incy,ix,iy,m,mp1,n c ddot = 0.0d0 dtemp = 0.0d0 if(n.le.0)return if(incx.eq.1.and.incy.eq.1)go to 20 c c code for unequal increments or equal increments c not equal to 1 c ix = 1 iy = 1 if(incx.lt.0)ix = (-n+1)*incx + 1 if(incy.lt.0)iy = (-n+1)*incy + 1 do 10 i = 1,n dtemp = dtemp + dx(ix)*dy(iy) ix = ix + incx iy = iy + incy 10 continue ddot = dtemp return c c code for both increments equal to 1 c c c clean-up loop c 20 m = mod(n,5) if( m .eq. 0 ) go to 40 do 30 i = 1,m dtemp = dtemp + dx(i)*dy(i) 30 continue if( n .lt. 5 ) go to 60 40 mp1 = m + 1 do 50 i = mp1,n,5 dtemp = dtemp + dx(i)*dy(i) + dx(i + 1)*dy(i + 1) + * dx(i + 2)*dy(i + 2) + dx(i + 3)*dy(i + 3) + dx(i + 4)*dy(i + 4) 50 continue 60 ddot = dtemp return end c====================== The end of ddot ================================ subroutine dpofa(a,lda,n,info) integer lda,n,info double precision a(lda,*) c c dpofa factors a double precision symmetric positive definite c matrix. c c dpofa is usually called by dpoco, but it can be called c directly with a saving in time if rcond is not needed. c (time for dpoco) = (1 + 18/n)*(time for dpofa) . c c on entry c c a double precision(lda, n) c the symmetric matrix to be factored. only the c diagonal and upper triangle are used. c c lda integer c the leading dimension of the array a . c c n integer c the order of the matrix a . c c on return c c a an upper triangular matrix r so that a = trans(r)*r c where trans(r) is the transpose. c the strict lower triangle is unaltered. c if info .ne. 0 , the factorization is not complete. c c info integer c = 0 for normal return. c = k signals an error condition. the leading minor c of order k is not positive definite. c c linpack. this version dated 08/14/78 . c cleve moler, university of new mexico, argonne national lab. c c subroutines and functions c c blas ddot c fortran sqrt c c internal variables c double precision ddot,t double precision s integer j,jm1,k c begin block with ...exits to 40 c c do 30 j = 1, n info = j s = 0.0d0 jm1 = j - 1 if (jm1 .lt. 1) go to 20 do 10 k = 1, jm1 t = a(k,j) - ddot(k-1,a(1,k),1,a(1,j),1) t = t/a(k,k) a(k,j) = t s = s + t*t 10 continue 20 continue s = a(j,j) - s c ......exit if (s .le. 0.0d0) go to 40 a(j,j) = sqrt(s) 30 continue info = 0 40 continue return end c====================== The end of dpofa =============================== subroutine dscal(n,da,dx,incx) c c scales a vector by a constant. c uses unrolled loops for increment equal to one. c jack dongarra, linpack, 3/11/78. c modified 3/93 to return if incx .le. 0. c double precision da,dx(*) integer i,incx,m,mp1,n,nincx c if( n.le.0 .or. incx.le.0 )return if(incx.eq.1)go to 20 c c code for increment not equal to 1 c nincx = n*incx do 10 i = 1,nincx,incx dx(i) = da*dx(i) 10 continue return c c code for increment equal to 1 c c c clean-up loop c 20 m = mod(n,5) if( m .eq. 0 ) go to 40 do 30 i = 1,m dx(i) = da*dx(i) 30 continue if( n .lt. 5 ) return 40 mp1 = m + 1 do 50 i = mp1,n,5 dx(i) = da*dx(i) dx(i + 1) = da*dx(i + 1) dx(i + 2) = da*dx(i + 2) dx(i + 3) = da*dx(i + 3) dx(i + 4) = da*dx(i + 4) 50 continue return end c====================== The end of dscal =============================== subroutine dtrsl(t,ldt,n,b,job,info) integer ldt,n,job,info double precision t(ldt,*),b(*) c c c dtrsl solves systems of the form c c t * x = b c or c trans(t) * x = b c c where t is a triangular matrix of order n. here trans(t) c denotes the transpose of the matrix t. c c on entry c c t double precision(ldt,n) c t contains the matrix of the system. the zero c elements of the matrix are not referenced, and c the corresponding elements of the array can be c used to store other information. c c ldt integer c ldt is the leading dimension of the array t. c c n integer c n is the order of the system. c c b double precision(n). c b contains the right hand side of the system. c c job integer c job specifies what kind of system is to be solved. c if job is c c 00 solve t*x=b, t lower triangular, c 01 solve t*x=b, t upper triangular, c 10 solve trans(t)*x=b, t lower triangular, c 11 solve trans(t)*x=b, t upper triangular. c c on return c c b b contains the solution, if info .eq. 0. c otherwise b is unaltered. c c info integer c info contains zero if the system is nonsingular. c otherwise info contains the index of c the first zero diagonal element of t. c c linpack. this version dated 08/14/78 . c g. w. stewart, university of maryland, argonne national lab. c c subroutines and functions c c blas daxpy,ddot c fortran mod c c internal variables c double precision ddot,temp integer case,j,jj c c begin block permitting ...exits to 150 c c check for zero diagonal elements. c do 10 info = 1, n c ......exit if (t(info,info) .eq. 0.0d0) go to 150 10 continue info = 0 c c determine the task and go to it. c case = 1 if (mod(job,10) .ne. 0) case = 2 if (mod(job,100)/10 .ne. 0) case = case + 2 go to (20,50,80,110), case c c solve t*x=b for t lower triangular c 20 continue b(1) = b(1)/t(1,1) if (n .lt. 2) go to 40 do 30 j = 2, n temp = -b(j-1) call daxpy(n-j+1,temp,t(j,j-1),1,b(j),1) b(j) = b(j)/t(j,j) 30 continue 40 continue go to 140 c c solve t*x=b for t upper triangular. c 50 continue b(n) = b(n)/t(n,n) if (n .lt. 2) go to 70 do 60 jj = 2, n j = n - jj + 1 temp = -b(j+1) call daxpy(j,temp,t(1,j+1),1,b(1),1) b(j) = b(j)/t(j,j) 60 continue 70 continue go to 140 c c solve trans(t)*x=b for t lower triangular. c 80 continue b(n) = b(n)/t(n,n) if (n .lt. 2) go to 100 do 90 jj = 2, n j = n - jj + 1 b(j) = b(j) - ddot(jj-1,t(j+1,j),1,b(j+1),1) b(j) = b(j)/t(j,j) 90 continue 100 continue go to 140 c c solve trans(t)*x=b for t upper triangular. c 110 continue b(1) = b(1)/t(1,1) if (n .lt. 2) go to 130 do 120 j = 2, n b(j) = b(j) - ddot(j-1,t(1,j),1,b(1),1) b(j) = b(j)/t(j,j) 120 continue 130 continue 140 continue 150 continue return end c====================== The end of dtrsl =============================== gpml-matlab-v3.5-2014-12-08/util/lbfgsb/matlabexception.cpp000664 000765 000024 00000001000 12425212207 023146 0ustar00carlstaff000000 000000 #include "matlabexception.h" // Function definitions for class MatlabException. // ----------------------------------------------------------------- MatlabException::MatlabException (const char* message) throw() : exception() { this->message = message; } MatlabException::MatlabException (const MatlabException& source) throw() : exception() { message = source.message; } MatlabException& MatlabException::operator= (const MatlabException& source) { message = source.message; return *this; } gpml-matlab-v3.5-2014-12-08/util/lbfgsb/arrayofmatrices.cpp000664 000765 000024 00000004741 12425212207 023201 0ustar00carlstaff000000 000000 #ifdef __linux__ #include "mex.h" #else #include "matrix.h" #endif #include "arrayofmatrices.h" // Function definitions for class ArrayOfMatrices. // ----------------------------------------------------------------- ArrayOfMatrices::ArrayOfMatrices (const mxArray* ptr) : Array(getnummatlabmatrices(ptr)) { setvalue(0); if (mxIsCell(ptr)) // Fill in the entries of the array with the entries of the Matlab // cell array. for (int i = 0; i < n; i++) elems[i] = new Matrix(mxGetCell(ptr,i)); else // Fill in the single entry of the array with the Matlab matrix. elems[0] = new Matrix(ptr); } ArrayOfMatrices::ArrayOfMatrices (const mxArray* ptrs[], int numptrs) : Array(numptrs) { setvalue(0); for (int i = 0; i < numptrs; i++) elems[i] = new Matrix(ptrs[i]); } ArrayOfMatrices::ArrayOfMatrices (mxArray* ptrs[], const ArrayOfMatrices& source) : Array(source.length()) { setvalue(0); // Create the new Matlab matrices and copy the data from the // source. for (int i = 0; i < n; i++) { int h = source[i]->height(); int w = source[i]->width(); elems[i] = new Matrix(ptrs[i],h,w); *elems[i] = *source[i]; } } ArrayOfMatrices::ArrayOfMatrices (const ArrayOfMatrices& source) : Array(source) { } ArrayOfMatrices::ArrayOfMatrices (double* data, const ArrayOfMatrices& model) : Array(model.length()) { setvalue(0); // Repeat for each matrix. for (int i = 0; i < n; i++) { const Matrix* A = model[i]; elems[i] = new Matrix(data,A->height(),A->width()); data += A->length(); } } ArrayOfMatrices::~ArrayOfMatrices() { for (int i = 0; i < n; i++) if (elems[i]) delete elems[i]; } int ArrayOfMatrices::numelems() const { int m = 0; // The return value. // Repeat for each matrix. for (int i = 0; i < n; i++) m += elems[i]->length(); return m; } ArrayOfMatrices& ArrayOfMatrices::operator= (const ArrayOfMatrices& source) { // Copy each matrix. for (int i = 0; i < n; i++) *elems[i] = *source.elems[i]; return *this; } int ArrayOfMatrices::getnummatlabmatrices (const mxArray* ptr) { int n; // The return value. if (mxIsCell(ptr)) n = mxGetNumberOfElements(ptr); else if (mxGetNumberOfDimensions(ptr) == 2 && mxIsDouble(ptr)) n = 1; else throw MatlabException("The Matlab array must either be a cell \ array or a double precision matrix"); return n; } gpml-matlab-v3.5-2014-12-08/util/lbfgsb/program.h000664 000765 000024 00000011647 12425212207 021125 0ustar00carlstaff000000 000000 #ifndef INCLUDE_PROGRAM #define INCLUDE_PROGRAM // Type definitions. // ----------------------------------------------------------------- // This defines the possible results of running the L-BFGS-B solver. enum SolverExitStatus { success, // The algorithm has converged to a stationary // point or has reached the maximum number of // iterations. abnormalTermination, // The algorithm has terminated abnormally and // was unable to satisfy the convergence // criteria. errorOnInput // The routine has detected an error in the // input parameters. }; // Constants. // ----------------------------------------------------------------- const int defaultm = 5; const int defaultmaxiter = 100; const double defaultfactr = 1e7; const double defaultpgtol = 1e-5; const int defaultprintlevel = -1; // Function declarations. // ----------------------------------------------------------------- // This is the L-BFGS-B routine implemented in Fortran 77. extern "C" void setulb_ (int* n, int* m, double x[], double l[], double u[], int nbd[], double* f, double g[], double* factr, double* pgtol, double wa[], int iwa[], char task[], int* iprint, char csave[], bool lsave[], int isave[], double dsave[]); // Class Program. // ----------------------------------------------------------------- // This class encapsulates execution of the L-BFGS-B routine for // solving a nonlinear optimization problem with bound constraints // using limited-memory approximations to the Hessian. // // This is an abstract class since some of the class methods have not // been implemented (they are "pure virtual" functions). In order to // use this class, one needs to define a child class and provide // definitions for the pure virtual methods. class Program { public: // The first input argument "n" is the number of variables, and "x" // must be set to the suggested starting point for the optimization // algorithm. See the L-BFGS-B documentation for more information on // the inputs to the constructor. Program (int n, double* x, double* lb, double* ub, int* btype, int m = defaultm, int maxiter = defaultmaxiter, double factr = defaultfactr, double pgtol = defaultpgtol); // This is the same constructor as above, except that the // appropriate amount of memory is dynamically allocated for the // variables, the bounds, and the bound types. The destructor also // makes sure that the memory is properly deallocated, but not so // for the other constructor. Program (int n, int m = defaultm, int maxiter = defaultmaxiter, double factr = defaultfactr, double pgtol = defaultpgtol); // The destructor. virtual ~Program(); // An implementation of this method should return the value of the // objective at the current value of the variables "x", where "n" is // the number of variables. virtual double computeObjective (int n, double* x) = 0; // An implementation of this method should fill in the values of the // gradient "g" and the current point "x". virtual void computeGradient (int n, double* x, double* g) = 0; // The child class may optionally override this method which is // called once per iteration of the L-BFGS-B routine. The input // arguments are the current iteration "t", the current value of the // variables "x" and the current value of the objective "f". virtual void iterCallback (int t, double* x, double f) { }; // Run the solver. Upon completion, the solution is stored in "x". SolverExitStatus runSolver(); protected: // The copy constructor and copy assignment operator are kept // private so that they are not used. Program (const Program& source) { }; Program& operator= (const Program& source) { return *this; }; int n; // The number of variables. double* x; // The current point. double* lb; // The lower bounds. double* ub; // The upper bounds. int* btype; // The bound types. private: // Execute a single step the L-BFGS-B solver routine. void callLBFGS (const char* cmd = 0); // Initialize some of the structures used by the L-BFGS-B routine. void initStructures(); bool owner; // If true, then the blocks of memory pointed to by // x, lb, ub and btype must be deallocated in the // destructor. double f; // The value of the objective. double* g; // The value of the gradient. int iprint; // The print level. int maxiter; // The maximum number of iterations. double pgtol; // Convergence parameter passed to L-BFGS-B. double factr; // Convergence parameter passed to L-BFGS-B. int m; // The number of variable corrections to the // limited-memory approximation to the Hessian. // These are structures used by the L-BFGS-B routine. double* wa; int* iwa; char task[60]; char csave[60]; bool lsave[4]; int isave[44]; double dsave[29]; }; #endif gpml-matlab-v3.5-2014-12-08/util/lbfgsb/matlabscalar.h000664 000765 000024 00000002264 12425212207 022077 0ustar00carlstaff000000 000000 #ifndef INCLUDE_MATLABSCALAR #define INCLUDE_MATLABSCALAR #include "mex.h" // Class MatlabScalar // ----------------------------------------------------------------- // The main appeal of this class is that one can create a scalar // object that accesses a MATLAB array. // // Note that the copy assignment operator is not valid for this class // because we cannot reassign a reference. class MatlabScalar { public: // This constructor accepts as input a pointer to a Matlab array // which must be a scalar in double precision. explicit MatlabScalar (const mxArray* ptr); // This constructor creates a new Matlab array which is a scalar // in double precision. MatlabScalar (mxArray*& ptr, double value); // The copy constructor. MatlabScalar (MatlabScalar& source); // The destructor. ~MatlabScalar() { }; // Access the value of the scalar. operator const double () const { return x; }; // Assign the value of the scalar. MatlabScalar& operator= (double value); protected: double& x; // The copy assignment operator is kept protected because it is // invalid. MatlabScalar& operator= (const MatlabScalar& source) { return *this; }; }; #endif gpml-matlab-v3.5-2014-12-08/util/lbfgsb/matlabmatrix.cpp000664 000765 000024 00000003156 12425212207 022472 0ustar00carlstaff000000 000000 #include "matlabmatrix.h" #include "matlabexception.h" // Function definitions. // ----------------------------------------------------------------- double* getMatlabMatrixDouble (const mxArray* ptr) { if (mxGetNumberOfDimensions(ptr) != 2) throw MatlabException("Matlab array must be a matrix"); if (!mxIsDouble(ptr)) throw MatlabException("Matlab array must be of type double"); return mxGetPr(ptr); } double* createMatlabMatrixDouble (mxArray*& ptr, int height, int width) { ptr = mxCreateDoubleMatrix(height,width,mxREAL); return mxGetPr(ptr); } // Function definitions for class Matrix. // ----------------------------------------------------------------- Matrix::Matrix (int height, int width) : Array(height*width) { h = height; w = width; } Matrix::Matrix (double* data, int height, int width) : Array(data,height*width) { h = height; w = width; } Matrix::Matrix (const mxArray* ptr) : Array(getMatlabMatrixDouble(ptr),mxGetNumberOfElements(ptr)) { h = mxGetM(ptr); w = mxGetN(ptr); } Matrix::Matrix (mxArray*& ptr, int height, int width) : Array(createMatlabMatrixDouble(ptr,height,width), height*width) { h = height; w = width; } Matrix::Matrix (const Matrix& source) : Array(source) { h = source.h; w = source.w; } Matrix& Matrix::operator= (const Matrix& source) { inject(source); return *this; } bool Matrix::operator== (const Matrix& X) const { return (h == X.h) && (w == X.w); } double& Matrix::entry (int r, int c) { return elems[h*c + r]; } double Matrix::entry (int r, int c) const { return elems[h*c + r]; } gpml-matlab-v3.5-2014-12-08/util/lbfgsb/matlabmatrix.h000664 000765 000024 00000004562 12425212207 022141 0ustar00carlstaff000000 000000 #ifndef INCLUDE_MATLABMATRIX #define INCLUDE_MATLABMATRIX #include "array.h" #include "mex.h" // Class Matrix // --------------------------------------------------------------- // A matrix object stores its elements in column-major format, as in // Fortran and Matlab. This means that columns are stored one after // another. For example, the matrix // // 1 2 3 // 4 5 6 // // is stored in memory as // // 1 4 2 5 3 6 // // Like Array objects, Matrix objects are not necessarily // encapsulated, and they exhibit analogous behaviour. class Matrix : public Array { public: // This constructor allocates memory for matrix of the specified // heigth and width. Matrix (int height, int width); // This constructor basically follows the lead of the analagous // constructor for the Array class. Matrix (double* data, int height, int width); // This constructor retrieves a matrix from a Matlab array. This // particular constructor is only defined for Matrix. // Since Matlab handles storage, the object created by this // constructor is not encapsulated. explicit Matrix (const mxArray* ptr); // This constructor creates a new Matlab array as a side effect. // Since Matlab handles storage, the object created by this // constructor is not encapsulated. Matrix (mxArray*& ptr, int height, int width); // The copy constructor makes a shallow copy of the data. Matrix (const Matrix& source); // The destructor. ~Matrix() { }; // Copy assignment operator that observes the same behaviour as // the Array copy assignment operator. Matrix& operator= (const Matrix& source); // Get the height and width of the matrix. int height() const { return h; }; int width () const { return w; }; // Returns true if the two matrices have the same dimensions // (i.e. the same height and width). bool operator== (const Matrix& X) const; bool operator!= (const Matrix& X) const { return !(*this == X); }; // If X is an object of type Matrix, X.entry(r,c) accesses the // entry of the rth row and cth column. double entry (int r, int c) const; double& entry (int r, int c); double operator() (int r, int c) const { return entry(r,c); }; double& operator() (int r, int c) { return entry(r,c); }; protected: int h; // The height of the matrix. int w; // The width of the matrix. }; #endif gpml-matlab-v3.5-2014-12-08/util/lbfgsb/matlabprogram.cpp000664 000765 000024 00000016452 12425212207 022640 0ustar00carlstaff000000 000000 #include "matlabprogram.h" #include "array.h" // Function definitions. // ----------------------------------------------------------------- // Copy the matrix elements to the destination array. It is assumed // that sufficient memory has been allocated for the destination // array. The copying preserves the column-major ordering of the // matrix elements. void copyElemsFromMatrix (const Matrix& source, double* dest) { Matrix destmatrix(dest,source.height(),source.width()); destmatrix = source; } // Copy the elements from the source array into the matrix. void copyElemsToMatrix (const double* source, Matrix& dest) { dest.inject(source); } // Copy the elements contained in all the matrices to the destination // array. It is assumed that sufficient memory has been allocated for // the destination array. The copying preserves the column-major // ordering of the matrix elements. void copyElemsFromArrayOfMatrices (const ArrayOfMatrices& source, double* dest) { // Repeat for each matrix. for (int i = 0; i < source.length(); i++) { copyElemsFromMatrix(*source[i],dest); dest += source[i]->length(); } } // Copy the elements from the source array into the array of matrices. void copyElemsToArrayOfMatrices (const double* source, ArrayOfMatrices& dest) { // Repeat for each matrix. for (int i = 0; i < dest.length(); i++) { copyElemsToMatrix(source,*dest[i]); source += dest[i]->length(); } } int getBoundType (double& lb, double& ub) { bool haslb = !mxIsInf(lb); bool hasub = !mxIsInf(ub); int btype = haslb + 3*hasub - 2*(haslb && hasub); lb = haslb ? lb : 0; ub = hasub ? ub : 0; return btype; } // Function definitions for class MatlabProgram. // ----------------------------------------------------------------- MatlabProgram::MatlabProgram (ArrayOfMatrices& variables, const ArrayOfMatrices& lowerbounds, const ArrayOfMatrices& upperbounds, const MatlabString* objFunc, const MatlabString* gradFunc, const MatlabString* iterFunc, mxArray* auxData, int m, int maxiter, double factr, double pgtol) : Program(variables.numelems(),0,0,0,0,m,maxiter,factr,pgtol), variables(variables) { x = new double[n]; lb = new double[n]; ub = new double[n]; btype = new int[n]; this->objFunc = objFunc; this->gradFunc = gradFunc; this->iterFunc = iterFunc; // Copy some pieces of information from the input arguments to the // respective program structures. copyElemsFromArrayOfMatrices(variables,x); copyElemsFromArrayOfMatrices(lowerbounds,lb); copyElemsFromArrayOfMatrices(upperbounds,ub); // Set the bound types. for (int i = 0; i < n; i++) btype[i] = getBoundType(lb[i],ub[i]); // Create a Matlab array for the variables. int nv = variables.length(); varInputs = new mxArray*[nv]; varMatlab = new ArrayOfMatrices(varInputs,variables); // Set up the inputs to the objective callback function. numInputsObjFunc = nv + (bool) auxData; inputsObjFunc = new mxArray*[numInputsObjFunc]; copymemory(varInputs,inputsObjFunc,nv); if (auxData) inputsObjFunc[nv] = auxData; // Set up the inputs to the gradient callback function. numInputsGradFunc = numInputsObjFunc; inputsGradFunc = inputsObjFunc; // If necessary, set up the inputs to the iterative callback // function. The first two inputs are the iteration and the function // objective. The next of inputs are the values of the variables // then, optionally, the auxiliary data. if (iterFunc) { numInputsIterFunc = 2 + nv + (bool) auxData; inputsIterFunc = new mxArray*[numInputsIterFunc]; mxArray** ptr = inputsIterFunc; tMatlab = new MatlabScalar(*ptr++,0); fMatlab = new MatlabScalar(*ptr++,0); copymemory(varInputs,ptr,nv); ptr += nv; if (auxData) *ptr = auxData; } } MatlabProgram::~MatlabProgram() { delete[] x; delete[] lb; delete[] ub; delete[] btype; // Deallocate the Matlab arrays. int nv = variables.length(); for (int i = 0; i < nv; i++) mxDestroyArray(varInputs[i]); delete varMatlab; delete[] varInputs; // If necessary, deallocate the additional Matlab arrays that act as // inputs to the iterative callback function. if (iterFunc) { delete fMatlab; delete tMatlab; // delete[] inputsIterFunc; } } double MatlabProgram::computeObjective (int n, double* x) { int nlhs = 1; // The number of outputs from Matlab. mxArray* plhs[nlhs]; // The outputs from the Matlab routine. // Copy the current value of the optimization variables. copyElemsToArrayOfMatrices(x,*varMatlab); // Call the designated Matlab routine for evaluating the objective // function. It takes as input the values of the variables, and // returns a single output, the value of the objective function. if (mexCallMATLAB(nlhs,plhs,numInputsObjFunc,inputsObjFunc,*objFunc)) throw MatlabException("Evaluation of objective function failed in \ call to MATLAB routine"); // Get the result passed back from the Matlab routine. MatlabScalar matlabOutput(plhs[0]); double objective = matlabOutput; // Free the dynamically allocated memory. mxDestroyArray(plhs[0]); return objective; } void MatlabProgram::computeGradient (int n, double* x, double* g) { int nlhs = variables.length(); // The number of outputs from Matlab. mxArray* plhs[nlhs]; // The outputs from the Matlab routine. // Copy the current value of the optimization variables. copyElemsToArrayOfMatrices(x,*varMatlab); // Call the designated Matlab routine for computing the gradient // of the objective function. It takes as input the values of the // variables, and returns as many outputs corresponding to the // partial derivatives of the objective function with respect to // the variables at their curent values. if (mexCallMATLAB(nlhs,plhs,numInputsGradFunc,inputsGradFunc,*gradFunc)) throw MatlabException("Evaluation of objective gradient failed in \ call to MATLAB routine"); // Get the result passed back from the Matlab routine. ArrayOfMatrices matlabOutput((const mxArray**) plhs,nlhs); for (int i = 0; i < variables.length(); i++) if (matlabOutput[i]->length() != variables[i]->length()) throw MatlabException("Invalid gradient passed back from MATLAB \ routine"); copyElemsFromArrayOfMatrices(matlabOutput,g); // Free the dynamically allocated Matlab outputs. for (int i = 0; i < variables.length(); i++) mxDestroyArray(plhs[i]); } void MatlabProgram::iterCallback (int t, double* x, double f) { if (iterFunc) { // Copy the current iteration, the current value of the objective // function, and the current value of the optimization variables. *tMatlab = t; *fMatlab = f; copyElemsToArrayOfMatrices(x,*varMatlab); // Call the Matlab iterative callback routine. Since there are no // outputs, there is no memory to deallocate. if (mexCallMATLAB(0,0,numInputsIterFunc,inputsIterFunc,*iterFunc)) throw MatlabException("Call to MATLAB iterative callback function \ failed"); } } SolverExitStatus MatlabProgram::runSolver() { SolverExitStatus status; // The return value. status = Program::runSolver(); // Copy the solution to the class member "variables". copyElemsToArrayOfMatrices(x,variables); return status; } gpml-matlab-v3.5-2014-12-08/mean/meanConst.m000664 000765 000024 00000001217 12425212207 020106 0ustar00carlstaff000000 000000 function A = meanConst(hyp, x, i) % Constant mean function. The mean function is parameterized as: % % m(x) = c % % The hyperparameter is: % % hyp = [ c ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-08-04. % % See also MEANFUNCTIONS.M. if nargin<2, A = '1'; return; end % report number of hyperparameters if numel(hyp)~=1, error('Exactly one hyperparameter needed.'), end c = hyp; if nargin==2 A = c*ones(size(x,1),1); % evaluate mean else if i==1 A = ones(size(x,1),1); % derivative else A = zeros(size(x,1),1); end endgpml-matlab-v3.5-2014-12-08/mean/meanSum.m000664 000765 000024 00000003511 12425212207 017563 0ustar00carlstaff000000 000000 function A = meanSum(mean, hyp, x, i) % meanSum - compose a mean function as the sum of other mean functions. % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls other mean functions to do the actual work. % % m(x) = \sum_i m_i(x) % % Copyright (c) by Carl Edward Rasmussen & Hannes Nickisch 2010-08-04. % % See also MEANFUNCTIONS.M. for ii = 1:numel(mean) % iterate over mean functions f = mean(ii); if iscell(f{:}), f = f{:}; end % expand cell array if necessary j(ii) = cellstr(feval(f{:})); % collect number hypers end if nargin<3 % report number of parameters A = char(j(1)); for ii=2:length(mean), A = [A, '+', char(j(ii))]; end; return end [n,D] = size(x); v = []; % v vector indicates to which mean parameters belong for ii = 1:length(mean), v = [v repmat(ii, 1, eval(char(j(ii))))]; end if nargin==3 % compute mean vector A = zeros(n,1); % allocate space for ii = 1:length(mean) % iteration over summand functions f = mean(ii); if iscell(f{:}), f = f{:}; end % expand cell array if needed A = A + feval(f{:}, hyp(v==ii), x); % accumulate means end else % compute derivative vector if i<=length(v) ii = v(i); % which mean function j = sum(v(1:i)==ii); % which parameter in that mean f = mean(ii); if iscell(f{:}), f = f{:}; end % dereference cell array if necessary A = feval(f{:}, hyp(v==ii), x, j); % compute derivative else A = zeros(n,1); end endgpml-matlab-v3.5-2014-12-08/mean/meanPoly.m000664 000765 000024 00000002436 12425212207 017747 0ustar00carlstaff000000 000000 function A = meanPoly(d, hyp, x, i) % meanPoly - compose a mean function as a polynomial. % % The degree d has to be a strictly positive integer. % % m(x) = sum_i=1..D sum_j=1..d a_ij * x_i^j % % The hyperparameter is: % % hyp = [ a_11 % a_21 % .. % a_D1 % a_12 % a_22 % .. % a_Dd] % % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls other mean function to do the actual work. % % Copyright (c) by Hannes Nickisch 2013-11-02. % % See also MEANFUNCTIONS.M. d = max(abs(floor(d)),1); % positive integer degree if nargin<3, A = ['D*',int2str(d)]; return; end % report number of hyperparams [n,D] = size(x); a = reshape(hyp,D,d); A = zeros(n,1); % allocate memory if nargin==3 % compute mean vector for i=1:d, A = A + (x.^i)*a(:,i); end % evaluate mean else % compute derivative vector if i<=d*D j = mod(i-1,D)+1; % which dimension A = x(:,j).^((i-j)/D+1); % derivative else A = zeros(n,1); end endgpml-matlab-v3.5-2014-12-08/mean/meanPref.m000664 000765 000024 00000001414 12425363740 017724 0ustar00carlstaff000000 000000 function A = meanPref(mean, hyp, x, varargin) % meanPref - mean function for preference learning. % % m(x) = m_0(x1)-m_0(x2), where x1=x(:,1:D), x2=x(:,D+1:2*D), D = size(x,2)/2. % % The hyperparameters are: % % hyp = [ hyp_m0 ] % % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls another mean function to do the actual work. % % See Collaborative Gaussian Processes for Preference Learning, NIPS 2014. % % Copyright (c) by Hannes Nickisch and Roman Garnett, 2014-11-01. % % See also MEANFUNCTIONS.M and COVPREF.M. if nargin<3, A = strrep(feval(mean{:}),'D','D/2'); return; end % no of params A = feval(mean{:}, hyp, x(:,1 :end/2), varargin{:}) ... - feval(mean{:}, hyp, x(:,1+end/2:end ), varargin{:}); gpml-matlab-v3.5-2014-12-08/mean/meanZero.m000664 000765 000024 00000000603 12425212207 017735 0ustar00carlstaff000000 000000 function A = meanZero(hyp, x, i) % Zero mean function. The mean function does not have any parameters. % % m(x) = 0 % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-01-10. % % See also MEANFUNCTIONS.M. if nargin<2, A = '0'; return; end % report number of hyperparameters A = zeros(size(x,1),1); % derivative and mean gpml-matlab-v3.5-2014-12-08/mean/meanPow.m000664 000765 000024 00000001713 12425212207 017566 0ustar00carlstaff000000 000000 function A = meanPow(d, mean, hyp, x, i) % meanPow - compose a mean function as the power of another mean function m0. % % The degree d has to be a strictly positive integer. % % m(x) = m_0(x) ^ d % % The hyperparameter is: % % hyp = [ ] % % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls other mean function to do the actual work. % % Copyright (c) by Carl Edward Rasmussen & Hannes Nickisch 2013-11-02. % % See also MEANFUNCTIONS.M. d = max(abs(floor(d)),1); % positive integer degree only if nargin<4 % report number of parameters A = feval(mean{:}); return end [n,D] = size(x); if nargin==4 % compute mean vector A = feval(mean{:},hyp,x).^d; else % compute derivative vector A = ( d*feval(mean{:},hyp,x).^(d-1) ).* feval(mean{:},hyp,x,i); endgpml-matlab-v3.5-2014-12-08/mean/meanMask.m000664 000765 000024 00000002136 12425212207 017714 0ustar00carlstaff000000 000000 function A = meanMask(mask, mean, hyp, x, i) % Apply a mean function to a subset of the dimensions only. The subset can % either be specified by a 0/1 mask by a boolean mask or by an index set. % % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls another mean function to do the actual work. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-11-02. % % See also MEANFUNCTIONS.M. nh_string = feval(mean{:}); % number of hyperparameters of the full mean if max(mask)<2 && length(mask)>1, mask = find(mask); end % convert 1/0->index D = length(mask); % masked dimension if nargin<4, A = num2str(eval(nh_string)); return; end % number of parameters if eval(nh_string)~=length(hyp) % check hyperparameters error('number of hyperparameters does not match size of masked data') end if nargin==4 A = feval(mean{:}, hyp, x(:,mask)); else % compute derivative A = feval(mean{:}, hyp, x(:,mask), i); end gpml-matlab-v3.5-2014-12-08/mean/meanLinear.m000664 000765 000024 00000001333 12425212207 020231 0ustar00carlstaff000000 000000 function A = meanLinear(hyp, x, i) % Linear mean function. The mean function is parameterized as: % % m(x) = sum_i a_i * x_i; % % The hyperparameter is: % % hyp = [ a_1 % a_2 % .. % a_D ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-01-10. % % See also MEANFUNCTIONS.M. if nargin<2, A = 'D'; return; end % report number of hyperparameters [n,D] = size(x); if any(size(hyp)~=[D,1]), error('Exactly D hyperparameters needed.'), end a = hyp(:); if nargin==2 A = x*a; % evaluate mean else if i<=D A = x(:,i); % derivative else A = zeros(n,1); end endgpml-matlab-v3.5-2014-12-08/mean/meanOne.m000664 000765 000024 00000000746 12425212207 017547 0ustar00carlstaff000000 000000 function A = meanOne(hyp, x, i) % One mean function. The mean function does not have any parameters. % % m(x) = 1 % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-08-04. % % See also MEANFUNCTIONS.M. if nargin<2, A = '0'; return; end % report number of hyperparameters if nargin==2 A = ones(size(x,1),1); % evaluate mean else A = zeros(size(x,1),1); % derivative endgpml-matlab-v3.5-2014-12-08/mean/meanGPexact.m000664 000765 000024 00000004547 12425363641 020375 0ustar00carlstaff000000 000000 function A = meanGPexact(mean,cov,x,y, hypz,z,i) % Mean function being the predictive mean of a GP model: % % m(z) = posterior mean of GP at location z as given by % m(z) = gp(hyp,@infExact,mean,cov,@likGauss,x,y, z) where % hyp.mean = hyp_mean; hyp.lik = log(sn); hyp.cov = hyp.cov; % % The hyperparameters are: % % hypz = [ hyp_cov % log(sn) % hyp_mean ] % % where hyp_cov are the covariance function hyperparameters, sn is the % noise variance of the Gaussian likelihood and hyp_mean are the mean % function hyperparameters. % % Copyright (c) by Hannes Nickisch, 2014-11-01. % % See also MEANFUNCTIONS.M and MEANGP.M. if nargin<4, error('GP must be specified.'), end % check for dimension if isempty(mean), mean = @meanZero; end % set default and make cell if ~iscell(mean), mean = {mean}; end if isempty(cov), cov = @covSEiso; end if ~iscell(cov), cov = {cov}; end nms = feval(mean{:}); ncs = feval(cov{:}); % number of hyperparameter string if nargin<6, A = [ncs,'+1+',nms]; return, end % report number of hyperparameters [nz,D] = size(z); n = size(x,1); nc = eval(ncs); nm = eval(nms); hyp = rewrap(struct('cov',zeros(nc,1),'lik',0,'mean',zeros(nm,1)),hypz); m = feval(mean{:},hyp.mean,x); mz = feval(mean{:},hyp.mean,z); K = feval(cov{:}, hyp.cov, x); kz = feval(cov{:}, hyp.cov, x,z); sn2 = exp(2*hyp.lik); % noise variance of likGauss if sn2<1e-6 % very tiny sn2 can lead to numerical trouble L = chol(K+sn2*eye(n)); sl = 1; % Cholesky factor of covariance with noise else L = chol(K/sn2+eye(n)); sl = sn2; % Cholesky factor of B end iKs = @(t) solve_chol(L,t)/sl; % iKs(t) = (K+sn2*eye(n))\t alpha = iKs(y-m); if nargin==6 % eval posterior mean A = mz+kz'*alpha; else if i<=nc % covariance function hypers dK = feval(cov{:},hyp.cov,x,[],i); dkz = feval(cov{:},hyp.cov,x,z,i); A = dkz'*alpha-kz'*iKs(dK*alpha); elseif i==nc+1 % likelihood function parameter sn A = -2*sn2*kz'*iKs(alpha); else % mean function parameter dm = feval(mean{:},hyp.mean,x,i-nc-1); dmz = feval(mean{:},hyp.mean,z,i-nc-1); A = dmz-kz'*iKs(dm); end end gpml-matlab-v3.5-2014-12-08/mean/meanGP.m000664 000765 000024 00000001307 12425363364 017341 0ustar00carlstaff000000 000000 function A = meanGP(hyp,inf,mean,cov,lik,x,y, hypz,z,i) % Mean function being the predictive mean of a GP model: % % m(z) = posterior mean of another GP at location z as given by % m(z) = gp(hyp,inf,mean,cov,lik,x,y, z) % % The hyperparameters are: % % hypz = [ ] % % Copyright (c) by Hannes Nickisch, 2014-11-01. % % See also MEANFUNCTIONS.M and MEANGPEXACT.M. if nargin<7, error('GP must be specified.'), end % check for dimension if nargin<9, A = '0'; return, end % report number of hyperparameters if nargin==9 A = gp(hyp,inf,mean,cov,lik,x,y, z); % evaluate posterior mean else A = zeros(size(x,1),1); % derivative end gpml-matlab-v3.5-2014-12-08/mean/meanDiscrete.m000664 000765 000024 00000001633 12425212207 020564 0ustar00carlstaff000000 000000 function A = meanDiscrete(s, hyp, x, i) % Mean function for discrete inputs x. Given a function defined on the % integers 1,2,3,..,s, the mean function is parametrized as: % % m(x) = mu_x, % % where mu is a fixed vector of length s. % % This implementation assumes that the inputs x are given as integers % between 1 and s, which simply index the provided vector. % % The hyperparameters are: % % hyp = [ mu_1 % mu_2 % .. % mu_s ] % % Copyright (c) by Roman Garnett, 2014-08-14. % % See also COVDISCRETE.M, MEANFUNCTIONS.M. if nargin==0, error('s must be specified.'), end % check for dimension if nargin<=2, A = num2str(s); return; end % report number of hyperparameters mu = hyp(:); if nargin==3 A = mu(x(:)); % evaluate mean else A = zeros(numel(x),1); % derivative A(x==i) = 1; endgpml-matlab-v3.5-2014-12-08/mean/meanNN.m000664 000765 000024 00000001414 12440011705 017327 0ustar00carlstaff000000 000000 function A = meanNN(c,m, hyp, x, i) % Nearest neighbor mean function. The mean function is parameterized as: % % m(z) = m_j, j = arg min_i d(ci,x) where d is the Euclidean distance and ci is % the ith cluster center. % % The hyperparameters are: % % hyp = [ ] % % Copyright (c) by Hannes Nickisch, 2014-12-04. % % See also MEANFUNCTIONS.M. if nargin<4, A = '0'; return; end % report number of hyperparameters if numel(hyp)~=0, error('No hyperparameters needed for this model.'), end if nargin==4 % evaluate mean [junk,j] = min(sq_dist(c',x')); A = m(j); A = A(:); else A = zeros(size(x,1),1); % derivative endgpml-matlab-v3.5-2014-12-08/mean/meanProd.m000664 000765 000024 00000003754 12425212207 017734 0ustar00carlstaff000000 000000 function A = meanProd(mean, hyp, x, i) % meanProd - compose a mean function as the product of other mean functions. % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls other mean functions to do the actual work. % % m(x) = \prod_i m_i(x) % % Copyright (c) by Carl Edward Rasmussen & Hannes Nickisch 2010-08-04. % % See also MEANFUNCTIONS.M. for ii = 1:numel(mean) % iterate over mean functions f = mean(ii); if iscell(f{:}), f = f{:}; end % expand cell array if necessary j(ii) = cellstr(feval(f{:})); % collect number hypers end if nargin<3 % report number of parameters A = char(j(1)); for ii=2:length(mean), A = [A, '+', char(j(ii))]; end; return end [n,D] = size(x); v = []; % v vector indicates to which mean parameters belong for ii = 1:length(mean), v = [v repmat(ii, 1, eval(char(j(ii))))]; end A = ones(n,1); % allocate space if nargin==3 % compute mean vector for ii = 1:length(mean) % iteration over summand functions f = mean(ii); if iscell(f{:}), f = f{:}; end % expand cell array if needed A = A.*feval(f{:}, hyp(v==ii), x); % accumulate means end else % compute derivative vector if i<=length(v) ii = v(i); % which mean function j = sum(v(1:i)==ii); % which parameter in that mean for jj = 1:length(mean) f = mean(jj); if iscell(f{:}), f = f{:}; end % dereference cell array if necessary if jj==ii A = A .* feval(f{:}, hyp(v==jj), x, j); % multiply derivative else A = A .* feval(f{:}, hyp(v==jj), x); % multiply mean end end else A = zeros(n,1); end endgpml-matlab-v3.5-2014-12-08/mean/meanScale.m000664 000765 000024 00000001622 12425223376 020060 0ustar00carlstaff000000 000000 function A = meanScale(mean, hyp, x, i) % meanScale - compose a mean function as a scaled version of another one. % % m(x) = a * m_0(x) % % The hyperparameters are: % % hyp = [ a; % hyp_m0 ] % % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls other mean functions to do the actual work. % % Copyright (c) by Carl Edward Rasmussen & Hannes Nickisch 2014-11-01. % % See also MEANFUNCTIONS.M. if nargin<3 % report number of parameters A = [feval(mean{:}),'+1']; return end [n,D] = size(x); a = hyp(1); if nargin==3 % compute mean vector A = a*feval(mean{:},hyp(2:end),x); else % compute derivative vector if i==1 A = feval(mean{:},hyp(2:end),x); else A = a*feval(mean{:},hyp(2:end),x,i-1); end end