inf/diaginv_lanczos.m0000644000175000017500000000702211425740436012765 0ustar hnhn% [z,Q,T] = DIAGINV_LANCZOS(X,R,B,P,kmax,output) % Approximates diag( B*inv(A)*B' ) with A = X'*R*X + B'*P*B through % Krylov subspace estimation via Lanczos iterations using full Gram-Schmidt % orthogonalisation in every step. % % ARGUMENTS % X [m,n] matrix or operator, m can be zero % R [m,m] square matrix % or [m,1] vector of diagonal R = diag(R(:)) % or [1,1] scalar multiple of identity R = R*eye(n) % % B [q,n] matrix or operator % P [n,n] square matrix % or [m,1] vector of diagonal P = diag(P(:)) % or [1,1] scalar multiple of identity P = P*eye(n) % % kmax [1,1] number of Lanczos vectors, kmax <= n [default 100] % output [1,1] flag saying whether some output is shown [default false] % % RESULTS % z [q,1] vector 0 <= z <= diag(B*inv(A)*B'), A = X'*R*X + B'*P*B % Q [n,k] orthogonal matrix of Lanczos vectors, Q'*Q = I % T [k,k] tridiagonal symmetric matrix, T = Q'*A*Q % % The eigensystem of T given by [W,D] = eig(T) can be used to approximate the % eigensystem of A because [Q*W,D] converges to an eigensystem of A. % As a result, inv(A) can be approximated by Q*inv(T)*Q'. % % The matrices X and B can be given implicitely through their multiplication. % % The algorithm is from Schneider & Willsky 2000: Krylov subspace estimation. % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 August 03 function [z,Q,T] = diaginv_lanczos(X,R,B,P,kmax,output) n = max(size(X,2),size(B,2)); if nargin<5, kmax = 100; end, kmax = min(n,kmax); % max. no of MVMs if nargin<6, output = 0; else zold = 0; end w = randn(n,1); w = w/norm(w,'fro'); % random starting vector % $9.2.1 from Golub & van Loan: only v, w and t are needed as storage of size n alpha = zeros(1,kmax); beta = zeros(1,kmax-1); % for T v = zeros(n,1); b = 1; k = 0; Q = zeros(n,kmax); z = 0; % memory for the Lanczos and result while abs(b)>1e-10 && k0, t = w; w = v/b; v = -b*t; end w(:) = w(:) - Q*(Q'*w(:)); % Gram-Schmidt orthogonalisation v = v + mvmA(X,R,B,P, w); % MVM with A: v = v + A*w k = k+1; a = real(w(:)'*v(:)); alpha(k) = a; % needed for T v = v - a*w; if k==1 Lkk = sqrt(a); % L(k,k), L = chol(T)' <=> L*L' = T p = w/Lkk; % Q(:,k) = w else Lkk1 = b/Lkk; % L(k,k-1) Lkk = sqrt( a - Lkk1*Lkk1 ); % L(k,k) p = (w-Lkk1*p)/Lkk; end b = norm(v,'fro'); if k2, T = diag(alpha)+diag(beta,+1)+diag(beta,-1); end function v = mvmA(X,R,B,P, w) % MVM with A where A = X'*R*X + B'*P*B Xw = X*w; if numel(R)==numel(Xw), RXw = R(:).*Xw; else RXw = R*Xw; end Bw = B*w; if numel(P)==numel(Bw), PBw = P(:).*Bw; else PBw = P*Bw; end v = [X']*RXw + [B']*PBw; % the [] around the transpose is needed by Octaveinf/diaginv_woodbury.m0000644000175000017500000000507611425743226013175 0ustar hnhn% z = DIAGINV_WOODBURY(X,R,P) % % Computes z = diag(inv(A)) with A = X'*R*X + P where X is an m-by-n matrix and % both R and P are positive definite. % % We use the Woodbury matrix identity % inv(X'*R*X + P) = inv(P) - inv(P)*X'*inv( inv(R)+X*inv(P)*X' )*X*inv(P) % whenever mO(M^3) where N=max(m,n) and M=min(m,n) % due to the Cholesky decomposition on a matrix of size MxM and a matrix-matrix % multiplication X'*X or X*X'. % % ARGUMENTS % X [m,n] full or sparse matrix, no operator allowed % % R [m,m] square matrix % or [m,1] vector of diagonal R = diag(R(:)) % or [1,1] scalar multiple of identity R = R*eye(n) % % P [n,n] square matrix % or [m,1] vector of diagonal P = diag(P(:)) % or [1,1] scalar multiple of identity P = P*eye(n) % % RESULTS % z [n,1] vector z = diag( inv(A) ), A = X'*R*X + P % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 August 03 function z = diaginv_woodbury(X,R,P) [m,n] = size(X); if m=0)-1; % strict sign mapping 0 to 1 r = sign_s.*sqrt(s.*s + z + eps); % signed stabilised z-smoothed variable [lp,dlp,ignore,b] = cols( pot(tau.*r) ); dlp = tau.*dlp; b = tau.*b; pi = abs( (b-dlp)./r ); % OUTER loop: marginal variances if incol, varargout{n}=[]; else varargout{n}=A(:,n); end,endmat/@catmat/catmat.m0000644000175000017500000000066111420141470012430 0ustar hnhn% concatenation of two matrices: constructor % % A = fd2mat(B,C) % % B [m,p]: operator % C [n,p]: operator % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 15 function A = catmat(B,C) A.B = B; A.C = C; if size(B,2)~=size(C,2), error('B and C need the same number of columns'), end sz = [size(B,1)+size(C,1),size(B,2)]; A.sz = sz; A.ctransp = 0; A = class(A,mfilename); % [generic]mat/@catmat/ctranspose.m0000644000175000017500000000032211430167323013340 0ustar hnhn% complex conjugate transpose [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function A = ctranspose(A) A.ctransp = xor(A.ctransp,1);mat/@catmat/disp.m0000644000175000017500000000033111430167323012116 0ustar hnhn% display something [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function disp(A) fprintf('<%dx%d %s>\n', size(A,1), size(A,2), class(A));mat/@catmat/full.m0000644000175000017500000000042211430167323012122 0ustar hnhn% convert to full matrix [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function B = full(A) [m,n] = size(A); B = zeros(m,n); ei = zeros(n,1); for i=1:n, ei(i) = 1; B(:,i) = A*ei; ei(i) = 0; end mat/@catmat/isempty.m0000644000175000017500000000032011430167323012647 0ustar hnhn% say whether matrix is empty [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function y = isempty(A) [m,n] = size(A); y = m*n==0;mat/@catmat/mtimes.m0000644000175000017500000000045211420141470012453 0ustar hnhn% concatenation of two matrices: matrix vector multiplication % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 16 function y = mtimes(A,x) if A.ctransp nb = size(A.B,1); Bt = A.B'; Ct = A.C'; y = Bt*x(1:nb) + Ct*x(nb+1:end); else y = [A.B*x; A.C*x]; endmat/@catmat/numel.m0000644000175000017500000000031211430167323012276 0ustar hnhn% number of elements in matrix [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function N = numel(A) [m,n] = size(A); N = n*m;mat/@catmat/size.m0000644000175000017500000000101511430167323012131 0ustar hnhn% return size [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function [sz1,sz2] = size(A,id) sz = A.sz; if A.ctransp, sz = fliplr(sz); end if nargin==1 if nargout==2 sz1 = sz(1); sz2 = sz(2); else sz1 = sz; end elseif nargin==2 if nargout>1, error('too many output arguments'), end if id(1)>numel(sz) sz1 = 1; else sz1 = sz(id); end else error('wrong number of input arguments') endmat/@catmat/transpose.m0000644000175000017500000000030011430167323013171 0ustar hnhn% transpose [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function A = transpose(A) A.ctransp = xor(A.ctransp,1); mat/@cnv2mat/cnv2mat.m0000644000175000017500000000261711422527031012640 0ustar hnhnfunction A = cnv2mat(f, sx, shape) %CNV2MAT creates a matrix object for 2d convolutions. % % Inputs: % f the left-hand-side of the convolution % sx the size of the right-hand-side of the convolution % shape one of 'circ', 'same' or 'valid' % % Example of usage: % F = cnv2mat(f, sx, 'same'); % y = F*x; % same as y = imfilter(x, f, 'same'); % % Michael Hirsch * 20/07/2010 sf = size(f); % size of the psf, not of the matrix if any(sf < sx) && any(sf > sx) error('[%s.m] size missmatch', mfilename); end sF = max(sx, sf); % the bigger one sw = min(sx, sf); % the smaller one switch shape case 'circ' sy = sF; case 'valid' sy = sF - sw + 1; case 'same' sy = sF; sF = sx + sf - 1; otherwise error('[cnv2mat.m] Unknown shape. Choose between ''circ'', ''same'' and ''valid'''); end % precompute fft already F = fftn(f, sF); % calculate its size sz = [prod(sy), prod(sx)]; % store all stuff in the structure A.shape = shape; % shape of the convolution A.f = f; % PSF A.F = F; % precomputed FFT A.sx = sx; % input dimension A.sy = sy; % output dimension A.sf = sf; % size of the PSF A.sF = sF; % size of the FFT A.sw = sw; % size of the PSF A.sz = sz; A.ctransp = 0; A = class(A,mfilename); % [generic]mat/@cnv2mat/ctranspose.m0000644000175000017500000000032211430167323013441 0ustar hnhn% complex conjugate transpose [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function A = ctranspose(A) A.ctransp = xor(A.ctransp,1);mat/@cnv2mat/disp.m0000644000175000017500000000033111430167323012217 0ustar hnhn% display something [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function disp(A) fprintf('<%dx%d %s>\n', size(A,1), size(A,2), class(A));mat/@cnv2mat/full.m0000644000175000017500000000042211430167323012223 0ustar hnhn% convert to full matrix [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function B = full(A) [m,n] = size(A); B = zeros(m,n); ei = zeros(n,1); for i=1:n, ei(i) = 1; B(:,i) = A*ei; ei(i) = 0; end mat/@cnv2mat/get_sin.m0000644000175000017500000000014011422527031012703 0ustar hnhnfunction sx = get_sin(obj) if obj.ctransp % transposed sx = obj.sy; else sx = obj.sx; end mat/@cnv2mat/get_sout.m0000644000175000017500000000015011422527031013105 0ustar hnhnfunction sout = get_sout(obj) if obj.ctransp % transposed sout = obj.sx; else sout = obj.sy; end mat/@cnv2mat/isempty.m0000644000175000017500000000032011430167323012750 0ustar hnhn% say whether matrix is empty [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function y = isempty(A) [m,n] = size(A); y = m*n==0;mat/@cnv2mat/mtimes.m0000644000175000017500000000206311422527031012557 0ustar hnhnfunction y = mtimes(A, x) sx = A.sx; sy = A.sy; sF = A.sF; sw = A.sw; % sizes if A.ctransp % transposed x = reshape(x,sy); switch A.shape case 'circ' x = circshift(x, floor(sw/2)); case 'same' sw2 = floor(sw/2); xx = zeros(sF); xx(sw2(1)+(1:sy(1)), sw2(2)+(1:sy(2))) = x; x = xx; case 'valid' x = [zeros(sw-1), zeros(sw(1)-1, sy(2)); zeros(sy(1), sw(2)-1), x]; % zero padding end y = ifftn(fftn(x) .* conj(A.F)); y = y(1:sx(1), 1:sx(2)); else % not transposed x = reshape(x,sx); y = ifftn(fftn(x, sF).* A.F); switch A.shape case 'circ' y = circshift(y, -floor(sw/2)); case 'same' sw2 = floor(sw/2); y = y((sw2(1)+1):end, (sw2(2)+1):end); y = y(1:sy(1), 1:sy(2)); case 'valid' y = y(sw(1):end, sw(2):end); end end y = real(y(:));mat/@cnv2mat/numel.m0000644000175000017500000000031211430167323012377 0ustar hnhn% number of elements in matrix [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function N = numel(A) [m,n] = size(A); N = n*m;mat/@cnv2mat/size.m0000644000175000017500000000101511430167323012232 0ustar hnhn% return size [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function [sz1,sz2] = size(A,id) sz = A.sz; if A.ctransp, sz = fliplr(sz); end if nargin==1 if nargout==2 sz1 = sz(1); sz2 = sz(2); else sz1 = sz; end elseif nargin==2 if nargout>1, error('too many output arguments'), end if id(1)>numel(sz) sz1 = 1; else sz1 = sz(id); end else error('wrong number of input arguments') endmat/@cnv2mat/transpose.m0000644000175000017500000000030011430167323013272 0ustar hnhn% transpose [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function A = transpose(A) A.ctransp = xor(A.ctransp,1); mat/@fftline2mat/ctranspose.m0000644000175000017500000000032211430167323014302 0ustar hnhn% complex conjugate transpose [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function A = ctranspose(A) A.ctransp = xor(A.ctransp,1);mat/@fftline2mat/disp.m0000644000175000017500000000033111430167323013060 0ustar hnhn% display something [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function disp(A) fprintf('<%dx%d %s>\n', size(A,1), size(A,2), class(A));mat/@fftline2mat/fftline2mat.m0000644000175000017500000000167011420141471014336 0ustar hnhn% 2d partial FFT matrix selecting rows: constructor % % A = fftline2mat(sz,id,complex) % % sz [1,2]: size of the image to transform % id [1,p]: row indices to keep, default id=1:sz(1) % complex [1,1]: flag saying whether we have complex output, default is true % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 15 % % The transform is NOT centered. function A = fftline2mat(sz,id,complex) A.imsz = sz; if numel(A.imsz) == 1, A.imsz = A.imsz*[1,1]; end % image size if nargin>1 id(id<1) = 1; id(id>A.imsz(1)) = A.imsz(1); % put into interval A.id = fix(id(:)); % subset of 1..imsz(1), double is ok else A.id = 1:A.sz(1); end sz = [numel(A.id)*A.imsz(2),prod(A.imsz)]; % matrix size if nargin>2 A.complex = complex; else A.complex = true; end A.sz = sz; A.ctransp = 0; A = class(A,mfilename); % [generic]mat/@fftline2mat/full.m0000644000175000017500000000042211430167323013064 0ustar hnhn% convert to full matrix [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function B = full(A) [m,n] = size(A); B = zeros(m,n); ei = zeros(n,1); for i=1:n, ei(i) = 1; B(:,i) = A*ei; ei(i) = 0; end mat/@fftline2mat/isempty.m0000644000175000017500000000032011430167323013611 0ustar hnhn% say whether matrix is empty [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function y = isempty(A) [m,n] = size(A); y = m*n==0;mat/@fftline2mat/mtimes.m0000644000175000017500000000076111420141471013421 0ustar hnhn% 2d partial FFT matrix selecting rows: matrix vector multiplication % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 15 function y = mtimes(A,x) if A.ctransp y = zeros(A.imsz); y(A.id,:) = reshape(x,[numel(A.id),A.imsz(2)]); y = ifft2(ifftshift(y))*sqrt(prod(A.imsz)); if ~A.complex, y = real(y); end else x = reshape(x,A.imsz); if ~A.complex, x = real(x); end y = fftshift( fft2(x)/sqrt(prod(A.imsz)) ); y = y(A.id,:); end y = y(:);mat/@fftline2mat/numel.m0000644000175000017500000000031211430167323013240 0ustar hnhn% number of elements in matrix [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function N = numel(A) [m,n] = size(A); N = n*m;mat/@fftline2mat/size.m0000644000175000017500000000101511430167323013073 0ustar hnhn% return size [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function [sz1,sz2] = size(A,id) sz = A.sz; if A.ctransp, sz = fliplr(sz); end if nargin==1 if nargout==2 sz1 = sz(1); sz2 = sz(2); else sz1 = sz; end elseif nargin==2 if nargout>1, error('too many output arguments'), end if id(1)>numel(sz) sz1 = 1; else sz1 = sz(id); end else error('wrong number of input arguments') endmat/@fftline2mat/transpose.m0000644000175000017500000000030011430167323014133 0ustar hnhn% transpose [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function A = transpose(A) A.ctransp = xor(A.ctransp,1); mat/@fftmask2mat/ctranspose.m0000644000175000017500000000032211430167323014306 0ustar hnhn% complex conjugate transpose [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function A = ctranspose(A) A.ctransp = xor(A.ctransp,1);mat/@fftmask2mat/disp.m0000644000175000017500000000033111430167323013064 0ustar hnhn% display something [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function disp(A) fprintf('<%dx%d %s>\n', size(A,1), size(A,2), class(A));mat/@fftmask2mat/fftmask2mat.m0000644000175000017500000000124711420141465014351 0ustar hnhn% 2d partial FFT matrix: constructor % % A = fftmask2mat(sz,id,complex) % % mask [m,n]: indices to keep, default id=1:sz(1) % complex [1,1]: flag saying whether we have complex output, default is true % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 15 % % The transform is centered. function A = fftmask2mat(mask,complex) A.mask = boolean(mask); A.imsz = size(A.mask); % image size sz = [sum(A.mask(:)),prod(A.imsz)]; % matrix size if nargin>1 A.complex = boolean(complex(1)); else A.complex = true; end A.sz = sz; A.ctransp = 0; A = class(A,mfilename); % [generic]mat/@fftmask2mat/full.m0000644000175000017500000000042211430167323013070 0ustar hnhn% convert to full matrix [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function B = full(A) [m,n] = size(A); B = zeros(m,n); ei = zeros(n,1); for i=1:n, ei(i) = 1; B(:,i) = A*ei; ei(i) = 0; end mat/@fftmask2mat/isempty.m0000644000175000017500000000032011430167323013615 0ustar hnhn% say whether matrix is empty [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function y = isempty(A) [m,n] = size(A); y = m*n==0;mat/@fftmask2mat/mtimes.m0000644000175000017500000000071011420141465013422 0ustar hnhn% 2d partial FFT matrix: matrix vector multiplication % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 15 function y = mtimes(A,x) if A.ctransp y = zeros(A.imsz); y(A.mask) = x(:); y = ifft2(ifftshift(y))*sqrt(prod(A.imsz)); if ~A.complex, y = real(y); end else x = reshape(x,A.imsz); if ~A.complex, x = real(x); end y = fftshift( fft2(x)/sqrt(prod(A.imsz)) ); y = y(A.mask); end y = y(:);mat/@fftmask2mat/numel.m0000644000175000017500000000031211430167323013244 0ustar hnhn% number of elements in matrix [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function N = numel(A) [m,n] = size(A); N = n*m;mat/@fftmask2mat/size.m0000644000175000017500000000101511430167323013077 0ustar hnhn% return size [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function [sz1,sz2] = size(A,id) sz = A.sz; if A.ctransp, sz = fliplr(sz); end if nargin==1 if nargout==2 sz1 = sz(1); sz2 = sz(2); else sz1 = sz; end elseif nargin==2 if nargout>1, error('too many output arguments'), end if id(1)>numel(sz) sz1 = 1; else sz1 = sz(id); end else error('wrong number of input arguments') endmat/@fftmask2mat/transpose.m0000644000175000017500000000030011430167323014137 0ustar hnhn% transpose [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function A = transpose(A) A.ctransp = xor(A.ctransp,1); mat/@fd2mat/ctranspose.m0000644000175000017500000000032211430167323013244 0ustar hnhn% complex conjugate transpose [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function A = ctranspose(A) A.ctransp = xor(A.ctransp,1);mat/@fd2mat/disp.m0000644000175000017500000000033111430167323012022 0ustar hnhn% display something [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function disp(A) fprintf('<%dx%d %s>\n', size(A,1), size(A,2), class(A));mat/@fd2mat/fd2mat.m0000644000175000017500000000121511420141471012235 0ustar hnhn% 2d finite difference transform matrix: constructor % % A = fd2mat(sz,qmf,nl) % % sz [1,2]: size of the tensor to transform % qmf [1,p]: quadrature mirror filter, default is Haar qmf = [1,1]/sqrt(2) % nl [1,1]: number of layers, default is full nl % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 15 function A = fd2mat(sz) imsz = sz(:)'; A.imsz = imsz; % image size if numel(A.imsz)==1, A.imsz = [A.imsz,1]; end sz = prod(imsz)*[2,1]; % matrix size A.sz = sz; A.ctransp = 0; A = class(A,mfilename); % [generic]mat/@fd2mat/full.m0000644000175000017500000000042211430167323012026 0ustar hnhn% convert to full matrix [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function B = full(A) [m,n] = size(A); B = zeros(m,n); ei = zeros(n,1); for i=1:n, ei(i) = 1; B(:,i) = A*ei; ei(i) = 0; end mat/@fd2mat/isempty.m0000644000175000017500000000032011430167323012553 0ustar hnhn% say whether matrix is empty [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function y = isempty(A) [m,n] = size(A); y = m*n==0;mat/@fd2mat/mtimes.m0000644000175000017500000000072211420141471012360 0ustar hnhn% 2d finite difference transform matrix: matrix vector multiplication % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 15 function y = mtimes(A,x) if A.ctransp x = reshape(x,[A.imsz,2]); y = [-x(1,:,1); x(1:end-2,:,1)-x(2:end-1,:,1); x(end-1,:,1)] ... + [-x(:,1,2), x(:,1:end-2,2)-x(:,2:end-1,2), x(:,end-1,2)]; else x = reshape(x,A.imsz); y = cat(3, x([2:end,end],:)-x, x(:,[2:end,end])-x); end y = y(:);mat/@fd2mat/numel.m0000644000175000017500000000031211430167323012202 0ustar hnhn% number of elements in matrix [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function N = numel(A) [m,n] = size(A); N = n*m;mat/@fd2mat/size.m0000644000175000017500000000101511430167323012035 0ustar hnhn% return size [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function [sz1,sz2] = size(A,id) sz = A.sz; if A.ctransp, sz = fliplr(sz); end if nargin==1 if nargout==2 sz1 = sz(1); sz2 = sz(2); else sz1 = sz; end elseif nargin==2 if nargout>1, error('too many output arguments'), end if id(1)>numel(sz) sz1 = 1; else sz1 = sz(id); end else error('wrong number of input arguments') endmat/@fd2mat/transpose.m0000644000175000017500000000030011430167323013075 0ustar hnhn% transpose [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function A = transpose(A) A.ctransp = xor(A.ctransp,1); mat/@wavmat/ctranspose.m0000644000175000017500000000032211430167323013366 0ustar hnhn% complex conjugate transpose [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function A = ctranspose(A) A.ctransp = xor(A.ctransp,1);mat/@wavmat/disp.m0000644000175000017500000000033111430167323012144 0ustar hnhn% display something [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function disp(A) fprintf('<%dx%d %s>\n', size(A,1), size(A,2), class(A));mat/@wavmat/full.m0000644000175000017500000000042211430167323012150 0ustar hnhn% convert to full matrix [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function B = full(A) [m,n] = size(A); B = zeros(m,n); ei = zeros(n,1); for i=1:n, ei(i) = 1; B(:,i) = A*ei; ei(i) = 0; end mat/@wavmat/isempty.m0000644000175000017500000000032011430167323012675 0ustar hnhn% say whether matrix is empty [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function y = isempty(A) [m,n] = size(A); y = m*n==0;mat/@wavmat/mtimes.m0000644000175000017500000000076111420141466012511 0ustar hnhn% 2d wavelet transform matrix: matrix vector multiplication % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 15 function y = mtimes(A,x) x = reshape(x,A.imsz); if A.ctransp if isreal(x) y = fwtn(x,A.nl,A.qmf,1); else y = fwtn(real(x),A.nl,A.qmf,1) + 1i*fwtn(imag(x),A.nl,A.qmf,1); end else if isreal(x) y = fwtn(x, A.nl, A.qmf); else y = fwtn(real(x), A.nl, A.qmf) + 1i*fwtn(imag(x), A.nl, A.qmf); end end y = y(:);mat/@wavmat/numel.m0000644000175000017500000000031211430167323012324 0ustar hnhn% number of elements in matrix [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function N = numel(A) [m,n] = size(A); N = n*m;mat/@wavmat/size.m0000644000175000017500000000101511430167323012157 0ustar hnhn% return size [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function [sz1,sz2] = size(A,id) sz = A.sz; if A.ctransp, sz = fliplr(sz); end if nargin==1 if nargout==2 sz1 = sz(1); sz2 = sz(2); else sz1 = sz; end elseif nargin==2 if nargout>1, error('too many output arguments'), end if id(1)>numel(sz) sz1 = 1; else sz1 = sz(id); end else error('wrong number of input arguments') endmat/@wavmat/transpose.m0000644000175000017500000000030011430167323013217 0ustar hnhn% transpose [generic routine, shared by all matrix objects] % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 22 function A = transpose(A) A.ctransp = xor(A.ctransp,1); mat/@wavmat/wavmat.m0000644000175000017500000000204611420141466012510 0ustar hnhn% wavelet transform matrix: constructor % % A = wavmat(sz,qmf,nl) % % sz [1,2]: size of the tensor to transform % qmf [1,p]: quadrature mirror filter, default is Haar qmf = [1,1]/sqrt(2) % nl [1,1]: number of layers, default is full nl % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 15 function A = wavmat(sz,qmf,nl) imsz = sz(:)'; A.imsz = imsz; % image size if numel(A.imsz)==1, A.imsz = [A.imsz,1]; end sz = prod(imsz)*[1,1]; % matrix size % wavelet type if nargin>1 A.qmf = qmf; else A.qmf = [1,1]/sqrt(2); % default is Haar end if nargin>2 A.nl = nl; else imsz = imsz/2; for nl = 1:min(fix(log2(imsz))) if any(imsz ~= fix(imsz)), nl = nl-1; break, end imsz = imsz/2; end A.nl = nl; % default is full end A.sz = sz; A.ctransp = 0; A = class(A,mfilename); % [generic]mat/@wavmat/private/fwtn.c0000644000175000017500000000643311420141465013632 0ustar hnhn/* FWTN.C - Periodized Wavelet Transform and its Inverse Transform The wavelet transform is an orthonormal transform i.e. norm(x(:))==norm(y(:)) with linear computational complexity O(numel(x)). The calling syntax is: y = FWTN(x,l,f{,transp}); x data whose size is l times divisable by 2 i.e. size(x) = sz*2^l l number of levels in the pyramid, 0 means FWTN reduces to y=x; f 1d quadrature mirror filter coefficients transp 0 for Fast Wavelet Transform and 1 for Inverse Fast Wavelet Transform, default is 0 y output data of same size (and norm) as input data x Examples for f corresponding to well-known kinds of wavelets [-0.075765714789341, -0.029635527645954, (Symmlet,4) 0.497618667632458, 0.803738751805216, 0.297857795605542, -0.099219543576935, -0.012603967262261, 0.032223100604071] [ 1+sqrt(3), 3+sqrt(3), 3-sqrt(3), 1-sqrt(3) ]/sqrt(32) (Daubechies, 4) [ 1, 1 ]/sqrt(2) (Haar) Written by Hannes Nickisch, 17 Jan 2008 */ #include "wavelet.h" #include "mex.h" #include #include /* Input Arguments */ #define SIG_IN prhs[0] /* X */ #define LEV_IN prhs[1] #define FIL_IN prhs[2] #define TRP_IN prhs[3] /* Output Arguments */ #define SIG_OUT plhs[0] /* Y */ void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) { double *in, *out, *f, *tmp, *dnl; int i, j, k, nf, d, *n, nl, max; mxArray *temp; int transpose=0; /* Check for proper number of arguments */ if (nrhs < 3) { mexErrMsgTxt("FWTN requires at least 3 input arguments."); } else if (nlhs != 1) { mexErrMsgTxt("FWTN requires one output argument."); } if (nrhs > 3) if(mxGetM(TRP_IN)>0) transpose = (int)mxGetPr(TRP_IN)[0]; /* Analyze input */ d = mxGetNumberOfDimensions(SIG_IN); n = mxMalloc(d*sizeof(int)); memcpy(n, mxGetDimensions(SIG_IN), d*sizeof(int)); if (mxGetM(LEV_IN)*mxGetN(LEV_IN)!=1) mexErrMsgTxt("FWTN requires a suitable numbers of layers"); nl = floor((mxGetPr(LEV_IN))[0]+.5); if(nl<0) mexErrMsgTxt("FWTN requires a suitable numbers of layers"); for(i=0; i #include /* 1d downsampling using high-pass filter * IN: x - vector of (even) length nx * f - high-pass filter of length nf * OUT: y - vector of length nx/2 */ void downhi( double *x, int nx, double *f, int nf, double *y ) { int nx2, nf2, h, i, j; double s; nx2 = nx/2; nf2 = nf/2-1; if(2*nf2+1-(nf-1) < 0) nf2++; for(i=nf2; i nx2) nf2 = nx2; for(i=0; i= nx) --dn2; if(dn2 < 0) dn2 = -1; for(i=0; i<=dn2; i++){ /* Calculate y[0] till y[nx2-nf2-1] */ s = 0.0; for(h=0; h= nx) j -= nx; s += f[h]*x[j]; j++; } y[i] = s; } } /* 1d upsampling using high-pass filter * IN: x - vector of length nx * f - high-pass filter of length nf * OUT: y - vector of length 2*nx */ void uphi(double *x, int nx, double *f, int nf, double *y) { int even, odd, min, h, i, j; double s; even = (nf+1)/2; odd = nf/2; min = nx-even; if(min < 0) min = 0; for(i=0; i+even= nx) j -= nx; s += f[2*h]*x[j]; j++; } y[2*i+1] = s; s = 0.0; j = i; for(h=0; h= nx) j -= nx; s += f[2*h+1]*x[j]; j++; } y[2*i] = s; } } /* 1d upsampling using low-pass filter * IN: x - vector of length nx * f - low-pass filter of length nf * OUT: y - vector of length 2*nx */ void uplo(double *x, int nx, double *f, int nf, double *y) { int even, odd, max, h, i, j; double s; even = (nf+1)/2; odd = nf/2; max = even; if(max > nx) max = nx; for(i=even; i=Nl[dd] && dd=0; l--) { /* Iterate over levels l */ for(d=D-1; d>=0; d--) { /* Iterate over dimensions d */ length = Nl[d]/2; Nl[d] = 1; /* Length of the 1d slice */ tmplo = &tmp[2*length]; tmphi = &tmp[3*length]; tmptop = &tmp[4*length]; /* Temp pointers */ offset=0; for(dd=0; dd=Nl[dd] && dd> cd ~/glm-ie/mat/@wavmat/private\n'); fprintf('>> mkoctfile --mex fwtn.c wavelet.c\n'); fprintf('>> delete fwtn.o\n'); fprintf('>> delete wavelet.o\n'); fprintf('>> cd ../../..\n'); fprintf('\n\n END GLM-IE MESSAGE\n\n') fprintf('##############################################################\n'); error end me = mfilename; % what is my filename mydir = which(me); mydir = mydir(1:end-2-numel(me)); % where am I located mycd = pwd; eval(['cd ',mydir]) mex -O fwtn.c wavelet.c clear functions y = fwtn(x,l,f,0); eval(['cd ',mycd]) endmat/@wavmat/private/wavelet.h0000644000175000017500000000167511420141465014333 0ustar hnhn/* FAST WAVELET TRANSFORM in D dimensions * X - array of size N[0]xN[1]x..xN[D-1] containing the data * N - D integers containing the size of X/W with N[i]=2^L*n[i] * L - number of levels in the pyramid * f - 1d low pass quadrature mirror filter * Nf - length of the 1d low pass quadrature mirror filter * W - the double array containing the wavelet coefficients same size as X * tmp - temporary array of size 3*max(N[i]) for auxiliary calculations * * Written by Hannes Nickisch, 17 Jan 2008 * * * The 4 auxiliary routines {up,down}{hi,lo} were inspired by code taken from * WaveLab Version 850. * http://www-stat.stanford.edu/~wavelab/Wavelab_850/download.html * */ /* Wavelet Transform */ void wtn(double *X, int *N, int D, int L, double *f, int Nf, double *W, double *tmp); /* Inverse Wavelet Transform */ void iwtn(double *W, int *N, int D, int L, double *f, int Nf, double *X, double *tmp); pen/penFromPot.m0000644000175000017500000000312111430004762011675 0ustar hnhn% [p,dp,d2p] = penFromPot(s,pot,tau,z,theta) - Potential derived penalty % % pen(s) = tau*b*(r-s) - log( pot(tau*r) ) where r = sign(s)*sqrt(s^2 + z). % % Here, theta are additional parameters for the potential function pot % which is invoked by the call feval(pot, tau.*r, theta{:}). % % For z->0, we have r->s and hence pen(s)->-log( pot(tau*s) ) => MAP estimation. % % See also PENFUNCTIONS.M. % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 August 05 function [p,dp,d2p] = penFromPot(s,pot,tau,z,varargin) if nargin<4, z = 1e-6; end % default value for smoothing parameter z sign_s = 2*(s>=0)-1; % strict sign mapping 0 to 1 r = sign_s.*sqrt(s.*s + z); % signed z-smoothed variable [lp,dlp,d2lp,b] = cols( feval(pot,tau.*r,varargin{:}) ); % evaluate potentials p = tau.*b.*(r-s) - lp; % penalty s_r = (abs(s)+eps)./(abs(r)+eps); % stabilised s./r to cover z=0 and s=0 dp = tau.*((b-dlp).*s_r - b); % first derivative r_e = r + sign_s*eps; z_r3 = z./(r_e.*r_e.*r_e); % stabilised z./r.^3 to cover z=0 and s=0 d2p = tau.*((b-dlp).*z_r3 - tau.*d2lp.*s_r.*s_r); % second derivative id = abs(s./sqrt(z))>1e10; % correct asymptotics of s -> +/-Inf p(id) = -lp(id); dp(id) = -dlp(id); d2p(id) = -d2lp(id); % split a matrix into its columns function varargout = cols(A) ncol = size(A,2); varargout = cell(1,nargout); for n=1:nargout, if n>ncol, varargout{n}=[]; else varargout{n}=A(:,n); end,endpen/penLog.m0000644000175000017500000000105311430004774011035 0ustar hnhn% [p,dp,d2p] = penLog(s,nu) - Logarithmic penalty % % pen(s) = log(s^2+nu), default nu=1 % % See also PENFUNCTIONS.M. % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 August 09 function [p,dp,d2p] = penLog(s,nu) if nargin<2, nu = 1; end % default a = s.*s+nu; p = log(a); % penalty dp = 2*s./a; % first derivative d2p = 2*(2*nu./a - 1)./a; % second derivativepen/penPow.m0000644000175000017500000000103211423621140011050 0ustar hnhn% [p,dp,d2p] = penPow(s,d) - Power penalty % % pen(s) = |s|^d, default d=1 % % See also PENFUNCTIONS.M. % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 23 function [p,dp,d2p] = penPow(s,d) if nargin<2, d = 1; end % default value p = abs(s).^d; % penalty dp = d*sign(s).*abs(s).^(d-1); % first derivative d2p = (d^2-d)*abs(s).^(d-2); % second derivativepen/penQuad.m0000644000175000017500000000067611422527032011216 0ustar hnhn% [p,dp,d2p] = penQuad(s) - Quadratic penalty % % pen(s) = s^2/2 % % See also PENFUNCTIONS.M. % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 23 function [p,dp,d2p] = penQuad(s) p = s.*s/2; % penalty dp = s; % first derivative d2p = ones(size(s)); % second derivativepen/penZero.m0000644000175000017500000000042511430004774011235 0ustar hnhn% [p,dp,d2p] = penZero(s) - Zero penalty % % pen(s) = 0 % % See also PENFUNCTIONS.M. % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 August 09 function [p,dp,d2p] = penZero(s) p = zeros(size(s)); dp = p; d2p = p; % penalty and first two derivativespls/brentmin.m0000755000175000017500000000730311415720432011451 0ustar hnhnfunction [xmin,fmin,funccount] = brentmin(xlow,xupp,Nitmax,tol,f,varargin) % BRENT Brent's minimization method in one dimension taken from: % § 10.2 Parabolic Interpolation and Brent's Method in One Dimension % Press, Teukolsky, Vetterling & Flannery % Numerical Recipes in C, Cambridge University Press, 2002 % % [xmin,fmin] = BRENTMIN(xlow,xupp,Nit,tol,f,varargin) % Given a function f, and given a search interval this routine isolates % the minimum of fractional precision of about tol using Brent's method. % % INPUT % xlow,xupp: search interval such that xlow<=xmin<=xupp % Nitmax: maximum number of function evaluations made by the routine % tol: fractional precision % f, varargin: f(x,varargin{:}) is the function % % OUTPUT % fmin: minimal function value % xmin: corresponding abscissa-value % funccount: number of function evaluations made % % Hannes Nickisch % revised 10/04/2006 % version 1.0.2 % tolerance is no smaller than machine's floating point precision tol = max(tol,eps); % Evaluate endpoints fa = f(xlow,varargin{:}); fb = f(xupp,varargin{:}); funccount = 2; % number of function evaluations % Compute the start point seps = sqrt(eps); c = 0.5*(3.0 - sqrt(5.0));% golden ratio a = xlow; b = xupp; v = a + c*(b-a); w = v; xf = v; d = 0.0; e = 0.0; x= xf; fx = f(x,varargin{:}); funccount = funccount + 1; fv = fx; fw = fx; xm = 0.5*(a+b); tol1 = seps*abs(xf) + tol/3.0; tol2 = 2.0*tol1; % Main loop while ( abs(xf-xm) > (tol2 - 0.5*(b-a)) ) gs = 1; % Is a parabolic fit possible if abs(e) > tol1 % Yes, so fit parabola gs = 0; r = (xf-w)*(fx-fv); q = (xf-v)*(fx-fw); p = (xf-v)*q-(xf-w)*r; q = 2.0*(q-r); if q > 0.0, p = -p; end q = abs(q); r = e; e = d; % Is the parabola acceptable if ( (abs(p)q*(a-xf)) && (p= xm, e = a-xf; else e = b-xf; end d = c*e; end % The function must not be evaluated too close to xf si = sign(d) + (d == 0); x = xf + si * max( abs(d), tol1 ); fu = f(x,varargin{:}); funccount = funccount + 1; % Update a, b, v, w, x, xm, tol1, tol2 if fu <= fx if x >= xf, a = xf; else b = xf; end v = w; fv = fw; w = xf; fw = fx; xf = x; fx = fu; else % fu > fx if x < xf, a = x; else b = x; end if ( (fu <= fw) || (w == xf) ) v = w; fv = fw; w = x; fw = fu; elseif ( (fu <= fv) || (v == xf) || (v == w) ) v = x; fv = fu; end end xm = 0.5*(a+b); tol1 = seps*abs(xf) + tol/3.0; tol2 = 2.0*tol1; if funccount >= Nitmax % BREAK fmin = fx; xmin = xf; % typically we should not get here % warning(sprintf('Maximum number of iterations (%d) exceeded: precision is not guaranteed',Nitmax)) % fprintf('[%1.3f,%1.3f,%1.3f]\n',xlow,xf,xupp) break end end % while % check that endpoints are less than the minimum found if ( (fa < fx) && (fa <= fb) ) xf = xlow; fx = fa; elseif fb < fx xf = xupp; fx = fb; end fmin = fx; xmin = xf; pls/lbfgsb.m0000644000175000017500000001144311422527036011072 0ustar hnhn% LBFGSB Call the nonlinear bound-constrained solver that uses % limited-memory BFGS quasi-Newton updates. % % The basic function call is % % LBFGSB(x0,lb,ub,objfunc,gradfunc) % % The first input argument x0 is either a matrix or a cell array of % matrices. It declares the starting point for the solver. % % The second and third input arguments lb and ub must be of the exact same % structure as x0. They declare the lower and upper bounds on the % variables, respectively. Set an entry to Inf to declare no bound. % % The next two input arguments must be the names of MATLAB routines % (M-files): % % objfunc Calculates the objective function at the current % point. The routine must accept as many inputs as cell % entry in x0 (or one input if x0 is a matrix). The % output must be a scalar representing the objective % evaluated at the current point. % % gradfunc Computes the gradient of the objective at the current % point. The input is the same as objfunc, but it must % return as many outputs as there are inputs, and each % of the outputs have the same matrix structure as its % corresponding input. % % Optionally, one may choose to pass additional auxiliary data to the % MATLAB callback routines listed above through the function call % LBFGSB(...,auxdata). If auxdata is the empty matrix, no extra % information is passed. It is important to observe that the auxiliary % data MAY NOT CHANGE through the course of the L-BFGS optimization! The % auxiliary data keep the same values as they possessed in the initial % call. If you need variables that change over time, you may want to % consider global variables (type HELP GLOBAL). % % LBFGSB(...,auxdata,iterfunc) specifies an additional callback routine % which is called once per algorithm iteration. The callback routine must % take the form ITERFUNC(T,F,X1,X2,...,AUXDATA). T is the current % iteration of the algorithm. F is the current value of the objective. The % inputs X1, X2,... are exactly the same as the ones passed to objfunc and % gradfunc. Finally, extra information may be passed through the input % AUXDATA. No outputs are expected from iterfunc. If iterfunc is the empty % string, no routine is called. % % These inputs may be followed by parameter/value pairs to modify the % default algorithm options. The algorithm option 'maxiter' sets the % maximum number of iterations. The rest of the options 'm', 'factr' and % 'pgtol' are described in the L-BFGS-B documentation. Note that in this % implementation we divide factr by the machine precision, so that the % convergence condition is satisfied when it is less than factr, not less % than factr times the machine precision. % % Interface has been written by Peter Carbonetto. % Compilation instructions by Hannes Nickisch, 2010-07-19 function x = lbfgsb(x,varargin) fprintf('\n\n\n') fprintf('##########################################################################\n') fprintf('In order to use the L-BFGS minimiser you have to compile Peter Carbonetto''s\n') fprintf('"Matlab interface for L-BFGS-B" [1]. The challenge here is the Fortran 77 code.\n') fprintf('We provide a Makefile suitable for Linux 32/64 bit and Mac whenever you have g77\n') fprintf('properly installed.\n\n') fprintf('Compilation is done by editing lbfgsb/Makefile. In any case, you need to\n') fprintf('provide MATLAB_HOME which can be found by the commands ''locate matlab'' or \n') fprintf('''find / -name "matlab"''. You can choose between two compilation modes:\n') fprintf('a) using the mex utility by Matlab [default]\n') fprintf(' provide MEX, then type ''make mex''\n\n') fprintf('b) without mex utility by Matlab\n') fprintf(' provide MEX_SUFFIX and MATLAB_LIB, then type ''make nomex''\n\n') fprintf('In Ubuntu 10.04 LTS, the libg2c library needed for both a) and b) is not\n') fprintf('included per default. If ''ls /usr/lib/libg2c.*'' does not list anything\n') fprintf('this is the case on your machine. You then whant to install the packages\n') fprintf('gcc-3.4-base and libg2c0 e.g. from\n') fprintf('http://packages.ubuntu.com/jaunty/gcc-3.4-base and\n') fprintf('http://packages.ubuntu.com/jaunty/libg2c0.\n') fprintf('After installation, you have to create a symbolic link by ''cd /usr/lib'' and\n') fprintf('''ln -s libg2c.so.0 libg2c.so''.\n\n') fprintf('[1] http://www.cs.ubc.ca/~pcarbo/lbfgsb-for-matlab.html\n') fprintf('[2] http://www.mathworks.com/support/compilers/R2010a\n') fprintf('##########################################################################\n') fprintf('\n\n\n') error pls/linsolve_lcg.m0000644000175000017500000000350311426544341012313 0ustar hnhn% c = LINSOLVE_LCG(X,R,B,P,b,kmax,c0) % % Computes c = A\b with A = X'*R*X + B*P*B by Linear Conjugate Gradients (LCG) % as described in § 10.2.6 of Golub, van Loan, Matrix Computations with at most % kmax MVMs starting from an intial value c0. % % ARGUMENTS % X [m,n] matrix or operator, m can be zero % R [m,m] square matrix % or [m,1] vector of diagonal R = diag(R(:)) % or [1,1] scalar multiple of identity R = R*eye(n) % % B [q,n] matrix or operator % P [n,n] square matrix % or [m,1] vector of diagonal P = diag(P(:)) % or [1,1] scalar multiple of identity P = P*eye(n) % % b [n,1] vector % kmax [1,1] scalar, default value is n % c0 [n,1] vector, default is zeros(n,1) % % RESULTS % c [n,1] vector c = A\b, A = X'*X/s2 + B*diag(pi)*B % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 August 05 function [c,k,res] = linsolve_lcg(X,R,B,P,b,kmax,c0) if nargin>6, c=c0; else c=zeros(size(b)); end if nargin<6, kmax=length(b); end r = b - mvmA(X,R,B,P, c); % r = b-A*c; nb = norm(b,'fro'); % |b| k = 0; % iteration counter while norm(r,'fro')>eps*nb && kO(M^3) where N=max(m,n) and M=min(m,n) % due to the Cholesky decomposition on a matrix of size MxM and a matrix-matrix % multiplication X'*X or X*X'. % % ARGUMENTS % X [m,n] full or sparse matrix, no operator allowed % % R [m,m] square matrix % or [m,1] vector of diagonal R = diag(R(:)) % or [1,1] scalar multiple of identity R = R*eye(n) % % P [n,n] square matrix % or [m,1] vector of diagonal P = diag(P(:)) % or [1,1] scalar multiple of identity P = P*eye(n) % % b [n,1] vector % % RESULTS % c [n,1] vector c = A\b, A = X'*R*X + P % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 August 05 function c = linsolve_woodbury(X,R,P,b) [m,n] = size(X); if m0, S='Linesearch'; else S=''; end i = 0; % zero the run length counter ls_failed = 0; % no previous line search has failed [f0 df0] = feval(f, X, varargin{:}); % get function value and gradient Z = X; X = unwrap(X); df0 = unwrap(df0); if exist('fflush','builtin'), fflush(stdout); end fX = f0; i = i + (length<0); % count epochs?! s = -df0; d0 = -s'*s; % initial search direction (steepest) and slope x3 = red/(1-d0); % initial step is red/(|s|+1) while i < abs(length) % while not finished i = i + (length>0); % count iterations?! X0 = X; F0 = f0; dF0 = df0; % make a copy of current values if length>0, M = MAX; else M = min(MAX, -length-i); end while 1 % keep extrapolating as long as necessary x2 = 0; f2 = f0; d2 = d0; f3 = f0; df3 = df0; success = 0; while ~success && M > 0 try M = M - 1; i = i + (length<0); % count epochs?! [f3 df3] = feval(f, rewrap(Z,X+x3*s), varargin{:}); df3 = unwrap(df3); if isnan(f3) || isinf(f3) || any(isnan(df3)+isinf(df3)), error(''), end success = 1; catch % catch any error which occured in f x3 = (x2+x3)/2; % bisect and try again end end if f3 < F0, X0 = X+x3*s; F0 = f3; dF0 = df3; end % keep best values d3 = df3'*s; % new slope if d3 > SIG*d0 || f3 > f0+x3*RHO*d0 || M == 0 % are we done extrapolating? break end x1 = x2; f1 = f2; d1 = d2; % move point 2 to point 1 x2 = x3; f2 = f3; d2 = d3; % move point 3 to point 2 A = 6*(f1-f2)+3*(d2+d1)*(x2-x1); % make cubic extrapolation B = 3*(f2-f1)-(2*d1+d2)*(x2-x1); x3 = x1-d1*(x2-x1)^2/(B+sqrt(B*B-A*d1*(x2-x1))); % num. error possible, ok! if ~isreal(x3) || isnan(x3) || isinf(x3) || x3 < 0 % num prob | wrong sign? x3 = x2*EXT; % extrapolate maximum amount elseif x3 > x2*EXT % new point beyond extrapolation limit? x3 = x2*EXT; % extrapolate maximum amount elseif x3 < x2+INT*(x2-x1) % new point too close to previous point? x3 = x2+INT*(x2-x1); end end % end extrapolation while (abs(d3) > -SIG*d0 || f3 > f0+x3*RHO*d0) && M > 0 % keep interpolating if d3 > 0 || f3 > f0+x3*RHO*d0 % choose subinterval x4 = x3; f4 = f3; d4 = d3; % move point 3 to point 4 else x2 = x3; f2 = f3; d2 = d3; % move point 3 to point 2 end if f4 > f0 x3 = x2-(0.5*d2*(x4-x2)^2)/(f4-f2-d2*(x4-x2)); % quadratic interpolation else A = 6*(f2-f4)/(x4-x2)+3*(d4+d2); % cubic interpolation B = 3*(f4-f2)-(2*d2+d4)*(x4-x2); x3 = x2+(sqrt(B*B-A*d2*(x4-x2)^2)-B)/A; % num. error possible, ok! end if isnan(x3) || isinf(x3) x3 = (x2+x4)/2; % if we had a numerical problem then bisect end x3 = max(min(x3, x4-INT*(x4-x2)),x2+INT*(x4-x2)); % don't accept too close [f3 df3] = feval(f, rewrap(Z,X+x3*s), varargin{:}); df3 = unwrap(df3); if f3 < F0, X0 = X+x3*s; F0 = f3; dF0 = df3; end % keep best values M = M - 1; i = i + (length<0); % count epochs?! d3 = df3'*s; % new slope end % end interpolation if abs(d3) < -SIG*d0 && f3 < f0+x3*RHO*d0 % if line search succeeded X = X+x3*s; f0 = f3; fX = [fX' f0]'; % update variables fprintf('%s%5i, phi=%4.4e; du=%1.3e\r', S, i, f0, norm(x3*s)); if exist('fflush','builtin'), fflush(stdout); end s = (df3'*df3-df0'*df3)/(df0'*df0)*s - df3; % Polack-Ribiere CG direction df0 = df3; % swap derivatives d3 = d0; d0 = df0'*s; if d0 > 0 % new slope must be negative s = -df0; d0 = -s'*s; % otherwise use steepest direction end x3 = x3 * min(RATIO, d3/(d0-realmin)); % slope ratio but max RATIO ls_failed = 0; % this line search did not fail else X = X0; f0 = F0; df0 = dF0; % restore best point so far if ls_failed || i > abs(length) % line search failed twice in a row break; % or we ran out of time, so we give up end s = -df0; d0 = -s'*s; % try steepest x3 = 1/(1-d0); ls_failed = 1; % this line search failed end end X = rewrap(Z,X); fprintf('\n'); if exist('fflush','builtin') fflush(stdout); end % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse rewrap.m. function v = unwrap(s) v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s))); % alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; unwrap(s{i})]; end end % other types are ignored % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also unwrap.m. function [s v] = rewrap(s, v) if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed pls/minimize_lbfgs.m0000644000175000017500000001055411420141474012627 0ustar hnhnfunction [X, fX, i] = minimize_lbfgs(X, f, length, varargin) % Minimize a differentiable multivariate function using quasi Newton. % % Usage: [X, fX, i] = minimize_lbfgs(X, f, length, P1, P2, P3, ... ) % % X initial guess; may be of any type, including struct and cell array % f the name or pointer to the function to be minimized. The function % f must return two arguments, the value of the function, and it's % partial derivatives wrt the elements of X. The partial derivative % must have the same type as X. % length length of the run; if it is positive, it gives the maximum number of % line searches, if negative its absolute gives the maximum allowed % number of function evaluations. Optionally, length can have a second % component, which will indicate the reduction in function value to be % expected in the first line-search (defaults to 1.0). % P1, P2 ... parameters are passed to the function f. % % X the returned solution % fX vector of function values indicating progress made % i number of iterations (line searches or function evaluations, % depending on the sign of "length") used at termination. % % The function returns when either its length is up, or if no further progress % can be made (ie, we are at a (local) minimum, or so close that due to % numerical problems, we cannot get any closer). NOTE: If the function % terminates within a few iterations, it could be an indication that the % function values and derivatives are not consistent (ie, there may be a bug in % the implementation of your "f" function). % % Copyright (C) 2010 by Hannes Nickisch, 2010-02-05 % global variables serve as communication interface between calls global minimize_lbfgs_iteration_number global minimize_lbfgs_objective global minimize_lbfgs_gradient global minimize_lbfgs_X % init global variables minimize_lbfgs_iteration_number = 0; minimize_lbfgs_objective = Inf; minimize_lbfgs_gradient = 0*unwrap(X); minimize_lbfgs_X = X; X0 = X; lb = -Inf*ones(size(unwrap(X0))); ub = Inf*ones(size(unwrap(X0))); maxiter = abs(length); % max number of iterations % no callback routine used so far % m is the number of saved vectors used to estimate the Hessian % factr is the precision 1e-12 X = lbfgsb( unwrap(X0), lb, ub, 'minimize_lbfgs_objfun', ... 'minimize_lbfgs_gradfun', ... {f,X0,varargin{:}}, [], ... 'maxiter',maxiter, 'm',4, 'factr',1e-12, 'pgtol',1e-5); i = minimize_lbfgs_iteration_number; fX = minimize_lbfgs_objective; X = rewrap(X0,X); % clear global variables clear minimize_lbfgs_iteration_number clear minimize_lbfgs_objective clear minimize_lbfgs_gradient clear minimize_lbfgs_X % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse rewrap.m. function v = unwrap(s) v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s)));% alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; unwrap(s{i})]; end end % other types are ignored % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also unwrap.m. function [s v] = rewrap(s, v) if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed pls/minimize_lbfgs_gradfun.m0000644000175000017500000000451311415720432014334 0ustar hnhnfunction G = minimize_lbfgs_gradfun(X,varargin) % extract input arguments varargin = varargin{1}; strctX = varargin{2}; f = varargin{1}; % global variables serve as communication interface between calls global minimize_lbfgs_iteration_number global minimize_lbfgs_objective global minimize_lbfgs_gradient global minimize_lbfgs_X if norm(unwrap(X)-unwrap(minimize_lbfgs_X))>1e-10 [y,G] = feval(f,rewrap(strctX,X),varargin{3:end}); else y = minimize_lbfgs_objective(minimize_lbfgs_iteration_number); G = minimize_lbfgs_gradient; end % memorise gradient and position minimize_lbfgs_gradient = G; minimize_lbfgs_X = X; G = unwrap(G); % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse rewrap.m. function v = unwrap(s) v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s)));% alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; unwrap(s{i})]; end end % other types are ignored % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also unwrap.m. function [s v] = rewrap(s, v) if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed pls/minimize_lbfgs_objfun.m0000644000175000017500000000540211422527036014172 0ustar hnhnfunction y = minimize_lbfgs_objfun(X,varargin) % extract input arguments varargin = varargin{1}; strctX = varargin{2}; f = varargin{1}; % global variables serve as communication interface between calls global minimize_lbfgs_iteration_number global minimize_lbfgs_objective global minimize_lbfgs_gradient global minimize_lbfgs_X recompute = 0; if minimize_lbfgs_iteration_number==0 recompute = 1; else if norm(unwrap(X)-unwrap(minimize_lbfgs_X))>1e-10 recompute = 1; end end if recompute [y,G] = feval(f,rewrap(strctX,X),varargin{3:end}); else y = minimize_lbfgs_objective(minimize_lbfgs_iteration_number); G = minimize_lbfgs_gradient; end % increase global counter, memorise objective function, gradient and position minimize_lbfgs_iteration_number = minimize_lbfgs_iteration_number+1; minimize_lbfgs_objective(minimize_lbfgs_iteration_number,1) = y; minimize_lbfgs_gradient = G; minimize_lbfgs_X = X; fprintf('%5i, phi=%4.4e; du=%1.3e\r', minimize_lbfgs_iteration_number, y, norm(G) ) if exist('fflush','builtin'), fflush(stdout); end % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse rewrap.m. function v = unwrap(s) v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s)));% alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; unwrap(s{i})]; end end % other types are ignored % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also unwrap.m. function [s v] = rewrap(s, v) if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed pls/phi.m0000644000175000017500000000231411426543407010414 0ustar hnhn% Penalised least squares objective function and derivatives % phi(u) = 1/(2*lambda) * ||X*u-y||_2^2 + sum( pen(s) ), s=B*u % % [f,df,d2f] = phi(u,X,y,B,lam,pen,varargin) where the penaliser is evaluated % by feval(pen,s,varargin{:}) % % The return arguments have the following meaning: % f = phi(u), [1,1] scalar % df = d phi(u) / du, and [n,1] gradient vector (same size as u) % d2f = d^2 phi(u) / du^2. [n,n] Hessian matrix % % See also PLSALGORITHMS.M. % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 August 05 function [f,df,d2f] = phi(u,X,y,B,lam,pen,varargin) s = B*u; r = X*u-y; if nargout==1 p = feval(pen,s,varargin{:}); elseif nargout==2 [p,dp] = feval(pen,s,varargin{:}); else [p,dp,d2p] = feval(pen,s,varargin{:}); end f = norm(r,2)^2/(2*lam) + sum(p); if nargout>1 df = [X']*r/lam + [B']*dp; if nargout>2 % evaluation of the Hessian is dangerous for large n if numel(dp)==numel(d2p) d2f = X'*X/lam + B'*diag(d2p)*B; % d2p is a vector else d2f = X'*X/lam + B'*d2p*B; % d2p is a scalar or a full matrix end end endpls/plsCG.m0000644000175000017500000000156211426551612010645 0ustar hnhn% Penalised least squares solved by CONJUGATE GRADIENTS using the function % minimize.m [1] by Carl E. Rasmussen. % % [u,phi] = plsCG(u,X,y,B,opt,lam,pen,varargin) % % See also PLSALGORITHMS.M. % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 August 05 % % [1] http://www.kyb.tuebingen.mpg.de/bs/people/carl/code/minimize/ function [u,phi] = plsCG(u,X,y,B,opt,lam,pen,varargin) % optimisation parameters if isfield(opt,'nMVM') % maximal number of steps nMVM = opt.nMVM; else nMVM = 100; end % information parameters if isfield(opt,'output') % flag saying whether some output is shown output = opt.output; else output = false; end if output [u,phi] = minimize(u,'phi',-nMVM,X,y,B,lam,pen,varargin{:}); else [t,u,phi] = evalc('minimize(u,''phi'',-nMVM,X,y,B,lam,pen,varargin{:})'); end phi = phi(end);pls/plsCGBT.m0000644000175000017500000000704111422622515011066 0ustar hnhn% Penalised least squares solved by CONJUGATE GRADIENTS with BACKTRACKING line % search using the Armijo rule. The algorithm is inspired by the code fnlCg.m % from Michael Lustig [1]. % % [u,phi] = plsTN(u,X,y,B,opt,lam,pen,varargin) % % Additional options: % opt. % nline : maximum number of line searche steps [default 20] % alpha : constant for Armijo condition [default 0.01] % beta : stepsize update factor [default 0.6] % cmax : max. stepsize (gradient scaling factor) [default 1] % eps_grad : gradient norm convergence threshold [default 1e-10] % % See also PLSALGORITHMS.M. % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 20 % % [1] http://www.stanford.edu/~mlustig/sparseMRI_v0.2.tar.gz function [u,phi] = plsCGBT(u,X,y,B,opt,lam,pen,varargin) % line search parameters if isfield(opt,'nline') % maximum number of line searche steps nline = opt.nline; else nline = 20; end if isfield(opt,'alpha') % constant for Armijo condition al = opt.alpha; else al = 0.01; end if isfield(opt,'beta') % stepsize update factor be = opt.beta; else be = 0.6; end if isfield(opt,'cmax') % max. stepsize (gradient scaling factor) cmax = opt.cmax; else cmax = 1.0; end % optimisation parameters if isfield(opt,'eps_grad') % gradient norm convergence threshold eps_grad = opt.eps_grad; else eps_grad = 1e-10; end if isfield(opt,'nMVM') % maximal number of conjugate gradient steps nMVM = opt.nMVM; else nMVM = 100; end % information parameters if isfield(opt,'output') % flag saying whether some output is shown output = opt.output; else output = false; end varargin = {lam,pen,varargin{:}}; if output, if exist('fflush','builtin'), fflush(stdout); end, end [g0,Xu,Bu] = gradPhi(u,X,y,B,varargin{:}); du = -g0; % initial gradient for n=0:nMVM Xdu = X*du; Bdu = B*du; % pre-calculate MVMs for speed c=0; f0 = Phi(Xu,Xdu,y,Bu,Bdu,c,varargin{:}); % objective for c=0 c=cmax; f1 = Phi(Xu,Xdu,y,Bu,Bdu,c,varargin{:}); % and for c=cmax for nl = 0:nline-1 % backtracking line-search if f1 <= f0-c*al*abs(g0(:)'*du(:)), break, end % check Armijo condition c = c*be; f1 = Phi(Xu,Xdu,y,Bu,Bdu,c,varargin{:}); end % control the number of line searches by adapting the initial step search if nl>2, cmax = cmax*be; end, if nl<1, cmax = cmax/be; end if nl==nline, warning(1,'Max line search; operator bug?'); return, end u = u + c*du; % update [g1,Xu,Bu] = gradPhi(u,X,y,B,varargin{:}); % Fletcher-Reeves CG update bk = g1(:)'*g1(:)/( g0(:)'*g0(:)+eps ); g0 = g1; du = bk*du-g1; if output fprintf('%5i, phi=%4.4e; du=%1.3e\r', n, f1, norm(c*du) ) if exist('fflush','builtin'), fflush(stdout); end end if norm(du(:)) < eps_grad, break, end % stopping criterion end phi = f1; if output, fprintf('\n'); if exist('fflush','builtin'), fflush(stdout); end, end function phi = Phi(Xu,Xdu,y,Bu,Bdu,c,lam,pen,varargin) % objective Phi(u+c*du) s = Bu+c*Bdu; r = Xu+c*Xdu - y; h = feval(pen,s,varargin{:}); phi = (r'*r)/(2*lam) + sum(h); function [g,Xu,Bu] = gradPhi(u,X,y,B,lam,pen,varargin) % gradient of objective Bu = B*u; Xu = X*u; r = Xu-y; [h,dh] = feval(pen,Bu,varargin{:}); g = [X']*r + lam*([B']*dh);pls/plsLBFGS.m0000644000175000017500000000217411422527036011210 0ustar hnhn% Penalised least squares solved by the LIMITED memory % BROYDEN-FLETCHER-GOLDFARB-SHANNO (L-BFGS) algorithm which is a quasi Newton or % variable metric method. % We use Peter Carbonetto's "Matlab interface for L-BFGS-B" [1]. % % [u,phi] = plsTN(u,X,y,B,opt,lam,pen,varargin) % % See also PLSALGORITHMS.M. % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 21 % % [1] http://www.cs.ubc.ca/~pcarbo/lbfgsb-for-matlab.html function [u,phi] = plsLBFGS(u,X,y,B,opt,lam,pen,varargin) % optimisation parameters if isfield(opt,'nMVM') % maximal number of steps nMVM = opt.nMVM; else nMVM = 100; end % information parameters if isfield(opt,'output') % flag saying whether some output is shown output = opt.output; else output = false; end if output if output, if exist('fflush','builtin'), fflush(stdout); end, end [u,phi] = minimize_lbfgs(u,'phi',-nMVM,X,y,B,lam,pen,varargin{:}); if output, fprintf('\n'); if exist('fflush','builtin') fflush(stdout); end, end else [t,u,phi] = evalc('minimize_lbfgs(u,''phi'',-nMVM,X,y,B,lam,pen,varargin{:})'); end phi = phi(end);pls/plsTN.m0000644000175000017500000000606411426546016010701 0ustar hnhn% Penalised least squares solved by TRUNCATED NEWTON % i.e. Newton steps approximated by conjugate gradients % % [u,phi] = plsTN(u,X,y,B,opt,lam,pen,varargin) % % Additional options: % opt. % nit: maximum number of Newton steps [default 15] % we have nMVM/nit MVMs to approximate the Newton dir. by CG % nline: max. bisection steps in Brent's line search [default 10] % exactNewt: flag indicating whether we want to compute the Newton % direction exactly instead of CG [default false] % this works only if B=1 and X is numeric % % See also PLSALGORITHMS.M. % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 August 05 function [u,phi] = plsTN(u,X,y,B,opt,lam,pen,varargin) % optimisation parameters if isfield(opt,'nMVM') % maximum number of MVMs with A nMVM = opt.nMVM; else nMVM = 100; end if isfield(opt,'nit') % maximum number of Newton steps nit = opt.nit; else nit = 15; end ncg = max(fix(nMVM/nit),1); % maximum number of CG steps for Newton direction if isfield(opt,'nline') % max. bisection steps in Brent's line search nline = opt.nline; else nline = 10; end if isfield(opt,'exactNewt') % Do we want to compute the Newton dir. exactly? exactNewt = opt.exactNewt; % B has to be the identity and X numeric exactPoss = isnumeric(B) && numel(B)==1 && isnumeric(X); if exactPoss, exactPoss = exactPoss && B==1; end % Is it possible at all? if ~exactPoss, error('Exact Newton direction cannot be computed.'), end else exactNewt = false; end % information parameters if isfield(opt,'output') % flag saying whether some output is shown output = opt.output; else output = false; end n = numel(u); if output, if exist('fflush','builtin'), fflush(stdout); end, end for i=1:nit s = B*u; r = X*u-y; % sparse coeffs, residuals [h,dh,d2h] = feval(pen,s,varargin{:}); if exactNewt % compute Newton direction by exploiting the Woodbury formula d = -H\g; g = X'*r/lam+dh; d = linsolve_woodbury( X, 1/lam, d2h, -g ); else % compute Newton direction by LCG: d = -H\g; % gradient g = X'*(X*u-y)/lam + B'*dh; % Hessian H = X'*X/lam + B'*diag(d2h)*B; % A = X'*R*X + B'*diag(pi)*B, R = eye(m)/lam, P = diag(d2h) g = [X']*r/lam+[B']*dh; d = linsolve_lcg( X, 1/lam, B, d2h, -g,min(ncg,n+1)); end % line search along Newton direction d, stepsize t Phi = @(t,r,X_d,s,B_d,lam,varargin) ... norm(r+t*X_d)^2/(2*lam) + sum(feval(pen,s+t*B_d,varargin{:})); [t,phi] = brentmin(0,2,nline,1e-8,Phi,r,X*d,s,B*d,lam,varargin{:}); % t=[0,2] % update estimate u = u+t*d; du = norm(t*d); if output fprintf('%5i, phi=%4.4e; du=%1.3e\r', i, phi, du ) if exist('fflush','builtin'), fflush(stdout); end end if du MATLAB interface for L-BFGS-B

A MATLAB interface for L-BFGS-B

by Peter Carbonetto
Dept. of Computer Science
University of British Columbia

L-BFGS-B is a collection of Fortran 77 routines for solving large-scale nonlinear optimization problems with bound constraints on the variables. One of the key features of the nonlinear solver is that knowledge of the Hessian is not required; the solver computes search directions by keeping track of a quadratic model of the objective function with a limited-memory BFGS (Broyden-Fletcher-Goldfarb-Shanno) approximation to the Hessian.1 The algorithm was developed by Ciyou Zhu, Richard Byrd and Jorge Nocedal. For more information, go to the original distribution site for the L-BFGS-B software package.

I've designed an interface to the L-BFGS-B solver so that it can be called like any other function in MATLAB.2 See the text below for more information on installing and calling it in MATLAB. Along the way, I've also developed a C++ class that encapsulates all the "messy" details in executing the L-BFGS-B code. See below for instructions on how to use this class for your own C++ code.

If you have any questions, praise, or comments, or would like to report a bug, do not hesitate to contact the author. I've tested this software in MATLAB version 7.2 on both the Mac OS X and Linux operating systems.

Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 2.5 Canada License. Please cite the source as Peter Carbonetto, Department of Computer Science, University of British Columbia.

Installation

These installation instructions assume you have a UNIX-based operating system, such as Mac OS X or Linux. It is surely possible to install this software on Windows, I'm just not sure how. These instructions also assume you have GNU make installed.

Download the source. First, download and unpack the tar archive. It contains some MATLAB files (ending in .m), some C++ source and header files (ending in .h and .cpp), a single Fortran 77 source file solver.f containing the L-BFGS-B routines, and a Makefile. I've included a version of the Fortran routines that is more recent than what is available for download at the distribution site at Northwestern University.

What we will do is create a MEX file, which is basically a file that contains a routine that can be called from MATLAB as if it were a built-it function. To learn about MEX files, I refer you to this document at the MathWorks website.

Install the C++ and Fortran 77 compilers. In order to build the MEX file, you will need both a C++ compiler and a Fortran 77 compiler. Unfortunately, you can't just use any compiler. You have to use the precise one supported by MATLAB. For instance, if you are running Mac OS X 7.2 on a Linux operating system, you will need to install the GNU compiler collection (GCC) version 3.4.4. Even if you already have a compiler installed on Linux, it may be the wrong version, and if you use the wrong version things could go horribly wrong! It is important that you use the correct version of the compiler, otherwise you will encounter linking errors. Refer to this document to find out which compiler is supported by your version of MATLAB.

Configure MATLAB. Next, you need to set up and configure MATLAB to build MEX Files. This is explained quite nicely in this MathWorks support document.

Modify the Makefile. You are almost ready to build the MEX file. But before you do so, you need to edit the Makefile to coincide with your system setup. At the top of the file are several variables you may need to modify. The variable MEX is the executable called to build the MEX file. The variables CXX and F77 must be the names of your C++ and Fortran 77 compilers. MEXSUFFIX is the MEX file extension specific to your platform. (For instance, the extension for Linux is mexglx.) The variable MATLAB_HOME must be the base directory of your MATLAB installation. Finally, CFLAGS and FFLAGS are options passed to the C++ and Fortran compilers, respectively. Often these flags coincide with your MEX options file (see here for more information on the options file). But often the settings in the MEX option file are incorrect, and so the options must be set manually. MATLAB requires some special compilation flags for various reasons, one being that it requires position-independent code. These instructions are vague (I apologize), and this step may require a bit of trial and error until before you get it right.

On my laptop running Mac OS X 10.3.9 with MATLAB 7.2, my settings for the Makefile are

  MEX         = mex
  MEXSUFFIX   = mexmac
  MATLAB_HOME = /Applications/MATLAB72
  CXX         = g++
  F77         = g77 
  CFLAGS      = -O3 -fPIC -fno-common -fexceptions \
                -no-cpp-precomp 
  FFLAGS      = -O3 -x f77-cpp-input -fPIC -fno-common

On my Linux machine, I set the variables in the Makefile like so:

  MEX         = mex
  MEXSUFFIX   = mexglx
  MATLAB_HOME = /cs/local/generic/lib/pkg/matlab-7.2
  CXX         = g++-3.4.5
  F77         = g77-3.4.5
  CFLAGS      = -O3 -fPIC -pthread 
  FFLAGS      = -O3 -fPIC -fexceptions

It may be helpful to look at the GCC documentation in order to understand what these various compiler flags mean.

Build the MEX file. If you are in the directory containing all the source files, typing make in the command prompt will first compile the Fortran and C++ source files into object code (.o files). After that, the make program calls the MEX script, which in turn links all the object files together into a single MEX file. If you didn't get any errors, then you are ready to try out the bound-constrained solver in MATLAB. Note that even if you didn't get any errors, there's still a possibility that you didn't link the MEX file properly, in which case executing the MEX file will cause MATLAB to crash. But there's only one way to find out: the hard way.

A brief tutorial

I've written a short script examplehs038.m which demonstrates how to call lbfgsb for solving a small optimization problem, the Hock & Schittkowski test problem no. 38.3 The optimization problem has four variables. They are bounded from below by -10 and bounded from above by +10. We've set the starting point to (-3, -1, -3, -1). We've written two MATLAB functions for computing the objective function and the gradient of the objective function at a given point. These functions can be found in the files computeObjectiveHS038.m and computeGradientHS038.m. If you run the script, the solver should very quickly progress toward the point (1, 1, 1, 1). The objective is 0 at this point.

The basic MATLAB function call is

  lbfgsb(x0,lb,ub,objfunc,gradfunc)

The first input argument x0 declares the starting point for the solver. The second and third input arguments define the lower and upper bounds on the variables, respectively. If an entry of lb is set to -Inf, then this is equivalent to no lower bound for that entry (the same applies to upper bounds, except that it must be +Inf instead). The next two input arguments must be the names of MATLAB routines (M-files). The first routine must calculate the objective function at the current point. The output must be a scalar representing the objective evaluated at the current point. The second function must compute the gradient of the objective at the current point. The input is the same as objfunc, but it must return as many outputs as there are inputs. For a complete guide on using the MATLAB interface, type help lbfgsb in the MATLAB prompt.

The script exampleldaimages.m is a more complicated example. It uses the L-BFGS-B solver to compute some statistics of a posterior distribution for a text analysis model called latent Dirichlet allication (LDA). The model is used to estimate the topics for a collection of documents. The optimization problem is to compute an approximation to the posterior, since it is not possible to compute the posterior exactly. In this case, the optimization problem corresponds to minimizing the distance between the true posterior distribution and an approximate distribution. For more information, see the paper of Blei, Ng and Jordan (2003).4 In order to run this example, you will need to install the lightspeed toolbox for MATLAB developed by Tom Minka at Microsoft Research.

The script starts by generating a bunch of canonical images that represent topics; if a pixel is white then a word at that position is more likely to appear in that topic. Note that the order of the pixels in the image is unimportant, since LDA is a "bag of words" model. In another figure we show a small sample of the data generated from the topics. Each image represents a document, and the pixels in the image depict the word proportions in that document.5

Next, the script runs the L-BFGS-B solver to find a local minimum to the variational objective. Buried in the M-file mflda.m is the call to the solver: 

  [xi gamma Phi] = lbfgsb({xi gamma Phi},...
      {repmat(lb,W,K)  repmat(lb,K,D)  repmat(lb,K,M)},...
      {repmat(inf,W,K) repmat(inf,K,D) repmat(inf,K,M)},...
      'computeObjectiveMFLDA','computeGradientMFLDA',...
      {nu eta L w lnZconst},'callbackMFLDA',...
      'm',m_lbfgs,'factr',1e-12,'pgtol',...
      tolerance,'maxiter',maxiter);

This example will give us the opportunity to demonstrate some of the more complicated aspects of the our MATLAB interface. The first input to lbfgsb sets the initial iterate of the optimization algorithm. Notice that we've passed a cell array with entries that are matrices. Indeed, it is possible to pass either a matrix or a cell array. Whatever structure is passed, the other input arguments (and outputs) must also abide by this structure. The lower and upper bounds on variables are also cell arrays. All the upper bounds are set to infinity, which means that the variables are only bounded from below.

It is instructive to examine the callback routines. They look like: 

  f = computeObjectiveMFLDA(xi,gamma,Phi,auxdata)

  [dxi, dgamma, dPhi] = ...
      computeGradientMFLDA(xi,gamma,Phi,auxdata)

Notice that each entry to the cell array is its own input argument to the callback routines. The same applies for the output arguments of the gradient callback function.

The sixth input argument is a cell array that contains auxiliary data. This data will be passed to the MATLAB functions that evaluate the objective and gradient. The seventh input argument specifies a callback routine that is called exactly once for every iteration. This callback routine is most useful for examining the progress of the solver. The remaining input arguments are label/value pairs that override some of the default settings of L-BFGS-B.

After converging to a solution, we display topic samples drawn from the variational approximation. A good solution should come close to recovering the true topics from which the data was generated (although they might not be in the same order).

Don't be surprised if it takes many iterations before the optimization algorithm converges on a local minimum. (In repeated trials, I found it as many as two thousand iterations to converge to a stationary point of the objective.) This particular problem demonstrates the limitations of limited-memory quasi-Newton approximations to the Hessian; the low storage requirements can come at the cost of slow convergence to the solution.

The C++ interface

One of the nice byproducts of writing a MATLAB interface for L-BFGS-B is that I've ended up with a neat little C++ class that encapsulates the nuts and bolts of executing the solver. A brief description of the Program class can be found in the header file program.h.

Program is an abstract class, which means that its impossible to instantiate a Program object. This is because the class has member functions that aren't yet implemented. These are called pure virtual functions. In order to use the Program class, one needs to define another class that inherits the Program class and that implements the pure virtual functions. In fact, the class MatlabProgram (in matlabprogram.h) is an example of such a class.

The new child class must declare and implement these two functions:

  virtual double computeObjective (int n, double* x);
  virtual void computeGradient (int n, double* x, double* g);

See the header file for a detailed description of these functions. The only remaining detail is calling the Program constructor. After that, it is just a matter of declaring a new object of type MyProgram then calling the function runSolver.

Contents of the tar archive

solver.f
Fortran 77 source for the L-BFGS-B solver routines.

program.h
program.cpp
Header and source file for the C++ interface.

array.h
arrayofmatrices.h
matlabexception.h
matlabmatrix.h
matlabprogram.h
matlabscalar.h
matlabstring.h
arrayofmatrices.cpp
matlabexception.cpp
matlabmatrix.cpp
matlabprogram.cpp
matlabscalar.cpp
matlabstring.cpp
lbfgsb.cpp
Header and source files for the MATLAB interface.

lbfgsb.m
Usage instructions for the MEX file.

examplehs038.m
MATLAB script for solving the H&S test example no. 38.

computeObjectiveHS038.m
computeGradientHS038.m
genericcallback.m
MATLAB functions used by the script examplehs038.m.

exampleldaimages.m
MATLAB script that generates synthetic documents and topics, then computes a mean field variational approximation to the posterior distribution of the latent Dirichlet allocation model, then displays the result.

mflda.m
This function computes a mean field variational approximation to the posterior distribution of the latent Dirichlet allocation model by minimizing the distance between the variational distribution and the true distribution. It uses the L-BFGS-B to minimize the objective function subject to the bound constraints. The objective function also acts as a lower bound on the logarithm of the denominator that appears from the application of Bayes' rule.4

callbackMFLDA.m
computeObjectiveMFLDA.m
computeGradientMFLDA.m
computelnZ.m
computelnZconst.m
computem.m
dirichletrnd.m
gammarnd.m
createsyntheticdata.m
mfldainit.m
Some functions used by exampleldaimages.m and mflda.m.

Footnotes

1 Ciyou Zhu, Richard H. Byrd, Peihuang Lu and Jorge Nocedal (1997). Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization. ACM Transactions on Mathematical Software, Vol. 23, No. 4., pp. 550-560.

1 Another MATLAB interface for the L-BFGS-B routines has been developed by Liam Stewart at the University of Toronto.

3 Willi Hock and Klaus Schittkowski (1981). Test Examples for Nonlinear Programming Codes. Lecture Notes in Economics and Mathematical Systems, Vol. 187, Springer-Verlag.

4 David M. Blei, Andrew Y. Ng, Michael I. Jordan (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, Vol. 3, pp. 993-1022.

5 Tom L. Griffiths and Mark Steyvers (2004). Finding scientific topics. Proceedings of the National Academy of Sciences, Vol. 101, pp. 5228-5235.


March 3, 2007
pls/lbfgsb/array.h0000644000175000017500000001071411415720432012200 0ustar hnhn#ifndef INCLUDE_ARRAY #define INCLUDE_ARRAY #include "matlabexception.h" #include // Function definitions. // ----------------------------------------------------------------- template void copymemory (const Type* source, Type* dest, int length) { memcpy(dest,source,sizeof(Type)*length); } // Class Array // ----------------------------------------------------------------- // An Array object stores a collection of ordered elements. The key // aspect of Array objects is that they do not necessarily achieve // encapsulation; they do not necessarily retain independent // ownership of their data. That is, the data could be modified // externally by another agent. This behaviour is determined by the // data member "owner". If "owner" is false, the array does not take // care of allocation and deallocation of the data in memory. // // Copy assignment behaves different than usual---it copies the // data, but requires that the destination already have the proper // resources allocated in memory. // // The copy constructor performs a shallow copy. template class Array { public: // This constructor allocates memory for an array of elements of // the specificed length. explicit Array (int length); // This constructor is set to point to an already existing // array. As such, it does not gain ownership of the elements. Array (Type* data, int length); // The copy constructor performs a shallow copy, so that both // arrays share the same copy of the data. Array (const Array& source); // The destructor. ~Array(); // Set all the elements of the array to the specified value. void setvalue (const Type& value); // Copy the elements to from the source array. Both the source and // destination array must have the same length. void inject (const Type* source); void inject (const Array& source); Array& operator= (const Type* source); Array& operator= (const Array& source); // Get the number of elements in the array. int length() const { return n; }; // Elements of an array A can be accessed the subscript operator; // e.g. A[i]. For the sake of efficiency, no checks are made to // ensure that the subscripting is legal. It is up to the user to // ensure that i is non-negative and less than A.length(). Note // that subscripts, as per C++ convention, start at 0. Type& operator[] (int i); const Type& operator[] (int i) const; // Returns true if the two arrays have the same dimensions // (i.e. the same lengths). bool operator== (const Array& a) const { return n == a.n; }; bool operator!= (const Array& a) const { return !(*this == a); }; protected: Type* elems; int n; // The number of elements in the array. bool owner; // Whether the object has ownership of the data. }; // Function definitions for class Array. // ----------------------------------------------------------------- template Array::Array (int length) { n = length; owner = true; elems = 0; elems = new Type[length]; } template Array::Array (Type* data, int length) { n = length; owner = false; elems = data; } template Array::Array (const Array& source) { n = source.n; owner = false; elems = source.elems; } template Array::~Array () { if (owner && elems) delete[] elems; } template void Array::setvalue (const Type& value) { for (int i = 0; i < n; i++) elems[i] = value; } template void Array::inject (const Type* source) { // Check for self-assignment. If there is self-assignment, there // is no need to copy because the objects share the same data! if (elems != source) copymemory(source,elems,n); } template void Array::inject (const Array& source) { if (n != source.n) throw MatlabException("Unable to perform copy; arrays have \ different lengths"); inject(source.elems); } template Array& Array::operator= (const Type* source) { inject(source); return *this; } template Array& Array::operator= (const Array& source) { inject(source); return *this; } template Type& Array::operator[] (int i) { return elems[i]; } template const Type& Array::operator[] (int i) const { return elems[i]; } #endif pls/lbfgsb/arrayofmatrices.cpp0000644000175000017500000000465511415720432014617 0ustar hnhn#include "arrayofmatrices.h" #include "matrix.h" // Function definitions for class ArrayOfMatrices. // ----------------------------------------------------------------- ArrayOfMatrices::ArrayOfMatrices (const mxArray* ptr) : Array(getnummatlabmatrices(ptr)) { setvalue(0); if (mxIsCell(ptr)) // Fill in the entries of the array with the entries of the Matlab // cell array. for (int i = 0; i < n; i++) elems[i] = new Matrix(mxGetCell(ptr,i)); else // Fill in the single entry of the array with the Matlab matrix. elems[0] = new Matrix(ptr); } ArrayOfMatrices::ArrayOfMatrices (const mxArray* ptrs[], int numptrs) : Array(numptrs) { setvalue(0); for (int i = 0; i < numptrs; i++) elems[i] = new Matrix(ptrs[i]); } ArrayOfMatrices::ArrayOfMatrices (mxArray* ptrs[], const ArrayOfMatrices& source) : Array(source.length()) { setvalue(0); // Create the new Matlab matrices and copy the data from the // source. for (int i = 0; i < n; i++) { int h = source[i]->height(); int w = source[i]->width(); elems[i] = new Matrix(ptrs[i],h,w); *elems[i] = *source[i]; } } ArrayOfMatrices::ArrayOfMatrices (const ArrayOfMatrices& source) : Array(source) { } ArrayOfMatrices::ArrayOfMatrices (double* data, const ArrayOfMatrices& model) : Array(model.length()) { setvalue(0); // Repeat for each matrix. for (int i = 0; i < n; i++) { const Matrix* A = model[i]; elems[i] = new Matrix(data,A->height(),A->width()); data += A->length(); } } ArrayOfMatrices::~ArrayOfMatrices() { for (int i = 0; i < n; i++) if (elems[i]) delete elems[i]; } int ArrayOfMatrices::numelems() const { int m = 0; // The return value. // Repeat for each matrix. for (int i = 0; i < n; i++) m += elems[i]->length(); return m; } ArrayOfMatrices& ArrayOfMatrices::operator= (const ArrayOfMatrices& source) { // Copy each matrix. for (int i = 0; i < n; i++) *elems[i] = *source.elems[i]; return *this; } int ArrayOfMatrices::getnummatlabmatrices (const mxArray* ptr) { int n; // The return value. if (mxIsCell(ptr)) n = mxGetNumberOfElements(ptr); else if (mxGetNumberOfDimensions(ptr) == 2 && mxIsDouble(ptr)) n = 1; else throw MatlabException("The Matlab array must either be a cell \ array or a double precision matrix"); return n; } pls/lbfgsb/arrayofmatrices.h0000644000175000017500000000414411415720432014255 0ustar hnhn#ifndef INCLUDE_ARRAYOFMATRICES #define INCLUDE_ARRAYOFMATRICES #include "array.h" #include "matlabmatrix.h" #include "mex.h" // Class ArrayOfMatrices. // ----------------------------------------------------------------- class ArrayOfMatrices : public Array { public: // This version of the constructor behaves just like its parent. explicit ArrayOfMatrices (int length) : Array(length) { }; // This constructor creates an array of matrices from a Matlab // array. It accepts either a matrix in double precision, or a cell // array with entries that are matrices. explicit ArrayOfMatrices (const mxArray* ptr); // This constructor creates an array of matrices from a collection // of Matlab arrays. The Matlab arrays must be matrices. ArrayOfMatrices (const mxArray* ptrs[], int numptrs); // This constructor creates an array of matrices and the // associated Matlab structures. The Matlab structures are // matrices. The second input argument acts as a template for the // creation of the matrices, but the data from "model" is not // actually copied into the new ArrayOfMatrices object. It is up // to the user to make sure that the array of mxArray pointers has // enough room for the pointers to the Matlab arrays. ArrayOfMatrices (mxArray* ptrs[], const ArrayOfMatrices& model); // This constructor creates an array of matrices using the second // input argument as a model. The input argument "data" contains // the element data. Note that the information is NOT copied from // the model! ArrayOfMatrices (double* data, const ArrayOfMatrices& model); // The copy constructor makes a shallow copy of the source object. ArrayOfMatrices (const ArrayOfMatrices& source); // The destructor. ~ArrayOfMatrices(); // Copy assignment operator that observes the same behaviour as // the Array copy assignment operator. ArrayOfMatrices& operator= (const ArrayOfMatrices& source); // Return the total number of elements in all the matrices. int numelems() const; protected: static int getnummatlabmatrices (const mxArray* ptr); }; #endif pls/lbfgsb/lbfgsb.cpp0000644000175000017500000000773411415720432012664 0ustar hnhn#include "mex.h" #include "matrix.h" #include "matlabexception.h" #include "matlabscalar.h" #include "matlabstring.h" #include "matlabmatrix.h" #include "arrayofmatrices.h" #include "program.h" #include "matlabprogram.h" #include #include extern void _main(); // Constants. // ----------------------------------------------------------------- const int minNumInputArgs = 5; // Function definitions. // ----------------------------------------------------------------- void mexFunction (int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) try { // Check to see if we have the correct number of input and output // arguments. if (nrhs < minNumInputArgs) throw MatlabException("Incorrect number of input arguments"); // Get the starting point for the variables. This is specified in // the first input argument. The variables must be either a single // matrix or a cell array of matrices. int k = 0; // The index of the current input argument. ArrayOfMatrices x0(prhs[k++]); // Create the output, which stores the solution obtained from // running IPOPT. There should be as many output arguments as cell // entries in X. if (nlhs != x0.length()) throw MatlabException("Incorrect number of output arguments"); ArrayOfMatrices x(plhs,x0); // Load the lower and upper bounds on the variables as // ArrayOfMatrices objects. They should have the same structure as // the ArrayOfMatrices object "x". ArrayOfMatrices lb(prhs[k++]); ArrayOfMatrices ub(prhs[k++]); // Check to make sure the bounds make sense. if (lb != x || ub != x) throw MatlabException("Input arguments LB and UB must have the same \ structure as X"); // Get the Matlab callback functions. MatlabString objFunc(prhs[k++]); MatlabString gradFunc(prhs[k++]); // Get the auxiliary data. const mxArray* auxData; const mxArray* ptr = prhs[k++]; if (nrhs > 5) { if (mxIsEmpty(ptr)) auxData = 0; else auxData = ptr; } else auxData = 0; // Get the intermediate callback function. MatlabString* iterFunc; ptr = prhs[k++]; if (nrhs > 6) { if (mxIsEmpty(ptr)) iterFunc = 0; else iterFunc = new MatlabString(ptr); } else iterFunc = 0; // Set the options for the L-BFGS algorithm to their defaults. int maxiter = defaultmaxiter; int m = defaultm; double factr = defaultfactr; double pgtol = defaultpgtol; // Process the remaining input arguments, which set options for // the IPOPT algorithm. while (k < nrhs) { // Get the option label from the Matlab input argument. MatlabString optionLabel(prhs[k++]); if (k < nrhs) { // Get the option value from the Matlab input argument. MatlabScalar optionValue(prhs[k++]); double value = optionValue; if (!strcmp(optionLabel,"maxiter")) maxiter = (int) value; else if (!strcmp(optionLabel,"m")) m = (int) value; else if (!strcmp(optionLabel,"factr")) factr = value / mxGetEps(); else if (!strcmp(optionLabel,"pgtol")) pgtol = value; else { if (iterFunc) delete iterFunc; throw MatlabException("Nonexistent option"); } } } // Create a new instance of the optimization problem. x = x0; MatlabProgram program(x,lb,ub,&objFunc,&gradFunc,iterFunc, (mxArray*) auxData,m,maxiter,factr,pgtol); // Run the L-BFGS-B solver. SolverExitStatus exitStatus = program.runSolver(); if (exitStatus == abnormalTermination) { if (iterFunc) delete iterFunc; throw MatlabException("Solver unable to satisfy convergence \ criteria due to abnormal termination"); } else if (exitStatus == errorOnInput) { if (iterFunc) delete iterFunc; throw MatlabException("Invalid inputs to L-BFGS routine"); } // Free the dynamically allocated memory. if (iterFunc) delete iterFunc; } catch (std::exception& error) { mexErrMsgTxt(error.what()); } pls/lbfgsb/license.txt0000644000175000017500000000302711415720432013073 0ustar hnhnCopyright (c) 2009, Peter Carbonetto All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution * Neither the name of the University of British Columbia nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. pls/lbfgsb/matlabexception.cpp0000644000175000017500000000100011415720432014560 0ustar hnhn#include "matlabexception.h" // Function definitions for class MatlabException. // ----------------------------------------------------------------- MatlabException::MatlabException (const char* message) throw() : exception() { this->message = message; } MatlabException::MatlabException (const MatlabException& source) throw() : exception() { message = source.message; } MatlabException& MatlabException::operator= (const MatlabException& source) { message = source.message; return *this; } pls/lbfgsb/matlabexception.h0000644000175000017500000000154511415720432014243 0ustar hnhn#ifndef INCLUDE_MATLABEXCEPTION #define INCLUDE_MATLABEXCEPTION #include // Class MatlabException // ----------------------------------------------------------------- // This class just makes it easier for me to throw exceptions. Its // functionality really has nothing to do with MATLAB. class MatlabException : public std::exception { public: MatlabException (const char* message) throw(); ~MatlabException() throw() { }; // The copy constructor makes a shallow copy. MatlabException (const MatlabException& source) throw(); // The copy assignment operator makes a shallow copy as well. MatlabException& operator= (const MatlabException& source); // Return the message string. virtual const char* what () const throw() { return message; }; private: const char* message; // The error message. }; #endif pls/lbfgsb/matlabmatrix.cpp0000644000175000017500000000315611415720432014104 0ustar hnhn#include "matlabmatrix.h" #include "matlabexception.h" // Function definitions. // ----------------------------------------------------------------- double* getMatlabMatrixDouble (const mxArray* ptr) { if (mxGetNumberOfDimensions(ptr) != 2) throw MatlabException("Matlab array must be a matrix"); if (!mxIsDouble(ptr)) throw MatlabException("Matlab array must be of type double"); return mxGetPr(ptr); } double* createMatlabMatrixDouble (mxArray*& ptr, int height, int width) { ptr = mxCreateDoubleMatrix(height,width,mxREAL); return mxGetPr(ptr); } // Function definitions for class Matrix. // ----------------------------------------------------------------- Matrix::Matrix (int height, int width) : Array(height*width) { h = height; w = width; } Matrix::Matrix (double* data, int height, int width) : Array(data,height*width) { h = height; w = width; } Matrix::Matrix (const mxArray* ptr) : Array(getMatlabMatrixDouble(ptr),mxGetNumberOfElements(ptr)) { h = mxGetM(ptr); w = mxGetN(ptr); } Matrix::Matrix (mxArray*& ptr, int height, int width) : Array(createMatlabMatrixDouble(ptr,height,width), height*width) { h = height; w = width; } Matrix::Matrix (const Matrix& source) : Array(source) { h = source.h; w = source.w; } Matrix& Matrix::operator= (const Matrix& source) { inject(source); return *this; } bool Matrix::operator== (const Matrix& X) const { return (h == X.h) && (w == X.w); } double& Matrix::entry (int r, int c) { return elems[h*c + r]; } double Matrix::entry (int r, int c) const { return elems[h*c + r]; } pls/lbfgsb/matlabmatrix.h0000644000175000017500000000456211415720432013553 0ustar hnhn#ifndef INCLUDE_MATLABMATRIX #define INCLUDE_MATLABMATRIX #include "array.h" #include "mex.h" // Class Matrix // --------------------------------------------------------------- // A matrix object stores its elements in column-major format, as in // Fortran and Matlab. This means that columns are stored one after // another. For example, the matrix // // 1 2 3 // 4 5 6 // // is stored in memory as // // 1 4 2 5 3 6 // // Like Array objects, Matrix objects are not necessarily // encapsulated, and they exhibit analogous behaviour. class Matrix : public Array { public: // This constructor allocates memory for matrix of the specified // heigth and width. Matrix (int height, int width); // This constructor basically follows the lead of the analagous // constructor for the Array class. Matrix (double* data, int height, int width); // This constructor retrieves a matrix from a Matlab array. This // particular constructor is only defined for Matrix. // Since Matlab handles storage, the object created by this // constructor is not encapsulated. explicit Matrix (const mxArray* ptr); // This constructor creates a new Matlab array as a side effect. // Since Matlab handles storage, the object created by this // constructor is not encapsulated. Matrix (mxArray*& ptr, int height, int width); // The copy constructor makes a shallow copy of the data. Matrix (const Matrix& source); // The destructor. ~Matrix() { }; // Copy assignment operator that observes the same behaviour as // the Array copy assignment operator. Matrix& operator= (const Matrix& source); // Get the height and width of the matrix. int height() const { return h; }; int width () const { return w; }; // Returns true if the two matrices have the same dimensions // (i.e. the same height and width). bool operator== (const Matrix& X) const; bool operator!= (const Matrix& X) const { return !(*this == X); }; // If X is an object of type Matrix, X.entry(r,c) accesses the // entry of the rth row and cth column. double entry (int r, int c) const; double& entry (int r, int c); double operator() (int r, int c) const { return entry(r,c); }; double& operator() (int r, int c) { return entry(r,c); }; protected: int h; // The height of the matrix. int w; // The width of the matrix. }; #endif pls/lbfgsb/matlabprogram.cpp0000644000175000017500000001645211415720432014252 0ustar hnhn#include "matlabprogram.h" #include "array.h" // Function definitions. // ----------------------------------------------------------------- // Copy the matrix elements to the destination array. It is assumed // that sufficient memory has been allocated for the destination // array. The copying preserves the column-major ordering of the // matrix elements. void copyElemsFromMatrix (const Matrix& source, double* dest) { Matrix destmatrix(dest,source.height(),source.width()); destmatrix = source; } // Copy the elements from the source array into the matrix. void copyElemsToMatrix (const double* source, Matrix& dest) { dest.inject(source); } // Copy the elements contained in all the matrices to the destination // array. It is assumed that sufficient memory has been allocated for // the destination array. The copying preserves the column-major // ordering of the matrix elements. void copyElemsFromArrayOfMatrices (const ArrayOfMatrices& source, double* dest) { // Repeat for each matrix. for (int i = 0; i < source.length(); i++) { copyElemsFromMatrix(*source[i],dest); dest += source[i]->length(); } } // Copy the elements from the source array into the array of matrices. void copyElemsToArrayOfMatrices (const double* source, ArrayOfMatrices& dest) { // Repeat for each matrix. for (int i = 0; i < dest.length(); i++) { copyElemsToMatrix(source,*dest[i]); source += dest[i]->length(); } } int getBoundType (double& lb, double& ub) { bool haslb = !mxIsInf(lb); bool hasub = !mxIsInf(ub); int btype = haslb + 3*hasub - 2*(haslb && hasub); lb = haslb ? lb : 0; ub = hasub ? ub : 0; return btype; } // Function definitions for class MatlabProgram. // ----------------------------------------------------------------- MatlabProgram::MatlabProgram (ArrayOfMatrices& variables, const ArrayOfMatrices& lowerbounds, const ArrayOfMatrices& upperbounds, const MatlabString* objFunc, const MatlabString* gradFunc, const MatlabString* iterFunc, mxArray* auxData, int m, int maxiter, double factr, double pgtol) : Program(variables.numelems(),0,0,0,0,m,maxiter,factr,pgtol), variables(variables) { x = new double[n]; lb = new double[n]; ub = new double[n]; btype = new int[n]; this->objFunc = objFunc; this->gradFunc = gradFunc; this->iterFunc = iterFunc; // Copy some pieces of information from the input arguments to the // respective program structures. copyElemsFromArrayOfMatrices(variables,x); copyElemsFromArrayOfMatrices(lowerbounds,lb); copyElemsFromArrayOfMatrices(upperbounds,ub); // Set the bound types. for (int i = 0; i < n; i++) btype[i] = getBoundType(lb[i],ub[i]); // Create a Matlab array for the variables. int nv = variables.length(); varInputs = new mxArray*[nv]; varMatlab = new ArrayOfMatrices(varInputs,variables); // Set up the inputs to the objective callback function. numInputsObjFunc = nv + (bool) auxData; inputsObjFunc = new mxArray*[numInputsObjFunc]; copymemory(varInputs,inputsObjFunc,nv); if (auxData) inputsObjFunc[nv] = auxData; // Set up the inputs to the gradient callback function. numInputsGradFunc = numInputsObjFunc; inputsGradFunc = inputsObjFunc; // If necessary, set up the inputs to the iterative callback // function. The first two inputs are the iteration and the function // objective. The next of inputs are the values of the variables // then, optionally, the auxiliary data. if (iterFunc) { numInputsIterFunc = 2 + nv + (bool) auxData; inputsIterFunc = new mxArray*[numInputsIterFunc]; mxArray** ptr = inputsIterFunc; tMatlab = new MatlabScalar(*ptr++,0); fMatlab = new MatlabScalar(*ptr++,0); copymemory(varInputs,ptr,nv); ptr += nv; if (auxData) *ptr = auxData; } } MatlabProgram::~MatlabProgram() { delete[] x; delete[] lb; delete[] ub; delete[] btype; // Deallocate the Matlab arrays. int nv = variables.length(); for (int i = 0; i < nv; i++) mxDestroyArray(varInputs[i]); delete varMatlab; delete[] varInputs; // If necessary, deallocate the additional Matlab arrays that act as // inputs to the iterative callback function. if (iterFunc) { delete fMatlab; delete tMatlab; // delete[] inputsIterFunc; } } double MatlabProgram::computeObjective (int n, double* x) { int nlhs = 1; // The number of outputs from Matlab. mxArray* plhs[nlhs]; // The outputs from the Matlab routine. // Copy the current value of the optimization variables. copyElemsToArrayOfMatrices(x,*varMatlab); // Call the designated Matlab routine for evaluating the objective // function. It takes as input the values of the variables, and // returns a single output, the value of the objective function. if (mexCallMATLAB(nlhs,plhs,numInputsObjFunc,inputsObjFunc,*objFunc)) throw MatlabException("Evaluation of objective function failed in \ call to MATLAB routine"); // Get the result passed back from the Matlab routine. MatlabScalar matlabOutput(plhs[0]); double objective = matlabOutput; // Free the dynamically allocated memory. mxDestroyArray(plhs[0]); return objective; } void MatlabProgram::computeGradient (int n, double* x, double* g) { int nlhs = variables.length(); // The number of outputs from Matlab. mxArray* plhs[nlhs]; // The outputs from the Matlab routine. // Copy the current value of the optimization variables. copyElemsToArrayOfMatrices(x,*varMatlab); // Call the designated Matlab routine for computing the gradient // of the objective function. It takes as input the values of the // variables, and returns as many outputs corresponding to the // partial derivatives of the objective function with respect to // the variables at their curent values. if (mexCallMATLAB(nlhs,plhs,numInputsGradFunc,inputsGradFunc,*gradFunc)) throw MatlabException("Evaluation of objective gradient failed in \ call to MATLAB routine"); // Get the result passed back from the Matlab routine. ArrayOfMatrices matlabOutput((const mxArray**) plhs,nlhs); for (int i = 0; i < variables.length(); i++) if (matlabOutput[i]->length() != variables[i]->length()) throw MatlabException("Invalid gradient passed back from MATLAB \ routine"); copyElemsFromArrayOfMatrices(matlabOutput,g); // Free the dynamically allocated Matlab outputs. for (int i = 0; i < variables.length(); i++) mxDestroyArray(plhs[i]); } void MatlabProgram::iterCallback (int t, double* x, double f) { if (iterFunc) { // Copy the current iteration, the current value of the objective // function, and the current value of the optimization variables. *tMatlab = t; *fMatlab = f; copyElemsToArrayOfMatrices(x,*varMatlab); // Call the Matlab iterative callback routine. Since there are no // outputs, there is no memory to deallocate. if (mexCallMATLAB(0,0,numInputsIterFunc,inputsIterFunc,*iterFunc)) throw MatlabException("Call to MATLAB iterative callback function \ failed"); } } SolverExitStatus MatlabProgram::runSolver() { SolverExitStatus status; // The return value. status = Program::runSolver(); // Copy the solution to the class member "variables". copyElemsToArrayOfMatrices(x,variables); return status; } pls/lbfgsb/matlabprogram.h0000644000175000017500000000550111415720432013710 0ustar hnhn#ifndef INCLUDE_MATLABPROGRAM #define INCLUDE_MATLABPROGRAM #include "mex.h" #include "program.h" #include "matlabscalar.h" #include "matlabstring.h" #include "arrayofmatrices.h" // Class MatlabProgram. // ----------------------------------------------------------------- // This is an implementation of the abstract class Program. class MatlabProgram: public Program { public: // On input, "variables" should contain the initial point for the // optimization routine. If no iterative callback function is // specified, "iterFunc" may be set to 0. Also, if no auxiliary data // is needed, it may also be set to 0. MatlabProgram (ArrayOfMatrices& variables, const ArrayOfMatrices& lowerbounds, const ArrayOfMatrices& upperbounds, const MatlabString* objFunc, const MatlabString* gradFunc, const MatlabString* iterFunc, mxArray* auxData, int m = defaultm, int maxiter = defaultmaxiter, double factr = defaultfactr, double pgtol = defaultpgtol); // The destructor. virtual ~MatlabProgram(); // These provide definitions for the pure virtual functions of the // abstract parent class. virtual double computeObjective (int n, double* x); virtual void computeGradient (int n, double* x, double* g); virtual void iterCallback (int t, double* x, double f); // Run the solver. Upon completion, the solution is stored in // "variables". SolverExitStatus runSolver(); protected: ArrayOfMatrices& variables; // Storage for the initial value and // solution. const MatlabString* objFunc; // The objective callback function. const MatlabString* gradFunc; // The gradient callback function. const MatlabString* iterFunc; // The iterative callback function. ArrayOfMatrices* varMatlab; // Inputs to the Matlab callback mxArray** varInputs; // functions representing the // current values of the variables. MatlabScalar* fMatlab; // Input to the Matlab callback // functions representing the // current value of the objective. MatlabScalar* tMatlab; // Input to the Matlab callback // functions representing the // current iteration. int numInputsObjFunc; // The number of inputs passed to the // objective callback function. int numInputsGradFunc; // The number of inputs passed to the // gradient callback function. int numInputsIterFunc; // The number of inputs passed to the // iterative callback function. mxArray** inputsObjFunc; // The inputs passed to the objective // callback function. mxArray** inputsGradFunc; // The inputs passed to the gradient // callback function. mxArray** inputsIterFunc; // The inputs passed to the iterative // callback function. }; #endif pls/lbfgsb/matlabscalar.cpp0000644000175000017500000000173011415720432014041 0ustar hnhn#include "matlabscalar.h" #include "matlabexception.h" // Function definitions. // ----------------------------------------------------------------- double& getMatlabScalar (const mxArray* ptr) { if (!mxIsDouble(ptr)) throw MatlabException("Matlab array must be of type double"); if (mxGetNumberOfElements(ptr) != 1) throw MatlabException("The Matlab array must be a scalar"); return *mxGetPr(ptr); } double& createMatlabScalar (mxArray*& ptr) { ptr = mxCreateDoubleScalar(0); return *mxGetPr(ptr); } // Function definitions for class MatlabScalar. // --------------------------------------------------------------- MatlabScalar::MatlabScalar (const mxArray* ptr) : x(getMatlabScalar(ptr)) { } MatlabScalar::MatlabScalar (mxArray*& ptr, double value) : x(createMatlabScalar(ptr)) { x = value; } MatlabScalar::MatlabScalar (MatlabScalar& source) : x(source.x) { } MatlabScalar& MatlabScalar::operator= (double value) { x = value; return *this; } pls/lbfgsb/matlabscalar.h0000644000175000017500000000226411415720432013511 0ustar hnhn#ifndef INCLUDE_MATLABSCALAR #define INCLUDE_MATLABSCALAR #include "mex.h" // Class MatlabScalar // ----------------------------------------------------------------- // The main appeal of this class is that one can create a scalar // object that accesses a MATLAB array. // // Note that the copy assignment operator is not valid for this class // because we cannot reassign a reference. class MatlabScalar { public: // This constructor accepts as input a pointer to a Matlab array // which must be a scalar in double precision. explicit MatlabScalar (const mxArray* ptr); // This constructor creates a new Matlab array which is a scalar // in double precision. MatlabScalar (mxArray*& ptr, double value); // The copy constructor. MatlabScalar (MatlabScalar& source); // The destructor. ~MatlabScalar() { }; // Access the value of the scalar. operator const double () const { return x; }; // Assign the value of the scalar. MatlabScalar& operator= (double value); protected: double& x; // The copy assignment operator is kept protected because it is // invalid. MatlabScalar& operator= (const MatlabScalar& source) { return *this; }; }; #endif pls/lbfgsb/matlabstring.cpp0000644000175000017500000000200711415720432014100 0ustar hnhn#include "matlabstring.h" #include "matlabexception.h" // Function definitions. // ----------------------------------------------------------------- char* copystring (const char* source) { int n = strlen(source); // The length of the string. char* dest = new char[n+1]; // The return value. strcpy(dest,source); return dest; } // Function definitions for class MatlabString. // ----------------------------------------------------------------- MatlabString::MatlabString (const mxArray* ptr) { s = 0; // Check to make sure the Matlab array is a string. if (!mxIsChar(ptr)) throw MatlabException("Matlab array must be a string (of type CHAR)"); // Get the string passed as a Matlab array. s = mxArrayToString(ptr); if (s == 0) throw MatlabException("Unable to obtain string from Matlab array"); } MatlabString::MatlabString (const MatlabString& source) { s = 0; // Copy the source string. s = copystring(source.s); } MatlabString::~MatlabString() { if (s) mxFree(s); } pls/lbfgsb/matlabstring.h0000644000175000017500000000220211422527036013545 0ustar hnhn#ifndef INCLUDE_MATLABSTRING #define INCLUDE_MATLABSTRING #include "mex.h" #include // Function declarations. // ----------------------------------------------------------------- // Copy a C-style string (i.e. a null-terminated character array). char* copystring (const char* source); // Class MatlabString. // ----------------------------------------------------------------- // This class encapsulates read-only access to a MATLAB character // array. class MatlabString { public: // The constructor accepts as input a pointer to a Matlab array, // which must be a valid string (array of type CHAR). explicit MatlabString (const mxArray* ptr); // The copy constructor makes a full copy of the source string. MatlabString (const MatlabString& source); // The destructor. ~MatlabString(); // Conversion operator for null-terminated string. operator const char* () const { return s; }; protected: char* s; // The null-terminated string. // The copy assignment operator is not proper, thus remains // protected. MatlabString& operator= (const MatlabString& source) { return *this; }; }; #endif pls/lbfgsb/program.cpp0000644000175000017500000000767511415720432013100 0ustar hnhn#include "program.h" #include // Function definitions. // ----------------------------------------------------------------- // Copy a C-style string (a null-terminated character array) to a // non-C-style string (a simple character array). The length of the // destination character array is given by "ndest". If the source is // shorter than the destination, the destination is padded with blank // characters. void copyCStrToCharArray (const char* source, char* dest, int ndest) { // Get the length of the source C string. int nsource = strlen(source); // Only perform the copy if the source can fit into the destination. if (nsource < ndest) { // Copy the string. strcpy(dest,source); // Fill in the rest of the string with blanks. for (int i = nsource; i < ndest; i++) dest[i] = ' '; } } // Return true if the two strings are the same. The second input // argument must be a C-style string (a null-terminated character // array), but the first input argument need not be one. We only // compare the two strings up to the length of "cstr". bool strIsEqualToCStr (const char* str, const char* cstr) { // Get the length of the C string. int n = strlen(cstr); return !strncmp(str,cstr,n); } // Function definitions for class Program. // ----------------------------------------------------------------- Program::Program (int n, double* x, double* lb, double* ub, int* btype, int m, int maxiter, double factr, double pgtol) { this->n = n; this->x = x; this->lb = lb; this->ub = ub; this->btype = btype; this->m = m; this->maxiter = maxiter; this->factr = factr; this->pgtol = pgtol; this->iprint = defaultprintlevel; this->owner = false; initStructures(); } Program::Program (int n, int m, int maxiter, double factr, double pgtol) { this->n = n; this->x = new double[n]; this->lb = new double[n]; this->ub = new double[n]; this->btype = new int[n]; this->m = m; this->maxiter = maxiter; this->factr = factr; this->pgtol = pgtol; this->iprint = defaultprintlevel; this->owner = true; initStructures(); } Program::~Program() { if (owner) { delete[] x; delete[] lb; delete[] ub; delete[] btype; } delete[] g; delete[] wa; delete[] iwa; } SolverExitStatus Program::runSolver() { SolverExitStatus status = success; // The return value. // Initialize the objective function and gradient to zero. f = 0; for (int i = 0; i < n; i++) g[i] = 0; // This initial call sets up the structures for L-BFGS. callLBFGS("START"); // Repeat until we've reached the maximum number of iterations. int t = 0; while (true) { // Do something according to the "task" from the previous call to // L-BFGS. if (strIsEqualToCStr(task,"FG")) { // Evaluate the objective function and the gradient of the // objective at the current point. f = computeObjective(n,x); computeGradient(n,x,g); } else if (strIsEqualToCStr(task,"NEW_X")) { // Go to the next iteration and call the iterative callback // routine. t++; iterCallback(t,x,f); // If we've reached the maximum number of iterations, terminate // the optimization. if (t == maxiter) { callLBFGS("STOP"); break; } } else if (strIsEqualToCStr(task,"CONV")) break; else if (strIsEqualToCStr(task,"ABNO")) { status = abnormalTermination; break; } else if (strIsEqualToCStr(task,"ERROR")) { status = errorOnInput; break; } // Call L-BFGS again. callLBFGS(); } return status; } void Program::callLBFGS (const char* cmd) { if (cmd) copyCStrToCharArray(cmd,task,60); setulb_(&n,&m,x,lb,ub,btype,&f,g,&factr,&pgtol,wa,iwa,task,&iprint, csave,lsave,isave,dsave); } void Program::initStructures() { f = 0; g = new double[n]; wa = new double[(2*m + 4)*n + 12*m*(m + 1)]; iwa = new int[3*n]; } pls/lbfgsb/program.h0000644000175000017500000001164711415720432012537 0ustar hnhn#ifndef INCLUDE_PROGRAM #define INCLUDE_PROGRAM // Type definitions. // ----------------------------------------------------------------- // This defines the possible results of running the L-BFGS-B solver. enum SolverExitStatus { success, // The algorithm has converged to a stationary // point or has reached the maximum number of // iterations. abnormalTermination, // The algorithm has terminated abnormally and // was unable to satisfy the convergence // criteria. errorOnInput // The routine has detected an error in the // input parameters. }; // Constants. // ----------------------------------------------------------------- const int defaultm = 5; const int defaultmaxiter = 100; const double defaultfactr = 1e7; const double defaultpgtol = 1e-5; const int defaultprintlevel = -1; // Function declarations. // ----------------------------------------------------------------- // This is the L-BFGS-B routine implemented in Fortran 77. extern "C" void setulb_ (int* n, int* m, double x[], double l[], double u[], int nbd[], double* f, double g[], double* factr, double* pgtol, double wa[], int iwa[], char task[], int* iprint, char csave[], bool lsave[], int isave[], double dsave[]); // Class Program. // ----------------------------------------------------------------- // This class encapsulates execution of the L-BFGS-B routine for // solving a nonlinear optimization problem with bound constraints // using limited-memory approximations to the Hessian. // // This is an abstract class since some of the class methods have not // been implemented (they are "pure virtual" functions). In order to // use this class, one needs to define a child class and provide // definitions for the pure virtual methods. class Program { public: // The first input argument "n" is the number of variables, and "x" // must be set to the suggested starting point for the optimization // algorithm. See the L-BFGS-B documentation for more information on // the inputs to the constructor. Program (int n, double* x, double* lb, double* ub, int* btype, int m = defaultm, int maxiter = defaultmaxiter, double factr = defaultfactr, double pgtol = defaultpgtol); // This is the same constructor as above, except that the // appropriate amount of memory is dynamically allocated for the // variables, the bounds, and the bound types. The destructor also // makes sure that the memory is properly deallocated, but not so // for the other constructor. Program (int n, int m = defaultm, int maxiter = defaultmaxiter, double factr = defaultfactr, double pgtol = defaultpgtol); // The destructor. virtual ~Program(); // An implementation of this method should return the value of the // objective at the current value of the variables "x", where "n" is // the number of variables. virtual double computeObjective (int n, double* x) = 0; // An implementation of this method should fill in the values of the // gradient "g" and the current point "x". virtual void computeGradient (int n, double* x, double* g) = 0; // The child class may optionally override this method which is // called once per iteration of the L-BFGS-B routine. The input // arguments are the current iteration "t", the current value of the // variables "x" and the current value of the objective "f". virtual void iterCallback (int t, double* x, double f) { }; // Run the solver. Upon completion, the solution is stored in "x". SolverExitStatus runSolver(); protected: // The copy constructor and copy assignment operator are kept // private so that they are not used. Program (const Program& source) { }; Program& operator= (const Program& source) { return *this; }; int n; // The number of variables. double* x; // The current point. double* lb; // The lower bounds. double* ub; // The upper bounds. int* btype; // The bound types. private: // Execute a single step the L-BFGS-B solver routine. void callLBFGS (const char* cmd = 0); // Initialize some of the structures used by the L-BFGS-B routine. void initStructures(); bool owner; // If true, then the blocks of memory pointed to by // x, lb, ub and btype must be deallocated in the // destructor. double f; // The value of the objective. double* g; // The value of the gradient. int iprint; // The print level. int maxiter; // The maximum number of iterations. double pgtol; // Convergence parameter passed to L-BFGS-B. double factr; // Convergence parameter passed to L-BFGS-B. int m; // The number of variable corrections to the // limited-memory approximation to the Hessian. // These are structures used by the L-BFGS-B routine. double* wa; int* iwa; char task[60]; char csave[60]; bool lsave[4]; int isave[44]; double dsave[29]; }; #endif pls/lbfgsb/solver.f0000644000175000017500000042341011415720432012373 0ustar hnhnc================ L-BFGS-B (version 2.4) ========================== subroutine setulb(n, m, x, l, u, nbd, f, g, factr, pgtol, wa, iwa, + task, iprint, csave, lsave, isave, dsave) character*60 task, csave logical lsave(4) integer n, m, iprint, + nbd(n), iwa(3*n), isave(44) double precision f, factr, pgtol, x(n), l(n), u(n), g(n), + wa(2*m*n+4*n+11*m*m+8*m), dsave(29) c ************ c c Subroutine setulb c c This subroutine partitions the working arrays wa and iwa, and c then uses the limited memory BFGS method to solve the bound c constrained optimization problem by calling mainlb. c (The direct method will be used in the subspace minimization.) c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c x is a double precision array of dimension n. c On entry x is an approximation to the solution. c On exit x is the current approximation. c c l is a double precision array of dimension n. c On entry l is the lower bound on x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound on x. c On exit u is unchanged. c c nbd is an integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, and c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c f is a double precision variable. c On first entry f is unspecified. c On final exit f is the value of the function at x. c c g is a double precision array of dimension n. c On first entry g is unspecified. c On final exit g is the value of the gradient at x. c c factr is a double precision variable. c On entry factr >= 0 is specified by the user. The iteration c will stop when c c (f^k - f^{k+1})/max{|f^k|,|f^{k+1}|,1} <= factr*epsmch c c where epsmch is the machine precision, which is automatically c generated by the code. Typical values for factr: 1.d+12 for c low accuracy; 1.d+7 for moderate accuracy; 1.d+1 for extremely c high accuracy. c On exit factr is unchanged. c c pgtol is a double precision variable. c On entry pgtol >= 0 is specified by the user. The iteration c will stop when c c max{|proj g_i | i = 1, ..., n} <= pgtol c c where pg_i is the ith component of the projected gradient. c On exit pgtol is unchanged. c c wa is a double precision working array of length c (2mmax + 4)nmax + 11mmax^2 + 8mmax. c c iwa is an integer working array of length 3nmax. c c task is a working string of characters of length 60 indicating c the current job when entering and quitting this subroutine. c c iprint is an integer variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c csave is a working string of characters of length 60. c c lsave is a logical working array of dimension 4. c On exit with 'task' = NEW_X, the following information is c available: c If lsave(1) = .true. then the initial X has been replaced by c its projection in the feasible set; c If lsave(2) = .true. then the problem is constrained; c If lsave(3) = .true. then each variable has upper and lower c bounds; c c isave is an integer working array of dimension 44. c On exit with 'task' = NEW_X, the following information is c available: c isave(22) = the total number of intervals explored in the c search of Cauchy points; c isave(26) = the total number of skipped BFGS updates before c the current iteration; c isave(30) = the number of current iteration; c isave(31) = the total number of BFGS updates prior the current c iteration; c isave(33) = the number of intervals explored in the search of c Cauchy point in the current iteration; c isave(34) = the total number of function and gradient c evaluations; c isave(36) = the number of function value or gradient c evaluations in the current iteration; c if isave(37) = 0 then the subspace argmin is within the box; c if isave(37) = 1 then the subspace argmin is beyond the box; c isave(38) = the number of free variables in the current c iteration; c isave(39) = the number of active constraints in the current c iteration; c n + 1 - isave(40) = the number of variables leaving the set of c active constraints in the current iteration; c isave(41) = the number of variables entering the set of active c constraints in the current iteration. c c dsave is a double precision working array of dimension 29. c On exit with 'task' = NEW_X, the following information is c available: c dsave(1) = current 'theta' in the BFGS matrix; c dsave(2) = f(x) in the previous iteration; c dsave(3) = factr*epsmch; c dsave(4) = 2-norm of the line search direction vector; c dsave(5) = the machine precision epsmch generated by the code; c dsave(7) = the accumulated time spent on searching for c Cauchy points; c dsave(8) = the accumulated time spent on c subspace minimization; c dsave(9) = the accumulated time spent on line search; c dsave(11) = the slope of the line search function at c the current point of line search; c dsave(12) = the maximum relative step length imposed in c line search; c dsave(13) = the infinity norm of the projected gradient; c dsave(14) = the relative step length in the line search; c dsave(15) = the slope of the line search function at c the starting point of the line search; c dsave(16) = the square of the 2-norm of the line search c direction vector. c c Subprograms called: c c L-BFGS-B Library ... mainlb. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: a c limited memory FORTRAN code for solving bound constrained c optimization problems'', Tech. Report, NAM-11, EECS Department, c Northwestern University, 1994. c c (Postscript files of these papers are available via anonymous c ftp to ece.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer l1,l2,l3,lws,lr,lz,lt,ld,lwa,lwy,lsy,lss,lwt,lwn,lsnd if (task .eq. 'START') then isave(1) = m*n isave(2) = m**2 isave(3) = 4*m**2 isave(4) = 1 isave(5) = isave(4) + isave(1) isave(6) = isave(5) + isave(1) isave(7) = isave(6) + isave(2) isave(8) = isave(7) + isave(2) isave(9) = isave(8) isave(10) = isave(9) + isave(2) isave(11) = isave(10) + isave(3) isave(12) = isave(11) + isave(3) isave(13) = isave(12) + n isave(14) = isave(13) + n isave(15) = isave(14) + n isave(16) = isave(15) + n endif l1 = isave(1) l2 = isave(2) l3 = isave(3) lws = isave(4) lwy = isave(5) lsy = isave(6) lss = isave(7) lwt = isave(9) lwn = isave(10) lsnd = isave(11) lz = isave(12) lr = isave(13) ld = isave(14) lt = isave(15) lwa = isave(16) call mainlb(n,m,x,l,u,nbd,f,g,factr,pgtol, + wa(lws),wa(lwy),wa(lsy),wa(lss),wa(lwt), + wa(lwn),wa(lsnd),wa(lz),wa(lr),wa(ld),wa(lt), + wa(lwa),iwa(1),iwa(n+1),iwa(2*n+1),task,iprint, + csave,lsave,isave(22),dsave) return end c======================= The end of setulb ============================= subroutine mainlb(n, m, x, l, u, nbd, f, g, factr, pgtol, ws, wy, + sy, ss, wt, wn, snd, z, r, d, t, wa, + index, iwhere, indx2, task, iprint, + csave, lsave, isave, dsave) character*60 task, csave logical lsave(4) integer n, m, iprint, nbd(n), index(n), + iwhere(n), indx2(n), isave(23) double precision f, factr, pgtol, + x(n), l(n), u(n), g(n), z(n), r(n), d(n), t(n), + wa(8*m), ws(n, m), wy(n, m), sy(m, m), ss(m, m), + wt(m, m), wn(2*m, 2*m), snd(2*m, 2*m), dsave(29) c ************ c c Subroutine mainlb c c This subroutine solves bound constrained optimization problems by c using the compact formula of the limited memory BFGS updates. c c n is an integer variable. c On entry n is the number of variables. c On exit n is unchanged. c c m is an integer variable. c On entry m is the maximum number of variable metric c corrections allowed in the limited memory matrix. c On exit m is unchanged. c c x is a double precision array of dimension n. c On entry x is an approximation to the solution. c On exit x is the current approximation. c c l is a double precision array of dimension n. c On entry l is the lower bound of x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound of x. c On exit u is unchanged. c c nbd is an integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c f is a double precision variable. c On first entry f is unspecified. c On final exit f is the value of the function at x. c c g is a double precision array of dimension n. c On first entry g is unspecified. c On final exit g is the value of the gradient at x. c c factr is a double precision variable. c On entry factr >= 0 is specified by the user. The iteration c will stop when c c (f^k - f^{k+1})/max{|f^k|,|f^{k+1}|,1} <= factr*epsmch c c where epsmch is the machine precision, which is automatically c generated by the code. c On exit factr is unchanged. c c pgtol is a double precision variable. c On entry pgtol >= 0 is specified by the user. The iteration c will stop when c c max{|proj g_i | i = 1, ..., n} <= pgtol c c where pg_i is the ith component of the projected gradient. c On exit pgtol is unchanged. c c ws, wy, sy, and wt are double precision working arrays used to c store the following information defining the limited memory c BFGS matrix: c ws, of dimension n x m, stores S, the matrix of s-vectors; c wy, of dimension n x m, stores Y, the matrix of y-vectors; c sy, of dimension m x m, stores S'Y; c ss, of dimension m x m, stores S'S; c wt, of dimension m x m, stores the Cholesky factorization c of (theta*S'S+LD^(-1)L'); see eq. c (2.26) in [3]. c c wn is a double precision working array of dimension 2m x 2m c used to store the LEL^T factorization of the indefinite matrix c K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c c where E = [-I 0] c [ 0 I] c c snd is a double precision working array of dimension 2m x 2m c used to store the lower triangular part of c N = [Y' ZZ'Y L_a'+R_z'] c [L_a +R_z S'AA'S ] c c z(n),r(n),d(n),t(n),wa(8*m) are double precision working arrays. c z is used at different times to store the Cauchy point and c the Newton point. c c c index is an integer working array of dimension n. c In subroutine freev, index is used to store the free and fixed c variables at the Generalized Cauchy Point (GCP). c c iwhere is an integer working array of dimension n used to record c the status of the vector x for GCP computation. c iwhere(i)=0 or -3 if x(i) is free and has bounds, c 1 if x(i) is fixed at l(i), and l(i) .ne. u(i) c 2 if x(i) is fixed at u(i), and u(i) .ne. l(i) c 3 if x(i) is always fixed, i.e., u(i)=x(i)=l(i) c -1 if x(i) is always free, i.e., no bounds on it. c c indx2 is an integer working array of dimension n. c Within subroutine cauchy, indx2 corresponds to the array iorder. c In subroutine freev, a list of variables entering and leaving c the free set is stored in indx2, and it is passed on to c subroutine formk with this information. c c task is a working string of characters of length 60 indicating c the current job when entering and leaving this subroutine. c c iprint is an INTEGER variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c csave is a working string of characters of length 60. c c lsave is a logical working array of dimension 4. c c isave is an integer working array of dimension 23. c c dsave is a double precision working array of dimension 29. c c c Subprograms called c c L-BFGS-B Library ... cauchy, subsm, lnsrlb, formk, c c errclb, prn1lb, prn2lb, prn3lb, active, projgr, c c freev, cmprlb, matupd, formt. c c Minpack2 Library ... timer, dpmeps. c c Linpack Library ... dcopy, ddot. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: FORTRAN c Subroutines for Large Scale Bound Constrained Optimization'' c Tech. Report, NAM-11, EECS Department, Northwestern University, c 1994. c c [3] R. Byrd, J. Nocedal and R. Schnabel "Representations of c Quasi-Newton Matrices and their use in Limited Memory Methods'', c Mathematical Programming 63 (1994), no. 4, pp. 129-156. c c (Postscript files of these papers are available via anonymous c ftp to ece.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ logical prjctd,cnstnd,boxed,updatd,wrk character*3 word integer i,k,nintol,itfile,iback,nskip, + head,col,iter,itail,iupdat, + nint,nfgv,info,ifun, + iword,nfree,nact,ileave,nenter double precision theta,fold,ddot,dr,rr,tol,dpmeps, + xstep,sbgnrm,ddum,dnorm,dtd,epsmch, + cpu1,cpu2,cachyt,sbtime,lnscht,time1,time2, + gd,gdold,stp,stpmx,time double precision one,zero parameter (one=1.0d0,zero=0.0d0) if (task .eq. 'START') then call timer(time1) c Generate the current machine precision. epsmch = dpmeps() fold = 0.0d0 dnorm = 0.0d0 cpu1 = 0.0d0 gd = 0.0d0 sbgnrm = 0.0d0 stp = 0.0d0 stpmx = 0.0d0 gdold = 0.0d0 dtd = 0.0d0 c Initialize counters and scalars when task='START'. c for the limited memory BFGS matrices: col = 0 head = 1 theta = one iupdat = 0 updatd = .false. iback = 0 itail = 0 ifun = 0 iword = 0 nact = 0 ileave = 0 nenter = 0 c for operation counts: iter = 0 nfgv = 0 nint = 0 nintol = 0 nskip = 0 nfree = n c for stopping tolerance: tol = factr*epsmch c for measuring running time: cachyt = 0 sbtime = 0 lnscht = 0 c 'word' records the status of subspace solutions. word = '---' c 'info' records the termination information. info = 0 itfile = 0 if (iprint .ge. 1) then c open a summary file 'iterate.dat' open (8, file = 'iterate.dat', status = 'unknown') itfile = 8 endif c Check the input arguments for errors. call errclb(n,m,factr,l,u,nbd,task,info,k) if (task(1:5) .eq. 'ERROR') then call prn3lb(n,x,f,task,iprint,info,itfile, + iter,nfgv,nintol,nskip,nact,sbgnrm, + zero,nint,word,iback,stp,xstep,k, + cachyt,sbtime,lnscht) return endif call prn1lb(n,m,l,u,x,iprint,itfile,epsmch) c Initialize iwhere & project x onto the feasible set. call active(n,l,u,nbd,x,iwhere,iprint,prjctd,cnstnd,boxed) c The end of the initialization. else c restore local variables. prjctd = lsave(1) cnstnd = lsave(2) boxed = lsave(3) updatd = lsave(4) nintol = isave(1) itfile = isave(3) iback = isave(4) nskip = isave(5) head = isave(6) col = isave(7) itail = isave(8) iter = isave(9) iupdat = isave(10) nint = isave(12) nfgv = isave(13) info = isave(14) ifun = isave(15) iword = isave(16) nfree = isave(17) nact = isave(18) ileave = isave(19) nenter = isave(20) theta = dsave(1) fold = dsave(2) tol = dsave(3) dnorm = dsave(4) epsmch = dsave(5) cpu1 = dsave(6) cachyt = dsave(7) sbtime = dsave(8) lnscht = dsave(9) time1 = dsave(10) gd = dsave(11) stpmx = dsave(12) sbgnrm = dsave(13) stp = dsave(14) gdold = dsave(15) dtd = dsave(16) c After returning from the driver go to the point where execution c is to resume. if (task(1:5) .eq. 'FG_LN') goto 666 if (task(1:5) .eq. 'NEW_X') goto 777 if (task(1:5) .eq. 'FG_ST') goto 111 if (task(1:4) .eq. 'STOP') then if (task(7:9) .eq. 'CPU') then c restore the previous iterate. call dcopy(n,t,1,x,1) call dcopy(n,r,1,g,1) f = fold endif goto 999 endif endif c Compute f0 and g0. task = 'FG_START' c return to the driver to calculate f and g; reenter at 111. goto 1000 111 continue nfgv = 1 c Compute the infinity norm of the (-) projected gradient. call projgr(n,l,u,nbd,x,g,sbgnrm) if (iprint .ge. 1) then write (6,1002) iter,f,sbgnrm write (itfile,1003) iter,nfgv,sbgnrm,f endif if (sbgnrm .le. pgtol) then c terminate the algorithm. task = 'CONVERGENCE: NORM OF PROJECTED GRADIENT <= PGTOL' goto 999 endif c ----------------- the beginning of the loop -------------------------- 222 continue if (iprint .ge. 99) write (6,1001) iter + 1 iword = -1 c if (.not. cnstnd .and. col .gt. 0) then c skip the search for GCP. call dcopy(n,x,1,z,1) wrk = updatd nint = 0 goto 333 endif cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Compute the Generalized Cauchy Point (GCP). c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc call timer(cpu1) call cauchy(n,x,l,u,nbd,g,indx2,iwhere,t,d,z, + m,wy,ws,sy,wt,theta,col,head, + wa(1),wa(2*m+1),wa(4*m+1),wa(6*m+1),nint, + iprint,sbgnrm,info,epsmch) if (info .ne. 0) then c singular triangular system detected; refresh the lbfgs memory. if(iprint .ge. 1) write (6, 1005) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. call timer(cpu2) cachyt = cachyt + cpu2 - cpu1 goto 222 endif call timer(cpu2) cachyt = cachyt + cpu2 - cpu1 nintol = nintol + nint c Count the entering and leaving variables for iter > 0; c find the index set of free and active variables at the GCP. call freev(n,nfree,index,nenter,ileave,indx2, + iwhere,wrk,updatd,cnstnd,iprint,iter) nact = n - nfree 333 continue c If there are no free variables or B=theta*I, then c skip the subspace minimization. if (nfree .eq. 0 .or. col .eq. 0) goto 555 cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Subspace minimization. c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc call timer(cpu1) c Form the LEL^T factorization of the indefinite c matrix K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c where E = [-I 0] c [ 0 I] if (wrk) call formk(n,nfree,index,nenter,ileave,indx2,iupdat, + updatd,wn,snd,m,ws,wy,sy,theta,col,head,info) if (info .ne. 0) then c nonpositive definiteness in Cholesky factorization; c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1006) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. call timer(cpu2) sbtime = sbtime + cpu2 - cpu1 goto 222 endif c compute r=-Z'B(xcp-xk)-Z'g (using wa(2m+1)=W'(xcp-x) c from 'cauchy'). call cmprlb(n,m,x,g,ws,wy,sy,wt,z,r,wa,index, + theta,col,head,nfree,cnstnd,info) if (info .ne. 0) goto 444 c call the direct method. call subsm(n,m,nfree,index,l,u,nbd,z,r,ws,wy,theta, + col,head,iword,wa,wn,iprint,info) 444 continue if (info .ne. 0) then c singular triangular system detected; c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1005) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. call timer(cpu2) sbtime = sbtime + cpu2 - cpu1 goto 222 endif call timer(cpu2) sbtime = sbtime + cpu2 - cpu1 555 continue cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Line search and optimality tests. c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c Generate the search direction d:=z-x. do 40 i = 1, n d(i) = z(i) - x(i) 40 continue call timer(cpu1) 666 continue call lnsrlb(n,l,u,nbd,x,f,fold,gd,gdold,g,d,r,t,z,stp,dnorm, + dtd,xstep,stpmx,iter,ifun,iback,nfgv,info,task, + boxed,cnstnd,csave,isave(22),dsave(17)) if (info .ne. 0 .or. iback .ge. 20) then c restore the previous iterate. call dcopy(n,t,1,x,1) call dcopy(n,r,1,g,1) f = fold if (col .eq. 0) then c abnormal termination. if (info .eq. 0) then info = -9 c restore the actual number of f and g evaluations etc. nfgv = nfgv - 1 ifun = ifun - 1 iback = iback - 1 endif task = 'ABNORMAL_TERMINATION_IN_LNSRCH' iter = iter + 1 goto 999 else c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1008) if (info .eq. 0) nfgv = nfgv - 1 info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. task = 'RESTART_FROM_LNSRCH' call timer(cpu2) lnscht = lnscht + cpu2 - cpu1 goto 222 endif else if (task(1:5) .eq. 'FG_LN') then c return to the driver for calculating f and g; reenter at 666. goto 1000 else c calculate and print out the quantities related to the new X. call timer(cpu2) lnscht = lnscht + cpu2 - cpu1 iter = iter + 1 c Compute the infinity norm of the projected (-)gradient. call projgr(n,l,u,nbd,x,g,sbgnrm) c Print iteration information. call prn2lb(n,x,f,g,iprint,itfile,iter,nfgv,nact, + sbgnrm,nint,word,iword,iback,stp,xstep) goto 1000 endif 777 continue c Test for termination. if (sbgnrm .le. pgtol) then c terminate the algorithm. task = 'CONVERGENCE: NORM OF PROJECTED GRADIENT <= PGTOL' goto 999 endif ddum = max(abs(fold), abs(f), one) if ((fold - f) .le. tol*ddum) then c terminate the algorithm. task = 'CONVERGENCE: REL_REDUCTION_OF_F <= FACTR*EPSMCH' if (iback .ge. 10) info = -5 c i.e., to issue a warning if iback>10 in the line search. goto 999 endif c Compute d=newx-oldx, r=newg-oldg, rr=y'y and dr=y's. do 42 i = 1, n r(i) = g(i) - r(i) 42 continue rr = ddot(n,r,1,r,1) if (stp .eq. one) then dr = gd - gdold ddum = -gdold else dr = (gd - gdold)*stp call dscal(n,stp,d,1) ddum = -gdold*stp endif if (dr .le. epsmch*ddum) then c skip the L-BFGS update. nskip = nskip + 1 updatd = .false. if (iprint .ge. 1) write (6,1004) dr, ddum goto 888 endif cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Update the L-BFGS matrix. c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc updatd = .true. iupdat = iupdat + 1 c Update matrices WS and WY and form the middle matrix in B. call matupd(n,m,ws,wy,sy,ss,d,r,itail, + iupdat,col,head,theta,rr,dr,stp,dtd) c Form the upper half of the pds T = theta*SS + L*D^(-1)*L'; c Store T in the upper triangular of the array wt; c Cholesky factorize T to J*J' with c J' stored in the upper triangular of wt. call formt(m,wt,sy,ss,col,theta,info) if (info .ne. 0) then c nonpositive definiteness in Cholesky factorization; c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1007) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. goto 222 endif c Now the inverse of the middle matrix in B is c [ D^(1/2) O ] [ -D^(1/2) D^(-1/2)*L' ] c [ -L*D^(-1/2) J ] [ 0 J' ] 888 continue c -------------------- the end of the loop ----------------------------- goto 222 999 continue call timer(time2) time = time2 - time1 call prn3lb(n,x,f,task,iprint,info,itfile, + iter,nfgv,nintol,nskip,nact,sbgnrm, + time,nint,word,iback,stp,xstep,k, + cachyt,sbtime,lnscht) 1000 continue c Save local variables. lsave(1) = prjctd lsave(2) = cnstnd lsave(3) = boxed lsave(4) = updatd isave(1) = nintol isave(3) = itfile isave(4) = iback isave(5) = nskip isave(6) = head isave(7) = col isave(8) = itail isave(9) = iter isave(10) = iupdat isave(12) = nint isave(13) = nfgv isave(14) = info isave(15) = ifun isave(16) = iword isave(17) = nfree isave(18) = nact isave(19) = ileave isave(20) = nenter dsave(1) = theta dsave(2) = fold dsave(3) = tol dsave(4) = dnorm dsave(5) = epsmch dsave(6) = cpu1 dsave(7) = cachyt dsave(8) = sbtime dsave(9) = lnscht dsave(10) = time1 dsave(11) = gd dsave(12) = stpmx dsave(13) = sbgnrm dsave(14) = stp dsave(15) = gdold dsave(16) = dtd 1001 format (//,'ITERATION ',i5) 1002 format + (/,'At iterate',i5,4x,'f= ',1p,d12.5,4x,'|proj g|= ',1p,d12.5) 1003 format (2(1x,i4),5x,'-',5x,'-',3x,'-',5x,'-',5x,'-',8x,'-',3x, + 1p,2(1x,d10.3)) 1004 format (' ys=',1p,e10.3,' -gs=',1p,e10.3,' BFGS update SKIPPED') 1005 format (/, +' Singular triangular system detected;',/, +' refresh the lbfgs memory and restart the iteration.') 1006 format (/, +' Nonpositive definiteness in Cholesky factorization in formk;',/, +' refresh the lbfgs memory and restart the iteration.') 1007 format (/, +' Nonpositive definiteness in Cholesky factorization in formt;',/, +' refresh the lbfgs memory and restart the iteration.') 1008 format (/, +' Bad direction in the line search;',/, +' refresh the lbfgs memory and restart the iteration.') return end c======================= The end of mainlb ============================= subroutine active(n, l, u, nbd, x, iwhere, iprint, + prjctd, cnstnd, boxed) logical prjctd, cnstnd, boxed integer n, iprint, nbd(n), iwhere(n) double precision x(n), l(n), u(n) c ************ c c Subroutine active c c This subroutine initializes iwhere and projects the initial x to c the feasible set if necessary. c c iwhere is an integer array of dimension n. c On entry iwhere is unspecified. c On exit iwhere(i)=-1 if x(i) has no bounds c 3 if l(i)=u(i) c 0 otherwise. c In cauchy, iwhere is given finer gradations. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer nbdd,i double precision zero parameter (zero=0.0d0) c Initialize nbdd, prjctd, cnstnd and boxed. nbdd = 0 prjctd = .false. cnstnd = .false. boxed = .true. c Project the initial x to the easible set if necessary. do 10 i = 1, n if (nbd(i) .gt. 0) then if (nbd(i) .le. 2 .and. x(i) .le. l(i)) then if (x(i) .lt. l(i)) then prjctd = .true. x(i) = l(i) endif nbdd = nbdd + 1 else if (nbd(i) .ge. 2 .and. x(i) .ge. u(i)) then if (x(i) .gt. u(i)) then prjctd = .true. x(i) = u(i) endif nbdd = nbdd + 1 endif endif 10 continue c Initialize iwhere and assign values to cnstnd and boxed. do 20 i = 1, n if (nbd(i) .ne. 2) boxed = .false. if (nbd(i) .eq. 0) then c this variable is always free iwhere(i) = -1 c otherwise set x(i)=mid(x(i), u(i), l(i)). else cnstnd = .true. if (nbd(i) .eq. 2 .and. u(i) - l(i) .le. zero) then c this variable is always fixed iwhere(i) = 3 else iwhere(i) = 0 endif endif 20 continue if (iprint .ge. 0) then if (prjctd) write (6,*) + 'The initial X is infeasible. Restart with its projection.' if (.not. cnstnd) + write (6,*) 'This problem is unconstrained.' endif if (iprint .gt. 0) write (6,1001) nbdd 1001 format (/,'At X0 ',i9,' variables are exactly at the bounds') return end c======================= The end of active ============================= subroutine bmv(m, sy, wt, col, v, p, info) integer m, col, info double precision sy(m, m), wt(m, m), v(2*col), p(2*col) c ************ c c Subroutine bmv c c This subroutine computes the product of the 2m x 2m middle matrix c in the compact L-BFGS formula of B and a 2m vector v; c it returns the product in p. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c sy is a double precision array of dimension m x m. c On entry sy specifies the matrix S'Y. c On exit sy is unchanged. c c wt is a double precision array of dimension m x m. c On entry wt specifies the upper triangular matrix J' which is c the Cholesky factor of (thetaS'S+LD^(-1)L'). c On exit wt is unchanged. c c col is an integer variable. c On entry col specifies the number of s-vectors (or y-vectors) c stored in the compact L-BFGS formula. c On exit col is unchanged. c c v is a double precision array of dimension 2col. c On entry v specifies vector v. c On exit v is unchanged. c c p is a double precision array of dimension 2col. c On entry p is unspecified. c On exit p is the product Mv. c c info is an integer variable. c On entry info is unspecified. c On exit info = 0 for normal return, c = nonzero for abnormal return when the system c to be solved by dtrsl is singular. c c Subprograms called: c c Linpack ... dtrsl. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,k,i2 double precision sum if (col .eq. 0) return c PART I: solve [ D^(1/2) O ] [ p1 ] = [ v1 ] c [ -L*D^(-1/2) J ] [ p2 ] [ v2 ]. c solve Jp2=v2+LD^(-1)v1. p(col + 1) = v(col + 1) do 20 i = 2, col i2 = col + i sum = 0.0d0 do 10 k = 1, i - 1 sum = sum + sy(i,k)*v(k)/sy(k,k) 10 continue p(i2) = v(i2) + sum 20 continue c Solve the triangular system call dtrsl(wt,m,col,p(col+1),11,info) if (info .ne. 0) return c solve D^(1/2)p1=v1. cc do 30 i = 1, col cc p(i) = v(i)/sqrt(sy(i,i)) cc30 continue c PART II: solve [ -D^(1/2) D^(-1/2)*L' ] [ p1 ] = [ p1 ] c [ 0 J' ] [ p2 ] [ p2 ]. c solve J^Tp2=p2. call dtrsl(wt,m,col,p(col+1),01,info) if (info .ne. 0) return c compute p1=-D^(-1/2)(p1-D^(-1/2)L'p2) c =-D^(-1/2)p1+D^(-1)L'p2. do 40 i = 1, col cc p(i) = -p(i)/sqrt(sy(i,i)) combined with do 30 loop cc into the next line p(i) = -v(i)/sy(i,i) 40 continue do 60 i = 1, col sum = 0.d0 do 50 k = i + 1, col sum = sum + sy(k,i)*p(col+k)/sy(i,i) 50 continue p(i) = p(i) + sum 60 continue return end c======================== The end of bmv =============================== subroutine cauchy(n, x, l, u, nbd, g, iorder, iwhere, t, d, xcp, + m, wy, ws, sy, wt, theta, col, head, p, c, wbp, + v, nint, iprint, sbgnrm, info, epsmch) integer n, m, head, col, nint, iprint, info, + nbd(n), iorder(n), iwhere(n) double precision theta, epsmch, + x(n), l(n), u(n), g(n), t(n), d(n), xcp(n), + wy(n, col), ws(n, col), sy(m, m), + wt(m, m), p(2*m), c(2*m), wbp(2*m), v(2*m) c ************ c c Subroutine cauchy c c For given x, l, u, g (with sbgnrm > 0), and a limited memory c BFGS matrix B defined in terms of matrices WY, WS, WT, and c scalars head, col, and theta, this subroutine computes the c generalized Cauchy point (GCP), defined as the first local c minimizer of the quadratic c c Q(x + s) = g's + 1/2 s'Bs c c along the projected gradient direction P(x-tg,l,u). c The routine returns the GCP in xcp. c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c x is a double precision array of dimension n. c On entry x is the starting point for the GCP computation. c On exit x is unchanged. c c l is a double precision array of dimension n. c On entry l is the lower bound of x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound of x. c On exit u is unchanged. c c nbd is an integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, and c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c g is a double precision array of dimension n. c On entry g is the gradient of f(x). g must be a nonzero vector. c On exit g is unchanged. c c iorder is an integer working array of dimension n. c iorder will be used to store the breakpoints in the piecewise c linear path and free variables encountered. On exit, c iorder(1),...,iorder(nleft) are indices of breakpoints c which have not been encountered; c iorder(nleft+1),...,iorder(nbreak) are indices of c encountered breakpoints; and c iorder(nfree),...,iorder(n) are indices of variables which c have no bound constraits along the search direction. c c iwhere is an integer array of dimension n. c On entry iwhere indicates only the permanently fixed (iwhere=3) c or free (iwhere= -1) components of x. c On exit iwhere records the status of the current x variables. c iwhere(i)=-3 if x(i) is free and has bounds, but is not moved c 0 if x(i) is free and has bounds, and is moved c 1 if x(i) is fixed at l(i), and l(i) .ne. u(i) c 2 if x(i) is fixed at u(i), and u(i) .ne. l(i) c 3 if x(i) is always fixed, i.e., u(i)=x(i)=l(i) c -1 if x(i) is always free, i.e., it has no bounds. c c t is a double precision working array of dimension n. c t will be used to store the break points. c c d is a double precision array of dimension n used to store c the Cauchy direction P(x-tg)-x. c c xcp is a double precision array of dimension n used to return the c GCP on exit. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c ws, wy, sy, and wt are double precision arrays. c On entry they store information that defines the c limited memory BFGS matrix: c ws(n,m) stores S, a set of s-vectors; c wy(n,m) stores Y, a set of y-vectors; c sy(m,m) stores S'Y; c wt(m,m) stores the c Cholesky factorization of (theta*S'S+LD^(-1)L'). c On exit these arrays are unchanged. c c theta is a double precision variable. c On entry theta is the scaling factor specifying B_0 = theta I. c On exit theta is unchanged. c c col is an integer variable. c On entry col is the actual number of variable metric c corrections stored so far. c On exit col is unchanged. c c head is an integer variable. c On entry head is the location of the first s-vector c (or y-vector) in S (or Y). c On exit col is unchanged. c c p is a double precision working array of dimension 2m. c p will be used to store the vector p = W^(T)d. c c c is a double precision working array of dimension 2m. c c will be used to store the vector c = W^(T)(xcp-x). c c wbp is a double precision working array of dimension 2m. c wbp will be used to store the row of W corresponding c to a breakpoint. c c v is a double precision working array of dimension 2m. c c nint is an integer variable. c On exit nint records the number of quadratic segments explored c in searching for the GCP. c c iprint is an INTEGER variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c sbgnrm is a double precision variable. c On entry sbgnrm is the norm of the projected gradient at x. c On exit sbgnrm is unchanged. c c info is an integer variable. c On entry info is 0. c On exit info = 0 for normal return, c = nonzero for abnormal return when the the system c used in routine bmv is singular. c c Subprograms called: c c L-BFGS-B Library ... hpsolb, bmv. c c Linpack ... dscal dcopy, daxpy. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: FORTRAN c Subroutines for Large Scale Bound Constrained Optimization'' c Tech. Report, NAM-11, EECS Department, Northwestern University, c 1994. c c (Postscript files of these papers are available via anonymous c ftp to ece.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ logical xlower,xupper,bnded integer i,j,col2,nfree,nbreak,pointr, + ibp,nleft,ibkmin,iter double precision f1,f2,dt,dtm,tsum,dibp,zibp,dibp2,bkmin, + tu,tl,wmc,wmp,wmw,ddot,tj,tj0,neggi,sbgnrm, + f2_org double precision one,zero parameter (one=1.0d0,zero=0.0d0) c Check the status of the variables, reset iwhere(i) if necessary; c compute the Cauchy direction d and the breakpoints t; initialize c the derivative f1 and the vector p = W'd (for theta = 1). if (sbgnrm .le. zero) then if (iprint .ge. 0) write (6,*) 'Subgnorm = 0. GCP = X.' call dcopy(n,x,1,xcp,1) return endif bnded = .true. nfree = n + 1 nbreak = 0 ibkmin = 0 bkmin = zero col2 = 2*col f1 = zero if (iprint .ge. 99) write (6,3010) c We set p to zero and build it up as we determine d. do 20 i = 1, col2 p(i) = zero 20 continue c In the following loop we determine for each variable its bound c status and its breakpoint, and update p accordingly. c Smallest breakpoint is identified. do 50 i = 1, n neggi = -g(i) if (iwhere(i) .ne. 3 .and. iwhere(i) .ne. -1) then c if x(i) is not a constant and has bounds, c compute the difference between x(i) and its bounds. if (nbd(i) .le. 2) tl = x(i) - l(i) if (nbd(i) .ge. 2) tu = u(i) - x(i) c If a variable is close enough to a bound c we treat it as at bound. xlower = nbd(i) .le. 2 .and. tl .le. zero xupper = nbd(i) .ge. 2 .and. tu .le. zero c reset iwhere(i). iwhere(i) = 0 if (xlower) then if (neggi .le. zero) iwhere(i) = 1 else if (xupper) then if (neggi .ge. zero) iwhere(i) = 2 else if (abs(neggi) .le. zero) iwhere(i) = -3 endif endif pointr = head if (iwhere(i) .ne. 0 .and. iwhere(i) .ne. -1) then d(i) = zero else d(i) = neggi f1 = f1 - neggi*neggi c calculate p := p - W'e_i* (g_i). do 40 j = 1, col p(j) = p(j) + wy(i,pointr)* neggi p(col + j) = p(col + j) + ws(i,pointr)*neggi pointr = mod(pointr,m) + 1 40 continue if (nbd(i) .le. 2 .and. nbd(i) .ne. 0 + .and. neggi .lt. zero) then c x(i) + d(i) is bounded; compute t(i). nbreak = nbreak + 1 iorder(nbreak) = i t(nbreak) = tl/(-neggi) if (nbreak .eq. 1 .or. t(nbreak) .lt. bkmin) then bkmin = t(nbreak) ibkmin = nbreak endif else if (nbd(i) .ge. 2 .and. neggi .gt. zero) then c x(i) + d(i) is bounded; compute t(i). nbreak = nbreak + 1 iorder(nbreak) = i t(nbreak) = tu/neggi if (nbreak .eq. 1 .or. t(nbreak) .lt. bkmin) then bkmin = t(nbreak) ibkmin = nbreak endif else c x(i) + d(i) is not bounded. nfree = nfree - 1 iorder(nfree) = i if (abs(neggi) .gt. zero) bnded = .false. endif endif 50 continue c The indices of the nonzero components of d are now stored c in iorder(1),...,iorder(nbreak) and iorder(nfree),...,iorder(n). c The smallest of the nbreak breakpoints is in t(ibkmin)=bkmin. if (theta .ne. one) then c complete the initialization of p for theta not= one. call dscal(col,theta,p(col+1),1) endif c Initialize GCP xcp = x. call dcopy(n,x,1,xcp,1) if (nbreak .eq. 0 .and. nfree .eq. n + 1) then c is a zero vector, return with the initial xcp as GCP. if (iprint .gt. 100) write (6,1010) (xcp(i), i = 1, n) return endif c Initialize c = W'(xcp - x) = 0. do 60 j = 1, col2 c(j) = zero 60 continue c Initialize derivative f2. f2 = -theta*f1 f2_org = f2 if (col .gt. 0) then call bmv(m,sy,wt,col,p,v,info) if (info .ne. 0) return f2 = f2 - ddot(col2,v,1,p,1) endif dtm = -f1/f2 tsum = zero nint = 1 if (iprint .ge. 99) + write (6,*) 'There are ',nbreak,' breakpoints ' c If there are no breakpoints, locate the GCP and return. if (nbreak .eq. 0) goto 888 nleft = nbreak iter = 1 tj = zero c------------------- the beginning of the loop ------------------------- 777 continue c Find the next smallest breakpoint; c compute dt = t(nleft) - t(nleft + 1). tj0 = tj if (iter .eq. 1) then c Since we already have the smallest breakpoint we need not do c heapsort yet. Often only one breakpoint is used and the c cost of heapsort is avoided. tj = bkmin ibp = iorder(ibkmin) else if (iter .eq. 2) then c Replace the already used smallest breakpoint with the c breakpoint numbered nbreak > nlast, before heapsort call. if (ibkmin .ne. nbreak) then t(ibkmin) = t(nbreak) iorder(ibkmin) = iorder(nbreak) endif c Update heap structure of breakpoints c (if iter=2, initialize heap). endif call hpsolb(nleft,t,iorder,iter-2) tj = t(nleft) ibp = iorder(nleft) endif dt = tj - tj0 if (dt .ne. zero .and. iprint .ge. 100) then write (6,4011) nint,f1,f2 write (6,5010) dt write (6,6010) dtm endif c If a minimizer is within this interval, c locate the GCP and return. if (dtm .lt. dt) goto 888 c Otherwise fix one variable and c reset the corresponding component of d to zero. tsum = tsum + dt nleft = nleft - 1 iter = iter + 1 dibp = d(ibp) d(ibp) = zero if (dibp .gt. zero) then zibp = u(ibp) - x(ibp) xcp(ibp) = u(ibp) iwhere(ibp) = 2 else zibp = l(ibp) - x(ibp) xcp(ibp) = l(ibp) iwhere(ibp) = 1 endif if (iprint .ge. 100) write (6,*) 'Variable ',ibp,' is fixed.' if (nleft .eq. 0 .and. nbreak .eq. n) then c all n variables are fixed, c return with xcp as GCP. dtm = dt goto 999 endif c Update the derivative information. nint = nint + 1 dibp2 = dibp**2 c Update f1 and f2. c temporarily set f1 and f2 for col=0. f1 = f1 + dt*f2 + dibp2 - theta*dibp*zibp f2 = f2 - theta*dibp2 if (col .gt. 0) then c update c = c + dt*p. call daxpy(col2,dt,p,1,c,1) c choose wbp, c the row of W corresponding to the breakpoint encountered. pointr = head do 70 j = 1,col wbp(j) = wy(ibp,pointr) wbp(col + j) = theta*ws(ibp,pointr) pointr = mod(pointr,m) + 1 70 continue c compute (wbp)Mc, (wbp)Mp, and (wbp)M(wbp)'. call bmv(m,sy,wt,col,wbp,v,info) if (info .ne. 0) return wmc = ddot(col2,c,1,v,1) wmp = ddot(col2,p,1,v,1) wmw = ddot(col2,wbp,1,v,1) c update p = p - dibp*wbp. call daxpy(col2,-dibp,wbp,1,p,1) c complete updating f1 and f2 while col > 0. f1 = f1 + dibp*wmc f2 = f2 + 2.0d0*dibp*wmp - dibp2*wmw endif f2 = max(epsmch*f2_org,f2) if (nleft .gt. 0) then dtm = -f1/f2 goto 777 c to repeat the loop for unsearched intervals. else if(bnded) then f1 = zero f2 = zero dtm = zero else dtm = -f1/f2 endif c------------------- the end of the loop ------------------------------- 888 continue if (iprint .ge. 99) then write (6,*) write (6,*) 'GCP found in this segment' write (6,4010) nint,f1,f2 write (6,6010) dtm endif if (dtm .le. zero) dtm = zero tsum = tsum + dtm c Move free variables (i.e., the ones w/o breakpoints) and c the variables whose breakpoints haven't been reached. call daxpy(n,tsum,d,1,xcp,1) 999 continue c Update c = c + dtm*p = W'(x^c - x) c which will be used in computing r = Z'(B(x^c - x) + g). if (col .gt. 0) call daxpy(col2,dtm,p,1,c,1) if (iprint .gt. 100) write (6,1010) (xcp(i),i = 1,n) if (iprint .ge. 99) write (6,2010) 1010 format ('Cauchy X = ',/,(4x,1p,6(1x,d11.4))) 2010 format (/,'---------------- exit CAUCHY----------------------',/) 3010 format (/,'---------------- CAUCHY entered-------------------') 4010 format ('Piece ',i3,' --f1, f2 at start point ',1p,2(1x,d11.4)) 4011 format (/,'Piece ',i3,' --f1, f2 at start point ', + 1p,2(1x,d11.4)) 5010 format ('Distance to the next break point = ',1p,d11.4) 6010 format ('Distance to the stationary point = ',1p,d11.4) return end c====================== The end of cauchy ============================== subroutine cmprlb(n, m, x, g, ws, wy, sy, wt, z, r, wa, index, + theta, col, head, nfree, cnstnd, info) logical cnstnd integer n, m, col, head, nfree, info, index(n) double precision theta, + x(n), g(n), z(n), r(n), wa(4*m), + ws(n, m), wy(n, m), sy(m, m), wt(m, m) c ************ c c Subroutine cmprlb c c This subroutine computes r=-Z'B(xcp-xk)-Z'g by using c wa(2m+1)=W'(xcp-x) from subroutine cauchy. c c Subprograms called: c c L-BFGS-B Library ... bmv. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,j,k,pointr double precision a1,a2 if (.not. cnstnd .and. col .gt. 0) then do 26 i = 1, n r(i) = -g(i) 26 continue else do 30 i = 1, nfree k = index(i) r(i) = -theta*(z(k) - x(k)) - g(k) 30 continue call bmv(m,sy,wt,col,wa(2*m+1),wa(1),info) if (info .ne. 0) then info = -8 return endif pointr = head do 34 j = 1, col a1 = wa(j) a2 = theta*wa(col + j) do 32 i = 1, nfree k = index(i) r(i) = r(i) + wy(k,pointr)*a1 + ws(k,pointr)*a2 32 continue pointr = mod(pointr,m) + 1 34 continue endif return end c======================= The end of cmprlb ============================= subroutine errclb(n, m, factr, l, u, nbd, task, info, k) character*60 task integer n, m, info, k, nbd(n) double precision factr, l(n), u(n) c ************ c c Subroutine errclb c c This subroutine checks the validity of the input data. c c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i double precision zero parameter (zero=0.0d0) c Check the input arguments for errors. if (n .le. 0) task = 'ERROR: N .LE. 0' if (m .le. 0) task = 'ERROR: M .LE. 0' if (factr .lt. zero) task = 'ERROR: FACTR .LT. 0' c Check the validity of the arrays nbd(i), u(i), and l(i). do 10 i = 1, n if (nbd(i) .lt. 0 .or. nbd(i) .gt. 3) then c return task = 'ERROR: INVALID NBD' info = -6 k = i endif if (nbd(i) .eq. 2) then if (l(i) .gt. u(i)) then c return task = 'ERROR: NO FEASIBLE SOLUTION' info = -7 k = i endif endif 10 continue return end c======================= The end of errclb ============================= subroutine formk(n, nsub, ind, nenter, ileave, indx2, iupdat, + updatd, wn, wn1, m, ws, wy, sy, theta, col, + head, info) integer n, nsub, m, col, head, nenter, ileave, iupdat, + info, ind(n), indx2(n) double precision theta, wn(2*m, 2*m), wn1(2*m, 2*m), + ws(n, m), wy(n, m), sy(m, m) logical updatd c ************ c c Subroutine formk c c This subroutine forms the LEL^T factorization of the indefinite c c matrix K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c where E = [-I 0] c [ 0 I] c The matrix K can be shown to be equal to the matrix M^[-1]N c occurring in section 5.1 of [1], as well as to the matrix c Mbar^[-1] Nbar in section 5.3. c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c nsub is an integer variable c On entry nsub is the number of subspace variables in free set. c On exit nsub is not changed. c c ind is an integer array of dimension nsub. c On entry ind specifies the indices of subspace variables. c On exit ind is unchanged. c c nenter is an integer variable. c On entry nenter is the number of variables entering the c free set. c On exit nenter is unchanged. c c ileave is an integer variable. c On entry indx2(ileave),...,indx2(n) are the variables leaving c the free set. c On exit ileave is unchanged. c c indx2 is an integer array of dimension n. c On entry indx2(1),...,indx2(nenter) are the variables entering c the free set, while indx2(ileave),...,indx2(n) are the c variables leaving the free set. c On exit indx2 is unchanged. c c iupdat is an integer variable. c On entry iupdat is the total number of BFGS updates made so far. c On exit iupdat is unchanged. c c updatd is a logical variable. c On entry 'updatd' is true if the L-BFGS matrix is updatd. c On exit 'updatd' is unchanged. c c wn is a double precision array of dimension 2m x 2m. c On entry wn is unspecified. c On exit the upper triangle of wn stores the LEL^T factorization c of the 2*col x 2*col indefinite matrix c [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c c wn1 is a double precision array of dimension 2m x 2m. c On entry wn1 stores the lower triangular part of c [Y' ZZ'Y L_a'+R_z'] c [L_a+R_z S'AA'S ] c in the previous iteration. c On exit wn1 stores the corresponding updated matrices. c The purpose of wn1 is just to store these inner products c so they can be easily updated and inserted into wn. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c ws, wy, sy, and wtyy are double precision arrays; c theta is a double precision variable; c col is an integer variable; c head is an integer variable. c On entry they store the information defining the c limited memory BFGS matrix: c ws(n,m) stores S, a set of s-vectors; c wy(n,m) stores Y, a set of y-vectors; c sy(m,m) stores S'Y; c wtyy(m,m) stores the Cholesky factorization c of (theta*S'S+LD^(-1)L') c theta is the scaling factor specifying B_0 = theta I; c col is the number of variable metric corrections stored; c head is the location of the 1st s- (or y-) vector in S (or Y). c On exit they are unchanged. c c info is an integer variable. c On entry info is unspecified. c On exit info = 0 for normal return; c = -1 when the 1st Cholesky factorization failed; c = -2 when the 2st Cholesky factorization failed. c c Subprograms called: c c Linpack ... dcopy, dpofa, dtrsl. c c c References: c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: a c limited memory FORTRAN code for solving bound constrained c optimization problems'', Tech. Report, NAM-11, EECS Department, c Northwestern University, 1994. c c (Postscript files of these papers are available via anonymous c ftp to ece.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer m2,ipntr,jpntr,iy,is,jy,js,is1,js1,k1,i,k, + col2,pbegin,pend,dbegin,dend,upcl double precision ddot,temp1,temp2,temp3,temp4 double precision zero parameter (zero=0.0d0) c Form the lower triangular part of c WN1 = [Y' ZZ'Y L_a'+R_z'] c [L_a+R_z S'AA'S ] c where L_a is the strictly lower triangular part of S'AA'Y c R_z is the upper triangular part of S'ZZ'Y. if (updatd) then if (iupdat .gt. m) then c shift old part of WN1. do 10 jy = 1, m - 1 js = m + jy call dcopy(m-jy,wn1(jy+1,jy+1),1,wn1(jy,jy),1) call dcopy(m-jy,wn1(js+1,js+1),1,wn1(js,js),1) call dcopy(m-1,wn1(m+2,jy+1),1,wn1(m+1,jy),1) 10 continue endif c put new rows in blocks (1,1), (2,1) and (2,2). pbegin = 1 pend = nsub dbegin = nsub + 1 dend = n iy = col is = m + col ipntr = head + col - 1 if (ipntr .gt. m) ipntr = ipntr - m jpntr = head do 20 jy = 1, col js = m + jy temp1 = zero temp2 = zero temp3 = zero c compute element jy of row 'col' of Y'ZZ'Y do 15 k = pbegin, pend k1 = ind(k) temp1 = temp1 + wy(k1,ipntr)*wy(k1,jpntr) 15 continue c compute elements jy of row 'col' of L_a and S'AA'S do 16 k = dbegin, dend k1 = ind(k) temp2 = temp2 + ws(k1,ipntr)*ws(k1,jpntr) temp3 = temp3 + ws(k1,ipntr)*wy(k1,jpntr) 16 continue wn1(iy,jy) = temp1 wn1(is,js) = temp2 wn1(is,jy) = temp3 jpntr = mod(jpntr,m) + 1 20 continue c put new column in block (2,1). jy = col jpntr = head + col - 1 if (jpntr .gt. m) jpntr = jpntr - m ipntr = head do 30 i = 1, col is = m + i temp3 = zero c compute element i of column 'col' of R_z do 25 k = pbegin, pend k1 = ind(k) temp3 = temp3 + ws(k1,ipntr)*wy(k1,jpntr) 25 continue ipntr = mod(ipntr,m) + 1 wn1(is,jy) = temp3 30 continue upcl = col - 1 else upcl = col endif c modify the old parts in blocks (1,1) and (2,2) due to changes c in the set of free variables. ipntr = head do 45 iy = 1, upcl is = m + iy jpntr = head do 40 jy = 1, iy js = m + jy temp1 = zero temp2 = zero temp3 = zero temp4 = zero do 35 k = 1, nenter k1 = indx2(k) temp1 = temp1 + wy(k1,ipntr)*wy(k1,jpntr) temp2 = temp2 + ws(k1,ipntr)*ws(k1,jpntr) 35 continue do 36 k = ileave, n k1 = indx2(k) temp3 = temp3 + wy(k1,ipntr)*wy(k1,jpntr) temp4 = temp4 + ws(k1,ipntr)*ws(k1,jpntr) 36 continue wn1(iy,jy) = wn1(iy,jy) + temp1 - temp3 wn1(is,js) = wn1(is,js) - temp2 + temp4 jpntr = mod(jpntr,m) + 1 40 continue ipntr = mod(ipntr,m) + 1 45 continue c modify the old parts in block (2,1). ipntr = head do 60 is = m + 1, m + upcl jpntr = head do 55 jy = 1, upcl temp1 = zero temp3 = zero do 50 k = 1, nenter k1 = indx2(k) temp1 = temp1 + ws(k1,ipntr)*wy(k1,jpntr) 50 continue do 51 k = ileave, n k1 = indx2(k) temp3 = temp3 + ws(k1,ipntr)*wy(k1,jpntr) 51 continue if (is .le. jy + m) then wn1(is,jy) = wn1(is,jy) + temp1 - temp3 else wn1(is,jy) = wn1(is,jy) - temp1 + temp3 endif jpntr = mod(jpntr,m) + 1 55 continue ipntr = mod(ipntr,m) + 1 60 continue c Form the upper triangle of WN = [D+Y' ZZ'Y/theta -L_a'+R_z' ] c [-L_a +R_z S'AA'S*theta] m2 = 2*m do 70 iy = 1, col is = col + iy is1 = m + iy do 65 jy = 1, iy js = col + jy js1 = m + jy wn(jy,iy) = wn1(iy,jy)/theta wn(js,is) = wn1(is1,js1)*theta 65 continue do 66 jy = 1, iy - 1 wn(jy,is) = -wn1(is1,jy) 66 continue do 67 jy = iy, col wn(jy,is) = wn1(is1,jy) 67 continue wn(iy,iy) = wn(iy,iy) + sy(iy,iy) 70 continue c Form the upper triangle of c WN= [ LL' L^-1(-L_a'+R_z')] c [(-L_a +R_z)L'^-1 S'AA'S*theta ] c first Cholesky factor (1,1) block of wn to get LL' c with L' stored in the upper triangle of wn. call dpofa(wn,m2,col,info) if (info .ne. 0) then info = -1 return endif c then form L^-1(-L_a'+R_z') in the (1,2) block. col2 = 2*col do 71 js = col+1 ,col2 call dtrsl(wn,m2,col,wn(1,js),11,info) 71 continue c Form S'AA'S*theta + (L^-1(-L_a'+R_z'))'L^-1(-L_a'+R_z') in the c upper triangle of (2,2) block of wn. do 72 is = col+1, col2 do 74 js = is, col2 wn(is,js) = wn(is,js) + ddot(col,wn(1,is),1,wn(1,js),1) 74 continue 72 continue c Cholesky factorization of (2,2) block of wn. call dpofa(wn(col+1,col+1),m2,col,info) if (info .ne. 0) then info = -2 return endif return end c======================= The end of formk ============================== subroutine formt(m, wt, sy, ss, col, theta, info) integer m, col, info double precision theta, wt(m, m), sy(m, m), ss(m, m) c ************ c c Subroutine formt c c This subroutine forms the upper half of the pos. def. and symm. c T = theta*SS + L*D^(-1)*L', stores T in the upper triangle c of the array wt, and performs the Cholesky factorization of T c to produce J*J', with J' stored in the upper triangle of wt. c c Subprograms called: c c Linpack ... dpofa. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,j,k,k1 double precision ddum double precision zero parameter (zero=0.0d0) c Form the upper half of T = theta*SS + L*D^(-1)*L', c store T in the upper triangle of the array wt. do 52 j = 1, col wt(1,j) = theta*ss(1,j) 52 continue do 55 i = 2, col do 54 j = i, col k1 = min(i,j) - 1 ddum = zero do 53 k = 1, k1 ddum = ddum + sy(i,k)*sy(j,k)/sy(k,k) 53 continue wt(i,j) = ddum + theta*ss(i,j) 54 continue 55 continue c Cholesky factorize T to J*J' with c J' stored in the upper triangle of wt. call dpofa(wt,m,col,info) if (info .ne. 0) then info = -3 endif return end c======================= The end of formt ============================== subroutine freev(n, nfree, index, nenter, ileave, indx2, + iwhere, wrk, updatd, cnstnd, iprint, iter) integer n, nfree, nenter, ileave, iprint, iter, + index(n), indx2(n), iwhere(n) logical wrk, updatd, cnstnd c ************ c c Subroutine freev c c This subroutine counts the entering and leaving variables when c iter > 0, and finds the index set of free and active variables c at the GCP. c c cnstnd is a logical variable indicating whether bounds are present c c index is an integer array of dimension n c for i=1,...,nfree, index(i) are the indices of free variables c for i=nfree+1,...,n, index(i) are the indices of bound variables c On entry after the first iteration, index gives c the free variables at the previous iteration. c On exit it gives the free variables based on the determination c in cauchy using the array iwhere. c c indx2 is an integer array of dimension n c On entry indx2 is unspecified. c On exit with iter>0, indx2 indicates which variables c have changed status since the previous iteration. c For i= 1,...,nenter, indx2(i) have changed from bound to free. c For i= ileave+1,...,n, indx2(i) have changed from free to bound. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer iact,i,k nenter = 0 ileave = n + 1 if (iter .gt. 0 .and. cnstnd) then c count the entering and leaving variables. do 20 i = 1, nfree k = index(i) if (iwhere(k) .gt. 0) then ileave = ileave - 1 indx2(ileave) = k if (iprint .ge. 100) write (6,*) + 'Variable ',k,' leaves the set of free variables' endif 20 continue do 22 i = 1 + nfree, n k = index(i) if (iwhere(k) .le. 0) then nenter = nenter + 1 indx2(nenter) = k if (iprint .ge. 100) write (6,*) + 'Variable ',k,' enters the set of free variables' endif 22 continue if (iprint .ge. 99) write (6,*) + n+1-ileave,' variables leave; ',nenter,' variables enter' endif wrk = (ileave .lt. n+1) .or. (nenter .gt. 0) .or. updatd c Find the index set of free and active variables at the GCP. nfree = 0 iact = n + 1 do 24 i = 1, n if (iwhere(i) .le. 0) then nfree = nfree + 1 index(nfree) = i else iact = iact - 1 index(iact) = i endif 24 continue if (iprint .ge. 99) write (6,*) + nfree,' variables are free at GCP ',iter + 1 return end c======================= The end of freev ============================== subroutine hpsolb(n, t, iorder, iheap) integer iheap, n, iorder(n) double precision t(n) c ************ c c Subroutine hpsolb c c This subroutine sorts out the least element of t, and puts the c remaining elements of t in a heap. c c n is an integer variable. c On entry n is the dimension of the arrays t and iorder. c On exit n is unchanged. c c t is a double precision array of dimension n. c On entry t stores the elements to be sorted, c On exit t(n) stores the least elements of t, and t(1) to t(n-1) c stores the remaining elements in the form of a heap. c c iorder is an integer array of dimension n. c On entry iorder(i) is the index of t(i). c On exit iorder(i) is still the index of t(i), but iorder may be c permuted in accordance with t. c c iheap is an integer variable specifying the task. c On entry iheap should be set as follows: c iheap .eq. 0 if t(1) to t(n) is not in the form of a heap, c iheap .ne. 0 if otherwise. c On exit iheap is unchanged. c c c References: c Algorithm 232 of CACM (J. W. J. Williams): HEAPSORT. c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c ************ integer i,j,k,indxin,indxou double precision ddum,out if (iheap .eq. 0) then c Rearrange the elements t(1) to t(n) to form a heap. do 20 k = 2, n ddum = t(k) indxin = iorder(k) c Add ddum to the heap. i = k 10 continue if (i.gt.1) then j = i/2 if (ddum .lt. t(j)) then t(i) = t(j) iorder(i) = iorder(j) i = j goto 10 endif endif t(i) = ddum iorder(i) = indxin 20 continue endif c Assign to 'out' the value of t(1), the least member of the heap, c and rearrange the remaining members to form a heap as c elements 1 to n-1 of t. if (n .gt. 1) then i = 1 out = t(1) indxou = iorder(1) ddum = t(n) indxin = iorder(n) c Restore the heap 30 continue j = i+i if (j .le. n-1) then if (t(j+1) .lt. t(j)) j = j+1 if (t(j) .lt. ddum ) then t(i) = t(j) iorder(i) = iorder(j) i = j goto 30 endif endif t(i) = ddum iorder(i) = indxin c Put the least member in t(n). t(n) = out iorder(n) = indxou endif return end c====================== The end of hpsolb ============================== subroutine lnsrlb(n, l, u, nbd, x, f, fold, gd, gdold, g, d, r, t, + z, stp, dnorm, dtd, xstep, stpmx, iter, ifun, + iback, nfgv, info, task, boxed, cnstnd, csave, + isave, dsave) character*60 task, csave logical boxed, cnstnd integer n, iter, ifun, iback, nfgv, info, + nbd(n), isave(2) double precision f, fold, gd, gdold, stp, dnorm, dtd, xstep, + stpmx, x(n), l(n), u(n), g(n), d(n), r(n), t(n), + z(n), dsave(13) c ********** c c Subroutine lnsrlb c c This subroutine calls subroutine dcsrch from the Minpack2 library c to perform the line search. Subroutine dscrch is safeguarded so c that all trial points lie within the feasible region. c c Subprograms called: c c Minpack2 Library ... dcsrch. c c Linpack ... dtrsl, ddot. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ********** integer i double precision ddot,a1,a2 double precision one,zero,big parameter (one=1.0d0,zero=0.0d0,big=1.0d+10) double precision ftol,gtol,xtol parameter (ftol=1.0d-3,gtol=0.9d0,xtol=0.1d0) if (task(1:5) .eq. 'FG_LN') goto 556 dtd = ddot(n,d,1,d,1) dnorm = sqrt(dtd) c Determine the maximum step length. stpmx = big if (cnstnd) then if (iter .eq. 0) then stpmx = one else do 43 i = 1, n a1 = d(i) if (nbd(i) .ne. 0) then if (a1 .lt. zero .and. nbd(i) .le. 2) then a2 = l(i) - x(i) if (a2 .ge. zero) then stpmx = zero else if (a1*stpmx .lt. a2) then stpmx = a2/a1 endif else if (a1 .gt. zero .and. nbd(i) .ge. 2) then a2 = u(i) - x(i) if (a2 .le. zero) then stpmx = zero else if (a1*stpmx .gt. a2) then stpmx = a2/a1 endif endif endif 43 continue endif endif if (iter .eq. 0 .and. .not. boxed) then stp = min(one/dnorm, stpmx) else stp = one endif call dcopy(n,x,1,t,1) call dcopy(n,g,1,r,1) fold = f ifun = 0 iback = 0 csave = 'START' 556 continue gd = ddot(n,g,1,d,1) if (ifun .eq. 0) then gdold=gd if (gd .ge. zero) then c the directional derivative >=0. c Line search is impossible. info = -4 return endif endif call dcsrch(f,gd,stp,ftol,gtol,xtol,zero,stpmx,csave,isave,dsave) xstep = stp*dnorm if (csave(1:4) .ne. 'CONV' .and. csave(1:4) .ne. 'WARN') then task = 'FG_LNSRCH' ifun = ifun + 1 nfgv = nfgv + 1 iback = ifun - 1 if (stp .eq. one) then call dcopy(n,z,1,x,1) else do 41 i = 1, n x(i) = stp*d(i) + t(i) 41 continue endif else task = 'NEW_X' endif return end c======================= The end of lnsrlb ============================= subroutine matupd(n, m, ws, wy, sy, ss, d, r, itail, + iupdat, col, head, theta, rr, dr, stp, dtd) integer n, m, itail, iupdat, col, head double precision theta, rr, dr, stp, dtd, d(n), r(n), + ws(n, m), wy(n, m), sy(m, m), ss(m, m) c ************ c c Subroutine matupd c c This subroutine updates matrices WS and WY, and forms the c middle matrix in B. c c Subprograms called: c c Linpack ... dcopy, ddot. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer j,pointr double precision ddot double precision one parameter (one=1.0d0) c Set pointers for matrices WS and WY. if (iupdat .le. m) then col = iupdat itail = mod(head+iupdat-2,m) + 1 else itail = mod(itail,m) + 1 head = mod(head,m) + 1 endif c Update matrices WS and WY. call dcopy(n,d,1,ws(1,itail),1) call dcopy(n,r,1,wy(1,itail),1) c Set theta=yy/ys. theta = rr/dr c Form the middle matrix in B. c update the upper triangle of SS, c and the lower triangle of SY: if (iupdat .gt. m) then c move old information do 50 j = 1, col - 1 call dcopy(j,ss(2,j+1),1,ss(1,j),1) call dcopy(col-j,sy(j+1,j+1),1,sy(j,j),1) 50 continue endif c add new information: the last row of SY c and the last column of SS: pointr = head do 51 j = 1, col - 1 sy(col,j) = ddot(n,d,1,wy(1,pointr),1) ss(j,col) = ddot(n,ws(1,pointr),1,d,1) pointr = mod(pointr,m) + 1 51 continue if (stp .eq. one) then ss(col,col) = dtd else ss(col,col) = stp*stp*dtd endif sy(col,col) = dr return end c======================= The end of matupd ============================= subroutine prn1lb(n, m, l, u, x, iprint, itfile, epsmch) integer n, m, iprint, itfile double precision epsmch, x(n), l(n), u(n) c ************ c c Subroutine prn1lb c c This subroutine prints the input data, initial point, upper and c lower bounds of each variable, machine precision, as well as c the headings of the output. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i if (iprint .ge. 0) then write (6,7001) epsmch write (6,*) 'N = ',n,' M = ',m if (iprint .ge. 1) then write (itfile,2001) epsmch write (itfile,*)'N = ',n,' M = ',m write (itfile,9001) if (iprint .gt. 100) then write (6,1004) 'L =',(l(i),i = 1,n) write (6,1004) 'X0 =',(x(i),i = 1,n) write (6,1004) 'U =',(u(i),i = 1,n) endif endif endif 1004 format (/,a4, 1p, 6(1x,d11.4),/,(4x,1p,6(1x,d11.4))) 2001 format ('RUNNING THE L-BFGS-B CODE',/,/, + 'it = iteration number',/, + 'nf = number of function evaluations',/, + 'nint = number of segments explored during the Cauchy search',/, + 'nact = number of active bounds at the generalized Cauchy point' + ,/, + 'sub = manner in which the subspace minimization terminated:' + ,/,' con = converged, bnd = a bound was reached',/, + 'itls = number of iterations performed in the line search',/, + 'stepl = step length used',/, + 'tstep = norm of the displacement (total step)',/, + 'projg = norm of the projected gradient',/, + 'f = function value',/,/, + ' * * *',/,/, + 'Machine precision =',1p,d10.3) 7001 format ('RUNNING THE L-BFGS-B CODE',/,/, + ' * * *',/,/, + 'Machine precision =',1p,d10.3) 9001 format (/,3x,'it',3x,'nf',2x,'nint',2x,'nact',2x,'sub',2x,'itls', + 2x,'stepl',4x,'tstep',5x,'projg',8x,'f') return end c======================= The end of prn1lb ============================= subroutine prn2lb(n, x, f, g, iprint, itfile, iter, nfgv, nact, + sbgnrm, nint, word, iword, iback, stp, xstep) character*3 word integer n, iprint, itfile, iter, nfgv, nact, nint, + iword, iback double precision f, sbgnrm, stp, xstep, x(n), g(n) c ************ c c Subroutine prn2lb c c This subroutine prints out new information after a successful c line search. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,imod c 'word' records the status of subspace solutions. if (iword .eq. 0) then c the subspace minimization converged. word = 'con' else if (iword .eq. 1) then c the subspace minimization stopped at a bound. word = 'bnd' else if (iword .eq. 5) then c the truncated Newton step has been used. word = 'TNT' else word = '---' endif if (iprint .ge. 99) then write (6,*) 'LINE SEARCH',iback,' times; norm of step = ',xstep write (6,2001) iter,f,sbgnrm if (iprint .gt. 100) then write (6,1004) 'X =',(x(i), i = 1, n) write (6,1004) 'G =',(g(i), i = 1, n) endif else if (iprint .gt. 0) then imod = mod(iter,iprint) if (imod .eq. 0) write (6,2001) iter,f,sbgnrm endif if (iprint .ge. 1) write (itfile,3001) + iter,nfgv,nint,nact,word,iback,stp,xstep,sbgnrm,f 1004 format (/,a4, 1p, 6(1x,d11.4),/,(4x,1p,6(1x,d11.4))) 2001 format + (/,'At iterate',i5,4x,'f= ',1p,d12.5,4x,'|proj g|= ',1p,d12.5) 3001 format(2(1x,i4),2(1x,i5),2x,a3,1x,i4,1p,2(2x,d7.1),1p,2(1x,d10.3)) return end c======================= The end of prn2lb ============================= subroutine prn3lb(n, x, f, task, iprint, info, itfile, + iter, nfgv, nintol, nskip, nact, sbgnrm, + time, nint, word, iback, stp, xstep, k, + cachyt, sbtime, lnscht) character*60 task character*3 word integer n, iprint, info, itfile, iter, nfgv, nintol, + nskip, nact, nint, iback, k double precision f, sbgnrm, time, stp, xstep, cachyt, sbtime, + lnscht, x(n) c ************ c c Subroutine prn3lb c c This subroutine prints out information when either a built-in c convergence test is satisfied or when an error message is c generated. c c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i if (task(1:5) .eq. 'ERROR') goto 999 if (iprint .ge. 0) then write (6,3003) write (6,3004) write(6,3005) n,iter,nfgv,nintol,nskip,nact,sbgnrm,f if (iprint .ge. 100) then write (6,1004) 'X =',(x(i),i = 1,n) endif if (iprint .ge. 1) write (6,*) ' F =',f endif 999 continue if (iprint .ge. 0) then write (6,3009) task if (info .ne. 0) then if (info .eq. -1) write (6,9011) if (info .eq. -2) write (6,9012) if (info .eq. -3) write (6,9013) if (info .eq. -4) write (6,9014) if (info .eq. -5) write (6,9015) if (info .eq. -6) write (6,*)' Input nbd(',k,') is invalid.' if (info .eq. -7) + write (6,*)' l(',k,') > u(',k,'). No feasible solution.' if (info .eq. -8) write (6,9018) if (info .eq. -9) write (6,9019) endif if (iprint .ge. 1) write (6,3007) cachyt,sbtime,lnscht write (6,3008) time if (iprint .ge. 1) then if (info .eq. -4 .or. info .eq. -9) then write (itfile,3002) + iter,nfgv,nint,nact,word,iback,stp,xstep endif write (itfile,3009) task if (info .ne. 0) then if (info .eq. -1) write (itfile,9011) if (info .eq. -2) write (itfile,9012) if (info .eq. -3) write (itfile,9013) if (info .eq. -4) write (itfile,9014) if (info .eq. -5) write (itfile,9015) if (info .eq. -8) write (itfile,9018) if (info .eq. -9) write (itfile,9019) endif write (itfile,3008) time endif endif 1004 format (/,a4, 1p, 6(1x,d11.4),/,(4x,1p,6(1x,d11.4))) 3002 format(2(1x,i4),2(1x,i5),2x,a3,1x,i4,1p,2(2x,d7.1),6x,'-',10x,'-') 3003 format (/, + ' * * *',/,/, + 'Tit = total number of iterations',/, + 'Tnf = total number of function evaluations',/, + 'Tnint = total number of segments explored during', + ' Cauchy searches',/, + 'Skip = number of BFGS updates skipped',/, + 'Nact = number of active bounds at final generalized', + ' Cauchy point',/, + 'Projg = norm of the final projected gradient',/, + 'F = final function value',/,/, + ' * * *') 3004 format (/,3x,'N',3x,'Tit',2x,'Tnf',2x,'Tnint',2x, + 'Skip',2x,'Nact',5x,'Projg',8x,'F') 3005 format (i5,2(1x,i4),(1x,i6),(2x,i4),(1x,i5),1p,2(2x,d10.3)) 3007 format (/,' Cauchy time',1p,e10.3,' seconds.',/ + ' Subspace minimization time',1p,e10.3,' seconds.',/ + ' Line search time',1p,e10.3,' seconds.') 3008 format (/,' Total User time',1p,e10.3,' seconds.',/) 3009 format (/,a60) 9011 format (/, +' Matrix in 1st Cholesky factorization in formk is not Pos. Def.') 9012 format (/, +' Matrix in 2st Cholesky factorization in formk is not Pos. Def.') 9013 format (/, +' Matrix in the Cholesky factorization in formt is not Pos. Def.') 9014 format (/, +' Derivative >= 0, backtracking line search impossible.',/, +' Previous x, f and g restored.',/, +' Possible causes: 1 error in function or gradient evaluation;',/, +' 2 rounding errors dominate computation.') 9015 format (/, +' Warning: more than 10 function and gradient',/, +' evaluations in the last line search. Termination',/, +' may possibly be caused by a bad search direction.') 9018 format (/,' The triangular system is singular.') 9019 format (/, +' Line search cannot locate an adequate point after 20 function',/ +,' and gradient evaluations. Previous x, f and g restored.',/, +' Possible causes: 1 error in function or gradient evaluation;',/, +' 2 rounding error dominate computation.') return end c======================= The end of prn3lb ============================= subroutine projgr(n, l, u, nbd, x, g, sbgnrm) integer n, nbd(n) double precision sbgnrm, x(n), l(n), u(n), g(n) c ************ c c Subroutine projgr c c This subroutine computes the infinity norm of the projected c gradient. c c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i double precision gi double precision zero parameter (zero=0.0d0) sbgnrm = zero do 15 i = 1, n gi = g(i) if (nbd(i) .ne. 0) then if (gi .lt. zero) then if (nbd(i) .ge. 2) gi = max((x(i)-u(i)),gi) else if (nbd(i) .le. 2) gi = min((x(i)-l(i)),gi) endif endif sbgnrm = max(sbgnrm,abs(gi)) 15 continue return end c======================= The end of projgr ============================= subroutine subsm(n, m, nsub, ind, l, u, nbd, x, d, ws, wy, theta, + col, head, iword, wv, wn, iprint, info) integer n, m, nsub, col, head, iword, iprint, info, + ind(nsub), nbd(n) double precision theta, + l(n), u(n), x(n), d(n), + ws(n, m), wy(n, m), + wv(2*m), wn(2*m, 2*m) c ************ c c Subroutine subsm c c Given xcp, l, u, r, an index set that specifies c the active set at xcp, and an l-BFGS matrix B c (in terms of WY, WS, SY, WT, head, col, and theta), c this subroutine computes an approximate solution c of the subspace problem c c (P) min Q(x) = r'(x-xcp) + 1/2 (x-xcp)' B (x-xcp) c c subject to l<=x<=u c x_i=xcp_i for all i in A(xcp) c c along the subspace unconstrained Newton direction c c d = -(Z'BZ)^(-1) r. c c The formula for the Newton direction, given the L-BFGS matrix c and the Sherman-Morrison formula, is c c d = (1/theta)r + (1/theta*2) Z'WK^(-1)W'Z r. c c where c K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c c Note that this procedure for computing d differs c from that described in [1]. One can show that the matrix K is c equal to the matrix M^[-1]N in that paper. c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c nsub is an integer variable. c On entry nsub is the number of free variables. c On exit nsub is unchanged. c c ind is an integer array of dimension nsub. c On entry ind specifies the coordinate indices of free variables. c On exit ind is unchanged. c c l is a double precision array of dimension n. c On entry l is the lower bound of x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound of x. c On exit u is unchanged. c c nbd is a integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, and c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c x is a double precision array of dimension n. c On entry x specifies the Cauchy point xcp. c On exit x(i) is the minimizer of Q over the subspace of c free variables. c c d is a double precision array of dimension n. c On entry d is the reduced gradient of Q at xcp. c On exit d is the Newton direction of Q. c c ws and wy are double precision arrays; c theta is a double precision variable; c col is an integer variable; c head is an integer variable. c On entry they store the information defining the c limited memory BFGS matrix: c ws(n,m) stores S, a set of s-vectors; c wy(n,m) stores Y, a set of y-vectors; c theta is the scaling factor specifying B_0 = theta I; c col is the number of variable metric corrections stored; c head is the location of the 1st s- (or y-) vector in S (or Y). c On exit they are unchanged. c c iword is an integer variable. c On entry iword is unspecified. c On exit iword specifies the status of the subspace solution. c iword = 0 if the solution is in the box, c 1 if some bound is encountered. c c wv is a double precision working array of dimension 2m. c c wn is a double precision array of dimension 2m x 2m. c On entry the upper triangle of wn stores the LEL^T factorization c of the indefinite matrix c c K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c where E = [-I 0] c [ 0 I] c On exit wn is unchanged. c c iprint is an INTEGER variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c info is an integer variable. c On entry info is unspecified. c On exit info = 0 for normal return, c = nonzero for abnormal return c when the matrix K is ill-conditioned. c c Subprograms called: c c Linpack dtrsl. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ logical temp1_updated integer pointr,m2,col2,ibd,jy,js,i,j,k double precision alpha,dk,temp1,temp2 double precision one,zero parameter (one=1.0d0,zero=0.0d0) if (nsub .le. 0) return if (iprint .ge. 99) write (6,1001) c Compute wv = W'Zd. pointr = head do 20 i = 1, col temp1 = zero temp2 = zero do 10 j = 1, nsub k = ind(j) temp1 = temp1 + wy(k,pointr)*d(j) temp2 = temp2 + ws(k,pointr)*d(j) 10 continue wv(i) = temp1 wv(col + i) = theta*temp2 pointr = mod(pointr,m) + 1 20 continue c Compute wv:=K^(-1)wv. m2 = 2*m col2 = 2*col call dtrsl(wn,m2,col2,wv,11,info) if (info .ne. 0) return do 25 i = 1, col wv(i) = -wv(i) 25 continue call dtrsl(wn,m2,col2,wv,01,info) if (info .ne. 0) return c Compute d = (1/theta)d + (1/theta**2)Z'W wv. pointr = head do 40 jy = 1, col js = col + jy do 30 i = 1, nsub k = ind(i) d(i) = d(i) + wy(k,pointr)*wv(jy)/theta + + ws(k,pointr)*wv(js) 30 continue pointr = mod(pointr,m) + 1 40 continue do 50 i = 1, nsub d(i) = d(i)/theta 50 continue c Backtrack to the feasible region. alpha = one temp1 = alpha do 60 i = 1, nsub k = ind(i) dk = d(i) if (nbd(k) .ne. 0) then temp1_updated = .false. if (dk .lt. zero .and. nbd(k) .le. 2) then temp2 = l(k) - x(k) if (temp2 .ge. zero) then temp1 = zero temp1_updated = .true. else if (dk*alpha .lt. temp2) then temp1 = temp2/dk temp1_updated = .true. endif else if (dk .gt. zero .and. nbd(k) .ge. 2) then temp2 = u(k) - x(k) if (temp2 .le. zero) then temp1 = zero temp1_updated = .true. else if (dk*alpha .gt. temp2) then temp1 = temp2/dk temp1_updated = .true. endif endif cc logical variable temp1_updated added to eliminate unexpected cc trigger of the if statement due to possible difference between cc hardware precision and double precision. if (temp1_updated .and. temp1 .lt. alpha) then alpha = temp1 ibd = i endif endif 60 continue if (alpha .lt. one) then dk = d(ibd) k = ind(ibd) if (dk .gt. zero) then x(k) = u(k) d(ibd) = zero else if (dk .lt. zero) then x(k) = l(k) d(ibd) = zero endif endif do 70 i = 1, nsub k = ind(i) x(k) = x(k) + alpha*d(i) 70 continue if (iprint .ge. 99) then if (alpha .lt. one) then write (6,1002) alpha else write (6,*) 'SM solution inside the box' end if if (iprint .gt.100) write (6,1003) (x(i),i=1,n) endif if (alpha .lt. one) then iword = 1 else iword = 0 endif if (iprint .ge. 99) write (6,1004) 1001 format (/,'----------------SUBSM entered-----------------',/) 1002 format ( 'ALPHA = ',f7.5,' backtrack to the BOX') 1003 format ('Subspace solution X = ',/,(4x,1p,6(1x,d11.4))) 1004 format (/,'----------------exit SUBSM --------------------',/) return end c====================== The end of subsm =============================== subroutine dcsrch(f,g,stp,ftol,gtol,xtol,stpmin,stpmax, + task,isave,dsave) character*(*) task integer isave(2) double precision f,g,stp,ftol,gtol,xtol,stpmin,stpmax double precision dsave(13) c ********** c c Subroutine dcsrch c c This subroutine finds a step that satisfies a sufficient c decrease condition and a curvature condition. c c Each call of the subroutine updates an interval with c endpoints stx and sty. The interval is initially chosen c so that it contains a minimizer of the modified function c c psi(stp) = f(stp) - f(0) - ftol*stp*f'(0). c c If psi(stp) <= 0 and f'(stp) >= 0 for some step, then the c interval is chosen so that it contains a minimizer of f. c c The algorithm is designed to find a step that satisfies c the sufficient decrease condition c c f(stp) <= f(0) + ftol*stp*f'(0), c c and the curvature condition c c abs(f'(stp)) <= gtol*abs(f'(0)). c c If ftol is less than gtol and if, for example, the function c is bounded below, then there is always a step which satisfies c both conditions. c c If no step can be found that satisfies both conditions, then c the algorithm stops with a warning. In this case stp only c satisfies the sufficient decrease condition. c c A typical invocation of dcsrch has the following outline: c c task = 'START' c 10 continue c call dcsrch( ... ) c if (task .eq. 'FG') then c Evaluate the function and the gradient at stp c goto 10 c end if c c NOTE: The user must no alter work arrays between calls. c c The subroutine statement is c c subroutine dcsrch(f,g,stp,ftol,gtol,xtol,stpmin,stpmax, c task,isave,dsave) c where c c f is a double precision variable. c On initial entry f is the value of the function at 0. c On subsequent entries f is the value of the c function at stp. c On exit f is the value of the function at stp. c c g is a double precision variable. c On initial entry g is the derivative of the function at 0. c On subsequent entries g is the derivative of the c function at stp. c On exit g is the derivative of the function at stp. c c stp is a double precision variable. c On entry stp is the current estimate of a satisfactory c step. On initial entry, a positive initial estimate c must be provided. c On exit stp is the current estimate of a satisfactory step c if task = 'FG'. If task = 'CONV' then stp satisfies c the sufficient decrease and curvature condition. c c ftol is a double precision variable. c On entry ftol specifies a nonnegative tolerance for the c sufficient decrease condition. c On exit ftol is unchanged. c c gtol is a double precision variable. c On entry gtol specifies a nonnegative tolerance for the c curvature condition. c On exit gtol is unchanged. c c xtol is a double precision variable. c On entry xtol specifies a nonnegative relative tolerance c for an acceptable step. The subroutine exits with a c warning if the relative difference between sty and stx c is less than xtol. c On exit xtol is unchanged. c c stpmin is a double precision variable. c On entry stpmin is a nonnegative lower bound for the step. c On exit stpmin is unchanged. c c stpmax is a double precision variable. c On entry stpmax is a nonnegative upper bound for the step. c On exit stpmax is unchanged. c c task is a character variable of length at least 60. c On initial entry task must be set to 'START'. c On exit task indicates the required action: c c If task(1:2) = 'FG' then evaluate the function and c derivative at stp and call dcsrch again. c c If task(1:4) = 'CONV' then the search is successful. c c If task(1:4) = 'WARN' then the subroutine is not able c to satisfy the convergence conditions. The exit value of c stp contains the best point found during the search. c c If task(1:5) = 'ERROR' then there is an error in the c input arguments. c c On exit with convergence, a warning or an error, the c variable task contains additional information. c c isave is an integer work array of dimension 2. c c dsave is a double precision work array of dimension 13. c c Subprograms called c c MINPACK-2 ... dcstep c c MINPACK-1 Project. June 1983. c Argonne National Laboratory. c Jorge J. More' and David J. Thuente. c c MINPACK-2 Project. October 1993. c Argonne National Laboratory and University of Minnesota. c Brett M. Averick, Richard G. Carter, and Jorge J. More'. c c ********** double precision zero,p5,p66 parameter(zero=0.0d0,p5=0.5d0,p66=0.66d0) double precision xtrapl,xtrapu parameter(xtrapl=1.1d0,xtrapu=4.0d0) logical brackt integer stage double precision finit,ftest,fm,fx,fxm,fy,fym,ginit,gtest, + gm,gx,gxm,gy,gym,stx,sty,stmin,stmax,width,width1 c Initialization block. if (task(1:5) .eq. 'START') then c Check the input arguments for errors. if (stp .lt. stpmin) task = 'ERROR: STP .LT. STPMIN' if (stp .gt. stpmax) task = 'ERROR: STP .GT. STPMAX' if (g .ge. zero) task = 'ERROR: INITIAL G .GE. ZERO' if (ftol .lt. zero) task = 'ERROR: FTOL .LT. ZERO' if (gtol .lt. zero) task = 'ERROR: GTOL .LT. ZERO' if (xtol .lt. zero) task = 'ERROR: XTOL .LT. ZERO' if (stpmin .lt. zero) task = 'ERROR: STPMIN .LT. ZERO' if (stpmax .lt. stpmin) task = 'ERROR: STPMAX .LT. STPMIN' c Exit if there are errors on input. if (task(1:5) .eq. 'ERROR') return c Initialize local variables. brackt = .false. stage = 1 finit = f ginit = g gtest = ftol*ginit width = stpmax - stpmin width1 = width/p5 c The variables stx, fx, gx contain the values of the step, c function, and derivative at the best step. c The variables sty, fy, gy contain the value of the step, c function, and derivative at sty. c The variables stp, f, g contain the values of the step, c function, and derivative at stp. stx = zero fx = finit gx = ginit sty = zero fy = finit gy = ginit stmin = zero stmax = stp + xtrapu*stp task = 'FG' goto 1000 else c Restore local variables. if (isave(1) .eq. 1) then brackt = .true. else brackt = .false. endif stage = isave(2) ginit = dsave(1) gtest = dsave(2) gx = dsave(3) gy = dsave(4) finit = dsave(5) fx = dsave(6) fy = dsave(7) stx = dsave(8) sty = dsave(9) stmin = dsave(10) stmax = dsave(11) width = dsave(12) width1 = dsave(13) endif c If psi(stp) <= 0 and f'(stp) >= 0 for some step, then the c algorithm enters the second stage. ftest = finit + stp*gtest if (stage .eq. 1 .and. f .le. ftest .and. g .ge. zero) + stage = 2 c Test for warnings. if (brackt .and. (stp .le. stmin .or. stp .ge. stmax)) + task = 'WARNING: ROUNDING ERRORS PREVENT PROGRESS' if (brackt .and. stmax - stmin .le. xtol*stmax) + task = 'WARNING: XTOL TEST SATISFIED' if (stp .eq. stpmax .and. f .le. ftest .and. g .le. gtest) + task = 'WARNING: STP = STPMAX' if (stp .eq. stpmin .and. (f .gt. ftest .or. g .ge. gtest)) + task = 'WARNING: STP = STPMIN' cc New warning statement added to eliminate the unexpected case cc of stp=stx due to possible difference between hardware precision cc and double precision. if (stp .eq. stx) + task = 'WARNING: STP = STX' c Test for convergence. if (f .le. ftest .and. abs(g) .le. gtol*(-ginit)) + task = 'CONVERGENCE' c Test for termination. if (task(1:4) .eq. 'WARN' .or. task(1:4) .eq. 'CONV') goto 1000 c A modified function is used to predict the step during the c first stage if a lower function value has been obtained but c the decrease is not sufficient. if (stage .eq. 1 .and. f .le. fx .and. f .gt. ftest) then c Define the modified function and derivative values. fm = f - stp*gtest fxm = fx - stx*gtest fym = fy - sty*gtest gm = g - gtest gxm = gx - gtest gym = gy - gtest c Call dcstep to update stx, sty, and to compute the new step. call dcstep(stx,fxm,gxm,sty,fym,gym,stp,fm,gm, + brackt,stmin,stmax) c Reset the function and derivative values for f. fx = fxm + stx*gtest fy = fym + sty*gtest gx = gxm + gtest gy = gym + gtest else c Call dcstep to update stx, sty, and to compute the new step. call dcstep(stx,fx,gx,sty,fy,gy,stp,f,g, + brackt,stmin,stmax) endif c Decide if a bisection step is needed. if (brackt) then if (abs(sty-stx) .ge. p66*width1) stp = stx + p5*(sty - stx) width1 = width width = abs(sty-stx) endif c Set the minimum and maximum steps allowed for stp. if (brackt) then stmin = min(stx,sty) stmax = max(stx,sty) else stmin = stp + xtrapl*(stp - stx) stmax = stp + xtrapu*(stp - stx) endif c Force the step to be within the bounds stpmax and stpmin. stp = max(stp,stpmin) stp = min(stp,stpmax) c If further progress is not possible, let stp be the best c point obtained during the search. if (brackt .and. (stp .le. stmin .or. stp .ge. stmax) + .or. (brackt .and. stmax-stmin .le. xtol*stmax)) stp = stx c Obtain another function and derivative. task = 'FG' 1000 continue c Save local variables. if (brackt) then isave(1) = 1 else isave(1) = 0 endif isave(2) = stage dsave(1) = ginit dsave(2) = gtest dsave(3) = gx dsave(4) = gy dsave(5) = finit dsave(6) = fx dsave(7) = fy dsave(8) = stx dsave(9) = sty dsave(10) = stmin dsave(11) = stmax dsave(12) = width dsave(13) = width1 end c====================== The end of dcsrch ============================== subroutine dcstep(stx,fx,dx,sty,fy,dy,stp,fp,dp,brackt, + stpmin,stpmax) logical brackt double precision stx,fx,dx,sty,fy,dy,stp,fp,dp,stpmin,stpmax c ********** c c Subroutine dcstep c c This subroutine computes a safeguarded step for a search c procedure and updates an interval that contains a step that c satisfies a sufficient decrease and a curvature condition. c c The parameter stx contains the step with the least function c value. If brackt is set to .true. then a minimizer has c been bracketed in an interval with endpoints stx and sty. c The parameter stp contains the current step. c The subroutine assumes that if brackt is set to .true. then c c min(stx,sty) < stp < max(stx,sty), c c and that the derivative at stx is negative in the direction c of the step. c c The subroutine statement is c c subroutine dcstep(stx,fx,dx,sty,fy,dy,stp,fp,dp,brackt, c stpmin,stpmax) c c where c c stx is a double precision variable. c On entry stx is the best step obtained so far and is an c endpoint of the interval that contains the minimizer. c On exit stx is the updated best step. c c fx is a double precision variable. c On entry fx is the function at stx. c On exit fx is the function at stx. c c dx is a double precision variable. c On entry dx is the derivative of the function at c stx. The derivative must be negative in the direction of c the step, that is, dx and stp - stx must have opposite c signs. c On exit dx is the derivative of the function at stx. c c sty is a double precision variable. c On entry sty is the second endpoint of the interval that c contains the minimizer. c On exit sty is the updated endpoint of the interval that c contains the minimizer. c c fy is a double precision variable. c On entry fy is the function at sty. c On exit fy is the function at sty. c c dy is a double precision variable. c On entry dy is the derivative of the function at sty. c On exit dy is the derivative of the function at the exit sty. c c stp is a double precision variable. c On entry stp is the current step. If brackt is set to .true. c then on input stp must be between stx and sty. c On exit stp is a new trial step. c c fp is a double precision variable. c On entry fp is the function at stp c On exit fp is unchanged. c c dp is a double precision variable. c On entry dp is the the derivative of the function at stp. c On exit dp is unchanged. c c brackt is an logical variable. c On entry brackt specifies if a minimizer has been bracketed. c Initially brackt must be set to .false. c On exit brackt specifies if a minimizer has been bracketed. c When a minimizer is bracketed brackt is set to .true. c c stpmin is a double precision variable. c On entry stpmin is a lower bound for the step. c On exit stpmin is unchanged. c c stpmax is a double precision variable. c On entry stpmax is an upper bound for the step. c On exit stpmax is unchanged. c c MINPACK-1 Project. June 1983 c Argonne National Laboratory. c Jorge J. More' and David J. Thuente. c c MINPACK-2 Project. October 1993. c Argonne National Laboratory and University of Minnesota. c Brett M. Averick and Jorge J. More'. c c ********** double precision zero,p66,two,three parameter(zero=0.0d0,p66=0.66d0,two=2.0d0,three=3.0d0) double precision gamma,p,q,r,s,sgnd,stpc,stpf,stpq,theta sgnd = dp*(dx/abs(dx)) c First case: A higher function value. The minimum is bracketed. c If the cubic step is closer to stx than the quadratic step, the c cubic step is taken, otherwise the average of the cubic and c quadratic steps is taken. if (fp .gt. fx) then theta = three*(fx - fp)/(stp - stx) + dx + dp s = max(abs(theta),abs(dx),abs(dp)) gamma = s*sqrt((theta/s)**2 - (dx/s)*(dp/s)) if (stp .lt. stx) gamma = -gamma p = (gamma - dx) + theta q = ((gamma - dx) + gamma) + dp r = p/q stpc = stx + r*(stp - stx) stpq = stx + ((dx/((fx - fp)/(stp - stx) + dx))/two)* + (stp - stx) if (abs(stpc-stx) .lt. abs(stpq-stx)) then stpf = stpc else stpf = stpc + (stpq - stpc)/two endif brackt = .true. c Second case: A lower function value and derivatives of opposite c sign. The minimum is bracketed. If the cubic step is farther from c stp than the secant step, the cubic step is taken, otherwise the c secant step is taken. else if (sgnd .lt. zero) then theta = three*(fx - fp)/(stp - stx) + dx + dp s = max(abs(theta),abs(dx),abs(dp)) gamma = s*sqrt((theta/s)**2 - (dx/s)*(dp/s)) if (stp .gt. stx) gamma = -gamma p = (gamma - dp) + theta q = ((gamma - dp) + gamma) + dx r = p/q stpc = stp + r*(stx - stp) stpq = stp + (dp/(dp - dx))*(stx - stp) if (abs(stpc-stp) .gt. abs(stpq-stp)) then stpf = stpc else stpf = stpq endif brackt = .true. c Third case: A lower function value, derivatives of the same sign, c and the magnitude of the derivative decreases. else if (abs(dp) .lt. abs(dx)) then c The cubic step is computed only if the cubic tends to infinity c in the direction of the step or if the minimum of the cubic c is beyond stp. Otherwise the cubic step is defined to be the c secant step. theta = three*(fx - fp)/(stp - stx) + dx + dp s = max(abs(theta),abs(dx),abs(dp)) c The case gamma = 0 only arises if the cubic does not tend c to infinity in the direction of the step. gamma = s*sqrt(max(zero,(theta/s)**2-(dx/s)*(dp/s))) if (stp .gt. stx) gamma = -gamma p = (gamma - dp) + theta q = (gamma + (dx - dp)) + gamma r = p/q if (r .lt. zero .and. gamma .ne. zero) then stpc = stp + r*(stx - stp) else if (stp .gt. stx) then stpc = stpmax else stpc = stpmin endif stpq = stp + (dp/(dp - dx))*(stx - stp) if (brackt) then c A minimizer has been bracketed. If the cubic step is c closer to stp than the secant step, the cubic step is c taken, otherwise the secant step is taken. if (abs(stpc-stp) .lt. abs(stpq-stp)) then stpf = stpc else stpf = stpq endif if (stp .gt. stx) then stpf = min(stp+p66*(sty-stp),stpf) else stpf = max(stp+p66*(sty-stp),stpf) endif else c A minimizer has not been bracketed. If the cubic step is c farther from stp than the secant step, the cubic step is c taken, otherwise the secant step is taken. if (abs(stpc-stp) .gt. abs(stpq-stp)) then stpf = stpc else stpf = stpq endif stpf = min(stpmax,stpf) stpf = max(stpmin,stpf) endif c Fourth case: A lower function value, derivatives of the c same sign, and the magnitude of the derivative does not c decrease. If the minimum is not bracketed, the step is either c stpmin or stpmax, otherwise the cubic step is taken. else if (brackt) then theta = three*(fp - fy)/(sty - stp) + dy + dp s = max(abs(theta),abs(dy),abs(dp)) gamma = s*sqrt((theta/s)**2 - (dy/s)*(dp/s)) if (stp .gt. sty) gamma = -gamma p = (gamma - dp) + theta q = ((gamma - dp) + gamma) + dy r = p/q stpc = stp + r*(sty - stp) stpf = stpc else if (stp .gt. stx) then stpf = stpmax else stpf = stpmin endif endif c Update the interval which contains a minimizer. if (fp .gt. fx) then sty = stp fy = fp dy = dp else if (sgnd .lt. zero) then sty = stx fy = fx dy = dx endif stx = stp fx = fp dx = dp endif c Compute the new step. stp = stpf end c====================== The end of dcstep ============================== subroutine timer(ttime) double precision ttime c ********* c c Subroutine timer c c This subroutine is used to determine user time. In a typical c application, the user time for a code segment requires calls c to subroutine timer to determine the initial and final time. c c The subroutine statement is c c subroutine timer(ttime) c c where c c ttime is an output variable which specifies the user time. c c Argonne National Laboratory and University of Minnesota. c MINPACK-2 Project. c c Modified October 1990 by Brett M. Averick. c c ********** real temp real tarray(2) real etime c The first element of the array tarray specifies user time temp = etime(tarray) ttime = dble(tarray(1)) return end c====================== The end of timer =============================== double precision function dnrm2(n,x,incx) integer n,incx double precision x(n) c ********** c c Function dnrm2 c c Given a vector x of length n, this function calculates the c Euclidean norm of x with stride incx. c c The function statement is c c double precision function dnrm2(n,x,incx) c c where c c n is a positive integer input variable. c c x is an input array of length n. c c incx is a positive integer variable that specifies the c stride of the vector. c c Subprograms called c c FORTRAN-supplied ... abs, max, sqrt c c MINPACK-2 Project. February 1991. c Argonne National Laboratory. c Brett M. Averick. c c ********** integer i double precision scale dnrm2 = 0.0d0 scale = 0.0d0 do 10 i = 1, n, incx scale = max(scale, abs(x(i))) 10 continue if (scale .eq. 0.0d0) return do 20 i = 1, n, incx dnrm2 = dnrm2 + (x(i)/scale)**2 20 continue dnrm2 = scale*sqrt(dnrm2) return end c====================== The end of dnrm2 =============================== double precision function dpmeps() c ********** c c Subroutine dpeps c c This subroutine computes the machine precision parameter c dpmeps as the smallest floating point number such that c 1 + dpmeps differs from 1. c c This subroutine is based on the subroutine machar described in c c W. J. Cody, c MACHAR: A subroutine to dynamically determine machine parameters, c ACM Trans. Math. Soft., 14, 1988, pages 303-311. c c The subroutine statement is: c c subroutine dpeps(dpmeps) c c where c c dpmeps is a double precision variable. c On entry dpmeps need not be specified. c On exit dpmeps is the machine precision. c c MINPACK-2 Project. February 1991. c Argonne National Laboratory and University of Minnesota. c Brett M. Averick. c c ******* integer i,ibeta,irnd,it,itemp,negep double precision a,b,beta,betain,betah,temp,tempa,temp1, + zero,one,two data zero,one,two /0.0d0,1.0d0,2.0d0/ c determine ibeta, beta ala malcolm. a = one b = one 10 continue a = a + a temp = a + one temp1 = temp - a if (temp1 - one .eq. zero) go to 10 20 continue b = b + b temp = a + b itemp = int(temp - a) if (itemp .eq. 0) go to 20 ibeta = itemp beta = dble(ibeta) c determine it, irnd. it = 0 b = one 30 continue it = it + 1 b = b * beta temp = b + one temp1 = temp - b if (temp1 - one .eq. zero) go to 30 irnd = 0 betah = beta/two temp = a + betah if (temp - a .ne. zero) irnd = 1 tempa = a + beta temp = tempa + betah if ((irnd .eq. 0) .and. (temp - tempa .ne. zero)) irnd = 2 c determine dpmeps. negep = it + 3 betain = one/beta a = one do 40 i = 1, negep a = a*betain 40 continue 50 continue temp = one + a if (temp - one .ne. zero) go to 60 a = a*beta go to 50 60 continue dpmeps = a if ((ibeta .eq. 2) .or. (irnd .eq. 0)) go to 70 a = (a*(one + a))/two temp = one + a if (temp - one .ne. zero) dpmeps = a 70 return end c====================== The end of dpmeps ============================== subroutine daxpy(n,da,dx,incx,dy,incy) c c constant times a vector plus a vector. c uses unrolled loops for increments equal to one. c jack dongarra, linpack, 3/11/78. c double precision dx(*),dy(*),da integer i,incx,incy,ix,iy,m,mp1,n c if(n.le.0)return if (da .eq. 0.0d0) return if(incx.eq.1.and.incy.eq.1)go to 20 c c code for unequal increments or equal increments c not equal to 1 c ix = 1 iy = 1 if(incx.lt.0)ix = (-n+1)*incx + 1 if(incy.lt.0)iy = (-n+1)*incy + 1 do 10 i = 1,n dy(iy) = dy(iy) + da*dx(ix) ix = ix + incx iy = iy + incy 10 continue return c c code for both increments equal to 1 c c c clean-up loop c 20 m = mod(n,4) if( m .eq. 0 ) go to 40 do 30 i = 1,m dy(i) = dy(i) + da*dx(i) 30 continue if( n .lt. 4 ) return 40 mp1 = m + 1 do 50 i = mp1,n,4 dy(i) = dy(i) + da*dx(i) dy(i + 1) = dy(i + 1) + da*dx(i + 1) dy(i + 2) = dy(i + 2) + da*dx(i + 2) dy(i + 3) = dy(i + 3) + da*dx(i + 3) 50 continue return end c====================== The end of daxpy =============================== subroutine dcopy(n,dx,incx,dy,incy) c c copies a vector, x, to a vector, y. c uses unrolled loops for increments equal to one. c jack dongarra, linpack, 3/11/78. c double precision dx(*),dy(*) integer i,incx,incy,ix,iy,m,mp1,n c if(n.le.0)return if(incx.eq.1.and.incy.eq.1)go to 20 c c code for unequal increments or equal increments c not equal to 1 c ix = 1 iy = 1 if(incx.lt.0)ix = (-n+1)*incx + 1 if(incy.lt.0)iy = (-n+1)*incy + 1 do 10 i = 1,n dy(iy) = dx(ix) ix = ix + incx iy = iy + incy 10 continue return c c code for both increments equal to 1 c c c clean-up loop c 20 m = mod(n,7) if( m .eq. 0 ) go to 40 do 30 i = 1,m dy(i) = dx(i) 30 continue if( n .lt. 7 ) return 40 mp1 = m + 1 do 50 i = mp1,n,7 dy(i) = dx(i) dy(i + 1) = dx(i + 1) dy(i + 2) = dx(i + 2) dy(i + 3) = dx(i + 3) dy(i + 4) = dx(i + 4) dy(i + 5) = dx(i + 5) dy(i + 6) = dx(i + 6) 50 continue return end c====================== The end of dcopy =============================== double precision function ddot(n,dx,incx,dy,incy) c c forms the dot product of two vectors. c uses unrolled loops for increments equal to one. c jack dongarra, linpack, 3/11/78. c double precision dx(*),dy(*),dtemp integer i,incx,incy,ix,iy,m,mp1,n c ddot = 0.0d0 dtemp = 0.0d0 if(n.le.0)return if(incx.eq.1.and.incy.eq.1)go to 20 c c code for unequal increments or equal increments c not equal to 1 c ix = 1 iy = 1 if(incx.lt.0)ix = (-n+1)*incx + 1 if(incy.lt.0)iy = (-n+1)*incy + 1 do 10 i = 1,n dtemp = dtemp + dx(ix)*dy(iy) ix = ix + incx iy = iy + incy 10 continue ddot = dtemp return c c code for both increments equal to 1 c c c clean-up loop c 20 m = mod(n,5) if( m .eq. 0 ) go to 40 do 30 i = 1,m dtemp = dtemp + dx(i)*dy(i) 30 continue if( n .lt. 5 ) go to 60 40 mp1 = m + 1 do 50 i = mp1,n,5 dtemp = dtemp + dx(i)*dy(i) + dx(i + 1)*dy(i + 1) + * dx(i + 2)*dy(i + 2) + dx(i + 3)*dy(i + 3) + dx(i + 4)*dy(i + 4) 50 continue 60 ddot = dtemp return end c====================== The end of ddot ================================ subroutine dpofa(a,lda,n,info) integer lda,n,info double precision a(lda,*) c c dpofa factors a double precision symmetric positive definite c matrix. c c dpofa is usually called by dpoco, but it can be called c directly with a saving in time if rcond is not needed. c (time for dpoco) = (1 + 18/n)*(time for dpofa) . c c on entry c c a double precision(lda, n) c the symmetric matrix to be factored. only the c diagonal and upper triangle are used. c c lda integer c the leading dimension of the array a . c c n integer c the order of the matrix a . c c on return c c a an upper triangular matrix r so that a = trans(r)*r c where trans(r) is the transpose. c the strict lower triangle is unaltered. c if info .ne. 0 , the factorization is not complete. c c info integer c = 0 for normal return. c = k signals an error condition. the leading minor c of order k is not positive definite. c c linpack. this version dated 08/14/78 . c cleve moler, university of new mexico, argonne national lab. c c subroutines and functions c c blas ddot c fortran sqrt c c internal variables c double precision ddot,t double precision s integer j,jm1,k c begin block with ...exits to 40 c c do 30 j = 1, n info = j s = 0.0d0 jm1 = j - 1 if (jm1 .lt. 1) go to 20 do 10 k = 1, jm1 t = a(k,j) - ddot(k-1,a(1,k),1,a(1,j),1) t = t/a(k,k) a(k,j) = t s = s + t*t 10 continue 20 continue s = a(j,j) - s c ......exit if (s .le. 0.0d0) go to 40 a(j,j) = sqrt(s) 30 continue info = 0 40 continue return end c====================== The end of dpofa =============================== subroutine dscal(n,da,dx,incx) c c scales a vector by a constant. c uses unrolled loops for increment equal to one. c jack dongarra, linpack, 3/11/78. c modified 3/93 to return if incx .le. 0. c double precision da,dx(*) integer i,incx,m,mp1,n,nincx c if( n.le.0 .or. incx.le.0 )return if(incx.eq.1)go to 20 c c code for increment not equal to 1 c nincx = n*incx do 10 i = 1,nincx,incx dx(i) = da*dx(i) 10 continue return c c code for increment equal to 1 c c c clean-up loop c 20 m = mod(n,5) if( m .eq. 0 ) go to 40 do 30 i = 1,m dx(i) = da*dx(i) 30 continue if( n .lt. 5 ) return 40 mp1 = m + 1 do 50 i = mp1,n,5 dx(i) = da*dx(i) dx(i + 1) = da*dx(i + 1) dx(i + 2) = da*dx(i + 2) dx(i + 3) = da*dx(i + 3) dx(i + 4) = da*dx(i + 4) 50 continue return end c====================== The end of dscal =============================== subroutine dtrsl(t,ldt,n,b,job,info) integer ldt,n,job,info double precision t(ldt,*),b(*) c c c dtrsl solves systems of the form c c t * x = b c or c trans(t) * x = b c c where t is a triangular matrix of order n. here trans(t) c denotes the transpose of the matrix t. c c on entry c c t double precision(ldt,n) c t contains the matrix of the system. the zero c elements of the matrix are not referenced, and c the corresponding elements of the array can be c used to store other information. c c ldt integer c ldt is the leading dimension of the array t. c c n integer c n is the order of the system. c c b double precision(n). c b contains the right hand side of the system. c c job integer c job specifies what kind of system is to be solved. c if job is c c 00 solve t*x=b, t lower triangular, c 01 solve t*x=b, t upper triangular, c 10 solve trans(t)*x=b, t lower triangular, c 11 solve trans(t)*x=b, t upper triangular. c c on return c c b b contains the solution, if info .eq. 0. c otherwise b is unaltered. c c info integer c info contains zero if the system is nonsingular. c otherwise info contains the index of c the first zero diagonal element of t. c c linpack. this version dated 08/14/78 . c g. w. stewart, university of maryland, argonne national lab. c c subroutines and functions c c blas daxpy,ddot c fortran mod c c internal variables c double precision ddot,temp integer case,j,jj c c begin block permitting ...exits to 150 c c check for zero diagonal elements. c do 10 info = 1, n c ......exit if (t(info,info) .eq. 0.0d0) go to 150 10 continue info = 0 c c determine the task and go to it. c case = 1 if (mod(job,10) .ne. 0) case = 2 if (mod(job,100)/10 .ne. 0) case = case + 2 go to (20,50,80,110), case c c solve t*x=b for t lower triangular c 20 continue b(1) = b(1)/t(1,1) if (n .lt. 2) go to 40 do 30 j = 2, n temp = -b(j-1) call daxpy(n-j+1,temp,t(j,j-1),1,b(j),1) b(j) = b(j)/t(j,j) 30 continue 40 continue go to 140 c c solve t*x=b for t upper triangular. c 50 continue b(n) = b(n)/t(n,n) if (n .lt. 2) go to 70 do 60 jj = 2, n j = n - jj + 1 temp = -b(j+1) call daxpy(j,temp,t(1,j+1),1,b(1),1) b(j) = b(j)/t(j,j) 60 continue 70 continue go to 140 c c solve trans(t)*x=b for t lower triangular. c 80 continue b(n) = b(n)/t(n,n) if (n .lt. 2) go to 100 do 90 jj = 2, n j = n - jj + 1 b(j) = b(j) - ddot(jj-1,t(j+1,j),1,b(j+1),1) b(j) = b(j)/t(j,j) 90 continue 100 continue go to 140 c c solve trans(t)*x=b for t upper triangular. c 110 continue b(1) = b(1)/t(1,1) if (n .lt. 2) go to 130 do 120 j = 2, n b(j) = b(j) - ddot(j-1,t(1,j),1,b(1),1) b(j) = b(j)/t(j,j) 120 continue 130 continue 140 continue 150 continue return end c====================== The end of dtrsl =============================== pot/potExpPower.m0000644000175000017500000000101611420141474012122 0ustar hnhn% P = potExpPower(s,alpha) - Exponential power family potential % % pott(s) = exp(-|s|^alpha) with shape parameter alpha>0 % % See also POTFUNCTIONS.M. % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 14 function P = potExpPower(s,alpha) P = zeros(numel(s),4); P(:,1) = -abs(s).^ alpha; % log potential P(:,2) = -alpha*sign(s) .*abs(s).^(alpha-1); % 1st derivative of log potential P(:,3) = -alpha*(alpha-1)*abs(s).^(alpha-2); % 2nd derivative of log potentialpot/potGauss.m0000644000175000017500000000100411420141474011430 0ustar hnhn% P = potGauss(s) - Gaussian potential % % pot(s) = exp(-s²/2) % % See also POTFUNCTIONS.M. % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 14 function P = potGauss(s) P = zeros(numel(s),4); % 2nd derivative and b are equal to zero P(:,1) = -s(:).*s(:)/2; % log potential P(:,2) = -s(:); % 1st derivative of log potential P(:,3) = -1; % 2nd derivative of log potentialpot/potLaplace.m0000644000175000017500000000071111420141474011713 0ustar hnhn% P = potLaplace(s) - Laplace (double exponential) potential % % pot(s) = exp(-|s|) % % See also POTFUNCTIONS.M. % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 14 function P = potLaplace(s) P = zeros(numel(s),4); % 2nd derivative and b are equal to zero P(:,1) = -abs(s(:)); % log potential P(:,2) = -sign(s(:)); % 1st derivative of log potentialpot/potLogistic.m0000644000175000017500000000107611423326701012135 0ustar hnhn% P = potLogistic(s) - Logistic potential for classification % % pot(s) = 1/(1+exp(-s)) % % See also POTFUNCTIONS.M. % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 26 function P = potLogistic(s) P = zeros(numel(s),4); % log potential, -log(1+exp(-s)) P(:,1) = min(0,s) - log(1+exp(-abs(s))); % safe since abs(s)>=0 p = exp(P(:,1)); P(:,2) = 1-p; % 1st derivative of log potential, exp(-s)/(1+exp(-s)) P(:,3) = p.*(p-1); % 2nd derivative of log potential, -exp(-s)/(1+exp(-s))^2 P(:,4) = 1/2;pot/potSech2.m0000644000175000017500000000111111422527037011316 0ustar hnhn% P = potSech2(s) - Sech-squared potential (Logistic distribution) % % pot(s) = 1 / cosh(s)² % % See also POTFUNCTIONS.M. % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 21 function P = potSech2(s) P = zeros(numel(s),4); a = exp(-2*abs(s)); % always between 0 and 1 and therefore safe to evaluate P(:,1) = -2*(abs(s) + log(1+a) - log(2)); % log potential, -2*log( cosh(s) ) b = 2*a./(1+a); P(:,2) = -2*sign(s).*(1-b); % 1st derivative of log potential P(:,3) = -4*b./(1+a); % 2nd derivative of log potentialpot/potT.m0000644000175000017500000000101511420141474010553 0ustar hnhn% P = potT(s,nu) - Student's t potential % % pot(s) = ( 1 + s²/nu )^(-nu/2-1/2) with nu>0 degrees of freedom % % See also POTFUNCTIONS.M. % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 14 function P = potT(s,nu) P = zeros(numel(s),4); ss = s.*s; P(:,1) = - (nu+1)*log( 1+ss./nu )/2; % log potential a = ss+nu; P(:,2) = - (nu+1)*s./a; % 1st derivative of log potential P(:,3) = (nu+1)*(ss-nu)./(a.*a); % 2nd derivative of log potentialLICENCE0000644000175000017500000000274011415720432007641 0ustar hnhnCopyright 2010 Hannes Nickisch. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY HANNES NICKISCH ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL HANNES NICKISCH OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. The views and conclusions contained in the software and documentation are those of the authors and should not be interpreted as representing official policies, either expressed or implied, of Hannes Nickisch. penFunctions.m0000644000175000017500000000170211430004775011504 0ustar hnhn% A penalty function pen(s) is a scalar function IR -> IR. % % [p,dp,d2p] = pen(s,psi) where psi contains additional parameters % % The return arguments are of the same size as s and have the following meaning: % p = pen(s), % dp = d pen(s) / ds, and % d2p = d^2 pen(s) / ds^2. % % Currently, we have implemented in pen/pen.m % Logarithmic: penLog(s,nu) = log(s^2+nu), % Power: penPow(s,d) = abs(s)^d, % Quadratic: penQuad(s) = s^2/2, % Zero: penZero(s) = 0, and % Derived from potential: penFromPot(s,pot,tau,z) = ... % tau*b*(r-s) -log( pot(tau*r) ), r=sign(s)*sqrt(s^2+z) % % Examples: % >> s=0.3; d=1.5; penPow(s,d) % >> s=-4; pot='potLaplace'; tau=2; z=1.3; penFromPot(s,pot,tau,z) % % See also POTFUNCTIONS.M. % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 August 09plsAlgorithms.m0000644000175000017500000000370011423621141011653 0ustar hnhn% A PLS (penalised least squares) algorithm is a program solving the % minimisation problem % phi(u) = 1/(2*lambda) * ||X*u-y||_2^2 + sum( pen(s) ), s=B*u, % where pen(s) is a penalty function. % % [u,phi] = pls(u0,X,y,B,opt,lam,pen,varargin) % % INPUT % ===== % u0 [n,1] initial vector % X [m,n] matrix or operator % y [m,1] vector % B [q,n] matrix or operator % opt. optimisation parameters % nMVM maximal number of steps = matrix vector multiplications ... % with A=X'*X+B'*D*B, D is diagonal, [default 100] % output flag saying whether something is printed [default false] % the function will show the current iteration number, the actual % function value phi and the current length of the step % in the format: 13, phi=8.6063e-01; du=2.325e-04 % lam [1x1] (positive) weight of the penaliser, can be Inf % pen penaliser function handle or function name % [p,dp,d2p] = feval(pen,Bu,varargin{:}) % varargin additional parameters for pen % % OUTPUT % ====== % u [n,1] optimal solution % phi [1,1] optimal function value % % Currently, we have implemented in pls/pls.m % LBFGS: LIMITED memory BROYDEN-FLETCHER-GOLDFARB-SHANNO % quasi Newton or variable metric method % TN: TRUNCATED NEWTON % optimisation with CG approximated Newton steps % CGBT: CONJUGATE GRADIENTS with BACKTRACKING line search % optimisation using the Armijo rule % CG: CONJUGATE GRADIENTS % optimisation with CG code minimize.m by Carl E. Rasmussen % % Examples: % >> lam=1; plsLBFGS(u,X,y,B,opt,lam,'penQuad') % >> lam=1; tau=2; z=1.3; plsCG(u,X,y,B,opt,lam,'penFromPot','potLaplace',tau,z) % % See also PENFUNCTIONS.M, POTFUNCTIONS.M. % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 27potFunctions.m0000644000175000017500000000243011423621141011515 0ustar hnhn% A potential pot(s) is a scalar positive and strongly super Gaussian % function IR -> IR_+ that can be symmetrised i.e. there is a number b % such that f(s) = pot(s)*exp(-b*s) is even. % % P = pot(s,theta) where theta contains additional parameters % % The return argument is a matrix of size [px4] where the columns do have the % following meaning: % lp = P(:,1), dlp = P(:,2), d2lp = P(:,3), b = P(:,4), where % % (i) lp(s) = log( pot(s) ), % (ii) dlp(s) = d lp(s) / ds, % (iii) d2lp(s) = d^2 lp(s) / ds^2, and % (iv) f(s) = pot(s)*exp(-b*s) is symmetric, i.e. f(s) = f(-s). % % We can split the matrix P by the command % [lp,dlp,d2lp,b] = col(P). % % Currently, we have implemented in pot/pot.m % Gaussian: potGauss(s) = exp(-s^2/2), % Sech-squared: potSech2(s) = 1 / cosh(s)^2, % Laplace: potLaplace(s) = exp(-abs(s)), % Student's t, nu>0: potT(s,nu) = (1 + s^2/nu)^(-nu/2-1/2), % Exponential power, al>0: potExpPower(s,a) = exp(-|s|^al), and % Logistic function: potLogistic(s) = 1/(1+exp(-s)). % % Examples: % >> s=0.3; potLogistic(s) % >> s=2.7; nu=3; potT(s,nu) % % See also PENFUNCTIONS.M. % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 27 startup.m0000644000175000017500000000152311420141475010532 0ustar hnhn% startup script to make Octave/Matlab aware of the glm-ie package % % (c) by Hannes Nickisch, MPI for Biological Cybernetics, 2010 July 16 disp(['executing glm-ie startup script...']); OCT = exist('OCTAVE_VERSION') ~= 0; % check if we run Matlab or Octave me = mfilename; % what is my filename mydir = which(me); mydir = mydir(1:end-2-numel(me)); % where am I located if OCT && numel(mydir)==2 if strcmp(mydir,'./'), mydir = [pwd,mydir(2:end)]; end end % OCTAVE 3.0.x relative, MATLAB and newer have absolute path addpath(mydir(1:end-1)) addpath([mydir,'inf']) addpath([mydir,'mat']) if OCT, addpath([mydir,'mat/@wavmat/private']), end addpath([mydir,'mat']) addpath([mydir,'pen']) addpath([mydir,'pls']) addpath([mydir,'pot']) addpath([mydir,'demo']) clear me mydirglm-ie.pdf0000644000175000017500000057141211430167323010531 0ustar hnhn%PDF-1.3 %쏢 29 0 obj <> stream x\rM_Oݔwo$J,ǎr{p@-f{If^W{WGEI?(iMm61ZFR ?8PޚXhTV !PI!:#,/CyOiN)Q_/y[zP^уqv~ufnlzH o0~Xp2`ˬ\T:RgT6!gN108u_th6RJ@YeuEE'1\2.<0RWcmnt`b"Q`'yUru~Z*V62If6w I~K΂sZ2ehJ E2h'a( qu(|2@v+~Ip&uЯ a7 ~YKaI0J80w"k*;@+re#[I˜Nd=8'S?bwPʨʄ *z: Ulj*\- 0*BXH ^*MEi6q`m1Z]NZA:igL/t(/@:!PE0W!%A*GbiYE멓M0$TXbXYͲj99R#M-*E_fM4iëXyJ !l4ГtߨAՊR'aI#(Ti?_.ƎKצ8 Гkam$FB#DHRGV<%0Z[ (~fl aI|6(ЁE`LLd0o}֧u}gM'%QR슕U.yJd|{WzlZ_g%q5AmMEuc) {B12٪5U!yT8tUbE06ݛ1v(t;u*ll] 46{"r1QB8~KO_KZy^4f=o^r>=AM@\Q5î;gw7Py ؽ=ojg[Ik)w>bIOzk`YMzYEfWcP+ őx5&à-:2@<$: -sbX&0ng^ֱoɗQO ,HQ˜#-'˖:z2QA< *?0o}go=U`-- Ԕ_ͳuzTn~`}o:gy>׎R CQj#ٗYͩ=Q\+[n8׭%r;G kQv̿zK2DܵΓ${qg;O"Z>H` zjr>O'EsrHX?5sP' d|) Mu&xnΥw+g^=rk\ 7&לF5H2 Ltkgé҅a&0A]88g夿׳Ɯ_yrIT@ʬ"]hǸ3m̖(=kqw8[PRm-$nS/<K{hwT{SZ=f'o]Xi>~r4cE mrڌ k--dXMrcζjXa4q zOhZq MLh ͝)< KhܬN8a_{Lx) P^`R8;cmqw(Ŏx-OyJw7O,w$E\]^$;4ZjT%YA]Mqz8c@nMoV}ݛwW蚔{yx;-kg}k{5'(x5q(wS/㭼 HC|WM 'B :QEK}OVW\1``J[ 4{OF _}f&4-eLμz ߳W{?z> stream x]͒$ErfW>Jt+.ڭ$òhEHr0 ' Qt{Dd{ddVvO7dAS῟{x}u<'AW(܃Fqp Yb2ƃf ͓Sǻ'3Im9>^Zs$'gS7OO烴N'NʨI{>x ߓ܇'j^x =>I;}4oQüeVKo/Oz2Ʒe=k3}QWѽ8EQjrAý^pZT``Dޘ<3HE!'x-P>~rz狷Fo$=ST01TxɷB"|eڈ>#Yd >Z9xK9N& !bOIʂ.NxemEb*;jC'ɐI)::L7#-[v3 mODXl$2ӇڑO&JBȴ*jkv/הw`{5h$]dge&&UM ^Ujo2a`Qq@g> Hʏ+R

i퀧}tpbChýu0u?_oǭ_I~DJ5:c/f:0LAk>*9wufB||0N׃901Կ 4s~D*J_vT=i!Q-#rCt 3#!>U' k%M .$Ho+&SR޼rF&jL*U)z'%8~_adT#pdlȂ=[Yi~jk 5utL6N@9a=_Zx%qǮfVj<:(/=oQ6D%+Y5DwڼrP "*%vh&afJ6Rhj| J ;Ѩb ,8H0A9TM _F,xr8Ȝ<1^ (l-qh:{,nK.RLVSH ĵZhW!z75x3Ic?I:ߦQ ͪ{]uS^>CbM0Lh^޼H= ? `sC$@"qʿ88 'cS !XEwXleeyF9ƞA׀CvI"P8SiCO7y z3ppwRF07!&y 2`Xu ߢ}/5wQ3ϫ 3]2ŝ?< D4akHl]Q0en)Z9zzD g \ldbS]k 9rU]hT8>13́s~W egS T&}C(ӝmzQ>+`)XMRKB!~LVNU%BZTDKg9* \䜟l4wP-{+@H)dU g0.]i䴯!&Jves |]Th2P]z;`dy7rE-rYY*-ZȎBpyID8G\Pb,.ٓ[鉋L@FJDuw6maF 3D46㇛pDDx J )ЌHY/%F^NM P|tċn+7)?:J"foʋJ"H/eu/qݼZ̓wU U[1ܩ}جxS7.'}7]Nuյd.((-1^?vLY+:ƴ51|3sbӘ/ѱp {۴4:,ɾ& `O_#YB_@tBQ, ]%.'Pm7bV/^a W8Y omʨ GHkNcyRUSšV U!p.\n7ڄ }XgrsyJ#I+Lꢚ#z&a0RKW@+R}Fуߗ2#Js{!DƉQxGKxk7 X3Fw v0Su2NnY'e=t_OrҴ$M" Zh匞W%%uePv% f3ȝ5cmp^Q)R(68~͒;Z7D ?@ %ƆkWBJpąS,AyzY3+}#>_HkA$+ΣOVx'n7MP‘)<[,@ig0Dㇷ6`KvX?%;RYcyE2B)ss"T[(j\zI^Q~ g8q~.cE`,)^ Թt60i뗈[ۼ=դ}v́WL$v#\3bĶe嶌wˌ_(3u6[u٧vu)E.$<Ob@KF6h'Ǩ_\dyɊ!F^k|^9g>3;ra\+Zk꯴$ 6 0ݬczOv崛Z7催"KlKݫSѵ03us\f@UL,e.HnYíH:v_pAٷڕ,[ZM2L,4!\+y,{ [̥<}S!5Z5`iO7KE-A>deίބEImr/p.\>o=NS{ cgpyCKʻצ ;7j/U8*5ta>ޑJgw< w~+! ?W}\ԛD3/]JV4(nQ*Tct憠*U%n TrS:=uV$2 J`g ̨_WY*[~@̀43] ʐmA:RӴU0+ȕ =b6v]K9Tԇ[9tlm=V^HU [Fc-LugVr5r]JsY^ ];_9J;/rk >KaVӖ}е3B~iR Zd\W%.OIa`_-8 ,FʵݵV9754ֻl'7]cG$EbfEț"3;{otCWVƭJ3B1=+R {^@T"5mDҁQV7,9 ekb )g&*olAvy;-j9@BIۙ9䲅fjZl [Lۀ,Q tz^h,Uk条YyC:JY~ k(ZH#^5t(D` e hRTBci%!FPpJ9%J=(*b Z>Eb˺@zܷ1f'ηy`[9kXESX_Z+яk,krǖwoj#:_A۠U8{ز?0pfTJvr>y[\IݵźNϳ'q%5T'a)^ EFWo](P1{ⱫY\dqW(ҷV&34L*YZfwD'oty0F[ըWVzw}֞^bgNJ\)]&w07#F7fiľ+\)꼫K.~ ׍= e0*4% A?t=܄VsJzuc.8Ig1$v8_a6N0Һ&^h4a3#~c¬?>ݒp=HyduȘJb6|?Ϳ?[uj]T x{+%1WV/k.c} !ٯo\ #߈1otop\9@,z]o\Ɇʯ~L:4ĜS WTсǺgre}=yt9HwjmN#t C]L2=F2VD2n=͝}'fh+ >gendstream endobj 120 0 obj 6877 endobj 152 0 obj <> stream x=ْ$q =ZK],܀#$)mɖ D-L8dۏoݠMN9n3>Ͽ:~3zs?2w/Y)pCH)׸`=DWeOv0Q H|@pèY(a_ݞ̠q`n( 2(-:ĸȥ1CӥQM_Kw2.t`vCSHșeC$% 8h8X!܈ei+{Y=;4MYWe`iɱl-, ~#NJz5Mhτ nH.0f:"uE3 e81<#܂BźɸJ`[,!uYn º gdfC_VDk-79Q/54&F}5-<.STYd4NtVeϨ|bcNepY$bEqg2yp~>,F&5`.K`gtbY8_^< 69ɕ4$ Q TQy ܦXxc lf3hu(!NF[r&;zv lJE]IT.LFZhyhK.|dUDObLR4% @5;OMcb+(Lƿ4g{}OZw><Mlѣ蚾M]%D壵ěOB/ɠkkp4hsU >̂2aű@dʯ$ ZՁp aI8pEe43pH:Q$ {͘4ބip0ZGgۙ44oWth:ij*=%Kh,2ޗʝ>\%= Fs/~. чIYa'k T!7.£&-uٚ_!̪NS8CzlW(a19+w2hN̒ϡ;W&[yLyVd@ 8Lxa*   nc`%"09DUEܜSP!G-\>L\|ycV2}=MK 8EZp_~~3UwChV{E깐B2S.27p_B٢=irZj[ TKx&whQ[<2Sf/:;)83CFj1@nfP.k7PayjԥAOb.&f3UU@ L؂~Rfȶ[|ĈNѹK\3t; BC4^)5,!"PgpϠiw;k뾫 `$ cDIKlIX-Q^ ۑoIv& 0%3)L#Q;ԩhssN٣@STTBqCXIQ Qv &wOF܎~7/бat]wlvpŃ ·DKD$S]q%zwMi^](L #i8 V@fӇ zJ lI4[z!z`maق:3Yy;j]8%+ xπz5`lM+ZIacEʨ7 06~/O\^Dq%4%GxQ|[ /nݐhg4a?YR.\V&_#֜eGT=4ǥc8B;ekٷ66YOYMX S}E-\i_*&DXw]E&t٭ 4E$8HDS*Ъ#3KxB.#XH~DW4GP&@<]ߥ Jǃ4sYXXAHa3BQ:dDڇaQ3jZrklÂApwFӢℎ8>Ed.V}k |J5fKcJ] M t&S\11J5-.3c*˿5º\Lǿ, 9Bu,lҋx9,B4㦈r|@0n?k05n̛+1d&0tdQds #, jk 3e$D"qLz tK}U\7SPQ>ӫVΜZfYlm[6=x;\T;!E{LS $ګ0ѹ ~mAȝLL!*9b[|߼9|,-`9?Ub@Gew{msk F"89[#*;nr:6q=^nIA-6ZCp6yiDU[t|ӤR-k2u,k|XʼnG窨b>/j}u, X]Wozxewz3W113?ɍ4"]!Հ5")}W3;Ͼ\{@]޳K6,W Q@+Qp gd/ی19BY}ԶVfzUI̙ss 77Gf4AKpZR"o2i"אTȫ,q85SQUL&_ Hڲbs(b%2mfv3f</6~^9^< .G--ƃJyϰǨ̮/Fn75aB^YQB-zK;yQbr${{MۋZyìBϴg[y鍞5w mexTs;Ř!9"#eGjykܷ]#rZWvfY_R=.)_EՍ4h[̌Ehv}v}8G:Ɉm Ȯ뤮 Ι{,^_( E&E.i(v1,Dr/J7l^@(lg,g[#McKVn3M6X19 ,Wȫ\MKY\z3q*eI iI0kOd(Xd'ylu<Ł Ʈv ߬ +5,fc@1.]8ۖ~tlMPӒrdۣ a ;f&f`x8M65k:{)>V Yd idHMNfuV2R7n.*:C̨;L 'FuFdqT&X\XY;}.jlu\Ca,*+s entN=˝̔ƯtA^yzt;[$K[*Zjai 0%#7}CSy{YَmzLi4'8媕]Ak`nukUkm\pWcm2ĵ; ֨k*ʮ]kEzVyYn;^D3Q1J9kUײYUfY jae6*,bMR016s 2nxj&'m˄=!EOr7jºl&qvR#Fg#]tb# uzFYNص4]0M)#.P;r\u x gNhz"lEWM¹Om:.ؗɹBƆչ tc ( hnO)++BS&ht#qSn9h\6S }^:{q)s20(-^O7r_OnE()ZSr_0MJxjG XB ! ܈ SuE&La-y_qT%Wڤ8D`uSTҚCv{iq\5s}Ib_j>dIv,bY$ݹ37cpIv]W0 l T;%3TK灬9Mp<y ,wH ghs#ּ煻مI4luXkw3 ]ҷOZ&zpka&P3X]ut*&b5yEsnsI¥%k~J7y˷[=rwQX ྍ(a3AJ5oLΌP.:< KO8z Wc @3쨗\Ptn*% u΂JH|-+t*g~;DPG'ŹG JE)h_ѭgf:wz^ĕ/,>pNpkM|0ZI jzwu04i_%҉y7^"kgJz`/<錂*{]n.%j@Ц״OM-rSjٹ4_qL/i.5p_s, f`{an@9Z 1S"wZH x1W1| t<9GgƢ'C$G-Ce<'P#;^5Fe[x}d_x$@Mn6TVw$үrxQdx)uoW zw7%bbZo]L*bF&lsBLoAg73q֗\;ӥGD:_.(X `HNӆWYWgs/^(zgD#s9ib ">pg,1:@-s9:=A(2p\N薴Oo4(t/n7TqO4D}dO?x4o"LuIY,.Nī9?5qoysھL5,!dU&}VLj:ypi\woZ!5 RRIjO>/<]mQVtW}Q1l4Ge{2_~(s(P *r-_G٫-jeڡ>,,e#;՗4GvC߼girLX5^썥¯pΫeaV?"0'#GyTb[Pv6! 9"W .1>rA.sR<5^Y^eKpI [+$]Y&SF&W$xB/U)6~v.$eVy_+gjM j^tM .xDLHrRf?|fendstream endobj 153 0 obj 8188 endobj 179 0 obj <> stream x][qv}9:Jl^ܞ׫;<~~&]9y|+w\'~.;>!^NYC&tJq ^l~ݾ 0n5;yBn){*#;!nx)pڋNtA}KHM~hA }9z="> ^n?Is 0{&nR6ʈ = Z`.Z'2b+xTLHmۊ7Rnܕ9?J/ `3̺IK zDSEt U3AzYЈ eP#l˃*\㘦 L;0A+O Is"IsBA"Moq~3S~9|gA3pFlfɛ1-ed bYCJ:aA*LzCIY,eˁD5ŧbKj(TR)) UdR\42_ %%O;փKYrX.E$Q*,jň>;ǿP] `lc8ZsNF'/WG5aa \@%дU[U&&xy+oYSDWԸ5k.-|Tã8XA>/%`PA-d_QI#+X ig13Hw|~vLl֨EE(9onY|L&PR+'&ЙԽV0CgY%rhVlA`м`(M}oy)h` v?ALQec7tD%ϵ#;Y54beݳ´#┥4)* -h" `ylr|'|ܹ?M`a#77O_|ӯ/ (xyq$_@S~Qۛ'/.z977,<|?S ~寧-`2݋NՓ{$ 8IA`D Q 0-r !kXl lF^R?'ݬ汞E'>>X3(6s٥ؗj])~2#.,&UwEXc*gB ֪&htrU6~ߡ4 @YYj˓޽8JO]W.] ׬W*U8ȝrƝΦ}Oc*6_8>6Nw*1+Rf\"cP4b&Fn =͢2JQI0C1ZF:aAA80d&=Y^(rƉD[/#mgTXJEJÝsU0mԋѬ5E]&>(!Y-"*  :Kf&g$ h89Lj{*s)=Ca| p?/MP`SvhY3.M{l㗇 Z닯 mKZajnlk)F6B,C{Э0HyyЕ hTu^mڹt|!N/vXB:0zFF'+ݾ탏$duڊl.c=[wo?ٛu/ mq _\{k7Fբ]^>KhM9w)d!+}L[ dqtgY̟ 68ЌݾXg> 3"M5Jf!%\C$U){uvEtR+*󍧴% b2Vm󍡘nr ߡ1.c @wc ߵB>*Vt˃۝BOR@xzw"P\\4 MiJ Y@7`g~(I/ZU4·=M?5g|f-NnO"tJ)ei0%feC-OeQ=;-;Y+ԍV8D8J5N1v5zŊQiv^҆8byF/FH _tNk[*󮜅(RCzTķ?IH]x< JrTGkl0>`V8<`ĦTW!l;4 6 QZڈ$([de uǢ7M!'g'Y{I] sdՇ8!u7#{u౜Hۋwo`5 ^~&J<{:1 VNV}3"m2J5JPFq8Pzv̨TjF+5FyHLGOy(8 ʈ̼qEFs.?'[>w xp'ik&+if`\{..W۽=O!aW8"lj Y$">g 3<䪕#8Ԭi,Ud/ݸϿ\dOf*eKVz' r y80LiIq<Įxs`=(Zs+ݑۈyr̗#N eZ⮊?-xw^iKbxLb.lx:vb5͸ʯ3b;kFSVv?-'y<$$}6wl0#}V9xc`|b:¢\R_o8MrhM&xږF^%Of88S|% V3X_}"5;_}.s^@ai|yGuCFU:M~v0lЇj:98P?,u![V7Apwe NMM)ã[O,-glFIxePc`֮Y2E,ߝSē/R"Oc| @;VUboޘ^BW;_?>:c ʁU8Wg}gi#>-?|W&quxgYpTx*KZ^HhQ'$tnf$!,qyJf$݅DqJ?=UU!p1C52꯱2~3fX;FyvA{1)?x 1PubgⴢГ-g֦Wahs~1,7tbCwv]:c:lco?]cű=kh:E5O ؚ\/nogh';'4F|W *I,ئ!k:zĪ.1/{B}a|+,/rrɖ]eW>cߟ9-Pq{6 )l^E{]:j7\fOGۢK  uL*=\ӶRvm G]#O% za$V[}- 4XUitPo&:HJhx?&dnܻN:gpI,%MQmMARp~o*:~Fi=֩eS(0LW <*^'YrL'%D`tm 4Cq _!)#I._^_{QFʈR&ʡp`|0ih'[ϲG#e>.0A[&Oj{>c>$@nDlo-d[yLd'Sw#/\Xzv"010T͛dce=^=hO-&J=+K/ߴfhr/.u)N(Υ8!o3.ur2ʖ]H١._Ftxћl)ބ,7a-߱p9>-7dx9fJ{v,یEB6c?f s,&XL?.d4-L;x#NF۲(ޔKZt%p$;x]+t8V$=Yws2z,-C:h&OF岙,TnOڬUCq@FJJv> iɈZ6K#Qۇ;  "m,eDh[N uHvlО;R~d>66Yhsʒꭍ=Q-ج0sR2ө*o+9k 2?ĵ -{G3ӅY Fb=Z= nD3>Xqo4YЄ*&KHhO۬ .` 6Wċ 6Yendstream endobj 180 0 obj 6385 endobj 224 0 obj <> stream x[ksF_/[F?N֩dYQƖxKL$a— Ҷ2>vn)ZىS2}{n8Hb2HOt~삑mudoH>8 NJ:z~Ċ(5HSd~pY*qUJB-N0# 3,ZL(7Lh6$q* lPOɨҘDciLf Nt^Ϣ .RH8t[YU14#"K|!C)e<qŒdµiN~W8bL4qJ#լz6*VJt: C83%qTx0S*MkhڗIeJ#|\EQMrUM?dOb<+`\pbxEPcxl[k̑lO9)AgCZ'%3*I!~)E*Fq?f#-Y$JgX,cS? Xڢp\e`HDBK1S"iL >2 q-:&32i r3X2y2ظĵR"Q6 _gЮ7q27f%SY#  ,#'f0MɉIe r0i1k>CR݂)6hMS>"uR/1z::O(Lc$$ R4ናO܀M6Hqae::;xvzYog c/'gv0`&|j(\]p.2:* e SV7pb*FsH@،%RB!ܚfB#$]<`Y;=HC |0es|jJ:GϦ+UGp\L5 1R us k;SLE6 ޺EЄq29?NQ:ͲD),`:Z"3`eOc+#/Ę* x!bަeƶ0Rg­|-LDSHJ,Jbg072M "c@-5N% l`.J 5BpbSkISg Z.xk( =9!S+mY Ȥ1201ȌLHS]3 u̐`eRKf捉{%g|O9S2\@؈BX:|1 @iS58QPIe$Asf`X}4l\7'` ؿ %Z:4g#"!;,7b?@!<cM+)n -bx1.w ('l!Xz HhDv.7w*?Ԕ՗/ 8yvkmv̠̅Nl=ԶYvo@;蠞( sȽ SDaF"zr<-qjO7oar*d!5A@aVC0 eXK!{1cP9a-^[7#$%c"8|ykU8#Zo dqp;-ʩڑ83C2oB;MPLk`S@W'MtZ/'i6$ܻv<@E)^F-*0=0PАzNebN'mCAl#gj@y)pф'^^`bsΣvml+7V¿.cf&ꚱsz-[)L4>mPKUK lF J(7hw7{fye(2Oy:-RŬӤ%2@<Ϟ](>`K $K--WRYpiN-thKeų  e5@ݗ BTH(r^ KV`P(}fq ]Ux]F= jޏGG#I#h_b,Z? 0 d$i н @@d_ynh{x"ۤňCQ7 w6e>[S1,=﫴ZСgG|.xdNi*y+AS>(CHoG=|}\m אz;*N4Zԗ_+n8ֲlSfTnj&ı# A qPv?|=yFԦ/*u. M]797:`үFz6vv})1k; VKz:O]B`.>nvnMoYUԺr F?|4Oɦ\f#-?G[Ou`"7!Lnk N"d؀a{o~g^)+j 0+&m#@pg0f(V?0EN+6f,|UF:})HKTSz̐N _Be 0Ey{ ךi L0NEgK'E :{TX+H?[vGD[Rwo2hDȑ]Jjt կ}ꦀl]h;j~/D%k̯KT`Cǜ^{F@0\!^Iv:`Gaj%_n`h9잀 %yܹ_c"D} ?7KT,+hFq6Y1ejګ2ɋ|2:x/N/_L8y7SBEoLsi/2qWeD7u}!'e R&@]V\|~|X_&oMz"?2ij^bSPlXlٿ~(O& ߜ)oӋg'wGa:gE i"_Olɴ{}XzVXogu-×پ@:>sFpGbi\^}|'qE^ͷUUBPɗ1^Pj+'ޝO\TjRr)@" O/^^= VUgxb9q\D4QPH]HBMz{r0yr]zTgS ޾8XNvhz ~VVӻQ?_=Gr-!2z|ؔjԁo(Mq̼|/=}h5pd$.~r.Uendstream endobj 225 0 obj 4158 endobj 300 0 obj <> stream x[mo.QK}Ծ%@8[HR@)(d%'9$jџ쒻w(b[Nk^RxaWk ooa=S9%}ܽui|*]^ }ZͿ6]8fSQ~%;U{Ok*A[R@$փi}ph#"7뎹`Pﷷ_^|ɓ۩b!!O݇CƗ>M#Tk8Dl-O޾;?:G'>}ɳQvp" 4 T7 _e% ;7__)0)S}󹑀 DKc:i"b$"!3%rp0 % 4"C|?ևVI8ޞII@ž 3^Z_-|@Qe V/W^c+$aӑq;,X8LZߖBiƖprԖ %Tfi1UY*e\SBfVYr#7bZ$L`8ɱ]A'| _ LVJLj.ԁ`O"1)0 lˤh&gx8eaxZpPLƣ}vƱ(a'L/b)gvX ocTi_I: GM~xbe ֓!07Svt9,rcɅHP'm3(mH\FGf(ӭl ՘dbuѡN!JlBD))e-hY" 2x*g-Ho DEYukS0vHt@k57bccPF<#:lV-RRu*,"=aj{|X)̈Q" lE-M_miRY5k眒侣E@;&L/ߋ8VlFeÜre0/ȊAQ@:Ň AnYg]bfzj = g>jaVǀocɿk VPݞ]\ޞ/x Oчbq]r|ĨFȲ,UKly"rT࢕q>3ĸ@-G",k:I 1o# q)b[WɜFݲ^gF`h^l) Qre ȁa쏆؈b]haaL*rMz{-땇}K hÒ%fs|EwU V EP|Gc+x7Qq*݊CG\<LH-zL7f.]U. 4LLvZ(< XP0mBćXP=C="O$gB~)Q=1ADPqW+Kg_/FSU IH91Q&i7C;ڦISy25!_TL܇fYNퟬ9zͅ\.jmDcpXNHB-i·?H(\7 2 |w&(%j;#A@mF!޴*s*_nZ5mYC["  T'GzAbOc;8H#yCY,bJ7/vQzLlT=j{t1<Y)!Q pR^z!i'=yM=%u?BWfjPXcOnD>x51u%ɭ`>krf ;7Q{20vzx$*7*)MTAv*Faj=Fخz4Pm6Չ2D$#|n ǂG ~p5#A[R5Z2@ f ڹjh E;Ltwĉ&?@f=6ЦṾA$"ղBE+h ʸEi?Ǩ`4fS#9s[xVNQPq&LhZl){te[m;OVM(_$ǝ|%J4QF,"Kkua<8p]+H M;x?{+mX+8]XI57kBda\~8 (~qN6azƲ;xHهbgH `~ '>#㎅0蓝 9Zȧz$F >_*/4%xx2TojWC!>T6:!CvLR x(̚(ྺ}7]L A)F tڇ/Cc&ݑr.;UnRɦa9&}(;(l*HSGQAH\w҂eD^H(ҷՅˤggR]ܕF7<օI{; U,`Iz>0V#S$LC Vxusq˫q C-J(@ PG]˽mdp1``9҇sX3Dzg%endstream endobj 301 0 obj 4000 endobj 326 0 obj <> stream x\{\k/Gx?쓫,G(Y\N]jńK]+Ǻq뺁Ç)Gf@x7" _Ƀ6'~ mAGΡ#J+%94Z;Z6|t~}jN'1ZR;nN'k.tbl/xʌvirD m[0n>a +#l?iDz!Ǜ0Z1Oe2 U8vJ ߌZ' Nqv|3[>8=ʏ*DU6zqxc ֚5Z#TKpKm!4HA[k-ht3RJRնԸ [Fark k-Z k`b)u:$ʈ`tFؕixG_CFb[ `R2i;R'm ,Id oRװ0ΤO7#Wk| l-io !֎Q,HFu_;@)aNS~_1Md$iR@aAÐ"\2ԪF1R>\w *9kД#'%(oHuR)%ki?k !ӁO`ؽ㗶 O>v_Z)@ 6VQ*#J=0XA`| B*Xgo6@5bECOsԀt_!sR!a @󯽰jL4r2V}hp1ъJ\ÇJq({ D/S 0|Imͪ ɗHe.=q0TM>4&aBUמqj?^#wn Y(BjX!Jd7-jJ𺃨{ = $q ]ym9Q@ܼ? Z"&LhoH #%5ty  wn&B;ѤgvzсMtIZx<^2ntswNlnqԪ1gـ|e XTȝ~K+pbQ5ُN:);[5@A\̀^2=$9߀J v\R\'0*Цn<I?%(O<8 /""ýL6A.zԶr³8S}qO ~)%XtZc?1?8 oP($i$80+hfE!X )"֖;T/˓%[_cXHS0ni//HquekPxJZ4UGX*1V}N``5)Ikh>s1Ez*:_ Sq պ)Ul*[^:(hI1LDK"ESL|p H{Zs=lF1"S}a8Oh<"nBը`>lU+|)}CY$>2y$p+xZA &eiҠ0U: Xw$㖗P蚜ykmBaKߏ"j 4aѱjg`\ЦVj^n.. ׃yIt!X($k{/nDOэm&:E|l-Eqvinqdt6EZ]R$z^Vm+(*ewz['|52WO;8B<),cn{ iti0&HТۂk{ӝ.n? 57-h#|^>Db]KL!1ړm4>|e E9 ѶW7g+4q*6lyog>f Q/V||zy9GLm 7zv;[o}BbA98;듴ޭEg[ON?] ]/mm *6߼[zo'VTcRn!LQ7$tz3}u>>]>*sz 5Οx>;a<,߸OL &U& إ^H.6'z}̋'&?zt9#_ C_?].g>K/>\={^fI,Ӈ3Jֶ)1Bޛs92`"-s)%&O{Yf{8V.7!O0B셕̡7!vIUHIIElHOTlK4ӑI3VS.1jЧݿjݿ%H5ꃟHHIan:8 >VtK22gk&Ppz[p@CE)8 HU͔kSk.Gō_[6-=pŪtFn2lē-*G/12SO '^Cj-22.+|BV>869V1SDzb oe^h/]nd^9\QJ[;͌1 ~ V`ev-{^F+%̇2XyJg/,H|XEnSVE9JTS%sٹ :/Hs OM,D)̳]\[C\a'Tx٠Nl6 \gh *}8}C*MDi*AapB!㑴UQ!.G_m]}Ȧ\Ǯt)(T0!npP7mt IAAhL԰Lj*ڪa]᲏D*rOd46ܨٶ럟Wѱ7([Q%K|Hu^M]`UV !27Ά3BZ*MkAJqgEE $'//0%5͌3lGƄ$'~s^E'Q@jM jGh(9.=0sI*sxYcd]6kVH֤'x,ͯx'K$LJ V* 1]5 oj>V^cS`J3\6UCEQRh 5Zs"zbWf2(%%ꆉK0qyYiJQfgՃ7zzJt֌D'RQ ~7Y{'Pi}31ڣ`ǣRloOʫ فrZPyPUtf{4WJ\'}^_wcJ{&VZUY52掓I_zL,:*tsxYhaݭsnwE3\o$os3W^K_h=B<65d޵{~sŽ |:IUȀzD.?R6 CG(]f!-T5?7z)^;`lOqʵFUg$>.~>L<ӱGEpr"@/7=;цU k9en* e*Ptfn4BY+BEM[pHlˆg@ip&="]U.e)o@,W=چ!p?V|cTcP2 @kaֽ~ؑ0z`/kTެ஀m3MFW1endstream endobj 327 0 obj 5143 endobj 341 0 obj <> stream x8>ba+ˬ*18;z3#V5F1;G^!xd׸ 3\ |p}*É/u34+g#.mŌ^Jjް&4/y%MhDlY[~: Тg(#-@+HUF%k[y>VRvU t\S)vqzVocI a 4(Tv`+EQhe5)m#W2B D+9}[^ %@UMc@ oPdu-@j<1C5@V^ tȔ ˪:5ćS"}$R)[Yz1~5HZ bJ~(Z¢%Ѽw#>_EϤML1JbuTSF  ` {0-5R9$W@>taO*h&)ˆ,)zL,8Zt7eݒGs7!5PHaWjv{&toGoJ9ZFJ˴EU<ytz8`G z߃?x9ֶE&& TCg]yV)sW%29PμT!zj c(,8LT9d` J+?i;)a]ÕbKh%6X =88_ER.) 7NJWWM1ym$edR1U¢+gNU6RR.gV(y8USїXb.,!b⸶ Ýtfݙ{jɑ-Jl}QM{"Wv )y+ )xBP,x 0#61qj`_ٟTvB]d_GgPӑ`]GЎc:BJ+ߒ6H42#YAO?Poe\$fNS޺SI!K63La+|kܽc/h8сP`iӸ3fEwbkF뇴F=NJ|={qb Aj ׌EUe^=`POj9c8DS(6d>pP8[ܜHf+#񫂊p+"w ޔl'rZػJm)2Q?lY(FTлŻ%3z_<#X[>"q^BRfVkL@` C?b:Cݟ\֓ Z=Q?r)yw_mKp/쑦w :b8^|SU_; 3$Up/;l$Xь80nU\GխLl_夈gxX/M:3\p9#FBj{ydsO'C ׸o9~_Y@Z$d[t^6U}7R(Na#$ePD*<ڝx-Z}ǯ<{Vp!OJJa|92s曧4ȥs+xl/^}:{~ kismgֻv)7W?CjU.kƺc[@& Ò!ϓ^mShv|XŲ&JYD4dK~:V 5P[na.mzO1ּca`cCڱˤJ Uv ɊW" DMp|i> stream x]$Q Hd|V {w^eYrH<UU=3G2%++/j^_A^ wۋKL`gp)C>X5}|/߽ŝ__ʋ;?|??wv'w/IW1|0x&p C*RW/J `/^\l}ɇ۫/>O-ߒT:^!:L aZ6;5OC~sC0BͳʐϞgZ L LЁ}AEmiټ?>nl`]4&< M9_i2-/aћ G(pw t(Cj+/ &_-. VƨmZJ5^P͠oA |-W tӵbXW+O~M'٦chɹ-+pKѧ AG^S 'Fa@(ݼziVBfg0b&]ljf e:#4jhҝe;<RX BHY^7D2 YؼZHyE ,uv ㎗mU5G:"s WGާOG.d&\9X2uo-81I n<-Y/U6@[ kShG5@)`%ORIseA(*GٍTME/[/5E=muАS( Oq N<0EېkVGXf {`yB&f(y׭>#gNL)M }U$*=H""P9Wæ(F|jtqtgY<  SG,uGCQ ڐ k!*!Ӯt7Q5XuJ| T$3Z5Mf'8鴡r#4$nj;9zYWY:EQgXiI&A7| 2/4^{۠DC.!ewӢ.D3{3c$IW ,MOriU8^}=e0:'X:z{ZT|! mI|k0p ]NN R궊#W=s+3z֖xkryguoC8 >7ćBeT؄%|'tkTp6ėΜS7w:g/-9*Z"19L_$MxZK]cXlvD\J0B3<hv)HRWB2/OZBʾV1ץjWnsQ= @OYI6҉N-.] |_T_ }7…V԰ijvgP\*,oRH5 &Ȭc(LJ5t~E)(x=_N0=Ss>In.FW߮i%SQ! ZB@KC"W*?i-]!5 ")ĂS1؉  -9XKp9fi>x1S7Ri%'$w95,H,ǝɺ5x7=) \uz53i5-Ы,`[dCٿj3Z~ݦ^1q. $'XH@[֋ނ^a!tEflY>5bjxdژ:HuBL=L.Տ=gZ?䒴~ļH)ĺ&t)0XһdI,>9k˕"4_omѥ][8wtY4N#HѨkB]|8Zb< 9o"4KZ?}"dOL smZ_=x؈[dptZq`ml3x:u396k5 YWg!?jv:N{DZ28(xeBHLmK60Z'2Z$]fAvY]fa4$?.kE|ӐJd j 6f ^i$ !Wr޶>U D fd)CFIY< >z{Xt) +,{ᎀW[}J` &TPG 02i<3G\cNCr`꼈IGD7-/"8Joq˲< !-aT=jzuetk#5dZ5ye V~$_!btNçI#Ţ"ag"ΔxfoZq)hZN=v8נyU;ƀ O"KC8@4wܥqlŠgWMQ2+I W:'~œ֤7 W64Igtʯ fL!nV*rMI2ީHԙvo[[&լ 1b3-2xxR)"ɋmj [n&X;yn˂@\ho`/Af-v-ޚhk Ej5mhsM3i.G9JOwvD͉y..e+&*m;5%,C8Y\yHiZnzƳtIDB,;fȝR*F9T÷YgT3ٷ8f?Ny֯2LɸnAmmuoE-݅)+#@c0N^5ev]}"7 ul7W4Yl6S";N'uPy*-"$t(>0ţgTin ~C#v[,[k'9tRn) Cendstream endobj 349 0 obj 6239 endobj 357 0 obj <> stream x\[%q8 F1iI?*ή#Kk8:0=s\=_$?5ϩ"b7gF@ `ӧyb߬Nzo+%Vg{J<}fO 竗o+;\=aG<+vNW *Xmqݬbg-+/;elX}]"ޡ$$Cai㯺k;u]H7C t` sx؜iGQ9^kFOCy]P& r]o#T=wJab,, 1RIޅ(^iw^-LP>_w7wҔ)@uB5ga>h!!2U=Y*\R養k:Ѡ^Ge Lh Ѽ\zk@9&3- 6IYl D AqX2cTza5HYT_ܽ_eӫΪzytk JXPMP-lN}Ir P{ǠuX@GtĒRu`zgU"1e󼍛ƏfkBaÌkj4M_}^7V;N+Crs"ɍG YTF+N?HIP1v% ty*Pk]—U9b=݃}qAhj沴[&L/U0 fK={*!bJ@{ԘӲ2%[ ؈hvEGk @U,ӓ #t #cablL,/k Ў4{`S`-YT zD ^;1L=pP"V*e낥#S*VƆ>}Io0i0U PCa^ͿlkzSsULC%v 1ab|zx3#N (JF +13v6Aw-v;M=:nH!ɃUVmLזA7/vV[ 0 Xyp3A)Rfu?ĩuH *TA!cm4gc0VKrԓTHX Vqk[~֨QXך [JY4Z?N0ryp$DiFZQZna9^]6"ʊH4z`ɿqnCBr5*1g?3]bxKġCJ`,QwA[sӠF+֟zJz&ƖGb20cD&_(Rɤk͉IG}hf6gmkg:o2RҖ aM,QV S$*~o~v`nz.> (>*uy}M_m }|;)_ܖۯb?^o7Ri] $  f3i."c$aM{%t6N a"1sm9I: 68ٖJ2ECRQK&ۉP9r ^. ;Xf qOno>;ߕ5>Y\c$Ri^ihI6~y IrrX-χ X5ߙy≨3ޔ6 c_!I7ܥk\tdBt&ict0MH" 7厛V6 U+DXGda'GtlOXF2O޾埿7-r*7CT-f?j1Dc*?41+(: BVUflԇ,oh`b?ΐS˳#jˇj!TNq%3Nտ_Gf=:^82f5x1^ubשׂ`'Njmn4 ~ʦ jAthŏ1 :A~>j`PPh\a`Q-rG$.ve$tP__-|7)Nl}ɉ{bBl> P:ܲxG %N_h/ "7_io0 3:EŞ їWA#ח<{>kiи~7.C 4XT#=Բ3]޹_!FJcVfsO@pIo ܗ͘z?(r\n/y1e.Ɇ.O'hcN<.ȕtpo |l@V !{p_$X%;`q)p~x&1)XSKTERŬ|޶c/1鸽b ,n߶G\)aO/`F>HL]_G;Q82Iek0;%Q&ifOKFFv5I_HG1QMPik$0(χR0k!#dz1wtnE>_i!˰f `Mr"dJ}Lrl2drNUI&y9 Q]f\02}9MNkuàx*KsSS6%bns UeqBn4k7)_İQ2(EV$X9rMttέ$EhP"K[B?I j~vt:77@(NCAXt_E8lj @-`o{;_ꯎ:uK0pLϤ &ԑh~s˿o(v޵mʅV.#: Fs3󶁯 _l#eubQ] 5N+s%]x\&-%0]'Ò+,-W)[E.F5Q1{js;}7۫.*#hMF9C[n~ԯ`viW<*.҉,J7QVj9Jѱ"sA>t4dӸeB;t$EoI$p ~D-p\̹<3X:ۀ~l_02'!u0j+;.ѻIJ|ʭwӆ'G9ea,Ob%g) `8{9Rf"͈Lefj'Xȭ'61ddRɳfh:D9$f4͔!cU Z[Mf^~35PO*TbsÑsx>giںG7 p z`I|(<a19m,"|FRԏu6 `9RPJa_Z)>GD:^O|<O {}x@bʤ)<,YqSғ&AcջXZITמ]ʽ50*N% *P@yfLe)TLsJH>| -en~h-J"#Z6s&V.Kz '(t:V1UY7C9΋J歵 06ӼB0oT ^~*(>#~1:<ӥU*cfӬ~@~YOv`j'ai^v# d3 7sZxκT#us0¦>K&q A=ʛzp1Rf58lFS8.DK6 -02Ls'%vxXkTcfOǰ"!:ͬvF%FdITNER Y$yL5zqDZVD3[!a^-XͮYYtBNi5?A ?f":sS#Vhpht5oJʀ>vD оvE |R'!~62c#60P6a59nn9>˃pu62$i'}GUx}Jx&b28aX,:? e޸U9?P7O23{9.0~۴(0/_ˆ,u5* B,nʛax#upXyXcR7W)#]wQ~dC4t4^YnBK'A|Aɦl%i\xSVoi-y9Go Ǝ4"]*u_Ԑ"vuۑ_\tb RDc}J@/IS55LD6ƮygC o J|FWL꜖N` ˓-~ ` oSD8-N痌oHN1P#27$iHǦ.w-D2qO%]Q/:9U*xطjjH!`DI w0Sx$47:j ym4L?|9ۻ8v`i㮜>9.;:1N5fGO|UxE]׻s{0(I)]j'{pE&.`&zh?u+LqKcdTʔ[9O0,}Tctrrtn]&UyuSVx|YdH*W-]\qxJ|ѽ/\JamH<;xIir BYG%jh+ڄ$Hn7RnKHyu%̱p(~[ =Njq[ SԊfG*LQnGE;;z<z}TtT#Sv̸!iȖ:!vnr#K<ْuN}:YMOG6۽}TY`v^  2hg~4$嫽7WboOyw{Z~|Ab|uZQ:m>r,'1@8\VDǒ ч˻{@Mv._/鯻ut g`ת:! hek-ĬPɟ8Fc`SƨaqǸـ1~񊰀+ie(P _sendstream endobj 358 0 obj 6319 endobj 4 0 obj <> /Annots[53 0 R 54 0 R 55 0 R 56 0 R 57 0 R 58 0 R 59 0 R 60 0 R 64 0 R 65 0 R 66 0 R 69 0 R 70 0 R 71 0 R 72 0 R 80 0 R 81 0 R 82 0 R 86 0 R 87 0 R 88 0 R 90 0 R 91 0 R 92 0 R 107 0 R 115 0 R]/Contents 29 0 R >> endobj 118 0 obj <> /Annots[134 0 R 135 0 R 136 0 R 137 0 R 138 0 R 139 0 R 142 0 R]/Contents 119 0 R >> endobj 151 0 obj <> /Annots[154 0 R 155 0 R 156 0 R 157 0 R 158 0 R 162 0 R 165 0 R 168 0 R 169 0 R 171 0 R 172 0 R 173 0 R 174 0 R 175 0 R]/Contents 152 0 R >> endobj 178 0 obj <> /Annots[181 0 R 182 0 R 183 0 R 184 0 R 185 0 R 220 0 R]/Contents 179 0 R >> endobj 223 0 obj <> /Annots[229 0 R 230 0 R 231 0 R]/Contents 224 0 R >> endobj 299 0 obj <> /Annots[307 0 R 309 0 R 313 0 R 314 0 R 315 0 R 316 0 R 321 0 R 322 0 R]/Contents 300 0 R >> endobj 325 0 obj <> /Annots[330 0 R 331 0 R 337 0 R]/Contents 326 0 R >> endobj 340 0 obj <> /Annots[344 0 R]/Contents 341 0 R >> endobj 347 0 obj <> /Annots[352 0 R 353 0 R]/Contents 348 0 R >> endobj 356 0 obj <> /Annots[363 0 R 368 0 R 379 0 R 380 0 R 381 0 R 382 0 R 383 0 R 385 0 R 386 0 R 387 0 R 388 0 R 389 0 R 390 0 R 391 0 R 392 0 R 393 0 R 394 0 R 401 0 R 406 0 R]/Contents 357 0 R >> endobj 3 0 obj << /Type /Pages /Kids [ 4 0 R 118 0 R 151 0 R 178 0 R 223 0 R 299 0 R 325 0 R 340 0 R 347 0 R 356 0 R ] /Count 10 >> endobj 6 0 obj << /Count 3 /First 7 0 R /Last 28 0 R >> endobj 1 0 obj <> endobj 8 0 obj << /Title(\376\377\000M\000A\000P\000 \000e\000s\000t\000i\000m\000a\000t\000i\000o\000n) /Dest/subsection.1.1 /Parent 7 0 R /Next 9 0 R >> endobj 9 0 obj << /Title(\376\377\000B\000a\000y\000e\000s\000i\000a\000n\000 \000i\000n\000f\000e\000r\000e\000n\000c\000e) /Dest/subsection.1.2 /Parent 7 0 R /Prev 8 0 R /Next 10 0 R >> endobj 10 0 obj << /Title(\376\377\000D\000o\000u\000b\000l\000e\000 \000l\000o\000o\000p\000 \000a\000l\000g\000o\000r\000i\000t\000h\000m) /Dest/subsection.1.3 /Parent 7 0 R /Prev 9 0 R /Next 11 0 R >> endobj 11 0 obj << /Title(\376\377\000P\000a\000c\000k\000a\000g\000e\000 \000o\000r\000g\000a\000n\000i\000s\000a\000t\000i\000o\000n) /Dest/subsection.1.4 /Parent 7 0 R /Prev 10 0 R >> endobj 7 0 obj << /Title(\376\377\000I\000n\000t\000r\000o\000d\000u\000c\000t\000i\000o\000n\000 \000a\000n\000d\000 \000m\000o\000d\000e\000l\000l\000i\000n\000g\000 \000f\000r\000a\000m\000e\000w\000o\000r\000k) /Dest/section.1 /Count -4 /Parent 6 0 R /Next 12 0 R /First 8 0 R /Last 11 0 R >> endobj 14 0 obj << /Title(\376\377\000I\000n\000t\000e\000r\000f\000a\000c\000e) /Dest/subsubsection.2.1.1 /Parent 13 0 R /Next 15 0 R >> endobj 15 0 obj << /Title(\376\377\000I\000m\000p\000l\000e\000m\000e\000n\000t\000a\000t\000i\000o\000n\000s) /Dest/subsubsection.2.1.2 /Parent 13 0 R /Prev 14 0 R >> endobj 13 0 obj << /Title(\376\377\000M\000a\000t\000r\000i\000x\000 \000o\000b\000j\000e\000c\000t\000s\000 \000i\000n\000 \000m\000a\000t\000/) /Dest/subsection.2.1 /Count -2 /Parent 12 0 R /Next 16 0 R /First 14 0 R /Last 15 0 R >> endobj 17 0 obj << /Title(\376\377\000I\000n\000t\000e\000r\000f\000a\000c\000e) /Dest/subsubsection.2.2.1 /Parent 16 0 R /Next 18 0 R >> endobj 18 0 obj << /Title(\376\377\000I\000m\000p\000l\000e\000m\000e\000n\000t\000a\000t\000i\000o\000n\000s) /Dest/subsubsection.2.2.2 /Parent 16 0 R /Prev 17 0 R /Next 19 0 R >> endobj 19 0 obj << /Title(\376\377\000A\000u\000x\000i\000l\000i\000a\000r\000y\000 \000r\000o\000u\000t\000i\000n\000e\000s) /Dest/subsubsection.2.2.3 /Parent 16 0 R /Prev 18 0 R >> endobj 16 0 obj << /Title(\376\377\000P\000e\000n\000a\000l\000i\000s\000e\000d\000 \000l\000e\000a\000s\000t\000 \000s\000q\000u\000a\000r\000e\000s\000 \000s\000o\000l\000v\000e\000r\000s\000 \000i\000n\000 \000p\000l\000s\000/) /Dest/subsection.2.2 /Count -3 /Parent 12 0 R /Prev 13 0 R /Next 20 0 R /First 17 0 R /Last 19 0 R >> endobj 21 0 obj << /Title(\376\377\000I\000n\000t\000e\000r\000f\000a\000c\000e) /Dest/subsubsection.2.3.1 /Parent 20 0 R /Next 22 0 R >> endobj 22 0 obj << /Title(\376\377\000I\000m\000p\000l\000e\000m\000e\000n\000t\000a\000t\000i\000o\000n\000s) /Dest/subsubsection.2.3.2 /Parent 20 0 R /Prev 21 0 R >> endobj 20 0 obj << /Title(\376\377\000P\000e\000n\000a\000l\000t\000y\000 \000f\000u\000n\000c\000t\000i\000o\000n\000s\000 \000\(\000s\000\)\000 \000i\000n\000 \000p\000e\000n\000/) /Dest/subsection.2.3 /Count -2 /Parent 12 0 R /Prev 16 0 R /Next 23 0 R /First 21 0 R /Last 22 0 R >> endobj 24 0 obj << /Title(\376\377\000I\000n\000t\000e\000r\000f\000a\000c\000e) /Dest/subsubsection.2.4.1 /Parent 23 0 R /Next 25 0 R >> endobj 25 0 obj << /Title(\376\377\000I\000m\000p\000l\000e\000m\000e\000n\000t\000a\000t\000i\000o\000n\000s) /Dest/subsubsection.2.4.2 /Parent 23 0 R /Prev 24 0 R >> endobj 23 0 obj << /Title(\376\377\000P\000o\000t\000e\000n\000t\000i\000a\000l\000 \000f\000u\000n\000c\000t\000i\000o\000n\000s\000 \000T\000\(\000s\000\)\000 \000i\000n\000 \000p\000o\000t\000/) /Dest/subsection.2.4 /Count -2 /Parent 12 0 R /Prev 20 0 R /Next 26 0 R /First 24 0 R /Last 25 0 R >> endobj 27 0 obj << /Title(\376\377\000A\000u\000x\000i\000l\000i\000a\000r\000y\000 \000r\000o\000u\000t\000i\000n\000e\000s\000 \000f\000o\000r\000 \000t\000h\000e\000 \000o\000u\000t\000e\000r\000 \000l\000o\000o\000p) /Dest/subsubsection.2.5.1 /Parent 26 0 R >> endobj 26 0 obj << /Title(\376\377\000D\000o\000u\000b\000l\000e\000 \000l\000o\000o\000p\000 \000i\000n\000f\000e\000r\000e\000n\000c\000e\000 \000e\000n\000g\000i\000n\000e\000 \000i\000n\000 \000i\000n\000f\000/) /Dest/subsection.2.5 /Count -1 /Parent 12 0 R /Prev 23 0 R /First 27 0 R /Last 27 0 R >> endobj 12 0 obj << /Title(\376\377\000C\000o\000d\000e) /Dest/section.2 /Count -5 /Parent 6 0 R /Prev 7 0 R /Next 28 0 R /First 13 0 R /Last 26 0 R >> endobj 31 0 obj <>endobj 53 0 obj <>endobj 54 0 obj <>endobj 55 0 obj <>endobj 56 0 obj <>endobj 57 0 obj <>endobj 58 0 obj <>endobj 59 0 obj <>endobj 60 0 obj <>endobj 64 0 obj <>endobj 65 0 obj <>endobj 66 0 obj <>endobj 69 0 obj <>endobj 70 0 obj <>endobj 71 0 obj <>endobj 72 0 obj <>endobj 80 0 obj <>endobj 81 0 obj <>endobj 82 0 obj <>endobj 86 0 obj <>endobj 87 0 obj <>endobj 88 0 obj <>endobj 90 0 obj <>endobj 91 0 obj <>endobj 92 0 obj <>endobj 107 0 obj <> /Subtype/Link>>endobj 115 0 obj <> /Subtype/Link>>endobj 116 0 obj <> endobj 117 0 obj <> endobj 134 0 obj <>endobj 135 0 obj <>endobj 136 0 obj <>endobj 137 0 obj <>endobj 138 0 obj <>endobj 139 0 obj <>endobj 142 0 obj <>endobj 149 0 obj <> endobj 150 0 obj <> endobj 154 0 obj <>endobj 155 0 obj <>endobj 156 0 obj <>endobj 157 0 obj <>endobj 158 0 obj <>endobj 162 0 obj <>endobj 165 0 obj <>endobj 168 0 obj <>endobj 169 0 obj <>endobj 171 0 obj <>endobj 172 0 obj <>endobj 173 0 obj <>endobj 174 0 obj <>endobj 175 0 obj <>endobj 176 0 obj <> endobj 177 0 obj <> endobj 181 0 obj <>endobj 182 0 obj <>endobj 183 0 obj <>endobj 184 0 obj <>endobj 185 0 obj <>endobj 220 0 obj <>endobj 221 0 obj <> endobj 222 0 obj <> endobj 229 0 obj <>endobj 230 0 obj <>endobj 231 0 obj <>endobj 297 0 obj <> endobj 298 0 obj <> endobj 307 0 obj <>endobj 309 0 obj <>endobj 313 0 obj <>endobj 314 0 obj <>endobj 315 0 obj <>endobj 316 0 obj <>endobj 321 0 obj <>endobj 322 0 obj <>endobj 323 0 obj <> endobj 324 0 obj <> endobj 330 0 obj <>endobj 331 0 obj <>endobj 337 0 obj <>endobj 338 0 obj <> endobj 339 0 obj <> endobj 344 0 obj <>endobj 345 0 obj <> endobj 346 0 obj <> endobj 352 0 obj <>endobj 353 0 obj <>endobj 354 0 obj <> endobj 355 0 obj <> endobj 363 0 obj <>endobj 368 0 obj <> /Subtype/Link>>endobj 379 0 obj <>endobj 380 0 obj <>endobj 381 0 obj <>endobj 382 0 obj <>endobj 383 0 obj <>endobj 385 0 obj <> /Subtype/Link>>endobj 386 0 obj <>endobj 387 0 obj <> /Subtype/Link>>endobj 388 0 obj <>endobj 389 0 obj <> /Subtype/Link>>endobj 390 0 obj <>endobj 391 0 obj <> /Subtype/Link>>endobj 392 0 obj <>endobj 393 0 obj <> /Subtype/Link>>endobj 394 0 obj <>endobj 401 0 obj <> /Subtype/Link>>endobj 406 0 obj <> /Subtype/Link>>endobj 407 0 obj <> endobj 408 0 obj <> endobj 312 0 obj <>stream x31Q0P0Q5T0T05RH1*T06 (+Dr.'BIQi*~8PP S!K!A,`c ͥSW bg颠(kۼz/8\n̓"*>Q¼Zw߲6qUw3:AY 01(pz*rGB endstream endobj 304 0 obj <>stream x31Q0P0bCSCCBcK 53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]Vğ\dG[>O~,9HSrA  \ \> endstream endobj 166 0 obj <>stream x31Q0P0T5P01P06SH1*T0((%dr.'BIQi*~8PPS!K!A,`c ΥSW bg颠bAZ&~p?>stream x31Q0PacKc3CBcS 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠YƇ-_/h0EW>8[ߘnzpa׭<: \ \H9 endstream endobj 123 0 obj <>stream x31Q0P5&& f )\ &@kg " ,T!9ɓKW4K?KK)YK%ڐ KM ZSW bg颠y C< }-M>9}nկ_Wk a%jcyU  \ \gE endstream endobj 336 0 obj <>stream x31Q0PacKSCCBcS 53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]v]/:ZTðV=n_6ʫt`aprT< 8m endstream endobj 219 0 obj <>stream x31Q0P0b#c3CB53T53 S\.'O.}O_T.pR.}J.}g %ڐ KM V8qy(x6ySO>?}۽<5Խy;_VB Cz endstream endobj 163 0 obj <>stream x31Q0Pccc3CBcK 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠XC× ?ܻb?"7C/^2bc|1X{S`á  \ \O < endstream endobj 160 0 obj <>stream x31Q0P0a3SCCBc53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<].x^=qu?[$y8\=J9 endstream endobj 405 0 obj <>stream x36U0P0V5T06T01RH1*T0 (+Dr.'BIQi*~8PP S!K!A,`c ͥSW bg颠ПEN {k_/U\6Ƿn`hrT: endstream endobj 377 0 obj <>stream x31Q0P0Q5T0T05RH1*T06 (+Dr.'BIQi*~8PP S!K!A,`c ͥSW bg颠AƇ-r:M^f/m䎛'io&>Vgֲ陹6Enݰ!1 B @ endstream endobj 374 0 obj <>stream x3P0CC L1*R.HrU()*MW0N %ڐ ES! p endstream endobj 371 0 obj <>stream x31Q0P0U5T0P05RH1*T06 L r@"9ɓKW4K?KK)YK%ڐ KM w)+jtQP[$"7o CK? ]:ۦqߺMGS! J7b endstream endobj 367 0 obj <>stream x31Q0PbcsS#CB# ĵ3T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠SCJG:>7 endstream endobj 364 0 obj <>stream x31Q0P0Q5T0T05RH1*T06 (+Dr.'BIQi*~8PP S!K!A,`c ͥSW bg颠 B^f_4`drT6 endstream endobj 359 0 obj <>stream x33T0P0bS scCBS3 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠S鲟UzZ.>ߙۿg݊]x?7PjH`Ᾱ \ \|Ji endstream endobj 318 0 obj <>stream x31Q0P0U5W0P0PH1*T06 (*Dr.'BIQi*~8PP S!K!A,`cﭠ q)+jtQP[0GGWWu0p3agQvOB~}gmB B< endstream endobj 306 0 obj <>stream x31Q0P0V5T01T05RH1*T0 (+Dr.'BIQi*~8PP S!K!A,`c ͥSW bg颠 v5_~i7˯Uo}_/W[u}oۻ07cLߨmHap`rTFx endstream endobj 294 0 obj <>stream x3T0P0Q54P06Q01WH1*T06PASs=C$LrU()*M*w pV0w62wwvvrllڹsJsZ<]emboٱeZwm$o\=,? endstream endobj 291 0 obj <>stream x3T0P0bc33CBcS 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠`ߣkAkU[7}/ /?~?y}_? wؿzŠ5E endstream endobj 288 0 obj <>stream x3T0P0bcs3CBc3 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠⅗U)?> h;E.i깂S! A endstream endobj 285 0 obj <>stream x3T0P0V5T06S01WH1*T06 (X(Dr.'BIQi*~8PP S!K!A,`c ͥSW bg颠 뱒&?n>ϑYˮOH?c~6`hrT?@c endstream endobj 282 0 obj <>stream x3T0P0Q5T06Q01WH1*T06PA =CHrU()*M*w pV0w62wwvvrllsJsZ<]embo`گYao}[&ݟ.WO@.*c7) endstream endobj 279 0 obj <>stream x3T0P0bc3sCBc 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠]~}K_ֿ+, zd<ߠ9"i \ \O6>stream x3T0P0V5T06U01WH1*T06 (X(Dr.'BIQi*~8PP S!K!A,`c ͥSW bg颠` ٗ^eWZw׫{k{bwݾm׷la6.WO@.?=g endstream endobj 273 0 obj <>stream x3T0P04F& F )\  *\.'O.}O_T.p*.}".}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠AG<:pN"AS! *%f endstream endobj 270 0 obj <>stream x3T0P0T5T06W01SH1*T06 (+Dr.'BIQi*~8PP S!K!A,`c ͥSW bg颠Mm򦬭|ۥ}m!߭ow#B  C endstream endobj 228 0 obj <>stream x35T0P0W5T01V01UH1*T06 ()Dr.'BIQi*~8PP S!K!A,`c ͥSW bg颠Г- {Յ8Xf;7\aԓ;ClmBͬ]+le AS! > endstream endobj 89 0 obj <>stream x3T0P0bcc3CBcC 53T53 S\.'O.}O_T.pR.}J.}g %ڐ KM V8qy(-#嵞ܫc[?|^q.WO@.6 endstream endobj 68 0 obj <>stream x3T0P0bccc#CB# 53T5ɹ\N\ %E\@E\@5\N \.ц\\nn@n.P9?47ΎEAm$ŊgII[V+6mo>Qk~jQI eM0XĠ"}/0 endstream endobj 62 0 obj <>stream x3T0P0bccCCBcC 53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]6M"vL֭, \ \Ð4 endstream endobj 51 0 obj <>stream x3T0P0bc33CBcS 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠BcГ٬AZχV?||̧C01(pz*r>stream x3T0P0bcsCB#kgkg`  ɹ\N\ %E\@\@\N \.ц\\nn@n.P9?47ΎEAm٦]SnW'a_[UX\=4 endstream endobj 44 0 obj <>stream x3T0P0bc3CBcrM t ,T!9ɓKW4K?KK)YK%ڐ KM V8qy(¡j@HoPCB &3 endstream endobj 36 0 obj <>stream x37R0P0f  )\ &@>khgkg`  ɹ\N\ %E\@\@\N \.ц\\nn@n.P9?47ΎEAm˗/֭9Oi+8\F.WO@.I endstream endobj 33 0 obj <>stream x37R0P04fF  )\ A@PA@,Ls<=}JJSÁj=J ] b݀}o]s~Nin^1PN.m?d0󰪰fҭ \ \VI^ endstream endobj 402 0 obj <>stream x36U0P0bcCCCB#s 53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]V?^TF.WO@.1 endstream endobj 398 0 obj <>stream x36U0P0S04R0P04WH1*T02 (**Y*$r9yr{*rp{(r;8+rDrry(\=( endstream endobj 395 0 obj <>stream x36U0P0bc#CCBc53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]L4v[(bj;3 \={4 endstream endobj 267 0 obj <>stream x3T0B]Cc 3CB$hb`gɹ\N\ %E\@E\@5\N \.ц\\nn@n7P7s~Nin^1PڊZ&G)|}ﺯ.p(:"17AS! @ endstream endobj 264 0 obj <>stream x3T0P0Bc abU`l &z z ɹ\N\ %E\@5\@%\N \.ц\\nn@n.P9?47ΎEAmŁTr><{ }8||CWGaˆK̮֠.WO@.u`1 endstream endobj 261 0 obj <>stream x3T0P0R5S02V05RH1*T02 (Z(Dr.'BIQi*~8PP Sme˥{+esJsZ<]V>bx?:PiŠn0 endstream endobj 258 0 obj <>stream x3T0P0bcc#CB#kgk$s<=}JJSÁ=j ] b݀}o]s~Nin^1P“,r—5}_rݱumM;s_uáhk$" \=. endstream endobj 255 0 obj <>stream x3T0P04V54P02S)\  4ddr.'BIQi*~8PKK)YK%ڐ KM jw)+jtQP[s^n8k*VF.WO@.%i endstream endobj 252 0 obj <>stream x3T0P0Q5W0 )\  h`gɹ\N\ %E\@5\ f\N \.ц\\nn@n7D.}ܼb;;.O08pz*r endstream endobj 249 0 obj <>stream x3T0P0U0W06V0TH1*T0()*Y*$r9yr{*rp{p;8+rDrr;;66 @ܼb;;.Ogۗ3|Ҏ{ zHLlm"'PUh>o >20rz*rD1B endstream endobj 246 0 obj <>stream x3T0P0V04Q06U06RH1*@hh`g`hgɥPRTʥTĥTå`ȥme˥{+esJsZ<]vOϬ, \ \T" endstream endobj 243 0 obj <>stream x3T0Cc 3CB$*XBr.'BIQi*~8P)P%S!K!A,`cﭠ q)+jtQP[1Yϔ?u~?1om3TU, \ \> endstream endobj 240 0 obj <>stream x3T0Cc c#CB,b`g "s<=}JJSÁ=j ] b݀}o]s~Nin^1Pڊ9GW[SOdxxRVUY08pz*r_5> endstream endobj 237 0 obj <>stream x3T0P04V5S06R05RH1*T0(Z @"9ɓKW4K?KK)YK%ڐ KM V8qy(-l]dץ_cO}&w÷, \ \"90 endstream endobj 234 0 obj <>stream x3T0P0bcS3CBc 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠Cݓ6|i``r&.WO@.< endstream endobj 170 0 obj <>stream x31Q0PacKc3CBcS 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠aB!~[[) &(oU[mmӳM;̙vM""1AS! 6@ endstream endobj 112 0 obj <>stream x36U0P0aKCCB#s 53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]Q07ۭr)\i9 <&.WO@.4B endstream endobj 109 0 obj <>stream x36U0P0R54S06V06PH1*T06()$dr.'BIQi*~8PPSme˥{s;qy(-X TM-2[f\=خKg;V$V|ʿQ*_]/M&`6 \=@# endstream endobj 104 0 obj <>stream x36U0P0bccCCBc# 53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]Vi3?um? Tz^c/}5(ޣ\Ug`rT>; endstream endobj 101 0 obj <>stream x36U0P0a#KCBcC 53T5ɹ\N\ %E\@E\@5\N @K!A,`cﭠ q)+jtQP[a˾ýˏWk'XC$WmHbP`rTI-f endstream endobj 98 0 obj <>stream x36U0P0bcc#KCBc# 53T5ɹ\N\ %E\@E\@5\N \.ц\\nn@n.P9?47ΎEAmłPOm80?'kﳣ=˿v< .WO@.M:2 endstream endobj 95 0 obj <>stream x36U0P0T54U06R0TH1*T06((%dr.'BIQi*~8PPSme˥{s;qy((?2h>ħɳ__zwBַ {O \ \{;$ endstream endobj 350 0 obj <>stream x35T0Pb3#CBc3 ĵ3T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠`gC@[$&\rO NAYSX\=';k endstream endobj 215 0 obj <>stream x31Q0P04U54R0T04TH1*T0 ( A@29ɓKW4K?KK)YK%ڐ KM jw)+jtQP[BUU{]tСP.߀F.WO@.G( endstream endobj 212 0 obj <>stream x31Q0PacKSCCBcS 53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]a勖Lvk-4M}_-z[w}ҽM wvtbf7\;tc 65p04pz*rMF endstream endobj 209 0 obj <>stream x31Q0P0WеP0T0PH1*T02 ()Dr.'BIQi*~8PP S!K!A,`cﭠ q)+jtQP[q@@U`P}}^CQAYM  \ \݆? endstream endobj 206 0 obj <>stream x31Q0P0bCSCCBcK 53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]V1f3k@W/}Y!/?_Z{/a/Z4\]D1 endstream endobj 203 0 obj <>stream x31Q0P04F f )\ @khgkg`  ɹ\N\ %E\@\@\N @K!A,`cﭠ q)+jtQP۠kk_mennrX\=\' endstream endobj 200 0 obj <>stream x31Q0P0bc3CBc 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠n9C)WÜk+|K;_z0oPCB <3u endstream endobj 197 0 obj <>stream x31Q0P04U54R0T06SH1*T0 (X A@29ɓKW4K?KK)YK%ڐ KM jw)+jtQP[ߺuOe_{ovE=Q XrAS! T3q endstream endobj 194 0 obj <>stream x31Q0P04U02R0T05TH1*T0D M  ,s<=}JJSÁ= ] b݀}o]s~Nin^1Pښ}Z}l쥿"#Vt`aprTm*m endstream endobj 191 0 obj <>stream x31Q0P0bCc3CBcK 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠Bc}S*݊.ݶDn|[u߼/Urorս_M}u/lAlɡ1(0qz*r= endstream endobj 187 0 obj <>stream x35T0P0U5T05P01UH1*@hb`gɹ\N\ %E\@E\@5\N \.ц\\nn@n7P7s~Nin^1PB/F/_~w+@Y1$`K‹ꚾ_o8iW+ڶ|8Ķ!!S! h? endstream endobj 333 0 obj <>stream x3T0P0V5T06U01WH1*T06 (X(Dr.'BIQi*~8PP S!K!A,`c ͥSW bg颠-rrP7nꗏ_ѿyt7l0gbk~ 8\=$? endstream endobj 328 0 obj <>stream x35T0P0U5T01Q01UH1*T0 ()Dr.'BIQi*~8PP S!K!A,`c ͥSW bg颠`/b""}V}׿SK.Ͽu;Fn׭[yt$48\=\;> endstream endobj 310 0 obj <>stream x31Q0P0bSCCBcs 53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]V,gb3kQ')`}r&.WO@.8 endstream endobj 302 0 obj <>stream x31Q0P0bCSCCBcK 53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]V,t]~7tvJރ"}YK08pz*r-iTx endstream endobj 167 0 obj <>stream x31Q0P0bc c3CBc# 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠HHE{.Gm]nSmr<6m'r>NmlDD”R". 6.&.WO@.R3 endstream endobj 335 0 obj <>stream x31Q0P0U54R0P05RH1*T06(($dr.'BIQi*~8PPS!K!A,`c ΥSW bg颠H֥En|u/x?2ku_]#mnbk B C endstream endobj 124 0 obj <>stream x31Q0P0W04S06S02TH1*T0 (**Y*$r9yr{*rp{(r;8+rDrry(?`rT$ endstream endobj 121 0 obj <>stream x31Q0P0R5T01R06WH1*T01PAS3#=C$LrU()*M*w pV0w62wwvvrllڹsJsZ<]\VM8kzM|&<^?|}={۴Üz܇߻ω}Cg$8_^wͬeysHnHM8pЁS! GuG endstream endobj 378 0 obj <>stream x31Q0P0bCS#CBcK 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠pÏ׿l7qAG 'lb>\S! ? endstream endobj 375 0 obj <>stream x31Q0B]CcSCCB$hj`gɹ\N\ %E\@E\@5\N \.ц\\nn@n7P7s~Nin^1Pڊ+J&G);en{'mp.WO@.YE endstream endobj 372 0 obj <>stream x31Q0PT02W06Q05RH1*T02 (K=C#3=K\.'O.}O_T.p,t pV0w62wwvvrllt29y@vv\. j;78l7l_{/AS! -f endstream endobj 369 0 obj <>stream x31Q0P0U5W0P0PH1*T06 (*Dr.'BIQi*~8PP S!K!A,`cﭠ q)+jtQP[U ? ?E%LI_[Z_w} 69l??Y[芸/Y)m۶`drTwO> endstream endobj 365 0 obj <>stream x31Q0P0R5T01V05TH1*T01 ()Dr.'BIQi*~8PP Sme˥{us;qy(O__qk?o zWXkrȬ# \=D endstream endobj 360 0 obj <>stream x33T0P0bS scCBS53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠OO @J `a Z>9t+#B J endstream endobj 343 0 obj <>stream x35T0P0S5T01Q01UH1*T0 ()Dr.'BIQi*~8PP S!K!A,`c ͥSW bg颠 wd2%~]3vg.p(Zt֭mb<" ,\ \3> endstream endobj 319 0 obj <>stream x31Q0P0U5W0P0PH1*T06 (*Dr.'BIQi*~8PP S!K!A,`cﭠ q)+jtQP[qAâUya߲>XPz&o7uZ" \=ӓ? endstream endobj 295 0 obj <>stream x3T0Ccs3CB$*XBr.'BIQi*~8P)P%S!K!A,`cﭠ q)+jtQP[1h̬O*U_I/{T_VM_s}߯]39ְ+#B qa? endstream endobj 292 0 obj <>stream x3T0P0Q54W0P01WH1*T02(($dr.'BIQi*~8PPS!K!A,`c ΥSW bg颠!wtaOP||&*hȬ6$:08pz*rrj8 endstream endobj 289 0 obj <>stream x3T0P0T5T06W01SH1*T06 (+Dr.'BIQi*~8PP S!K!A,`c ͥSW bg颠b򟦞k [OsM_>_~k I endstream endobj 286 0 obj <>stream x3T0P0T5T0P01SH1*T06 (+Dr.'BIQi*~8PP S!K!A,`c ͥSW bg颠Ezrȧu{k^~߿_ە|$\=x@ endstream endobj 164 0 obj <>stream x31Q0P0bcs3CBcS 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠з?V }߶{':&n4 endstream endobj 161 0 obj <>stream x31Q0P0U5W0P0PH1*T06 (*Dr.'BIQi*~8PP S!K!A,`cﭠ q)+jtQP[8kQ_}\ʔ-ʚ.;P endstream endobj 79 0 obj <>stream x3T0Cc c#CB,b`g "s<=}JJSÁ=j ] b݀}o]s~Nin^1Pڊ9iG9\Up?}GIZrS! V6 endstream endobj 63 0 obj <>stream x3T0P0Q5S06Q05RH1*T06PAS =CHrU()*M*w pV0w62wwvvrllt29y@-vv\. j l[>xG(_ B(k  B ~7N endstream endobj 403 0 obj <>stream x36U0P0bc#CB# 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠E??)? 3k)#B 0`0o endstream endobj 399 0 obj <>stream x36U0P0V5T06T01RH1*T0 (+Dr.'BIQi*~8PP S!K!A,`c ͥSW bg颠HENDIiz0orϚ_4u8&ڡ \ \+b9 endstream endobj 396 0 obj <>stream x36U0P0V54U02Q01RH1*T02(+$dr.'BIQi*~8PPSme˥{s;qy(-2W~mJ.uRBw&.WO@.̉5 endstream endobj 283 0 obj <>stream x3T0P0V5T06U01WH1*T06 (X(Dr.'BIQi*~8PP S!K!A,`c ͥSW bg颠 (^wd|*n_~ۓ'WOd컼w}[ozc0hnfVF.WO@.J< endstream endobj 280 0 obj <>stream x3T0P0Q5S06Q05RH1*T06PAS =CHrU()*M*w pV0w62wwvvrllt29y@-vv\. j 7&gz=?:&\mɇBkȝ[]CB (3: endstream endobj 277 0 obj <>stream x3T0P04V54P02S06RH1*gɹ\N\ %E\@I.}*.}gC.}hC.X.}7}gg7 7FK9?47ΎEAmC_pvk_Mt \ hJ/ endstream endobj 274 0 obj <>stream x3T0P0V5T06U01WH1*T06 (X(Dr.'BIQi*~8PP S!K!A,`c ͥSW bg颠A9nUcw]/0w6V] z;iۆCDX\=: endstream endobj 271 0 obj <>stream x3T0P04V02T02S01SH1* )1гTHrU()*Mq{q;8+rDrr;;66 @ܼb;;.O57__[aW`(oPCB J)D endstream endobj 268 0 obj <>stream x3T0P0bc33CBc 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠yW[s_+#B 4 endstream endobj 265 0 obj <>stream x3T0P0V5T06U01WH1*T06 (X(Dr.'BIQi*~8PP S!K!A,`c ͥSW bg颠Q˗/eR_/4?Wwvn&᝴m۶ \=? endstream endobj 262 0 obj <>stream x3T0P0Bc abU`l &z z ɹ\N\ %E\@5\@%\N \.ц\\nn@n.P9?47ΎEAm s3YMp08{r?vo2u`aprTo0 endstream endobj 259 0 obj <>stream x3T0P04F&  )\  A2z @"9ɓKW4K?(ͥ& ] b<]@[S! Qo endstream endobj 256 0 obj <>stream x3T0P0bcS3CBc 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠Ci|￿a衇pݟ>PK5>stream x3T0P0U06T06V01VH1*T0 ()% ,s<=}JJSÁ=j ] b݀}o]s~Nin^1Pڂ' sNDv Ǔ8L\=$# endstream endobj 250 0 obj <>stream x% @D-+JJB6 ."BL:G+S 9z1s- UuQEH<ȉ->[ Z]˞MM_5rQ<,#A endstream endobj 247 0 obj <>stream x3T0P0bccsCB#3 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠 _^{\=N6 endstream endobj 244 0 obj <>stream x3T0P0bc3c#CBc 53T5ɹ\N\ %E\@E\@5\N \.ц\\nn@n.P9?47ΎEAmEU)gﭿ>.;?P[02pz*rz3H endstream endobj 241 0 obj <>stream x3T0P0bc3c#CBc 53T5ɹ\N\ %E\@E\@5\N \.ц\\nn@n.P9?47ΎEAmEl1 YN7._M\=6. endstream endobj 238 0 obj <>stream x3T0P0bc 3CBc3 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠pΜOMvn_jvzuw4Y  \ \e? endstream endobj 226 0 obj <>stream x35T0P0T52R01W01QH1*T01()%dr.'BIQi*~8PPS!K!A,`cﭠ q)+jtQP[%C9_X:}a}o_zWzÜ1oc`{.w?S! P endstream endobj 113 0 obj <>stream x36U0P0a#KCB#s 53T5ɹ\N\ %E\@E\@5\N \.ц\\nn@n.P9?47ΎEAm!ϑ~R͵vvf2NJ7uytL۸\=o+ endstream endobj 110 0 obj <>stream x36U0P0bcc#KCBc# 53T5ɹ\N\ %E\@E\@5\N \.ц\\nn@n.P9?47ΎEDC}>ssVlB 90 endstream endobj 105 0 obj <>stream x36U0Pa#KJ1*T02"z @"9ɓKW4K?KK)YK%ڐ KM V8qy(-V?t:G;=&X`;םǶ}"󉤊qj+20rz*r",j endstream endobj 102 0 obj <>stream x36U0P0bc#KCB#3 53T5ɹ\N\ %E\@E\@5\N \.ц\\nn@n.P9?47ΎEAm!?K?2!_o3km̲<# \=X.< endstream endobj 99 0 obj <>stream x36U0P04FF )\@-z @"9ɓKW4K?\KD88+rDrry(13oz;.WO@.` endstream endobj 96 0 obj <>stream x36U0P04FF F )\ @ghgk$s<=}JJS,=j ] b݀}o]s~Nin^ڎ:ߠ[lNm \=ſ%? endstream endobj 93 0 obj <>stream x36U0P0bccCCBc# 53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]LX{Ώ'~Cɛ$1(0qz*rֹ6 endstream endobj 52 0 obj <>stream x3T0P0bc33CBc 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠S`.?9_u{c_ookﭾ_k__[7OmCW04ppz*rB, endstream endobj 48 0 obj <>stream x3T0P0ac#CBcC 53T5ɹ\N\ %E\@E\@5\N @K!A,`cﭠ q)+jtQP[xɉF}*_{ KӸ 1N>pЁS! s/| endstream endobj 45 0 obj <>stream x3T0P5& F )\ & >kg " s<=}JJSÁ=hC.X.}7}gg7 7Fh8s~Nin^1Pڊs4Qxt(kQŢ5qH8b/=9d&.WO@.c= endstream endobj 37 0 obj <>stream x37R0PP5T03V03QH1*T05 (*XDr.'BIQi*~8PP S!K!A,`c ͥSW bg颠`3GG*_ب/_8`C>Z{=UW_gZ~]ɡӶ}n.p=qCњG:&pPAS! L endstream endobj 34 0 obj <>stream x37R0P0bs3cCB3K 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠)-2le*|b6 __߬V-'55ׁS! _ endstream endobj 351 0 obj <>stream x35T0P0bc3#CBr t ,T!9ɓKW4K?KK)YK%ڐ KM V8qy(-Qaɱ+Q?M9-`bPrT?* endstream endobj 235 0 obj <>stream x3T0P0bcS3CBc 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠;KӸM?/ z1F.WO@.|5 endstream endobj 232 0 obj <>stream x3T0P0V5S06U05RH1*T06 (Z(Dr.'BIQi*~8PP S!K!A,`cﭠ q)+jtQP[s o[|Wf-Vj鯯_~M_/o娿u20rz*rQG endstream endobj 216 0 obj <>stream x31Q0P0b#SCCBcK 53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<].XU@ӞEE~N/~_~i7|>stream x31Q0P04F )\ $l3T5ɹ\N\ %E\@i.}0\ b<]\=} endstream endobj 210 0 obj <>stream x31Q0P04W5W01P0PH1*T02( @"9ɓKW4K?KK)YK%ڐ KM V8qy(t}U!|ⵙAS! e5,| endstream endobj 207 0 obj <>stream x31Q0P04QеP06S0PH1*T02( @"9ɓKW4K?KK)YK%ڐ KM V8qy(-p:TVWu)\ۛ}m믯)U߷oWd1(0qz*r-E>d endstream endobj 204 0 obj <>stream x31Q0P0bcKc3CBcc 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠Q?޼#/ė~6!'W{by۶K7 bnHL8#k62 endstream endobj 201 0 obj <>stream x31Q0P0U0Q0P01SH1*T06()*Y*$r9yr{*rp{p;8+rDrr;;66 @ܼb;;.OJ[(lt(CWGa3C5:Çyoo'd1o}GS! 1 endstream endobj 198 0 obj <>stream x31Q0P0bCSCCBcK 53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]V~>AGZU30rz*r$6 endstream endobj 195 0 obj <>stream x31Q0P0S0W06W01VH1*T06()Br.'BIQi*~8P P Sme˥{+esJsZ<]sDDbnbǗ[d|gdlx]un`hrT7 endstream endobj 192 0 obj <>stream x31Q0P0V04S0& )\ @AC c=CCS=K\.'O.}O_T.p".}.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠UDO `drT4;! endstream endobj 334 0 obj <>stream x31Q0P0bcKSCCBcs 53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]VRx'3kG2k>1`׃ CU?M\=t> endstream endobj 329 0 obj <>stream x35T0PaKCB 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠`9 ͗9,U\%+/<&oPv .WO@.m!A endstream endobj 311 0 obj <>stream x31Q0P0bSCCBc 53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]V>?l}_o#8{wMm`hrTtP< endstream endobj 303 0 obj <>stream x31Q0P0bCSCCBcK 53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]V̑j6{sa W} !??o??.  B &S endstream endobj 188 0 obj <>stream x35T0P0R5T01V05UH1*T0 ()Dr.'BIQi*~8PP Sme˥{us;qy(-xvն?XS2?d|^evvv6A#B .; endstream endobj 308 0 obj <>stream x31Q0P0bSCCBcK 53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]VLoxrNλ_VqƖ~`o(~}+V:02pz*rXC endstream endobj 125 0 obj <>stream x31Q0P0bc S#CBcC 53T53 S\.'O.}O_T.pR.}J.}g %ڐ KM V8qy(-hchkwanxr^a$!ԋAS! D5 endstream endobj 122 0 obj <>stream x31Q0P0bcKSCCBckgkg`  ɹ\N\ %E\@\@\N @%ڐ KM V8qy(X4L/NoPCB Y6 endstream endobj 384 0 obj <>stream x31Q0P0S01T06W05TH1*T06 (e L ,s<=}JJSÁ=jhC.X.}7}gg7 7F[A(㜟SW bg颠؆oC_sTc˖UYAYX\={$ endstream endobj 376 0 obj <>stream x31Q0P0bcsSCCBcr t ,T!9ɓKW4K?KK)YHDrr;;66 @ܼb;;.OO괿 ,4bP`rT58 endstream endobj 373 0 obj <>stream x31Q0P0bCSCCBcK 53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]v^[k׽,-q|x Z~Bk2ʊ]V[k&E endstream endobj 370 0 obj <>stream x31Q0P0bCSCCBcK 53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]VP59Nx#O[2?AS! D endstream endobj 320 0 obj <>stream x31Q0P0UеP0P5RH1*T06 ()Dr.'BIQi*~8P S!K!A,`cﭠ q)+갳tQP[)k  B v endstream endobj 317 0 obj <>stream x31Q0P0bSCCBcK 53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]V,0:g:~b\#ߵkA2c}+nÜ'+sٸ\=q7BA endstream endobj 305 0 obj <>stream x31Q0P0V5T01P05RH1*T06  z@9 ɥPRTʥTĥTå`ȥme˥{us;qy(-9׊7ѮݭuA%~GyM+4wB|]/ݶ?GV4ЍkL7lHXĠ< endstream endobj 159 0 obj <>stream x31Q0P0b#c3CB53T53 S\.'O.}O_T.pR.}J.}g %ڐ KM V8qy(X{˗'Wy__G>stream x36U0P0aC#CB# 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠H-b;|]meَ_{mۛ33CWWGؤqg9ߠ.WO@.2 endstream endobj 400 0 obj <>stream x36U0P0bc##CBcr t ,T!9ɓKW4K?KK)YK%ڐ KM V8qy(-0/]חemߪ'WE|-20rz*r$8 endstream endobj 397 0 obj <>stream x36U0P0R54U06R0TH1*T06PA#=C$LrU()*M*w pV0w62wwvvrllڹsJsZ<]T\Vݺm^tci绗Q.Š@7X endstream endobj 366 0 obj <>stream x31Q0P0Q5T0T05RH1*T06 (+Dr.'BIQi*~8PP S!K!A,`c ͥSW bg颠-r}por&uwi.emvn&᝴mۆC\=bD endstream endobj 361 0 obj <>stream x33T0P0bSsscCBS 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠q˗v&cw}Jܭ^{Bu_Xo}uD5Qe_4\5[8 endstream endobj 296 0 obj <>stream x3T0P0V5T06U01WH1*T06 (X(Dr.'BIQi*~8PP S!K!A,`c ͥSW bg颠Fj'C.MY-zuq@Y L\ \ 4# endstream endobj 293 0 obj <>stream x3T0P0bcc#CBm1R03T5ɹ\N\ %E\@Y.}0`ȥme˥{+esJs󊁊<]6]җK6z.(qVy^zY+r&.WO@.* endstream endobj 290 0 obj <>stream x3T0P0U5T06U01SH1*T06PAsS=CHrU()*M*w pV0w62wwvvrllsJsZ<]Ii?)WلߺuAS! a5K endstream endobj 287 0 obj <>stream x3T0P0bcs3CBc3 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠Mm򦬭1~^|[wUg >]=%= endstream endobj 284 0 obj <>stream x3T0P0bc33CBc 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠ՔͲ[/F@NrUS! f8 endstream endobj 281 0 obj <>stream x3T0P0Q5S06Q05RH1*T06PAS =CHrU()*M*w pV0w62wwvvrllt29y@-vv\. j+ΘqX>stream x3T0P0Q5T06Q01SH1*T06PAs=CHrU()*M*w pV0w62wwvvrllsJsZ<]6 ?HOK;[~3owַmB7֮]c B x:" endstream endobj 275 0 obj <>stream x3T0P0V5T06U01WH1*T06 (X(Dr.'BIQi*~8PP S!K!A,`c ͥSW bg颠̇92bY/8.-t;᠎!7͇ElY//cC.z;eGbv6L \ \#A: endstream endobj 272 0 obj <>stream x3T0P0bcS3CBc 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠b:vj\o? `pk{zJ7 FS! > endstream endobj 269 0 obj <>stream x3T0P0bc3c#CBc 53T5ɹ\N\ %E\@E\@5\N \.ц\\nn@n.P9?47ΎEAm¡.?KGg ;|{i KI|Ϭ%?02pz*r6 endstream endobj 266 0 obj <>stream x3T0P0bcc3CB# 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠U)u! \=_(3< endstream endobj 263 0 obj <>stream x3T0P0bcs3CBc3 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠bRY#~Sw@ùF.&~U, \ \K endstream endobj 260 0 obj <>stream x3T0P04U5S06S05RH1*T02(Z @"9ɓKW4K?KK)Yq62wwvvrllt29y@-vv\. j+]_UG|8\=)+U endstream endobj 257 0 obj <>stream x3T0P0bcsc#CBc3 53T5ɹ\N\ %E\@E\@5\N \.ц\\nn@n.P9?47ΎEAm /9&cd/lW(?W2Pf,&.WO@.ނ: endstream endobj 254 0 obj <>stream x3T0P0V0S06U01PH1*T06EM L,s<=}JJSÁj=J ] b݀}o]s~Nin^1Pf/󌎚YsX  \ \y- endstream endobj 227 0 obj <>stream x35T0P0bS3CCBcK 53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]v O?\rm \=V>stream x36U0P0bc##KCBcr t Dr.'BIQi*~8PP S!K!A,`cﭠ q)+jtQPr5 }ﺯo:08pz*rQ. endstream endobj 111 0 obj <>stream x36U0P0bcc#KCBc# 53T5ɹ\N\ %E\@E\@5\N \.ц\\nn@n.P9?47ΎEDfi?_߰ǯq#6.WO@.2 endstream endobj 106 0 obj <>stream x36U0Pac#KCB#53T5ɹ\N\ %E\@E\@5\N \.ц\\nn@n.P9?47ΎEAm!9 zN/ثK>3nͣmnbk B - endstream endobj 85 0 obj <>stream x3T0P0bcc#CBcrM t Dr.'BIQi*~8PP S!K!A,`cﭠ q)+jtQP[s? {~/ߕQ&t;i۶ s1(0qz*r%1* endstream endobj 67 0 obj <>stream x3T0B]Csc3c#CB$j3I\.'O.}O_T.p*.}".}gC.}hC.X.}7}gg7 7FK9?47ΎEAmŜC6 \ȲP~wջeGJ 8 05$F.WO@.? endstream endobj 61 0 obj <>stream x3T0P0a c#CBcS 53T5ɹ\N\ %E\@E\@5\N \.ц\\nn@n.P9?47ΎEAm Ӈ Ujپ>_azB/V[G<);=넃l \=r1 endstream endobj 50 0 obj <>stream x3T0P0bcs3CBc3 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠b[Cqωl׃_CI ogYtOu`aprTK endstream endobj 46 0 obj <>stream x3T0P0S04Q06R0TH1*T02 (**Y*$r9yr{*rp{(r;8+rDrry(0 sz*r# endstream endobj 43 0 obj <>stream x3T0P0R5P06W06VH1*T06(*$dr.'BIQi*~8PPSr62wwvvrllڹsJsZ<]NVQoh26ĻǗ_`9 c3;yߧ8;/ 9GF=l=m(> C endstream endobj 35 0 obj <>stream x @D-+ !EƘ`ea ډ?_z)ΜhhgsbkkI,*"$yWQG-3HZH}0gH#!4mQȎs5q[eXsvzM6Ub endstream endobj 32 0 obj <>stream x37R0P0Q56Q0T03QH1*T03(XZ($dr.'BIQi*~8PPS!K!A,`cﭠ q)+jtQP[,KF`@¡˝ 5Gq>stream x3T0P0R54W06S06RH1*T06(X*$dr.'BIQi*~8PPS!K!A,`c ΥSW bg颠K*ߔMi=i.nw?~}V}zm׭\ L\ \L@* endstream endobj 248 0 obj <>stream x3T0P0bcsCBcrM t ,T!9ɓKW4K?KK)YK%ڐ KM V8qy(-e:ȾC &ךK_/?*>noҥ@ӥC./ս^8>9dB 683 endstream endobj 245 0 obj <>stream x3T0Cc33CB$*XBr.'BIQi*~8P)P%S!K!A,`cﭠ q)+jtQP[1ic'G;/~(~̲Ksΐ`hrTbK< endstream endobj 242 0 obj <>stream x3T0P0S5S02U05RH1*T0 (Z(Dr.'BIQi*~8PP S!K!A,`cﭠ q)+jtQP[q !zrȷ}~]__+~Q/u5Vt`aprTd9:x endstream endobj 239 0 obj <>stream x3T0P0R54W01P06RH1*T0(X*$dr.'BIQi*~8PPS!K!A,`c ΥSW bg颠̉Q|@|/}o]-uGM?y.WO@.;@; endstream endobj 236 0 obj <>stream x3T0P0Q5T06Q01WH1*T06PA =CHrU()*M*w pV0w62wwvvrllsJsZ<]I (\>$Ņ? /?oo^pmԱmo$$?G~V@+'K*$:08pz*r?> endstream endobj 233 0 obj <>stream x3T0P0bc3sCBc 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠y2r.<>g5gZ.WO@.?p endstream endobj 218 0 obj <>stream x31Q0P0bCc3CBcK 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠╩V?߰M=|Wrm[Ġi9 endstream endobj 217 0 obj <>stream x31Q0P0b#c3CB53T53 S\.'O.}O_T.pR.}J.}g %ڐ KM V8qy(Xɡ5_ߖ20rz*r)< endstream endobj 214 0 obj <>stream x31Q0P0aKc3CBm1S03T53 S\.'O.}O_T.p.}0`ȥme˥{+esJs󊁊<]v]^}KE4K K/K6ʫt`aprTNs3 endstream endobj 211 0 obj <>stream x31Q0P0V5W02S0PH1*T02 (*Dr.'BIQi*~8PP S!K!A,`cﭠ q)+jtQP[j_ᓫ \ \75 endstream endobj 208 0 obj <>stream x31Q0PacKS#CBcS 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠,^GE/rǾ_r_}yUm}ҥ@`˖-U^8>9dB b;F endstream endobj 103 0 obj <>stream x36U0P0bc##KCBcr t Dr.'BIQi*~8PP S!K!A,`cﭠ q)+jtQP[ 0rڊ"7±/bL \ \ +( endstream endobj 100 0 obj <>stream x36U0P5Ʀ F )\ f@k& s<=}JJSÁ= ] b݀}o\9y@-vv\. j+>[7us4wY{? C'ZĠH8M endstream endobj 97 0 obj <>stream x36U0P0Q5S0f )\ Ff@AS#=CHrU()*M*w pV0w62wwvvrllt29y@-vv\. j fz?nk  zrUS! 6b endstream endobj 94 0 obj <>stream x36U0P0aKcsCB#s 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠H'?eMs=> ϾScaB / endstream endobj 332 0 obj <>stream x3T0P0aS3CBc# 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠獭/_VſU_WlkD%//{iOj04ppz*r28Q endstream endobj 205 0 obj <>stream x31Q0P0b#SCCB53T53 S\.'O.}O_T.pR.}J.}g me˥{+esJsZ<]V,0v3k@dC?ꗿ_#AEz`hrT0C endstream endobj 202 0 obj <>stream x31Q0P0U0Q0P01SH1*T06()*Y*$r9yr{*rp{p;8+rDrr;;66 @ܼb;;.O2R,E>;|"mrW>lpLem"&ᘙb ]Z2 B f3I endstream endobj 199 0 obj <>stream x31Q0P0R5T01P05RH1*T0  Fz@9 ɥPRTʥTĥTå`ȥme˥{us;qy(->stream x31Q0P0R5P01T06SH1*T0(($dr.'BIQi*~8PPS!K!A,`c ΥSW bg颠╩xNSM=V~oKlo[VڷVOWLصk:08pz*rD@ endstream endobj 193 0 obj <>stream x31Q0P0bCS#CBcK 53T53 S\.'O.}O_T.pR.}J.}gC.}hC.X.}7}gg7 7F[A(㜟SW bg颠`燸_o_C%|`K_UcVƛQ/01(pz*rG> endstream endobj 189 0 obj <>stream x35T0P0SеP01Q0TH1*T0 (+Dr.'BIQi*~8PP S!K!A,`cﭠ q)+jtQP[dk`o2 D\ziCB >BO endstream endobj 186 0 obj <>stream x35T0CSCB\=C]=K 0UHrU()*M*w pV0w62wwvvrllt29y@-vv\. j v>XtjcLؼEY}k~o<T8J+ ,\ \TM[ endstream endobj 140 0 obj <> endobj 422 0 obj <> endobj 132 0 obj <> endobj 423 0 obj <> endobj 41 0 obj <> endobj 424 0 obj <>stream x]O O zAcp(J<ƸN̰~D #ƪ[ q24*q ,=aM#M|3g۲j* K 턤;N%Zw4J 9K Y1PXC@Kb2{3̢Z endstream endobj 147 0 obj <> endobj 425 0 obj <> endobj 145 0 obj <> endobj 426 0 obj <> endobj 427 0 obj <>stream x]O9 y?(,q(J Ea@}EVbfvgIzk"&"ƪ[pz90]H endstream endobj 127 0 obj <>/FontMatrix[1 0 0 -1 0 0]/FontBBox[0 0 1000 1000]/FirstChar 23/LastChar 126/Widths[ 0 0 0 0 0 0 0 0 0 0 0 44 0 44 0 44 44 44 44 44 0 44 44 44 44 44 44 44 44 44 0 0 44 0 0 44 44 44 44 44 0 44 44 44 44 0 44 44 44 44 44 0 0 44 44 44 44 44 44 44 44 44 44 0 0 44 0 44 44 0 44 0 44 0 44 44 44 44 44 44 44 44 44 0 44 44 44 44 44 44 0 44 44 44 44 44 44 44 44 44 44 0 44 44] /Subtype/Type3>> endobj 428 0 obj <> endobj 39 0 obj <> endobj 429 0 obj <> endobj 362 0 obj <>/FontMatrix[1 0 0 -1 0 0]/FontBBox[0 0 1000 1000]/FirstChar 69/LastChar 88/Widths[ 61 0 0 0 0 0 0 0 61 0 0 0 0 0 0 0 0 0 0 61] /Subtype/Type3>> endobj 430 0 obj <> endobj 130 0 obj <> endobj 431 0 obj <> endobj 38 0 obj <>/FontMatrix[1 0 0 -1 0 0]/FontBBox[0 0 1000 1000]/FirstChar 45/LastChar 109/Widths[ 72 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 72 0 72 0 72 0 0 72 72] /Subtype/Type3>> endobj 432 0 obj <> endobj 143 0 obj <> endobj 433 0 obj <> endobj 434 0 obj <>stream x]10 E7 ЅU!qP(o C-ҷo>LV16܅|B\@XڦͭzU:! uU1z4}mR+sd r)dQ2bSκDALdUK endstream endobj 108 0 obj <>/FontMatrix[1 0 0 -1 0 0]/FontBBox[0 0 1000 1000]/FirstChar 45/LastChar 121/Widths[ 35 35 35 35 0 35 35 35 0 0 0 0 0 35 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 35 35 35 0 35 35 35 35 35 35 35 35 35 35 35 35 0 35 35 35 35 35 35 0 35] /Subtype/Type3>> endobj 435 0 obj <> endobj 436 0 obj <>stream x]n0 @|E H/ݥMӶT;Eېx"/%N9|Ҧ)+HtbA Sʊn~_R)FYִ$y>^:҃1=XQmHV? h@V;'`a*ŝXiD jDQ#+Z;9g8H˧NYqyS[ ee-&+#wV ̪"t(; N'm;nuxeJq Ƀ1Eڇl[⴨ endstream endobj 83 0 obj <> endobj 437 0 obj <> endobj 75 0 obj <> endobj 77 0 obj <> endobj 73 0 obj <> endobj 438 0 obj <> endobj 190 0 obj <>/FontMatrix[1 0 0 -1 0 0]/FontBBox[0 0 1000 1000]/FirstChar 47/LastChar 116/Widths[ 51 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 51 0 0 0 51 51 0 0 51 0 0 51 51 51 51 51 0 0 51 51] /Subtype/Type3>> endobj 439 0 obj <> endobj 128 0 obj <> endobj 440 0 obj <>stream x]O9 y?;N*i\8|bQxA}EVڑ {6R N!RcQX&*SA5KOX7Hz6? 4KADkR F}bǺȫEu9Y1;PKb2{gN|fU3Yz endstream endobj 49 0 obj <>/FontMatrix[1 0 0 -1 0 0]/FontBBox[0 0 1000 1000]/FirstChar 37/LastChar 125/Widths[ 39 0 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 0 0 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 0 39 39 39 0 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39 39] /Subtype/Type3>> endobj 441 0 obj <> endobj 141 0 obj <> endobj 409 0 obj <>stream xeSmL[e7\襽ɲ% #i"Y0M6J{iVtwC䣣TVZ>3,HT,#!nH8un 1YL?h\8m-xs@.n _s\RnܱS/ (1:;P&jƋO-# s:VX) o@E"utvkwB7Ƒq3TQ>m(Qy+n,-vˊ6'+PC"@lR, !LMko3 ĴkHD vbi>mo7(;rDHpGehݩȦUWR5MH d,ArmQ (5h1,,V%F"Jj[ҮV+? en|.ľ*36h 7p @8鵥Ef#I.mL쇄%LY{OA#3=LV.iS> endobj 410 0 obj <>stream x}QLe+ƥ,?>]HB0E9l֌jsd _) Fqb3 PY6@ nD猉щY?cwd̛|?}{$$l7II*E8 I)Жm"Xve%WO^ yrA"/DE6AԱLp5՛kKKȶe!#9Ȁ^&Ĉ4aZ9 ^&$1<(³UUH]jR㗫  $eC@`6V3`%鐮, y䵼@yަU4*7/ ^d]Mˇ$m ~{mTi6$iuVXI訦Q "ĆW ТcM^W4.!@ЂE7f׿,1ߥ[ tOy?vh#^iR0~4AB2AI¤O9AI}Νbgl:Y֕|*yN=Rs]3k::r5]g;IJJ‰M}$I35s^+5-ԥ79qJ5AAc#3궙d$/5nɧ&q,'M'tNAhAهQ:t|DM~T?|ipkJ tAqxޡE+M>z9eƱ=cc`ҫ_VTv2Į=ro$WO;ф(2%Hx"DUqTK(Ì5h`O;YHs4b ݤ> endobj 411 0 obj <>stream xX TڞɌhdjkւֺU+ֺ a HH,$vYպj[ֶn:kmrr|<>'ƎD"opLp<.1a/i"n1'W0&NylǮrS'ɘH|2ɞsvmS%Kxd5 yDl%5<&>!6<.yz;&F  DŽG{S=Z๧"4'xLp5-xczȦʰ-[eQѻbb>9/6ۊ-fa6 l;{ۉy?[`>^l {{[-^a1olۄ=mc[G1O,=b$6E(XABtijN3.8N2Wb ĮqqNHW\ܞɃ]e$FLyvJGkP:aj+vNӺ UkpYn]aKyY xm_?ɻ#{>'2$S}qVt sZ9SAAмO`l ?}"uP]ܥJt._{G](y:`xuv㡫y F"Sjwй8OḇP>:J@} ),ޟ]ʙ"cd|kCA~y欰H-{şԱmm=q|듳ׯJ@_@#;+v*ՠ am8@QW$|e4_[hJQbs ğxGEgT%7>R;P f'ܷo5)$ 6Ho'EBW]8(^o.ABj?8Tdt$Zpon0g SI | \xb?tzhgZP G#1FAВ>(1:/C)~ć 6 b>H)gi'׵iQח g<t~ڭj|)9a1I:ޛst ,*돷6 7RAjWkA /|΄(վCP]򃶛9^yabK&?!&M*ӂ4_}S'-FDz/g&H),#r@'Tkrcn4c~SNkgNp*-m*,34F4İ{ WXA. Ժ\f6o.q՚f:Z[ JZ\ΎTww Q販XMJ^؟j 1R/=`!{\g 8NWȳS4q&$_NK6x]~ʳtȄB ]. ,Rr'ᅿ !B. U8T_Lt\ƭU&] .QKfcY: {kՁI&L#3'G\.3&aE8T+$kβ* KGKSH&D,QS]CC2YTKyUj\b tǣEШ= 3]4C%9P_,4do I~΍Lk&.x%ޚPhJPH'l~ ฺ A-Fһ%g_f,iFu_ 3i]^z?^ APgRb Lhx 7CCL}rlԝejA}($##XqMKϨx{W gE\0T?@1Q=+:qPĭH~~2$z i0&nS=T;ӠW!mWSs;C*:#w?6|wKGߊ8sG(e2,U:\V[2*oq1OfЂ 3r2 ~$pSS Ӆ^#ViZi]1S ' $&5B78Ǜn2V LĔRw7 de0 UC*kh}2*C nz- ߔSt˓ǡr$Ȼ\#72])LgRLdCZlQ:륙2[T}GuZ#g.i^AA*j"2.'%T S~JҟtvWHh#መ{yqԣ)A .V[T|J >:}(ht6w (Ғq://75-Nc${W8N0|(IȈR*x$GǗ9K;xw ^,pbFq!G/ol9Zķr'2KY/b޿njʊ TRGRV[!brWbx9Hq@毨 ~RA|XEa S}h-In#TSAQmޞ4_mrͼ*TDV06dk#mW|Fu=xO4o;~]%pn]pC{7Q#yUyZ;+Ͻ\NUT}fI KKˡ&~Тr@vJ GV~]wa S~5fGdkA`EB[a #_,, ;wSBb!~GnzHJ9`& 2r&l 6h3jw0h5ʍV5,bi6[pTyl>tѡLhqnihP\Z\^6F:p8z}~tto_ί@ u1 ЈpW}LN|21GɁ'18(&'giO})Ts6SNp.׉g8d "m!EVFReBY@n3&6'p(4˫smŵ܈ͨ~((vdPpV^''nGzp;]!én95NЯDDfؿC"A%*L6hX~vp\}o ˉsFĤ#Pe!=ΞYW(Z_ nOְrP˲Ȟ^Vpڠ3Nzh |}Wޅk褦9oǝr>e5 RP@}FTSJ3^jtO; ) FB0JHY=OF_b8K "{B.mӐmȁؐ+fC>_9%RemI-]º-EU1 J@ jY$_Yj7^}A3%mKK_1@{Ua+WKBwՊ;WGm55)>+q jwIDn8r+"K:E10q̗7q HuM|6ǯ\OϤJbZk*~RڭU%wIoKI-py"L)H endstream endobj 148 0 obj <> endobj 412 0 obj <>stream x%AkPk3T' > endobj 413 0 obj <>stream x}QMkAiӦֵjR-o=RoUE% Nwg7afbiI#D6R < v Vo^f}}BcnbU-\e"d%g.%3Amgz k(hg>O_ԯ*9sHϠ8 shp]q! ]"@U o݁2 ,ԁ!$*/JzQ xV\H 9q +ኲ0ҼJSgŀNc.LHUTA&x<XBiƐ<2'?)LC gK%5R\"IC{p"VK1dJ ,sן*I iz2)Z [Bq{,8:Lw<gOOtuKÝ˓ûmo"zς =}Ƈ>?+ ͥ 1d'ַD~9r]jk7Sck/;llg4=:3:}yneP1 endstream endobj 40 0 obj <> endobj 414 0 obj <>stream xz XSIcrNM֡ZZuPgQ@F9@$9@ L<ժJjko>{oZ9{z׻޵^a66oDx{ͷYNkjn'((0f6f̘YцXhcooQqi@AN+W؆ [n&5>z<7zmeV_m?3yeb}ίM|/o3f}6E6P㨍Gxj5zLMP$ʁLmޥSS2ʑZNG9Q+i3NR3ULj55ާP=5ZGͥSR#QPoQVh.ES2^ޠPP*ZDYP0N,)FI&Tj~mkuA|à ] fS7|8?x!7"haç /j^ݢofSض#28H$IrMVc.[ϱ6J6o̴jms킱Sǽ6nW/1L{'Npb70)ȻqSMY_mx(훜x*c-@1G3ҞZo, %PPx ;0]!cSB. ?O.dmpFb璾ih D]qU/i V %PH^(tpIhxgDJtb;CZ#b{TI`3ݚb'4+R|̖\ԨkjTה2/%țn}ERTrףgMHoD'(۟F7,* Z?}@љ $}4x&ǧ'hDruUAITȵ WJjñ6O"]wne}ft|mḰ2+dum[{<8 L r YQJM/^12I-Ñ ^c9O0Oo?X4 zYM~H^Gcz&=Ecѐ_I-LF=<t1ge4钴+ciXSlYMz쀷;ipWXN67)O{,sp_7q[GJ09F0 =īos\ D##GBd|^w:Z}4@9e`6| Wo ؊~h.=p{hD8H7(ѳ\n<,GP=='P"8|M1>ZWĭ՝?%?ČWR\%hKɍ#_[+EhkQqjPӭEP<<65!luf(܄{CQ.0 68m ~`cinh'!Od@Sn'AuC4\еD剝vTNuug<4$#B|hpP\,,%N]zݔ$53(CfFI]˼d3o8IOqI2'R6XeZOH Pa%[gZ8;x xUz_|/%bYL 16hv5Ryv8s(T<&kfmWdItr\Ga*Q_p*^?Ua̚&evqwC7j*v'iH #2}IrE(?%%''ju{6Q۵J-\%֣xbra b'xJqSv7C7Jg N%=`;)#QA"L0k\_-揽6(64[ޞ)Q?N^Wd^q:M֢WyH zn[W11fVl\!I8{-=,<GdֲMGm)#Ib&Y[ :!u9yxC&4μ^zQ{7z 4ԩ5*, RaDOR!T2PeBzJbԆx;Db =GRZgˆnE9KK|TYwǥE=V-EEM:*h@0,,07!0/,80oJ_iHKiZXSuT *QjE,D= 3G~܅^pYAXVo0/H)Ur"N 6%(mqB@BLJx0j* .*Vwkz2uQrV; LoT)3SBMJrAg,hTz+: Kl'$-Qf.6t+8 h(xEܧ Y' ,LSjəf~o.U䨲![EUV7ޠ4( "J :ߣō!4?t+V\Iƫ%ZIF`瑥nU&^&s6HyF)>Eig#'9nY%Bw`.j[P:2=wM:=%h-&̞a|kꔂܬ|=3&E. v[Z̲Fћl;0{7뛌eҪc n&nAm(%z#]V^F H x8^1ƸFD3˕x "E.KMILayC*+[r_R J̉NNazs x #MP'd'M0+Lޕ^!krcٻc%/HK}^4S,Nh >t$:'#M&%vq޺E~j{| aK+Eà ~_~͂jb B$SՍx*hc:Q&8JJHflv la8>p>`[+עԒ"¼Ju TYO, .2 -u1EL,8 $Wh7-nJk0% ZMRPHQӓq#{8W.Бٖs`~<7p-QK>!A21Z'=pߔ2s:ZL6CZN ࢾ㸄;.߈d4Nns~ ,! ;6?'>phQJfuE YȖW6R'is HTD4xgE ^hup،ZE$u H2 [ zT&tZngn A"{x2k6t.1Y"TMw:*Ԧr`,ϊK:J 5$YX Q_w-&iy| IsomPGcL+-j9l54y8Ҝ_ױjUDGh^pݿ c;1t_8 <ђg1+]&'p jH1(0h:^Ss AxY<}`9xnw9& ClBveʋ]jL̨(.?r ZVT/+@楮M098 OœpD Eh> Bq4ƕE;Mǯxއ}}(%+w&W6";A+_ d''v+m'gWltq<*]%Ҕ:jgYUQx)߼@,dpjO5?4wؘm49ӗmL2ijBL/js=.\)~Nm.Ѿ9Ć6jcuŧ *{ G t dxFcrl2ã j~dwê#H9sɚqlKN*H]Cΐ|_,Ft xP(wދ:؜Wc m[ >7I˷<7n! VDK ύ$=g2Z[6u^Gv6 ((0&`6W:b((4`7đM OX feU Ӎ*³I^fD3V37kaNq}@hOߤ0W ~5-*G8i:lF bQy&ڵ.ms;=eL)'܌iYJY7i*;n8޴F^_ŽFs wtvVg3ucϒ)כ&y.Qf50r,}\yms[] 9*,p-+OFn00uZF|&GQ*H8 $)S>1v2T*-dH ??Kf,~Y25Jy;{Ւ8ZuyzJN4XlWb Ӂ]^FtylUWjlA X۵/iт?#BlFP6fנ&47BSo;IqSުhR䥩c%J}b!~N(FٵEZk≜"_ͩX[5SUhUE-r$hp_~xQaN}Nb40K܄yD("y_S*4ؗ9K!5״L]* K2#i+9)TqggMa70Oϭ]yҝ_Wn--1ĵUQ{kpAasٮ*Allֳd[ݒ` c<;E#F#ێ2։r*1~V`1Iv^4gMVSFY^s 4-yr Nq_\ACZE?ӽU2pRk~R@;!MP0CHu*BVvszy~<֛hb-7ok;zY]1~0X|sй86<{Xk#XUf;[koќ߬N}QJW]6qG6PCHtUxva!~7L$o}x03D]0l[L#[D*帋mݢ7^s k MC F  C|?(߿&PSg 4I4f3I1AhɣSť,YiJa[.HVuo0-O{}?{)EF#FXY E#2f.>;nyahUD]%:Aسi|EQz7WBo6͡"H0t0EY!N&7 帗os v p>|L֢]*\.Eqj||Z HpN+]pT9E7FTҖm)`G #:lGH-]|=3kѽg*wIeN4AsPnn>DMX<4 П ,PiǞr@앐:?yQQQ8=xQpW$} _et2Zm|R?y#pћ8WCL:j +[ NPMg X$!'AOG@^H7%-Y&7_z\!bϨjF*틦a{bbKlԢiyV$I#jW5A(sv6)z$t|3|%;KxKh֠`n.QfP$> endobj 415 0 obj <>stream x} P%N1T;`MS{qH\c&6~ =]kW!t@BBB ^0~`ۓv5qitt2tۙδ޹?|g!QFreYd5;%Sryt6>8ZID +de~Nϟ>(+%E!2t9tzzJBw^!)V1zC$"7h7.74T B EڇlG TąS"ZXh2 k ՂV2Z /`j 4Dh UcZF)#|B:5шV$R `7hP^د$ pN)9I3j=F1GE!d5IB4BV 75ڠCg#hAk E4MJ+HQzl)#zTkp^MuE`2T !JKҭ1Fʂe$IaF1ACa4)7 Q(S +T$UtJHS1f:rAnu@^'Z! (C4 U5%\ʺW-ꮳߋkI3~f@xoѕ'ïȯ 4a(gEs3gY8W| 6{م~5hʼn:hHI- f^=71 Р#oͪf߰Wa{ptjWqe}QTISHpp]d\a-1L :oT|nV`8/TO$/jg<7LMo^ i->$2y񉷮Ed3)w=l봏 @a_@Hxn|γLg\,rr_|7!7nsvung ގ+0+-S 㞄.#jz0:9% "a?<[ ϛMɥuZfqL&:GnNGƀooIgT?}~,>?ηOgU&Nm6<_h# ]y_]lEmdӸ$~~nTZrB.39돺{]詓:^EO:ҚJK}1{e ]5!{?Ǘ\svе-m{nrpaSK%UH <' F7:WH_:6m6ǭUSٛ6*B1a5N\|׈~@> endobj 416 0 obj <>stream xesint10%MM  ~<73X_Generated from MetaFont bitmap by mftrace 1.0.20, http://www.cs.uu.nl/~hanwen/mftrace/ esint10}~8|l{k\VkY؄֏TUUі؍~zpgwxxPLwfYR?%~1pTUyUD>nwyjPuc}Yng{ʚʣǔWr5ơxsZbXi_saV.rq$JQY[iT|L|LsOss{w}y~ ē ď endstream endobj 84 0 obj <> endobj 417 0 obj <>stream xUktSUi .iQZ#/):BAS BH+:hk<-/,[..;+Sx2zZ@eSK>EijmIh7ntԴR)>2/e ς\TQjg. I9?!eŠ '\ߖ耳'g/hԈ.hґ%# `.,gai zf }\,xCG=z,ہk:hfLv[,z2ҙP;`5ށ,H˻`]S6g UmXG ?t :dwD;Q{ÄM8kIϩJOJ2C)ą쁬_{6¼hڔt/0Q]#Abc"Bx#E)q?4Ї,ZQ0 jڌ~<1 3!O 9>mR*ھ>6Ž\#o>n;!t0+{6׈ _.Uʬ6b4{or$i ^(aNHx|vqk@vp E0d*)*ŽCC H׷<8@$ɽjQD~^q <ŠMn va ~H1G}4 1'` Xaz[˂;- _@Yqc231>;Q7;i~}Iop˰: -\搹)$x=bq.?,'1FH'_DJ[!pt[q' brS=W* &g$ x YM~,F=F 4d5B^tPpNZUvs (MnvXM0lq=gY!(> |.d:$5N]q!1#' _&U16Zo_qq{e6r: ^j qJr:l:`z[sO<-Zܨ֎_A$rxF͏G{W6xOw$ f6z-یb0Pb3)Hr\0U 6TqBiL^Frh*]C3eߖx(~/Z%Fn♀EhjR K$ O9lW/g|;(k3Wշtpo#O##G6nZ[--_g#<yۏ^>ji1W_FLbCY#(dlo4G6l[Ww mz L?S˂`8N |98'w>]pc]/bTDԍXj l`q̦f2lcN83uz7Npz!~ՙ^ScerRQyxpd*zښVOVlY>$ ?âtQY2eqȸIf^ޜ,9xނgGFsHqۂfGFϱx5h%Z$h:h俄kct=/.T\"W$<)|GbC`VIz igEPC@v1όNfc%,K[22,=\7GDI) pz:iwPn endstream endobj 76 0 obj <> endobj 418 0 obj <>stream xM]HSqgg;sn~p m&6&a0C+b͓[M3eƔhMl]vn&B袲n3"-E}xy0 o~Ƞ;YJA,L`cf|oW ur2Lt0 !IkƖfpE8` *t HEq[â4,xZC4p^e_ 2kBB8(}Bg,* Qعͷ3$ƅXB6O͵#H Vb#\|I'T3}LJ`>yWc ~/@Uto ,_ uF[?]'٣2?N[xv+/(ٌo> endobj 419 0 obj <>stream x}X tSeN,I/WƤ"NR kMפifmr76M -ݠ .#(3zϙ=sO]}ycbٳcYyY{OQsoѡVOp`f̜qЗQlV]j['ɗf23C`zL{%حY&FQ=&H**QJ,u!qLVѻ[ko5g9?rc,c:u_zGO|L+ W$iJ7V$(EuE|P`Ԗ3V,^Ohv4!v"HB~LXY8Yr}*`Z_k'PR 5D@S]!Ir4*Im2cF*'CiHpK N>dmYBXlF\ԧ"'%N.3WQN=Z`Q;!H#,:q1ۊ[9| pLl0!UcԲ"u ?o}Jn+ⶺTz|% ?ӷhP]/T $lɳ*8sLs1*Q^pf h:sg&/W8yrx' EJV{WÉ{iT1<Ʃ')1_Ζsx!_J-d„bz&Ē|YƾƮ>@߱Shn(3ʡe+}M~,5TI@+4 4UZ;XV/٥qYq_ܱ+0eUMv4#aAEXU7_7|*Ǩ4&O[gp,1rMǽk5 r @(ңj^}H=:r5TkZTX5@Bh{^_=1s D˨rܯ> qSO{oʑgPٌvU{  0HcQ]D_ؓPaK̸A%x|ZGnj~Xss˶kr4e=Qߪ olPkv_P!tZTډW#t&IUf݇fo om*wHdҸ21smvT -܇b->h'te{#6ڀufʤS*'>֧sM@7_n{xB3שF-Lu]p[| 8kD{&?c tdp`_TH/bOhPOipOA}&n&$UQč7ѦԵ_=6 g[舍[s [9xě׾Eh 9x"Z_Qʹ١,')31P(Y|Q<-؋ZT ʹ hZa @W:O+hNTPNj E;JJ݊WL[ї7=t\d#!)o0z[t:{k`1a ̮aJ|BJA2`愵2nkYbhn*l fcqYMzf[DfZĽ҄3:m RH Hc5sJq;p':q=b"C)ij6ot15s=NfDU.H]v)Ҥ=y`=\^0{) iu@T*W,Iibq|#9-9V7f}m{QpF^0?, C61yg; יWz+$Z? ;ro|Bfl)R?O+%3ij6©\W"gBƭҵx:^#2@)Q r4 #FU8O Վ#uD 8=_Jh2ԔW3Ҁ!O epא AOʅdIy~"rU.HQ`'9LGɼ 滉l~k]81joy^C p['PW,I+HʒI,9Vw！qG?r4dUkӌ">10}9>8q\&|nn|2'I٨}C6tra+^]w-Wg~_1Ȍ`9g,H ggd d endstream endobj 74 0 obj <> endobj 420 0 obj <>stream x{pSUs)Da+3Fay B@6[6i &}$iI4> -Mi 츺`sٽEٙ;gw]5y A2!%[񂉽EHjޤLtb\<^<%Og4>ԼKfϘL$uHGrEE#cȗ?oOЪpž,XmR% `+WXqrOR*<_MEZ!0J&6|,VUB9s'iێ{RFQĺ52BL./`Jj/J˰ILgRK +K0B*IYc8IUBDeJZ$f31&Uvp”*,s&L'N )@ 19)R1J+HRuŘ' PeR*^ jWHSBH.&P%'^( BkU;K}"rɲ n>(_$T ĥE§T2%TBH*%eӡPNȄ)!!å¢"!ʐ+*:j%R+bCH=@ q#ċ?``Ś*c}&4!dÔ|?ux݊~u1ml|f6;֩4z-p|Ϗsyev>4tƿ~-.ShHCՀT 'PTZq=fq,~shFVK5=(ǧ}9QԖ@'ȋ([O0y.zW7u`V ETg<Җ@)"3E*u~:Tƅ ~9+V}ХMhpyǾwvd[6gsUnSPew2qvę? mM&{#{CNmf}' ,*֝;w O32H>MIj[$ ZʉZ D̍fOm[nq`o/ 6{r|tq_G|voa-n-Ur~Q.@eđm 0j^IYk#I~g5 :mvV6FZ &Cʞ[qwK0+ >Ҿĝ{bN?_YWQ)eQBSHUfNN|\ XmqGG.޳ΠPZ|,Es %ź"*LނSጡAw6Fé]Hx GR7SFajSOKXXnz7N*7&ai$|SrP<~m۰4 r@V*=X@i_V߽fH^8 iwչ[xY8(RչMrPN&amDر3n Y;PJVVZ) y{ )<{N{o''nЌ S5cKsإP**,&_c`=Ȳ?.7U :G zI0Tj| T.׬ʑ^ Qv68<}bBcJE9>WThI/? {}-U$>~ÑɪrPtvCUՆuD0;3jxZ[ƆRVJ2k*]5wIuiVuz:x- 6u6:߂=]]C\Z-G+T%bQ WBuSAxK":st"k'F/ܮA>%-4q 5o }:NI(m8ϱ9 >A[> \{(3+K$W7X,ux2-UjA]jFPo/UTJuc㘃=@ R4Nkf&Q7u''M\O N=i9tf N Q4(\ gIpٲU@6=Q4{g}=ifc_~=7W);Iг8VR8u>I(J!@^{Hp7q^l_@OȲs+w T @ec5U܌6Y }ӑVޝWFf3QooCv?ޒ B/~G{ 19g\y`ƍ>37wW g>HM"pt-ZtMϡ=Cutfgg2%61c8D#pڑ󼣗/ŝ.f#,- endstream endobj 129 0 obj <> endobj 421 0 obj <>stream xR}lunw@HӬ`h|,Š(qL@, 1֞jJ׍u[K{oۭh(6 E&$1Q&'ׁJ/}yS`y9B(5Ru5a`S4O,T3sY Ufng&E* TKTvo(^&^+88a9kw;h'璕4efIl(!.X`⹰Tx])鴙([(rڍJF9 ԋ! hf.te1$o0MBuGEp3#G#}m~RJI{!*_зEna0!PR ;'?7g㧷D6pP#puJf( #:nOO[`n|#Kж֊_ -RN3h Ep۱t4jR"?CS$F_ ?IOeiJ=2 «[C'Ovk;®T]ꎝϊ͇{$Ь~EoIOY,qĴĴ} S6{xsP\0Wkn߰]K'b8H{ќڡ]/Z~=[aTGoX3nh1}~M~M *x)͗^XE2+f*4hiDK/J/\BjNkAqޅY3Od{>>'Y?n#ɮcQ?T/.X'~/=';ijF%ZzʈzÉBu<=,b]1'9Ş#tamגII P@V>3{dfnR!ΓEp)'x|RNCDsܮCkRz,uh? qЅx"KE),T>l 2hONxMVT% mBtHt$ķ54C}!+]*a!|Կ'qo|W2j<9Ԣ)eJu/z"u endstream endobj 28 0 obj << /Title(\376\377\000I\000n\000s\000t\000a\000l\000l\000a\000t\000i\000o\000n\000 \000a\000n\000d\000 \000c\000o\000m\000p\000i\000l\000a\000t\000i\000o\000n\000 \000o\000f\000 \000M\000E\000X\000 \000c\000o\000d\000e) /Dest/section.3 /Parent 6 0 R /Prev 12 0 R >> endobj 5 0 obj <>endobj 2 0 obj <>endobj xref 0 442 0000000000 65535 f 0000059721 00000 n 0000183388 00000 n 0000059533 00000 n 0000057028 00000 n 0000178874 00000 n 0000059665 00000 n 0000060579 00000 n 0000059845 00000 n 0000060000 00000 n 0000060188 00000 n 0000060392 00000 n 0000064019 00000 n 0000061182 00000 n 0000060876 00000 n 0000061014 00000 n 0000061919 00000 n 0000061417 00000 n 0000061555 00000 n 0000061736 00000 n 0000062558 00000 n 0000062252 00000 n 0000062390 00000 n 0000063149 00000 n 0000062843 00000 n 0000062981 00000 n 0000063714 00000 n 0000063449 00000 n 0000178592 00000 n 0000000015 00000 n 0000004209 00000 n 0000064170 00000 n 0000122352 00000 n 0000087225 00000 n 0000108200 00000 n 0000122152 00000 n 0000086989 00000 n 0000107910 00000 n 0000134454 00000 n 0000132704 00000 n 0000152517 00000 n 0000129531 00000 n 0000144492 00000 n 0000121876 00000 n 0000086775 00000 n 0000107675 00000 n 0000121701 00000 n 0000086551 00000 n 0000107439 00000 n 0000139745 00000 n 0000121449 00000 n 0000086309 00000 n 0000107180 00000 n 0000064212 00000 n 0000064346 00000 n 0000064480 00000 n 0000064612 00000 n 0000064749 00000 n 0000064886 00000 n 0000065023 00000 n 0000065160 00000 n 0000121207 00000 n 0000086079 00000 n 0000100778 00000 n 0000065292 00000 n 0000065429 00000 n 0000065570 00000 n 0000120958 00000 n 0000085836 00000 n 0000065712 00000 n 0000065849 00000 n 0000065990 00000 n 0000066132 00000 n 0000138123 00000 n 0000173835 00000 n 0000137472 00000 n 0000167498 00000 n 0000137742 00000 n 0000168467 00000 n 0000100557 00000 n 0000066274 00000 n 0000066410 00000 n 0000066551 00000 n 0000136699 00000 n 0000164339 00000 n 0000120722 00000 n 0000066693 00000 n 0000066829 00000 n 0000066970 00000 n 0000085608 00000 n 0000067112 00000 n 0000067249 00000 n 0000067391 00000 n 0000106949 00000 n 0000126317 00000 n 0000092253 00000 n 0000106739 00000 n 0000126101 00000 n 0000092015 00000 n 0000106575 00000 n 0000125867 00000 n 0000091778 00000 n 0000106345 00000 n 0000125643 00000 n 0000091530 00000 n 0000106113 00000 n 0000120496 00000 n 0000067523 00000 n 0000135417 00000 n 0000091257 00000 n 0000105894 00000 n 0000120279 00000 n 0000091030 00000 n 0000105663 00000 n 0000120058 00000 n 0000067686 00000 n 0000067859 00000 n 0000067892 00000 n 0000057391 00000 n 0000004230 00000 n 0000011181 00000 n 0000096840 00000 n 0000112959 00000 n 0000079760 00000 n 0000096664 00000 n 0000112732 00000 n 0000079508 00000 n 0000130995 00000 n 0000139289 00000 n 0000176884 00000 n 0000133958 00000 n 0000161273 00000 n 0000129141 00000 n 0000143063 00000 n 0000068015 00000 n 0000068163 00000 n 0000068310 00000 n 0000068453 00000 n 0000068595 00000 n 0000068747 00000 n 0000128552 00000 n 0000141528 00000 n 0000068899 00000 n 0000134935 00000 n 0000163502 00000 n 0000130502 00000 n 0000151634 00000 n 0000130244 00000 n 0000150859 00000 n 0000069038 00000 n 0000069071 00000 n 0000057629 00000 n 0000011203 00000 n 0000019465 00000 n 0000069263 00000 n 0000069400 00000 n 0000069539 00000 n 0000069678 00000 n 0000069815 00000 n 0000114803 00000 n 0000080750 00000 n 0000100338 00000 n 0000069954 00000 n 0000080492 00000 n 0000100102 00000 n 0000070093 00000 n 0000079250 00000 n 0000096152 00000 n 0000070232 00000 n 0000070371 00000 n 0000090780 00000 n 0000070510 00000 n 0000070649 00000 n 0000070788 00000 n 0000070927 00000 n 0000071066 00000 n 0000071205 00000 n 0000071238 00000 n 0000057923 00000 n 0000019487 00000 n 0000025946 00000 n 0000071417 00000 n 0000071556 00000 n 0000071695 00000 n 0000071834 00000 n 0000071973 00000 n 0000128305 00000 n 0000094893 00000 n 0000112237 00000 n 0000128083 00000 n 0000138692 00000 n 0000094638 00000 n 0000111043 00000 n 0000127827 00000 n 0000094414 00000 n 0000110792 00000 n 0000127562 00000 n 0000094179 00000 n 0000110571 00000 n 0000127279 00000 n 0000093945 00000 n 0000110321 00000 n 0000127027 00000 n 0000093729 00000 n 0000110074 00000 n 0000126787 00000 n 0000093473 00000 n 0000109829 00000 n 0000125380 00000 n 0000093229 00000 n 0000109620 00000 n 0000125169 00000 n 0000092946 00000 n 0000109450 00000 n 0000124938 00000 n 0000092726 00000 n 0000109191 00000 n 0000124709 00000 n 0000124478 00000 n 0000080242 00000 n 0000072112 00000 n 0000072247 00000 n 0000072280 00000 n 0000058153 00000 n 0000025968 00000 n 0000030200 00000 n 0000105390 00000 n 0000119841 00000 n 0000085337 00000 n 0000072409 00000 n 0000072546 00000 n 0000072685 00000 n 0000108924 00000 n 0000124229 00000 n 0000090536 00000 n 0000108703 00000 n 0000123965 00000 n 0000090294 00000 n 0000105132 00000 n 0000123709 00000 n 0000090076 00000 n 0000104907 00000 n 0000123464 00000 n 0000089839 00000 n 0000104676 00000 n 0000123214 00000 n 0000089630 00000 n 0000104453 00000 n 0000122966 00000 n 0000089385 00000 n 0000104252 00000 n 0000122709 00000 n 0000089188 00000 n 0000104032 00000 n 0000119628 00000 n 0000088975 00000 n 0000103777 00000 n 0000119389 00000 n 0000088742 00000 n 0000103610 00000 n 0000119181 00000 n 0000088533 00000 n 0000103377 00000 n 0000118928 00000 n 0000088299 00000 n 0000103121 00000 n 0000118715 00000 n 0000088073 00000 n 0000102900 00000 n 0000118471 00000 n 0000085094 00000 n 0000102680 00000 n 0000118225 00000 n 0000084877 00000 n 0000102418 00000 n 0000117950 00000 n 0000084614 00000 n 0000102188 00000 n 0000117704 00000 n 0000084368 00000 n 0000101954 00000 n 0000117468 00000 n 0000084135 00000 n 0000101700 00000 n 0000117231 00000 n 0000083875 00000 n 0000099850 00000 n 0000116986 00000 n 0000083645 00000 n 0000099598 00000 n 0000116764 00000 n 0000083387 00000 n 0000099365 00000 n 0000116544 00000 n 0000083140 00000 n 0000099113 00000 n 0000116310 00000 n 0000072824 00000 n 0000072857 00000 n 0000058359 00000 n 0000030222 00000 n 0000034296 00000 n 0000095904 00000 n 0000111980 00000 n 0000079011 00000 n 0000114542 00000 n 0000082870 00000 n 0000072997 00000 n 0000112473 00000 n 0000073136 00000 n 0000095686 00000 n 0000111742 00000 n 0000078733 00000 n 0000073272 00000 n 0000073411 00000 n 0000073547 00000 n 0000073686 00000 n 0000114288 00000 n 0000082630 00000 n 0000098871 00000 n 0000114089 00000 n 0000073820 00000 n 0000073959 00000 n 0000074098 00000 n 0000074131 00000 n 0000058605 00000 n 0000034318 00000 n 0000039535 00000 n 0000095426 00000 n 0000111498 00000 n 0000074210 00000 n 0000074347 00000 n 0000126546 00000 n 0000095163 00000 n 0000111253 00000 n 0000096410 00000 n 0000080007 00000 n 0000074486 00000 n 0000074625 00000 n 0000074658 00000 n 0000058811 00000 n 0000039557 00000 n 0000044256 00000 n 0000098619 00000 n 0000074822 00000 n 0000074959 00000 n 0000074992 00000 n 0000059001 00000 n 0000044278 00000 n 0000050591 00000 n 0000092497 00000 n 0000108466 00000 n 0000075156 00000 n 0000075295 00000 n 0000075434 00000 n 0000075467 00000 n 0000059199 00000 n 0000050613 00000 n 0000057006 00000 n 0000082341 00000 n 0000098376 00000 n 0000116036 00000 n 0000133619 00000 n 0000075659 00000 n 0000082104 00000 n 0000098117 00000 n 0000115773 00000 n 0000081878 00000 n 0000075805 00000 n 0000097823 00000 n 0000113857 00000 n 0000081640 00000 n 0000097604 00000 n 0000113604 00000 n 0000081498 00000 n 0000097375 00000 n 0000113392 00000 n 0000081229 00000 n 0000097128 00000 n 0000075993 00000 n 0000076125 00000 n 0000076257 00000 n 0000076388 00000 n 0000076519 00000 n 0000113173 00000 n 0000076650 00000 n 0000076835 00000 n 0000076966 00000 n 0000077139 00000 n 0000077270 00000 n 0000077461 00000 n 0000077592 00000 n 0000077781 00000 n 0000077913 00000 n 0000078112 00000 n 0000087832 00000 n 0000101471 00000 n 0000115529 00000 n 0000087659 00000 n 0000101217 00000 n 0000115286 00000 n 0000078243 00000 n 0000087445 00000 n 0000100993 00000 n 0000115036 00000 n 0000080979 00000 n 0000078427 00000 n 0000078606 00000 n 0000078639 00000 n 0000141867 00000 n 0000143317 00000 n 0000144862 00000 n 0000151082 00000 n 0000151854 00000 n 0000153060 00000 n 0000161542 00000 n 0000163736 00000 n 0000164773 00000 n 0000167769 00000 n 0000168782 00000 n 0000174133 00000 n 0000177126 00000 n 0000128926 00000 n 0000129396 00000 n 0000130006 00000 n 0000130415 00000 n 0000130658 00000 n 0000130746 00000 n 0000132346 00000 n 0000133518 00000 n 0000133861 00000 n 0000134307 00000 n 0000134819 00000 n 0000135087 00000 n 0000135185 00000 n 0000136136 00000 n 0000136329 00000 n 0000137137 00000 n 0000138498 00000 n 0000139152 00000 n 0000139504 00000 n 0000141179 00000 n trailer << /Size 442 /Root 1 0 R /Info 2 0 R /ID [<49124F58990E02D8C81D3270985B5CFF><49124F58990E02D8C81D3270985B5CFF>] >> startxref 184293 %%EOF