bayespy-0.4.0/0000755000175000017500000000000012615647103014712 5ustar jluttinejluttine00000000000000bayespy-0.4.0/PKG-INFO0000644000175000017500000001316612615647103016016 0ustar jluttinejluttine00000000000000Metadata-Version: 1.1 Name: bayespy Version: 0.4.0 Summary: Variational Bayesian inference tools for Python Home-page: http://bayespy.org Author: Jaakko Luttinen Author-email: jaakko.luttinen@iki.fi License: MIT Description: BayesPy - Bayesian Python ========================= BayesPy provides tools for Bayesian inference with Python. The user constructs a model as a Bayesian network, observes data and runs posterior inference. The goal is to provide a tool which is efficient, flexible and extendable enough for expert use but also accessible for more casual users. Currently, only variational Bayesian inference for conjugate-exponential family (variational message passing) has been implemented. Future work includes variational approximations for other types of distributions and possibly other approximate inference methods such as expectation propagation, Laplace approximations, Markov chain Monte Carlo (MCMC) and other methods. Contributions are welcome. Project information ------------------- Copyright (C) 2011-2015 Jaakko Luttinen and other contributors (see below) BayesPy including the documentation is licensed under the MIT License. See LICENSE file for a text of the license or visit http://opensource.org/licenses/MIT. * Documentation: * http://bayespy.org * `PDF file `_ * RST format in ``doc`` directory * Repository: https://github.com/bayespy/bayespy.git * Bug reports: https://github.com/bayespy/bayespy/issues * Mailing list: bayespy@googlegroups.com * IRC: #bayespy @ `freenode `_ * Author: Jaakko Luttinen jaakko.luttinen@iki.fi * Latest release: .. image:: https://badge.fury.io/py/bayespy.svg :target: https://pypi.python.org/pypi/bayespy * Build status: .. image:: https://travis-ci.org/bayespy/bayespy.png?branch=master :target: https://travis-ci.org/bayespy/bayespy/ * Unit test coverage: .. image:: https://coveralls.io/repos/bayespy/bayespy/badge.png?branch=master :target: https://coveralls.io/r/bayespy/bayespy?branch=master Similar projects ---------------- `VIBES `_ (http://vibes.sourceforge.net/) allows variational inference to be performed automatically on a Bayesian network. It is implemented in Java and released under revised BSD license. `Bayes Blocks `_ (http://research.ics.aalto.fi/bayes/software/) is a C++/Python implementation of the variational building block framework. The framework allows easy learning of a wide variety of models using variational Bayesian learning. It is available as free software under the GNU General Public License. `Infer.NET `_ (http://research.microsoft.com/infernet/) is a .NET framework for machine learning. It provides message-passing algorithms and statistical routines for performing Bayesian inference. It is partly closed source and licensed for non-commercial use only. `PyMC `_ (https://github.com/pymc-devs/pymc) provides MCMC methods in Python. It is released under the Academic Free License. `OpenBUGS `_ (http://www.openbugs.info) is a software package for performing Bayesian inference using Gibbs sampling. It is released under the GNU General Public License. `Dimple `_ (http://dimple.probprog.org/) provides Gibbs sampling, belief propagation and a few other inference algorithms for Matlab and Java. It is released under the Apache License. `Stan `_ (http://mc-stan.org/) provides inference using MCMC with an interface for R and Python. It is released under the New BSD License. `PBNT - Python Bayesian Network Toolbox `_ (http://pbnt.berlios.de/) is Bayesian network library in Python supporting static networks with discrete variables. There was no information about the license. Contributors ------------ The list of contributors: * Jaakko Luttinen * Hannu Hartikainen Each file or the git log can be used for more detailed information. Keywords: variational Bayes,probabilistic programming,Bayesian networks,graphical models,variational message passing Platform: UNKNOWN Classifier: Programming Language :: Python :: 3 :: Only Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Development Status :: 4 - Beta Classifier: Environment :: Console Classifier: Intended Audience :: Developers Classifier: Intended Audience :: Science/Research Classifier: License :: OSI Approved :: MIT License Classifier: Operating System :: OS Independent Classifier: Topic :: Scientific/Engineering Classifier: Topic :: Scientific/Engineering :: Information Analysis bayespy-0.4.0/setup.py0000644000175000017500000000625012615643655016437 0ustar jluttinejluttine00000000000000#!/usr/bin/env python ################################################################################ # Copyright (C) 2011-2015 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ # Read version number __version__ from the file # See: https://packaging.python.org/en/latest/single_source_version/#single-sourcing-the-version import os version = {} base_dir = os.path.dirname(os.path.abspath(__file__)) with open(os.path.join(base_dir, 'bayespy', 'version.py')) as fp: exec(fp.read(), version) __version__ = version['__version__'] NAME = 'bayespy' DESCRIPTION = 'Variational Bayesian inference tools for Python' AUTHOR = 'Jaakko Luttinen' AUTHOR_EMAIL = 'jaakko.luttinen@iki.fi' URL = 'http://bayespy.org' LICENSE = 'MIT' VERSION = __version__ if __name__ == "__main__": import os import sys python_version = int(sys.version.split('.')[0]) if python_version < 3: raise RuntimeError("BayesPy requires Python 3. You are running Python " "{0}.".format(python_version)) # Utility function to read the README file. # Used for the long_description. It's nice, because now 1) we have a top level # README file and 2) it's easier to type in the README file than to put a raw # string in below ... def read(fname): return open(os.path.join(os.path.dirname(__file__), fname)).read() from setuptools import setup, find_packages # Setup for BayesPy setup( install_requires = [ 'numpy>=1.8.0', # 1.8 implements broadcasting in numpy.linalg 'scipy>=0.13.0', # <0.13 have a bug in special.multigammaln 'matplotlib>=1.2.0', 'h5py' ], packages = find_packages(), package_data = { NAME: ["tests/baseline_images/test_plot/*.png"] }, name = NAME, version = VERSION, author = AUTHOR, author_email = AUTHOR_EMAIL, description = DESCRIPTION, license = LICENSE, url = URL, long_description = read('README.rst'), keywords = [ 'variational Bayes', 'probabilistic programming', 'Bayesian networks', 'graphical models', 'variational message passing' ], classifiers = [ 'Programming Language :: Python :: 3 :: Only', 'Programming Language :: Python :: 3.3', 'Programming Language :: Python :: 3.4', 'Development Status :: 4 - Beta', 'Environment :: Console', 'Intended Audience :: Developers', 'Intended Audience :: Science/Research', 'License :: OSI Approved :: MIT License', 'Operating System :: OS Independent', 'Topic :: Scientific/Engineering', 'Topic :: Scientific/Engineering :: Information Analysis' ], entry_points = { 'nose.plugins': [ 'warnaserror = bayespy.testing:WarnAsError', ] }, ) bayespy-0.4.0/bayespy/0000755000175000017500000000000012615647103016366 5ustar jluttinejluttine00000000000000bayespy-0.4.0/bayespy/tests/0000755000175000017500000000000012615647103017530 5ustar jluttinejluttine00000000000000bayespy-0.4.0/bayespy/tests/test_plot.py0000644000175000017500000001173512615643655022136 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2015 Hannu Hartikainen, Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Tests for the module bayespy.plot. This file mostly contains functional tests. Since testing the plotting capabilities relies on image comparisons, it's difficult to create strict unit tests. """ import numpy as np from matplotlib.testing.decorators import image_comparison import bayespy.plot as bpplt from bayespy.nodes import Bernoulli, Beta, Categorical, Dirichlet, \ Gamma, Gaussian, GaussianARD, Mixture, SumMultiply, Wishart from bayespy.inference import VB from bayespy.utils import random @image_comparison(baseline_images=['gaussian_mixture'], extensions=['png'], remove_text=True) def test_gaussian_mixture_plot(): """ Test the gaussian_mixture plotting function. The code is from http://www.bayespy.org/examples/gmm.html """ np.random.seed(1) y0 = np.random.multivariate_normal([0, 0], [[1, 0], [0, 0.02]], size=50) y1 = np.random.multivariate_normal([0, 0], [[0.02, 0], [0, 1]], size=50) y2 = np.random.multivariate_normal([2, 2], [[1, -0.9], [-0.9, 1]], size=50) y3 = np.random.multivariate_normal([-2, -2], [[0.1, 0], [0, 0.1]], size=50) y = np.vstack([y0, y1, y2, y3]) bpplt.pyplot.plot(y[:,0], y[:,1], 'rx') N = 200 D = 2 K = 10 alpha = Dirichlet(1e-5*np.ones(K), name='alpha') Z = Categorical(alpha, plates=(N,), name='z') mu = Gaussian(np.zeros(D), 1e-5*np.identity(D), plates=(K,), name='mu') Lambda = Wishart(D, 1e-5*np.identity(D), plates=(K,), name='Lambda') Y = Mixture(Z, Gaussian, mu, Lambda, name='Y') Z.initialize_from_random() Q = VB(Y, mu, Lambda, Z, alpha) Y.observe(y) Q.update(repeat=1000) bpplt.gaussian_mixture_2d(Y, scale=2) # Have to define these limits because on some particular environments these # may otherwise differ and thus result in an image comparsion failure bpplt.pyplot.xlim([-3, 6]) bpplt.pyplot.ylim([-3, 5]) @image_comparison(baseline_images=['hinton_r'], extensions=['png'], remove_text=True) def test_hinton_plot_dirichlet(): (R,P,Z) = _setup_bernoulli_mixture() bpplt.hinton(R) @image_comparison(baseline_images=['hinton_p'], extensions=['png'], remove_text=True) def test_hinton_plot_beta(): (R,P,Z) = _setup_bernoulli_mixture() bpplt.hinton(P) @image_comparison(baseline_images=['hinton_z'], extensions=['png'], remove_text=True) def test_hinton_plot_categorical(): (R,P,Z) = _setup_bernoulli_mixture() bpplt.hinton(Z) @image_comparison(baseline_images=['pdf'], extensions=['png'], remove_text=True) def test_pdf_plot(): data = _setup_linear_regression() bpplt.pdf(data['tau'], np.linspace(1e-6,1,100), color='k') bpplt.pyplot.axvline(data['s']**(-2), color='r') @image_comparison(baseline_images=['contour'], extensions=['png'], remove_text=True) def test_contour_plot(): data = _setup_linear_regression() bpplt.contour(data['B'], np.linspace(1,3,1000), np.linspace(1,9,1000), n=10, colors='k') bpplt.plot(data['c'], x=data['k'], color='r', marker='x', linestyle='None', markersize=10, markeredgewidth=2) def _setup_bernoulli_mixture(): """ Setup code for the hinton tests. This code is from http://www.bayespy.org/examples/bmm.html """ np.random.seed(1) p0 = [0.1, 0.9, 0.1, 0.9, 0.1, 0.9, 0.1, 0.9, 0.1, 0.9] p1 = [0.1, 0.1, 0.1, 0.1, 0.1, 0.9, 0.9, 0.9, 0.9, 0.9] p2 = [0.9, 0.9, 0.9, 0.9, 0.9, 0.1, 0.1, 0.1, 0.1, 0.1] p = np.array([p0, p1, p2]) z = random.categorical([1/3, 1/3, 1/3], size=100) x = random.bernoulli(p[z]) N = 100 D = 10 K = 10 R = Dirichlet(K*[1e-5], name='R') Z = Categorical(R, plates=(N,1), name='Z') P = Beta([0.5, 0.5], plates=(D,K), name='P') X = Mixture(Z, Bernoulli, P) Q = VB(Z, R, X, P) P.initialize_from_random() X.observe(x) Q.update(repeat=1000) return (R,P,Z) def _setup_linear_regression(): """ Setup code for the pdf and contour tests. This code is from http://www.bayespy.org/examples/regression.html """ np.random.seed(1) k = 2 # slope c = 5 # bias s = 2 # noise standard deviation x = np.arange(10) y = k*x + c + s*np.random.randn(10) X = np.vstack([x, np.ones(len(x))]).T B = GaussianARD(0, 1e-6, shape=(2,)) F = SumMultiply('i,i', B, X) tau = Gamma(1e-3, 1e-3) Y = GaussianARD(F, tau) Y.observe(y) Q = VB(Y, B, tau) Q.update(repeat=1000) xh = np.linspace(-5, 15, 100) Xh = np.vstack([xh, np.ones(len(xh))]).T Fh = SumMultiply('i,i', B, Xh) return locals() bayespy-0.4.0/bayespy/tests/__init__.py0000644000175000017500000000061112614666421021642 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2015 Hannu Hartikainen # # This file is licensed under the MIT License. ################################################################################ import bayespy.plot as bpplt def setup(): for i in bpplt.pyplot.get_fignums(): fig = bpplt.pyplot.figure(i) fig.clear() bayespy-0.4.0/bayespy/tests/baseline_images/0000755000175000017500000000000012615647103022637 5ustar jluttinejluttine00000000000000bayespy-0.4.0/bayespy/tests/baseline_images/test_plot/0000755000175000017500000000000012615647103024654 5ustar jluttinejluttine00000000000000bayespy-0.4.0/bayespy/tests/baseline_images/test_plot/contour.png0000664000175000017500000022276612473034010027061 0ustar jluttinejluttine00000000000000PNG  IHDR XvpsBIT|d pHYsaa?i IDATxgXTW55X JWaF4Ě"&V +R**yMo 3̀u{CNY!T@C QBDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*BDDDDD*/>Ddd$̠v޼y~ ڵC5NRDFFn>`۶mׯ(@333\жm>z&L@hh۠  5RL6H+[[[8;;:#FkT˗000UPAAAn*^k*ֈ QZ#UFT>00000000000000000h]QQPTT hkkC*B"5""""c!RPaa!ӑ \v ;޽Ǐ͛7g& PZ5nݺWаaC4jT7""""*= D%pM8p8s ^zD333XYY uAաJ*ASSBx9=zΝ;o<дiS Q'"""*9"cذaΜ9mmmb̙TAVVӑdl߾с+7o-[G9TvZ 77~~~ضmt}}}ΥSSSM6åKgbXp!퍶mۢ}prr ""w*Dȑ#_|v!33'N@~>1~x޽ҥKadd^z>|8=\DDDDe {x BΝaeeTl޼666n H`cc1cx!SNSNY&ޅDDDDBW^;?"""n?ikkUVXxGLL|'066BuKDDD$$$A:$ 0c \xYYY4iѡCԫW_|RSS*U@ D_bb"ڴiF!&&nI)1k,dff…  ֭[777]ϟ?WwDDDTA0q:t[[[DFFH-)D" .]߿kƘ1cPN 6 .\PwDDDc ͛7֭ pn.RݺuÇq-L:QQQpss;~'EDDD*ɓ'ʕ+ؿ?jԨvT3gč7p!TV )fΜ;wE"""0Pv\!!!Ww;jΝ;#""YYY ҥKaffŋn> TaaررcNbmm˗Xp!Μ9'''iQQQBE"""*@ CJJ -[ >8q"]0<}ڵ3vڅ"uHDDD ﺨB*..ѩS'xxx2OKK HHH'PF ?޾}` )22/_ԩSJ"H營 hҤ  +++Yn86l؀M[ݭ[GZZ1f4lWf!""BӧOq <D{7FXX222닱c ׯGAA#""2*#GzRw+l۶ QШQ#lٲՉO T;v ...011Qw+%ҥKhڴ) 8pU,BDGG_ݭ|~šnݺ޽;qiuFDDDjBq=4kLݭTq :uBZZ[#"""5` %117ŤZ~~~Ν;MbUwkDDDB TN:Y[444ЧOdffbɒ%̙ׯ_="""RP`cc6*TwkDDD TaxNHUVҥKqEԭ[۷G=pMuFDDD ` #??PR%5wBhҤ Xb…|-RwDRTT*ߗ}nn.W֭[>?~ׯ_:::\2V5jN:055`kk H$uH$ D@@k̘1[l5kHNnA.8s ۇ\z`ddFAhڴ)T !޾}/_ٳgxrss[n`hhMD l(ŋ1h |ghٲ% K eЄ `hh? 5uqJPn^y֭[5k~)0|xzz^z% y&222/Xt)hݺ5Q~}e5988̙3شiLÇLjx""',, aaagϞ=SS7(o%%%pvvVw;VZ8pvv۷o|r̟?_FPPooR~bbbp:u B;vDNмy2 1qDl߾-[ĺu`mm#5q:UvzVYgggL:AAAq~Gok֬]GBB=z={aaahݺ51x`>|Efllm۶!** n݂,XuFDDDBFJOڸq# JիWnݺj={ƍqmcшC.]PV-|8qğ ƬY₄uEDDDBD"ACu?9s&C"66M6Uw[CCCnnn7o233ٳ2e 233gʕHRxzz/ׯ*Zj޽{n|ט7oBBBzj註jܸ1̙+W ..ݻwƍaggoool޼^R[NNN÷~UV'OT[?DDDP֭u~ s… 1ydu#7Dwww\wݻCnݺ?~ڞhiiaɸt֭VZaez QEB)~wuHKKȑ#1tr>IGGzBdd$1fڵ vvvhӦ :"8y$VX-['NPyDDDPp uB 4 6Ċ+>+ͱ`ܺu ۶m˗/ѵkW4j+V˗/UڏƎTXXX}U(x_شituuKiA~Xa„ _>O88qVZ[rm0P+Wׯ_cΜ98p <<<ևyxx ,, ׯ_ǧ~˗ApuѣG#%% 4@V̝T*kkkH$nyf<|fR[T~},^n̙3o>X[[c|@tt4.]*b EWWHKKSBZ =z􀅅Zz(+0}toXd ~W!00*ACCx"3f۷*"b .]R Ĉ#2YTre?׮]իqy4iAAARI5¹s7 $$* ADDD U8x"*{Ϟ=066FVT>wYQFիXf Ξ= ;;; ā0c tM61 TRR+WFznݺa„ Xf Nn݊={ >>BԮ]FzPZ?GqqOC=zw֭[8vV\ӑ ^^^h֬!JKH$tXv-fϞݻwc7n\chڴ)~GtQs}@BH$hժ~lq͛m۶yӾyfnݺkR p5\t III̙3TR-Z@۶mѡCXYYgRƍC߾}1k,L4 7nKmױ6m %%F@@&Noڥ2ǂ`Q .ǥ>W~~>ahhXs۷o1k,ѣ7oܹp 4HR ĢEpI<{ >}:1yd4j/S&WUV!)) 5jԀ?zTY&9%K`ŊAvvvEDD` m۶Bĉ>כ7o+W.޹q<==pBL<٘4iUVjsJRxxx`ԩ8q?~CUVضm|||`jj?qqq{UӦMq)l۶ gϞ -Zo*}.D & &&O<vޭy> TaիW7K}wk%Tcbb"3aܹSΝ;cӧѳgOڵ ܹsqM-HЯ_?deeaĈ6mpiHJJB@@QF|y͵c[?<==qCDDT1P֩S'ܻw :ϻɥ}r{KKKDEEz:_IijjM6q]lڴ ^B޽Q~}̘1C66msaݺuؿ?llleBD#F ..yyypqqΝ;:QyB7UC<W^B 48|0 Jm.eR >S;wٳ'/_sss gϞUZH#pem ,wpp ХKaWƌ =z 5V^۷GZZ>3|WŋRD#GKK?q#""*@k۶-wR+7n`R.ر#~$%%ƍBBB ŋo 3gTRHLL/v튯JO\tuuѩS'ٳT111ANNcŊݻ7,--^prrΝ;qt3g΄ϟWܐ3g"$$ΈWBZ*ߏEﾃR""@>}pҥR} nRzݘ\v #GTzׯGvv60w\axB1k,$&&BWW^^^:u*dM4 ̄Ν;DDDe:@__aaa6_m_ 4o\u:SSS\۷/XXX 44T`ooX̝;pqq ZhdXZZe˖X|JΉ!""R7"*UB=cǎR «WԺԩ44*&&&Xr%^]bҤI¦MZciӦ!11*U'fΜԃԩh;߿nLDDTT;ׯ^Zj233V޽{BVV_>6l؀ x{{cذapppC M4All,f͚ J[*"44;wU* D_֭QNlݺT[XX@GGJ wwwՔWAA_=z޽{qADGG#99(..VI?صk.\uk׮hٲBR*b֬YCAA\]]wߡHi}˃+;DDDe *+455ѿXx1Z_KK 7V9ׇj~ȓ'Opq/CVVL:IRQFhܸ1sssH$₟4iѿ,X% .`ƌ2e 9~Ii?&M >> @N7`:""8o54h=z#GJ}'''$%%)^vv66lX*7ti|'/E IDATU뇄4k ˖-CTTRSSGɓ'wC!44;vDAAn݊@XZZvOb dff*u D"A표#** 57|ׯ_fJ_olٲEi}WZٳ1k,|'JfTf$&& "11QݭThnnn" Tj]VhjjW^)^ǎEΝRߤ6mN۷o+\޽{ȑ#byB* ~bԨQرc"//O /Ϟ=SLTڵKӧOŀѫW/c%v+ġCYYYJMDD%5 ? :Ǐ/C=<S^iԩS OOO}% :u*bbbc899a۶m%4440w\ٳFfp-&""R%#֬YڭZϕDSSSӾܹs_W^J,h׮ك۷ocو+5k={Q~}ȑ#B&M0=qssCrr2w`РAxe={x \]]qY%""RH$5j=7o*;q kU\o޼QBW5j1}t,MƘ:u*_phiiwްŠ+JR@@ٳgϟ>lق-[`߾}pqqQ0pкukJKDD D_~R ֮]ԺRZBddµ 3%tرiiiXr%455RS455ѽ{wSlذyyyJݡC;wO>UNj#BGNp=uqvvƎ;p5b033Ksjڴ)Ο?%K~@&M!wkkkbĈ;v,z{+k֬+b tYi01cѣG Sj:HEkݻw'!!7nµ333\7nȑ#t?;gijj?GZZ5j:`r+o>8qJY$H0vXDDD 66J}40+++`ٲeJ={ApppÇcbb۷o+ݻQV-)\,]6-Z~ GҥKanns:9"##i&8p7ơCGHNNFWڴi?-+V 88zB۶mѢE }ӧƍݻ%:㿘bӦMHMM= OOONH$4h >\'z… wwwlݺUޯHR^K.Ehh(z衴sHB$~K,QZMD={"<<ۧ֬Y5kT7/_T*EÆ K\a˖-h߾=/?3>} CCC4hR =z_ ,@^*UF֭[v<'OB-ZgϞq5LLLp1[;w\Aթ޽{cz5}$ qaDGGyJY/DDD0AeFbb  󅎎{j^pAQQQ%'uV&M%o޾}+BCC |}}ҥKŋEAAL5-VZ%>Sakk+J;VDDD<-**[l&&&BGGGL6MxBGH$1e/bn:-ŭ[o.]$LMMEݺu""~Mq e/BOOOL6Mi5EÆ K\cҤIԴ(|||J<}ӅÆ W\QZ{QF B___7o(T˗b̙RJDboVHR(!!!Aԯ__ԨQC8qBs*ġCR" 0b^Z);&ӯ_?۷_.Q 777%>ٳg044, 6JikԨO>k֬7I&!##ݻwGڵ1|p={DaΜ9̄;ЪU+4^SSSNE\\ꊕ+WՋ+䄶m"$$DzԩSNm۶ڵ+/_P=""b!ĉ+]Vi5/^L ?D򠫫[Ν;qqqpvvVJ#H`oo3gҥKĸqp 4o666k!<<ͅ#&O,bn'''$&&bذa7n:u<5pqL:SNE^+W޽{_ 88ǏYDDzj~CGzÐ!CDڵ~］EK<~b„ %/zU߹q022m۶{탲_~EWmmm1`_zyyybܹRJTs}FÇ_~A˖-K ,,,ܓM0AmN@�P4h@}V)={&tuużyJ4~ӦMB"G=gϞ߿Dw^DWBqѢE @4lP[N'Yb׮]vB___ZJ4H,^XHR"^*O>;wD̟?_&))IԩSG }x8>!*iӦ͛J94 ЫW/rN6m @ttcV'O=!Ν k׮uJ&vSN!>>5j,,,*&{FFF1ftH&Nٳgpvv馡!83gbݻ :99!66UT7N>P=""a!*{{{t ,(!4b\~D!~AddcZOϟǥK0iҤP5777ٳ/_Fv0ydc2###_'OݻwѴiS,\Pܹ3Ç<#"">>>r>Ǚ3g ڵKzDDDf͚ll߾])aggW:tcǎɽ]k:u['/sNԫW~~~%Nؼy3\.]`ԩʕ+e^닔?ӧO'RSS?8۶mƍ}vxxx2޽{w?/_N<)Z*?޽{#00[Q 9::[n7oRH$5 ĝ;w;w 99Yqfff(((@nnscйsghhcÆ BvFa֭2-VŢEpyy...? DC">>EEEpuu+6i ptt?֬Y#Ɩ-[0m4L4 9%""+w Deٳq5l۶M)lذA-Z!<(8KKKkݻFV[YXXDjj*0p@ **JHJJ_|Yf.Hȑ#'ӜժUga=zBI$̟?k֬ʕ+ѧO{!""ѽ{w̙3Gl?֭kVJԩ/8KKKhjjtO/^ǚT\\!22Ʊcp)ɓ'J}]ሉA*UЮ];tAutt"&&ϟ?BCC?~'lذ[l7eW*bX~=6l؀v27FBxx8=m*QP\?["u)^jj v!p@dee5V=Z/_.eކV/_-[UT.*V^-Daa}} KKK)F->|(ׯ_?\-[L㒓044Ó'Oի LƾOLLVnRQy5 ѧO̝;W)4i[Ʋe۾}{TRwkkGۨWR9s~~~AHH 1c :tɸy&rssqmdeeܹsؽ{7M[[[$&&bpvvFjХK\;EI$ m`eeUV}p.BCCl888`֭|ZDj ]vW_}%:#___C[[e兘z 2="""ON@&+++KhjjK*ޡC#ؾ}ƍ5@qÆ nnnr7/^ w^~-Μ9#Ν+Zl)R _Г{C D"ę3gdѯ_?@G<~cEHH~~~23ѡC)V^-#DժUeDD#ޯ) ] 2DԬYSxBZEEEJSGŋes]{AAAe˖{{{QJqF3gѯ_?/gggtRn.>>^ bРA޽{2 Tb7o<@^0aR}BWWW@s@=zT13gժU#Fz]v jXDGGݻ QZ51c _z_B*ٳgbbb044{i7 mmm%rrrd/==]ڵkǽOqqo\*;E_SH 0g-rb֬Yr+..r=8u ׍)S\SXX(6l(:w\"88XJ*1c͛7bƌBKKKىp̓'OD޽1rHuuԑiw޽+<<<8pիD"o߾UQY51!?/^jWJ`add$?.׸e˖ ---q]>[/%K]]sA*=zH̙3GT^]HR1bBeJJpuuDL8uu։J* {{{qΑ+|||شi̽~ZSH$*{n!JE~}@^k "99YZnRT|wr{Toe'?g@gϞIq h"QF !Jѣe~@,ZH+++qIII666BOOOl߾Çe^STT$&M8Emzzz[-+ޯ) (((666O) t*jժ%ӫ97p@ann.?H$"77WϿfV?~>bܹrS/^oV ]]]1uTfff ///!H_|M^x!b}XZJhii ???#ך5kڵO/DՅsB*ޯ) qy< իWX|\Ο?/W<bڵ2}زe\}%%% @Oӧʕ+jժ%K+,,- $]9~WQzuaii)זGzzz]!LLL[H_SҏV'"@ΝI&PZ 6D߾}>#--M %%(׸õkгgO|嗰Cxx8:NSS&MBRRtuu9s!H0|ppqqC-[">>ǎ;ԩSy&pUƽ-Ξ= 4k %EDD'R"Hxbddd`ƍ כ1crss?رcqqo*{hݴ1 IDAT~~~HMMq7Fll,+̙3͚5õk󎎎HLLDVеkW|W(**[XX &&ܹ3,YB[[ވcffg-Z@|||kGHݏ`/|q8pYxµ'LLL>&޼y#j֬)b2}~ΜyEu}~ 3 e轋 أbk{Khbbh,آbl"6@"̜~9s`N{|:kهYkVyήD:zBTTh{14 ///4oܹLVb]]]T)N^^lllD+ 11QYQfM\,^o999f.W^ԩSQ|yB(cGPPr9ƍgt.55H$8q6Gaa!z Bc{}6BBB`oo#G:?˲9s2.#\SN`?CP(O ̇ n֮] B~wٳ't:]+W+V޽;Ev=NNN@u놘Q\ٳ ` .\!CBjԨ7/((  l21ydH$#--M~ٲeZIlԮ]r+WVL&C&M 1 ۷/!;B| f>4 wò,U0~Iޔ(JL8D4ieˊ A}L>|={V9Q%''hذa֭͡Cj@,XP+WJ*o3Gt:v-h 777xxx )**Bn@qDp{쁵5TR"ÿ²,F BNAH BP|hbA ܹs`|{ :4w9q!زeI[eQ\9lҤmQQZ-&O, ,uω'BR!55kΝ;:@&'N ,\j& 4j5J0p}۾իW ˲,f̘񶄭PJaaax5|?uԷE OkC ^={m.233j_h]||4_3XBwDtt2S,[ ю#ر_ҍ\|/&m7o.zUhh(z-HѦM4 oal_tY#44 3,ˢbŊ^5u7~zBpM{v +Vu^#...%8<ԩBj׮K.\sECP`޼y;wx"QV-rOMB*4y^M_Nbb"$5|lٲr͚5OP(ꯙ @,z{tf7ڵ l۶McDLqcǎ 0\d d2^z%z'bDz,v܉ HR 0@0c;dBPn]<~5֯_kWTTVpĈ^G>}@ID)@_ٵkJ%5jdRBP>_3XBHOOvj>,ˢv]bU"TlY4mIII5Pl,? B~WQׯ1{lj8;;67߿?\]]舝;w/!Æ `Y0hٲ ˲6mp1},b̘1 `̘1f߿* 3KBP>_3XB71N2]0Ņ J1b%0 ]&hw1Q%, FuLBi&Qby&f͚M"((rE&M0rHlذl푴4m$$$[q `R)֭+8RwPՈ6b HR4mTt 0gBЯ_?Z:t֨Y,B| f>4 ob0PR%+Wݻ^˲GDDIgnnnHJJ2g2eh۵k'Z;$''R#\x 6!֨[. sbҥo1m4 6! {Ν;K]"~@P`ҤIB.#::w=x t:{NRRRooo)w.e0Sh4ǗHuBP>$_3XBwIIID"ٳӧ)PAuLΙ3rd&8To? B_70%W,Y\2e`ӧشiB`cc;b%f1c@&!,, 'O?s akk+{.áV>|hZepFP ׯL&C-j(?qlllPZ5Q* Ă͠A`mm-JOj+WDiܸ1Lf`rssaooo2̄R̙3222 Hl2QtrrԩSEr`8,˘4iʔ)B0i$QrDGG*uzZjBKޒ/_ꫯx1rssѠAd2XBFt///cFvRiv/ɓ'jQJŠ ĂMnn.<==Ѹqcx :u?xb'OɬENk23PF 4lP9н{wQĉH$6lX,Oj2 m۶-dqq1fϞ R 8qBrf &MBy3=o'dM8Qu:;x j5WnVp)ڢr4P(ꯙ @,z)[l!YGn޼YN:dKvv6Z- "hwBuVAEA.#++۴i5j+,"** 111f ? |7!5j޽{E׮]C\\$ F!?GGG~ * 111e1}tBеkWLQrej߿_{%''*UGř3g`ggXxP(9P|hbA Mf͚ͤ^,ˢaÆ(JQǏJӧO*Wl2`xh}I&ѱ:tDJ^ǦMB*UݻwԘ1c ʗ//zaL6wsfrԯ__0}6l\.:X>wt:"""3QkABP|hbA M4k4۬}nܸ\I&^3bX[[cʂV~WQwvHHH0y[RqݺuCPPYmeY۷UT!UTѣGEpʗ/B9szƍ!͛7-Ozamm-acc(ڵD"-yelٲ%yg B|2f>4 腦Yhߢ 2rHXYYݻ쳳D #N8VVVdqq1Ю];V\ BGD_~e,={ ** 4nXԐ :Ԯ]iii۷oVE2exЪU+0 sdϟpM4 ۷/!'ݸq ăD˜ =! B5A/4ň^Gll,BBB̚KxzzI&,^ ݋/>}-\Rׯ?$ 郌 8wwwt:l۶ƍ `Q@A^xu:߿899Ӽɲ,Ǝ["wc.Μ91P( ꯙ @,z)r%r?ެ}eP;ve_TT$$$t&g͚\.(N~ խ[7lҥ mŋ 99YMaa!Ν [[[a…&222дiSB0`477-Z!SL/_L&|޽+zjj*)e1ӧjQjUP(̇ Д2ad2\x{,zWî]DM˃ڷo/h7uT{OB{n֬Y#Z9Bnܸ!c={a/_ǏgY-B@ p-N;ɓ'?^;plmm+0ƍC&_~t0 ˗ 8˜R͙3gN޽Q))) 6lkyfjDDD[\|^^^ 8mйsgB7mݺudhٲ%P<}aaapqq)x_INNFABBY b ꯙ @,z)\4 腦1h T*ޒ1 005j=vPՂӗ`ƍ  edd@`ȑ6yyy79wΜ9˗/_"ǏѪU+BФI)c,bɒ%P(rzP(Xz5Mnn.4hLkg!}8p 7###pttĈȑ#PThРY( ꯙ @,z)|z ǁ@+Dٿx&,ʕ+#22R|GZ5jZh{b'x|Bi&by6l؀ &w֭ k׮ťKV\߼y3\]]akk+W gΜt::yaa!tBƎ=k׮ `̙?B" 11Qj%sDEEA 4i" BPJ̇ !:B/^l>۷N,z+V#G%'' n233j1d^4r̛7OиqcAMphDTqE4mB ErB!h4hذ!R<;v| jD ުs,b̙ M6eJ,bom6mB@Æ ˬN> {{{DEE!33;geebŊppp09 Z={@PUV(***> Wf>4 腦wPf5T?{ ر({8tڴiwwwID'NR,ܹ3<==^x1d2`6HLL :wlNe˖A.#((}o0ÇcS !_-Jlݺt^6xhٲ%o͛R)$ Zn[ʴ~X[[zꂊϟʗ//Btt4ʄl۶ 2 ڵ3;#EP(Ff>4 腦"77>>>YYX˗/!\篜;wdV޽{P*%999tٳ'˗AO?k BaLУG/_ޤk׮}PҒ\lذ͛7\.B@bb"EoJɌ]vضmlllʫqiϏAP*UoI\rr2%\]r([,?~,ox7bNoA*k׮fP(#_3XBS`oK˲HHH葥F1ِ>zhXYY6̓T*h֬n۶-BBBL:ƞ>>} Vۗ(`"##CPP!ŦMD;,⧟~FA@@N:k{5ΎWD oѣGakk(<}pvvFhh`7ၠ AN31}͚5`}5;P(ꯙ @,z)bիjqBܼyJRP]֭[ mڴ)((ZjksiBn:^Oq(C.F VkR$ ر5j!X~@֭[Td2f͚ŻŋhԨ Ԯ] k׮… puuE`` ݻisuxzz" @0}6;]GGG9GB0tPP(Ă"\Ff̘1gΜeoT!߳gݪULFGXX;,`A˲l~[>}h]I8q4hBñsNQrQQF Bׯ[u҅ogϞwA@@qeϺw[e`b爌gaѢE `¯n IDAT ރBPf>4 腦T z"DFF"<<\Ԥ eQvm n˗/bN^TR6NZ~=g-\R><{n"hW߿BmVuĉo3"5j}nnn8z(իT*QjU h,bܸq oO4 :>>M4ͦCPPbccy3 b=z4r)S@&իZ~EFoD!Xr%Cd|@AӧOOOR@,AUVj `ڵd41X4'OB"૯*([,bbbD2([,bccLt:A͑k׮A*b̙6-Z/Td2#y۶mpAw1+Tٳ'AŊEaΝC@@䨜4jR=aoopα_T5kp T\ZV'--DAȣGQ\,ݻC"`ƍڃB7ĂRZƌLRa dw9v!XpqPChx3%W^D",ܹ3<<|u]xzz"$$DT EEEϠRLM)  @,z)p%( 5(S *U$+''hذ .@* _|w ɓҠR0f=Zn ???^CN]~:tȤ\2!C`HRԫWYYYv6mWΩwhhZI[,bԨQ `ԩ:u=sjj*T^;#7oބ+ʗ/_j|$$$@՚EP( f>4 腦ܹsAtLa,Ś>}({cY?,h7|pXYYۜz=*T+\;VVVŋA?{cP׷.EEEXhIۏ^ǜ9sT*ji>p;699:e˖hVZΝ;92e !;v,g@CRN4 00Pb+Whx¤=YS(ʿꯙ @,z)b0OOO_0b( ڵ>}k+N:&u?.]<77Сn>>> kF\\w, \|={(۷ Vp ݝO͛7L&U3gxAA ݻB2eIIIUVꞝL˫ ̇ Дڵ+ E D Z۽{lIRRyk-[Bޕ+W0 /^[n]vB >ԩSGY7NH$Y'OB (..Fǎm 0enݺwwwr''OBVnݺz\ kC ^h߅qz/R=Ξ= T &߰a!_ڵkNPޞ7`TPſSNpqq%\#00mڴ<+?")SrٙWxF svP*ؼy{  Z|=!ӧn"csu ""BTǡC`eef͚ u\\\U. kC ^h_|Vk} N?\nժU 63Grmcy 2|899gϞgZj^z&L,֯_XB/((]vyӦMD"AZˈDFFrji N}e1uTB0zh dʕ`ݺu Qvmj?~U_!Hлwo mB?~* ͚55B7Ăw@"fLq5XYYaׯ_B֯_kò,4h_/^Lɓ's>OMM5Lڢo߾1c >|(F%K J!h'`@N@AO?!99ϟѣGb ˗!J9[_> gggxxx6===K.eY?`!3?o<B0x`?3$ 8%// h4g?hPV-QY ȑ#Ma̦tڵԁ BwA5A/4C0vXHRQ |,\#,ˢupppl~4 zk3j(( ѱ3g΄T*5gHR^a\888_~o/E& 6b̙`FT_1w\+Wbɒ%&JZZT\ΫTn->i_={rfԾ[Bзo_ cՂA˗/QZ5̙3g=z(T*7n+l.e͚5 `ރB{Ă!(**Bll,*[.ݑi>##hҤK.!|@s:_FHHUhРMBaRI&X ^Vŗ_~Yu,ĉhݺ5$ \]]`$~mx=xԩS 6lYb$ ڵk?|^ à{srr {{{1F B6mڈ*/0a!A)؜9sJBw@5A/4Cqmؠcǎ#-- EGO65kքrss9m:$ b||r7o7ͅ#w.>۶m!gϞZ{֭5r-tR^^^XjUJ/_BU굼~۷0Xp!ƍ!ѴiS@XjAϞ=9Qvv6ě-[@*~λ,a@S3Z{P(>_3XBS>$WoBVX!ʾK.{xm޽ Z- ޽{wjtӷXEڵ>o|(jl۶nق=ZF||}T%7ޝ')**g}kkk$''碸͚5J*QGP=P|hbA M" 111xG\\Ѥ0={6 ˲S<<ΎSeYԬY~~~:999pssC-86 PCӉҢ0W_Q}Iy۲-Zj1 oLnٲJ|ӧO111z1ҵkW,؂ 3w܁;ʕ+'8.bwww@!oooj τkC ^hbΜ9 `Ϟ=cʔ)H$?L"<<aaaV\ B6m<33...8[n C \o(ٱcscCШC&a޼y6eݺu .*-䄭[Zl2HR|gAVQZ5LGJJ k֬5}tB0k,Ͽz*;eY[lze'Oy!<==Q|yQ  ̇ Д`@Æ ęM^G_tJ%kò,ZhN[rcrJg0',ˢ^zu[hOOO GRR\\\Dohq'Ϟ=CӦMo9={h͛=?agg+rf".\GGGDFFr>7Nӧgh;\h;wZ5kChݺ5 :$LsqqAtt4^|iҞ˗/ C(ʿ ꯙ @,z)tf͚xЪU+QF-Kzz:Ѹqc=;v[[[NbDGGlٲٖ۷o CΝ;P*;v,ܹL&Z`0@aܸqͅeY|PT 됒8\\\v)ԩStr9FŹ~ƌH$P(pM3:Ncbbb K˕+WP\9T*QwEpp0isUL2.]#*TG +ٳr۷u_|X899ƍ^B_|QbӦM`.z ρkC ^hʧ`ҤIH$8|pիT*._lɓ'prrp'6Ǿ}Kv+H$:ugEEE@dd$/yyyAyϘ+++YkׂRҒzM#&&6668rͭ[兀,ŋPbEΩf!'O~`@!ɰ{n߰axL-[>>> W0 ?$ŋz bP|hbA MzԬYnnn'q090f8^W^!88+V-ׯT*(Tc)VHHgٳg!H0m4νc~QFAR Jӧ,_JќAûN:P*Z!݃/|}}q?TgW_}Bg3QQ6m J;!D0{Io,Z̝;פ-i\6l( ŲĂx1QnR\|*JP]B!8ٳd9r$<#::3q(J_G B+Wp>o޼9x'xܹ3;K(g8~maa!ZjT5kpkc?KFFѵkW0 &55AȩS`cce1`0 9877111,bРA`F%''CRy&.,N:A.9S~5kք=^Z=(B5A/4S3qDH$Q#L`Yڵ nݺeڵkFRR`@Z NSB"رc=gɓ'!H0c ν.\B'|򈍍5=bYhԨ݇ѣ!>|   ې r*;v ֨_{j tT۷퍇rCcwT^z@~j:!Aw σkC ^hʧFףVZpqq)Ӕ@DEEE駟LAӡiӦΣ^GjY]J1 s\^G\\ʔ)}b[3C²,ϟaвeK$e6T*ѢEMhٲ%J%0#EEEhР4˗"*˗(sUbxQbE,BgA5A/4x ܐY$( Ϥ-˲h߾=4m۶ NwlmmXo=+((@XX"##9ޯ\Bۋ۷I"x{{]vvm۶qqq&4i$UAAAx{wL:u),,D:ux;_GGGq:^BJ̫cɖB|r^#O}6<<<5 IDAToذ}}/P4w8::r_>s j56l9t ##fJصkoR:߳gR)P(ꯙ @,z)Č3@Ν;KeYm666 #b&Dpp0T%ҥ j5gၚ5kr={I&DbŊ tRBx3~hٲ +.\ h;sL pqqAdd${nTD 233~rtԉәw\]]{GѴiSjQjJ%R=a ςkC ^h%a0ФIۗZ}`c3g`ccZjqdF k׮AӡJ* ,w˖-=+**BJ)2hTnݺzT^Y@T*ř3gx!WqƐw16=zVVVh֬{M,ˢgϞJN54i=l00 g?'͛pttDMfXE>} Jo>A[>Qre:HP>-_3XBS,~AtG׮]RLaee=z\r ,aY-Z=߿;w@բm۶Bβ,j9DXN:qQQQ 5~pss f>_F֭!JEXŨQ@ҥK9mvLN:{ѲeKXYYq}޽R=zx{2 h׮긱i^hDrrr2hR˥5 .^(hGzz:S(ˆkC ^h%+++ TGDDD9]FUV\v j۷,ʂbcc9-[޳lrʜ}>ҥKP(>|kƗ_~)h)(..F HuZX0 ~gNkׂa <猪 G1P4i{ UǍYL&@qF0 ѣGy%"##9nX ׮]իۧBP,ꯙ @,z)L~~>*TFbuZ-ZhaeYt*J_~|O:\s>ٳ'833'OT*tJSlmm93,0k,0 뼑9s@",ztdb0T[r,^/^@xx8<==sc&K#;;ʕU7k!36ƋHKKY"L&C޽-BC5A/4ҹ{.ѨQ#*|lݺ̚5ˤm^^$XԧO(J;_5!ؼy{2epr͘1 pz oooTZG/#)..Fdd$""",p61Ѻuk( acǎRAP>_3XBS)L:,B^ڵkɉSllٲ3-3Y&LDDV$BĖ[kѪRUEQJkOjUj/U^ YȾ9w 5_/<{;sx]>̘1ise"))6ݻ ajF۷oM6f:>C( l۶zo`KӮƏZǏK=+L&uF72Q~}xxxPE(M&:t8={@RQn]PKĦ7xzo'N^g)Il@zz:9E>3f/f3RRR$kHypzxbCy^0h߾=\\\$:Ȁ?ׯ/KcDNdƍ >$:vvv7;wR˭L&ˋڃzU\\HDDDTXwiSZZC1P f  jVjʕP(T[n! 5k֤-\̙3zo6mRlz*ѠAyj5Bʕe=5 4yݻ E5,8t0h gMhoE&MB5>hňA*Ut1!ҡtx{{y7빹 Attdpq Iu[-[^իETTrss˭߹s ;wʭϚ59`̙3pqqAbb"o߆Rd5͟?̟?y=^~eM⥥hժ Ο?/iF-=5 4y̙3prrB.]*|Z `dee!00kצ{^z\2222ʭo޼h@DGGSRO:fܹsAʕ+o6oooܾ}isV fpܬFJu8Nnݺ ʐ^O"hTTZf2жm[8;;̙3A& ш-ZTxُ?B> ]p._@sWƍ'Ydb4l~~~ȢE$?jݻ7vѲeKjchO+9z(Z-_nl6#99tRuAХKzj͛7 x'r |||PNRKœzj( 7Znn.jժl wS님j0z=ԬYAAA20o֫W/t:jh4x,ڲ4i!XfMp8OY@l~9+dP4x{{QFTG:t(4 ˴E ?3;R#T-5ڋܴGbN:L7,, "v[lB~ȴ?^6mP9,_>c Bwߕ[v|||MᖦH=z֭5|2<<<<{EEEhР|||dMnAy,0ܹ3mv*okhLNN΃tZñ'qII 7n ooo[PTرcuO?eܽ{aaaF^{Y:v1t:ۗ:u k'LRk+lݺբ ӧ쨁 ߿?4 U=^쏠v)Rb)SgaOOO?QPP@y~hKgR<׬ 6?М睳gB㥗^pِX­[ 0ТE իA@Ϟ=aooOmV;uT1-))All,|}}=bܹs&jki- GջM4͚5;mŊ PB׮]bccA,c߈Y-رcppp@׮]-6UVpuupfP|ƍ r8׬ 6?Мkׂɓ'Wx!C@`Ϟ=m:Vlp7KWRkצ.,,D:u@矙N͛7XoBV353F)rʈb?H;wjf)33lTA˰*+5lժU(999 Cjp=4g1;p9f=<!(?s/--E\\<==e_xdpOpUxxxYfEȺuʭ߿ ypwwǕ+W][VV-[ݝ969NW^yŦPQ\KN:}4z=RRRٲ3g@ף}qbɓ']dB0{rkf5L&iArHɓA/´9w\\\T_ae8?׬ 6?МKn߾ʕ+N:LTO?BNJ]߹s'j5 VnMltf9d.(f`M癙 DddIRR%N@nn.j֬`dgg3~W( L8~z( J~~>jժ@dff[3ׯK5jժY ٌ; Of>Ze1bT s8'׬ 6?МDxSKrO0/,\Î5DZtI8::SN7NѣGCT2uF)w?/^+Zn~{7>#l>jժrkW\;ZhQndBbb"\]]q…rݻWrh׮CDDe ~' hK pssp8Y@l~9/*kD-G-;;\#[iv0l0j.]777A|g͚5 0]!b2Ю];03c$!A-[M0q":w '''={ ڵ+q[oUn-;;!!!Y&5$6ͳd6lB &0ҥK0HHHA,>9dee!88#9Ӄkh΋Ȯ]e8qlQ6u۶moΔe˖pssŋ˭̑bNժUSZQ)1}tBxb3|+WNٽ~!y(3Ӕ`ԨQUV@HH;@_nprrB.]Y4Kz2zjo۶'(bJ@@222,e;w~ Ekh΋8Zwl6SNptt =͛T*1j(͕+W$粲dS޽[n]|? WWWǗ˒޽{Cb߾}FaN1"::Zo㑠B Bf3ڵkWWM ٙ:.\@-S{{{h2HĒ\$T/ɰˇ4obք8p_@s^dnI\e3g!o6\~N_nݚ(9J6m*CVc...FƍA/))A-3رcprrB۶m+\DaYYY<fS=k,ׯg-,,DTTƍc6ߺu QyyyY&BCC(Ur,ڈ#,ڲ6lP=85 4ER, = 8pL8O>oPTx7ʭL&$''31C}ZjT9;;ժUCPPdejIFruPC߆ZfDI)_t HJJ*%1LhժܨFN@TM]6e5 `ٲemif$''`0TX<>_@s^t*ŁjѻwowII 7o777$$cBP`ڵ 2GHOO/>f( j͛S߼_|^^^l8Z3hb4楥hԨL&7n[߸q# 5KP/,Yzobg˗a4hQC:MwEXXjԨaxοY@l~9RSSrY|Xb/Gxx8#S!{Ĉ׮]bbbi(8tPkw;;;ՋP=zHNN5k!9s& |___jp!ruF$''S qdĉ,ɑ#GjCA+"YF5yd( j֭ٝ[T*1~xHaa!ԩ*UH~'R={Vўģuօͺ8ɉ9rTB4hԞ~͛7CV׸q@_ʹGy$7n܀h,9pGRQ>& 6WLKL0h۶-\]]%Y\rnnn&,ӧWzCg|8X @lQsMN mA@nݘGٱcj5 t|o޼ DEEQYPNTT[J#G,`TZ -XzoK, RСCP(K FӦaŊ&F1vvv$qd1-qm"..\D,}rppj\x)))Գ@DGGzo %%...3Ĭ U3E&cǎe6s8'׬ 6?Мz~Ƶk,/ZHN8^d贴4TT QQQ->oJw")p }߾}R$+*z@0muH)..FTT˕uڵ *f͚֭<+'Z- ļw"441kn:4nrJp8ĆoIz# LΥM6AR_?y$ѦMj¨QP(?[ۿ?ѩSrdB׮]jk.꽽; Э[7jY%@OG Ν;KnwYcȐ!۷o-[N:u* nZn#f̘1PTؽ{7)gRũS>}X,7:t5*Unݺ ñY@l~9FzߺuB{>}Z2 xٌnݺAL/|[BVcKٌ]BQɮ]Jypqq. @RI*×SNh4~ذA/"AafʶmBA0hժ<==!ʭI&󣪖?ܴ.@/[ |0m IDATR ~$$$(,,hOѣj0`@p8pzxbC"NW^-[@RaС4!66."e5La"h4ܹs6޽{]6qr& ]t;Hii):vFcɷKnQx8*UD-g[sRԘ1cVqȑrkBBB5K2`f$&&R-((@DDV* C ɓG߾}+D BZߖƍpuuEݩ%%%S¨)[n!3f(VXX5kz^[BP`ԩ>wн{wfP\\u"00P8t૯hbjS8' ׬ 6?М;k֬B1c*_<TV/**B&Mϗ[ZƖ-[ʭ_v ~~~UV ^Rf(._ ___DDDPB4oz^ =zRWI"b2i>Cj_Lܹs"GffIΜ9{{{ 8zq$AQReV.]`@de6 VԦDQQQJYG_@s8ԫRNȞ={jѽ{wf%++ ժUCPPujSYYڶm '''SN`0YfM&:w V?ܵgΜ4h@-CF"Yc20`B0|U0sKEEE D6m$A@t~hޒDGG3$b)A*U,{נj;"k׮+**B:uѴ̞T"_f=<!p;C JoB ,گ\ ʕ+Att4ulk~~>իOOOg޽:;v,7-[3ÇѢE j̽{P~}J-o!lJ5{ժU X₾}RA@bb"|||Aʹs@+t\zk`4"ԪU UVQFAVcOzK$rظq# W<>_@s8)++C6mq 푙P1X?̧~j1;p1znݚPS3%+b^^իwwwj)Ү]`oomۢzNN֭ ;|!x+]7[ލ7`0УG… zG&Mߍ;w.us{fii)6lʕ+*_$'a„ P*ؾ}{p8Ć/BJV=.\www4jHt#F@PHߠh.]oooԩSZ2%N'?~|LԨQ*U mٲvvvԩupvv R… T*ѥKыصkaA@ӦM]&no)Ik.(Jf_,ZtE}W۷x ɄxxzzVù׬ 6?r Iq98p)(f]tNÞ={v˗/!.upvvF-9vZZB~n:jtޝ<999+>,Xf t:4ib3MʭZB5,fAΝ;;;;L0.ڵkIOO;Il:?tP"DDDzo߾ptt.)saڈԽ޽ z\۷nnn /ٻw䤫y`gg>}PW^ B,YRnuEhh(at:t(u＼AT*B<<<]n|/pRīJI&prrŽ;$Fhh($5E)РAv@۶m6| !ؼy3ugϞpqqX_V!N_GFȑ#'N ƽ{v" 3f!ׯοY@l~96Rr;v,!,^~`@}Zf:};ܚd/ FC0 hذ!uҲeˠP(Я_?jxt:lܸQyЬY3h4g-&cN] ___ɉ`Ok׮@~ XJJJPZ54lؐ)̄':vH^j/ 8A,+/ϐԭ[WVVbŊ X *9s&pY@l~9y!::>>>T 9#88aaa1"}C͛F2f{Z6.N:Nھ};t:ڴiC BVXZ]RXA0qDB[oYuƍCVO>hEe޸q:9B7PE6Ko^^^dyyy Bll,3S6x`K=tj)U=3f  ׯ_ο Y@l~9ܺu tN… DLL QķR.^ԪU3PVV^z Zڇx{{Soth۶-5Yf 4 ڷo[?> ^~e; M6}d鰷x"Gd@٣GxxxP !ފ7oՕY{n( L>PHf8>c( ˀիWGDDD%333QR%Ȅkh8<+KѣG둘cƌXtiFRNjGGGݻzzz:QR%\x֭[%3!7nNCBB`ժUjh޼9sף{zxx 6mh$?AΝ;egffBK4ɉ s58991G.^[oOmqq1Ç36lZ=zP34ϯ^@RѴiSjˉ'兰0f#==Ճ~~@jj*@%ǥSNkF \ʕ+Aas>x  s(PRwhXgϞNÈ#$gϞpvv5l!##^^^HLLqdpRRRyEkh:>#Bp [*"5LG]Q*1wyxNd)ٳgz=bcceA DJeGԩ f̘!4X`E=IY*4hX\v jsa "**y 5{d6ѰaCSKRfiYP(03''h֬v H>%6lBfϞ]=8Y@l~9oJ%V^]=xQLJ0."}7۷QZ5… ֯\  ?x SPfMF߿zfƍ!}=}͛޽;!y!ݻwGӦM+t]PPӾgBSܹш~QO:Z)SPBO>n2ФIJNaAY63|phZ}}7JڵoAAA̱oj;v,Z-ud2pߑ씽= &y#FVQTT5kZ5QQQZS6<___LG1Xt)P(йsg)nܸRY!oAPV-HڝΝ;Wn  !SLa:7oDÆ agg<4JJJ_#((4i˗/F^ {;w.jEė_~̒۷~~zBb -[@a|fM4APPdСCPT̒GOP(T_UVUxEkhɒ0>V)ܻwuօ'SNh4~S&M$YڵV ֭[^: iiiQݩ?ߟV|4LXjՃY...۷/V^ ѧOBlٲ̄ZfN9{, ۣrCezhdCb{Pղz3f35k%A@ǎׯWhEkhs!""B4;wZj%w8;;#.. Ǝ BA~8|ddd 22T433Ճ^gNZZx14 Zj%ioPv gΝ;{ժU! 11{F=РAT*8;;K%)) 5h׵kWTR)x9j(˗0fzFFF#zż3gBP`߾}LԪU ())x\rʂZhħq8#_@s8O3gh4QFoRܸq %w^8::"!!\ `Tׯ#$$)[YYY`NC˖-jfgl߾...я;8;;[]se,^#F@JJ 7nMgϞ/*,>ʷ~ BtIǏ[:t( Sdʔ)h4L7|BmF]7LA%';v jq[E'-C{UzEkhq8::M6xi\x%'*'%%1V #FHj=*UN[w7n '''j#sqq1t"}TR^^^̉Op]t 6S)222P(ZmӦ "##=1pttѣ녅R I& c'OZƤI$uPTJA@JJ -bRh4(pzxbC<]n Fݻ3,q#&&FVs͛agg;2Q{DJD͛rHHHN˭L& :;KaÆt̆j~,XVIF[ ::=zh'ߺu+f„ tq˫V!9P1~xh4ɑ%%%DZdbݾ}HII¹8z HpzxbC<}VZR!CT?&M*_Vѭ[7f(HaMOOG5SN[/..FJJ j5ub އ B ŠУG'Nzt;wnϧͨQoN`3@FC aѴiSTV[=: иqcɞ?b^,YD=ӧOCbރyĆgBƎ[=GGGjJbիRгgOf U(QoлwoɩN_5T*RRRoչsgѰ6l!HLLb|V\Y;P($U?ch4 p?0P(̾|8rrr,^4]6sr3f@IZppp1cj5S=Ν;0  TZZ#LBz :Фnݺprr͛{L&֭HYn 99>>>Vl۶ 4q8_@s8CFFUJ*ҥKg̙ ޓ\bݺutWXZիagg[/++{7**((/XqAA B^z%dffZ|V~z4hԨQ_}.9ܸqNNN3fc_[ZZ GGGf!!!zE߾}6bl2͡C@ $ȑ#P(5kȵkנ%0tP88809Y@l~9"-- !!!R_)f̘!?իVѱcGѫׯVEBBu.p_]#&&Ä6 ɸ{FR3v܉vAP`0_Ǒ#G*GFSa-[}OV+9gφJС%W^ynnniذaprr="w `׮]i!00qqq5 4c{\z e A˘8q,zݺuCrr2sp?prrBll,|?BCCٜZqqqƍ₰0~7n >>돝͸x"ƌoooBcbϞ=˨:Zj cϢ]zz:T*ϟϴχ^uͩSP($y&1rH{Ɂlrٌ "< 2_@s8ŋի30mڴʄlڴ :Zt<WWWԪU9… s޽{fKjj*jԨGGGɾٌ?: 7oƫ 777B䄖-[W_}M6aϞ=ؽ{7~g!V YYYmQ^=Iɓ'K@^ |4 RSS%?ﾓո.r)jIm9X/^jĆñ]Ο?///DFFo!fBl߾hڴ(ީSバf۷Qn]zŋQZ5[n `ȑYsΡ~P(5jT{;L&8iӦ=(SR( cŊr1BdeQVXBdPp>`\p*J*UBN$l6#665jԐ=7n rE@@7o 9/_@s8ӧڵkW>IQGٳgѠAϽx"ӧOSmЦMj`NNV_RmAٳVѤI>=!!![W^ř3gpYܾ}; `Ŋm 舏>Hn,s߿?<===ݻWO( ̙3GB[#痕aС 7dsݳg|}}m۶I>ùsиqcBпYeMAh0o-5 4|pQ 4hݫ>s!o ɓBjp ]vv65jlڴj#&O B 0ϟJfۈBI$Kf3 /WWWL>]3bijZBBׯ/iӿxyyI$DܹXcvu}999ARR\f=<!p:ggg4jԈ!y7xCvy#00PPՒ-ZF֭=&۷ohDHHΞ=K1L2e J%7onq͛74ifh,6''* _|ݚ5k,{/X&ղeKTVbg}BT4zkY/ ￷jVĆy8pz=$k-_Nz*BCC#9 !Ov[n3jժ,pW^u1ڱc|}} 6X|~ UVRİaìjV>L57K/$iSVV???^z_R oܫh֬l:tH=]]r5 4w^899iӦV!-B*kzSzz:"##Ç3A1c@ѣΩSPre2"%%L2WFFڶmDJJJJ0sL8;;`0`֬YVP>Z4XvvvHHH`ر# f͚tzW8;;[TE?~8jԨV9s (Μ9:<矟YԩS([CT*Y{ѢE I;wbwlZ3P*UVַo_FJA@-PJuY5 4|k׮'[vvvHJJXԩT*/^,ic8;;#**(n6m 8?^*zaa!B-NٷoZl Bjժ~鉈 EDEEYZoP($ ue˖{BR߿?ܐc_BluCzjj*t:Fm>γkB>>*%!111K~x\QG#ЩS>\ll,U^?rZ|9IR(ܫW_~PBB}$ȈLRb)bUdgg hѢTǿϟOҥddjjjVWE{f̘Aڂ呝yzzt$.//iӦE1u8DfQt1[>! dhhH...K.^xIٳM6Qb{UZ:ܜWf͚O7nTjCJ&L @mڴ[7oސ/͛7K!q֭@a26{ld M׏V￩z4dk,Eh׮]|JvLx:N@*~={VJm'˗ؘlmm1زe I$ (8iΝD@"BCC?\FFH$L߿efy6o\|BGGuF[n/^(QF$(**T1r್?)ӧOI**<~ D/({ٲe$Jv*s""6lӧOUXy8DfOzzzSBܺuKP>|tttӓ^|ryqi݁~Gaa!͘1PϞ=?{,X@b9vYXX)Uq),,6mJ/і-[ŋ\\.{͛חRew?ƍN>-tޝ~1֭իWOwޑu]:9楩Ig|Tӽ bRXXHmnݚ4ikI~~>Pǎ?5jЀ\N-[+Xe5qR ؗҥKTZ5rrrRi[]WWN<)h̓'OQFd``@111%>{_>UVbcc?\ff&oߞb1}v+&---rppPК5kАiٲe:Un߾MJ0קs*|v̙TjUw0;&x Eh$T.{uJ4g0V^x:N@*~]zLLLޞK'''ڵkGZZZ~Acu֤hNff&mڴ@ݻw/+W/_\aW 1bD"GVXGteUyRl}ח|}}>w]Ar9S.],,,${{{j߾瓭-nZ.SL!mmmwJq+_S' Ќ}n޼Ij"п{wN$C߿޽{H$R LR{RժUaÆ%O+zv/^$1v۷o'JHfݝ>~zD%+o _C:|1M悒%*T H%/4c_]6Y[[x[6lŋ SXXXܰ088B.Ӓ%KH,SΝKq-*U(|dd$rO 4 HD})&0ydPjLf ]|9ihhX,蟄Zj0fW:^\.'W`Qt0Vx:N@*~5ծ][o?G.6m0W&XL]vUxѣG%nsrrwƌSLٳ'NPż գG233+E,XRT H%/4c_gϞQFz*Wںu+IRҥ`RRŽؙԶm[H$?%))ԩC /ʧS@@@qCU^~M ,իD"~ѕ+W-+cǎ%##Rmhт]p!ikk H޹sgrvvul@?ǔJh"c>gĉ$63'Ph IDATd~MuT"B3̤&MRP?ѣ:dggG]LhȐ!%y%uڵIIrڱcQi킏ҥKvŗՏ9Җݼy444h޼y߱cGAыz@P<>}ZG#Geee)5cYYYdll6 T H%/4c_<==J*{|·Ѕ&իWԶm[t`ӦMEɓ>'ˋ/M*3ѣ ???Aю;I&駟~gϞ QZL666>4h rww\.'+++3fg ܜ.(+H"(uɓ'C?1Sԝ̙3*bLx:N@*~c999Ծ}{R7‚j׮-~I^^^qݐ_̌j֬|^zK۷oW9d2Zh{-Dl_~E:wL,)˗CtR?~< ~~Ĉdcc#~APCjժѸq']]];R&Mťܿb1T H%/4c| IRP)Ǐё 6Y.… I$w}Ϟ=#///РUVxlԷo_@+oƍ#XL58A?Ç233iʕԴiS@2;ڶm[8qАN>RiӦJ 駒J%D$,UT|诿UҤIf7nT9c5qR +O}%HDׯW)֫We˖E{}Zyyy4vX@ PlݺtuuZН4hŠ]B ,(%q4~xݻ~ ڴihтZ(<{LPW"^^^?(44TzfΜI !f͚ UT H%/4cC4zh@U;ѣD"ZrqdffF*tmݺtttqt޽{.5k֌$ ͞=[Zz5>XBcY{)mڴGu!_HϏOG1cח:pRcׯOcǎuH,SzzLfff'x=^"o}ʃH[[BBBTŘ~MuT"B3>a3f-,, &2eO<&ML&ԧҥKdmmM ̙3I,4|pDdggG~E={FǎŋӨQ( <<<ɉy@'N;v+_~ԢE ۗ6m*LJb A_rE/,E̙CZZZ%'j餭MT|tt4P͚569$":rmٲEאӲe˨ZjMSLKȏSiU:tR?N|ff&I$Zfܜ Ժ^zEzzzjUPP@6T'c=zСC/Jqqqx=qyA[᯿B&Mp̙K`kk6m`}cǎv:u O>-qMMM#55=VZ+++̜9^sUǏC"Sq5jԀ޽+y###x{{J>|8vڅ[ k"33SOH$Xp!bccq!b1*΀5cL8244$'''<I&{y{{[N?H$v No>Y&ƍ4i$&===1cZz~%OOO%r?֨Q#5j,YBZZZ=y$ ZJueddL&SKwt\Nmڴ![[[1*ޯJ_hƘ2^Jj"kkk{Pҷ~K"-[&x\^^^q#F"211(gffR kNayӤIH&L&WjJ~R󣀀ߺu(RrvvVzmdhhHJXߓʱ+-ޯJ_hƘҨ~dbbB.]R)Vaa!}ƌoׯ_O!rszz:mۖD"?tرcdaaA2/^222hƌdhhH)!!A$LJ4hZߟ<<عs'kx1""")>|85c}3jƘ*޽{G=z XLk׮U9޵kרN:dbbBΝ<Dh.deeQ޽ Ӈϧ%K.жmJU5%%NJ5k$dggGsΥ[n)K(\N ihh7|SRc\߮}[ҥKdmmM쥥'D"6mTTڒX,QFTv777G ccc@2ڶmKf͢#GǏYdddЎ;SN$‚[}lԴiSuڕ|||J:::8\NԸy%N c8DfӦMH"P@@r9͟?w}TW^QΝ TPP@ , TJtUAsђ%KHOO hɒ%*(,,SXXuԉ }}}rvv.]СCiĉ4uT8q" 2|}}/._޽{Қ>ֱcGܹ͛GJ%EÕoÆ $(%%E鱟2|Ԭٗk1nG!LF͛7WK#={L&#WWWzqJbZn- !х ޞ455),,Lpϟӈ#H,SzW{)44N~~~F 6zQÆ iӦԥK4iر?~?ֶT_>LRݕ/77UFJׯ_Sijǘ_S' Ќ@5jԠ (s.^Hkצ5k*u9(::LLLf͚K}&OL"<<<Ν;z*j޼9>}Z}R)^Ziii;ԸS*UJUxڴiƄDD-"Tw1!x:.c_M"..h޼9.\R;v QQQxйsg\|Y+diӦJܹԸ]͛7?sȑ;)GFj0o1Bd2L2[nŽ{T+{0WaaV 6OKK 6lҥKdtIpc8Ts"22.\3N:p\*Ub prrBXX իnܸܹsppp@`` _.hݕ ~CΝ!HJ>TjH$?=Z$bȑ|2xFFFX`Z1' 1R ۇaÆaС9sJD?~<?$4mT ۷˗d͚5KЗooo\|AAA AӦMqEAsjhh`СHIIիt III^DGG0`@cJǏK}/O) eܿ_-1ceHR^;w.Tvmo6m*ܟ2fb2ԏy{{#>>o߾ D"BBBpQ={nnnvFFFسg֮][U#** ۷oGLL 4hK ga̘17o׫HO>/jԨu֩O,7^~WjRnݺR+cN@cA8wвeK۷O嘾8}/gO<7FLL?,XGa000ȑ#QfMt;޼ytҺx"<<<{DFF@J ŕj|@@}R`r\Rm1c c1=y$bɒ%*i[.7 00&L@~~񶶶8w1emVpX\IIIՅz%455ѽ{w;v =¼y󐖖5j@׮]i&+[\̞=VΞ= kkkɁn ߽{wرC-=AF:t(VXQ!cL ؅}$99PrrrE/1+,,iӦ1bS.I*=}T'O$sssקm۶\.<6oL&&&CgϦ7o(O?D$9::Rpp0⧥ٳĄ444hڴi;aÆkǏ5J=? PKKK#DB+VP[LƊ~Mu""5ʁ… puuErr2\\\*z9ƍ1bj Z={}Vj|VVƌ;w{Xf W.xׯ1o<,[ &&& E޽!~ ##'NSN_755P^=XXXR $ #==w˗WB&_~:u*T^:uDj1rH@GGG񅅅033C`` .]Z5|J>}c55}3jXeRjjZbI$ZpR_2DDDB]v%ԤIV:"?={PHHSzH*԰aC4hSvvK=ܹs*5f277R؅ EDD-&cD_SRpFn߾ deea߾}T9fAAf̘0`JaIOOǰap 8K.U:Fll,&O$t pttT*2 3<{ YYYx r9455QjUԬY.5|(''zzzX~= RömзoRňмyR6m ''ΝSW c:cUȲG6muVcJRO8x bbbdbСCذa ;vLHHH@xx8nݺ'''S*PRhҤ ڶmt:u7lll-D5kVUV)ܹS066޽{KS&NRWb N@c)TZ58q}ŀԽ#>Թsg\pժUV^T5$HիGǎ1x`zJ={č7rJDEEA9r$?~\_ĉ066JqUJ$ t˖-Sk\Ƙj8a1&&~W,Zaaa֭rrrTkee3g`3f zׯ_+X~={{{/UkjjbHMM￣nݺ *Uʎg| B?֭[*X,Fpp0'|3cqcL0Hɓ'?ѢE |''RwttDll,6n܈ǏAXlK>sss̙3>ā`nn`" jנwaСXp!-[@/*%JUK.HT*Ő!CsNdgg56cL9TB&L@.]?ve1Ƙ VVV/uYfv-peW^߿6b ۷ѻwoL87V7RӧOxb<{ Qv-[\1JϟGfͰ}vlٲj#77?_T\c}lȐ!Exxc/Ӯ]Ϟl„ ?bFrr2^ cL.I$Q.](++KlBUV:uЙ3gTL[JKKSBGjj*-\I$H$MҌ3ɓ.\@z"HDNNNt%QdҥXdhhHgVKRf$6:~Mu1XD Cf͚͛jݿ\t jՂ7f̘Rsqq3g}vǣaÆ>}ڎX[[G||<ӱaXZZb͚5hݺ5`lٲ/^TgE^^1|qqqXf Ο?''',5k%X,cC ABBnܸQ&cT1… puuErr2\\\*z916)))o#lݺ|b ,, ?#}vٕ:^NN°xbcܹW˽ Ez/^ӱqF%އ޿333 < .T{|8Df}ɲ1h ٳӧOٳպONNF߾}PTfÇ ;`oo O-Jׯ֭[HMMŃL~ԭabb:uM4 455tsrrBVaw}ϟX-1?4vXٳ>,}Yx{}2ƾtrBCCI,S(33Ssss)88P˖-޽{*LLL-[sΩa_oߒT*իW-۷ EEE- ?~L/Tw@cHiӦ!22IIIpuuUkc8,[ 'ODZZ5j_~EnnnơC_ݺu;#11pwwW[Lkkkhjjǚ4il߾L3J cr׮]ↀشiZj W^E^0bѣG'K._wpp@~םщ'`ddF-T*EqmH$B߾}o߾Œ c aii3gΠ_~|y*ʦM+V@[[L000deeI"]vѣG<2' 1*%oorڵÜ9sPXX9tttdd2xzzbԨQjD"j ;wлwoZ VVVҥ 9%''#((FΝlU3|Y @vv6t cJǏ̙3?ϟ?W}:RRR6WYy&:vGGG\L2?KKK.X91X&H0k,DEEիprr*#Ycǎ͛7>}]vj/#F… 8<:wUV3ΩgϞ7qȑ]lWw!5}:dFD={s .}M8a1U¢EQFظqf}`nnkΝ;ܹ3N +++,Z7G\\RSSX!;;;ʕ+|2 cJ˖-qU|2dٳr +ܹL>3gNqݪUPn]TEZZZ׀c_===lڴ @BBQn[YY_~ݻwѻwo&MǏmMJJ """0~ 9~7*d~ƾ0juׯ_G6mЫW/6~gܿƍƍaee~ҥK嶎b055Ő!C*l fffR!N@c}ժW~ 8y$gϞr] ϟbѢEEƍѺuk8p̫vUǏݻ  [H$Bzp [c_:N@c1={ݻZ*ƏTDDDݻwoP^=}M 3220x`k}nݺw^E//' 1011={sN6mHTK%>|X` c1H@ܸqZB`` v튧OVz4i-[ǘ?>  44_}i}ٳggUS6_LXe c1 ߰g;wvvv_+lSZzuL<wAtt45ksڵÖ-[ Y[iyݺuþ}//^R1SSS";;cݺuÍ7еkW 6 mڴAJJJG$۶mógϰ~zc066Fnݰk׮Jܼy7Tccc/*x%}8a1022¦MUE=44.=== 2NÇ1o<<~{F5бcG]=uy-ϟƍΝ_E/022nX1m۶v?Νej׮ɓ'#11GXXrss 888`ҤI8zh-z VZ7nU6P"1o c1LE!)) ѣUK+fiiԩS@xx8N:M4App0v܉r!//'FZjaܸqױp H*U81OZ `17nZ ӧO޽{tRիR-bhh={gϞ " &&OƑ#Gb .aooԭ[0551>_.իWx!RRRp$&&"..o޼%0|pXZZ^jE?+x%}8a1JI"`ܸq֭Ə޽{cƍǍ*H`ذaL$''ҥKv._z]z}A$tHISOaYkTQNml͹(wٖ:*VlrJP ܌$Avp+vެ9??;};'|ޟe˖ݟ544SWWw}7wΎ;IcƌIkkk:::֖O<_Q__$C Pqeٲeys嗧%W]uUoCP_ikkK[[۾}k7|3ӟo7{ͰaА#G)ǏOsss+nZhdٰ̙aCp Ytin̚5k@}jillLcccN<ĪftPK`pr:G k~L2%_3cƌ UOzw *I_2>`6mڔ,\Ѯ;` pj̚5+6lHGGG~D~ӟco?&IF]` /38#_W3uԬ^i===I; 2~tvvfժUIiӦeΜ9yW*^e˖~_ ?kfɒ%yr檫WOS^|477W=-_M6ꫯ-ܒtM] 6dʔ)UπAK@=,^87oٳ=w\n7W`Ϟ=yO~)0h >oϺuҒy極5˗/dnue׮]9S~dʔ)y衇z1"3fiĬB~f:ujS` }sˣ>_iӦeƌyꩧ6-_<ӦM!RSZ-w^,[lɧ?̚5+Owww,Y _B֬YS;9眓>)0 z+Ν;7sέhɰarg޼y{}/zj8|[gZVڵk)#vQњl~+ys'W=b޽yru駟Nkkk.\|+65~3fȫקy?\]]]fϞ'|2+Vȑ#sd;wV=_o~˗kPw Z-mmmyGSOeԩ+3~|֭[/g?ٜUρ!A ښμK׾n-'NL{{{~ykyzB bsLnl۶-?aÆ~iii7>l}׮]3yZ*{or!UO!MԤIhѢ]6۶m7ߜ#GnN:餌3&sߞ7V$zd„ Yxq.lܸуV``;vl,X 7=X~_eʕgϞ=:gr)STN:uQlk~8˖-ʕ+АK.$72a„umÇϙg3<3Iog͚5Yzux|[o$ijjc=6&Mĉя~4cǎMcccͶm/;]]]YfM^zg[s8 )7vX֖ӧ^~8ٻwoz{{sl߾}_$ICCCZZZ2sL>=guVFU{58`jZܜٳg{/oFnݚv_ٳ'93bĈ5*GuTƍ'c@5W2f̘3)@aN @1(F#@bP @1(F#@bP @1(F#@bP @1(F#@bP @1(F:;;Z (?)ŵF)5P @1ê۽{w矯x C;U (in_Wz뮻2gO,]4ͫzƀ$@7|3+VȄ Pݻwg˖-9s2z H(M@1(F#@bP @1(F#@bP @1(F#@bP @1(F#@bP @1(F#@bP @1(F#@bP @1(F#@bP @1<,VZIENDB`bayespy-0.4.0/bayespy/tests/baseline_images/test_plot/hinton_z.png0000664000175000017500000001470212471107140027210 0ustar jluttinejluttine00000000000000PNG  IHDR XvpsBIT|d pHYsaa?idIDATx1Gڀᢚ G z6w^EVKE:o!ue,KdS/3ޙm5dn!@l @6F d#@l @6F d#@l @6F d#@l @6F d#@l @6F d#@l @6F d#@l @6F d#@lٮ=Ǐ?lO?4{?zg/N |1|4?_ 03w_5@~Eq }>Wu]os_ "@(v'1UUifYq*R [۶p846c;|Kp>}Mw81\.&npBu~RřN}+N@ LwJc!\ wkno+ $ $w> $]  9; @ 9; @ 9; @ 9; @ 9; @ 9; @_`f݆g8"@:޻ >}Za" qCu7xn.pnvFx-Z;,E0c8Dݶϊwvozy v> -Q>)ZMvTv@~E$@iz\ώ  wk řWUUilV 7\Bmۆm!@rDw>;|Kp Mt?(_ [;!GřN}+N@ LwJc!\ @qx\ɹ @ 9wA $1$Çf; _qq  $Mv@^%@H_r@r#@qfb %@(; PbZnwx-|~;`&Q>{e^sG@r#@!@T,;"sG@r#@Hb2.ř*4M6͊S8mJ 9wD $1(N^8{!߿< ̓8{!5$f w<1B''@HBΧc<9@qwDNS> @qwDc gCRUUh @(έmpXi"R $7&@Hn_8D@q g|}IHMP|ϙ' 5@q.ww@~|x\x\x\x\v u]qA\]eu ONPqCu_>NW Ź\.x@qvg97?yR r|^' 5@q<.@q<.@q<.@q<.Hn݆g8F @H2{΅t''@(8뺯_N}+N@ \.~Rr<0 v@8B4<@qn퐴mJ Ƿ<7@rB; s@qx\3v8X`K $1 FPJUUilV ŹҶm8+M@*;,E0csG@r#@(tt:~_q*R gr<0 z8)[{v{[w nw w G^ Fb Fb Fb Fb2.n_WU fũHAPwomۆD"@Hnuw >}Htq-<9@rv8B:?xrLwHNS> @q;$1 SC$V!@Hn۽!@H|>!@H.Q> ;,E0K $7;M|!@H 1#@R3v8X`K `)#@H 1#@H 1;,E0cfp"@H 1#@(t4Ml6+N@ Ai6& "@Rɹ#@ 9wD gr:Ba߯8)3A9ap^W 3v8X`K $1 FPJUUilV ŹҶm8+M@*;,E0cf`)w@X8~ũHAPx 0T @H 1#@qf`);,E0cC@ 9;$ļ[{ʳT^K0%@9oz >jGp/"@R3v@X`K˼g݆~]UUh&l6 @q///߽mJ`fuw`rO>=H03bGE0sk#"@(tt:~_q*R gr<0 z8) %@qf`) F;" nx @q;$1 S 9;$C@`"@q3!*4M6͊S8vHڶ aHEb! F0K ̸RřN}+N@ LwHc!\ @qx\3x;,v{cv@~|v߿< s>o9$&@(rOP; KP; KP; KP; !Bfmz>c453u]{u.p@qq ]}t:~_q*R r|ařTUqskmp8VT3o"@qf`)qq C@ 9;$f`)w@X8B4a٬8)skmp8VT! Fb3x;,EPt }߇~T @(tx gCr<0 z8)[{sk u>|ۇn.[Vws ^˯`03MZwx-Ly^/ܭ;"!b&@H2{Χe<9,E0K gCRUUh&l6 @qn퐴mJ 9;$C@`"@q3!9NW ř0 C^+N@ #@H 1w<X`"@R3v8X 9;$C@/???Nx<)''DU! m0 ^׵`]{ׯ7M駟f/~Oټ[{y d#@l @6F d#@l @6F d#@l @6F d#@l @6F d#@l @6F d#@l @6F d#@l @6F d#@l @6F d#@l @6F d#@l @6F d#@l @6F d#@l @6F d#@l @6F d#@l @6F d#@l @6F d#@l @6F d#@l @6R3.IENDB`bayespy-0.4.0/bayespy/tests/baseline_images/test_plot/gaussian_mixture.png0000664000175000017500000011047312500307603030750 0ustar jluttinejluttine00000000000000PNG  IHDR XvpsBIT|d pHYsaa?i IDATx{|{ ]-&r%>!c5:D:(T/TjNs!9MsHJ%,UJ~{z߻]/彽cz_Ö-0(; LC`@4!0 i LC`@4!0 i LC`@4!0 i LC`@4!0 i LC`@4!0 i LC`@4!0 i LC`@4!0 i L`u ==]WnWPP SN)--M]vUHHx%Yz.3g du^Av$t&M-G}TSN _׷mݽ{_ 9I&j޼@V‡m~}_ _˗4!_׷m~}-;;;"-Zh۶m ,z`@4!0 i LC`@4!0 i LC`@4!0 i LC`@4!0 i LC`@4!0 i LC8)--R!SD{qIKs<aEU aK3g:FH*Kv+@RfԻ70 rKMujm=.: t| AWii5{K :  k@ɹU=pJJ;d8wJJ*| # kv;!0 S<У>Ux.**JQQQUP+>>>sǏaζ8Eڶm7onu9ȅc @4!0 i LC`@&ۜ:uJ?~'駟~RFFBBBTf d<h˖->}ϟ3gHTn]U^]9ӧOԾ}{uA۷Wz o>}Z~vyiF 4P:ux:qℎ=#GoզMfM>]T^=uAԫW/FHe  rJ 0@!!!JLLT=m6JڵӰa$I|S_/{h}teGՏ?XA ӧkT|ylK!!Rbǐ5O?T޽[ÇٳհaCCWVVVqu`?Vu5T)3uf{7ɓuA}:t萺ukFӦM_e|MݻW;wT߾}spɹsV`<VJJ6lؠp9Ru#<}>܈,_\*TPn݌?xZ.-5lxLOw<ͦ:hJKK#|X_^G{+p???5mԸ8*f#l6?ю;o(!!AW_}|M;wZe@P-[_|ҟ;&&ݮ?JVԠA4rHjJɆ ' 食Y[l18hQUwyG7oVVVZnxm{nAJoTZ`N&4+,L֭SjΜ9jܸΝnYC@.W>}`I #Fh݊Pttn}wm O殉0YaAԩ hժU駟gyFN mEY;FMÒ,iVX]jΝz/Ӛ5k,  4 =  &hΝڵ"u@@Æ Ӂ׽'fiذ>3Zh:}V2( 7ܠ;^sk1 ?R<.̙{'{ftKe}A|K.رcﾳgK!!ngΨͦ?ݧCY](|Pu՘1c.;J,SG|´xbT|yM8QK.͛.KU];ܹS7t맡Cĉ0 J`РA דO>l\iiRzԻ԰1=]K `5kL_L@Ӌ/7jV㙜]͜)-\(}qL)6V~Rllo߮o^}?[#@ uU;w֨QrCR#lnw<$I Ն O聅 U+}{ϰu/ fӴit Vybbjnw@M\e'&J!!(1cƨ~Yns*LKI&q PJ*T;Ｃ$͚5r|Y[aB'@]tQTTqedd8BP=sm秨m2^.pG}TUV\TT,@QkjܸF>؃GDu[+9jkFZc+Ery=VeIae;Z~{=W?!>>^9;~E[6iѢEjݺvsE7xլY34P}~cY4dmٲEYYYjٲΝJn8ݻWSjjZn_gVzzFau9A)""B[lQrԦMZʴs G7+o?֒%K.L+yfo^ݻwo!g .EFF{Ç.LR-[QFiȑ{o (6fiƌ TLL.L+>Hqqqҥӭ. 4$$$Dgg}ɓ'[]Nr]wiݺuڽ{Zjo"n7N>:i߾mO>$( `{N;vuQ ˿?GZuӠAmڴI7|nv,N@d7oΞ=[$""&ΦVTv+W֒%KSO'Pll25g]V/5E8JK9 /hΜ97lh!H.]4vX=3Z~5ERH(1$ħÇA/޽{ղeKر@B=nFEEEYw>5Ut| iӦUzukN. |,y)33Ӻ R`@eLzauM/S<飨(s;s'$8F<94oj) 7T5x` ȍ^bŊZlZn=zhÆ vI !=JfiҤI]z!>|Xqqq*_| [3e.rXBSTTΞ=knnXKa9ТEnMoG/ԴiS-ZHVҨQ=RvWk֬QJJ:ucǎ~VL/յkWM6MÇW_~؜q-w!88y4ecEǎ]CZvׯo91^רQ4rHX¼iG_AIL_IK:Α89nD4pc`~0/|a'/7l'7̙3j߾;ކ^_VϞ=5`m߾ݼ8wNZ)Ssznq'5wc<)>myjƍZ:t蠭[<- 5w\5nX=zСC9q:9.y_iY7 ;[f~}qn 6#*nSN_^ԩ֭[s#T-[&ͦ={ĉ=aZڅ :OrL4爌*WvvZb٣+ym4'׵ :_Θ ][rv?{V*4&v*Hʕ|rK}G}d@iɒ%,O'1 }i.s588F@+Qt,>/HC*66V /J5:[tEgVTTj֬W_}U6͸qZcV>9MK+t)suD3=fzTzu=cФI="t]DB4`;vL?jժ'|ҽ'tQ-hfL&Il}#H  |C=Gj QllN |[Șs,6̔wL!|i„ :zowωuHC%\;SYYY;w.!b:cVfi銌ԀacO"v@/Eiҥ4hv1@xR63gڶm={jǎܹ.f B{&yCR6*TPbbJuM7r^RZf${/m̙ܹ6mZbcc><ѣ5rH=C3g/%:GH)8ؼ^1n!vf{GUjEEE)**ʢ$M>+WNQQQ7o^z饋֊:ͦ)S㊉fӠA.\/]CO~ϻCQF|JYCdd,XhРA;wn?~ܢj|-D)))jѢmۦ͛[]f͚/ }֭[ղeKZJ]v5BϓaÆ)..Nfj7 4ſ#.bɒ%0`9sBpVz JLLT6m>$E}Z;y4!CHw~7v;aY>}ΏH" NȮ$!$;;[fSTTToε]yFspOKP4LGX۸߯oPxy;C~ժUKcǎ5(!rY_[R7x$iׇ~(ICً‹82 iZFpѣG5n8ըQC{q@1@ݻwj޼ySZ55mTׯא!C η+s4U1F ewƍGj Q>},@ESm&IK};j͚5>/+rqhSQG2M4}lz7t1EEEiժUԩ%( |JJJ6lRcǎ>}>/܀|SBH~V,R/4-O<~)i֬Yѣzꥤ$C:)ZKǎ%I6l0xB_xb\^iZ|R*W/^ƍ-@]v{.Sݺu}vߐc2wWAIlӦ;sZb:t蠮]jӦMˬ. @ -EV;vVsh 4H_RԹnQF YFOVntqKPF@}I\ĩ?s !gvW4h@k֬сtV {TڟQ#cBaaa:x~wrXpt)-k VIص^O>DɊٳg. #ӥ^K/h(ԟ0IΝ;/!qo,hY״kN-5|pegg[]"t>c߾}EpUԟ \rڱc]PtR|y}裏^xCQ5=H5sLĨf͚Y]E3J@ V&M(ժ5 >_WQF)##C'Nf+^r3I1ِ!Ct1=YFauI|طoZlY[u]orͦɓ'zz'iӦɿ[֚ ѣG5rHըQC$>';wN?ׯ_[u&M駟*;;q%㏫Zj{k֬Y*W\`F0(b0e/;U^]]v$>'9rDgϞUzbLiܸ~w=zTj22)66V\rԫW/%$$RJЌ`P`l6f̘cǎ;ЦMԴiS# O8x$}$&& IrHݻKQ!\G~6nܨ[om]…; ۢLˈ+44TݻwÇ. Chh ܹ>s}(׬`P`ZT\Y˗/WVVz쩿pAUT=@\r w}g˺-[jԡC/)uJ:uO>w}h;wx9pիWmě4i\s5ڴiվ}{nX;KRf4|-[LG^'iJ)((]|lDG?|PW^'ddd($$moРu!RժUj*uM{̙3wh6DΖ֭s4jx-{Nnݺ;Ю]." ^ێO?p ҢEt=hС4iah;!J#"?x !IוW^ݻ_-~y#zΝokĉzgmk41"aCcFƅL 11i~:yd7(~#z1kŊUF l6ƍ_3fPu?F"v$9w` tj~}sbbb.R(KOOAw^4hp!3l0%&&jʕE.\aD^ ҤSnjrם=/0nWEZ~ F;!A222$))XymР# ѣ>s}7?k4uDd _#"ο7wOdF&Z@pk@x=g)pw^ׯOP6mi&8qBm۶7|ZFD 7)s3* +Kw$'~wC^9`UV7k q~eߟ7n/RժUSqƜooF$rkLgr /#Cg53X^/##CUVUj~JyAYelԗ2vZ~5k.](11‹VHĝӥΔ+PMN)Q53XVQx TGpy-I>:6|LlX8x`WoxLjC~vk~?ӥ w3*Μ;՛Fi_xQ!e+ aJ>g/=jݛ'/_^רQjݚ:uXM[oS''C` QpBTVM*UrlkHwջﾫnhZȿ3u.=|ӽ^T_yEh}ƞ#zN*B"le'r}i͚5JIIQ6m?X]Ō+TGDH>!ڂI6Mvz?ôs|^)Xy)D 4ЩݻuI_պuk].0zBsDg=I,4^re%&&رc4hŋY^)Xy)D_z+vPڼyڶmnMӦMSvvElJZr^z%Kjy Zvvvvʭڞ9a6gmV˗kĈz5|pe:^X9]*))gp1rO֭ƌqiÆ 3@x?SΝ+,)Ů[Ϟ_:}/c= _SL|3g[oUFF58K5Qk43ڵKZvGvP||2335dփeW;y$bŊb fs"^|EiΜ9[̽5fr9s A2:uJR np^FJg2d֭[^-[Ν;R_a&փeW+\ ֬YS{23ݿܪ]vJNN%\vi=0z7Z]@x5g *⤄B^ ֨QñúNׯ7K.ի^~e-N5Ź}նm[EGG?,xvI(й(""O?yH߾N:núN*W3FG]wݕwɲ&Ɖ 4k,ozGJ}<ށ???M4IsՂ t7_5$&(ʌƉrlo*..N/6<[b>V㹨(EEEYTJ@ӧ23ЋpQu>hPKZҲeԬY3cQ}qqs7^X^8bbb|r{j۶j׮m񁒊W|||?nQ5⁦N͛[]J@OY.Oeu>8IBWjժիW/o^gV>}J`ȿ m6{=5mTZrl6+)))jѢE`jN*y)5j(BґɓGխ[W6lPշo_=s:w\lFO2iaHHfΜիWw9xVR4QfIB;*V hҤI4iv#Gfl\`T1qmݦaÆr#j'O,=@ƌ^~9 y,Q$Al;v֮]]vYfJ*Y68 LjժK: E# E9g^9}*F e7߬۷qƺ /#'`+*U裏>RRRMVxj'OTʕ/lĈBQ1uH%\2yd衇r_wz jNx .8!-]')q4izɹQ9󹦌T^]vjР"""ꫯ~Sp6jNq`=Jʕ[oiʕ4(| W;sʕ+w ٳ<RTTιpѢŅ96td#(UT_֟ nݺZnF'xB{nuY(z^zo׈#@f"jYYY_e7 f~EzI);O ?/:uaqp:ҡCN &@z饗l2mذA͛7WrreyR.뭷tq7έe0Wξ88/x"#%u甬zusG }6aEszYڷozu !Qׯ &hںuK~^-H\qc ڵ/g[/ms]_4e4h 6#2!F {Cꫯfb[Gj;ϣwoAA^9@PTW\q M0A/:t.Aqo^xAVnu9J;{S y8[gE%qiӦMPfnz {>}uz'eu9{yH;ur.\b*UH"dZn۷;԰aԷo_edd@2l6^z%m۶M V Wx8ΫSZcV>9/ҮĻ1Ҫ\>-^XIII ڵk~{u]cƌљ3g A|yeT߼.ҺwwyYǣ.BݻKO9[飝;wuh IDATkխުQFӅР._Ծ}[] b >.bXpf͹ʕ,PgF]VZSjjժvQ 2nR6S'-%T\ =i 3!d Gp.PΖ3QHa4rH%''KZh &")>:zΝ,5MIpo ȇk?""L”~Z'NTV}vr)6lP={kVeeePhhwQS=\/xV8|DF:֒jSCTB[/_^?l٢llR=oLӷ~5kָL!jYYY:R]Tש .6#|8ג,Y"EEj5o\;v^xp ڶmeoFhBSLqCqFί).jKs͚5K7n7|2_~Z~~WKm]W]TT̹!ir{rvu=;txf_xͦנA4rHj7S޽%KX] |@x;NTpFF/vAwNkMGRӦ9wjoaK/;Ｃ͛7+++K[CCO#Fhݺ馛4x`uܹL4ݻrJK#AAwA }rXv@` gN3G:u`ZJ?5kÇرc)BBBtUW)#GuM一{ iذ##3f8vB\v@` gL3,@׮]k.M2EWhhL)iǎV ^-00qUPt)Gc¾}s>ߡC޻cL p;6Ͱʕ+#GjϞ=֮՝ !6Z.\||F[Z d`ԨQ(((@NN~G\Q\\lc8[TTp zDJoi WcSR֧& !PWWZ _)? ;>>ƥ3gxF?STP}.!jjxCPz{22UUhwDr2O*!6|}WWS|}}!VW+^Ɵ~br=lL-ڃMɓ'w믿E>iѵks˓M>ݐR " 3!"c @E\J/Zݺ|^SX=qqXc).ƧjTX!Θ1}k+#|6%xM(/,D6m?R߾x;%jj< СC:t耐x{{7YDE;~VxgckcUH>/[E}1!ȫݦNR^Ŭ0n'N@EE*++QQQ/b,p{KKq_U>JKàh ((ht30(̙3~| ߔVML xm^@ΫVv ?aaxoY"wx"=jף UUU +5ڷoo*;vD.]ХKDFF"<<9*y1uu@q}4f"s|^AAA+N|=~<~h-qqӹ3ss虜,TADѵ ={ʼn^‘˗QTTcǎ_~}sOï{yyCtt4н{wtСѭsm-bo*! Q!!GeU`` \&sV%%1@a!T))*/G7$%U@}h__>OrKÕی^u+Wb!.\*TVV6%Wwqu5%n:tJt@r3QS[쉘~]R"WjUռy={6w[ҥK# DFAA󑟟ݻnz:*H|6~7HM*aCWn XJ&_Zo+zI{$JJ./OGLLL^FٳgqɆ[YYY;vɓ'Q^^i@uHR;v˗ӑ.]4"W>Lb^~F xbdffǘ1c*jB^\="2j8zA?7l޽; )SիiNxi4HƎEO> |۶s1!ޗ-/7\Kr˖=^-WCjke-} r%eÆ=>>> ADS._s?vO>gDFh 1/p(^ϟDt>22^Ǐ;<RB xbq,.N ɚW!G]^2ۅ>\p(DmJ92ɓ'OnCmm8|XnX`xGDFF^^^ /I~~6mGg}V|bΝy=ﻩגr&քBՊ;wl#ڴi#yqAM̙3_[.y.َ45Z|'bĉ" @_[lb*XES'&M V ѳYYX1dEpp %'ɯBH"4Th!ڵb`ݼ'OBBc{i +>˗/#G/BlwL R];W_}U_^!kgtIEDDDbȑ⣏>Ovt}PQ^^ҵkcFMzJ bРA7ǎk@\l1KNL}̬,2HR ms_]۷!e[ zҶVeɊV VjjjD~~裏?. &ڷo߰bru׉[nE+bƍ`/+Wjkk,ӅJXtiY;[l }]]<_7?hwE ___"^|EVWM(韘_mG\j^D}lWs'Q2HN" @*;9m[kQnRx2jEqqXjxȑ#EpppCP kf{̑[)++oHKK8qX|vz`-M&Z&51q#&E޽E``xlbdoa||7ukϼU{s'M`-WnʊwU+0 jo9 YYB趨Y`zq'xBݻaşg}vqˎkϿrk%%%bb=X|(--muuubbСxꩧD]]fO$|v*!hIL{nwtEлwocŊݻm:røJgK2yj+-Xa۸JxyAA+W ||P"TUU!cʒM4}l%}vcr־6aӦMظq#6oތJo#F@ff&233ѵkWǝ2E7Bʱc_cǎPXXAZZѩS't ԩڶm'N%%%PꫯPZZ!C+222\|v,Kfy\xvBTT*fM|;P *)O,;vī~]cz({DPLu N0x{[mlNRc ^^l>\I̕.,`]CAV"##q{E}}=vލ7bƍxP__=z`̙{бcGDeQ񈏏̙3gΜw}?v܉3g4t47֡C!33?0OZW/H!z!{1}ʶ]6I\B2Y\jԵ &<\njV&ڂdn^YY2ߣsg!h^de{j\ BDDqmgY{}v[իw!6m(XJW<@`UZV?^;w7ϻzjt ܧe)ڷo#[e^U~k[B~\- + k `z 3S|<з/͛ByijVy(}|l)畚 l,2p^55@@^yrhzE$=%y+ II^YIh);cĉOp Kؾ};޾6=};wBo}mV[/.`}O}5|ѩޛ=yN'""aB*ykӰ5;;ј3ggXAȘ ̝ |UdGU̗XVP'ѷk صK67O&cGF_-o3Gw"Q׮5 yyL EE}99(C9w~=Y8iv &k!"cB䮌U߾:u*8p~;.\dDEE{QQQÞY?r2){+fe6oג$gfxJrb!̑ ۷_D^DD({#p92M֞=Śgލ_mØ9cammxqQ%Ixf;%r7[vL5{< Gѣ1|aӦMظq#-^/*5#GĨQ?}_KKZBҟixePP+[ ˼^okjt`\00Oepi4r5kJyތ 6̲yy__Ko!Yq=K嚓")nuLwMGwEжmǎ:HDDtQn4صk7o ŠAģ>*,Y"8 4ݦov~NJ򧃃W2uSÍ6W p$K>3{ϳXhZ'233E, v_?^Q<l"wTIƠA0h ⋨={O?᧟~–-[pB@PP郔)))HNNF[XV Ȝ6mJ検>mجx-Xj+ ҂U3>Y+WZF>}\ 8k XZ,ߨq!]#1d 2j`Ϟ=سg~,Z* ѣuĆx뗓ZvV`j5:\* dQS#;WUBC~~@s%- #u{WKnQPPXr%JKK[o'Oucx㷪gDDZ#F`ĈS=X IDAT ]|ž={w^=z_|Z-ɰbDG 41>qtWuj*9yAUR"{?|90m<dއes^[[lY5ŋسgnG;wn:nA+ɿ |9"2}o߾k4+ QXX/ǎE׮]ѭ[7$$$ ** ]tAl݊yc5WG]jW /O"RWxyƈ55Z%OF[lҥK |ƭ[;<| ??V?RRR0b"9Wĝ߰^B/pAAA ?n֭)IbB@ѠvFԠFK.>@rr?t]>ݐih[-& pL&r.V7 셇$Md}!ƵIDs 1!"fᕽW@_O'M_,1x۟e樕rҿ$+{F, @O7?㸸ìQ۹sq`YRRteo]’樕 ^ٳ8^Oǯ?:3ٝ_0ȹoZ){j=%DJNUKLwBw`l yT3\n G]]~wۏt6>WNADD q ypsl`th4smDD q(o.^s96(//+"""+p,Q`YyRػ>"܂EDDd DQbccm[#jYy>"j܂EDDd&GEYY4M˛b+ DDD-"(111j(++kL7+4#R^^5BD%66|m5& `W@ȣ0ӌNz @ȣiaaamByy9ڴiS1!"^*{;{,6>'cBD[n8z/--mEDDD->ޮc]KGѣ|||/uܢ" DDD6`BD={khPRRIDDtq~egg#;;E3"j}mǏ+ DD׈\}oŴiqqE2 RRrx\ ?)s'""T\!"6l0|cZQydQ[[0?F @ȣ 6 +N:e{:u] C}}=l<ڰa*HJ rPPPooo$""f0!"ֹsg$%% j0HJj/_^z! )&""TLB'"gT2N8߱7X:T_]Qԁ8k""""xÆ QQQJ_Ce)@ر1t(Љy<%d+}AzUڰǁq DDDv<^ll,bcc @ SC%PXxyy!Du"""j"&4D >Xѣڴiiy DtM1bvލ2@S+ >&L(//hEDDDaBDׄ &˖-=DŽ 8+~Om4NC1!RSFbVdGw}7ۇ_~E`b"RɯV>Y>}ƍ}fDDD9aaxJnܴZoaaaXt)HH1cBcPT!1uԉ<"}98닩Sbٲee6\#|,;cƸnDDD1:=yiӦw+WbuܹW֭(,G}1cƠ]v6`U/_Q ͵>>W\q^#Go6-Mu Ν~<Ӯ2G njZ ӧ7;G-[p!={gΜiUWW#"GA@tt4bbb41:t%Ǜ7on\S̬x0`rJu*Dp:բEfZ^^-[`ؼy3JKK닞={"""8p """S$? "3/\DZvZ,X* =z4F48-!ۤa̘1xg1_?j> _5VZR!B%IwFjj* пWOǽ(թك˗c!@^ѣGcذahӦMcp[nŖ-[e;w1zh}݈@vTZZ[RRHB %eeH+َ9 :W߫5XB`˖-@~h"/Ɖ'~[3f.0n<8}.0n<hWll,f̘e˖ػw/^~exyyaܹ#30^}F˗7l*//DzepI|ǘ>}:"##Mgƃ^^^HNN3< 6 /"V\nݺ!;;{eڵߢÆ Ô)S15ݶoG~B"""P둒"rA,, ݻرc=z46oތ#GZ~ځCCC /O/^7xCff&HOOuWuc\݋ 00=z􀗗0h'HDD䙸BZ- p%.;vШ(޽6m¨QZ~R.[Ge˖ԩS>|8222PZZjc믿^DDD0?-[o#5O~~|x{_hm;񠏏O6?F>}tR]Kٿ?<|I:t<բE:-RRdU T*?øq0m4L4 Νs1 r{<~<>[MTj]HJ,$$K.g}<6mr1rkV¨QУG޽Çm@һ W6nƎ%;8I&aׯ;DDDDￏɓ'[oŦMйs `ЕݱxU]+һJu,[vYs/裏ƬYjrl""""gc^rKǬYL%uyq〘``޵7~m駟ɓ'.0!YfaΜ9xWT4`&@JK +KVIrĆ'|]tAvv6d%"""ʆ p}a̙ >uu 2hh4 @7"44ɓj`dB""" @mfܒv<Сt 999ŋ-zNdz#""""k0!Zp!Q?p T'U]7w2.W imSgȕY h3~䈼oVWUK?8_kȑ#1m4̚5 gϞmsMO!8""""j) 2_|!%%@}q"xsW-=6 &W@T* ;f:fx<'&瑘e7|^^^xgz~#v Zexe}\|/\i/{e}9}7TMeˀA]d5_c)y#L5Tƞ8plZ>_qpaaax裏GϞ=ǀ#""""Kq\xw0m4_hS isۦU@&+x[rޱM`Y;`2eƌćs'QxUR @%>S>}&On|Բ s'M)[!M=o惓-x1nZX?oq2}K#""""+q Ŀoddd %ooˮʛbVMȓzc)_ɭ_B@ڒ(*me&y,aȚ5Md&[Qk'"""BNw|w:u<3>ֿ*_on\SղjƁIII[zvmƯM_\e'&Ž;uBE嬥* XaBNCRaŋ UyKNL(ׯ};jlTr%`S22ƛo ƍkM?{u-3dQA"""jW@֬Yt 2+`x-Yi˓۫d1ta8Y+< ]M$''#%%K.md=)&""r TBܹ#Glܿ׀3a閥椧۶VB[k;tlD ehZ ^8Ye[ȩP]]aoqq J[.Yxm6k4(~४*۷),vBDDD:\!*((tUfZ=VV&eYXMǚoj9r_~QQQ =fx+ AADž۲=4jiLn̙4hviմ VV#""cBNUZZ:*q3|g%|MIL?Wi@{(,,y22DDDrӧOsg7SM5-ܷHH0|=caY;㙙2]Q 1!!}PTVCFXaSUVVk׮Kwؓf`8`ݺ@{|a`c8_?A 5UMmLHH@uu5*++ѱcG """r&n"A``0 *%w5@C`\*mۀ3g4Cy6nQVΝ;gDDDDr*???)hj+TP#CvлaIѣe7ϝ>]>w޼%vA{JSlrc97O#88ldgghF˗/wPkl(Ӂl`*iJU[UU+>%/D ~Jұ#P[++l9=%""""77js0qC ,@]= Cmma\+2ßi7$hpٳ1vM87W@ݻw3cBNSN9fp[:'=DE5cG :u֯}^/^5!"""rcLB'JHH@QQcZ-ƍU5kw0W UW&;Dㆆu\""""GaBNR˶ d3Bg(JIDAT^*0`ж-njWVu դ*DDDDn9Unݠj믿6JIrҟnE#^IL$HO*+e$䶪= ;믬*X552 V>>ؾ};z/[Z*/;T]jL@W/}=ҽ'? W6MVƻVȬ_oyEڽ{7k1+ Tm۶7XVr0O=ȯ:@$&>&ǎDY?wD&?_/rKX۶֬Yk=$"""r# @鼼p]wa…xWp2׼~))<#OL~k<`8 "8}Z&=n@?hsV0?g-c?\(ȕsW_a޼y߳&m`KL6 R^ \S?Ւ`hYFTdDj*6]e+ZdȦ~99||d欿rr >y̦Ɉĉ RDDDDrݻ#++ pwD,U|eA|geɊV,{2|lJd/\A̹s$rVݴIvg̐+&l33`rhŃX9s'!˼8tVZe`jɤI2(2c@` YjĈ jĞ=rŋ达ܹ hJk Rɭ[tq߹s'>߻z5] q\fyp뭷WKs+7̭ ]M%F!.]tK4/.BQQj*5`TL6Lw-""""wr^{ k֬?5kX9)) hFhJW*_m`/ nz+@j,)ے,Y\ 3[VMp\.88 .{s+vۜÛ+r~ YF2jӟp]w!…'&"""r#\!0qD% OoI8}Zဓ/駟}v4|܂En}4 ƎϷ| n'; nNNW^!C>>1! 6ѣ[p%WOɼY pH 6̙31{lMDDD* @ׯǮ]p-ŋRcjCܹ&MBVV{=Y,'j9cBniСX~=v܉n OvtW*lYaÆ 5jRSS|r\MJO7=or[7toߎ'Ob8p$YnKb5j6n܈ϘLt.Z ]v!88iiiu^__sbڴi6mVZ@+շaa >`Bn/** ۷oGVVL:ݜ>>:u*nvl۶ ֭Ö-[xb@rr2FQFaذaPWW~7oƦMO?A")) YYY;v,222X`B#W0LjSP) bYN T*FPYԩSغu+lق\Cbb"bbbhDDD}Fpp0T*Ξ=3g4/--ŋѡC5 3gѣkADDDi})۫W2+8)P]w݅ B9r[nѣGqq`ը07RDDDK.ׯ~a'""" @ȾZ*%E|9JB=УGF?jpQ]] VN:!,,DDDDdQYJ~Vm5о}{o.bB*Xi0!""""air h Y\!kn{US ADDD}q{5[Ⱦ/n""""&p9 """""r DɑLQω DHOepz+fEDDDXJY30ÆuBDDDd"kcZ(.|-XDػW߻׵!"""rs @/k__=~Ԫl|=?_ MP9&II|u,r;gz6~9 DP]'{p:yy :V^+fEDDDBd,.DDDDf0q#555Cx&ݻ]= r~gUӔ6j9Bz$}'뮻\= """"j;hrlܸqqq ttHMM j5233J @iX9 """""r DDDDD4 @i0!"""""aBDDDDDN9 """""r DDDDD4 @i0!"""""aBDDDDDN9 """""r DDDDD4 @i0!"""""aBDDDDDN9 """""r DDDDD4 @i0!"""""aBDDDDDN9 """""r DDDDD4 @i0!"""""aBDDDDDN9 """""r DDDDD4 @i0!"""""aBDDDDDN9 """""r DDDDD4 @i0!"""""ױ Y* P5|sIENDB`bayespy-0.4.0/bayespy/tests/baseline_images/test_plot/hinton_r.png0000664000175000017500000000677512471107140027213 0ustar jluttinejluttine00000000000000PNG  IHDR XvpsBIT|d pHYsaa?i IDATxݽm@a&p'Pmo%<&9 ֪(E4=|Ҁ9/}wq\.@ @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @Fd @FdF@t:M˲~v`#-2=>>///1aOOO<ϣYu:`cȆ<ߏ q #@2 #@𡻻-]u:ÞvnCޙgrN';Lقd @FN͟5>q>v؂d `w&@p  #@2ȸ͹y 6>Tw4rX72cB2 #@2 ; #Em`aql̏r=4-2z Fiۍ @!t #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2 #@2i^ǟIENDB`bayespy-0.4.0/bayespy/tests/baseline_images/test_plot/hinton_p.png0000664000175000017500000001356212471107140027201 0ustar jluttinejluttine00000000000000PNG  IHDR XvpsBIT|d pHYsaa?iIDATx1׹ᣫLE ʀ 9Swك{^*N' Dx#/=T?O(qZ?j bUN}Ǐ z F1 @b#@ F1 @b#@ F1 @b#@ F1 @b#@ F1 @b#@ F1 @b#@ F1 @b#@ F1 @b#@ F1 @b#@ F1 @̡/c0W^0,y||l_=ŋիclz||l]Q`X?M{ ~C_{M.'@Y4~#W!t F1 @bc^p8eYں.S}>W}>`l6LԎ<ݻw.S}>W}>`l`1 @Ꙁ}f L@n%X@; r"ˆL kˉ,2K F1 @mx,K Y{>%X@; \k¾s %U_V}>[b #@o>~͛?p8i,K[1eǾU?z߷rww׿ޣ<7eǾU?, F1 @b#@z s}[{z}{||= OiJzg/۱/A|mC1`x(ڝeYz!@V/=jwyn׿z @V/.X@b,($LgEz3 "@`õDʕǯ|[|l֕[\U}~U F1 @mxa?Г>@O`1 @b<;,K;ݮr,b#@ F1 [xϽՏ_ط_ =~ǾU?Mr+ϟ@W;1 a9eN#@gfJ\Jd @9KjW+v84Mox0<'\yix=ƦoG qaUY,K[׵C ry-[8%X@b#@Æks+T?~cߪ&@`õv׹roϿc q\\Dz"@`'te9EN%З? 3 @; r;L k,2K F1 @mx,K Y{>%X@; _[H ˹۟v."~ ltYɗ>T?~cߪ&@`e%~ǾU?, F1 @ֵvk:gN#@ݚc 8mclZ{ =<<~O?'@4Mx<c<ϖ]˗/}16㏽GYĸ,, taUY.fY,b#@x6\kw[->V>06έSU_Ͽc E=@_3 @b#@ vjk W[= ة-\m d #@ CS˲y?z S (, F1 @bPVʗ|&@`sނǯ|JyϪ&@`/~ǾU?3 @b#@0ktF0,5, >QeY˝=<<~O?'@eMv{M?c`x`1 @b#@xl8mgeYںT?~cߪ&@`4Mx<>|/y~ǾU?, F1`psH=P}>["@ع >- 3^ueAΜzוq}-2% |-2%  @b#@؆`0˲y*;/si]W`XvƗS2NK>- \`1 6fϽՏ_ط_ `eoϿc  @b#@ {@Цi=ƦeYںvZ/A|߿oǀ (hv<{ig/)P5.~U~ ۷oۇz T_|K0*wjܙ8`XUԸ3p:`1 @b<έSU}~U ]Vv˩~ǾU?, F1 @]8u.qA_؅Swe #@ F1 @b؅eY<'9ºp  @b!tpn*|-~ǾU?M\>V>06K F1B`Xڭ98`Xvkʕ0.===clzzz=C5MS[ŝF@A/_l4ϳ/cl F1 @b= Z/%o{WU}~U zYܭ>V>06KX}f L@/EpsH=P}>. c  @b#@ F1 @bs=[|%TX[׵{e~a,`1 6ϽՏ_ط_ `MeoϿc  @b#@؆9mc|4MmYkQ_k00L{Emg 8ӵ^F8Zk?Yk7B?==six={v F1 @bc^p7oϿc Zi78W?~cߪf #@Kؽsi~a,;,b #@ F1 @b#@ {eYsoAW}>W}>`l6XS|ǯ|[|,b#@ F1ZY4T}{vN@1˲v{MhSU7k/W˲u]{1Цi={xg LԎc1cJĉr U%F7^l  F1:l:[NW}>W}>`l6\kw[->V>06ٞJ+CpR8 ν!@{'Ν76@b#@ F1 @b#@C.,K0u]ۻwzHps<s  ues{ oϿc T?~cߪf^ F1 @ֵvk:gN#@:W#~[xs84Miڲ,m]ޣ ͗gnbij6ϳ/;U=/clc #@ F1 @kV^VU}~U zۭD>V>06Kw@a]\̲0 ޹_R}9bA`) 9J+T%@r|W.X@b#@ F1"B,y>@E` z9C ީw>sV>06K F1 @mxYUo{p߷o+~:0o>~͛cl޼y{k?{xzۇz ?z ܜ_~=>>c7|~7nѫWw}{ b< F1 @b#@ F1 @b#@ F1 @b#@ F1 @b#@ F1 @b#@ F1 @b#@ F1 @b#@ F1 @b#@ F1 @b#@ F1 @b#@ F1 @b#@ F1 @b#@ F1 @b#@ F1 @b#@ F1 @oR4eIENDB`bayespy-0.4.0/bayespy/tests/baseline_images/test_plot/pdf.png0000664000175000017500000003517312473034010026133 0ustar jluttinejluttine00000000000000PNG  IHDR XvpsBIT|d pHYsaa?i IDATx{}33 wPP!4df``55v 6'ݘmmMOmNb[w9 9[϶ڒ][\DP⠫ru.6e.:0'yH7|d*@ @fdF @fdF @fdF @fdF @fdF @fdF @fdF @fdF @fdF @fdF @fdF @fdF @fdF @fdF @fjR/?3f̈ѣG_rȑغuk~QWWz$@?SE<(I̘1#"=wܴÐ+Vʕ+8G#YL$ߕ]98RD;wn477',L81{=wn?kI~)we~WM@fdF@B˗/O=~Wwe~C Hȿ*x+O |>rKS 7n-Z[[?iBQO~ַo1E[[[BQƍ\.wyg[.ū---O|" 񪪪cӦMqزeKI(Bۿx饗555O~2&Lt 8q"x뮻弿nرq=W8yd0ױux.kعsg<L@!ѣG}C1o޼X`A|_`: AP4/Ů]ׯr},oݻy: AP>=P|#9ss=1bĈxGq: EP~|3}\rIoF|_|>?LP(;gh̜9s{W_}5x@A [n]Ľ;M7ٳmF(֮]#Gr{{x: Iܚ5k9jkk9>ĩSG-d!ku]7q>җd3:@ $gϞxW omF߸qc]0I=3qsӧیPI]6.처5k֐?WuuuobժUqСL@ ZfM\wu qС~V@a vڂ<~gܹQWWO>d>'#@Hf˖-S }r\|ͱz}N GLiU Kƚ5kȑ# !5kĬYoǏ; )2o޼K(B$N8mmm݀ާ:nF@! 6ѣGe$} ?Bk׮Xp|o99=ܰ|~GĚ5kb1za1vX@!ڀާ&n@!sof3-]4z8y䰾'@ܳ>|~XW@"  /0@ 2v?~|\s5:-QFy 2f͚kjx92/^l#:@ dn7o'ۛpaLر#v5,]46m0B֮] _#FH2f͚x;SNƌ]w} EB,cՑ3}]NSNų>y,]4:;;_ux;Bfm@dɒ2GDDsss;~hnn2SL &d7|s<$&@LGGG̚5+k/]4v[nMEW_}5Yx$Bf:::⪫Jڗ^zi]$[8rHڵ+ HD… cݺu^BF^ Xn]& 2id…qxWPژ:uj.\1,舙3gFUU?r'Oӧۈ!)9i 2Q,mmmnDHD0|… 'vܙz$@vq@ #xS݂~iӦɓ@"a 3gL_phoo#GbMWWW>|h9N:6lH= @ b=OcccTWWۈ!°yW#"b̙'9QFECC}  (l CaS,\0֯_'NH= @E R8vX"MGGG\uUǸrȈaX=z4vY+ Ǐw]6dD0,n{°(;@6l|>z'@Q[[SNM=E566FOOOtuu ū3f̈ԣ\Լy""bƍ'(aQ G5kV5Jd@0,J)@^d@Pp|$^Ç\lڴ)z{{SPwo߂~yc۶mG(k 3g&#IXMPp1y?~|Qmڴi1qD0u֘1cF1$Eccf۵kWL6-$,'@(]vŕW^z7o^l޼9N8z%@(ݻwԩSS1`qxWRPu.ihh'a 'BA%V]]]\q`  j%a#:p Ԯ]"4W@"QLPPwK=ʠ̛7/^y8rHQʒ୪*?Z磽=(e4Jh }#IXEPPzHq̙3mD&*J}$⭍`x oA?Scccر#xԣB-gjllM6% oA3gΜ` oAS[[g϶` oA?CP0~ G(+"E{ԣB }N@a  f׮]e 3gΌѣG; {y:뭀 39 Q.!6m$, D܂~8tPl۶-(eCP}SGDċ/x!@(['Mz>}z;V >UUU1w\P@"I-gjhh $@(rL/: @QN>|عsgQʂ 39 CV1cF=Za>}'amڴ)(eA0dx 60t!+[$, Y9ނ~x7N?jդ[bEL8-_q5؈P!+'aa}$ٴiH0$| b޽ݝz&@i=@#"܈0D!ꪫbĈ aH>5551g0D!)[ @H0$~ N`FU%ҧ>0!;@ޞx%@JZꫣ>! I%ٳRnA?C#@JLV@@0hr _{$ RnA?F0hw#FT-}fϞ$ ڵ+⊊ȑ#ꫯ`*+G 9'@JL`V+ sؿQJa:;;/O=FN² 0pA ٳgGUU ݻ7z{{cʔ)Gѣc֬Y`'ONoofQJaPbĈ1qԣ$'(-A)SD.K=Js̉\.1, JWWWn@3fL̚5Ft Jwwwc#: JV__o`"@ڈm۶8p@QJaP[+ 7oN< @ c߾} \sMDǰ@0`===} VDرcc̙6 a#roA?F0`]]]!@"0P J+⭓nJ= @I XWWW;6ƎzNjooO< @i #xaܹa<~7.w ~ Xww3؈  gkhh-[ÇSP&@V__|>6oޜz'@CšClB?O0 3M0!On#:@?D#@ La@.$ť!^}8rHQa@K/ԣ^zԣ5€tuu9N²sKbڴi\a@y؈pqr~€; ѣG(Z~룷7^~ԣ-B߿?N8󨯯'a\܂~a^ziL:U\ߺ#B\Hccclܸ1EKoV@.N\ߺ*.ԣC( +ꢪill|>G(J{q}'ay &@.nر1k,p5VX'NDӔRV\ͩx7o^1^ccc<#\---&*(IΫ/@OR>'vڕz$e𚚚"Ftr Ϋ'FcǎM=J٘1cFL0AKp^-\.g#:P%@*hjj_~99z K 퍍7 s ã1<T$y 1z={*N8 FtR iϞ=ׯ|>zL ΩēؿkGȔzzz" pijj>$ S SNI&T9uwwLj#bĉG)K\Ft" Ω;"˥l-X@GpN~MMMofQ2#@8'26oذ!$ ~s΍#Fx (s ï6Ν+@"@8' 'aF6g ԩSGȄmx8u@sss9r$6oޜzLަē戈xgO 7a„={v̙c#:Po#@g#:P)o#@/B8q"(J6===$c-GƋ/za%@8K> lD*,cǎ 7.Νk#:Pg.$ Pg + [hQ_>?za#@8I%;6mJ=  #X,X ʚ,1qĨM=J;vl5YVKK "@ZhQ qرԣ Y܂VKKK8q"6nܘza!@8cX@ E5f&@8IoѢEV@%@8ѣqAXKKKlذFt, Ns aqXhQ8q"6lؐz  V@Қ?~TWW%i8=:ʒ4R<-Zd(KӺc̘11f̘ԣTظqc=z4(%@8cѢEqxSPPzzzHhjj#GO?z fxƢEg?YQ Jp).K,((i,^8v۷oO= @N eXIDATVA"@'Nľ}.(W\qE̜9SeE{%K؈BDX-^8֭[GI= @A"B%Kĉ'K= @A"Byرcʆ!" ژ0aBQ8CMMM\wueCqG.K= d<|QLoȔ)SR9,Y$:;;c˖-G2BDbw;"\HBDb6iҤ3g} @Y DDDWW)b}@J!"ŋ /L= qaЋؒ%K7yԣ !:;;#"0a} @ PUU~J!"B;@Q]]&MJ= dɒػwo˩G4BtvvɓbvG.P|Itvv:L81lDJ %ą@ dɒشiS߿?("@.R"/^|>֬YzA X)!gώI&SO=zA ѣ~KD.KƏԣ p.!,=f͚8|pQLTΈ 5N8?OS0` S__SLPH{kɉ'r\= @I 3&M#FH= p-3<H=  Nr/PrHsHiz׻SNGGRHirjPrH %5֭[oFQMT8+ 5z{{cթG7RN<{ D͜93~W~cX@I ;[n @I -襯5^xسgOQ&݊+bĉg}l|򂾎)}O?@yYjUZ꬏߿?4C+WFssNWWWD= %lqUWŏcvnkkD`U?~|=:( @R"@*#xCkkk⋧(f p>b'@*)W^ye\s5J`|Ə~c\`]]]L7vܙz  7[&DDx (zBݻ7N:eLL<9PHr z[{^ԣP,[,vׯO= y tM1vwz 3FƍK= 2rx{+@&@*T \.(вe駟={Hr ayz㏧H iԩpBaEKT(R-[NJ=  %@Բeb޽OmHSLI= k:aEIT7|3;fLUWWǿBEIT e˖cǎG8@~Q]]wSpRHKcɒ%@]]]QSS\rIQF˖-qԣ&@*P XUUrlٲ8|p<䓩G8W%!O1, $@*C.e˖wRHXlYttt/z KT[n%F1,h Է 7f̘hmm @ qA+ ~bձo߾ԣJ/忌SN׿ԣJ#@*ԩScҥG FT;~FwwwQ '@*LWWWr4iRQ?ēNTΨԣɓ'-1, 9Ra\BXx'_O= PHٽ{w\q +z }>}z1H.n-`l߾=~W~%$rwOSa rرؽ{ | jkk?AQ %@*Ν;#"T'wRT(RAo}ݱqӦcJTm۶EDxy1jcJT۷ǤIb̘1G!qM7ݔz B m6DDmXȊ ۶m""lF2'@*;@3jԨxOTRA|88@ en1y=ztQ(AG{쉯~Gʄ)sa(f͚Їs\8q"8@ e ǶmRR欀0T͋%N|>t ͛77ԣ%NCw;Z[[㡇|>z 2^ {.~(a [oEşٟYM۷GMMM\qG r?SO=_S(RƶmӦMԣP&n;ⓟd;v,8@  e kԣ%H1w0Ν>`tuu(1rGMMM|3I= PbH:uTܹ bҤIqǗذaCq"@믿'O°ݸcŊM۷G;@>#F018@ e-daٲeq뭷!?z2m۶7n\\r%GrJ<éJ)S۷oӧG.K= en޼y},xؽ{wq"'@ʔ#xҟȑ#c  2t4iҤw+_Jq"&@P>B~~-XbEttt(R m߾=MMMG| _8uTq"$@s=---'Ҍ?>o&z(B _W^yeQ@K.+VğɟƍSRڢ$qWG>g e/@ QF36l>`q""@ݻ_ $??OR Rf""En!~=nݚz2se]|;S1bĈxbqwƁR$&@ʌ Zs= '@ʌ ww_qH۷ ws\MqDjR@[."l@x}M6?oL=1+ e-ƏW]uUQr\_E,Y$زeK 2ҷJ񪭭{,&NzkkG2+2sy0iҤK.D}EGGd'|2FK.W^y%H@Hx#tJ;x'c̘1txS 3R&b1gΜԣL:5x≸Kbҥޞz$` 2 ,ԣ]q8&Oy{bƍG)mmmҒz )Sď~:ujtMOOG)^zJ^]]]<q7Ʋe⡇|>z,Hx" ĉf|3O}Sտ (Rڢ6SQUU7~\N2!@@[[[̟??Fz(3D>k61H 2+֜9sb͚5q-ĝw7H=0H9r$^|E'`Q&L=X<}-FA %nq)+ شiS]wz %-jjj1(ӧǷ׾O>d̝;7կ:J)qmmmFJ= d&w?>яƵ^,D g:lҤI#O<#G;#nXzu %aB[tiK.n-֮]z4}/N8]w]Q \.wqGY~hii|+k׮g?x{^zivm_{)1<@TWWǧ>ԣ@Yrq>زeK[.|x -Z'O~>V^L=6,RB^z˿ԧ>&MJ=jժ#p\. ,}|ߍ}ի;>OҥKcĉۿ?p?7x㢟]Yߕ '@JȟɟĴiSBVqn)O4V^of WWq7D,Y$.Ҙ6mZz?'x"^{ӏpy+xjR@?W5*8Pjjjb1G?G͛7G{{{/xOGMMML>=8{o̘1#MSNSƕW^&Mr6A|>'cq=8èQb`>~舭[Ɩ-[b˖-裏?7bϞ=gڸ+cʔ)QWWo]vYL81.䒳9rdq`H:ydl޼96lׯg}6~ڸkk9͛7Ƿ8vXk׮ؽ{wڵ+v===[ng}6zzzbϞ={9rd?>ƍocƌѣGϣF#GǵQ[[#Fx555?QSS7RD97pC|􏻺رcqر8zoǎǏ#?_UUUTWWGuuuTUU-Euuur|ϝ}ξ=vs_?wq_\~M{{{p 9啎M67ߜz 2wbm \._1hSE<(I?3f̈ѣG_rȑغuk~QWWz$@5dF @fdF @fdF @fdF @fdF @fdF @fdF @fdF @fdF @fdF @fdF @fdF @fdF @fdF @fds{og/IENDB`bayespy-0.4.0/bayespy/utils/0000755000175000017500000000000012615647103017526 5ustar jluttinejluttine00000000000000bayespy-0.4.0/bayespy/utils/tests/0000755000175000017500000000000012615647103020670 5ustar jluttinejluttine00000000000000bayespy-0.4.0/bayespy/utils/tests/__init__.py0000664000175000017500000000000012174013072022761 0ustar jluttinejluttine00000000000000bayespy-0.4.0/bayespy/utils/tests/test_misc.py0000644000175000017500000003523212615643655023251 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for bayespy.utils.misc module. """ import unittest import warnings import numpy as np from numpy import testing from .. import misc class TestCeilDiv(misc.TestCase): def test_ceildiv(self): """ Test the ceil division """ self.assertEqual(misc.ceildiv(3, 1), 3) self.assertEqual(misc.ceildiv(6, 3), 2) self.assertEqual(misc.ceildiv(7, 3), 3) self.assertEqual(misc.ceildiv(8, 3), 3) self.assertEqual(misc.ceildiv(-6, 3), -2) self.assertEqual(misc.ceildiv(-7, 3), -2) self.assertEqual(misc.ceildiv(-8, 3), -2) self.assertEqual(misc.ceildiv(-9, 3), -3) self.assertEqual(misc.ceildiv(6, -3), -2) self.assertEqual(misc.ceildiv(7, -3), -2) self.assertEqual(misc.ceildiv(8, -3), -2) self.assertEqual(misc.ceildiv(9, -3), -3) class TestAddAxes(misc.TestCase): def test_add_axes(self): """ Test the add_axes method. """ f = lambda X, **kwargs: np.shape(misc.add_axes(X, **kwargs)) # By default, add one leading axis self.assertEqual(f(np.ones((3,))), (1,3)) # By default, add leading axes self.assertEqual(f(np.ones((3,)), num=3), (1,1,1,3)) # By default, add one axis self.assertEqual(f(np.ones((3,)), axis=1), (3,1)) # Add axes to the beginning self.assertEqual(f(np.ones((2,3,4,)), axis=0, num=3), (1,1,1,2,3,4)) # Add axes to the middle self.assertEqual(f(np.ones((2,3,4,)), axis=1, num=3), (2,1,1,1,3,4)) # Test negative axis index self.assertEqual(f(np.ones((2,3,4,)), axis=-4, num=3), (1,1,1,2,3,4)) self.assertEqual(f(np.ones((2,3,4,)), axis=-1, num=1), (2,3,4,1)) self.assertEqual(f(np.ones((2,3,4,)), axis=-2, num=3), (2,3,1,1,1,4)) # Add axes to the end self.assertEqual(f(np.ones((2,3,4,)), axis=3, num=3), (2,3,4,1,1,1)) class TestBroadcasting(unittest.TestCase): def test_is_shape_subset(self): f = misc.is_shape_subset self.assertTrue(f( (), () )) self.assertTrue(f( (), (3,) )) self.assertTrue(f( (1,), (1,) )) self.assertTrue(f( (1,), (3,) )) self.assertTrue(f( (1,), (4,1) )) self.assertTrue(f( (1,), (4,3) )) self.assertTrue(f( (1,), (1,3) )) self.assertTrue(f( (3,), (1,3) )) self.assertTrue(f( (3,), (4,3) )) self.assertTrue(f( (5,1,3), (6,5,4,3) )) self.assertTrue(f( (5,4,3), (6,5,4,3) )) self.assertFalse(f( (1,), () )) self.assertFalse(f( (3,), (1,) )) self.assertFalse(f( (4,3,), (3,) )) self.assertFalse(f( (4,3,), (1,3,) )) self.assertFalse(f( (6,1,4,3,), (6,1,1,3,) )) class TestMultiplyShapes(unittest.TestCase): def test_multiply_shapes(self): f = lambda *shapes: tuple(misc.multiply_shapes(*shapes)) # Basic test self.assertEqual(f((2,), (3,)), (6,)) # Test multiple arguments self.assertEqual(f((2,), (3,), (4,)), (24,)) # Test different lengths and multiple arguments self.assertEqual(f(( 2,3,), (4,5,6,), ( 7,)), (4,10,126,)) # Test empty shapes self.assertEqual(f((), ()), ()) self.assertEqual(f((), (5,)), (5,)) class TestSumMultiply(unittest.TestCase): def check_sum_multiply(self, *shapes, **kwargs): # The set of arrays x = list() for (ind, shape) in enumerate(shapes): x += [np.random.randn(*shape)] # Result from the function yh = misc.sum_multiply(*x, **kwargs) axis = kwargs.get('axis', None) sumaxis = kwargs.get('sumaxis', True) keepdims = kwargs.get('keepdims', False) # Compute the product y = 1 for xi in x: y = y * xi # Compute the sum if sumaxis: y = np.sum(y, axis=axis, keepdims=keepdims) else: axes = np.arange(np.ndim(y)) # TODO/FIXME: np.delete has a bug that it doesn't accept negative # indices. Thus, transform negative axes to positive axes. if len(axis) > 0: axis = [i if i >= 0 else i+np.ndim(y) for i in axis] elif axis < 0: axis += np.ndim(y) axes = np.delete(axes, axis) axes = tuple(axes) if len(axes) > 0: y = np.sum(y, axis=axes, keepdims=keepdims) # Check the result testing.assert_allclose(yh, y, err_msg="Incorrect value.") def test_sum_multiply(self): """ Test misc.sum_multiply. """ # Check empty list returns error self.assertRaises(ValueError, self.check_sum_multiply) # Check scalars self.check_sum_multiply(()) self.check_sum_multiply((), (), ()) # Check doing no summation self.check_sum_multiply((3,), axis=()) self.check_sum_multiply((3,1,5), ( 4,1), ( 5,), ( ), axis=(), keepdims=True) # Check AXES_TO_SUM self.check_sum_multiply((3,1), (1,4), (3,4), axis=(1,)) self.check_sum_multiply((3,1), (1,4), (3,4), axis=(-2,)) self.check_sum_multiply((3,1), (1,4), (3,4), axis=(1,-2)) # Check AXES_TO_SUM and KEEPDIMS self.check_sum_multiply((3,1), (1,4), (3,4), axis=(1,), keepdims=True) self.check_sum_multiply((3,1), (1,4), (3,4), axis=(-2,), keepdims=True) self.check_sum_multiply((3,1), (1,4), (3,4), axis=(1,-2,), keepdims=True) self.check_sum_multiply((3,1,5,6), ( 4,1,6), ( 4,1,1), ( ), axis=(1,-2), keepdims=True) # Check AXES_TO_KEEP self.check_sum_multiply((3,1), (1,4), (3,4), sumaxis=False, axis=(1,)) self.check_sum_multiply((3,1), (1,4), (3,4), sumaxis=False, axis=(-2,)) self.check_sum_multiply((3,1), (1,4), (3,4), sumaxis=False, axis=(1,-2)) # Check AXES_TO_KEEP and KEEPDIMS self.check_sum_multiply((3,1), (1,4), (3,4), sumaxis=False, axis=(1,), keepdims=True) self.check_sum_multiply((3,1), (1,4), (3,4), sumaxis=False, axis=(-2,), keepdims=True) self.check_sum_multiply((3,1), (1,4), (3,4), sumaxis=False, axis=(1,-2,), keepdims=True) self.check_sum_multiply((3,1,5,6), ( 4,1,6), ( 4,1,1), ( ), sumaxis=False, axis=(1,-2,), keepdims=True) # Check errors # Inconsistent shapes self.assertRaises(ValueError, self.check_sum_multiply, (3,4), (3,5)) # Axis index out of bounds self.assertRaises(ValueError, self.check_sum_multiply, (3,4), (3,4), axis=(-3,)) self.assertRaises(ValueError, self.check_sum_multiply, (3,4), (3,4), axis=(2,)) self.assertRaises(ValueError, self.check_sum_multiply, (3,4), (3,4), sumaxis=False, axis=(-3,)) self.assertRaises(ValueError, self.check_sum_multiply, (3,4), (3,4), sumaxis=False, axis=(2,)) # Same axis several times self.assertRaises(ValueError, self.check_sum_multiply, (3,4), (3,4), axis=(1,-1)) self.assertRaises(ValueError, self.check_sum_multiply, (3,4), (3,4), sumaxis=False, axis=(1,-1)) class TestLogSumExp(misc.TestCase): def test_logsumexp(self): """ Test the ceil division """ self.assertAllClose(misc.logsumexp(3), np.log(np.sum(np.exp(3)))) with warnings.catch_warnings(): warnings.simplefilter("ignore", RuntimeWarning) self.assertAllClose(misc.logsumexp(-np.inf), -np.inf) self.assertAllClose(misc.logsumexp(np.inf), np.inf) with warnings.catch_warnings(): warnings.simplefilter("ignore", RuntimeWarning) self.assertAllClose(misc.logsumexp(np.nan), np.nan) with warnings.catch_warnings(): warnings.simplefilter("ignore", RuntimeWarning) self.assertAllClose(misc.logsumexp([-np.inf, -np.inf]), -np.inf) self.assertAllClose(misc.logsumexp([[1e10, 1e-10], [-1e10, -np.inf]], axis=-1), [1e10, -1e10]) # Test keeping dimensions self.assertAllClose(misc.logsumexp([[1e10, 1e-10], [-1e10, -np.inf]], axis=-1, keepdims=True), [[1e10], [-1e10]]) # Test multiple axes self.assertAllClose(misc.logsumexp([[1e10, 1e-10], [-1e10, -np.inf]], axis=(-1,-2)), 1e10) pass class TestMean(misc.TestCase): def test_mean(self): """ Test the ceil division """ self.assertAllClose(misc.mean(3), 3) with warnings.catch_warnings(): warnings.simplefilter("ignore", RuntimeWarning) self.assertAllClose(misc.mean(np.nan), np.nan) self.assertAllClose(misc.mean([[2,3], [np.nan,np.nan]], axis=-1), [2.5,np.nan]) self.assertAllClose(misc.mean([[2,3], [np.nan,np.nan]], axis=-1, keepdims=True), [[2.5],[np.nan]]) self.assertAllClose(misc.mean([[2,3], [np.nan,np.nan]], axis=-2), [2,3]) self.assertAllClose(misc.mean([[2,3], [np.nan,np.nan]], axis=-2, keepdims=True), [[2,3]]) self.assertAllClose(misc.mean([[2,3], [np.nan,np.nan]]), 2.5) self.assertAllClose(misc.mean([[2,3], [np.nan,np.nan]], axis=(-1,-2)), 2.5) self.assertAllClose(misc.mean([[2,3], [np.nan,np.nan]], keepdims=True), [[2.5]]) pass bayespy-0.4.0/bayespy/utils/tests/test_linalg.py0000644000175000017500000001407012615643655023561 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for bayespy.utils.linalg module. """ import numpy as np from .. import misc from .. import linalg class TestDot(misc.TestCase): def test_dot(self): """ Test dot product multiple multi-dimensional arrays. """ # If no arrays, return 0 self.assertAllClose(linalg.dot(), 0) # If only one array, return itself self.assertAllClose(linalg.dot([[1,2,3], [4,5,6]]), [[1,2,3], [4,5,6]]) # Basic test of two arrays: (2,3) * (3,2) self.assertAllClose(linalg.dot([[1,2,3], [4,5,6]], [[7,8], [9,1], [2,3]]), [[31,19], [85,55]]) # Basic test of four arrays: (2,3) * (3,2) * (2,1) * (1,2) self.assertAllClose(linalg.dot([[1,2,3], [4,5,6]], [[7,8], [9,1], [2,3]], [[4], [5]], [[6,7]]), [[1314,1533], [3690,4305]]) # Test broadcasting: (2,2,2) * (2,2,2,2) self.assertAllClose(linalg.dot([[[1,2], [3,4]], [[5,6], [7,8]]], [[[[1,2], [3,4]], [[5,6], [7,8]]], [[[9,1], [2,3]], [[4,5], [6,7]]]]), [[[[ 7, 10], [ 15, 22]], [[ 67, 78], [ 91, 106]]], [[[ 13, 7], [ 35, 15]], [[ 56, 67], [ 76, 91]]]]) # Inconsistent shapes: (2,3) * (2,3) self.assertRaises(ValueError, linalg.dot, [[1,2,3], [4,5,6]], [[1,2,3], [4,5,6]]) # Other axes do not broadcast: (2,2,2) * (3,2,2) self.assertRaises(ValueError, linalg.dot, [[[1,2], [3,4]], [[5,6], [7,8]]], [[[1,2], [3,4]], [[5,6], [7,8]], [[9,1], [2,3]]]) # Do not broadcast matrix axes: (2,1) * (3,2) self.assertRaises(ValueError, linalg.dot, [[1], [2]], [[1,2,3], [4,5,6]]) # Do not accept less than 2-D arrays: (2) * (2,2) self.assertRaises(ValueError, linalg.dot, [1,2], [[1,2,3], [4,5,6]]) class TestBandedSolve(misc.TestCase): def test_block_banded_solve(self): """ Test the Gaussian elimination algorithm for block-banded matrices. """ # # Create a block-banded matrix # # Number of blocks N = 40 # Random sizes of the blocks #D = np.random.randint(5, 10, size=N) # Fixed sizes of the blocks D = 5*np.ones(N, dtype=np.int) # Some helpful variables to create the covariances W = [np.random.randn(D[i], 2*D[i]) for i in range(N)] # The diagonal blocks (covariances) A = [np.dot(W[i], W[i].T) for i in range(N)] # The superdiagonal blocks (cross-covariances) B = [np.dot(W[i][:,-1:], W[i+1][:,:1].T) for i in range(N-1)] C = misc.block_banded(A, B) # Create the system to be solved: y=C*x x_true = np.random.randn(np.sum(D)) y = np.dot(C, x_true) x_true = np.reshape(x_true, (N, -1)) y = np.reshape(y, (N, -1)) # # Run tests # # The correct inverse invC = np.linalg.inv(C) # Inverse from the function that is tested (invA, invB, x, ldet) = linalg.block_banded_solve(np.asarray(A), np.asarray(B), np.asarray(y)) # Check that you get the correct number of blocks self.assertEqual(len(invA), N) self.assertEqual(len(invB), N-1) # Check each block i0 = 0 for i in range(N-1): i1 = i0 + D[i] i2 = i1 + D[i+1] # Check diagonal block self.assertTrue(np.allclose(invA[i], invC[i0:i1, i0:i1])) # Check super-diagonal block self.assertTrue(np.allclose(invB[i], invC[i0:i1, i1:i2])) i0 = i1 # Check last block self.assertTrue(np.allclose(invA[-1], invC[i0:, i0:])) # Check the solution of the system self.assertTrue(np.allclose(x_true, x)) # Check the log determinant self.assertAlmostEqual(ldet/np.linalg.slogdet(C)[1], 1) bayespy-0.4.0/bayespy/utils/tests/test_random.py0000644000175000017500000002274112615643655023577 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for bayespy.utils.random module. """ import numpy as np from .. import misc from .. import random class TestCeilDiv(misc.TestCase): def test_categorical(self): # Test dummy one category y = random.categorical([1]) self.assertEqual(y, 0) # Test multiple categories y = random.categorical([1,0,0]) self.assertEqual(y, 0) y = random.categorical([0,1,0]) self.assertEqual(y, 1) y = random.categorical([0,0,1]) self.assertEqual(y, 2) # Test un-normalized probabilities y = random.categorical([0,0.1234]) self.assertEqual(y, 1) # Test multiple distributions y = random.categorical([ [1,0,0], [0,0,1], [0,1,0] ]) self.assertArrayEqual(y, [0,2,1]) # Test multiple samples y = random.categorical([0,1,0], size=(4,)) self.assertArrayEqual(y, [1,1,1,1]) # # ERRORS # # Negative probablities self.assertRaises(ValueError, random.categorical, [0, -1]) # Requested size and probability array size mismatch self.assertRaises(ValueError, random.categorical, [[1,0],[0,1]], size=(3,)) pass class TestDirichlet(misc.TestCase): """ Unit tests for the Dirichlet random sampling """ def test(self): """ Test random sampling from the Dirichlet distribution. """ # Test computations p = random.dirichlet([1e-10, 1e-10, 1e10, 1e-10]) self.assertAllClose(p, [0, 0, 1, 0], atol=1e-5) p = random.dirichlet([1e20, 1e20, 1e20, 5*1e20]) self.assertAllClose(p, [0.125, 0.125, 0.125, 0.625]) # Test array p = random.dirichlet([ [1e20, 1e-20], [3*1e20, 1e20] ]) self.assertAllClose(p, [[1.0, 0.0], [0.75, 0.25]]) # Test size argument p = random.dirichlet([ [1e20, 1e-20] ], size=3) self.assertAllClose(p, [[1, 0], [1, 0], [1, 0]]) p = random.dirichlet([ [3*1e20, 1e20] ], size=(2,3)) self.assertAllClose(p, [ [[0.75, 0.25], [0.75, 0.25], [0.75, 0.25]], [[0.75, 0.25], [0.75, 0.25], [0.75, 0.25]] ]) pass class TestAlphaBetaRecursion(misc.TestCase): def test(self): """ Test the results of alpha-beta recursion for Markov chains """ np.seterr(divide='ignore') # Deterministic oscillator p0 = np.array([1.0, 0.0]) P = np.array(3*[[[0.0, 1.0], [1.0, 0.0]]]) (z0, zz, g) = random.alpha_beta_recursion(np.log(p0), np.log(P)) self.assertAllClose(z0, [1.0, 0]) self.assertAllClose(zz, [ [[0.0, 1.0], [0.0, 0.0]], [[0.0, 0.0], [1.0, 0.0]], [[0.0, 1.0], [0.0, 0.0]] ]) self.assertAllClose(g, -np.log(np.einsum('a,ab,bc,cd->', p0, P[0], P[1], P[2])), msg="Cumulant generating function incorrect") # Maximum randomness p0 = np.array([0.5, 0.5]) P = np.array(3*[[[0.5, 0.5], [0.5, 0.5]]]) (z0, zz, g) = random.alpha_beta_recursion(np.log(p0), np.log(P)) self.assertAllClose(z0, [0.5, 0.5]) self.assertAllClose(zz, [ [[0.25, 0.25], [0.25, 0.25]], [[0.25, 0.25], [0.25, 0.25]], [[0.25, 0.25], [0.25, 0.25]] ]) self.assertAllClose(g, -np.log(np.einsum('a,ab,bc,cd->', p0, P[0], P[1], P[2])), msg="Cumulant generating function incorrect") # Unnormalized probabilities p0 = np.array([2, 2]) P = np.array([ [[4, 4], [4, 4]], [[8, 8], [8, 8]], [[20, 20], [20, 20]] ]) (z0, zz, g) = random.alpha_beta_recursion(np.log(p0), np.log(P)) self.assertAllClose(z0, [0.5, 0.5]) self.assertAllClose(zz, [ [[0.25, 0.25], [0.25, 0.25]], [[0.25, 0.25], [0.25, 0.25]], [[0.25, 0.25], [0.25, 0.25]] ]) self.assertAllClose(g, -np.log(np.einsum('a,ab,bc,cd->', p0, P[0], P[1], P[2])), msg="Cumulant generating function incorrect") p0 = np.array([2, 6]) P = np.array([ [[0, 3], [4, 1]], [[3, 5], [6, 4]], [[9, 2], [8, 1]] ]) (z0, zz, g) = random.alpha_beta_recursion(np.log(p0), np.log(P)) y0 = np.einsum('a,ab,bc,cd->a', p0, P[0], P[1], P[2]) y1 = np.einsum('a,ab,bc,cd->ab', p0, P[0], P[1], P[2]) y2 = np.einsum('a,ab,bc,cd->bc', p0, P[0], P[1], P[2]) y3 = np.einsum('a,ab,bc,cd->cd', p0, P[0], P[1], P[2]) self.assertAllClose(z0, y0 / np.sum(y0)) self.assertAllClose(zz, [ y1 / np.sum(y1), y2 / np.sum(y2), y3 / np.sum(y3) ]) self.assertAllClose(g, -np.log(np.einsum('a,ab,bc,cd->', p0, P[0], P[1], P[2])), msg="Cumulant generating function incorrect") # Test plates p0 = np.array([ [1.0, 0.0], [0.5, 0.5] ]) P = np.array([ [ [[0.0, 1.0], [1.0, 0.0]] ], [ [[0.5, 0.5], [0.5, 0.5]] ] ]) (z0, zz, g) = random.alpha_beta_recursion(np.log(p0), np.log(P)) self.assertAllClose(z0, [[1.0, 0.0], [0.5, 0.5]]) self.assertAllClose(zz, [ [ [[0.0, 1.0], [0.0, 0.0]] ], [ [[0.25, 0.25], [0.25, 0.25]] ] ]) self.assertAllClose(g, -np.log(np.einsum('...a,...ab->...', p0, P[...,0,:,:])), msg="Cumulant generating function incorrect") # Test overflow logp0 = np.array([1e5, -np.inf]) logP = np.array([[[-np.inf, 1e5], [-np.inf, 1e5]]]) (z0, zz, g) = random.alpha_beta_recursion(logp0, logP) self.assertAllClose(z0, [1.0, 0]) self.assertAllClose(zz, [ [[0.0, 1.0], [0.0, 0.0]] ]) ## self.assertAllClose(g, ## -np.log(np.einsum('a,ab,bc,cd->', ## p0, P[0], P[1], P[2]))) # Test underflow logp0 = np.array([-1e5, -np.inf]) logP = np.array([[[-np.inf, -1e5], [-np.inf, -1e5]]]) (z0, zz, g) = random.alpha_beta_recursion(logp0, logP) self.assertAllClose(z0, [1.0, 0]) self.assertAllClose(zz, [ [[0.0, 1.0], [0.0, 0.0]] ]) ## self.assertAllClose(g, ## -np.log(np.einsum('a,ab,bc,cd->', ## p0, P[0], P[1], P[2]))) # Test stability of the algorithm logp0 = np.array([-1e5, -np.inf]) logP = np.array(10*[[[-np.inf, 1e5], [1e0, -np.inf]]]) (z0, zz, g) = random.alpha_beta_recursion(logp0, logP) self.assertTrue(np.all(~np.isnan(z0)), msg="Nans in results, algorithm not stable") self.assertTrue(np.all(~np.isnan(zz)), msg="Nans in results, algorithm not stable") self.assertTrue(np.all(~np.isnan(g)), msg="Nans in results, algorithm not stable") pass bayespy-0.4.0/bayespy/utils/random.py0000644000175000017500000002436712533543154021374 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ r""" General functions random sampling and distributions. """ import numpy as np from scipy import special from . import linalg from . import misc def intervals(N, length, amount=1, gap=0): r""" Return random non-overlapping parts of a sequence. For instance, N=16, length=2 and amount=4: [0, |1, 2|, 3, 4, 5, |6, 7|, 8, 9, |10, 11|, |12, 13|, 14, 15] that is, [1,2,6,7,10,11,12,13] However, the function returns only the indices of the beginning of the sequences, that is, in the example: [1,6,10,12] """ if length * amount + gap * (amount-1) > N: raise ValueError("Too short sequence") # In practice, we draw the sizes of the gaps between the sequences total_gap = N - length*amount - gap*(amount-1) gaps = np.random.multinomial(total_gap, np.ones(amount+1)/(amount+1)) # And then we get the beginning index of each sequence intervals = np.cumsum(gaps[:-1]) + np.arange(amount)*(length+gap) return intervals def mask(*shape, p=0.5): r""" Return a boolean array of the given shape. Parameters ---------- d0, d1, ..., dn : int Shape of the output. p : value in range [0,1] A probability that the elements are `True`. """ return np.random.rand(*shape) < p def wishart(nu, V): r""" Draw a random sample from the Wishart distribution. Parameters ---------- nu : int """ # TODO/FIXME: Are these correct.. D = np.shape(V)[0] if nu < D: raise ValueError("Degrees of freedom must be equal or greater than the " "dimensionality of the matrix.") X = np.random.multivariate_normal(np.zeros(D), V, size=nu) return np.dot(X, X.T) wishart_rand = wishart def invwishart_rand(nu, V): # TODO/FIXME: Are these correct.. return np.linalg.inv(wishart_rand(nu, V)) def covariance(D, size=(), nu=None): r""" Draw a random covariance matrix. Draws from inverse-Wishart distribution. The distribution of each element is independent of the dimensionality of the matrix. C ~ Inv-W(I, D) Parameters ---------- D : int Dimensionality of the covariance matrix. Returns: -------- C : (D,D) ndarray Positive-definite symmetric :math:`D\times D` matrix. """ if nu is None: nu = D if nu < D: raise ValueError("nu must be greater than or equal to D") try: size = tuple(size) except TypeError: size = (size,) shape = size + (D,nu) C = np.random.randn(*shape) C = linalg.dot(C, np.swapaxes(C, -1, -2)) / nu return linalg.inv(C) #return np.linalg.inv(np.dot(C, C.T)) def correlation(D): r""" Draw a random correlation matrix. """ X = np.random.randn(D,D); s = np.sqrt(np.sum(X**2, axis=-1, keepdims=True)) X = X / s return np.dot(X, X.T) def gaussian_logpdf(yVy, yVmu, muVmu, logdet_V, D): r""" Log-density of a Gaussian distribution. :math:`\mathcal{G}(\mathbf{y}|\boldsymbol{\mu},\mathbf{V}^{-1})` Parameters ----------- yVy : ndarray or double :math:`\mathbf{y}^T\mathbf{Vy}` yVmu : ndarray or double :math:`\mathbf{y}^T\mathbf{V}\boldsymbol{\mu}` muVmu : ndarray or double :math:`\boldsymbol{\mu}^T\mathbf{V}\boldsymbol{\mu}` logdet_V : ndarray or double Log-determinant of the precision matrix, :math:`\log|\mathbf{V}|`. D : int Dimensionality of the distribution. """ return -0.5*yVy + yVmu - 0.5*muVmu + 0.5*logdet_V - 0.5*D*np.log(2*np.pi) def gaussian_entropy(logdet_V, D): r""" Compute the entropy of a Gaussian distribution. If you want to get the gradient, just let each parameter be a gradient of that term. Parameters ---------- logdet_V : ndarray or double The log-determinant of the precision matrix. D : int The dimensionality of the distribution. """ return -0.5*logdet_V + 0.5*D + 0.5*D*np.log(2*np.pi) def gamma_logpdf(bx, logx, a_logx, a_logb, gammaln_a): r""" Log-density of :math:`\mathcal{G}(x|a,b)`. If you want to get the gradient, just let each parameter be a gradient of that term. Parameters ---------- bx : ndarray :math:`bx` logx : ndarray :math:`\log(x)` a_logx : ndarray :math:`a \log(x)` a_logb : ndarray :math:`a \log(b)` gammaln_a : ndarray :math:`\log\Gamma(a)` """ return a_logb - gammaln_a + a_logx - logx - bx #def gamma_logpdf(a, log_b, gammaln_a, def gamma_entropy(a, log_b, gammaln_a, psi_a, a_psi_a): r""" Entropy of :math:`\mathcal{G}(a,b)`. If you want to get the gradient, just let each parameter be a gradient of that term. Parameters ---------- a : ndarray :math:`a` log_b : ndarray :math:`\log(b)` gammaln_a : ndarray :math:`\log\Gamma(a)` psi_a : ndarray :math:`\psi(a)` a_psi_a : ndarray :math:`a\psi(a)` """ return a - log_b + gammaln_a + psi_a - a_psi_a def orth(D): r""" Draw random orthogonal matrix. """ Q = np.random.randn(D,D) (Q, _) = np.linalg.qr(Q) return Q def svd(s): r""" Draw a random matrix given its singular values. """ D = len(s) U = orth(D) * s V = orth(D) return np.dot(U, V.T) def sphere(N=1): r""" Draw random points uniformly on a unit sphere. Returns (latitude,longitude) in degrees. """ lon = np.random.uniform(-180, 180, N) lat = (np.arccos(np.random.uniform(-1, 1, N)) * 180 / np.pi) - 90 return (lat, lon) def bernoulli(p, size=None): r""" Draw random samples from the Bernoulli distribution. """ if isinstance(size, int): size = (size,) if size is None: size = np.shape(p) return (np.random.rand(*size) < p) def categorical(p, size=None): r""" Draw random samples from a categorical distribution. """ if size is None: size = np.shape(p)[:-1] if isinstance(size, int): size = (size,) if np.any(np.asanyarray(p)<0): raise ValueError("Array contains negative probabilities") if not misc.is_shape_subset(np.shape(p)[:-1], size): raise ValueError("Probability array shape and requested size are " "inconsistent") size = tuple(size) # Normalize probabilities p = p / np.sum(p, axis=-1, keepdims=True) # Compute cumulative probabilities (p_1, p_1+p_2, ..., p_1+...+p_N): P = np.cumsum(p, axis=-1) # Draw samples from interval [0,1] x = np.random.rand(*size) # For simplicity, repeat p to the size of the output (plus probability axis) K = np.shape(p)[-1] P = P * np.ones(tuple(size)+(K,)) if size == (): z = np.searchsorted(P, x) else: # Seach the indices z = np.zeros(size) inds = misc.nested_iterator(size) for ind in inds: z[ind] = np.searchsorted(P[ind], x[ind]) return z.astype(np.int) def dirichlet(alpha, size=None): r""" Draw random samples from the Dirichlet distribution. """ if isinstance(size, int): size = (size,) if size is None: size = np.shape(alpha) else: size = size + np.shape(alpha)[-1:] p = np.random.gamma(alpha, size=size) p /= np.sum(p, axis=-1, keepdims=True) return p def logodds_to_probability(x): r""" Solves p from log(p/(1-p)) """ return 1 / (1 + np.exp(-x)) def alpha_beta_recursion(logp0, logP): r""" Compute alpha-beta recursion for Markov chain Initial state log-probabilities are in `p0` and state transition log-probabilities are in P. The probabilities do not need to be scaled to sum to one, but they are interpreted as below: logp0 = log P(z_0) + log P(y_0|z_0) logP[...,n,:,:] = log P(z_{n+1}|z_n) + log P(y_{n+1}|z_{n+1}) """ logp0 = misc.atleast_nd(logp0, 1) logP = misc.atleast_nd(logP, 3) D = np.shape(logp0)[-1] N = np.shape(logP)[-3] plates = misc.broadcasted_shape(np.shape(logp0)[:-1], np.shape(logP)[:-3]) if np.shape(logP)[-2:] != (D,D): raise ValueError("Dimension mismatch %s != %s" % (np.shape(logP)[-2:], (D,D))) # # Run the recursion algorithm # # Allocate memory logalpha = np.zeros(plates+(N,D)) logbeta = np.zeros(plates+(N,D)) g = np.zeros(plates) # Forward recursion logalpha[...,0,:] = logp0 for n in range(1,N): # Compute: P(z_{n-1},z_n|x_1,...,x_n) v = logalpha[...,n-1,:,None] + logP[...,n-1,:,:] c = misc.logsumexp(v, axis=(-1,-2)) # Sum over z_{n-1} to get: log P(z_n|x_1,...,x_n) logalpha[...,n,:] = misc.logsumexp(v - c[...,None,None], axis=-2) g -= c # Compute the normalization of the last term v = logalpha[...,N-1,:,None] + logP[...,N-1,:,:] g -= misc.logsumexp(v, axis=(-1,-2)) # Backward recursion logbeta[...,N-1,:] = 0 for n in reversed(range(N-1)): v = logbeta[...,n+1,None,:] + logP[...,n+1,:,:] c = misc.logsumexp(v, axis=(-1,-2)) logbeta[...,n,:] = misc.logsumexp(v - c[...,None,None], axis=-1) v = logalpha[...,:,:,None] + logbeta[...,:,None,:] + logP[...,:,:,:] c = misc.logsumexp(v, axis=(-1,-2)) zz = np.exp(v - c[...,None,None]) # The logsumexp normalization is not numerically accurate, so do # normalization again: zz /= np.sum(zz, axis=(-1,-2), keepdims=True) z0 = np.sum(zz[...,0,:,:], axis=-1) z0 /= np.sum(z0, axis=-1, keepdims=True) return (z0, zz, g) def gaussian_gamma_to_t(mu, Cov, a, b, ndim=1): r""" Integrates gamma distribution to obtain parameters of t distribution """ alpha = a/b nu = 2*a S = Cov / misc.add_trailing_axes(alpha, 2*ndim) return (mu, S, nu) def t_logpdf(z2, logdet_cov, nu, D): r""" """ return (special.gammaln((nu+D)/2) - special.gammaln(nu/2) - 0.5 * D * np.log(nu*np.pi) - 0.5 * logdet_cov - 0.5 * (nu+D) * np.log(1 + z2/nu)) bayespy-0.4.0/bayespy/utils/__init__.py0000644000175000017500000000052312533543154021637 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2013 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ from . import misc from . import linalg from . import random from . import optimize bayespy-0.4.0/bayespy/utils/linalg.py0000644000175000017500000003727312533543154021362 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ General numerical functions and methods. """ import itertools import numpy as np import scipy as sp #import scipy.linalg.decomp_cholesky as decomp import scipy.linalg as linalg import scipy.special as special import scipy.optimize as optimize import scipy.sparse as sparse #import scikits.sparse.cholmod as cholmod # THIS IS SOME NEW GENERALIZED UFUNC FOR LINALG FEATURE, NOT IN OFFICIAL NUMPY # REPO YET #import numpy.linalg._gufuncs_linalg as gula #import numpy.core.gufuncs_linalg as gula from . import misc def chol(C): if sparse.issparse(C): # Sparse Cholesky decomposition (returns a Factor object) return cholmod.cholesky(C) else: # Computes Cholesky decomposition for a collection of matrices. # The last two axes of C are considered as the matrix. C = np.atleast_2d(C) U = np.empty(np.shape(C)) for i in misc.nested_iterator(np.shape(U)[:-2]): try: U[i] = linalg.cho_factor(C[i])[0] except np.linalg.linalg.LinAlgError: raise Exception("Matrix not positive definite") return U def chol_solve(U, b, out=None, matrix=False): if isinstance(U, np.ndarray): if sparse.issparse(b): b = b.toarray() if matrix: if np.ndim(b) < 2: raise ValueError("b is not a matrix") b = np.swapaxes(b, -1, -2) U = U[...,None,:,:] # Allocate memory U = np.atleast_2d(U) B = np.atleast_1d(b) sh_u = U.shape[:-2] sh_b = B.shape[:-1] l_u = len(sh_u) l_b = len(sh_b) # Check which axis are iterated over with B along with U ind_b = [slice(None)] * l_b l_min = min(l_u, l_b) jnd_b = tuple(i for i in range(-l_min,0) if sh_b[i]==sh_u[i]) if out == None: # Shape of the result (broadcasting rules) sh = misc.broadcasted_shape(sh_u, sh_b) #out = np.zeros(np.shape(B)) out = np.zeros(sh + B.shape[-1:]) for i in misc.nested_iterator(np.shape(U)[:-2]): # The goal is to run Cholesky solver once for all vectors of B # for which the matrices of U are the same (according to the # broadcasting rules). Thus, we collect all the axes of B for # which U is singleton and form them as a 2-D matrix and then # run the solver once. # Select those axes of B for which U and B are not singleton for j in jnd_b: ind_b[j] = i[j] # Collect all the axes for which U is singleton b = B[tuple(ind_b) + (slice(None),)] # Reshape it to a 2-D (or 1-D) array orig_shape = b.shape if b.ndim > 1: b = b.reshape((-1, b.shape[-1])) # slice(None) to all preceeding axes and ellipsis for the last # axis: if len(ind_b) < len(sh): ind_out = (slice(None),) + tuple(ind_b) + (slice(None),) else: ind_out = tuple(ind_b) + (slice(None),) out[ind_out] = linalg.cho_solve((U[i], False), b.T).T.reshape(orig_shape) if matrix: out = np.swapaxes(out, -1, -2) return out elif isinstance(U, cholmod.Factor): if matrix: raise NotImplementedError() if sparse.issparse(b): b = b.toarray() return U.solve_A(b) else: raise ValueError("Unknown type of Cholesky factor") def chol_inv(U): if isinstance(U, np.ndarray): # Allocate memory V = np.tile(np.identity(np.shape(U)[-1]), np.shape(U)[:-2]+(1,1)) for i in misc.nested_iterator(np.shape(U)[:-2]): V[i] = linalg.cho_solve((U[i], False), V[i], overwrite_b=True) # This would need Fortran order return V elif isinstance(U, cholmod.Factor): raise NotImplementedError ## if sparse.issparse(b): ## b = b.toarray() ## return U.solve_A(b) else: raise ValueError("Unknown type of Cholesky factor") def chol_logdet(U): if isinstance(U, np.ndarray): return 2*np.sum(np.log(np.einsum('...ii->...i',U)), axis=-1) elif isinstance(U, cholmod.Factor): return np.sum(np.log(U.D())) else: raise ValueError("Unknown type of Cholesky factor") def logdet_chol(U): if isinstance(U, np.ndarray): # Computes Cholesky decomposition for a collection of matrices. return 2*np.sum(np.log(np.einsum('...ii->...i', U)), axis=(-1,)) elif isinstance(U, cholmod.Factor): return np.sum(np.log(U.D())) def logdet_tri(R): """ Logarithm of the absolute value of the determinant of a triangular matrix. """ return np.sum(np.log(np.abs(np.einsum('...ii->...i', R)))) def logdet_cov(C): return logdet_chol(chol(C)) def solve_triangular(U, B, **kwargs): # Allocate memory U = np.atleast_2d(U) B = np.atleast_1d(B) sh_u = U.shape[:-2] sh_b = B.shape[:-1] l_u = len(sh_u) l_b = len(sh_b) # Check which axis are iterated over with B along with U ind_b = [slice(None)] * l_b l_min = min(l_u, l_b) jnd_b = tuple(i for i in range(-l_min,0) if sh_b[i]==sh_u[i]) # Shape of the result (broadcasting rules) sh = misc.broadcasted_shape(sh_u, sh_b) out = np.zeros(sh + B.shape[-1:]) for i in misc.nested_iterator(np.shape(U)[:-2]): # The goal is to run triangular solver once for all vectors of # B for which the matrices of U are the same (according to the # broadcasting rules). Thus, we collect all the axes of B for # which U is singleton and form them as a 2-D matrix and then # run the solver once. # Select those axes of B for which U and B are not singleton for j in jnd_b: ind_b[j] = i[j] # Collect all the axes for which U is singleton b = B[tuple(ind_b) + (slice(None),)] # Reshape it to a 2-D (or 1-D) array orig_shape = b.shape if b.ndim > 1: b = b.reshape((-1, b.shape[-1])) # slice(None) to all preceeding axes and ellipsis for the last # axis: if len(ind_b) < len(sh): ind_out = (slice(None),) + tuple(ind_b) + (slice(None),) else: ind_out = tuple(ind_b) + (slice(None),) out[ind_out] = linalg.solve_triangular(U[i], b.T, **kwargs).T.reshape(orig_shape) return out def inner(*args, ndim=1): """ Compute inner product. The number of arrays is arbitrary. The number of dimensions is arbitrary. """ axes = tuple(range(-ndim,0)) return misc.sum_product(*args, axes_to_sum=axes) def outer(A, B, ndim=1): """ Computes outer product over the last axes of A and B. The other axes are broadcasted. Thus, if A has shape (..., N) and B has shape (..., M), then the result has shape (..., N, M). Using the argument `ndim` it is possible to change that how many axes trailing axes are used for the outer product. For instance, if ndim=3, A and B have shapes (...,N1,N2,N3) and (...,M1,M2,M3), the result has shape (...,N1,M1,N2,M2,N3,M3). """ if not isinstance(ndim, int) or ndim < 0: raise ValueError('ndim must be non-negative integer') if ndim > 0: if ndim > np.ndim(A): raise ValueError('Argument ndim larger than ndim of the first ' 'parameter') if ndim > np.ndim(B): raise ValueError('Argument ndim larger than ndim of the second ' 'parameter') shape_A = np.shape(A) + (1,)*ndim shape_B = np.shape(B)[:-ndim] + (1,)*ndim + np.shape(B)[-ndim:] A = np.reshape(A, shape_A) B = np.reshape(B, shape_B) return np.asanyarray(A) * np.asanyarray(B) def _dot(A, B): """ Dot product which handles broadcasting properly. Future NumPy will have a better built-in implementation for this. """ A_plates = np.shape(A)[:-2] B_plates = np.shape(B)[:-2] M = np.shape(A)[-2] N = np.shape(B)[-1] Y_plates = misc.broadcasted_shape(A_plates, B_plates) if Y_plates == (): return np.dot(A, B) indices = misc.nested_iterator(Y_plates) Y_shape = Y_plates + (M, N) Y = np.zeros(Y_shape) for i in indices: Y[i] = np.dot(A[misc.safe_indices(i, A_plates)], B[misc.safe_indices(i, B_plates)]) return Y def dot(*arrays): """ Compute matrix-matrix product. You can give multiple arrays, the dot product is computed from left to right: A1*A2*A3*...*AN. The dot product is computed over the last two axes of each arrays. All other axes must be broadcastable. """ if len(arrays) == 0: return 0 else: Y = np.asanyarray(arrays[0]) for X in arrays[1:]: X = np.asanyarray(X) if np.ndim(Y) < 2 or np.ndim(X) < 2: raise ValueError("Must be at least 2-D arrays") if np.shape(Y)[-1] != np.shape(X)[-2]: raise ValueError("Dimensions do not match") # Replace this with numpy.dot when NumPy implements broadcasting in dot Y = _dot(Y, X) #Y = np.einsum('...ik,...kj->...ij', Y, X) #Y = gula.matrix_multiply(Y, X) return Y def tracedot(A, B): """ Computes trace(A*B). """ return np.einsum('...ij,...ji->...', A, B) def inv(A, ndim=1): """ General array inversion. Supports broadcasting and inversion of multidimensional arrays. For instance, an array with shape (4,3,2,3,2) could mean that there are four (3*2) x (3*2) matrices to be inverted. This can be done by inv(A, ndim=2). For inverting scalars, ndim=0. For inverting matrices, ndim=1. """ A = np.asanyarray(A) if ndim == 0: return 1 / A elif ndim == 1: return np.linalg.inv(A) else: raise NotImplementedError() def mvdot(A, b, ndim=1): """ Compute matrix-vector product. Applies broadcasting. """ # TODO/FIXME: A bug in inner1d: # https://github.com/numpy/numpy/issues/3338 # # b = np.asanyarray(b) # return gula.inner1d(A, b[...,np.newaxis,:]) # # Use einsum instead: if ndim > 0: b = misc.add_axes(b, num=ndim, axis=-1-ndim) return inner(A, b, ndim=ndim) ## if ndim != 1: ## raise NotImplementedError("mvdot not yet implemented for ndim!=1") ## return _dot(A, b[...,None])[...,0] ## #return np.einsum('...ik,...k->...i', A, b) def mmdot(A, B, ndim=1): """ Compute matrix-matrix product. Applies broadcasting. """ if ndim == 0: return A * B elif ndim == 1: return _dot(A, B) else: raise Exception("mmdot not yet implemented for ndim>1") #return np.einsum('...ik,...kj->...ij', A, B) def transpose(X, ndim=1): """ Transpose the matrix. """ for n in range(ndim): X = np.swapaxes(X, -1-n, -1-ndim-n) return X ## if ndim != 1: ## raise Exception("transpose not yet implemented for ndim!=1") ## return np.swapaxes(X, -1, -2) def m_dot(A,b): raise DeprecationWarning() # Compute matrix-vector product over the last two axes of A and # the last axes of b. Other axes are broadcasted. If A has shape # (..., M, N) and b has shape (..., N), then the result has shape # (..., M) #b = reshape(b, shape(b)[:-1] + (1,) + shape(b)[-1:]) #return np.dot(A, b) return np.einsum('...ik,...k->...i', A, b) # TODO: Use einsum!! #return np.sum(A*b[...,np.newaxis,:], axis=(-1,)) def block_banded_solve(A, B, y): """ Invert symmetric, banded, positive-definite matrix. A contains the diagonal blocks. B contains the superdiagonal blocks (their transposes are the subdiagonal blocks). Shapes: A: (..., N, D, D) B: (..., N-1, D, D) y: (..., N, D) The algorithm is basically LU decomposition. Computes only the diagonal and super-diagonal blocks of the inverse. The true inverse is dense, in general. Assume each block has the same size. Return: * inverse blocks * solution to the system * log-determinant """ # Number of time instance and dimensionality N = np.shape(y)[-2] D = np.shape(y)[-1] # Check the shape of the diagonal blocks if np.shape(A)[-3] != N: raise ValueError("The number of diagonal blocks is incorrect") if np.shape(A)[-2:] != (D,D): raise ValueError("The diagonal blocks have wrong shape") # Check the shape of the super-diagonal blocks if np.shape(B)[-3] != N-1: raise ValueError("The number of super-diagonal blocks is incorrect") if np.shape(B)[-2:] != (D,D): raise ValueError("The diagonal blocks have wrong shape") plates_VC = misc.broadcasted_shape(np.shape(A)[:-3], np.shape(B)[:-3]) plates_y = misc.broadcasted_shape(plates_VC, np.shape(y)[:-2]) V = np.empty(plates_VC+(N,D,D)) C = np.empty(plates_VC+(N-1,D,D)) x = np.empty(plates_y+(N,D)) # # Forward recursion # # In the forward recursion, store the Cholesky factor in V. So you # don't need to recompute them in the backward recursion. # TODO: This whole algorithm could be implemented as in-place operation. # Might be a nice feature (optional?) x[...,0,:] = y[...,0,:] V[...,0,:,:] = chol(A[...,0,:,:]) ldet = chol_logdet(V[...,0,:,:]) for n in range(N-1): # Compute the solution of the system x[...,n+1,:] = (y[...,n+1,:] - mvdot(misc.T(B[...,n,:,:]), chol_solve(V[...,n,:,:], x[...,n,:]))) # Compute the superdiagonal block of the inverse C[...,n,:,:] = chol_solve(V[...,n,:,:], B[...,n,:,:], matrix=True) # Compute the diagonal block V[...,n+1,:,:] = (A[...,n+1,:,:] - mmdot(misc.T(B[...,n,:,:]), C[...,n,:,:])) # Ensure symmetry by 0.5*(V+V.T) V[...,n+1,:,:] = 0.5 * (V[...,n+1,:,:] + misc.T(V[...,n+1,:,:])) # Compute and store the Cholesky factor of the diagonal block V[...,n+1,:,:] = chol(V[...,n+1,:,:]) # Compute the log-det term here, too ldet += chol_logdet(V[...,n+1,:,:]) # # Backward recursion # x[...,-1,:] = chol_solve(V[...,-1,:,:], x[...,-1,:]) V[...,-1,:,:] = chol_inv(V[...,-1,:,:]) for n in reversed(range(N-1)): # Compute the solution of the system x[...,n,:] = chol_solve(V[...,n,:,:], x[...,n,:] - mvdot(B[...,n,:,:], x[...,n+1,:])) # Compute the diagonal block of the inverse V[...,n,:,:] = (chol_inv(V[...,n,:,:]) + mmdot(C[...,n,:,:], mmdot(V[...,n+1,:,:], misc.T(C[...,n,:,:])))) C[...,n,:,:] = - mmdot(C[...,n,:,:], V[...,n+1,:,:]) # Ensure symmetry by 0.5*(V+V.T) V[...,n,:,:] = 0.5 * (V[...,n,:,:] + misc.T(V[...,n,:,:])) return (V, C, x, ldet) bayespy-0.4.0/bayespy/utils/optimize.py0000644000175000017500000000303612615643655021752 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2013 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import numpy as np from scipy import optimize def minimize(f, x0, maxiter=None, verbose=False): """ Simple wrapper for SciPy's optimize. The given function must return a tuple: (value, gradient). """ options = {'disp': verbose} if maxiter is not None: options['maxiter'] = maxiter opt = optimize.minimize(f, x0, jac=True, method='CG', options=options) return opt.x def check_gradient(f, x0, verbose=True, epsilon=optimize.optimize._epsilon, return_abserr=False): """ Simple wrapper for SciPy's gradient checker. The given function must return a tuple: (value, gradient). Returns absolute and relative errors """ df = f(x0)[1] df_num = optimize.approx_fprime(x0, lambda x: f(x)[0], epsilon) abserr = np.linalg.norm(df-df_num) norm_num = np.linalg.norm(df_num) if abserr == 0 and norm_num == 0: err = 0 else: err = abserr / norm_num if verbose: print("Norm of numerical gradient: %g" % np.linalg.norm(df_num)) print("Norm of function gradient: %g" % np.linalg.norm(df)) print("Gradient relative error = %g and absolute error = %g" % (err, abserr)) return (abserr, err) bayespy-0.4.0/bayespy/utils/covfunc/0000755000175000017500000000000012615647103021171 5ustar jluttinejluttine00000000000000bayespy-0.4.0/bayespy/utils/covfunc/covariance.py0000644000175000017500000002622412533543154023663 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2012 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import itertools import numpy as np #import scipy as sp import scipy.sparse as sp # prefer CSC format #import scipy.spatial.distance as dist from bayespy.utils import misc #from bayespy.utils.covfunc import distance from scipy.spatial import distance # Covariance matrices can be either arrays or matrices so be careful # with products and powers! Use explicit multiply or dot instead of # *-operator. def gp_cov_se(D2, overwrite=False): if overwrite: K = D2 K *= -0.5 np.exp(K, out=K) else: K = np.exp(-0.5*D2) return K def gp_cov_pp2_new(r, d, derivative=False): # Dimension dependent parameter q = 2 j = np.floor(d/2) + q + 1 # Polynomial coefficients a2 = j**2 + 4*j + 3 a1 = 3*j + 6 a0 = 3 # Two parts of the covariance function k1 = (1-r) ** (j+2) k2 = (a2*r**2 + a1*r + 3) # TODO: Check that derivative is 0, 1 or 2! if derivative == 0: # Return covariance return k1 * k2 / 3 dk1 = - (j+2) * (1-r)**(j+1) dk2 = 2*a2*r + a1 if derivative == 1: # Return first derivative of the covariance return (k1 * dk2 + dk1 * k2) / 3 ddk1 = (j+2) * (j+1) * (1-r)**j ddk2 = 2*a2 if derivative == 2: # Return second derivative of the covariance return (ddk1*k2 + 2*dk1*dk2 + k1*ddk2) / 3 def gp_cov_pp2(r, d, gradient=False): # Dimension dependent parameter j = np.floor(d/2) + 2 + 1 # Polynomial coefficients a2 = j**2 + 4*j + 3 a1 = 3*j + 6 a0 = 3 # Two parts of the covariance function k1 = (1-r) ** (j+2) k2 = (a2*r**2 + a1*r + 3) # The covariance function k = k1 * k2 / 3 if gradient: # The gradient w.r.t. r dk = k * (j+2) / (r-1) + k1 * (2*a2*r + a1) / 3 return (k, dk) else: return k def gp_cov_delta(N): # TODO: Use sparse matrices here! if N > 0: #print('in gpcovdelta', N, sp.identity(N).shape) return sp.identity(N) else: # Sparse matrices do not allow zero-length dimensions return np.identity(N) #return np.identity(N) #return np.asmatrix(np.identity(N)) def squared_distance(x1, x2): ## # Reshape arrays to 2-D arrays ## sh1 = np.shape(x1)[:-1] ## sh2 = np.shape(x2)[:-1] ## d = np.shape(x1)[-1] ## x1 = np.reshape(x1, (-1,d)) ## x2 = np.reshape(x2, (-1,d)) (m1,n1) = x1.shape (m2,n2) = x2.shape if m1 == 0 or m2 == 0: D2 = np.empty((m1,m2)) else: D2 = distance.cdist(x1, x2, metric='sqeuclidean') #D2 = distance.cdist(x1, x2, metric='sqeuclidean') #D2 = np.asmatrix(D2) # Reshape the result #D2 = np.reshape(D2, sh1 + sh2) return D2 # General rule for the parameters for covariance functions: # # (value, [ [dvalue1, ...], [dvalue2, ...], [dvalue3, ...], ...]) # # For instance, # # k = covfunc_se((1.0, []), (15, [ [1,update_grad] ])) # K = k((x1, [ [dx1,update_grad] ]), (x2, [])) # # Plain values are converted as: # value -> (value, []) def gp_standardize_input(x): if np.size(x) == 0: x = np.reshape(x, (0,0)) elif np.ndim(x) == 0: x = np.reshape(x, (1,1)) elif np.ndim(x) == 1: x = np.reshape(x, (-1,1)) elif np.ndim(x) == 2: x = np.atleast_2d(x) else: raise Exception("Standard GP inputs must be 2-dimensional") return x def gp_preprocess_inputs(x1,x2=None): #args = list(args) #if len(args) < 1 or len(args) > 2: #raise Exception("Number of inputs must be one or two") if x2 is None: x1 = gp_standardize_input(x1) return x1 else: if x1 is x2: x1 = gp_standardize_input(x1) x2 = x1 else: x1 = gp_standardize_input(x1) x2 = gp_standardize_input(x2) return (x1, x2) #return args ## def gp_preprocess_inputs(x1,x2=None): ## #args = list(args) ## #if len(args) < 1 or len(args) > 2: ## #raise Exception("Number of inputs must be one or two") ## if x2 is not None: len(args) == 2: ## if args[0] is args[1]: ## args[0] = gp_standardize_input(args[0]) ## args[1] = args[0] ## else: ## args[1] = gp_standardize_input(args[1]) ## args[0] = gp_standardize_input(args[0]) ## else: ## args[0] = gp_standardize_input(args[0]) ## return args # TODO: # General syntax for these covariance functions: # covfunc(hyper1, # hyper2, # ... # hyperN, # x1, # x2=None, # gradient=list_of_booleans_for_each_hyperparameter) def covfunc_zeros(x1, x2=None, gradient=False): inputs = gp_preprocess_inputs(*inputs) # Compute distance and covariance matrix if x2 is None: x1 = gp_preprocess_inputs(x1) # Only variance vector asked N = np.shape(x1)[0] # TODO: Use sparse matrices! K = np.zeros(N) #K = np.asmatrix(np.zeros((N,1))) else: (x1,x2) = gp_preprocess_inputs(x1,x2) # Full covariance matrix asked #x1 = inputs[0] #x2 = inputs[1] # Number of inputs x1 N1 = np.shape(x1)[0] N2 = np.shape(x2)[0] # TODO: Use sparse matrices! K = np.zeros((N1,N2)) #K = np.asmatrix(np.zeros((N1,N2))) if gradient is not False: return (K, []) else: return K def covfunc_delta(amplitude, x1, x2=None, gradient=False): # Make sure that amplitude is a scalar, not an array object amplitude = misc.array_to_scalar(amplitude) ## if gradient: ## gradient_amplitude = gradient[0] ## else: ## gradient_amplitude = [] ## inputs = gp_preprocess_inputs(*inputs) # Compute distance and covariance matrix if x2 is None: x1 = gp_preprocess_inputs(x1) # Only variance vector asked #x = inputs[0] N = np.shape(x1)[0] K = np.ones(N) * amplitude**2 else: (x1,x2) = gp_preprocess_inputs(x1,x2) # Full covariance matrix asked #x1 = inputs[0] #x2 = inputs[1] # Number of inputs x1 N1 = np.shape(x1)[0] # x1 == x2? if x1 is x2: delta = True # Delta covariance # # FIXME: Broadcasting doesn't work with sparse matrices, # so must use scalar multiplication K = gp_cov_delta(N1) * amplitude**2 #K = gp_cov_delta(N1).multiply(amplitude**2) else: delta = False # Number of inputs x2 N2 = np.shape(x2)[0] # Zero covariance if N1 > 0 and N2 > 0: K = sp.csc_matrix((N1,N2)) else: K = np.zeros((N1,N2)) # Gradient w.r.t. amplitude if gradient: # FIXME: Broadcasting doesn't work with sparse matrices, # so must use scalar multiplication gradient_amplitude = K*(2/amplitude) print("noise grad", gradient_amplitude) return (K, (gradient_amplitude,)) else: return K def covfunc_pp2(amplitude, lengthscale, x1, x2=None, gradient=False): # Make sure that hyperparameters are scalars, not an array objects amplitude = misc.array_to_scalar(amplitude) lengthscale = misc.array_to_scalar(lengthscale) #amplitude = theta[0] #lengthscale = theta[1] ## if gradient: ## gradient_amplitude = gradient[0] ## gradient_lengthscale = gradient[1] ## else: ## gradient_amplitude = [] ## gradient_lengthscale = [] ## inputs = gp_preprocess_inputs(*inputs) # Compute covariance matrix if x2 is None: x1 = gp_preprocess_inputs(x1) # Compute variance vector K = np.ones(np.shape(x)[:-1]) K *= amplitude**2 # Compute gradient w.r.t. lengthscale if gradient: gradient_lengthscale = np.zeros(np.shape(x1)[:-1]) else: (x1,x2) = gp_preprocess_inputs(x1,x2) # Compute (sparse) distance matrix if x1 is x2: x1 = x1 / (lengthscale) x2 = x1 D2 = distance.sparse_pdist(x1, 1.0, form="full", format="csc") else: x1 = x1 / (lengthscale) x2 = x2 / (lengthscale) D2 = distance.sparse_cdist(x1, x2, 1.0, format="csc") r = np.sqrt(D2.data) N1 = np.shape(x1)[0] N2 = np.shape(x2)[0] # Compute the covariances if gradient: (k, dk) = gp_cov_pp2(r, np.shape(x1)[-1], gradient=True) else: k = gp_cov_pp2(r, np.shape(x1)[-1]) k *= amplitude**2 # Compute gradient w.r.t. lengthscale if gradient: if N1 >= 1 and N2 >= 1: dk *= r * (-amplitude**2 / lengthscale) gradient_lengthscale = sp.csc_matrix((dk, D2.indices, D2.indptr), shape=(N1,N2)) else: gradient_lengthscale = np.empty((N1,N2)) # Form sparse covariance matrix if N1 >= 1 and N2 >= 1: ## K = sp.csc_matrix((k, ij), shape=(N1,N2)) K = sp.csc_matrix((k, D2.indices, D2.indptr), shape=(N1,N2)) else: K = np.empty((N1, N2)) #print(K.__class__) # Gradient w.r.t. amplitude if gradient: gradient_amplitude = K * (2 / amplitude) # Return values if gradient: print("pp2 grad", gradient_lengthscale) return (K, (gradient_amplitude, gradient_lengthscale)) else: return K def covfunc_se(amplitude, lengthscale, x1, x2=None, gradient=False): # Make sure that hyperparameters are scalars, not an array objects amplitude = misc.array_to_scalar(amplitude) lengthscale = misc.array_to_scalar(lengthscale) # Compute covariance matrix if x2 is None: x1 = gp_preprocess_inputs(x1) #x = inputs[0] # Compute variance vector N = np.shape(x1)[0] K = np.ones(N) np.multiply(K, amplitude**2, out=K) # Compute gradient w.r.t. lengthscale if gradient: # TODO: Use sparse matrices? gradient_lengthscale = np.zeros(N) else: (x1,x2) = gp_preprocess_inputs(x1,x2) x1 = x1 / (lengthscale) x2 = x2 / (lengthscale) # Compute distance matrix K = squared_distance(x1, x2) # Compute gradient partly if gradient: gradient_lengthscale = np.divide(K, lengthscale) # Compute covariance matrix gp_cov_se(K, overwrite=True) np.multiply(K, amplitude**2, out=K) # Compute gradient w.r.t. lengthscale if gradient: gradient_lengthscale *= K # Gradient w.r.t. amplitude if gradient: gradient_amplitude = K * (2 / amplitude) # Return values if gradient: print("se grad", gradient_amplitude, gradient_lengthscale) return (K, (gradient_amplitude, gradient_lengthscale)) else: return K bayespy-0.4.0/bayespy/utils/covfunc/__init__.py0000664000175000017500000000000012174013072023262 0ustar jluttinejluttine00000000000000bayespy-0.4.0/bayespy/utils/misc.py0000644000175000017500000011711012615643655021044 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2013 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ General numerical functions and methods. """ import functools import itertools import operator import sys import getopt import numpy as np import scipy as sp import scipy.linalg as linalg import scipy.special as special import scipy.optimize as optimize import scipy.sparse as sparse import tempfile as tmp import unittest from numpy import testing def find_set_index(index, set_lengths): """ Given set sizes and an index, returns the index of the set The given index is for the concatenated list of the sets. """ # Negative indices to positive if index < 0: index += np.sum(set_lengths) # Indices must be on range (0, N-1) if index >= np.sum(set_lengths) or index < 0: raise Exception("Index out bounds") return np.searchsorted(np.cumsum(set_lengths), index, side='right') def parse_command_line_arguments(mandatory_args, *optional_args_list, argv=None): """ Parse command line arguments of style "--parameter=value". Parameter specification is tuple: (name, converter, description). Some special handling: * If converter is None, the command line does not accept any value for it, but instead use either "--option" to enable or "--no-option" to disable. * If argument name contains hyphens, those are converted to underscores in the keys of the returned dictionaries. Parameters ---------- mandatory_args : list of tuples Specs for mandatory arguments optional_args_list : list of lists of tuples Specs for each optional arguments set argv : list of strings (optional) The command line arguments. By default, read sys.argv. Returns ------- args : dictionary The parsed mandatory arguments kwargs : dictionary The parsed optional arguments Examples -------- >>> from pprint import pprint as print >>> (args, kwargs) = parse_command_line_arguments( ... # Mandatory arguments ... [ ... ('name', str, "Full name"), ... ('age', int, "Age (years)"), ... ('employed', None, "Working"), ... ], ... # Optional arguments ... [ ... ('phone', str, "Phone number"), ... ('favorite-color', str, "Favorite color") ... ], ... argv=['--name=John Doe', ... '--age=42', ... '--no-employed', ... '--favorite-color=pink'] ... ) >>> print(args) {'age': 42, 'employed': False, 'name': 'John Doe'} >>> print(kwargs) {'favorite_color': 'pink'} It is possible to have several optional argument sets: >>> (args, kw_info, kw_fav) = parse_command_line_arguments( ... # Mandatory arguments ... [ ... ('name', str, "Full name"), ... ], ... # Optional arguments (contact information) ... [ ... ('phone', str, "Phone number"), ... ('email', str, "E-mail address") ... ], ... # Optional arguments (preferences) ... [ ... ('favorite-color', str, "Favorite color"), ... ('favorite-food', str, "Favorite food") ... ], ... argv=['--name=John Doe', ... '--favorite-color=pink', ... '--email=john.doe@email.com', ... '--favorite-food=spaghetti'] ... ) >>> print(args) {'name': 'John Doe'} >>> print(kw_info) {'email': 'john.doe@email.com'} >>> print(kw_fav) {'favorite_color': 'pink', 'favorite_food': 'spaghetti'} """ if argv is None: argv = sys.argv[1:] mandatory_arg_names = [arg[0] for arg in mandatory_args] # Sizes of each optional argument list optional_args_lengths = [len(opt_args) for opt_args in optional_args_list] all_args = mandatory_args + functools.reduce(operator.add, optional_args_list) # Create a list of arg names for the getopt parser arg_list = [] for arg in all_args: arg_name = arg[0].lower() if arg[1] is None: arg_list.append(arg_name) arg_list.append("no-" + arg_name) else: arg_list.append(arg_name + "=") if len(set(arg_list)) < len(arg_list): raise Exception("Argument names are not unique") # Use getopt parser try: (cl_opts, cl_args) = getopt.getopt(argv, "", arg_list) except getopt.GetoptError as err: print(err) print("Usage:") for arg in all_args: if arg[1] is None: print("--{0}\t{1}".format(arg[0].lower(), arg[2])) else: print("--{0}=<{1}>\t{2}".format(arg[0].lower(), str(arg[1].__name__).upper(), arg[2])) sys.exit(2) # A list of all valid flag names: ["--first-argument", "--another-argument"] valid_flags = [] valid_flag_arg_indices = [] for (ind, arg) in enumerate(all_args): valid_flags.append("--" + arg[0].lower()) valid_flag_arg_indices.append(ind) if arg[1] is None: valid_flags.append("--no-" + arg[0].lower()) valid_flag_arg_indices.append(ind) # Go through all the given command line arguments and store them in the # correct dictionaries args = dict() kwargs_list = [dict() for i in range(len(optional_args_list))] handled_arg_names = [] for (cl_opt, cl_arg) in cl_opts: # Get the index of the argument try: ind = valid_flag_arg_indices[valid_flags.index(cl_opt.lower())] except ValueError: print("Invalid command line argument: {0}".format(cl_opt)) raise Exception("Invalid argument given") # Check that the argument wasn't already given and then mark the # argument as handled if all_args[ind][0] in handled_arg_names: raise Exception("Same argument given multiple times") else: handled_arg_names.append(all_args[ind][0]) # Check whether to add the argument to the mandatory or optional # argument dictionary if ind < len(mandatory_args): dict_to = args else: dict_index = find_set_index(ind - len(mandatory_args), optional_args_lengths) dict_to = kwargs_list[dict_index] # Convert and store the argument convert_function = all_args[ind][1] arg_name = all_args[ind][0].replace('-', '_') if convert_function is None: if cl_opt[:5] == "--no-": dict_to[arg_name] = False else: dict_to[arg_name] = True else: dict_to[arg_name] = convert_function(cl_arg) # Check if some mandatory argument was not given for arg_name in mandatory_arg_names: if arg_name not in handled_arg_names: raise Exception("Mandatory argument --{0} not given".format(arg_name)) return tuple([args] + kwargs_list) def composite_function(function_list): """ Construct a function composition from a list of functions. Given a list of functions [f,g,h], constructs a function :math:`h \circ g \circ f`. That is, returns a function :math:`z`, for which :math:`z(x) = h(g(f(x)))`. """ def composite(X): for function in function_list: X = function(X) return X return composite def ceildiv(a, b): """ Compute a divided by b and rounded up. """ return -(-a // b) def rmse(y1, y2, axis=None): return np.sqrt(np.mean((y1-y2)**2, axis=axis)) def is_callable(f): return hasattr(f, '__call__') def atleast_nd(X, d): if np.ndim(X) < d: sh = (d-np.ndim(X))*(1,) + np.shape(X) X = np.reshape(X, sh) return X def T(X): """ Transpose the matrix. """ return np.swapaxes(X, -1, -2) class TestCase(unittest.TestCase): """ Simple base class for unit testing. Adds NumPy's features to Python's unittest. """ def assertAllClose(self, A, B, msg="Arrays not almost equal", rtol=1e-4, atol=0): self.assertEqual(np.shape(A), np.shape(B), msg=msg) testing.assert_allclose(A, B, err_msg=msg, rtol=rtol, atol=atol) pass def assertArrayEqual(self, A, B, msg="Arrays not equal"): self.assertEqual(np.shape(A), np.shape(B), msg=msg) testing.assert_array_equal(A, B, err_msg=msg) pass def assertMessage(self, M1, M2): if len(M1) != len(M2): self.fail("Message lists have different lengths") for (m1, m2) in zip(M1, M2): self.assertAllClose(m1, m2) pass def assertMessageToChild(self, X, u): self.assertMessage(X._message_to_child(), u) pass def symm(X): """ Make X symmetric. """ return 0.5 * (X + np.swapaxes(X, -1, -2)) def unique(l): """ Remove duplicate items from a list while preserving order. """ seen = set() seen_add = seen.add return [ x for x in l if x not in seen and not seen_add(x)] def tempfile(prefix='', suffix=''): return tmp.NamedTemporaryFile(prefix=prefix, suffix=suffix).name def write_to_hdf5(group, data, name): """ Writes the given array into the HDF5 file. """ try: # Try using compression. It doesn't work for scalars. group.create_dataset(name, data=data, compression='gzip') except TypeError: group.create_dataset(name, data=data) except ValueError: raise ValueError('Could not write %s' % data) def nans(size=()): return np.tile(np.nan, size) def trues(shape): return np.ones(shape, dtype=np.bool) def identity(*shape): return np.reshape(np.identity(np.prod(shape)), shape+shape) def array_to_scalar(x): # This transforms an N-dimensional array to a scalar. It's most # useful when you know that the array has only one element and you # want it out as a scalar. return np.ravel(x)[0] #def diag(x): def grid(x1, x2): """ Returns meshgrid as a (M*N,2)-shape array. """ (X1, X2) = np.meshgrid(x1, x2) return np.hstack((X1.reshape((-1,1)),X2.reshape((-1,1)))) class CholeskyDense(): def __init__(self, K): self.U = linalg.cho_factor(K) def solve(self, b): if sparse.issparse(b): b = b.toarray() return linalg.cho_solve(self.U, b) def logdet(self): return 2*np.sum(np.log(np.diag(self.U[0]))) def trace_solve_gradient(self, dK): return np.trace(self.solve(dK)) class CholeskySparse(): def __init__(self, K): self.LD = cholmod.cholesky(K) def solve(self, b): if sparse.issparse(b): b = b.toarray() return self.LD.solve_A(b) def logdet(self): return self.LD.logdet() #np.sum(np.log(LD.D())) def trace_solve_gradient(self, dK): # WTF?! numpy.multiply doesn't work for two sparse # matrices.. It returns a result but it is incorrect! # Use the identity trace(K\dK)=sum(inv(K).*dK) by computing # the sparse inverse (lower triangular part) iK = self.LD.spinv(form='lower') return (2*iK.multiply(dK).sum() - iK.diagonal().dot(dK.diagonal())) # Multiply by two because of symmetry (remove diagonal once # because it was taken into account twice) #return np.multiply(self.LD.inv().todense(),dK.todense()).sum() #return self.LD.inv().multiply(dK).sum() # THIS WORKS #return np.multiply(self.LD.inv(),dK).sum() # THIS NOT WORK!! WTF?? iK = self.LD.spinv() return iK.multiply(dK).sum() #return (2*iK.multiply(dK).sum() # - iK.diagonal().dot(dK.diagonal())) #return (2*np.multiply(iK, dK).sum() # - iK.diagonal().dot(dK.diagonal())) # THIS NOT WORK!! #return np.trace(self.solve(dK)) def cholesky(K): if isinstance(K, np.ndarray): return CholeskyDense(K) elif sparse.issparse(K): return CholeskySparse(K) else: raise Exception("Unsupported covariance matrix type") # Computes log probability density function of the Gaussian # distribution def gaussian_logpdf(y_invcov_y, y_invcov_mu, mu_invcov_mu, logdetcov, D): return (-0.5*D*np.log(2*np.pi) -0.5*logdetcov -0.5*y_invcov_y +y_invcov_mu -0.5*mu_invcov_mu) def zipper_merge(*lists): """ Combines lists by alternating elements from them. Combining lists [1,2,3], ['a','b','c'] and [42,666,99] results in [1,'a',42,2,'b',666,3,'c',99] The lists should have equal length or they are assumed to have the length of the shortest list. This is known as alternating merge or zipper merge. """ return list(sum(zip(*lists), ())) def remove_whitespace(s): return ''.join(s.split()) def is_numeric(a): return (np.isscalar(a) or isinstance(a, list) or isinstance(a, np.ndarray)) def isinteger(x): t = np.asanyarray(x).dtype.type return ( issubclass(t, np.integer) or issubclass(t, np.bool_) ) def is_string(s): return isinstance(s, str) def multiply_shapes(*shapes): """ Compute element-wise product of lists/tuples. Shorter lists are concatenated with leading 1s in order to get lists with the same length. """ # Make the shapes equal length shapes = make_equal_length(*shapes) # Compute element-wise product f = lambda X,Y: (x*y for (x,y) in zip(X,Y)) shape = functools.reduce(f, shapes) return tuple(shape) def make_equal_length(*shapes): """ Make tuples equal length. Add leading 1s to shorter tuples. """ # Get maximum length max_len = max((len(shape) for shape in shapes)) # Make the shapes equal length shapes = ((1,)*(max_len-len(shape)) + tuple(shape) for shape in shapes) return shapes def make_equal_ndim(*arrays): """ Add trailing unit axes so that arrays have equal ndim """ shapes = [np.shape(array) for array in arrays] shapes = make_equal_length(*shapes) arrays = [np.reshape(array, shape) for (array, shape) in zip(arrays, shapes)] return arrays def sum_to_dim(A, dim): """ Sum leading axes of A such that A has dim dimensions. """ dimdiff = np.ndim(A) - dim if dimdiff > 0: axes = np.arange(dimdiff) A = np.sum(A, axis=axes) return A def broadcasting_multiplier(plates, *args): """ Compute the plate multiplier for given shapes. The first shape is compared to all other shapes (using NumPy broadcasting rules). All the elements which are non-unit in the first shape but 1 in all other shapes are multiplied together. This method is used, for instance, for computing a correction factor for messages to parents: If this node has non-unit plates that are unit plates in the parent, those plates are summed. However, if the message has unit axis for that plate, it should be first broadcasted to the plates of this node and then summed to the plates of the parent. In order to avoid this broadcasting and summing, it is more efficient to just multiply by the correct factor. This method computes that factor. The first argument is the full plate shape of this node (with respect to the parent). The other arguments are the shape of the message array and the plates of the parent (with respect to this node). """ # Check broadcasting of the shapes for arg in args: broadcasted_shape(plates, arg) # Check that each arg-plates are a subset of plates? for arg in args: if not is_shape_subset(arg, plates): print("Plates:", plates) print("Args:", args) raise ValueError("The shapes in args are not a sub-shape of " "plates") r = 1 for j in range(-len(plates),0): mult = True for arg in args: # if -j <= len(arg) and arg[j] != 1: if not (-j > len(arg) or arg[j] == 1): mult = False if mult: r *= plates[j] return r def sum_multiply_to_plates(*arrays, to_plates=(), from_plates=None, ndim=0): """ Compute the product of the arguments and sum to the target shape. """ arrays = list(arrays) def get_plates(x): if ndim == 0: return x else: return x[:-ndim] plates_arrays = [get_plates(np.shape(array)) for array in arrays] product_plates = broadcasted_shape(*plates_arrays) if from_plates is None: from_plates = product_plates r = 1 else: r = broadcasting_multiplier(from_plates, product_plates, to_plates) for ind in range(len(arrays)): plates_others = plates_arrays[:ind] + plates_arrays[(ind+1):] plates_without = broadcasted_shape(to_plates, *plates_others) ax = axes_to_collapse(plates_arrays[ind], #get_plates(np.shape(arrays[ind])), plates_without) if ax: ax = tuple([a-ndim for a in ax]) arrays[ind] = np.sum(arrays[ind], axis=ax, keepdims=True) plates_arrays = [get_plates(np.shape(array)) for array in arrays] product_plates = broadcasted_shape(*plates_arrays) ax = axes_to_collapse(product_plates, to_plates) if ax: ax = tuple([a-ndim for a in ax]) y = sum_multiply(*arrays, axis=ax, keepdims=True) else: y = functools.reduce(np.multiply, arrays) y = squeeze_to_dim(y, len(to_plates) + ndim) return r * y def sum_multiply(*args, axis=None, sumaxis=True, keepdims=False): # Computes sum(arg[0]*arg[1]*arg[2]*..., axis=axes_to_sum) without # explicitly computing the intermediate product if len(args) == 0: raise ValueError("You must give at least one input array") # Dimensionality of the result max_dim = 0 for k in range(len(args)): max_dim = max(max_dim, np.ndim(args[k])) if sumaxis: if axis is None: # Sum all axes axes = [] else: if np.isscalar(axis): axis = [axis] axes = [i for i in range(max_dim) if i not in axis and (-max_dim+i) not in axis] else: if axis is None: # Keep all axes axes = range(max_dim) else: # Find axes that are kept if np.isscalar(axis): axes = [axis] axes = [i if i >= 0 else i+max_dim for i in axis] axes = sorted(axes) if len(axes) > 0 and (min(axes) < 0 or max(axes) >= max_dim): raise ValueError("Axis index out of bounds") # Form a list of pairs: the array in the product and its axes pairs = list() for i in range(len(args)): a = args[i] a_dim = np.ndim(a) pairs.append(a) pairs.append(range(max_dim-a_dim, max_dim)) # Output axes are those which are not summed pairs.append(axes) # Compute the sum-product try: y = np.einsum(*pairs) except ValueError as err: if str(err) == ("If 'op_axes' or 'itershape' is not NULL in " "theiterator constructor, 'oa_ndim' must be greater " "than zero"): # TODO/FIXME: Handle a bug in NumPy. If all arguments to einsum are # scalars, it raises an error. For scalars we can just use multiply # and forget about summing. Hopefully, in the future, einsum handles # scalars properly and this try-except becomes unnecessary. y = functools.reduce(np.multiply, args) else: raise err # Restore summed axes as singleton axes if keepdims: d = 0 s = () for k in range(max_dim): if k in axes: # Axis not summed s = s + (np.shape(y)[d],) d += 1 else: # Axis was summed s = s + (1,) y = np.reshape(y, s) return y def sum_product(*args, axes_to_keep=None, axes_to_sum=None, keepdims=False): if axes_to_keep is not None: return sum_multiply(*args, axis=axes_to_keep, sumaxis=False, keepdims=keepdims) else: return sum_multiply(*args, axis=axes_to_sum, sumaxis=True, keepdims=keepdims) def moveaxis(A, axis_from, axis_to): """ Move the axis `axis_from` to position `axis_to`. """ if ((axis_from < 0 and abs(axis_from) > np.ndim(A)) or (axis_from >= 0 and axis_from >= np.ndim(A)) or (axis_to < 0 and abs(axis_to) > np.ndim(A)) or (axis_to >= 0 and axis_to >= np.ndim(A))): raise ValueError("Can't move axis %d to position %d. Axis index out of " "bounds for array with shape %s" % (axis_from, axis_to, np.shape(A))) axes = np.arange(np.ndim(A)) axes[axis_from:axis_to] += 1 axes[axis_from:axis_to:-1] -= 1 axes[axis_to] = axis_from return np.transpose(A, axes=axes) def safe_indices(inds, shape): """ Makes sure that indices are valid for given shape. The shorter shape determines the length. For instance, .. testsetup:: from bayespy.utils.misc import safe_indices >>> safe_indices( (3, 4, 5), (1, 6) ) (0, 5) """ m = min(len(inds), len(shape)) if m == 0: return () inds = inds[-m:] maxinds = np.array(shape[-m:]) - 1 return tuple(np.fmin(inds, maxinds)) def broadcasted_shape(*shapes): """ Computes the resulting broadcasted shape for a given set of shapes. Uses the broadcasting rules of NumPy. Raises an exception if the shapes do not broadcast. """ dim = 0 for a in shapes: dim = max(dim, len(a)) S = () for i in range(-dim,0): s = 1 for a in shapes: if -i <= len(a): if s == 1: s = a[i] elif a[i] != 1 and a[i] != s: raise ValueError("Shapes %s do not broadcast" % (shapes,)) S = S + (s,) return S def broadcasted_shape_from_arrays(*arrays): """ Computes the resulting broadcasted shape for a given set of arrays. Raises an exception if the shapes do not broadcast. """ shapes = [np.shape(array) for array in arrays] return broadcasted_shape(*shapes) def is_shape_subset(sub_shape, full_shape): """ """ if len(sub_shape) > len(full_shape): return False for i in range(len(sub_shape)): ind = -1 - i if sub_shape[ind] != 1 and sub_shape[ind] != full_shape[ind]: return False return True def add_axes(X, num=1, axis=0): for i in range(num): X = np.expand_dims(X, axis=axis) return X shape = np.shape(X)[:axis] + num*(1,) + np.shape(X)[axis:] return np.reshape(X, shape) def add_leading_axes(x, n): return add_axes(x, axis=0, num=n) def add_trailing_axes(x, n): return add_axes(x, axis=-1, num=n) ## def add_axes(x, lead, trail): ## shape = (1,)*lead + np.shape(x) + (1,)*trail ## return np.reshape(x, shape) def nested_iterator(max_inds): s = [range(i) for i in max_inds] return itertools.product(*s) def first(L): """ """ for (n,l) in enumerate(L): if l: return n return None def squeeze(X): """ Remove leading axes that have unit length. For instance, a shape (1,1,4,1,3) will be reshaped to (4,1,3). """ shape = np.array(np.shape(X)) inds = np.nonzero(shape != 1)[0] if len(inds) == 0: shape = () else: shape = shape[inds[0]:] return np.reshape(X, shape) def squeeze_to_dim(X, dim): s = tuple(range(np.ndim(X)-dim)) return np.squeeze(X, axis=s) def axes_to_collapse(shape_x, shape_to): # Solves which axes of shape shape_x need to be collapsed in order # to get the shape shape_to s = () for j in range(-len(shape_x), 0): if shape_x[j] != 1: if -j > len(shape_to) or shape_to[j] == 1: s += (j,) elif shape_to[j] != shape_x[j]: print('Shape from: ' + str(shape_x)) print('Shape to: ' + str(shape_to)) raise Exception('Incompatible shape to squeeze') return tuple(s) def sum_to_shape(X, s): """ Sum axes of the array such that the resulting shape is as given. Thus, the shape of the result will be s or an error is raised. """ # First, sum and remove axes that are not in s if np.ndim(X) > len(s): axes = tuple(range(-np.ndim(X), -len(s))) else: axes = () Y = np.sum(X, axis=axes) # Second, sum axes that are 1 in s but keep the axes axes = () for i in range(-np.ndim(Y), 0): if s[i] == 1: if np.shape(Y)[i] > 1: axes = axes + (i,) else: if np.shape(Y)[i] != s[i]: raise ValueError("Shape %s can't be summed to shape %s" % (np.shape(X), s)) Y = np.sum(Y, axis=axes, keepdims=True) return Y def repeat_to_shape(A, s): # Current shape t = np.shape(A) if len(t) > len(s): raise Exception("Can't repeat to a smaller shape") # Add extra axis t = tuple([1]*(len(s)-len(t))) + t A = np.reshape(A,t) # Repeat for i in reversed(range(len(s))): if s[i] != t[i]: if t[i] != 1: raise Exception("Can't repeat non-singular dimensions") else: A = np.repeat(A, s[i], axis=i) return A #def spinv_chol(L): def chol(C): if sparse.issparse(C): # Sparse Cholesky decomposition (returns a Factor object) return cholmod.cholesky(C) else: # Dense Cholesky decomposition return linalg.cho_factor(C)[0] def chol_solve(U, b): if isinstance(U, np.ndarray): if sparse.issparse(b): b = b.toarray() return linalg.cho_solve((U, False), b) elif isinstance(U, cholmod.Factor): if sparse.issparse(b): b = b.toarray() return U.solve_A(b) else: raise ValueError("Unknown type of Cholesky factor") def chol_inv(U): if isinstance(U, np.ndarray): I = np.identity(np.shape(U)[-1]) return linalg.cho_solve((U, False), I) elif isinstance(U, cholmod.Factor): raise NotImplementedError ## if sparse.issparse(b): ## b = b.toarray() ## return U.solve_A(b) else: raise ValueError("Unknown type of Cholesky factor") def chol_logdet(U): if isinstance(U, np.ndarray): return 2*np.sum(np.log(np.diag(U))) elif isinstance(U, cholmod.Factor): return np.sum(np.log(U.D())) else: raise ValueError("Unknown type of Cholesky factor") def logdet_chol(U): if isinstance(U, np.ndarray): return 2*np.sum(np.log(np.diag(U))) elif isinstance(U, cholmod.Factor): return np.sum(np.log(U.D())) def m_solve_triangular(U, B, **kwargs): # Allocate memory U = np.atleast_2d(U) B = np.atleast_1d(B) sh_u = U.shape[:-2] sh_b = B.shape[:-1] l_u = len(sh_u) l_b = len(sh_b) # Check which axis are iterated over with B along with U ind_b = [Ellipsis] * l_b l_min = min(l_u, l_b) jnd_b = tuple(i for i in range(-l_min,0) if sh_b[i]==sh_u[i]) # Shape of the result (broadcasting rules) sh = broadcasted_shape(sh_u, sh_b) #out = np.zeros(np.shape(B)) out = np.zeros(sh + B.shape[-1:]) ## if out == None: ## # Shape of the result (broadcasting rules) ## sh = broadcasted_shape(sh_u, sh_b) ## #out = np.zeros(np.shape(B)) ## out = np.zeros(sh + B.shape[-1:]) for i in nested_iterator(np.shape(U)[:-2]): # The goal is to run triangular solver once for all vectors of # B for which the matrices of U are the same (according to the # broadcasting rules). Thus, we collect all the axes of B for # which U is singleton and form them as a 2-D matrix and then # run the solver once. # Select those axes of B for which U and B are not singleton for j in jnd_b: ind_b[j] = i[j] # Collect all the axes for which U is singleton b = B[tuple(ind_b) + (Ellipsis,)] # Reshape it to a 2-D (or 1-D) array orig_shape = b.shape if b.ndim > 1: b = b.reshape((-1, b.shape[-1])) # Ellipsis to all preceeding axes and ellipsis for the last # axis: if len(ind_b) < len(sh): ind_out = (Ellipsis,) + tuple(ind_b) + (Ellipsis,) else: ind_out = tuple(ind_b) + (Ellipsis,) #print('utils.m_solve_triangular', np.shape(U[i]), np.shape(b)) out[ind_out] = linalg.solve_triangular(U[i], b.T, **kwargs).T.reshape(orig_shape) #out[ind_out] = out[ind_out].T.reshape(orig_shape) return out def multidigamma(a, d): """ Returns the derivative of the log of multivariate gamma. """ return np.sum(special.digamma(a[...,None] - 0.5*np.arange(d)), axis=-1) m_digamma = multidigamma def m_outer(A,B): # Computes outer product over the last axes of A and B. The other # axes are broadcasted. Thus, if A has shape (..., N) and B has # shape (..., M), then the result has shape (..., N, M) A = np.asanyarray(A) B = np.asanyarray(B) return A[...,np.newaxis]*B[...,np.newaxis,:] def diagonal(A): return np.diagonal(A, axis1=-2, axis2=-1) def get_diag(X, ndim=1): """ Get the diagonal of an array. If ndim>1, take the diagonal of the last 2*ndim axes. """ if ndim == 0: return X if ndim < 0: raise ValueError("Parameter ndim must be non-negative integer") if np.ndim(X) < 2*ndim: raise ValueError("The array does not have enough axes") if np.shape(X)[-ndim:] != np.shape(X)[-2*ndim:-ndim]: raise ValueError("The array X is not square") axes_out = tuple(range(np.ndim(X)-ndim, 0, -1)) axes_dim = tuple(range(ndim, 0, -1)) return np.einsum(X, axes_out+axes_dim, axes_out) def diag(X, ndim=1): """ Create a diagonal array given the diagonal elements. The diagonal array can be multi-dimensional. By default, the last axis is transformed to two axes (diagonal matrix) but this can be changed using ndim keyword. For instance, an array with shape (K,L,M,N) can be transformed to a set of diagonal 4-D tensors with shape (K,L,M,N,M,N) by giving ndim=2. If ndim=3, the result has shape (K,L,M,N,L,M,N), and so on. Diagonality means that for the resulting array Y holds: Y[...,i_1,i_2,..,i_ndim,j_1,j_2,..,j_ndim] is zero if i_n!=j_n for any n. """ X = atleast_nd(X, ndim) if ndim > 0: I = identity(*(np.shape(X)[-ndim:])) X = add_axes(X, axis=np.ndim(X), num=ndim) X = I * X return X def m_dot(A,b): # Compute matrix-vector product over the last two axes of A and # the last axes of b. Other axes are broadcasted. If A has shape # (..., M, N) and b has shape (..., N), then the result has shape # (..., M) #b = reshape(b, shape(b)[:-1] + (1,) + shape(b)[-1:]) #return np.dot(A, b) return np.einsum('...ik,...k->...i', A, b) # TODO: Use einsum!! #return np.sum(A*b[...,np.newaxis,:], axis=(-1,)) def block_banded(D, B): """ Construct a symmetric block-banded matrix. `D` contains square diagonal blocks. `B` contains super-diagonal blocks. The resulting matrix is: D[0], B[0], 0, 0, ..., 0, 0, 0 B[0].T, D[1], B[1], 0, ..., 0, 0, 0 0, B[1].T, D[2], B[2], ..., ..., ..., ... ... ... ... ... ..., B[N-2].T, D[N-1], B[N-1] 0, 0, 0, 0, ..., 0, B[N-1].T, D[N] """ D = [np.atleast_2d(d) for d in D] B = [np.atleast_2d(b) for b in B] # Number of diagonal blocks N = len(D) if len(B) != N-1: raise ValueError("The number of super-diagonal blocks must contain " "exactly one block less than the number of diagonal " "blocks") # Compute the size of the full matrix M = 0 for i in range(N): if np.ndim(D[i]) != 2: raise ValueError("Blocks must be 2 dimensional arrays") d = np.shape(D[i]) if d[0] != d[1]: raise ValueError("Diagonal blocks must be square") M += d[0] for i in range(N-1): if np.ndim(B[i]) != 2: raise ValueError("Blocks must be 2 dimensional arrays") b = np.shape(B[i]) if b[0] != np.shape(D[i])[1] or b[1] != np.shape(D[i+1])[0]: raise ValueError("Shapes of the super-diagonal blocks do not match " "the shapes of the diagonal blocks") A = np.zeros((M,M)) k = 0 for i in range(N-1): (d0, d1) = np.shape(B[i]) # Diagonal block A[k:k+d0, k:k+d0] = D[i] # Super-diagonal block A[k:k+d0, k+d0:k+d0+d1] = B[i] # Sub-diagonal block A[k+d0:k+d0+d1, k:k+d0] = B[i].T k += d0 A[k:,k:] = D[-1] return A def kalman_filter(y, U, A, V, mu0, Cov0, out=None): """ Perform Kalman filtering to obtain filtered mean and covariance. The parameters of the process may vary in time, thus they are given as iterators instead of fixed values. Parameters ---------- y : (N,D) array "Normalized" noisy observations of the states, that is, the observations multiplied by the precision matrix U (and possibly other transformation matrices). U : (N,D,D) array or N-list of (D,D) arrays Precision matrix (i.e., inverse covariance matrix) of the observation noise for each time instance. A : (N-1,D,D) array or (N-1)-list of (D,D) arrays Dynamic matrix for each time instance. V : (N-1,D,D) array or (N-1)-list of (D,D) arrays Covariance matrix of the innovation noise for each time instance. Returns ------- mu : array Filtered mean of the states. Cov : array Filtered covariance of the states. See also -------- rts_smoother """ mu = mu0 Cov = Cov0 # Allocate memory for the results (N,D) = np.shape(y) X = np.empty((N,D)) CovX = np.empty((N,D,D)) # Update step for t=0 M = np.dot(np.dot(Cov, U[0]), Cov) + Cov L = chol(M) mu = np.dot(Cov, chol_solve(L, np.dot(Cov,y[0]) + mu)) Cov = np.dot(Cov, chol_solve(L, Cov)) X[0,:] = mu CovX[0,:,:] = Cov #for (yn, Un, An, Vn) in zip(y, U, A, V): for n in range(len(y)-1): #(yn, Un, An, Vn) in zip(y, U, A, V): # Prediction step mu = np.dot(A[n], mu) Cov = np.dot(np.dot(A[n], Cov), A[n].T) + V[n] # Update step M = np.dot(np.dot(Cov, U[n+1]), Cov) + Cov L = chol(M) mu = np.dot(Cov, chol_solve(L, np.dot(Cov,y[n+1]) + mu)) Cov = np.dot(Cov, chol_solve(L, Cov)) # Force symmetric covariance (for numeric inaccuracy) Cov = 0.5*Cov + 0.5*Cov.T # Store results X[n+1,:] = mu CovX[n+1,:,:] = Cov return (X, CovX) def rts_smoother(mu, Cov, A, V, removethis=None): """ Perform Rauch-Tung-Striebel smoothing to obtain the posterior. The function returns the posterior mean and covariance of each state. The parameters of the process may vary in time, thus they are given as iterators instead of fixed values. Parameters ---------- mu : (N,D) array Mean of the states from Kalman filter. Cov : (N,D,D) array Covariance of the states from Kalman filter. A : (N-1,D,D) array or (N-1)-list of (D,D) arrays Dynamic matrix for each time instance. V : (N-1,D,D) array or (N-1)-list of (D,D) arrays Covariance matrix of the innovation noise for each time instance. Returns ------- mu : array Posterior mean of the states. Cov : array Posterior covariance of the states. See also -------- kalman_filter """ N = len(mu) #n = N-1 # Start from the last time instance and smoothen backwards x = mu[-1,:] Covx = Cov[-1,:,:] for n in reversed(range(N-1)):#(An, Vn) in zip(reversed(A), reversed(V)): #n = n - 1 #if n <= 0: # break # The predicted value of n x_p = np.dot(A[n], mu[n,:]) Cov_p = np.dot(np.dot(A[n], Cov[n,:,:]), A[n].T) + V[n] # Temporary variable S = np.linalg.solve(Cov_p, np.dot(A[n], Cov[n,:,:])) # Smoothed value of n x = mu[n,:] + np.dot(S.T, x-x_p) Covx = Cov[n,:,:] + np.dot(np.dot(S.T, Covx-Cov_p), S) # Force symmetric covariance (for numeric inaccuracy) Covx = 0.5*Covx + 0.5*Covx.T # Store results mu[n,:] = x Cov[n,:] = Covx return (mu, Cov) def dist_haversine(c1, c2, radius=6372795): # Convert coordinates to radians lat1 = np.atleast_1d(c1[0])[...,:,None] * np.pi / 180 lon1 = np.atleast_1d(c1[1])[...,:,None] * np.pi / 180 lat2 = np.atleast_1d(c2[0])[...,None,:] * np.pi / 180 lon2 = np.atleast_1d(c2[1])[...,None,:] * np.pi / 180 dlat = lat2 - lat1 dlon = lon2 - lon1 A = np.sin(dlat/2)**2 + np.cos(lat1)*np.cos(lat2)*(np.sin(dlon/2)**2) C = 2 * np.arctan2(np.sqrt(A), np.sqrt(1-A)) return radius * C def logsumexp(X, axis=None, keepdims=False): """ Compute log(sum(exp(X)) in a numerically stable way """ X = np.asanyarray(X) maxX = np.amax(X, axis=axis, keepdims=True) if np.ndim(maxX) > 0: maxX[~np.isfinite(maxX)] = 0 elif not np.isfinite(maxX): maxX = 0 X = X - maxX if not keepdims: maxX = np.squeeze(maxX, axis=axis) return np.log(np.sum(np.exp(X), axis=axis, keepdims=keepdims)) + maxX def invpsi(x): r""" Inverse digamma (psi) function. The digamma function is the derivative of the log gamma function. This calculates the value Y > 0 for a value X such that digamma(Y) = X. See: http://www4.ncsu.edu/~pfackler/ """ L = 1.0 y = np.exp(x) while (L > 1e-10): y += L*np.sign(x-special.psi(y)) L /= 2 # Ad hoc by Jaakko y[x<-10] = -1/x[x<-10] return y def invgamma(x): r""" Inverse gamma function. See: http://mathoverflow.net/a/28977 """ k = 1.461632 c = 0.036534 L = np.log((x+c)/np.sqrt(2*np.pi)) W = special.lambertw(L/np.exp(1)) return L/W + 0.5 def mean(X, axis=None, keepdims=False): """ Compute the mean, ignoring NaNs. """ if np.ndim(X) == 0: if axis is not None: raise ValueError("Axis out of bounds") return X X = np.asanyarray(X) nans = np.isnan(X) X = X.copy() X[nans] = 0 m = (np.sum(X, axis=axis, keepdims=keepdims) / np.sum(~nans, axis=axis, keepdims=keepdims)) return m bayespy-0.4.0/bayespy/testing.py0000644000175000017500000000360012615643655020424 0ustar jluttinejluttine00000000000000# Copyright (c) 2015 Bernhard Thiel, Jaakko Luttinen # MIT License # From: https://github.com/Bernhard10/WarnAsError/blob/master/warnaserror.py __author__ = 'Bernhard Thiel' from nose.plugins import Plugin import nose import warnings class WarnAsError(Plugin): enabled = False def options(self, parser, env): """ Add options to command line. """ super().options(parser, env) parser.add_option("--warn-as-error", action="store_true", default=False, dest="enabled", help="Treat warnings that occur WITHIN tests as errors.") def configure(self, options, conf): """ Configure plugin. """ super().configure(options, conf) if options.enabled: self.enabled = True def prepareTestRunner(self, runner): """ Treat warnings as errors. """ return WarnAsErrorTestRunner(runner) class WarnAsErrorTestRunner(object): def __init__(self, runner): self.runner=runner def run(self, test): with warnings.catch_warnings(): warnings.simplefilter("error") # Filter out some deprecation warnings inside nose 1.3.7 when run # on python 3.5b2. See # https://github.com/nose-devs/nose/issues/929 warnings.filterwarnings( "ignore", message=".*getargspec.*", category=DeprecationWarning, module="nose|scipy" ) # Filter out some deprecation warnings inside matplotlib on Python # 3.4 warnings.filterwarnings( "ignore", message=".*elementwise.*", category=DeprecationWarning, module="matplotlib" ) return self.runner.run(test) bayespy-0.4.0/bayespy/discrete_example.py0000644000175000017500000000356612615643655022277 0ustar jluttinejluttine00000000000000# This example could be simplified a little bit by using Bernoulli instead of # Categorical, but Categorical makes it possible to use more categories than # just TRUE and FALSE. import numpy as np from bayespy.nodes import Categorical, Mixture from bayespy.inference import VB # NOTE: Python's built-in booleans don't work nicely for indexing, thus define # own variables: FALSE = 0 TRUE = 1 def _or(p_false, p_true): """ Build probability table for OR-operation of two parents p_false: Probability table to use if both are FALSE p_true: Probability table to use if one or both is TRUE """ return np.take([p_false, p_true], [[FALSE, TRUE], [TRUE, TRUE]], axis=0) asia = Categorical([0.5, 0.5]) tuberculosis = Mixture(asia, Categorical, [[0.99, 0.01], [0.8, 0.2]]) smoking = Categorical([0.5, 0.5]) lung = Mixture(smoking, Categorical, [[0.98, 0.02], [0.25, 0.75]]) bronchitis = Mixture(smoking, Categorical, [[0.97, 0.03], [0.08, 0.92]]) xray = Mixture(tuberculosis, Mixture, lung, Categorical, _or([0.96, 0.04], [0.115, 0.885])) dyspnea = Mixture(bronchitis, Mixture, tuberculosis, Mixture, lung, Categorical, [_or([0.6, 0.4], [0.18, 0.82]), _or([0.11, 0.89], [0.04, 0.96])]) # Mark observations tuberculosis.observe(TRUE) smoking.observe(FALSE) bronchitis.observe(TRUE) # not a "chance" observation as in the original example # Run inference Q = VB(dyspnea, xray, bronchitis, lung, smoking, tuberculosis, asia) Q.update(repeat=100) # Show results print("P(asia):", asia.get_moments()[0][TRUE]) print("P(tuberculosis):", tuberculosis.get_moments()[0][TRUE]) print("P(smoking):", smoking.get_moments()[0][TRUE]) print("P(lung):", lung.get_moments()[0][TRUE]) print("P(bronchitis):", bronchitis.get_moments()[0][TRUE]) print("P(xray):", xray.get_moments()[0][TRUE]) print("P(dyspnea):", dyspnea.get_moments()[0][TRUE]) bayespy-0.4.0/bayespy/__init__.py0000644000175000017500000000056412615643655020514 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2013 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ from .version import __version__ from . import utils from . import inference from . import nodes from . import plot bayespy-0.4.0/bayespy/nodes/0000755000175000017500000000000012615647103017476 5ustar jluttinejluttine00000000000000bayespy-0.4.0/bayespy/nodes/__init__.py0000644000175000017500000000310312533543154021604 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Package for nodes used to construct the model. Stochastic nodes ================ .. currentmodule:: bayespy.nodes Nodes for Gaussian variables: .. autosummary:: :toctree: generated/ Gaussian GaussianARD Nodes for precision and scale variables: .. autosummary:: :toctree: generated/ Gamma Wishart Exponential Nodes for modelling Gaussian and precision variables jointly (useful as prior for Gaussian nodes): .. autosummary:: :toctree: generated/ GaussianGammaISO GaussianGammaARD GaussianWishart Nodes for discrete count variables: .. autosummary:: :toctree: generated/ Bernoulli Binomial Categorical Multinomial Poisson Nodes for probabilities: .. autosummary:: :toctree: generated/ Beta Dirichlet Nodes for dynamic variables: .. autosummary:: :toctree: generated/ CategoricalMarkovChain GaussianMarkovChain SwitchingGaussianMarkovChain VaryingGaussianMarkovChain Other stochastic nodes: .. autosummary:: :toctree: generated/ Mixture Deterministic nodes =================== .. autosummary:: :toctree: generated/ Dot SumMultiply Gate """ # Currently, model construction and the inference network are not separated so # the model is constructed using variational message passing nodes. from bayespy.inference.vmp.nodes import * bayespy-0.4.0/bayespy/plot.py0000644000175000017500000010552512615643655017736 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2013 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Functions for plotting nodes. Functions ========= .. currentmodule:: bayespy.plot .. autosummary:: :toctree: generated/ pdf contour plot hinton gaussian_mixture_2d Plotters ======== .. autosummary:: :toctree: generated/ Plotter PDFPlotter ContourPlotter HintonPlotter FunctionPlotter GaussianTimeseriesPlotter CategoricalMarkovChainPlotter """ import os, sys ############################################################################ # A STUPID WORKAROUND FOR A MATPLOTLIB 1.4.0 BUG RELATED TO INTERACTIVE MODE # See: https://github.com/matplotlib/matplotlib/issues/3505 import __main__ if hasattr(__main__, '__file__'): sys.ps1 = ('WORKAROUND FOR A BUG #3505 IN MATPLOTLIB.\n' 'IF YOU SEE THIS MESSAGE, TRY MATPLOTLIB!=1.4.0.') # This workaround does not work on Python shell, only on stand-alone scripts # and IPython. A better solution: require MPL!=1.4.0. ############################################################################# import numpy as np import scipy.sparse as sp import scipy from scipy import special import matplotlib.pyplot as plt from matplotlib import animation #from matplotlib.pyplot import * from bayespy.inference.vmp.nodes.categorical import CategoricalMoments from bayespy.inference.vmp.nodes.gaussian import (GaussianMoments, GaussianWishartMoments) from bayespy.inference.vmp.nodes.beta import BetaMoments from bayespy.inference.vmp.nodes.beta import DirichletMoments from bayespy.inference.vmp.nodes.bernoulli import BernoulliMoments from bayespy.inference.vmp.nodes.categorical import CategoricalMoments from bayespy.inference.vmp.nodes.gamma import GammaMoments from bayespy.inference.vmp.nodes.node import Node, Moments from bayespy.utils import (misc, random, linalg) # Users can use pyplot via this module import matplotlib mpl = matplotlib pyplot = plt def interactive(function): """A decorator for forcing functions to use the interactive mode. Parameters ---------- function : callable The function to be decorated """ def new_function(*args, **kwargs): if mpl.is_interactive(): was_interactive = True else: was_interactive = False mpl.interactive(True) retval = function(*args, **kwargs) if not was_interactive: mpl.interactive(False) return retval return new_function def _subplots(plotfunc, *args, fig=None, kwargs=None): """Create a collection of subplots Each subplot is created with the same plotting function. Inputs are given as pairs: (x, 3), (y, 2), ... where x,y,... are the input arrays and 3,2,... are the ndim parameters. The last ndim axes of each array are interpreted as a single element to the plotting function. All high-level plotting functions should wrap low-level plotting functions with this function in order to generate subplots for plates. """ if kwargs is None: kwargs = {} if fig is None: fig = plt.gcf() # Parse shape and plates of each input array shapes = [np.shape(x)[-n:] if n > 0 else () for (x,n) in args] plates = [np.shape(x)[:-n] if n > 0 else np.shape(x) for (x,n) in args] # Get the full grid shape of the subplots broadcasted_plates = misc.broadcasted_shape(*plates) # Subplot indexing layout M = np.prod(broadcasted_plates[-2::-2]) N = np.prod(broadcasted_plates[-1::-2]) strides_subplot = [np.prod(broadcasted_plates[(j+2)::2]) * N if ((len(broadcasted_plates)-j) % 2) == 0 else np.prod(broadcasted_plates[(j+2)::2]) for j in range(len(broadcasted_plates))] # Plot each subplot for ind in misc.nested_iterator(broadcasted_plates): # Get the list of inputs for this subplot broadcasted_args = [] for n in range(len(args)): i = misc.safe_indices(ind, plates[n]) broadcasted_args.append(args[n][0][i]) # Plot the subplot using the given function ind_subplot = np.einsum('i,i', ind, strides_subplot) axes = fig.add_subplot(M, N, ind_subplot+1) plotfunc(*broadcasted_args, axes=axes, **kwargs) def pdf(Z, x, *args, name=None, axes=None, fig=None, **kwargs): """ Plot probability density function of a scalar variable. Parameters ---------- Z : node or function Stochastic node or log pdf function x : array Grid points """ # TODO: Make it possible to plot a plated variable using _subplots function. if axes is None and fig is None: axes = plt.gca() else: if fig is None: fig = plt.gcf() axes = fig.add_subplot(111) try: lpdf = Z.logpdf(x) except AttributeError: lpdf = Z(x) p = np.exp(lpdf) retval = axes.plot(x, p, *args, **kwargs) if name is None: try: name = Z.name except AttributeError: pass if name: axes.set_title(r'$q(%s)$' % (name)) axes.set_xlabel(r'$%s$' % (name)) return retval def contour(Z, x, y, n=None, axes=None, fig=None, **kwargs): """ Plot 2-D probability density function of a 2-D variable. Parameters ---------- Z : node or function Stochastic node or log pdf function x : array Grid points on x axis y : array Grid points on y axis """ # TODO: Make it possible to plot a plated variable using _subplots function. if axes is None and fig is None: axes = plt.gca() else: if fig is None: fig = plt.gcf() axes = fig.add_subplot(111) XY = misc.grid(x, y) try: lpdf = Z.logpdf(XY) except AttributeError: lpdf = Z(XY) p = np.exp(lpdf) shape = (np.size(x), np.size(y)) X = np.reshape(XY[:,0], shape) Y = np.reshape(XY[:,1], shape) P = np.reshape(p, shape) if n is not None: levels = np.linspace(0, np.amax(P), num=n+2)[1:-1] return axes.contour(X, Y, P, levels, **kwargs) else: return axes.contour(X, Y, P, **kwargs) def plot_gaussian_mc(X, scale=2, **kwargs): """ Plot Gaussian Markov chain as a 1-D function Parameters ---------- X : node Node with Gaussian Markov chain moments. """ timeseries_gaussian(X, axis=-2, scale=scale, **kwargs) def plot_bernoulli(X, axis=-1, scale=2, **kwargs): """ Plot Bernoulli node as a 1-D function """ X = X._convert(BernoulliMoments) u_X = X.get_moments() z = u_X[0] return _timeseries_mean_and_error(z, None, axis=axis, **kwargs) def plot_gaussian(X, axis=-1, scale=2, **kwargs): """ Plot Gaussian node as a 1-D function Parameters ---------- X : node Node with Gaussian moments. axis : int The index of the time axis. """ X = X._convert(GaussianMoments) u_X = X.get_moments() x = u_X[0] xx = misc.get_diag(u_X[1], ndim=len(X.dims[0])) std = scale * np.sqrt(xx - x**2) #std = scale * np.sqrt(np.einsum('...ii->...i', xx) - x**2) return _timeseries_mean_and_error(x, std, axis=axis, **kwargs) def plot(Y, axis=-1, scale=2, center=False, **kwargs): """ Plot a variable or an array as 1-D function with errorbars """ if misc.is_numeric(Y): return _timeseries_mean_and_error(Y, None, axis=axis, center=center, **kwargs) if isinstance(Y, Node): # Try Bernoulli plotting try: Y = Y._convert(BernoulliMoments) except BernoulliMoments.NoConverterError: pass else: return plot_bernoulli(Y, axis=axis, scale=scale, center=center, **kwargs) # Try Gaussian plotting try: Y = Y._convert(GaussianMoments) except GaussianMoments.NoConverterError: pass else: return plot_gaussian(Y, axis=axis, scale=scale, center=center, **kwargs) (mu, var) = Y.get_mean_and_variance() std = np.sqrt(var) return _timeseries_mean_and_error(mu, std, axis=axis, scale=scale, center=center, **kwargs) # Some backward compatibility def timeseries_gaussian_mc(*args, center=True, **kwargs): return plot_gaussian_mc(*args, center=center, **kwargs) def timeseries_gaussian(*args, center=True, **kwargs): return plot_gaussian(*args, center=center, **kwargs) timeseries_normal = timeseries_gaussian def timeseries(*args, center=True, **kwargs): return plot(*args, center=center, **kwargs) def _timeseries_mean_and_error(y, std, *args, axis=-1, center=True, fig=None, axes=None, **kwargs): # TODO/FIXME: You must multiply by ones(plates) in order to plot # broadcasted plates properly if fig is None: fig = plt.gcf() y = np.atleast_1d(y) shape = list(np.shape(y)) # Get and remove the length of the time axis T = shape.pop(axis) # Move time axis to first y = np.rollaxis(y, axis) if std is not None: std = np.rollaxis(std, axis) y = np.reshape(y, (T, -1)) if std is not None: std = np.reshape(std, (T, -1)) # Remove 1s shape = [s for s in shape if s > 1] # Calculate number of rows and columns shape = misc.multiply_shapes(shape, (1,1)) if len(shape) > 2: raise Exception("Can plot only in 2 dimensions (rows and columns)") (M, N) = shape # Prefer plotting to rows if M == 1: M = N N = 1 # Plot each timeseries if axes is None: ax0 = fig.add_subplot(M, N, 1) for i in range(M*N): if axes is None: if i > 0: # Share x axis between all subplots ax = fig.add_subplot(M, N, i+1, sharex=ax0) else: ax = ax0 # Autoscale the axes to data and use tight y and x axes ax.autoscale(enable=True, tight=True) ax.set_ylim(auto=True) if i < (M-1)*N: # Remove x tick labels from other than the last row plt.setp(ax.get_xticklabels(), visible=False) else: ax = axes[i] if std is None: errorplot(y=y[:,i], axes=ax, **kwargs) else: if len(args) > 0: raise Exception("Can't handle extra arguments") errorplot(y=y[:,i], error=std[:,i], axes=ax, **kwargs) if center: # Center the zero level on y-axis ylim = ax.get_ylim() vmax = np.max(np.abs(ylim)) ax.set_ylim([-vmax, vmax]) if axes is None: # Remove height space between subplots fig.subplots_adjust(hspace=0) def _blob(axes, x, y, area, colour): """ Draws a square-shaped blob with the given area (< 1) at the given coordinates. """ hs = np.sqrt(area) / 2 xcorners = np.array([x - hs, x + hs, x + hs, x - hs]) ycorners = np.array([y - hs, y - hs, y + hs, y + hs]) axes.fill(xcorners, ycorners, colour, edgecolor=colour) def _rectangle(axes, x, y, width, height, **kwargs): _x = x - width/2 _y = y - height/2 rectangle = plt.Rectangle((_x, _y), width, height, **kwargs) axes.add_patch(rectangle) return def gaussian_mixture_2d(X, alpha=None, scale=2, fill=False, axes=None, **kwargs): """ Plot Gaussian mixture as ellipses in 2-D Parameters ---------- X : Mixture node alpha : Dirichlet-like node (optional) Probabilities for the clusters scale : float (optional) Scale for the covariance ellipses (by default, 2) """ if axes is None: axes = plt.gca() mu_Lambda = X.parents[1]._convert(GaussianWishartMoments) (mu, _, Lambda, _) = mu_Lambda.get_moments() mu = np.linalg.solve(Lambda, mu) if len(mu_Lambda.plates) != 1: raise NotImplementedError("Not yet implemented for more plates") K = mu_Lambda.plates[0] width = np.zeros(K) height = np.zeros(K) angle = np.zeros(K) for k in range(K): m = mu[k] L = Lambda[k] (u, W) = scipy.linalg.eigh(L) u[0] = np.sqrt(1/u[0]) u[1] = np.sqrt(1/u[1]) width[k] = 2*u[0] height[k] = 2*u[1] angle[k] = np.arctan(W[0,1] / W[0,0]) angle = 180 * angle / np.pi mode_height = 1 / (width * height) # Use cluster probabilities to adjust alpha channel if alpha is not None: # Compute the normalized probabilities in a numerically stable way logsum_p = misc.logsumexp(alpha.u[0], axis=-1, keepdims=True) logp = alpha.u[0] - logsum_p p = np.exp(logp) # Visibility is based on cluster mode peak height visibility = mode_height * p visibility /= np.amax(visibility) else: visibility = np.ones(K) for k in range(K): ell = mpl.patches.Ellipse(mu[k], scale*width[k], scale*height[k], angle=(180+angle[k]), fill=fill, alpha=visibility[k], **kwargs) axes.add_artist(ell) plt.axis('equal') # If observed, plot the data too if np.any(X.observed): mask = np.array(X.observed) * np.ones(X.plates, dtype=np.bool) y = X.u[0][mask] plt.plot(y[:,0], y[:,1], 'r.') return def _hinton(W, error=None, vmax=None, square=True, axes=None): """ Draws a Hinton diagram for visualizing a weight matrix. Temporarily disables matplotlib interactive mode if it is on, otherwise this takes forever. Originally copied from http://wiki.scipy.org/Cookbook/Matplotlib/HintonDiagrams """ if axes is None: axes = plt.gca() W = misc.atleast_nd(W, 2) (height, width) = W.shape if not vmax: #vmax = 2**np.ceil(np.log(np.max(np.abs(W)))/np.log(2)) if error is not None: vmax = np.max(np.abs(W) + error) else: vmax = np.max(np.abs(W)) axes.fill(0.5+np.array([0,width,width,0]), 0.5+np.array([0,0,height,height]), 'gray') axes.axis('off') if square: axes.axis('equal') axes.invert_yaxis() for x in range(width): for y in range(height): _x = x+1 _y = y+1 w = W[y,x] _w = np.abs(w) if w > 0: _c = 'white' else: _c = 'black' if error is not None: e = error[y,x] if e < 0: print(e, _w, vmax) raise Exception("BUG? Negative error") if _w + e > vmax: print(e, _w, vmax) raise Exception("BUG? Value+error greater than max") _rectangle(axes, _x, _y, min(1, np.sqrt((_w+e)/vmax)), min(1, np.sqrt((_w+e)/vmax)), edgecolor=_c, fill=False) _blob(axes, _x, _y, min(1, _w/vmax), _c) def matrix(A, axes=None): if axes is None: axes = plt.gca() A = np.atleast_2d(A) vmax = np.max(np.abs(A)) return axes.imshow(A, interpolation='nearest', cmap='RdBu_r', vmin=-vmax, vmax=vmax) def new_matrix(A, vmax=None): A = np.atleast_2d(A) if vmax is None: vmax = np.max(np.abs(A)) (M, N) = np.shape(A) for i in range(M): for j in range(N): pass def gaussian_hinton(X, rows=None, cols=None, scale=1, fig=None): """ Plot the Hinton diagram of a Gaussian node """ if fig is None: fig = plt.gcf() # Get mean and second moment X = X._convert(GaussianMoments) (x, xx) = X.get_moments() ndim = len(X.dims[0]) shape = X.get_shape(0) size = len(X.get_shape(0)) # Compute standard deviation xx = misc.get_diag(xx, ndim=ndim) std = np.sqrt(xx - x**2) # Force explicit elements when broadcasting x = x * np.ones(shape) std = std * np.ones(shape) if rows is None: rows = np.nan if cols is None: cols = np.nan # Preprocess the axes to 0,...,ndim if rows < 0: rows += size if cols < 0: cols += size if rows < 0 or rows >= size: raise ValueError("Row axis invalid") if cols < 0 or cols >= size: raise ValueError("Column axis invalid") # Remove non-row and non-column axes that have length 1 squeezed_shape = list(shape) for i in reversed(range(len(shape))): if shape[i] == 1 and i != rows and i != cols: squeezed_shape.pop(i) if i < cols: cols -= 1 if i < rows: rows -= 1 x = np.reshape(x, squeezed_shape) std = np.reshape(std, squeezed_shape) if np.ndim(x) < 2: cols += 2 - np.ndim(x) rows += 2 - np.ndim(x) x = np.atleast_2d(x) std = np.atleast_2d(std) size = np.ndim(x) if np.isnan(cols): if rows != size - 1: cols = size - 1 else: cols = size - 2 if np.isnan(rows): if cols != size - 1: rows = size - 1 else: rows = size - 2 # Put the row and column axes to the end axes = [i for i in range(size) if i not in (rows, cols)] + [rows, cols] x = np.transpose(x, axes=axes) std = np.transpose(std, axes=axes) vmax = np.max(np.abs(x) + scale*std) if scale == 0: _subplots(_hinton, (x, 2), fig=fig, kwargs=dict(vmax=vmax)) else: def plotfunc(z, e, **kwargs): return _hinton(z, error=e, **kwargs) _subplots(plotfunc, (x, 2), (scale*std, 2), fig=fig, kwargs=dict(vmax=vmax)) def _hinton_figure(x, rows=None, cols=None, fig=None, square=True): """ Plot the Hinton diagram of a Gaussian node """ scale = 0 std = 0 if fig is None: fig = plt.gcf() # Get mean and second moment shape = np.shape(x) size = np.ndim(x) if rows is None: rows = np.nan if cols is None: cols = np.nan # Preprocess the axes to 0,...,ndim if rows < 0: rows += size if cols < 0: cols += size if rows < 0 or rows >= size: raise ValueError("Row axis invalid") if cols < 0 or cols >= size: raise ValueError("Column axis invalid") # Remove non-row and non-column axes that have length 1 squeezed_shape = list(shape) for i in reversed(range(len(shape))): if shape[i] == 1 and i != rows and i != cols: squeezed_shape.pop(i) if i < cols: cols -= 1 if i < rows: rows -= 1 x = np.reshape(x, squeezed_shape) size = np.ndim(x) if np.isnan(cols): if rows != size - 1: cols = size - 1 else: cols = size - 2 if np.isnan(rows): if cols != size - 1: rows = size - 1 else: rows = size - 2 # Put the row and column axes to the end if np.ndim(x) >= 2: axes = [i for i in range(size) if i not in (rows, cols)] + [rows, cols] x = np.transpose(x, axes=axes) #std = np.transpose(std, axes=axes) vmax = np.max(np.abs(x) + scale*std) kw = dict(vmax=vmax, square=square) if scale == 0: _subplots(_hinton, (x, 2), fig=fig, kwargs=kw) else: def plotfunc(z, e, **kwargs): return _hinton(z, error=e, **kwargs) _subplots(plotfunc, (x, 2), (scale*std, 2), fig=fig, kwargs=kw) # For backwards compatibility: gaussian_array = gaussian_hinton def timeseries_categorical_mc(Z, fig=None): if fig is None: fig = plt.gcf() # Make sure that the node is categorical Z = Z._convert(CategoricalMoments) # Get expectations (and broadcast explicitly) z = Z._message_to_child()[0] * np.ones(Z.get_shape(0)) # Compute the subplot layout z = misc.atleast_nd(z, 4) if np.ndim(z) != 4: raise ValueError("Can not plot arrays with over 4 axes") M = np.shape(z)[0] N = np.shape(z)[1] # Plot Hintons for i in range(M): for j in range(N): axes = fig.add_subplot(M, N, i*N+j+1) _hinton(z[i,j].T, vmax=1.0, square=False, axes=axes) def gamma_hinton(alpha, square=True, **kwargs): """ Plot a beta distributed random variable as a Hinton diagram """ # Make sure that the node is beta alpha = alpha._convert(GammaMoments) # Compute exp( ) x = alpha.get_moments()[0] # Explicit broadcasting x = x * np.ones(alpha.plates) # Plot Hinton diagram return _hinton_figure(x, square=square, **kwargs) def beta_hinton(P, square=True): """ Plot a beta distributed random variable as a Hinton diagram """ # Make sure that the node is beta P = P._convert(BetaMoments) # Compute exp( ) p = np.exp(P._message_to_child()[0][...,0]) # Explicit broadcasting p = p * np.ones(P.plates) # Plot Hinton diagram return _hinton(p, vmax=1.0, square=square) def dirichlet_hinton(P, square=True): """ Plot a beta distributed random variable as a Hinton diagram """ # Make sure that the node is beta P = P._convert(DirichletMoments) # Compute exp( ) p = np.exp(P._message_to_child()[0]) # Explicit broadcasting p = p * np.ones(P.plates+(1,)) # Plot Hinton diagram return _hinton(p, vmax=1.0, square=square) def bernoulli_hinton(Z, square=True): """ Plot a Bernoulli distributed random variable as a Hinton diagram """ # Make sure that the node is Bernoulli Z = Z._convert(BernoulliMoments) # Get z = Z._message_to_child()[0] # Explicit broadcasting z = z * np.ones(Z.plates) # Plot Hinton diagram return _hinton(z, vmax=1.0, square=square) def categorical_hinton(Z, square=True): """ Plot a Bernoulli distributed random variable as a Hinton diagram """ # Make sure that the node is Bernoulli Z = Z._convert(CategoricalMoments) # Get z = Z._message_to_child()[0] # Explicit broadcasting z = z * np.ones(Z.plates+(1,)) # Plot Hinton diagram return _hinton(np.squeeze(z), vmax=1.0, square=square) def hinton(X, **kwargs): r""" Plot the Hinton diagram of a node The keyword arguments depend on the node type. For some node types, the diagram also shows uncertainty with non-filled rectangles. Currently, beta-like, Gaussian-like and Dirichlet-like nodes are supported. Parameters ---------- X : node """ if hasattr(X, "_convert"): try: X = X._convert(GaussianMoments) except Moments.NoConverterError: pass else: return gaussian_hinton(X, **kwargs) try: X = X._convert(GammaMoments) except Moments.NoConverterError: pass else: return gamma_hinton(X, **kwargs) try: X = X._convert(BetaMoments) except Moments.NoConverterError: pass else: return beta_hinton(X, **kwargs) try: X = X._convert(DirichletMoments) except Moments.NoConverterError: pass else: return dirichlet_hinton(X, **kwargs) try: X = X._convert(BernoulliMoments) except Moments.NoConverterError: pass else: return bernoulli_hinton(X, **kwargs) try: X = X._convert(CategoricalMoments) except Moments.NoConverterError: pass else: return categorical_hinton(X, **kwargs) return _hinton_figure(X, **kwargs) class Plotter(): r""" Wrapper for plotting functions and base class for node plotters The purpose of this class is to collect all the parameters needed by a plotting function and provide a callable interface which needs only the node as the input. Plotter instances are callable objects that plot a given node using a specified plotting function. Parameters ---------- plotter : function Plotting function to use args : defined by the plotting function Additional inputs needed by the plotting function kwargs : defined by the plotting function Additional keyword arguments supported by the plotting function Examples -------- First, create a gamma variable: >>> import numpy as np >>> from bayespy.nodes import Gamma >>> x = Gamma(4, 5) The probability density function can be plotted as: >>> import bayespy.plot as bpplt >>> bpplt.pdf(x, np.linspace(0.1, 10, num=100)) # doctest: +ELLIPSIS [] However, this can be problematic when one needs to provide a plotting function for the inference engine as the inference engine gives only the node as input. Thus, we need to create a simple plotter wrapper: >>> p = bpplt.Plotter(bpplt.pdf, np.linspace(0.1, 10, num=100)) Now, this callable object ``p`` needs only the node as the input: >>> p(x) # doctest: +ELLIPSIS [] Thus, it can be given to the inference engine to use as a plotting function: >>> x = Gamma(4, 5, plotter=p) >>> x.plot() # doctest: +ELLIPSIS [] """ def __init__(self, plotter, *args, **kwargs): self._args = args self._kwargs = kwargs self._plotter = plotter def __call__(self, X, fig=None, **kwargs): """ Plot the node using the specified plotting function Parameters ---------- X : node The plotted node """ kwargs_all = self._kwargs.copy() kwargs_all.update(kwargs) return self._plotter(X, *self._args, fig=fig, **kwargs_all) class PDFPlotter(Plotter): r""" Plotter of probability density function of a scalar node Parameters ---------- x_grid : array Numerical grid on which the density function is computed and plotted See also -------- pdf """ def __init__(self, x_grid, **kwargs): super().__init__(pdf, x_grid, **kwargs) class ContourPlotter(Plotter): r""" Plotter of probability density function of a two-dimensional node Parameters ---------- x1_grid : array Grid for the first dimension x2_grid : array Grid for the second dimension See also -------- contour """ def __init__(self, x1_grid, x2_grid, **kwargs): super().__init__(contour, x1_grid, x2_grid, **kwargs) class HintonPlotter(Plotter): r""" Plotter of the Hinton diagram of a node See also -------- hinton """ def __init__(self, **kwargs): super().__init__(hinton, **kwargs) class FunctionPlotter(Plotter): r""" Plotter of a node as a 1-dimensional function See also -------- plot """ def __init__(self, **kwargs): super().__init__(plot, **kwargs) class GaussianMarkovChainPlotter(Plotter): r""" Plotter of a Gaussian Markov chain as a timeseries """ def __init__(self, **kwargs): super().__init__(timeseries_gaussian_mc, **kwargs) class GaussianTimeseriesPlotter(Plotter): r""" Plotter of a Gaussian node as a timeseries """ def __init__(self, **kwargs): super().__init__(timeseries_gaussian, **kwargs) class GaussianHintonPlotter(Plotter): r""" Plotter of a Gaussian node as a Hinton diagram """ def __init__(self, **kwargs): super().__init__(gaussian_array, **kwargs) class CategoricalMarkovChainPlotter(Plotter): r""" Plotter of a Categorical timeseries """ def __init__(self, **kwargs): super().__init__(timeseries_categorical_mc, **kwargs) def matrix_animation(A, filename=None, fps=25, fig=None, **kwargs): if fig is None: fig = plt.gcf() axes = fig.add_subplot(111) A = np.atleast_3d(A) vmax = np.max(np.abs(A)) x = axes.imshow(A[0], interpolation='nearest', cmap='RdBu_r', vmin=-vmax, vmax=vmax, **kwargs) s = axes.set_title('t = %d' % 0) def animate(nframe): s.set_text('t = %d' % nframe) x.set_array(A[nframe]) return (x, s) anim = animation.FuncAnimation(fig, animate, frames=np.shape(A)[0], interval=1000/fps, blit=False, repeat=False) return anim def save_animation(anim, filename, fps=25, bitrate=5000, fig=None): # A bug in numpy/matplotlib causes this not to work in python3.3: # https://github.com/matplotlib/matplotlib/issues/1891 # # So the following command does not work currently.. # # anim.save(filename, fps=fps) if fig is None: fig = plt.gcf() writer = animation.FFMpegFileWriter(fps=fps, bitrate=bitrate) writer.setup(fig, filename, 100) anim.save(filename, fps=fps, writer=writer, bitrate=bitrate) return def binary_matrix(A, axes=None): if axes is None: axes = plt.gca() A = np.atleast_2d(A) G = np.zeros(np.shape(A) + (3,)) G[A] = [0,0,0] G[np.logical_not(A)] = [1,1,1] axes.imshow(G, interpolation='nearest') def gaussian_mixture_logpdf(x, w, mu, Sigma): # Shape(x) = (N, D) # Shape(w) = (K,) # Shape(mu) = (K, D) # Shape(Sigma) = (K, D, D) # Shape(result) = (N,) # Dimensionality D = np.shape(x)[-1] # Cholesky decomposition of the covariance matrix U = linalg.chol(Sigma) # Reshape x: # Shape(x) = (N, 1, D) x = np.expand_dims(x, axis=-2) # (x-mu) and (x-mu)'*inv(Sigma)*(x-mu): # Shape(v) = (N, K, D) # Shape(z) = (N, K) v = x - mu z = np.einsum('...i,...i', v, linalg.chol_solve(U, v)) # Log-determinant of Sigma: # Shape(ldet) = (K,) ldet = linalg.chol_logdet(U) # Compute log pdf for each cluster: # Shape(lpdf) = (N, K) lpdf = misc.gaussian_logpdf(z, 0, 0, ldet, D) def matrixplot(A, colorbar=False, axes=None): if axes is None: axes = plt.gca() if sp.issparse(A): A = A.toarray() axes.imshow(A, interpolation='nearest') if colorbar: plt.colorbar(ax=axes) def contourplot(x1, x2, y, colorbar=False, filled=True, axes=None): """ Plots 2D contour plot. x1 and x2 are 1D vectors, y contains the function values. y.size must be x1.size*x2.size. """ if axes is None: axes = plt.gca() y = np.reshape(y, (len(x2),len(x1))) if filled: axes.contourf(x1, x2, y) else: axes.contour(x1, x2, y) if colorbar: plt.colorbar(ax=axes) def errorplot(y=None, error=None, x=None, lower=None, upper=None, color=(0,0,0,1), fillcolor=(0,0,0,0.4), axes=None, **kwargs): if axes is None: axes = plt.gca() # Default inputs if x is None: x = np.arange(np.size(y)) # Parse errors (lower=lower/error/upper, upper=upper/error/lower) if lower is None: if error is not None: lower = error elif upper is not None: lower = upper if upper is None: if error is not None: upper = error elif lower is not None: upper = lower # Plot errors if (lower is not None) and (upper is not None): l = y - lower u = y + upper axes.fill_between(x, l, u, facecolor=fillcolor, edgecolor=(0, 0, 0, 0), linewidth=1, interpolate=True) # Plot function axes.plot(x, y, color=color, **kwargs) def plotmatrix(X): """ Creates a matrix of marginal plots. On diagonal, are marginal plots of each variable. Off-diagonal plot (i,j) shows the joint marginal density of x_i and x_j. """ return X.plotmatrix() def _pdf_t(mu, s2, nu, axes=None, scale=4, color='k'): """ """ if axes is None: axes = plt.gca() s = np.sqrt(s2) x = np.linspace(mu-scale*s, mu+scale*s, num=100) y2 = (x-mu)**2 / s2 lpdf = random.t_logpdf(y2, np.log(s2), nu, 1) p = np.exp(lpdf) return axes.plot(x, p, color=color) def _pdf_gamma(a, b, axes=None, scale=4, color='k'): """ """ if axes is None: axes = plt.gca() if np.size(a) != 1 or np.size(b) != 1: raise ValueError("Parameters must be scalars") mean = a/b v = scale*np.sqrt(a/b**2) m = max(0, mean-v) n = mean + v x = np.linspace(m, n, num=100) logx = np.log(x) lpdf = random.gamma_logpdf(b*x, logx, a*logx, a*np.log(b), special.gammaln(a)) p = np.exp(lpdf) return axes.plot(x, p, color=color) def _contour_t(mu, Cov, nu, axes=None, scale=4, transpose=False, colors='k'): """ """ if axes is None: axes = plt.gca() if np.shape(mu) != (2,) or np.shape(Cov) != (2,2) or np.shape(nu) != (): print(np.shape(mu), np.shape(Cov), np.shape(nu)) raise ValueError("Only 2-d t-distribution allowed") if transpose: mu = mu[[1,0]] Cov = Cov[np.ix_([1,0],[1,0])] s = np.sqrt(np.diag(Cov)) x0 = np.linspace(mu[0]-scale*s[0], mu[0]+scale*s[0], num=100) x1 = np.linspace(mu[1]-scale*s[1], mu[1]+scale*s[1], num=100) X0X1 = misc.grid(x0, x1) Y = X0X1 - mu L = linalg.chol(Cov) logdet_Cov = linalg.chol_logdet(L) Z = linalg.chol_solve(L, Y) Z = linalg.inner(Y, Z, ndim=1) lpdf = random.t_logpdf(Z, logdet_Cov, nu, 2) p = np.exp(lpdf) shape = (np.size(x0), np.size(x1)) X0 = np.reshape(X0X1[:,0], shape) X1 = np.reshape(X0X1[:,1], shape) P = np.reshape(p, shape) return axes.contour(X0, X1, P, colors=colors) def _contour_gaussian_gamma(mu, s2, a, b, axes=None, transpose=False): """ """ pass bayespy-0.4.0/bayespy/version.py0000644000175000017500000000002612615643674020434 0ustar jluttinejluttine00000000000000__version__ = '0.4.0' bayespy-0.4.0/bayespy/demos/0000755000175000017500000000000012615647103017475 5ustar jluttinejluttine00000000000000bayespy-0.4.0/bayespy/demos/black_box.py0000644000175000017500000000654512533543154022005 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2015 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Black-box variational inference """ import numpy as np import scipy import matplotlib.pyplot as plt import bayespy.plot as myplt from bayespy.utils import misc from bayespy.utils import random from bayespy.nodes import GaussianARD, LogPDF, Dot from bayespy.inference.vmp.vmp import VB from bayespy.inference.vmp import transformations import bayespy.plot as bpplt from bayespy.demos import pca def run(M=10, N=100, D=5, seed=42, maxiter=100, plot=True): """ Run deterministic annealing demo for 1-D Gaussian mixture. """ raise NotImplementedError("Black box variational inference not yet implemented, sorry") if seed is not None: np.random.seed(seed) # Generate data data = np.dot(np.random.randn(M,D), np.random.randn(D,N)) # Construct model C = GaussianARD(0, 1, shape=(2,), plates=(M,1), name='C') X = GaussianARD(0, 1, shape=(2,), plates=(1,N), name='X') F = Dot(C, X) # Some arbitrary log likelihood def logpdf(y, f): """ exp(f) / (1 + exp(f)) = 1/(1+exp(-f)) -log(1+exp(-f)) = -log(exp(0)+exp(-f)) also: 1 - exp(f) / (1 + exp(f)) = (1 + exp(f) - exp(f)) / (1 + exp(f)) = 1 / (1 + exp(f)) = -log(1+exp(f)) = -log(exp(0)+exp(f)) """ return -np.logaddexp(0, -f * np.where(y, -1, +1)) Y = LogPDF(logpdf, F, samples=10, shape=()) #Y = GaussianARD(F, 1) Y.observe(data) Q = VB(Y, C, X) Q.ignore_bound_checks = True delay = 1 forgetting_rate = 0.7 for n in range(maxiter): # Observe a mini-batch #subset = np.random.choice(N, N_batch) #Y.observe(data[subset,:]) # Learn intermediate variables #Q.update(Z) # Set step length step = (n + delay) ** (-forgetting_rate) # Stochastic gradient for the global variables Q.gradient_step(C, X, scale=step) if plot: bpplt.pyplot.plot(np.cumsum(Q.cputime), Q.L, 'r:') bpplt.pyplot.xlabel('CPU time (in seconds)') bpplt.pyplot.ylabel('VB lower bound') return if __name__ == '__main__': import sys, getopt, os try: opts, args = getopt.getopt(sys.argv[1:], "", ["n=", "batch=", "seed=", "maxiter="]) except getopt.GetoptError: print('python stochastic_inference.py ') print('--n= Number of data points') print('--batch= Mini-batch size') print('--maxiter= Maximum number of VB iterations') print('--seed= Seed (integer) for the random number generator') sys.exit(2) kwargs = {} for opt, arg in opts: if opt == "--maxiter": kwargs["maxiter"] = int(arg) elif opt == "--seed": kwargs["seed"] = int(arg) elif opt in ("--n",): kwargs["N"] = int(arg) elif opt in ("--batch",): kwargs["N_batch"] = int(arg) run(**kwargs) plt.show() bayespy-0.4.0/bayespy/demos/pattern_search.py0000644000175000017500000000755012535540151023054 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2015 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Demonstration of the pattern search method for PCA. The pattern searches are compared to standard VB-EM algorithm in CPU time. For more info on the pattern search method, see :cite:`Honkela:2002`. """ import numpy as np import scipy import matplotlib.pyplot as plt import bayespy.plot as myplt from bayespy.utils import misc from bayespy.utils import random from bayespy import nodes from bayespy.inference.vmp.vmp import VB from bayespy.inference.vmp import transformations import bayespy.plot as bpplt from bayespy.demos import pca def run(M=40, N=100, D_y=6, D=8, seed=42, rotate=False, maxiter=1000, debug=False, plot=True): """ Run pattern search demo for PCA. """ if seed is not None: np.random.seed(seed) # Generate data w = np.random.normal(0, 1, size=(M,1,D_y)) x = np.random.normal(0, 1, size=(1,N,D_y)) f = misc.sum_product(w, x, axes_to_sum=[-1]) y = f + np.random.normal(0, 0.2, size=(M,N)) # Construct model Q = pca.model(M, N, D) # Data with missing values mask = random.mask(M, N, p=0.5) # randomly missing y[~mask] = np.nan Q['Y'].observe(y, mask=mask) # Initialize some nodes randomly Q['X'].initialize_from_random() Q['W'].initialize_from_random() # Use a few VB-EM updates at the beginning Q.update(repeat=10) Q.save() # Standard VB-EM as a baseline Q.update(repeat=maxiter) if plot: bpplt.pyplot.plot(np.cumsum(Q.cputime), Q.L, 'k-') # Restore initial state Q.load() # Pattern search method for comparison for n in range(maxiter): Q.pattern_search('W', 'tau', maxiter=3, collapsed=['X', 'alpha']) Q.update(repeat=20) if Q.has_converged(): break if plot: bpplt.pyplot.plot(np.cumsum(Q.cputime), Q.L, 'r:') bpplt.pyplot.xlabel('CPU time (in seconds)') bpplt.pyplot.ylabel('VB lower bound') bpplt.pyplot.legend(['VB-EM', 'Pattern search'], loc='lower right') if __name__ == '__main__': import sys, getopt, os try: opts, args = getopt.getopt(sys.argv[1:], "", ["m=", "n=", "d=", "k=", "seed=", "maxiter=", "debug"]) except getopt.GetoptError: print('python demo_pca.py ') print('--m= Dimensionality of data vectors') print('--n= Number of data vectors') print('--d= Dimensionality of the latent vectors in the model') print('--k= Dimensionality of the true latent vectors') print('--maxiter= Maximum number of VB iterations') print('--seed= Seed (integer) for the random number generator') print('--debug Check that the rotations are implemented correctly') sys.exit(2) kwargs = {} for opt, arg in opts: if opt == "--rotate": kwargs["rotate"] = True elif opt == "--maxiter": kwargs["maxiter"] = int(arg) elif opt == "--debug": kwargs["debug"] = True elif opt == "--seed": kwargs["seed"] = int(arg) elif opt in ("--m",): kwargs["M"] = int(arg) elif opt in ("--n",): kwargs["N"] = int(arg) elif opt in ("--d",): kwargs["D"] = int(arg) elif opt in ("--k",): kwargs["D_y"] = int(arg) run(**kwargs) plt.show() bayespy-0.4.0/bayespy/demos/lssm_sd.py0000644000175000017500000002472312535534626021531 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Demonstrate the linear state-space model with switching dynamics. The model differs from the classical linear state-space model in that it has a set of state dynamics matrices of which one is used at each time instance. A hidden Markov model is used to select the dynamics matrix. Some functions in this module are re-usable: * ``model`` can be used to construct the LSSM with switching dynamics. * ``infer`` can be used to apply the model to given data. """ import numpy as np import matplotlib.pyplot as plt from bayespy.nodes import (GaussianARD, SwitchingGaussianMarkovChain, CategoricalMarkovChain, Dirichlet, Mixture, Gamma, SumMultiply) from bayespy.inference.vmp.vmp import VB from bayespy.inference.vmp import transformations import bayespy.plot as bpplt def model(M=20, N=100, D=10, K=3): """ Construct the linear state-space model with switching dynamics. """ # # Switching dynamics (HMM) # # Prior for initial state probabilities rho = Dirichlet(1e-3*np.ones(K), name='rho') # Prior for state transition probabilities V = Dirichlet(1e-3*np.ones(K), plates=(K,), name='V') v = 10*np.identity(K) + 1*np.ones((K,K)) v /= np.sum(v, axis=-1, keepdims=True) V.initialize_from_value(v) # Hidden states (with unknown initial state probabilities and state # transition probabilities) Z = CategoricalMarkovChain(rho, V, states=N-1, name='Z', plotter=bpplt.CategoricalMarkovChainPlotter(), initialize=False) Z.u[0] = np.random.dirichlet(np.ones(K)) Z.u[1] = np.reshape(np.random.dirichlet(0.5*np.ones(K*K), size=(N-2)), (N-2, K, K)) # # Linear state-space models # # Dynamics matrix with ARD # (K,D) x () alpha = Gamma(1e-5, 1e-5, plates=(K,1,D), name='alpha') # (K,1,1,D) x (D) A = GaussianARD(0, alpha, shape=(D,), plates=(K,D), name='A', plotter=bpplt.GaussianHintonPlotter()) A.initialize_from_value(np.identity(D)*np.ones((K,D,D)) + 0.1*np.random.randn(K,D,D)) # Latent states with dynamics # (K,1) x (N,D) X = SwitchingGaussianMarkovChain(np.zeros(D), # mean of x0 1e-3*np.identity(D), # prec of x0 A, # dynamics Z, # dynamics selection np.ones(D), # innovation n=N, # time instances name='X', plotter=bpplt.GaussianMarkovChainPlotter()) X.initialize_from_value(10*np.random.randn(N,D)) # Mixing matrix from latent space to observation space using ARD # (K,1,1,D) x () gamma = Gamma(1e-5, 1e-5, plates=(D,), name='gamma') # (K,M,1) x (D) C = GaussianARD(0, gamma, shape=(D,), plates=(M,1), name='C', plotter=bpplt.GaussianHintonPlotter(rows=-3,cols=-1)) C.initialize_from_value(np.random.randn(M,1,D)) # Underlying noiseless function # (K,M,N) x () F = SumMultiply('i,i', C, X, name='F') # # Mixing the models # # Observation noise tau = Gamma(1e-5, 1e-5, name='tau') tau.initialize_from_value(1e2) # Emission/observation distribution Y = GaussianARD(F, tau, name='Y') Q = VB(Y, F, Z, rho, V, C, gamma, X, A, alpha, tau) return Q def infer(y, D, K, rotate=True, debug=False, maxiter=100, mask=True, plot_C=True, monitor=False, update_hyper=0, autosave=None): """ Apply LSSM with switching dynamics to the given data. """ (M, N) = np.shape(y) # Construct model Q = model(M=M, K=K, N=N, D=D) if not plot_C: Q['C'].set_plotter(None) if autosave is not None: Q.set_autosave(autosave, iterations=10) Q['Y'].observe(y, mask=mask) # Set up rotation speed-up if rotate: # Initial rotate the D-dimensional state space (X, A, C) # Do not update hyperparameters rotA_init = transformations.RotateGaussianARD(Q['A']) rotX_init = transformations.RotateSwitchingMarkovChain(Q['X'], Q['A'], Q['Z'], rotA_init) rotC_init = transformations.RotateGaussianARD(Q['C']) R_init = transformations.RotationOptimizer(rotX_init, rotC_init, D) # Rotate the D-dimensional state space (X, A, C) rotA = transformations.RotateGaussianARD(Q['A'], Q['alpha']) rotX = transformations.RotateSwitchingMarkovChain(Q['X'], Q['A'], Q['Z'], rotA) rotC = transformations.RotateGaussianARD(Q['C'], Q['gamma']) R = transformations.RotationOptimizer(rotX, rotC, D) if debug: rotate_kwargs = {'maxiter': 10, 'check_bound': True, 'check_gradient': True} else: rotate_kwargs = {'maxiter': 10} # Run inference if monitor: Q.plot() for n in range(maxiter): if n < update_hyper: Q.update('X', 'C', 'A', 'tau', 'Z', plot=monitor) if rotate: R_init.rotate(**rotate_kwargs) else: Q.update(plot=monitor) if rotate: R.rotate(**rotate_kwargs) return Q def simulate_data(N): """ Generate time-series data with switching dynamics. """ # Two states: 1) oscillation, 2) random walk w1 = 0.02 * 2*np.pi A = [ [[np.cos(w1), -np.sin(w1)], [np.sin(w1), np.cos(w1)]], [[ 1.0, 0.0], [ 0.0, 0.0]] ] C = [[1.0, 0.0]] # State switching probabilities q = 0.993 # probability to stay in the same state r = (1-q)/(2-1) # probability to switch P = q*np.identity(2) + r*(np.ones((2,2))-np.identity(2)) X = np.zeros((N, 2)) Z = np.zeros(N) Y = np.zeros(N) F = np.zeros(N) z = np.random.randint(2) x = np.random.randn(2) Z[0] = z X[0,:] = x for n in range(1,N): x = np.dot(A[z], x) + np.random.randn(2) f = np.dot(C, x) y = f + 5*np.random.randn() z = np.random.choice(2, p=P[z]) Z[n] = z X[n,:] = x Y[n] = y F[n] = f Y = Y[None,:] return (Y, F) @bpplt.interactive def demo(N=1000, maxiter=100, D=3, K=2, seed=42, plot=True, debug=False, rotate=True, monitor=True): """ Run the demo for linear state-space model with switching dynamics. """ # Use deterministic random numbers if seed is not None: np.random.seed(seed) # Generate data (Y, F) = simulate_data(N) # Plot observations if plot: plt.figure() bpplt.timeseries(F, linestyle='-', color='b') bpplt.timeseries(Y, linestyle='None', color='r', marker='x') # Apply the linear state-space model with switching dynamics Q = infer(Y, D, K, debug=debug, maxiter=maxiter, monitor=monitor, rotate=rotate, update_hyper=5) # Show results if plot: Q.plot() return if __name__ == '__main__': import sys, getopt, os try: opts, args = getopt.getopt(sys.argv[1:], "", ["n=", "d=", "k=", "seed=", "debug", "no-rotation", "no-monitor", "no-plot", "maxiter="]) except getopt.GetoptError: print('python lssm_sd.py ') print('--n= Number of data vectors') print('--d= Latent space dimensionality') print('--k= Number of mixed models') print('--maxiter= Maximum number of VB iterations') print('--seed= Seed (integer) for the random number generator') print('--no-rotation Do not peform rotation speed ups') print('--no-plot Do not plot results') print('--no-monitor Do not plot distributions during VB learning') print('--debug Check that the rotations are implemented correctly') sys.exit(2) kwargs = {} for opt, arg in opts: if opt == "--maxiter": kwargs["maxiter"] = int(arg) elif opt == "--d": kwargs["D"] = int(arg) elif opt == "--k": kwargs["K"] = int(arg) elif opt == "--seed": kwargs["seed"] = int(arg) elif opt == "--no-rotation": kwargs["rotate"] = False elif opt == "--no-monitor": kwargs["monitor"] = False elif opt == "--no-plot": kwargs["plot"] = False elif opt == "--debug": kwargs["debug"] = True elif opt in ("--n",): kwargs["N"] = int(arg) else: raise ValueError("Unhandled option given") demo(**kwargs) plt.show() bayespy-0.4.0/bayespy/demos/annealing.py0000644000175000017500000000640412533543154022007 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2015 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Demonstration of deterministic annealing. Deterministic annealing aims at avoiding convergence to local optima and finding the global optimum :cite:`Katahira:2008`. """ import numpy as np import scipy import matplotlib.pyplot as plt import bayespy.plot as myplt from bayespy.utils import misc from bayespy.utils import random from bayespy.nodes import GaussianARD, Categorical, Mixture from bayespy.inference.vmp.vmp import VB from bayespy.inference.vmp import transformations import bayespy.plot as bpplt from bayespy.demos import pca def run(N=500, seed=42, maxiter=100, plot=True): """ Run deterministic annealing demo for 1-D Gaussian mixture. """ if seed is not None: np.random.seed(seed) mu = GaussianARD(0, 1, plates=(2,), name='means') Z = Categorical([0.3, 0.7], plates=(N,), name='classes') Y = Mixture(Z, GaussianARD, mu, 1, name='observations') # Generate data z = Z.random() data = np.empty(N) for n in range(N): data[n] = [4, -4][z[n]] Y.observe(data) # Initialize means closer to the inferior local optimum in which the # cluster means are swapped mu.initialize_from_value([0, 6]) Q = VB(Y, Z, mu) Q.save() # # Standard VB-EM algorithm # Q.update(repeat=maxiter) mu_vbem = mu.u[0].copy() L_vbem = Q.compute_lowerbound() # # VB-EM with deterministic annealing # Q.load() beta = 0.01 while beta < 1.0: beta = min(beta*1.2, 1.0) print("Set annealing to %.2f" % beta) Q.set_annealing(beta) Q.update(repeat=maxiter, tol=1e-4) mu_anneal = mu.u[0].copy() L_anneal = Q.compute_lowerbound() print("==============================") print("RESULTS FOR VB-EM vs ANNEALING") print("Fixed component probabilities:", np.array([0.3, 0.7])) print("True component means:", np.array([4, -4])) print("VB-EM component means:", mu_vbem) print("VB-EM lower bound:", L_vbem) print("Annealed VB-EM component means:", mu_anneal) print("Annealed VB-EM lower bound:", L_anneal) return if __name__ == '__main__': import sys, getopt, os try: opts, args = getopt.getopt(sys.argv[1:], "", ["n=", "seed=", "maxiter="]) except getopt.GetoptError: print('python annealing.py ') print('--n= Number of data points') print('--maxiter= Maximum number of VB iterations') print('--seed= Seed (integer) for the random number generator') sys.exit(2) kwargs = {} for opt, arg in opts: if opt == "--maxiter": kwargs["maxiter"] = int(arg) elif opt == "--seed": kwargs["seed"] = int(arg) elif opt in ("--n",): kwargs["N"] = int(arg) run(**kwargs) plt.show() bayespy-0.4.0/bayespy/demos/categorical.py0000644000175000017500000000161312533543154022325 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2013 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import numpy as np from bayespy.utils import random from bayespy import nodes from bayespy.inference.vmp.vmp import VB def run(M=30, D=5): # Generate data y = np.random.randint(D, size=(M,)) # Construct model p = nodes.Dirichlet(1*np.ones(D), name='p') z = nodes.Categorical(p, plates=(M,), name='z') # Observe the data with randomly missing values mask = random.mask(M, p=0.5) z.observe(y, mask=mask) # Run VB-EM Q = VB(p, z) Q.update() # Show results z.show() p.show() if __name__ == '__main__': run() bayespy-0.4.0/bayespy/demos/__init__.py0000644000175000017500000000000012533543154021574 0ustar jluttinejluttine00000000000000bayespy-0.4.0/bayespy/demos/lssm_tvd.py0000644000175000017500000003460612533543154021713 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2013-2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Demonstrate the linear state-space model with time-varying dynamics. The observation is 1-D signal with changing frequency. The frequency oscillates so it can be learnt too. Missing values are used to create a few gaps in the data so the task is to reconstruct the gaps. For reference, see the following publication: (TODO) Some functions in this module are re-usable: * ``model`` can be used to construct the LSSM with switching dynamics. * ``infer`` can be used to apply the model to given data. """ import numpy as np import scipy import matplotlib.pyplot as plt from bayespy.nodes import (GaussianMarkovChain, VaryingGaussianMarkovChain, GaussianARD, Gamma, SumMultiply) from bayespy.utils import misc from bayespy.utils import random from bayespy.inference.vmp.vmp import VB from bayespy.inference.vmp import transformations from bayespy.inference.vmp.nodes.gaussian import GaussianMoments import bayespy.plot as bpplt def model(M, N, D, K): """ Construct the linear state-space model with time-varying dynamics For reference, see the following publication: (TODO) """ # # The model block for the latent mixing weight process # # Dynamics matrix with ARD # beta : (K) x () beta = Gamma(1e-5, 1e-5, plates=(K,), name='beta') # B : (K) x (K) B = GaussianARD(np.identity(K), beta, shape=(K,), plates=(K,), name='B', plotter=bpplt.GaussianHintonPlotter(rows=0, cols=1, scale=0), initialize=False) B.initialize_from_value(np.identity(K)) # Mixing weight process, that is, the weights in the linear combination of # state dynamics matrices # S : () x (N,K) S = GaussianMarkovChain(np.ones(K), 1e-6*np.identity(K), B, np.ones(K), n=N, name='S', plotter=bpplt.GaussianMarkovChainPlotter(scale=2), initialize=False) s = 10*np.random.randn(N,K) s[:,0] = 10 S.initialize_from_value(s) # # The model block for the latent states # # Projection matrix of the dynamics matrix # alpha : (K) x () alpha = Gamma(1e-5, 1e-5, plates=(D,K), name='alpha') alpha.initialize_from_value(1*np.ones((D,K))) # A : (D) x (D,K) A = GaussianARD(0, alpha, shape=(D,K), plates=(D,), name='A', plotter=bpplt.GaussianHintonPlotter(rows=0, cols=1, scale=0), initialize=False) # Initialize S and A such that A*S is almost an identity matrix a = np.zeros((D,D,K)) a[np.arange(D),np.arange(D),np.zeros(D,dtype=int)] = 1 a[:,:,0] = np.identity(D) / s[0,0] a[:,:,1:] = 0.1/s[0,0]*np.random.randn(D,D,K-1) A.initialize_from_value(a) # Latent states with dynamics # X : () x (N,D) X = VaryingGaussianMarkovChain(np.zeros(D), # mean of x0 1e-3*np.identity(D), # prec of x0 A, # dynamics matrices S._convert(GaussianMoments)[1:], # temporal weights np.ones(D), # innovation n=N, # time instances name='X', plotter=bpplt.GaussianMarkovChainPlotter(scale=2), initialize=False) X.initialize_from_value(np.random.randn(N,D)) # # The model block for observations # # Mixing matrix from latent space to observation space using ARD # gamma : (D) x () gamma = Gamma(1e-5, 1e-5, plates=(D,), name='gamma') gamma.initialize_from_value(1e-2*np.ones(D)) # C : (M,1) x (D) C = GaussianARD(0, gamma, shape=(D,), plates=(M,1), name='C', plotter=bpplt.GaussianHintonPlotter(rows=0, cols=2, scale=0)) C.initialize_from_value(np.random.randn(M,1,D)) # Noiseless process # F : (M,N) x () F = SumMultiply('d,d', C, X, name='F') # Observation noise # tau : () x () tau = Gamma(1e-5, 1e-5, name='tau') tau.initialize_from_value(1e2) # Observations # Y: (M,N) x () Y = GaussianARD(F, tau, name='Y') # Construct inference machine Q = VB(Y, F, C, gamma, X, A, alpha, tau, S, B, beta) return Q def infer(y, D, K, mask=True, maxiter=100, rotate=False, debug=False, precompute=False, update_hyper=0, start_rotating=0, start_rotating_weights=0, plot_C=True, monitor=True, autosave=None): """ Run VB inference for linear state-space model with time-varying dynamics. """ y = misc.atleast_nd(y, 2) (M, N) = np.shape(y) # Construct the model Q = model(M, N, D, K) if not plot_C: Q['C'].set_plotter(None) if autosave is not None: Q.set_autosave(autosave, iterations=10) # Observe data Q['Y'].observe(y, mask=mask) # Set up rotation speed-up if rotate: # Initial rotate the D-dimensional state space (X, A, C) # Does not update hyperparameters rotA_init = transformations.RotateGaussianARD(Q['A'], axis=0, precompute=precompute) rotX_init = transformations.RotateVaryingMarkovChain(Q['X'], Q['A'], Q['S']._convert(GaussianMoments)[...,1:,None], rotA_init) rotC_init = transformations.RotateGaussianARD(Q['C'], axis=0, precompute=precompute) R_X_init = transformations.RotationOptimizer(rotX_init, rotC_init, D) # Rotate the D-dimensional state space (X, A, C) rotA = transformations.RotateGaussianARD(Q['A'], Q['alpha'], axis=0, precompute=precompute) rotX = transformations.RotateVaryingMarkovChain(Q['X'], Q['A'], Q['S']._convert(GaussianMoments)[...,1:,None], rotA) rotC = transformations.RotateGaussianARD(Q['C'], Q['gamma'], axis=0, precompute=precompute) R_X = transformations.RotationOptimizer(rotX, rotC, D) # Rotate the K-dimensional latent dynamics space (S, A, C) rotB = transformations.RotateGaussianARD(Q['B'], Q['beta'], precompute=precompute) rotS = transformations.RotateGaussianMarkovChain(Q['S'], rotB) rotA = transformations.RotateGaussianARD(Q['A'], Q['alpha'], axis=-1, precompute=precompute) R_S = transformations.RotationOptimizer(rotS, rotA, K) if debug: rotate_kwargs = {'maxiter': 10, 'check_bound': True, 'check_gradient': True} else: rotate_kwargs = {'maxiter': 10} # Plot initial distributions if monitor: Q.plot() # Run inference using rotations for ind in range(maxiter): if ind < update_hyper: # It might be a good idea to learn the lower level nodes a bit # before starting to learn the upper level nodes. Q.update('X', 'C', 'A', 'tau', plot=monitor) if rotate and ind >= start_rotating: # Use the rotation which does not update alpha nor beta R_X_init.rotate(**rotate_kwargs) else: Q.update(plot=monitor) if rotate and ind >= start_rotating: # It might be a good idea to not rotate immediately because it # might lead to pruning out components too efficiently before # even estimating them roughly R_X.rotate(**rotate_kwargs) if ind >= start_rotating_weights: R_S.rotate(**rotate_kwargs) # Return the posterior approximation return Q def simulate_data(N): """ Generate a signal with changing frequency """ t = np.arange(N) a = 0.1 * 2*np.pi # base frequency b = 0.01 * 2*np.pi # frequency of the frequency change c = 8 # magnitude of the frequency change f = np.sin( a * (t + c*np.sin(b*t)) ) y = f + 0.1*np.random.randn(N) return (y, f) @bpplt.interactive def demo(N=1000, D=5, K=4, seed=42, maxiter=200, rotate=True, debug=False, precompute=False, plot=True): # Seed for random number generator if seed is not None: np.random.seed(seed) # Create data (y, f) = simulate_data(N) # Create some gaps mask_gaps = misc.trues(N) for m in range(100, N, 140): start = m end = min(m+15, N-1) mask_gaps[start:end] = False # Randomly missing values mask_random = np.logical_or(random.mask(N, p=0.8), np.logical_not(mask_gaps)) # Remove the observations mask = np.logical_and(mask_gaps, mask_random) y[~mask] = np.nan # BayesPy doesn't require NaNs, they're just for plotting. # Add row axes y = y[None,...] f = f[None,...] mask = mask[None,...] mask_gaps = mask_gaps[None,...] mask_random = mask_random[None,...] # Run the method Q = infer(y, D, K, mask=mask, maxiter=maxiter, rotate=rotate, debug=debug, precompute=precompute, update_hyper=10, start_rotating_weights=20, monitor=True) if plot: # Plot observations plt.figure() bpplt.timeseries_normal(Q['F'], scale=2) bpplt.timeseries(f, linestyle='-', color='b') bpplt.timeseries(y, linestyle='None', color='r', marker='.') plt.ylim([-2, 2]) # Plot latent space Q.plot('X') # Plot mixing weight space Q.plot('S') # Compute RMSE rmse_random = misc.rmse(Q['Y'].get_moments()[0][~mask_random], f[~mask_random]) rmse_gaps = misc.rmse(Q['Y'].get_moments()[0][~mask_gaps], f[~mask_gaps]) print("RMSE for randomly missing values: %f" % rmse_random) print("RMSE for gap values: %f" % rmse_gaps) if __name__ == '__main__': import sys, getopt, os try: opts, args = getopt.getopt(sys.argv[1:], "", [ "n=", "d=", "k=", "seed=", "maxiter=", "debug", "precompute", "no-plot", "no-rotation"]) except getopt.GetoptError: print('python lssm_tvd.py ') print('--n= Number of data vectors') print('--d= Dimensionality of the latent vectors in the model') print('--k= Dimensionality of the latent mixing weights') print('--no-rotation Do not apply speed-up rotations') print('--maxiter= Maximum number of VB iterations') print('--seed= Seed (integer) for the random number generator') print('--debug Check that the rotations are implemented correctly') print('--no-plot Do not plot results') print('--precompute Precompute some moments when rotating. May ' 'speed up or slow down.') sys.exit(2) kwargs = {} for opt, arg in opts: if opt == "--no-rotation": kwargs["rotate"] = False elif opt == "--maxiter": kwargs["maxiter"] = int(arg) elif opt == "--debug": kwargs["debug"] = True elif opt == "--precompute": kwargs["precompute"] = True elif opt == "--seed": kwargs["seed"] = int(arg) elif opt == "--n": kwargs["N"] = int(arg) elif opt == "--d": kwargs["D"] = int(arg) elif opt == "--k": if int(arg) == 0: kwargs["K"] = None else: kwargs["K"] = int(arg) elif opt == "--no-plot": kwargs["plot"] = False else: raise ValueError("Unhandled argument given") demo(**kwargs) plt.show() bayespy-0.4.0/bayespy/demos/pca.py0000644000175000017500000001106112533543154020611 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import numpy as np import matplotlib.pyplot as plt import bayespy.plot as myplt from bayespy.utils import misc from bayespy.utils import random from bayespy import nodes from bayespy.inference.vmp.vmp import VB from bayespy.inference.vmp import transformations import bayespy.plot as bpplt def model(M, N, D): # Construct the PCA model with ARD # ARD alpha = nodes.Gamma(1e-2, 1e-2, plates=(D,), name='alpha') # Loadings W = nodes.GaussianARD(0, alpha, shape=(D,), plates=(M,1), name='W') # States X = nodes.GaussianARD(0, 1, shape=(D,), plates=(1,N), name='X') # PCA F = nodes.SumMultiply('i,i', W, X, name='F') # Noise tau = nodes.Gamma(1e-2, 1e-2, name='tau') # Noisy observations Y = nodes.GaussianARD(F, tau, name='Y') # Initialize some nodes randomly X.initialize_from_random() W.initialize_from_random() return VB(Y, F, W, X, tau, alpha) @bpplt.interactive def run(M=10, N=100, D_y=3, D=5, seed=42, rotate=False, maxiter=1000, debug=False, plot=True): if seed is not None: np.random.seed(seed) # Generate data w = np.random.normal(0, 1, size=(M,1,D_y)) x = np.random.normal(0, 1, size=(1,N,D_y)) f = misc.sum_product(w, x, axes_to_sum=[-1]) y = f + np.random.normal(0, 0.1, size=(M,N)) # Construct model Q = model(M, N, D) # Data with missing values mask = random.mask(M, N, p=0.5) # randomly missing y[~mask] = np.nan Q['Y'].observe(y, mask=mask) # Run inference algorithm if rotate: # Use rotations to speed up learning rotW = transformations.RotateGaussianARD(Q['W'], Q['alpha']) rotX = transformations.RotateGaussianARD(Q['X']) R = transformations.RotationOptimizer(rotW, rotX, D) if debug: Q.callback = lambda : R.rotate(check_bound=True, check_gradient=True) else: Q.callback = R.rotate # Use standard VB-EM alone Q.update(repeat=maxiter) # Plot results if plot: plt.figure() bpplt.timeseries_normal(Q['F'], scale=2) bpplt.timeseries(f, color='g', linestyle='-') bpplt.timeseries(y, color='r', linestyle='None', marker='+') if __name__ == '__main__': import sys, getopt, os try: opts, args = getopt.getopt(sys.argv[1:], "", ["m=", "n=", "d=", "k=", "seed=", "maxiter=", "debug", "rotate"]) except getopt.GetoptError: print('python demo_pca.py ') print('--m= Dimensionality of data vectors') print('--n= Number of data vectors') print('--d= Dimensionality of the latent vectors in the model') print('--k= Dimensionality of the true latent vectors') print('--rotate Apply speed-up rotations') print('--maxiter= Maximum number of VB iterations') print('--seed= Seed (integer) for the random number generator') print('--debug Check that the rotations are implemented correctly') sys.exit(2) kwargs = {} for opt, arg in opts: if opt == "--rotate": kwargs["rotate"] = True elif opt == "--maxiter": kwargs["maxiter"] = int(arg) elif opt == "--debug": kwargs["debug"] = True elif opt == "--seed": kwargs["seed"] = int(arg) elif opt in ("--m",): kwargs["M"] = int(arg) elif opt in ("--n",): kwargs["N"] = int(arg) elif opt in ("--d",): kwargs["D"] = int(arg) elif opt in ("--k",): kwargs["D_y"] = int(arg) run(**kwargs) plt.show() bayespy-0.4.0/bayespy/demos/saving.py0000644000175000017500000000767412533543154021354 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2013 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import numpy as np import matplotlib.pyplot as plt import h5py import tempfile import bayespy.plot as bpplt from bayespy.utils import misc from bayespy.utils import random from bayespy.inference.vmp import nodes from bayespy.inference.vmp.vmp import VB def pca_model(M, N, D): # Construct the PCA model with ARD # ARD alpha = nodes.Gamma(1e-2, 1e-2, plates=(D,), name='alpha') # Loadings W = nodes.Gaussian(np.zeros(D), alpha.as_diagonal_wishart(), name="W", plates=(M,1)) # States X = nodes.Gaussian(np.zeros(D), np.identity(D), name="X", plates=(1,N)) # PCA WX = nodes.Dot(W, X, name="WX") # Noise tau = nodes.Gamma(1e-2, 1e-2, name="tau", plates=()) # Noisy observations Y = nodes.GaussianARD(WX, tau, name="Y", plates=(M,N)) return (Y, WX, W, X, tau, alpha) @bpplt.interactive def run(M=10, N=100, D_y=3, D=5): seed = 45 print('seed =', seed) np.random.seed(seed) # Check HDF5 version. if h5py.version.hdf5_version_tuple < (1,8,7): print("WARNING! Your HDF5 version is %s. HDF5 versions <1.8.7 are not " "able to save empty arrays, thus you may experience problems if " "you for instance try to save before running any iteration steps." % str(h5py.version.hdf5_version_tuple)) # Generate data w = np.random.normal(0, 1, size=(M,1,D_y)) x = np.random.normal(0, 1, size=(1,N,D_y)) f = misc.sum_product(w, x, axes_to_sum=[-1]) y = f + np.random.normal(0, 0.5, size=(M,N)) # Construct model (Y, WX, W, X, tau, alpha) = pca_model(M, N, D) # Data with missing values mask = random.mask(M, N, p=0.9) # randomly missing mask[:,20:40] = False # gap missing y[~mask] = np.nan Y.observe(y, mask=mask) # Construct inference machine Q = VB(Y, W, X, tau, alpha, autosave_iterations=5) # Initialize some nodes randomly X.initialize_from_value(X.random()) W.initialize_from_value(W.random()) # Save the state into a HDF5 file filename = tempfile.NamedTemporaryFile(suffix='hdf5').name Q.update(X, W, alpha, tau, repeat=1) Q.save(filename=filename) # Inference loop. Q.update(X, W, alpha, tau, repeat=10) # Reload the state from the HDF5 file Q.load(filename=filename) # Inference loop again. Q.update(X, W, alpha, tau, repeat=10) # NOTE: Saving and loading requires that you have the model # constructed. "Save" does not store the model structure nor does "load" # read it. They are just used for reading and writing the contents of the # nodes. Thus, if you want to load, you first need to construct the same # model that was used for saving and then use load to set the states of the # nodes. plt.clf() WX_params = WX.get_parameters() fh = WX_params[0] * np.ones(y.shape) err_fh = 2*np.sqrt(WX_params[1] + 1/tau.get_moments()[0]) * np.ones(y.shape) for m in range(M): plt.subplot(M,1,m+1) #errorplot(y, error=None, x=None, lower=None, upper=None): bpplt.errorplot(fh[m], x=np.arange(N), error=err_fh[m]) plt.plot(np.arange(N), f[m], 'g') plt.plot(np.arange(N), y[m], 'r+') plt.figure() Q.plot_iteration_by_nodes() plt.figure() plt.subplot(2,2,1) bpplt.binary_matrix(W.mask) plt.subplot(2,2,2) bpplt.binary_matrix(X.mask) plt.subplot(2,2,3) #bpplt.binary_matrix(WX.get_mask()) plt.subplot(2,2,4) bpplt.binary_matrix(Y.mask) if __name__ == '__main__': run() plt.show() bayespy-0.4.0/bayespy/demos/hmm.py0000644000175000017500000001134712533543154020636 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Demonstrate categorical Markov chain with hidden Markov model (HMM) """ import numpy as np import matplotlib.pyplot as plt from bayespy.nodes import Gaussian, \ CategoricalMarkovChain, \ Dirichlet, \ Mixture, \ Categorical from bayespy.inference.vmp.vmp import VB import bayespy.plot as bpplt def hidden_markov_model(distribution, *args, K=3, N=100): # Prior for initial state probabilities alpha = Dirichlet(1e-3*np.ones(K), name='alpha') # Prior for state transition probabilities A = Dirichlet(1e-3*np.ones(K), plates=(K,), name='A') # Hidden states (with unknown initial state probabilities and state # transition probabilities) Z = CategoricalMarkovChain(alpha, A, states=N, name='Z') # Emission/observation distribution Y = Mixture(Z, distribution, *args, name='Y') Q = VB(Y, Z, alpha, A) return Q def mixture_model(distribution, *args, K=3, N=100): # Prior for state probabilities alpha = Dirichlet(1e-3*np.ones(K), name='alpha') # Cluster assignments Z = Categorical(alpha, plates=(N,), name='Z') # Observation distribution Y = Mixture(Z, distribution, *args, name='Y') Q = VB(Y, Z, alpha) return Q @bpplt.interactive def run(N=200, maxiter=10, seed=42, std=2.0, plot=True): # Use deterministic random numbers if seed is not None: np.random.seed(seed) # # Generate data # mu = np.array([ [0,0], [3,4], [6,0] ]) K = 3 p0 = np.ones(K) / K q = 0.9 # probability to stay in the same state r = (1-q)/(K-1) P = q*np.identity(K) + r*(np.ones((3,3))-np.identity(3)) y = np.zeros((N,2)) z = np.zeros(N) state = np.random.choice(K, p=p0) for n in range(N): z[n] = state y[n,:] = std*np.random.randn(2) + mu[state] state = np.random.choice(K, p=P[state]) plt.figure() # Plot data plt.subplot(1,3,1) plt.axis('equal') plt.title('True classification') colors = [ [[1,0,0], [0,1,0], [0,0,1]][int(state)] for state in z ] plt.plot(y[:,0], y[:,1], 'k-', zorder=-10) plt.scatter(y[:,0], y[:,1], c=colors, s=40) # # Use HMM # # Run VB inference for HMM Q_hmm = hidden_markov_model(Gaussian, mu, K*[std**(-2)*np.identity(2)], K=K, N=N) Q_hmm['Y'].observe(y) Q_hmm.update(repeat=maxiter) # Plot results plt.subplot(1,3,2) plt.axis('equal') plt.title('Classification with HMM') colors = Q_hmm['Y'].parents[0]._message_to_child()[0] plt.plot(y[:,0], y[:,1], 'k-', zorder=-10) plt.scatter(y[:,0], y[:,1], c=colors, s=40) # # Use mixture model # # For comparison, run VB for Gaussian mixture Q_mix = mixture_model(Gaussian, mu, K*[std**(-2)*np.identity(2)], K=K, N=N) Q_mix['Y'].observe(y) Q_mix.update(repeat=maxiter) # Plot results plt.subplot(1,3,3) plt.axis('equal') plt.title('Classification with mixture') colors = Q_mix['Y'].parents[0]._message_to_child()[0] plt.plot(y[:,0], y[:,1], 'k-', zorder=-10) plt.scatter(y[:,0], y[:,1], c=colors, s=40) if __name__ == '__main__': import sys, getopt, os try: opts, args = getopt.getopt(sys.argv[1:], "", ["n=", "seed=", "std=", "maxiter="]) except getopt.GetoptError: print('python demo_lssm.py ') print('--n= Number of data vectors') print('--std= Standard deviation of the Gaussians') print('--maxiter= Maximum number of VB iterations') print('--seed= Seed (integer) for the random number generator') sys.exit(2) kwargs = {} for opt, arg in opts: if opt == "--maxiter": kwargs["maxiter"] = int(arg) elif opt == "--std": kwargs["std"] = float(arg) elif opt == "--seed": kwargs["seed"] = int(arg) elif opt in ("--n",): kwargs["N"] = int(arg) else: raise ValueError("Unhandled option given") run(**kwargs) plt.show() bayespy-0.4.0/bayespy/demos/stochastic_inference.py0000644000175000017500000001162512533543154024236 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2015 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Stochastic variational inference on mixture of Gaussians Stochastic variational inference is a scalable variational Bayesian learning method which utilizes stochastic gradient. For details, see :cite:`Hoffman:2013`. """ import numpy as np import scipy import matplotlib.pyplot as plt import bayespy.plot as myplt from bayespy.utils import misc from bayespy.utils import random from bayespy.nodes import Gaussian, Categorical, Mixture, Dirichlet from bayespy.inference.vmp.vmp import VB from bayespy.inference.vmp import transformations import bayespy.plot as bpplt from bayespy.demos import pca def run(N=100000, N_batch=50, seed=42, maxiter=100, plot=True): """ Run deterministic annealing demo for 1-D Gaussian mixture. """ if seed is not None: np.random.seed(seed) # Number of clusters in the model K = 20 # Dimensionality of the data D = 5 # Generate data K_true = 10 spread = 5 means = spread * np.random.randn(K_true, D) z = random.categorical(np.ones(K_true), size=N) data = np.empty((N,D)) for n in range(N): data[n] = means[z[n]] + np.random.randn(D) # # Standard VB-EM algorithm # # Full model mu = Gaussian(np.zeros(D), np.identity(D), plates=(K,), name='means') alpha = Dirichlet(np.ones(K), name='class probabilities') Z = Categorical(alpha, plates=(N,), name='classes') Y = Mixture(Z, Gaussian, mu, np.identity(D), name='observations') # Break symmetry with random initialization of the means mu.initialize_from_random() # Put the data in Y.observe(data) # Run inference Q = VB(Y, Z, mu, alpha) Q.save(mu) Q.update(repeat=maxiter) if plot: bpplt.pyplot.plot(np.cumsum(Q.cputime), Q.L, 'k-') max_cputime = np.sum(Q.cputime[~np.isnan(Q.cputime)]) # # Stochastic variational inference # # Construct smaller model (size of the mini-batch) mu = Gaussian(np.zeros(D), np.identity(D), plates=(K,), name='means') alpha = Dirichlet(np.ones(K), name='class probabilities') Z = Categorical(alpha, plates=(N_batch,), plates_multiplier=(N/N_batch,), name='classes') Y = Mixture(Z, Gaussian, mu, np.identity(D), name='observations') # Break symmetry with random initialization of the means mu.initialize_from_random() # Inference engine Q = VB(Y, Z, mu, alpha, autosave_filename=Q.autosave_filename) Q.load(mu) # Because using mini-batches, messages need to be multiplied appropriately print("Stochastic variational inference...") Q.ignore_bound_checks = True maxiter *= int(N/N_batch) delay = 1 forgetting_rate = 0.7 for n in range(maxiter): # Observe a mini-batch subset = np.random.choice(N, N_batch) Y.observe(data[subset,:]) # Learn intermediate variables Q.update(Z) # Set step length step = (n + delay) ** (-forgetting_rate) # Stochastic gradient for the global variables Q.gradient_step(mu, alpha, scale=step) if np.sum(Q.cputime[:n]) > max_cputime: break if plot: bpplt.pyplot.plot(np.cumsum(Q.cputime), Q.L, 'r:') bpplt.pyplot.xlabel('CPU time (in seconds)') bpplt.pyplot.ylabel('VB lower bound') bpplt.pyplot.legend(['VB-EM', 'Stochastic inference'], loc='lower right') bpplt.pyplot.title('VB for Gaussian mixture model') return if __name__ == '__main__': import sys, getopt, os try: opts, args = getopt.getopt(sys.argv[1:], "", ["n=", "batch=", "seed=", "maxiter="]) except getopt.GetoptError: print('python stochastic_inference.py ') print('--n= Number of data points') print('--batch= Mini-batch size') print('--maxiter= Maximum number of VB iterations') print('--seed= Seed (integer) for the random number generator') sys.exit(2) kwargs = {} for opt, arg in opts: if opt == "--maxiter": kwargs["maxiter"] = int(arg) elif opt == "--seed": kwargs["seed"] = int(arg) elif opt in ("--n",): kwargs["N"] = int(arg) elif opt in ("--batch",): kwargs["N_batch"] = int(arg) run(**kwargs) plt.show() bayespy-0.4.0/bayespy/demos/gamma_shape.py0000664000175000017500000000074412556654271022330 0ustar jluttinejluttine00000000000000 from bayespy import nodes from bayespy.inference import VB def run(): a = nodes.GammaShape(name='a') b = nodes.Gamma(1e-5, 1e-5, name='b') tau = nodes.Gamma(a, b, plates=(1000,), name='tau') tau.observe(nodes.Gamma(10, 20, plates=(1000,)).random()) Q = VB(tau, a, b) Q.update(repeat=1000) print("True gamma parameters:", 10.0, 20.0) print("Estimated parameters from 1000 samples:", a.u[0], b.u[0]) if __name__ == "__main__": run() bayespy-0.4.0/bayespy/demos/collapsed_cg.py0000644000175000017500000000632412535533206022472 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014-2015 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Demonstrate Riemannian conjugate gradient """ import numpy as np from bayespy.nodes import (GaussianARD, Gamma, SumMultiply) from bayespy.utils import random from bayespy.inference.vmp.vmp import VB import bayespy.plot as bpplt from bayespy.demos import mog def pca(): np.random.seed(41) M = 10 N = 3000 D = 5 # Construct the PCA model alpha = Gamma(1e-3, 1e-3, plates=(D,), name='alpha') W = GaussianARD(0, alpha, plates=(M,1), shape=(D,), name='W') X = GaussianARD(0, 1, plates=(1,N), shape=(D,), name='X') tau = Gamma(1e-3, 1e-3, name='tau') W.initialize_from_random() F = SumMultiply('d,d->', W, X) Y = GaussianARD(F, tau, name='Y') # Observe data data = np.sum(np.random.randn(M,1,D-1) * np.random.randn(1,N,D-1), axis=-1) + 1e-1 * np.random.randn(M,N) Y.observe(data) # Initialize VB engine Q = VB(Y, X, W, alpha, tau) # Take one update step (so phi is ok) Q.update(repeat=1) Q.save() # Run VB-EM Q.update(repeat=200) bpplt.pyplot.plot(np.cumsum(Q.cputime), Q.L, 'k-') # Restore the state Q.load() # Run Riemannian conjugate gradient #Q.optimize(X, alpha, maxiter=100, collapsed=[W, tau]) Q.optimize(W, tau, maxiter=100, collapsed=[X, alpha]) bpplt.pyplot.plot(np.cumsum(Q.cputime), Q.L, 'r:') bpplt.pyplot.show() def mixture_of_gaussians(): """Collapsed Riemannian conjugate gradient demo This is similar although not exactly identical to an experiment in (Hensman et al 2012). """ np.random.seed(41) # Number of samples N = 1000 # Number of clusters in the model (five in the data) K = 10 # Overlap parameter of clusters R = 2 # Construct the model Q = mog.gaussianmix_model(N, K, 2, covariance='diagonal') # Generate data from five Gaussian clusters mu = np.array([[0, 0], [R, R], [-R, R], [R, -R], [-R, -R]]) Z = random.categorical(np.ones(5), size=N) data = np.empty((N, 2)) for n in range(N): data[n,:] = mu[Z[n]] + np.random.randn(2) Q['Y'].observe(data) # Take one update step (so phi is ok) Q.update(repeat=1) Q.save() # Run standard VB-EM Q.update(repeat=1000, tol=0) bpplt.pyplot.plot(np.cumsum(Q.cputime), Q.L, 'k-') # Restore the initial state Q.load() # Run Riemannian conjugate gradient Q.optimize('alpha', 'X', 'Lambda', collapsed=['z'], maxiter=300, tol=0) bpplt.pyplot.plot(np.cumsum(Q.cputime), Q.L, 'r:') bpplt.pyplot.xlabel('CPU time (in seconds)') bpplt.pyplot.ylabel('VB lower bound') bpplt.pyplot.legend(['VB-EM', 'Collapsed Riemannian CG'], loc='lower right') ## bpplt.pyplot.figure() ## bpplt.pyplot.plot(data[:,0], data[:,1], 'rx') ## bpplt.pyplot.title('Data') bpplt.pyplot.show() if __name__ == "__main__": #pca() mixture_of_gaussians() bayespy-0.4.0/bayespy/demos/lssm.py0000644000175000017500000002470212533543154021032 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2013-2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Demonstrate linear Gaussian state-space model. Some of the functions in this module are re-usable: * ``model`` can be used to construct the classical linear state-space model. * ``infer`` can be used to apply linear state-space model to given data. """ import numpy as np import scipy import matplotlib.pyplot as plt from bayespy.nodes import GaussianMarkovChain from bayespy.nodes import Gaussian, GaussianARD from bayespy.nodes import Gamma from bayespy.nodes import SumMultiply from bayespy.inference.vmp.nodes.gamma import diagonal from bayespy.utils import random from bayespy.inference.vmp.vmp import VB from bayespy.inference.vmp import transformations import bayespy.plot as bpplt def model(M=10, N=100, D=3): """ Construct linear state-space model. See, for instance, the following publication: "Fast variational Bayesian linear state-space model" Luttinen (ECML 2013) """ # Dynamics matrix with ARD alpha = Gamma(1e-5, 1e-5, plates=(D,), name='alpha') A = GaussianARD(0, alpha, shape=(D,), plates=(D,), plotter=bpplt.GaussianHintonPlotter(rows=0, cols=1, scale=0), name='A') A.initialize_from_value(np.identity(D)) # Latent states with dynamics X = GaussianMarkovChain(np.zeros(D), # mean of x0 1e-3*np.identity(D), # prec of x0 A, # dynamics np.ones(D), # innovation n=N, # time instances plotter=bpplt.GaussianMarkovChainPlotter(scale=2), name='X') X.initialize_from_value(np.random.randn(N,D)) # Mixing matrix from latent space to observation space using ARD gamma = Gamma(1e-5, 1e-5, plates=(D,), name='gamma') gamma.initialize_from_value(1e-2*np.ones(D)) C = GaussianARD(0, gamma, shape=(D,), plates=(M,1), plotter=bpplt.GaussianHintonPlotter(rows=0, cols=2, scale=0), name='C') C.initialize_from_value(np.random.randn(M,1,D)) # Observation noise tau = Gamma(1e-5, 1e-5, name='tau') tau.initialize_from_value(1e2) # Underlying noiseless function F = SumMultiply('i,i', C, X, name='F') # Noisy observations Y = GaussianARD(F, tau, name='Y') Q = VB(Y, F, C, gamma, X, A, alpha, tau, C) return Q def infer(y, D, mask=True, maxiter=100, rotate=True, debug=False, precompute=False, update_hyper=0, start_rotating=0, plot_C=True, monitor=True, autosave=None): """ Apply linear state-space model for the given data. """ (M, N) = np.shape(y) # Construct the model Q = model(M, N, D) if not plot_C: Q['C'].set_plotter(None) if autosave is not None: Q.set_autosave(autosave, iterations=10) # Observe data Q['Y'].observe(y, mask=mask) # Set up rotation speed-up if rotate: # Initial rotate the D-dimensional state space (X, A, C) # Does not update hyperparameters rotA_init = transformations.RotateGaussianARD(Q['A'], axis=0, precompute=precompute) rotX_init = transformations.RotateGaussianMarkovChain(Q['X'], rotA_init) rotC_init = transformations.RotateGaussianARD(Q['C'], axis=0, precompute=precompute) R_X_init = transformations.RotationOptimizer(rotX_init, rotC_init, D) # Rotate the D-dimensional state space (X, A, C) rotA = transformations.RotateGaussianARD(Q['A'], Q['alpha'], axis=0, precompute=precompute) rotX = transformations.RotateGaussianMarkovChain(Q['X'], rotA) rotC = transformations.RotateGaussianARD(Q['C'], Q['gamma'], axis=0, precompute=precompute) R_X = transformations.RotationOptimizer(rotX, rotC, D) # Keyword arguments for the rotation if debug: rotate_kwargs = {'maxiter': 10, 'check_bound': True, 'check_gradient': True} else: rotate_kwargs = {'maxiter': 10} # Plot initial distributions if monitor: Q.plot() # Run inference using rotations for ind in range(maxiter): if ind < update_hyper: # It might be a good idea to learn the lower level nodes a bit # before starting to learn the upper level nodes. Q.update('X', 'C', 'A', 'tau', plot=monitor) if rotate and ind >= start_rotating: # Use the rotation which does not update alpha nor beta R_X_init.rotate(**rotate_kwargs) else: Q.update(plot=monitor) if rotate and ind >= start_rotating: # It might be a good idea to not rotate immediately because it # might lead to pruning out components too efficiently before # even estimating them roughly R_X.rotate(**rotate_kwargs) # Return the posterior approximation return Q def simulate_data(M, N): """ Generate a dataset using linear state-space model. The process has two latent oscillation components and one random walk component. """ # Simulate some data D = 3 c = np.random.randn(M, D) w = 0.3 a = np.array([[np.cos(w), -np.sin(w), 0], [np.sin(w), np.cos(w), 0], [0, 0, 1]]) x = np.empty((N,D)) f = np.empty((M,N)) y = np.empty((M,N)) x[0] = 10*np.random.randn(D) f[:,0] = np.dot(c,x[0]) y[:,0] = f[:,0] + 3*np.random.randn(M) for n in range(N-1): x[n+1] = np.dot(a,x[n]) + np.random.randn(D) f[:,n+1] = np.dot(c,x[n+1]) y[:,n+1] = f[:,n+1] + 3*np.random.randn(M) return (y, f) @bpplt.interactive def demo(M=6, N=200, D=3, maxiter=100, debug=False, seed=42, rotate=True, precompute=False, plot=True, monitor=True): """ Run the demo for linear state-space model. """ # Use deterministic random numbers if seed is not None: np.random.seed(seed) # Get data (y, f) = simulate_data(M, N) # Add missing values randomly mask = random.mask(M, N, p=0.3) # Add missing values to a period of time mask[:,30:80] = False y[~mask] = np.nan # BayesPy doesn't require this. Just for plotting. # Run inference Q = infer(y, D, mask=mask, rotate=rotate, debug=debug, monitor=monitor, maxiter=maxiter) if plot: # Show results plt.figure() bpplt.timeseries_normal(Q['F'], scale=2) bpplt.timeseries(f, linestyle='-', color='b') bpplt.timeseries(y, linestyle='None', color='r', marker='.') if __name__ == '__main__': import sys, getopt, os try: opts, args = getopt.getopt(sys.argv[1:], "", ["m=", "n=", "d=", "seed=", "maxiter=", "debug", "precompute", "no-plot", "no-monitor", "no-rotation"]) except getopt.GetoptError: print('python lssm.py ') print('--m= Dimensionality of data vectors') print('--n= Number of data vectors') print('--d= Dimensionality of the latent vectors in the model') print('--no-rotation Do not apply speed-up rotations') print('--maxiter= Maximum number of VB iterations') print('--seed= Seed (integer) for the random number generator') print('--debug Check that the rotations are implemented correctly') print('--no-plot Do not plot the results') print('--no-monitor Do not plot distributions during learning') print('--precompute Precompute some moments when rotating. May ' 'speed up or slow down.') sys.exit(2) kwargs = {} for opt, arg in opts: if opt == "--no-rotation": kwargs["rotate"] = False elif opt == "--maxiter": kwargs["maxiter"] = int(arg) elif opt == "--debug": kwargs["debug"] = True elif opt == "--precompute": kwargs["precompute"] = True elif opt == "--seed": kwargs["seed"] = int(arg) elif opt in ("--m",): kwargs["M"] = int(arg) elif opt in ("--n",): kwargs["N"] = int(arg) elif opt in ("--d",): kwargs["D"] = int(arg) elif opt in ("--no-plot"): kwargs["plot"] = False elif opt in ("--no-monitor"): kwargs["monitor"] = False else: raise ValueError("Unhandled option given") demo(**kwargs) plt.show() bayespy-0.4.0/bayespy/demos/lda.py0000644000175000017500000001611112537637533020620 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2015 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import numpy as np from bayespy import nodes from bayespy.inference import VB from bayespy.inference.vmp.nodes.constant import Constant from bayespy.inference.vmp.nodes.categorical import CategoricalMoments import bayespy.plot as bpplt def model(n_documents, n_topics, n_vocabulary, corpus, word_documents, plates_multiplier=1): ''' Construct Latent Dirichlet Allocation model. Parameters ---------- documents : int The number of documents topics : int The number of topics vocabulary : int The number of words in the vocabulary corpus : integer array The vocabulary index of each word in the corpus word_documents : integer array The document index of each word in the corpus ''' # Topic distributions for each document p_topic = nodes.Dirichlet(np.ones(n_topics), plates=(n_documents,), name='p_topic') # Word distributions for each topic p_word = nodes.Dirichlet(np.ones(n_vocabulary), plates=(n_topics,), name='p_word') # Use a simple wrapper node so that the value of this can be changed if one # uses stocahstic variational inference word_documents = Constant(CategoricalMoments(n_documents), word_documents, name='word_documents') # Choose a topic for each word in the corpus topics = nodes.Categorical(nodes.Gate(word_documents, p_topic), plates=(len(corpus),), plates_multiplier=(plates_multiplier,), name='topics') # Choose each word in the corpus from the vocabulary words = nodes.Categorical(nodes.Gate(topics, p_word), name='words') # Observe the corpus words.observe(corpus) # Break symmetry by random initialization p_topic.initialize_from_random() p_word.initialize_from_random() return VB(words, topics, p_word, p_topic, word_documents) def generate_data(n_documents, n_topics, n_vocabulary, n_words): # Generate random data from the generative model # Generate document assignments for the words word_documents = nodes.Categorical(np.ones(n_documents)/n_documents, plates=(n_words,)).random() # Topic distribution for each document p_topic = nodes.Dirichlet(1e-1*np.ones(n_topics), plates=(n_documents,)).random() # Word distribution for each topic p_word = nodes.Dirichlet(1e-1*np.ones(n_vocabulary), plates=(n_topics,)).random() # Topic for each word in each document topic = nodes.Categorical(p_topic[word_documents], plates=(n_words,)).random() # Each word in each document corpus = nodes.Categorical(p_word[topic], plates=(n_words,)).random() bpplt.pyplot.figure() bpplt.hinton(p_topic) bpplt.pyplot.title("True topic distribution for each document") bpplt.pyplot.xlabel("Topics") bpplt.pyplot.ylabel("Documents") bpplt.pyplot.figure() bpplt.hinton(p_word) bpplt.pyplot.title("True word distributions for each topic") bpplt.pyplot.xlabel("Words") bpplt.pyplot.ylabel("Topics") return (corpus, word_documents) def run(n_documents=30, n_topics=5, n_vocabulary=10, n_words=50000, stochastic=False, maxiter=1000, seed=None): if seed is not None: np.random.seed(seed) (corpus, word_documents) = generate_data(n_documents, n_topics, n_vocabulary, n_words) if not stochastic: Q = model(n_documents=n_documents, n_topics=n_topics, n_vocabulary=n_vocabulary, corpus=corpus, word_documents=word_documents) Q.update(repeat=maxiter) else: subset_size = 1000 Q = model(n_documents=n_documents, n_topics=n_topics, n_vocabulary=n_vocabulary, corpus=corpus[:subset_size], word_documents=word_documents[:subset_size], plates_multiplier=n_words/subset_size) Q.ignore_bound_checks = True delay = 1 forgetting_rate = 0.7 for n in range(maxiter): # Observe a mini-batch subset = np.random.choice(n_words, subset_size) Q['words'].observe(corpus[subset]) Q['word_documents'].set_value(word_documents[subset]) # Learn intermediate variables Q.update('topics') # Set step length step = (n + delay) ** (-forgetting_rate) # Stochastic gradient for the global variables Q.gradient_step('p_topic', 'p_word', scale=step) bpplt.pyplot.figure() bpplt.pyplot.plot(Q.L) bpplt.pyplot.figure() bpplt.hinton(Q['p_topic']) bpplt.pyplot.title("Posterior topic distribution for each document") bpplt.pyplot.xlabel("Topics") bpplt.pyplot.ylabel("Documents") bpplt.pyplot.figure() bpplt.hinton(Q['p_word']) bpplt.pyplot.title("Posterior word distributions for each topic") bpplt.pyplot.xlabel("Words") bpplt.pyplot.ylabel("Topics") return if __name__ == '__main__': import sys, getopt, os try: opts, args = getopt.getopt(sys.argv[1:], "", ["documents=", "topics=", "vocabulary=", "words=", "stochastic", "seed=", "maxiter="]) except getopt.GetoptError: print('python lda.py ') print('--documents= The number of documents') print('--topics= The number of topics') print('--vocabulary= The size of the vocabulary') print('--words= The size of the corpus') print('--maxiter= Maximum number of VB iterations') print('--seed= Seed (integer) for the RNG') print('--stochastic Use stochastic variational inference') sys.exit(2) kwargs = {} for opt, arg in opts: if opt == "--maxiter": kwargs["maxiter"] = int(arg) elif opt == "--seed": kwargs["seed"] = int(arg) elif opt == "--documents": kwargs["n_documents"] = int(arg) elif opt == "--topics": kwargs["n_topics"] = int(arg) elif opt == "--vocabulary": kwargs["n_vocabulary"] = int(arg) elif opt == "--words": kwargs["n_words"] = int(arg) elif opt == "--stochastic": kwargs["stochastic"] = True #raise NotImplementedError("Work in progress.. This demo is not yet finished") run(**kwargs) bpplt.pyplot.show() bayespy-0.4.0/bayespy/demos/mog.py0000644000175000017500000000604612533543154020637 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2012 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import numpy as np import matplotlib.pyplot as plt import time from bayespy.utils import misc import bayespy.plot as bpplt from bayespy.inference.vmp import nodes from bayespy.inference.vmp.vmp import VB def gaussianmix_model(N, K, D, covariance='full'): # N = number of data vectors # K = number of clusters # D = dimensionality # Construct the Gaussian mixture model # K prior weights (for components) alpha = nodes.Dirichlet(1e-3*np.ones(K), name='alpha') # N K-dimensional cluster assignments (for data) z = nodes.Categorical(alpha, plates=(N,), name='z') # K D-dimensional component means X = nodes.GaussianARD(0, 1e-3, shape=(D,), plates=(K,), name='X') if covariance.lower() == 'full': # K D-dimensional component covariances Lambda = nodes.Wishart(D, 0.01*np.identity(D), plates=(K,), name='Lambda') # N D-dimensional observation vectors Y = nodes.Mixture(z, nodes.Gaussian, X, Lambda, plates=(N,), name='Y') elif covariance.lower() == 'diagonal': # Inverse variances Lambda = nodes.Gamma(1e-3, 1e-3, plates=(K, D), name='Lambda') # N D-dimensional observation vectors Y = nodes.Mixture(z, nodes.GaussianARD, X, Lambda, plates=(N,), name='Y') elif covariance.lower() == 'isotropic': # Inverse variances Lambda = nodes.Gamma(1e-3, 1e-3, plates=(K, 1), name='Lambda') # N D-dimensional observation vectors Y = nodes.Mixture(z, nodes.GaussianARD, X, Lambda, plates=(N,), name='Y') z.initialize_from_random() return VB(Y, X, Lambda, z, alpha) @bpplt.interactive def run(N=50, K=5, D=2): # Generate data N1 = np.floor(0.5*N) N2 = N - N1 y = np.vstack([np.random.normal(0, 0.5, size=(N1,D)), np.random.normal(10, 0.5, size=(N2,D))]) # Construct model Q = gaussianmix_model(N,K,D) # Observe data Q['Y'].observe(y) # Run inference Q.update(repeat=30) # Run predictive model zh = nodes.Categorical(Q['alpha'], name='zh') Yh = nodes.Mixture(zh, nodes.Gaussian, Q['X'], Q['Lambda'], name='Yh') zh.update() # Plot predictive pdf N1 = 400 N2 = 400 x1 = np.linspace(-3, 15, N1) x2 = np.linspace(-3, 15, N2) xh = misc.grid(x1, x2) lpdf = Yh.integrated_logpdf_from_parents(xh, 0) pdf = np.reshape(np.exp(lpdf), (N2,N1)) plt.clf() plt.contourf(x1, x2, pdf, 100) plt.scatter(y[:,0], y[:,1]) print('integrated pdf:', np.sum(pdf)*(18*18)/(N1*N2)) #Q['X'].show() #Q['alpha'].show() if __name__ == '__main__': run() plt.show() bayespy-0.4.0/bayespy/inference/0000755000175000017500000000000012615647103020324 5ustar jluttinejluttine00000000000000bayespy-0.4.0/bayespy/inference/vmp/0000755000175000017500000000000012615647103021126 5ustar jluttinejluttine00000000000000bayespy-0.4.0/bayespy/inference/vmp/tests/0000755000175000017500000000000012615647103022270 5ustar jluttinejluttine00000000000000bayespy-0.4.0/bayespy/inference/vmp/tests/test_transformations.py0000644000175000017500000012225212615643655027146 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for `transformations` module. """ import numpy as np from bayespy.inference.vmp.nodes.gaussian import GaussianARD from bayespy.inference.vmp.nodes.gaussian import Gaussian from bayespy.inference.vmp.nodes.gamma import Gamma from bayespy.inference.vmp.nodes.wishart import Wishart from bayespy.inference.vmp.nodes.dot import SumMultiply from bayespy.inference.vmp.nodes.gaussian_markov_chain import GaussianMarkovChain from bayespy.utils import linalg from bayespy.utils import random from bayespy.utils import optimize from ..transformations import RotateGaussianARD from ..transformations import RotateGaussianMarkovChain from ..transformations import RotateVaryingMarkovChain from bayespy.utils.misc import TestCase class TestRotateGaussianARD(TestCase): def test_cost_function(self): """ Test the speed-up rotation of Gaussian ARD arrays. """ # Use seed for deterministic testing np.random.seed(42) def test(shape, plates, axis=-1, alpha_plates=None, plate_axis=None, mu=3): if plate_axis is not None: precomputes = [False, True] else: precomputes = [False] for precompute in precomputes: # Construct the model D = shape[axis] if alpha_plates is not None: alpha = Gamma(2, 2, plates=alpha_plates) alpha.initialize_from_random() else: alpha = 2 X = GaussianARD(mu, alpha, shape=shape, plates=plates) # Some initial learning and rotator constructing X.initialize_from_random() Y = GaussianARD(X, 1) Y.observe(np.random.randn(*(Y.get_shape(0)))) X.update() if alpha_plates is not None: alpha.update() true_cost0_alpha = alpha.lower_bound_contribution() rotX = RotateGaussianARD(X, alpha, axis=axis, precompute=precompute) else: rotX = RotateGaussianARD(X, axis=axis, precompute=precompute) true_cost0_X = X.lower_bound_contribution() # Rotation matrices I = np.identity(D) R = np.random.randn(D, D) if plate_axis is not None: C = plates[plate_axis] Q = np.random.randn(C, C) Ic = np.identity(C) else: Q = None Ic = None # Compute bound terms rotX.setup(plate_axis=plate_axis) rot_cost0 = rotX.get_bound_terms(I, Q=Ic) rot_cost1 = rotX.get_bound_terms(R, Q=Q) self.assertAllClose(sum(rot_cost0.values()), rotX.bound(I, Q=Ic)[0], msg="Bound terms and total bound differ") self.assertAllClose(sum(rot_cost1.values()), rotX.bound(R, Q=Q)[0], msg="Bound terms and total bound differ") # Perform rotation rotX.rotate(R, Q=Q) # Check bound terms true_cost1_X = X.lower_bound_contribution() self.assertAllClose(true_cost1_X - true_cost0_X, rot_cost1[X] - rot_cost0[X], msg="Incorrect rotation cost for X") if alpha_plates is not None: true_cost1_alpha = alpha.lower_bound_contribution() self.assertAllClose(true_cost1_alpha - true_cost0_alpha, rot_cost1[alpha] - rot_cost0[alpha], msg="Incorrect rotation cost for alpha") return # Rotating a vector (zero mu) test( (3,), (), axis=-1, mu=0) test( (3,), (), axis=-1, alpha_plates=(1,), mu=0) test( (3,), (), axis=-1, alpha_plates=(3,), mu=0) test( (3,), (2,4), axis=-1, mu=0) test( (3,), (2,4), axis=-1, alpha_plates=(1,), mu=0) test( (3,), (2,4), axis=-1, alpha_plates=(3,), mu=0) test( (3,), (2,4), axis=-1, alpha_plates=(2,4,3), mu=0) test( (3,), (2,4), axis=-1, alpha_plates=(1,4,3), mu=0) # Rotating a vector (full mu) test( (3,), (), axis=-1, mu=3*np.ones((3,))) test( (3,), (), axis=-1, alpha_plates=(), mu=3*np.ones((3,))) test( (3,), (), axis=-1, alpha_plates=(1,), mu=3*np.ones((3,))) test( (3,), (), axis=-1, alpha_plates=(3,), mu=3*np.ones((3,))) test( (3,), (2,4), axis=-1, mu=3*np.ones((2,4,3))) test( (3,), (2,4), axis=-1, alpha_plates=(1,), mu=3*np.ones((2,4,3))) test( (3,), (2,4), axis=-1, alpha_plates=(3,), mu=3*np.ones((2,4,3))) test( (3,), (2,4), axis=-1, alpha_plates=(2,4,3), mu=3*np.ones((2,4,3))) test( (3,), (2,4), axis=-1, alpha_plates=(1,4,3), mu=3*np.ones((2,4,3))) # Rotating a vector (broadcast mu) test( (3,), (), axis=-1, mu=3*np.ones((1,))) test( (3,), (), axis=-1, alpha_plates=(1,), mu=3*np.ones((1,))) test( (3,), (), axis=-1, alpha_plates=(3,), mu=3*np.ones((1,))) test( (3,), (2,4,5), axis=-1, mu=3*np.ones((4,1,1))) test( (3,), (2,4,5), axis=-1, alpha_plates=(1,), mu=3*np.ones((4,1,1))) test( (3,), (2,4,5), axis=-1, alpha_plates=(3,), mu=3*np.ones((4,1,1))) test( (3,), (2,4,5), axis=-1, alpha_plates=(2,4,5,3), mu=3*np.ones((4,1,1))) #!! test( (3,), (2,4,5), axis=-1, alpha_plates=(1,4,5,3), mu=3*np.ones((4,1,1))) # Rotating an array test( (2,3,4), (), axis=-1) test( (2,3,4), (), axis=-2) test( (2,3,4), (), axis=-3) test( (2,3,4), (5,6), axis=-1) test( (2,3,4), (5,6), axis=-2) test( (2,3,4), (5,6), axis=-3) test( (2,3,4), (5,6), axis=-1, alpha_plates=(3,1)) test( (2,3,4), (5,6), axis=-2, alpha_plates=(3,1)) test( (2,3,4), (5,6), axis=-3, alpha_plates=(3,1)) test( (2,3), (4,5,6), axis=-1, alpha_plates=(5,1,2,1)) test( (2,3), (4,5,6), axis=-2, alpha_plates=(5,1,2,1)) # Test mu array broadcasting test( (2,3,4), (5,6,7), axis=-2, mu=GaussianARD(3, 1, shape=(3,1), plates=(6,1,1))) test( (2,3,4), (5,6,7), axis=-3, mu=GaussianARD(3, 1, shape=(3,1), plates=(6,1,1))) test( (2,3,4), (5,6,7), axis=-2, alpha_plates=(5,1,7,2,1,1), mu=GaussianARD(3, 1, shape=(3,1), plates=(6,1,1))) # Plate rotation test( (3,), (5,), axis=-1, plate_axis=-1) test( (3,), (4,5,6), axis=-1, plate_axis=-2) test( (2,3), (4,5,6), axis=-2, plate_axis=-2) test( (2,3,4), (5,6,7), axis=-2, plate_axis=-3) # Plate rotation with alpha test( (2,3,4), (5,6,7), axis=-2, alpha_plates=(3,1), plate_axis=-2) test( (2,3,4), (5,6,7), axis=-2, alpha_plates=(6,1,2,1,4), plate_axis=-3) # Plate rotation with mu test( (2,3,4), (5,6,7), axis=-2, plate_axis=-2, mu=GaussianARD(3, 1, shape=(3,1), plates=(6,1,1))) test( (2,3,4), (5,6,7), axis=-3, plate_axis=-2, mu=GaussianARD(3, 1, shape=(3,1), plates=(6,1,1))) test( (2,3,4), (5,6,7), axis=-2, alpha_plates=(5,1,7,2,1,1), plate_axis=-2, mu=GaussianARD(3, 1, shape=(3,1), plates=(6,1,1))) # # Plate rotation with mu and alpha # # Basic, matching sizes test( (3,), (4,), axis=-1, plate_axis=-1, alpha_plates=(4,3), mu=GaussianARD(3, 1, shape=(3,), plates=(4,))) # Broadcast for mu test( (3,), (4,), axis=-1, plate_axis=-1, alpha_plates=(4,3), mu=GaussianARD(3, 1, shape=(1,), plates=(4,))) test( (3,), (4,), axis=-1, plate_axis=-1, alpha_plates=(4,3), mu=GaussianARD(3, 1, shape=(), plates=(1,))) test( (3,), (4,), axis=-1, plate_axis=-1, alpha_plates=(4,3), mu=GaussianARD(3, 1, shape=(3,), plates=(1,))) # Broadcast for alpha test( (3,), (4,), axis=-1, plate_axis=-1, alpha_plates=(4,1), mu=GaussianARD(3, 1, shape=(3,), plates=(4,))) test( (3,), (4,), axis=-1, plate_axis=-1, alpha_plates=(3,), mu=GaussianARD(3, 1, shape=(3,), plates=(4,))) # Several variable dimensions test( (3,4,5), (2,), axis=-2, plate_axis=-1, alpha_plates=(2,3,4,5), mu=GaussianARD(3, 1, shape=(3,4,5), plates=(2,))) test( (3,4,5), (2,), axis=-2, plate_axis=-1, alpha_plates=(2,3,1,5), mu=GaussianARD(3, 1, shape=(4,1), plates=(2,1))) # Several plate dimensions test( (5,), (2,3,4), axis=-1, plate_axis=-2, alpha_plates=(2,3,4,5), mu=GaussianARD(3, 1, shape=(5,), plates=(2,3,4))) # Several plate dimensions, rotated plate broadcasted in alpha test( (5,), (2,3,4), axis=-1, plate_axis=-2, alpha_plates=(2,1,4,5), mu=GaussianARD(3, 1, shape=(5,), plates=(2,3,4))) test( (5,), (2,3,4), axis=-1, plate_axis=-2, alpha_plates=(4,5), mu=GaussianARD(3, 1, shape=(5,), plates=(2,3,4))) # Several plate dimensions, rotated plate broadcasted in mu test( (5,), (2,3,4), axis=-1, plate_axis=-2, alpha_plates=(2,3,4,5), mu=GaussianARD(3, 1, shape=(5,), plates=(2,1,4))) test( (5,), (2,3,4), axis=-1, plate_axis=-2, alpha_plates=(2,3,4,5), mu=GaussianARD(3, 1, shape=(5,), plates=(4,))) # Several plate dimensions, rotated plate broadcasted in mu and alpha test( (5,), (2,3,4), axis=-1, plate_axis=-2, alpha_plates=(2,1,4,5), mu=GaussianARD(3, 1, shape=(5,), plates=(2,1,4))) test( (5,), (2,3,4), axis=-1, plate_axis=-2, alpha_plates=(4,5), mu=GaussianARD(3, 1, shape=(5,), plates=(4,))) # TODO: Missing values pass def test_cost_gradient(self): """ Test gradient of the rotation cost function for Gaussian ARD arrays. """ # Use seed for deterministic testing np.random.seed(42) def test(shape, plates, axis=-1, alpha_plates=None, plate_axis=None, mu=3): if plate_axis is not None: precomputes = [False, True] else: precomputes = [False] for precompute in precomputes: # Construct the model D = shape[axis] if alpha_plates is not None: alpha = Gamma(3, 5, plates=alpha_plates) alpha.initialize_from_random() else: alpha = 2 X = GaussianARD(mu, alpha, shape=shape, plates=plates) # Some initial learning and rotator constructing X.initialize_from_random() Y = GaussianARD(X, 1) Y.observe(np.random.randn(*(Y.get_shape(0)))) X.update() if alpha_plates is not None: alpha.update() rotX = RotateGaussianARD(X, alpha, axis=axis, precompute=precompute) else: rotX = RotateGaussianARD(X, axis=axis, precompute=precompute) try: mu.update() except: pass # Rotation matrices R = np.random.randn(D, D) if plate_axis is not None: C = plates[plate_axis] Q = np.random.randn(C, C) else: Q = None # Compute bound terms rotX.setup(plate_axis=plate_axis) if plate_axis is None: def f_r(r): (b, dr) = rotX.bound(np.reshape(r, np.shape(R))) return (b, np.ravel(dr)) else: def f_r(r): (b, dr, dq) = rotX.bound(np.reshape(r, np.shape(R)), Q=Q) return (b, np.ravel(dr)) def f_q(q): (b, dr, dq) = rotX.bound(R, Q=np.reshape(q, np.shape(Q))) return (b, np.ravel(dq)) # Check gradient with respect to R err = optimize.check_gradient(f_r, np.ravel(R), verbose=False)[1] self.assertAllClose(err, 0, atol=1e-4, msg="Gradient incorrect for R") # Check gradient with respect to Q if plate_axis is not None: err = optimize.check_gradient(f_q, np.ravel(Q), verbose=False)[1] self.assertAllClose(err, 0, atol=1e-4, msg="Gradient incorrect for Q") return # # Basic rotation # test((3,), (), axis=-1) test((2,3,4), (), axis=-1) test((2,3,4), (), axis=-2) test((2,3,4), (), axis=-3) test((2,3,4), (5,6), axis=-2) # # Rotation with mu # # Simple test((1,), (), axis=-1, mu=GaussianARD(2, 4, shape=(1,), plates=())) test((3,), (), axis=-1, mu=GaussianARD(2, 4, shape=(3,), plates=())) # Broadcast mu over rotated dim test((3,), (), axis=-1, mu=GaussianARD(2, 4, shape=(1,), plates=())) test((3,), (), axis=-1, mu=GaussianARD(2, 4, shape=(), plates=())) # Broadcast mu over dim when multiple dims test((2,3), (), axis=-1, mu=GaussianARD(2, 4, shape=(1,3), plates=())) test((2,3), (), axis=-1, mu=GaussianARD(2, 4, shape=(3,), plates=())) # Broadcast mu over rotated dim when multiple dims test((2,3), (), axis=-2, mu=GaussianARD(2, 4, shape=(1,3), plates=())) test((2,3), (), axis=-2, mu=GaussianARD(2, 4, shape=(3,), plates=())) # Broadcast mu over plates test((3,), (4,5), axis=-1, mu=GaussianARD(2, 4, shape=(3,), plates=(4,1))) test((3,), (4,5), axis=-1, mu=GaussianARD(2, 4, shape=(3,), plates=(5,))) # # Rotation with alpha # # Simple test((1,), (), axis=-1, alpha_plates=()) test((3,), (), axis=-1, alpha_plates=(3,)) # Broadcast alpha over rotated dim test((3,), (), axis=-1, alpha_plates=()) test((3,), (), axis=-1, alpha_plates=(1,)) # Broadcast alpha over dim when multiple dims test((2,3), (), axis=-1, alpha_plates=(1,3)) test((2,3), (), axis=-1, alpha_plates=(3,)) # Broadcast alpha over rotated dim when multiple dims test((2,3), (), axis=-2, alpha_plates=(1,3)) test((2,3), (), axis=-2, alpha_plates=(3,)) # Broadcast alpha over plates test((3,), (4,5), axis=-1, alpha_plates=(4,1,3)) test((3,), (4,5), axis=-1, alpha_plates=(5,3)) # # Rotation with alpha and mu # # Simple test((1,), (), axis=-1, alpha_plates=(1,), mu=GaussianARD(2, 4, shape=(1,), plates=())) test((3,), (), axis=-1, alpha_plates=(3,), mu=GaussianARD(2, 4, shape=(3,), plates=())) # Broadcast mu over rotated dim test((3,), (), axis=-1, alpha_plates=(3,), mu=GaussianARD(2, 4, shape=(1,), plates=())) test((3,), (), axis=-1, alpha_plates=(3,), mu=GaussianARD(2, 4, shape=(), plates=())) # Broadcast alpha over rotated dim test((3,), (), axis=-1, alpha_plates=(1,), mu=GaussianARD(2, 4, shape=(3,), plates=())) test((3,), (), axis=-1, alpha_plates=(), mu=GaussianARD(2, 4, shape=(3,), plates=())) # Broadcast both mu and alpha over rotated dim test((3,), (), axis=-1, alpha_plates=(1,), mu=GaussianARD(2, 4, shape=(1,), plates=())) test((3,), (), axis=-1, alpha_plates=(), mu=GaussianARD(2, 4, shape=(), plates=())) # Broadcast mu over plates test((3,), (4,5), axis=-1, alpha_plates=(4,5,3), mu=GaussianARD(2, 4, shape=(3,), plates=(4,1))) test((3,), (4,5), axis=-1, alpha_plates=(4,5,3), mu=GaussianARD(2, 4, shape=(3,), plates=(5,))) # Broadcast alpha over plates test((3,), (4,5), axis=-1, alpha_plates=(4,1,3), mu=GaussianARD(2, 4, shape=(3,), plates=(4,5))) test((3,), (4,5), axis=-1, alpha_plates=(5,3), mu=GaussianARD(2, 4, shape=(3,), plates=(4,5))) # Broadcast both mu and alpha over plates test((3,), (4,5), axis=-1, alpha_plates=(4,1,3), mu=GaussianARD(2, 4, shape=(3,), plates=(4,1))) test((3,), (4,5), axis=-1, alpha_plates=(5,3), mu=GaussianARD(2, 4, shape=(3,), plates=(5,))) # Broadcast both mu and alpha over plates but different plates test((3,), (4,5), axis=-1, alpha_plates=(4,1,3), mu=GaussianARD(2, 4, shape=(3,), plates=(5,))) test((3,), (4,5), axis=-1, alpha_plates=(5,3), mu=GaussianARD(2, 4, shape=(3,), plates=(4,1))) # # Rotation with missing values # # TODO # # Plate rotation # # Simple test((2,), (3,), axis=-1, plate_axis=-1) test((2,), (3,4,5), axis=-1, plate_axis=-1) test((2,), (3,4,5), axis=-1, plate_axis=-2) test((2,), (3,4,5), axis=-1, plate_axis=-3) test((2,3), (4,5), axis=-2, plate_axis=-2) # With mu test((2,), (3,), axis=-1, plate_axis=-1, mu=GaussianARD(3, 4, shape=(2,), plates=(3,))) # With mu broadcasted test((2,), (3,), axis=-1, plate_axis=-1, mu=GaussianARD(3, 4, shape=(2,), plates=(1,))) test((2,), (3,), axis=-1, plate_axis=-1, mu=GaussianARD(3, 4, shape=(2,), plates=())) # With mu multiple plates test((2,), (3,4,5), axis=-1, plate_axis=-2, mu=GaussianARD(3, 4, shape=(2,), plates=(3,4,5))) # With mu multiple dims test((2,3,4), (5,), axis=-2, plate_axis=-1, mu=GaussianARD(3, 4, shape=(2,3,4), plates=(5,))) # # With alpha # print("Test: Plate rotation with alpha. Scalars.") test((1,), (1,), axis=-1, plate_axis=-1, alpha_plates=(1,1), mu=0) print("Test: Plate rotation with alpha. Plates.") test((1,), (3,), axis=-1, plate_axis=-1, alpha_plates=(3,1), mu=0) print("Test: Plate rotation with alpha. Dims.") test((3,), (1,), axis=-1, plate_axis=-1, alpha_plates=(1,3), mu=0) print("Test: Plate rotation with alpha. Broadcast alpha over rotated plates.") test((1,), (3,), axis=-1, plate_axis=-1, alpha_plates=(1,1), mu=0) test((1,), (3,), axis=-1, plate_axis=-1, alpha_plates=(1,), mu=0) print("Test: Plate rotation with alpha. Broadcast alpha over dims.") test((3,), (1,), axis=-1, plate_axis=-1, alpha_plates=(1,1), mu=0) test((3,), (1,), axis=-1, plate_axis=-1, alpha_plates=(), mu=0) print("Test: Plate rotation with alpha. Multiple dims.") test((2,3,4,5), (6,), axis=-2, plate_axis=-1, alpha_plates=(6,2,3,4,5), mu=0) print("Test: Plate rotation with alpha. Multiple plates.") test((2,), (3,4,5), axis=-1, plate_axis=-1, alpha_plates=(3,4,5,2), mu=0) test((2,), (3,4,5), axis=-1, plate_axis=-2, alpha_plates=(3,4,5,2), mu=0) test((2,), (3,4,5), axis=-1, plate_axis=-3, alpha_plates=(3,4,5,2), mu=0) # # With alpha and mu # print("Test: Plate rotation with alpha and mu. Scalars.") test((1,), (1,), axis=-1, plate_axis=-1, alpha_plates=(1,1), mu=GaussianARD(2, 3, shape=(1,), plates=(1,))) print("Test: Plate rotation with alpha and mu. Plates.") test((1,), (3,), axis=-1, plate_axis=-1, alpha_plates=(3,1), mu=GaussianARD(2, 3, shape=(1,), plates=(3,))) print("Test: Plate rotation with alpha and mu. Dims.") test((3,), (1,), axis=-1, plate_axis=-1, alpha_plates=(1,3), mu=GaussianARD(2, 3, shape=(3,), plates=(1,))) print("Test: Plate rotation with alpha and mu. Broadcast over rotated " "plates.") test((1,), (3,), axis=-1, plate_axis=-1, alpha_plates=(1,1), mu=GaussianARD(2, 3, shape=(1,), plates=(1,))) test((1,), (3,), axis=-1, plate_axis=-1, alpha_plates=(1,), mu=GaussianARD(2, 3, shape=(1,), plates=())) print("Test: Plate rotation with alpha and mu. Broadcast over dims.") test((3,), (1,), axis=-1, plate_axis=-1, alpha_plates=(1,1), mu=GaussianARD(2, 3, shape=(1,), plates=(1,))) test((3,), (1,), axis=-1, plate_axis=-1, alpha_plates=(), mu=GaussianARD(2, 3, shape=(), plates=(1,))) print("Test: Plate rotation with alpha and mu. Multiple dims.") test((2,3,4,5), (6,), axis=-2, plate_axis=-1, alpha_plates=(6,2,3,4,5), mu=GaussianARD(2, 3, shape=(2,3,4,5), plates=(6,))) print("Test: Plate rotation with alpha and mu. Multiple plates.") test((2,), (3,4,5), axis=-1, plate_axis=-1, alpha_plates=(3,4,5,2), mu=GaussianARD(2, 3, shape=(2,), plates=(3,4,5,))) test((2,), (3,4,5), axis=-1, plate_axis=-2, alpha_plates=(3,4,5,2), mu=GaussianARD(2, 3, shape=(2,), plates=(3,4,5,))) test((2,), (3,4,5), axis=-1, plate_axis=-3, alpha_plates=(3,4,5,2), mu=GaussianARD(2, 3, shape=(2,), plates=(3,4,5,))) # TODO: With missing values pass class TestRotateGaussianMarkovChain(TestCase): def test_cost_function(self): """ Test the cost function of the speed-up rotation for Markov chain """ np.random.seed(42) def check(D, N, mu=None, Lambda=None, rho=None, A=None): if mu is None: mu = np.zeros(D) if Lambda is None: Lambda = np.identity(D) if rho is None: rho = np.ones(D) if A is None: A = GaussianARD(3, 5, shape=(D,), plates=(D,)) V = np.identity(D) + np.ones((D,D)) # Construct model X = GaussianMarkovChain(mu, Lambda, A, rho, n=N+1, initialize=False) Y = Gaussian(X, V, initialize=False) # Posterior estimation Y.observe(np.random.randn(*(Y.get_shape(0)))) X.update() try: A.update() except: pass try: mu.update() except: pass try: Lambda.update() except: pass try: rho.update() except: pass # Construct rotator rotA = RotateGaussianARD(A, axis=-1) rotX = RotateGaussianMarkovChain(X, rotA) # Rotation true_cost0 = X.lower_bound_contribution() rotX.setup() I = np.identity(D) R = np.random.randn(D, D) rot_cost0 = rotX.get_bound_terms(I) rot_cost1 = rotX.get_bound_terms(R) self.assertAllClose(sum(rot_cost0.values()), rotX.bound(I)[0], msg="Bound terms and total bound differ") self.assertAllClose(sum(rot_cost1.values()), rotX.bound(R)[0], msg="Bound terms and total bound differ") rotX.rotate(R) true_cost1 = X.lower_bound_contribution() self.assertAllClose(true_cost1 - true_cost0, rot_cost1[X] - rot_cost0[X], msg="Incorrect rotation cost for X") return self._run_checks(check) pass def test_cost_gradient(self): """ Test the gradient of the speed-up rotation for Markov chain """ # Use seed for deterministic testing np.random.seed(42) def check(D, N, mu=None, Lambda=None, rho=None, A=None): if mu is None: mu = np.zeros(D) if Lambda is None: Lambda = np.identity(D) if rho is None: rho = np.ones(D) if A is None: A = GaussianARD(3, 5, shape=(D,), plates=(D,)) V = np.identity(D) + np.ones((D,D)) # Construct model X = GaussianMarkovChain(mu, Lambda, A, rho, n=N+1, initialize=False) Y = Gaussian(X, V, initialize=False) # Posterior estimation Y.observe(np.random.randn(*(Y.get_shape(0)))) X.update() try: A.update() except: pass try: mu.update() except: pass try: Lambda.update() except: pass try: rho.update() except: pass # Construct rotator rotA = RotateGaussianARD(A, axis=-1) rotX = RotateGaussianMarkovChain(X, rotA) rotX.setup() # Check gradient with respect to R R = np.random.randn(D, D) def cost(r): (b, dr) = rotX.bound(np.reshape(r, np.shape(R))) return (b, np.ravel(dr)) err = optimize.check_gradient(cost, np.ravel(R), verbose=False)[1] self.assertAllClose(err, 0, atol=1e-5, msg="Gradient incorrect") return self._run_checks(check) pass def _run_checks(self, check): # Basic test check(2, 3) # Test mu check(2, 3, mu=GaussianARD(2, 4, shape=(2,), plates=())) check(2, 3, mu=GaussianARD(2, 4, shape=(2,), plates=(5,))) # Test Lambda check(2, 3, Lambda=Wishart(3, random.covariance(2))) check(2, 3, Lambda=Wishart(3, random.covariance(2), plates=(5,))) # Test A check(2, 3, A=GaussianARD(2, 4, shape=(2,), plates=(2,))) check(2, 3, A=GaussianARD(2, 4, shape=(2,), plates=(3,2))) check(2, 3, A=GaussianARD(2, 4, shape=(2,), plates=(5,3,2))) # Test Lambda and mu check(2, 3, mu=GaussianARD(2, 4, shape=(2,), plates=()), Lambda=Wishart(2, random.covariance(2))) check(2, 3, mu=GaussianARD(2, 4, shape=(2,), plates=(5,)), Lambda=Wishart(2, random.covariance(2), plates=(5,))) # Test mu and A check(2, 3, mu=GaussianARD(2, 4, shape=(2,), plates=()), A=GaussianARD(2, 4, shape=(2,), plates=(2,))) check(2, 3, mu=GaussianARD(2, 4, shape=(2,), plates=(5,)), A=GaussianARD(2, 4, shape=(2,), plates=(5,1,2,))) # Test Lambda and A check(2, 3, Lambda=Wishart(2, random.covariance(2)), A=GaussianARD(2, 4, shape=(2,), plates=(2,))) check(2, 3, Lambda=Wishart(2, random.covariance(2), plates=(5,)), A=GaussianARD(2, 4, shape=(2,), plates=(5,1,2,))) # Test mu, Lambda and A check(2, 3, mu=GaussianARD(2, 4, shape=(2,), plates=()), Lambda=Wishart(2, random.covariance(2)), A=GaussianARD(2, 4, shape=(2,), plates=(2,))) check(2, 3, mu=GaussianARD(2, 4, shape=(2,), plates=(5,)), Lambda=Wishart(2, random.covariance(2), plates=(5,)), A=GaussianARD(2, 4, shape=(2,), plates=(5,1,2,))) pass class TestRotateVaryingMarkovChain(TestCase): def test_cost_function(self): """ Test the speed-up rotation of Markov chain with time-varying dynamics """ # Use seed for deterministic testing np.random.seed(42) def check(D, N, K, mu=None, Lambda=None, rho=None): if mu is None: mu = np.zeros(D) if Lambda is None: Lambda = np.identity(D) if rho is None: rho = np.ones(D) V = np.identity(D) + np.ones((D,D)) # Construct model B = GaussianARD(3, 5, shape=(D,K), plates=(1,D)) S = GaussianARD(2, 4, shape=(K,), plates=(N,1)) A = SumMultiply('dk,k->d', B, S) X = GaussianMarkovChain(mu, Lambda, A, rho, n=N+1, initialize=False) Y = Gaussian(X, V, initialize=False) # Posterior estimation Y.observe(np.random.randn(N+1,D)) X.update() B.update() S.update() try: mu.update() except: pass try: Lambda.update() except: pass try: rho.update() except: pass # Construct rotator rotB = RotateGaussianARD(B, axis=-2) rotX = RotateVaryingMarkovChain(X, B, S, rotB) # Rotation true_cost0 = X.lower_bound_contribution() rotX.setup() I = np.identity(D) R = np.random.randn(D, D) rot_cost0 = rotX.get_bound_terms(I) rot_cost1 = rotX.get_bound_terms(R) self.assertAllClose(sum(rot_cost0.values()), rotX.bound(I)[0], msg="Bound terms and total bound differ") self.assertAllClose(sum(rot_cost1.values()), rotX.bound(R)[0], msg="Bound terms and total bound differ") rotX.rotate(R) true_cost1 = X.lower_bound_contribution() self.assertAllClose(true_cost1 - true_cost0, rot_cost1[X] - rot_cost0[X], msg="Incorrect rotation cost for X") return self._run_checks(check) pass def test_cost_gradient(self): """ Test the gradient of the rotation for MC with time-varying dynamics """ # Use seed for deterministic testing np.random.seed(42) def check(D, N, K, mu=None, Lambda=None, rho=None): if mu is None: mu = np.zeros(D) if Lambda is None: Lambda = np.identity(D) if rho is None: rho = np.ones(D) V = np.identity(D) + np.ones((D,D)) # Construct model B = GaussianARD(3, 5, shape=(D,K), plates=(1,D)) S = GaussianARD(2, 4, shape=(K,), plates=(N,1)) A = SumMultiply('dk,k->d', B, S) X = GaussianMarkovChain(mu, Lambda, A, rho, n=N+1, initialize=False) Y = Gaussian(X, V, initialize=False) # Posterior estimation Y.observe(np.random.randn(N+1,D)) X.update() B.update() S.update() try: mu.update() except: pass try: Lambda.update() except: pass try: rho.update() except: pass # Construct rotator rotB = RotateGaussianARD(B, axis=-2) rotX = RotateVaryingMarkovChain(X, B, S, rotB) rotX.setup() # Check gradient with respect to R R = np.random.randn(D, D) def cost(r): (b, dr) = rotX.bound(np.reshape(r, np.shape(R))) return (b, np.ravel(dr)) err = optimize.check_gradient(cost, np.ravel(R), verbose=False)[1] self.assertAllClose(err, 0, atol=1e-6, msg="Gradient incorrect") return self._run_checks(check) pass def _run_checks(self, check): # Basic test check(1, 1, 1) check(2, 1, 1) check(1, 2, 1) check(1, 1, 2) check(3, 4, 2) # Test mu check(2, 3, 4, mu=GaussianARD(2, 4, shape=(2,), plates=())) # Test Lambda check(2, 3, 4, Lambda=Wishart(3, random.covariance(2))) # Test Lambda and mu check(2, 3, 4, mu=GaussianARD(2, 4, shape=(2,), plates=()), Lambda=Wishart(2, random.covariance(2))) # TODO: Test plates pass bayespy-0.4.0/bayespy/inference/vmp/tests/test_annealing.py0000644000175000017500000000621612615643655025652 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2015 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for `vmp` module. """ import numpy as np from scipy import special from numpy import testing from bayespy.nodes import (Gaussian, GaussianARD, GaussianGammaISO, Gamma, Wishart) from ..vmp import VB from bayespy.utils import misc from bayespy.utils import linalg from bayespy.utils import random from bayespy.utils.misc import TestCase class TestVB(TestCase): def test_annealing(self): X = GaussianARD(3, 4) X.initialize_from_parameters(-1, 6) Q = VB(X) Q.set_annealing(0.1) # # Check that the gradient is correct # # Initial parameters phi0 = X.phi # Gradient rg = X.get_riemannian_gradient() g = X.get_gradient(rg) # Numerical gradient of the first parameter eps = 1e-6 p0 = X.get_parameters() l0 = Q.compute_lowerbound(ignore_masked=False) g_num = [(), ()] e = eps p1 = p0[0] + e X.set_parameters([p1, p0[1]]) l1 = Q.compute_lowerbound(ignore_masked=False) g_num[0] = (l1 - l0) / eps # Numerical gradient of the second parameter p1 = p0[1] + e X.set_parameters([p0[0], p1]) l1 = Q.compute_lowerbound(ignore_masked=False) g_num[1] = (l1 - l0) / (eps) # Check self.assertAllClose(g[0], g_num[0]) self.assertAllClose(g[1], g_num[1]) # # Gradient should be zero after updating # X.update() # Initial parameters phi0 = X.phi # Numerical gradient of the first parameter eps = 1e-8 p0 = X.get_parameters() l0 = Q.compute_lowerbound(ignore_masked=False) g_num = [(), ()] e = eps p1 = p0[0] + e X.set_parameters([p1, p0[1]]) l1 = Q.compute_lowerbound(ignore_masked=False) g_num[0] = (l1 - l0) / eps # Numerical gradient of the second parameter p1 = p0[1] + e X.set_parameters([p0[0], p1]) l1 = Q.compute_lowerbound(ignore_masked=False) g_num[1] = (l1 - l0) / (eps) # Check self.assertAllClose(0, g_num[0], atol=1e-5) self.assertAllClose(0, g_num[1], atol=1e-5) # Not at the optimum X.initialize_from_parameters(-1, 6) # Initial parameters phi0 = X.phi # Gradient g = X.get_riemannian_gradient() # Parameters after VB-EM update X.update() phi1 = X.phi # Check self.assertAllClose(g[0], phi1[0] - phi0[0]) self.assertAllClose(g[1], phi1[1] - phi0[1]) pass bayespy-0.4.0/bayespy/inference/vmp/tests/__init__.py0000664000175000017500000000000012174013072024361 0ustar jluttinejluttine00000000000000bayespy-0.4.0/bayespy/inference/vmp/transformations.py0000644000175000017500000015752312615643655024756 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2013-2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import numpy as np import warnings import scipy from bayespy.utils import optimize from bayespy.utils import random from bayespy.utils import linalg from bayespy.utils import misc from bayespy.utils.linalg import dot, tracedot from .nodes import gaussian from .nodes.categorical import CategoricalMoments class RotationOptimizer(): r""" Optimizer for rotation parameter expansion in state-space models Rotates one model block with :math:`\mathbf{R}` and one model block with :math:`\mathbf{R}^{-1}`. Parameters ---------- block1 : rotator object The first rotation parameter expansion object block2 : rotator object The second rotation parameter expansion object D : int Dimensionality of the latent space References ---------- :cite:`Luttinen:2010`, :cite:`Luttinen:2013` """ def __init__(self, block1, block2, D): self.block1 = block1 self.block2 = block2 self.D = D def rotate(self, maxiter=10, check_gradient=False, verbose=False, check_bound=False): """ Optimize the rotation of two separate model blocks jointly. If some variable is the dot product of two Gaussians, rotating the two Gaussians optimally can make the inference algorithm orders of magnitude faster. First block is rotated with :math:`\mathbf{R}` and the second with :math:`\mathbf{R}^{-T}`. Blocks must have methods: `bound(U,s,V)` and `rotate(R)`. """ I = np.identity(self.D) piv = np.arange(self.D) def cost(r): # Make vector-r into matrix-R R = np.reshape(r, (self.D,self.D)) # Compute SVD invR = np.linalg.inv(R) logdetR = np.linalg.slogdet(R)[1] # Compute lower bound terms (b1,db1) = self.block1.bound(R, logdet=logdetR, inv=invR) (b2,db2) = self.block2.bound(invR.T, logdet=-logdetR, inv=R.T) # Apply chain rule for the second gradient: # d b(invR.T) # = tr(db.T * d(invR.T)) # = tr(db * d(invR)) # = -tr(db * invR * (dR) * invR) # = -tr(invR * db * invR * dR) db2 = -dot(invR.T, db2.T, invR.T) # Compute the cost function c = -(b1+b2) dc = -(db1+db2) return (c, np.ravel(dc)) def get_bound_terms(r, gradient=False): """ Returns a dictionary of bound terms for the nodes. """ # Gradient not yet implemented.. if gradient: raise NotImplementedError() # Make vector-r into matrix-R R = np.reshape(r, (self.D,self.D)) # Compute SVD invR = np.linalg.inv(R) logdetR = np.linalg.slogdet(R)[1] # Compute lower bound terms dict1 = self.block1.get_bound_terms(R, logdet=logdetR, inv=invR) dict2 = self.block2.get_bound_terms(invR.T, logdet=-logdetR, inv=R.T) if not gradient: dict1.update(dict2) return dict1 else: terms = dict1[0].copy() terms = terms.update(dict2[0]) grad = dict1[1].copy() grad = grad.update(dict2[1]) return (terms, grad) def get_true_bound_terms(): nodes = set(self.block1.nodes()) | set(self.block2.nodes()) D = {} # TODO/FIXME: Also compute bound for child nodes as they could be # affected in practice although they shouldn't. Just checking that. for node in nodes: L = node.lower_bound_contribution() D[node] = L return D self.block1.setup() self.block2.setup() if check_gradient: R = np.random.randn(self.D, self.D) err = optimize.check_gradient(cost, np.ravel(R), verbose=verbose)[1] if err > 1e-5: warnings.warn("Rotation gradient has relative error %g" % err) # Initial rotation is identity matrix r0 = np.ravel(np.identity(self.D)) (cost_begin, _) = cost(r0) if check_bound: bound_terms_begin = get_bound_terms(r0) true_bound_terms_begin = get_true_bound_terms() # Run optimization r = optimize.minimize(cost, r0, maxiter=maxiter, verbose=verbose) (cost_end, _) = cost(r) if check_bound: bound_terms_end = get_bound_terms(r) # Apply the optimal rotation R = np.reshape(r, (self.D,self.D)) invR = np.linalg.inv(R) logdetR = np.linalg.slogdet(R)[1] self.block1.rotate(R, inv=invR, logdet=logdetR) self.block2.rotate(invR.T, inv=R.T, logdet=-logdetR) # Check that the cost function and the true lower bound changed equally cost_change = cost_end - cost_begin # Check that we really have improved the bound. if cost_change > 0: warnings.warn("Rotation optimization made the cost function worse " "by %g. Probably a bug in the gradient of the " "rotation functions." % (cost_change,)) if check_bound: true_bound_terms_end = get_true_bound_terms() bound_change = 0 for node in bound_terms_begin.keys(): node_bound_change = (bound_terms_end[node] - bound_terms_begin[node]) bound_change += node_bound_change true_node_bound_change = 0 try: true_node_bound_change += (true_bound_terms_end[node] - true_bound_terms_begin[node]) except KeyError: raise Exception("The node %s is part of the " "transformation but not part of the " "model. Check your VB construction." % node.name) if not np.allclose(node_bound_change, true_node_bound_change): warnings.warn("Rotation cost function is not consistent " "with the true lower bound for node %s. " "Bound changed %g but optimized function " "changed %g." % (node.name, true_node_bound_change, node_bound_change)) # Check that we really have improved the bound. # TODO/FIXME: Also compute bound for child nodes as they could be # affected in practice although they shouldn't. Just checking that. if bound_change < 0: warnings.warn("Rotation made the true lower bound worse by %g. " "Probably a bug in the rotation functions." % (bound_change,)) class RotateGaussian(): r""" Rotation parameter expansion for :class:`bayespy.nodes.Gaussian` """ def __init__(self, X): self.X = X def rotate(self, R, inv=None, logdet=None): self.X.rotate(R, inv=inv, logdet=logdet) def setup(self): """ This method should be called just before optimization. """ mask = self.X.mask[...,np.newaxis,np.newaxis] # Number of plates self.N = self.X.plates[0] #np.sum(mask) # Compute the sum over plates self.XX = misc.sum_multiply(self.X.get_moments()[1], mask, axis=(-1,-2), sumaxis=False, keepdims=False) # Parent's moments self.Lambda = self.X.parents[1].get_moments()[0] def _compute_bound(self, R, logdet=None, inv=None, gradient=False): """ Rotate q(X) as X->RX: q(X)=N(R*mu, R*Cov*R') Assume: :math:`p(\mathbf{X}) = \prod^M_{m=1} N(\mathbf{x}_m|0, \mathbf{\Lambda})` """ # TODO/FIXME: X and alpha should NOT contain observed values!! Check # that. # TODO/FIXME: Allow non-zero prior mean! # Assume constant mean and precision matrix over plates.. # Compute rotated moments XX_R = dot(R, self.XX, R.T) inv_R = inv logdet_R = logdet # Compute entropy H(X) logH_X = random.gaussian_entropy(-2*self.N*logdet_R, 0) # Compute logp_X = random.gaussian_logpdf(np.vdot(XX_R, self.Lambda), 0, 0, 0, 0) # Compute the bound if terms: bound = {self.X: bound} else: bound = logp_X + logH_X if not gradient: return bound # Compute dH(X) dlogH_X = random.gaussian_entropy(-2*self.N*inv_R.T, 0) # Compute d dXX = 2*dot(self.Lambda, R, self.XX) dlogp_X = random.gaussian_logpdf(dXX, 0, 0, 0, 0) if terms: d_bound = {self.X: dlogp_X + dlogH_X} else: d_bound = dlogp_X + dlogH_X return (bound, d_bound) def bound(self, R, logdet=None, inv=None): return self._compute_bound(R, logdet=logdet, inv=inv, gradient=True) def get_bound_terms(self, R, logdet=None, inv=None): return self._compute_bound(R, logdet=logdet, inv=inv, gradient=False, terms=True) def nodes(self): return [self.X] def covariance_to_variance(C, ndim=1, covariance_axis=None): # Force None to empty list if covariance_axis is None: covariance_axis = [] # Force a list from integer if isinstance(covariance_axis, int): covariance_axis = [covariance_axis] # Force positive axis indices covariance_axis = [axis + ndim if axis < 0 else axis for axis in covariance_axis] # Make a set of the axes covariance_axis = set(covariance_axis) keys = [i+ndim if i in covariance_axis else i for i in range(ndim)] keys += [i+2*ndim if i in covariance_axis else i for i in range(ndim)] out_keys = sorted(list(set(keys))) return np.einsum(C, [Ellipsis]+keys, [Ellipsis]+out_keys) def sum_to_plates(V, plates_to, plates_from=None, ndim=0): if ndim == 0: if plates_from is not None: r = gaussian.Gaussian.broadcasting_multiplier(plates_from, np.shape(V)) else: r = 1 return r * misc.sum_to_shape(V, plates_to) else: dims_V = np.shape(V)[-ndim:] plates_V = np.shape(V)[:-ndim] shape_to = tuple(plates_to) + dims_V if plates_from is not None: r = gaussian.Gaussian.broadcasting_multiplier(plates_from, plates_V) else: r = 1 return r * misc.sum_to_shape(V, shape_to) class RotateGaussianARD(): """ Rotation parameter expansion for :class:`bayespy.nodes.GaussianARD` The model: alpha ~ N(a, b) X ~ N(mu, alpha) X can be an array (e.g., GaussianARD). Transform q(X) and q(alpha) by rotating X. Requirements: * X and alpha do not contain any observed values """ def __init__(self, X, *alpha, axis=-1, precompute=False, subset=None): """ Precompute tells whether to compute some moments once in the setup function instead of every time in the bound function. However, they are computed a bit differently in the bound function so it can be useful too. Precomputation is probably beneficial only when there are large axes that are not rotated (by R nor Q) and they are not contained in the plates of alpha, and the dimensions for R and Q are quite small. """ self.precompute = precompute self.node_parent = X.parents[0] if len(alpha) == 0: self.update_alpha = False elif len(alpha) == 1: self.node_alpha = alpha[0] self.update_alpha = True else: raise ValueError("Too many arguments") self.node_X = X #self.node_mu = X.parents[0] self.ndim = len(X.dims[0]) # Force negative rotation axis indexing if not isinstance(axis, int): raise ValueError("Axis must be integer") if axis >= 0: axis -= self.ndim if axis < -self.ndim or axis >= 0: raise ValueError("Axis out of bounds") self.axis = axis # Allow rotation of only subset of elements/slices self.D = X.dims[0][axis] if subset is None: #self.subset = np.ones(self.D, dtype=bool) self.subset = None #tuple(range(self.D)) else: #self.subset = tuple(range(self.D)) self.subset = subset #self.subset[subset] if axis != -1: raise NotImplementedError("Subset indexing for non-last " "axis not yet implemented") ## self.subset = np.zeros(self.D, dtype=bool) ## self.subset[list(subset)] = True def nodes(self): if self.update_alpha: return [self.node_X, self.node_alpha] else: return [self.node_X] def _full_rotation_matrix(self, R): if self.subset is not None: R_full = np.identity(self.D) indices = np.ix_(self.subset, self.subset) R_full[indices] = R return R_full else: return R def rotate(self, R, inv=None, logdet=None, Q=None): ## R = self._full_rotation_matrix(R) ## if inv is not None: ## inv = self._full_rotation_matrix(inv) self.node_X.rotate(R, inv=inv, logdet=logdet, subset=self.subset, axis=self.axis) if self.plate_axis is not None: self.node_X.rotate_plates(Q, plate_axis=self.plate_axis) if self.update_alpha: self.node_alpha.update() def setup(self, plate_axis=None): """ This method should be called just before optimization. For efficiency, sum over axes that are not in mu, alpha nor rotation. If using Q, set rotate_plates to True. """ # Store the original plate_axis parameter for later use in other methods self.plate_axis = plate_axis # Manipulate the plate_axis parameter to suit the needs of this method if plate_axis is not None: if not isinstance(plate_axis, int): raise ValueError("Plate axis must be integer") if plate_axis >= 0: plate_axis -= len(self.node_X.plates) if plate_axis < -len(self.node_X.plates) or plate_axis >= 0: raise ValueError("Axis out of bounds") plate_axis -= self.ndim - 1 # Why -1? Because one axis is preserved! # Get the mean parameter. It will not be rotated. This assumes that mu # and alpha are really independent. (alpha_mu, alpha_mu2, alpha, _) = self.node_parent.get_moments() (X, XX) = self.node_X.get_moments() # mu = alpha_mu / alpha mu2 = alpha_mu2 / alpha # For simplicity, force mu to have the same shape as X mu = mu * np.ones(self.node_X.dims[0]) mu2 = mu2 * np.ones(self.node_X.dims[0]) ## (mu, mumu) = gaussian.reshape_gaussian_array(self.node_mu.dims[0], ## self.node_X.dims[0], ## mu, ## mumu) # Take diagonal of covariances to variances for axes that are not in R # (and move those axes to be the last) XX = covariance_to_variance(XX, ndim=self.ndim, covariance_axis=self.axis) ## mumu = covariance_to_variance(mumu, ## ndim=self.ndim, ## covariance_axis=self.axis) # Move axes of X and mu and compute their outer product X = misc.moveaxis(X, self.axis, -1) mu = misc.moveaxis(mu, self.axis, -1) mu2 = misc.moveaxis(mu2, self.axis, -1) Xmu = linalg.outer(X, mu, ndim=1) D = np.shape(X)[-1] # Move axes of alpha related variables def safe_move_axis(x): if np.ndim(x) >= -self.axis: return misc.moveaxis(x, self.axis, -1) else: return x[...,np.newaxis] if self.update_alpha: a = safe_move_axis(self.node_alpha.phi[1]) a0 = safe_move_axis(self.node_alpha.parents[0].get_moments()[0]) b0 = safe_move_axis(self.node_alpha.parents[1].get_moments()[0]) plates_alpha = list(self.node_alpha.plates) else: alpha = safe_move_axis(self.node_parent.get_moments()[2]) plates_alpha = list(self.node_parent.get_shape(2)) # Move plates of alpha for R if len(plates_alpha) >= -self.axis: plate = plates_alpha.pop(self.axis) plates_alpha.append(plate) else: plates_alpha.append(1) plates_X = list(self.node_X.get_shape(0)) plates_X.pop(self.axis) def sum_to_alpha(V, ndim=2): # TODO/FIXME: This could be improved so that it is not required to # explicitly repeat to alpha plates. Multiplying by ones was just a # simple bug fix. return sum_to_plates(V * np.ones(plates_alpha[:-1]+ndim*[1]), plates_alpha[:-1], ndim=ndim, plates_from=plates_X) if plate_axis is not None: # Move plate axis just before the rotated dimensions (which are # last) def safe_move_plate_axis(x, ndim): if np.ndim(x)-ndim >= -plate_axis: return misc.moveaxis(x, plate_axis-ndim, -ndim-1) else: inds = (Ellipsis,None) + ndim*(slice(None),) return x[inds] X = safe_move_plate_axis(X, 1) mu = safe_move_plate_axis(mu, 1) XX = safe_move_plate_axis(XX, 2) mu2 = safe_move_plate_axis(mu2, 1) if self.update_alpha: a = safe_move_plate_axis(a, 1) a0 = safe_move_plate_axis(a0, 1) b0 = safe_move_plate_axis(b0, 1) else: alpha = safe_move_plate_axis(alpha, 1) # Move plates of X and alpha plate = plates_X.pop(plate_axis) plates_X.append(plate) if len(plates_alpha) >= -plate_axis+1: plate = plates_alpha.pop(plate_axis-1) else: plate = 1 plates_alpha = plates_alpha[:-1] + [plate] + plates_alpha[-1:] CovX = XX - linalg.outer(X, X) self.CovX = sum_to_plates(CovX, plates_alpha[:-2], ndim=3, plates_from=plates_X[:-1]) # Broadcast mumu to ensure shape #mumu = np.ones(np.shape(XX)[-3:]) * mumu mu2 = mu2 * np.ones(np.shape(X)[-2:]) self.mu2 = sum_to_alpha(mu2, ndim=1) if self.precompute: # Precompute some stuff for the gradient of plate rotation # # NOTE: These terms may require a lot of memory if alpha has the # same or almost the same plates as X. self.X_X = sum_to_plates(X[...,:,:,None,None] * X[...,None,None,:,:], plates_alpha[:-2], ndim=4, plates_from=plates_X[:-1]) self.X_mu = sum_to_plates(X[...,:,:,None,None] * mu[...,None,None,:,:], plates_alpha[:-2], ndim=4, plates_from=plates_X[:-1]) else: self.X = X self.mu = mu else: # Sum axes that are not in the plates of alpha self.XX = sum_to_alpha(XX) self.mu2 = sum_to_alpha(mu2, ndim=1) self.Xmu = sum_to_alpha(Xmu) if self.update_alpha: self.a = a self.a0 = a0 self.b0 = b0 else: self.alpha = alpha self.plates_X = plates_X self.plates_alpha = plates_alpha # Take only a subset of the matrix for rotation if self.subset is not None: if self.precompute: raise NotImplementedError("Precomputation not implemented when " "using a subset") # from X self.X = self.X[...,self.subset] self.mu2 = self.mu2[...,self.subset] if plate_axis is not None: # from CovX inds = [] for i in range(np.ndim(self.CovX)-2): inds.append(range(np.shape(self.CovX)[i])) inds.append(self.subset) inds.append(self.subset) indices = np.ix_(*inds) self.CovX = self.CovX[indices] # from mu self.mu = self.mu[...,self.subset] else: # from XX inds = [] for i in range(np.ndim(self.XX)-2): inds.append(range(np.shape(self.XX)[i])) inds.append(self.subset) inds.append(self.subset) indices = np.ix_(*inds) self.XX = self.XX[indices] # from Xmu self.Xmu = self.Xmu[...,self.subset] # from alpha if self.update_alpha: if np.shape(self.a)[-1] > 1: self.a = self.a[...,self.subset] if np.shape(self.a0)[-1] > 1: self.a0 = self.a0[...,self.subset] if np.shape(self.b0)[-1] > 1: self.b0 = self.b0[...,self.subset] else: if np.shape(self.alpha)[-1] > 1: self.alpha = self.alpha[...,self.subset] self.plates_alpha[-1] = min(self.plates_alpha[-1], len(self.subset)) ## # from mu ## # from alpha ## alpha_mu = alpha_mu[...,self.subset] ## alpha_mu2 = alpha_mu2[...,self.subset] ## alpha = alpha[...,self.subset] ## dims = list(self.node_X.dims[0]) ## dims[-1] = len(self.subset) ## else: ## dims = list(self.node_X.dims[0]) def _compute_bound(self, R, logdet=None, inv=None, Q=None, gradient=False, terms=False): """ Rotate q(X) and q(alpha). Assume: p(X|alpha) = prod_m N(x_m|0,diag(alpha)) p(alpha) = prod_d G(a_d,b_d) """ ## R = self._full_rotation_matrix(R) ## if inv is not None: ## inv = self._full_rotation_matrix(inv) # # Transform the distributions and moments # plates_alpha = self.plates_alpha plates_X = self.plates_X # Compute rotated second moment if self.plate_axis is not None: # The plate axis has been moved to be the last plate axis if Q is None: raise ValueError("Plates should be rotated but no Q give") # Transform covariance sumQ = np.sum(Q, axis=0) QCovQ = sumQ[:,None,None]**2 * self.CovX # Rotate plates if self.precompute: QX_QX = np.einsum('...kalb,...ik,...il->...iab', self.X_X, Q, Q) XX = QX_QX + QCovQ XX = sum_to_plates(XX, plates_alpha[:-1], ndim=2) Xmu = np.einsum('...kaib,...ik->...iab', self.X_mu, Q) Xmu = sum_to_plates(Xmu, plates_alpha[:-1], ndim=2) else: X = self.X mu = self.mu QX = np.einsum('...ik,...kj->...ij', Q, X) XX = (sum_to_plates(QCovQ, plates_alpha[:-1], ndim=2) + sum_to_plates(linalg.outer(QX, QX), plates_alpha[:-1], ndim=2, plates_from=plates_X)) Xmu = sum_to_plates(linalg.outer(QX, self.mu), plates_alpha[:-1], ndim=2, plates_from=plates_X) mu2 = self.mu2 D = np.shape(XX)[-1] logdet_Q = D * np.log(np.abs(sumQ)) else: XX = self.XX mu2 = self.mu2 Xmu = self.Xmu logdet_Q = 0 # Compute transformed moments #mu2 = np.einsum('...ii->...i', mu2) RXmu = np.einsum('...ik,...ki->...i', R, Xmu) RXX = np.einsum('...ik,...kj->...ij', R, XX) RXXR = np.einsum('...ik,...ik->...i', RXX, R) # <(X-mu) * (X-mu)'>_R XmuXmu = (RXXR - 2*RXmu + mu2) D = np.shape(R)[0] # Compute q(alpha) if self.update_alpha: # Parameters a0 = self.a0 b0 = self.b0 a = self.a b = b0 + 0.5*sum_to_plates(XmuXmu, plates_alpha, plates_from=None, ndim=0) # Some expectations alpha = a / b logb = np.log(b) logalpha = -logb # + const b0_alpha = b0 * alpha a0_logalpha = a0 * logalpha else: alpha = self.alpha logalpha = 0 # # Compute the cost # def sum_plates(V, *plates): full_plates = misc.broadcasted_shape(*plates) r = self.node_X.broadcasting_multiplier(full_plates, np.shape(V)) return r * np.sum(V) XmuXmu_alpha = XmuXmu * alpha if logdet is None: logdet_R = np.linalg.slogdet(R)[1] inv_R = np.linalg.inv(R) else: logdet_R = logdet inv_R = inv # Compute entropy H(X) logH_X = random.gaussian_entropy(-2*sum_plates(logdet_R + logdet_Q, plates_X), 0) # Compute logp_X = random.gaussian_logpdf(sum_plates(XmuXmu_alpha, plates_alpha[:-1] + [D]), 0, 0, sum_plates(logalpha, plates_X + [D]), 0) if self.update_alpha: # Compute entropy H(alpha) # This cancels out with the log(alpha) term in log(p(alpha)) logH_alpha = 0 # Compute logp_alpha = random.gamma_logpdf(sum_plates(b0_alpha, plates_alpha), 0, sum_plates(a0_logalpha, plates_alpha), 0, 0) else: logH_alpha = 0 logp_alpha = 0 # Compute the bound if terms: bound = {self.node_X: logp_X + logH_X} if self.update_alpha: bound.update({self.node_alpha: logp_alpha + logH_alpha}) else: bound = (0 + logp_X + logp_alpha + logH_X + logH_alpha ) if not gradient: return bound # # Compute the gradient with respect R # broadcasting_multiplier = self.node_X.broadcasting_multiplier def sum_plates(V, plates): ones = np.ones(np.shape(R)) r = broadcasting_multiplier(plates, np.shape(V)[:-2]) return r * misc.sum_multiply(V, ones, axis=(-1,-2), sumaxis=False, keepdims=False) D_XmuXmu = 2*RXX - 2*gaussian.transpose_covariance(Xmu) DXmuXmu_alpha = np.einsum('...i,...ij->...ij', alpha, D_XmuXmu) if self.update_alpha: D_b = 0.5 * D_XmuXmu XmuXmu_Dalpha = np.einsum('...i,...i,...i,...ij->...ij', sum_to_plates(XmuXmu, plates_alpha, plates_from=None, ndim=0), alpha, -1/b, D_b) D_b0_alpha = np.einsum('...i,...i,...i,...ij->...ij', b0, alpha, -1/b, D_b) D_logb = np.einsum('...i,...ij->...ij', 1/b, D_b) D_logalpha = -D_logb D_a0_logalpha = a0 * D_logalpha else: XmuXmu_Dalpha = 0 D_logalpha = 0 D_XmuXmu_alpha = DXmuXmu_alpha + XmuXmu_Dalpha D_logR = inv_R.T # Compute dH(X) dlogH_X = random.gaussian_entropy(-2*sum_plates(D_logR, plates_X), 0) # Compute d dlogp_X = random.gaussian_logpdf(sum_plates(D_XmuXmu_alpha, plates_alpha[:-1]), 0, 0, (sum_plates(D_logalpha, plates_X) * broadcasting_multiplier((D,), plates_alpha[-1:])), 0) if self.update_alpha: # Compute dH(alpha) # This cancels out with the log(alpha) term in log(p(alpha)) dlogH_alpha = 0 # Compute d dlogp_alpha = random.gamma_logpdf(sum_plates(D_b0_alpha, plates_alpha[:-1]), 0, sum_plates(D_a0_logalpha, plates_alpha[:-1]), 0, 0) else: dlogH_alpha = 0 dlogp_alpha = 0 if terms: raise NotImplementedError() dR_bound = {self.node_X: dlogp_X + dlogH_X} if self.update_alpha: dR_bound.update({self.node_alpha: dlogp_alpha + dlogH_alpha}) else: dR_bound = (0*dlogp_X + dlogp_X + dlogp_alpha + dlogH_X + dlogH_alpha ) if self.subset: indices = np.ix_(self.subset, self.subset) dR_bound = dR_bound[indices] if self.plate_axis is None: return (bound, dR_bound) # # Compute the gradient with respect to Q (if Q given) # # Some pre-computations Q_RCovR = np.einsum('...ik,...kl,...il,...->...i', R, self.CovX, R, sumQ) if self.precompute: Xr_rX = np.einsum('...abcd,...jb,...jd->...jac', self.X_X, R, R) QXr_rX = np.einsum('...akj,...ik->...aij', Xr_rX, Q) RX_mu = np.einsum('...jk,...akbj->...jab', R, self.X_mu) else: RX = np.einsum('...ik,...k->...i', R, X) QXR = np.einsum('...ik,...kj->...ij', Q, RX) QXr_rX = np.einsum('...ik,...jk->...kij', QXR, RX) RX_mu = np.einsum('...ik,...jk->...kij', RX, mu) QXr_rX = sum_to_plates(QXr_rX, plates_alpha[:-2], ndim=3, plates_from=plates_X[:-1]) RX_mu = sum_to_plates(RX_mu, plates_alpha[:-2], ndim=3, plates_from=plates_X[:-1]) def psi(v): """ Compute: d/dQ 1/2*trace(diag(v)*<(X-mu)*(X-mu)>) = Q*'*R'*diag(v)*R* + ones * Q diag( tr(R'*diag(v)*R*Cov) ) + mu*diag(v)*R* """ # Precompute all terms to plates_alpha because v has shape # plates_alpha. # Gradient of 0.5*v** v_QXrrX = np.einsum('...kij,...ik->...ij', QXr_rX, v) # Gradient of 0.5*v*Cov Q_tr_R_v_R_Cov = np.einsum('...k,...k->...', Q_RCovR, v)[...,None,:] # Gradient of mu*v*x mu_v_R_X = np.einsum('...ik,...kji->...ij', v, RX_mu) return v_QXrrX + Q_tr_R_v_R_Cov - mu_v_R_X def sum_plates(V, plates): ones = np.ones(np.shape(Q)) r = self.node_X.broadcasting_multiplier(plates, np.shape(V)[:-2]) return r * misc.sum_multiply(V, ones, axis=(-1,-2), sumaxis=False, keepdims=False) if self.update_alpha: D_logb = psi(1/b) XX_Dalpha = -psi(alpha/b * sum_to_plates(XmuXmu, plates_alpha)) D_logalpha = -D_logb else: XX_Dalpha = 0 D_logalpha = 0 DXX_alpha = 2*psi(alpha) D_XX_alpha = DXX_alpha + XX_Dalpha D_logdetQ = D / sumQ N = np.shape(Q)[-1] # Compute dH(X) dQ_logHX = random.gaussian_entropy(-2*sum_plates(D_logdetQ, plates_X[:-1]), 0) # Compute d dQ_logpX = random.gaussian_logpdf(sum_plates(D_XX_alpha, plates_alpha[:-2]), 0, 0, (sum_plates(D_logalpha, plates_X[:-1]) * broadcasting_multiplier((N,D), plates_alpha[-2:])), 0) if self.update_alpha: D_alpha = -psi(alpha/b) D_b0_alpha = b0 * D_alpha D_a0_logalpha = a0 * D_logalpha # Compute dH(alpha) # This cancels out with the log(alpha) term in log(p(alpha)) dQ_logHalpha = 0 # Compute d dQ_logpalpha = random.gamma_logpdf(sum_plates(D_b0_alpha, plates_alpha[:-2]), 0, sum_plates(D_a0_logalpha, plates_alpha[:-2]), 0, 0) else: dQ_logHalpha = 0 dQ_logpalpha = 0 if terms: raise NotImplementedError() dQ_bound = {self.node_X: dQ_logpX + dQ_logHX} if self.update_alpha: dQ_bound.update({self.node_alpha: dQ_logpalpha + dQ_logHalpha}) else: dQ_bound = (0*dQ_logpX + dQ_logpX + dQ_logpalpha + dQ_logHX + dQ_logHalpha ) return (bound, dR_bound, dQ_bound) def bound(self, R, logdet=None, inv=None, Q=None): return self._compute_bound(R, logdet=logdet, inv=inv, Q=Q, gradient=True) def get_bound_terms(self, R, logdet=None, inv=None, Q=None): return self._compute_bound(R, logdet=logdet, inv=inv, Q=Q, gradient=False, terms=True) class RotateGaussianMarkovChain(): r""" Rotation parameter expansion for :class:`bayespy.nodes.GaussianMarkovChain` Assume the following model. Constant, unit isotropic innovation noise. Unit variance only? Maybe: Assume innovation noise with unit variance? Would it help make this function more general with respect to A. TODO: Allow constant A or not rotating A. :math:`A` may vary in time. Shape of A: (N,D,D) Shape of AA: (N,D,D,D) No plates for X. """ def __init__(self, X, *args): self.X_node = X self.A_node = X.parents[2] if len(args) == 0: raise NotImplementedError() elif len(args) == 1: self.A_rotator = args[0] else: raise ValueError("Wrong number of arguments") self.N = X.dims[0][0] def nodes(self): return [self.X_node] + self.A_rotator.nodes() def rotate(self, R, inv=None, logdet=None): if inv is None: inv = np.linalg.inv(R) if logdet is None: logdet = np.linalg.slogdet(R)[1] self.X_node.rotate(R, inv=inv, logdet=logdet) self.A_rotator.rotate(inv.T, inv=R.T, logdet=-logdet, Q=R) def _computations_for_A_and_X(self, XpXn, XpXp): # Get moments of the state dynamics matrix (A, AA) = self.A_node.get_moments() # Ignore axes that correspond to input signals D = np.shape(A)[-2] A = A[...,:D] AA = AA[...,:D,:D] # Make sure time axis is in the arrays A = misc.atleast_nd(A, 3) AA = misc.atleast_nd(AA, 4) CovA = AA - A[...,:,np.newaxis]*A[...,np.newaxis,:] # # Expectations with respect to A and X # # TODO: In case A does not depend on time, use a bit more efficient # formulas # Compute: \sum_n A_XpXn = np.einsum('...nik,...nkj->...ij', A, XpXn) A_XpXn = sum_to_plates(A_XpXn, (), ndim=2, plates_from=self.X_node.plates) # Compute: \sum_n ^T A_XpXp = np.einsum('...nik,...nkj->...nij', A, XpXp) A_XpXp_A = np.einsum('...nik,...njk->...ij', A_XpXp, A) A_XpXp_A = sum_to_plates(A_XpXp_A, (), ndim=2, plates_from=self.X_node.plates) # Compute: \sum_n tr(CovA_n ) CovA_XpXp = np.einsum('...ndij,...nij->...d', CovA, XpXp) CovA_XpXp = sum_to_plates(CovA_XpXp, (), ndim=1, plates_from=self.X_node.plates) return (A_XpXn, A_XpXp_A, CovA_XpXp) def setup(self): """ This method should be called just before optimization. """ # Get moments of X (X, XnXn, XpXn) = self.X_node.get_moments() # TODO/FIXME: Sum to plates of A/CovA XpXp = XnXn[...,:-1,:,:] # # Expectations with respect to X # self.X0 = X[...,0,:] self.X0X0 = XnXn[...,0,:,:] #self.XnXn = np.sum(XnXn[...,1:,:,:], axis=-3) self.XnXn = sum_to_plates(XnXn[...,1:,:,:], (), plates_from=self.X_node.plates + (self.N-1,), ndim=2) # Get moments of the fixed parameter nodes mu = self.X_node.parents[0].get_moments()[0] self.Lambda = self.X_node.parents[1].get_moments()[0] self.Lambda_mu_X0 = linalg.outer(np.einsum('...ik,...k->...i', self.Lambda, mu), self.X0) self.Lambda_mu_X0 = sum_to_plates(self.Lambda_mu_X0, (), plates_from=self.X_node.plates, ndim=2) # # Prepare the rotation for A # (self.A_XpXn, self.A_XpXp_A, self.CovA_XpXp) = self._computations_for_A_and_X(XpXn, XpXp) self.A_rotator.setup(plate_axis=-1) # Innovation noise is assumed to be I #self.v = self.X_node.parents[3].get_moments()[0] def _compute_bound(self, R, logdet=None, inv=None, gradient=False, terms=False): """ Rotate q(X) as X->RX: q(X)=N(R*mu, R*Cov*R') Assume: :math:`p(\mathbf{X}) = \prod^M_{m=1} N(\mathbf{x}_m|0, \mathbf{\Lambda})` Assume unit innovation noise covariance. """ # TODO/FIXME: X and alpha should NOT contain observed values!! Check # that. # Assume constant mean and precision matrix over plates.. if inv is None: invR = np.linalg.inv(R) else: invR = inv if logdet is None: logdetR = np.linalg.slogdet(R)[1] else: logdetR = logdet # Transform moments of X and A: Lambda_R_X0X0 = sum_to_plates(dot(self.Lambda, R, self.X0X0), (), plates_from=self.X_node.plates, ndim=2) R_XnXn = dot(R, self.XnXn) RA_XpXp_A = dot(R, self.A_XpXp_A) sumr = np.sum(R, axis=0) R_CovA_XpXp = sumr * self.CovA_XpXp # Compute entropy H(X) M = self.N*np.prod(self.X_node.plates) # total number of rotated vectors logH_X = random.gaussian_entropy(-2 * M * logdetR, 0) # Compute yy = tracedot(R_XnXn, R.T) + tracedot(Lambda_R_X0X0, R.T) yz = tracedot(dot(R,self.A_XpXn),R.T) + tracedot(self.Lambda_mu_X0, R.T) zz = tracedot(RA_XpXp_A, R.T) + np.einsum('...k,...k->...', R_CovA_XpXp, sumr) logp_X = random.gaussian_logpdf(yy, yz, zz, 0, 0) # Compute the bound if terms: bound = {self.X_node: logp_X + logH_X} else: bound = logp_X + logH_X if not gradient: return bound # Compute dH(X) dlogH_X = random.gaussian_entropy(-2 * M * invR.T, 0) # Compute d dyy = 2 * (R_XnXn + Lambda_R_X0X0) dyz = dot(R, self.A_XpXn + self.A_XpXn.T) + self.Lambda_mu_X0 dzz = 2 * (RA_XpXp_A + R_CovA_XpXp[None,:]) dlogp_X = random.gaussian_logpdf(dyy, dyz, dzz, 0, 0) if terms: d_bound = {self.X_node: dlogp_X + dlogH_X} else: d_bound = ( + dlogp_X + dlogH_X ) return (bound, d_bound) def bound(self, R, logdet=None, inv=None): if inv is None: inv = np.linalg.inv(R) if logdet is None: logdet = np.linalg.slogdet(R)[1] (bound_X, d_bound_X) = self._compute_bound(R, logdet=logdet, inv=inv, gradient=True) # Compute cost and gradient from A (bound_A, dR_bound_A, dQ_bound_A) = self.A_rotator.bound(inv.T, inv=R.T, logdet=-logdet, Q=R) dR_bound_A = -dot(inv.T, dR_bound_A.T, inv.T) # Compute the bound bound = bound_X + bound_A d_bound = d_bound_X + dR_bound_A + dQ_bound_A return (bound, d_bound) def get_bound_terms(self, R, logdet=None, inv=None): if inv is None: inv = np.linalg.inv(R) if logdet is None: logdet = np.linalg.slogdet(R)[1] terms_A = self.A_rotator.get_bound_terms(inv.T, inv=R.T, logdet=-logdet, Q=R) terms_X = self._compute_bound(R, logdet=logdet, inv=inv, gradient=False, terms=True) terms_X.update(terms_A) return terms_X class RotateVaryingMarkovChain(RotateGaussianMarkovChain): r""" Rotation for :class:`bayespy.nodes.SwitchingGaussianMarkovChain` Assume the following model. Constant, unit isotropic innovation noise. :math:`A_n = \sum_k B_k s_{kn}` Gaussian B: (1,D) x (D,K) Gaussian S: (N,1) x (K) MC X: () x (N+1,D) No plates for X. """ def __init__(self, X, B, S, B_rotator): self.X_node = X self.B_node = B self.S_node = S self.B_rotator = B_rotator if len(S.plates) > 0 and S.plates[-1] > 1: raise ValueError("The length of the last plate of S must be 1.") if len(B.plates) > 1 and B.plates[-2] > 1: raise ValueError("The length of the last plate of B must be 1.") if len(S.dims[0]) != 1: raise ValueError("S should have exactly one variable axis") if len(B.dims[0]) != 2: raise ValueError("B should have exactly two variable axes") super().__init__(X, B_rotator) def _computations_for_A_and_X(self, XpXn, XpXp): # Get moments of B and S (B, BB) = self.B_node.get_moments() CovB = BB - B[...,:,:,None,None]*B[...,None,None,:,:] u_S = self.S_node.get_moments() S = u_S[0] SS = u_S[1] # # Expectations with respect to A and X # # TODO/FIXME: If S and B have overlapping plates, then these will give # wrong results, because those plates of S are summed before multiplying # by the plates of B. There should be some "smart einsum" function which # would compute sum-multiplys intelligently given a number of inputs. # Compute: \sum_n # Axes: (N, D, D, D, K) S_XpXn = misc.sum_multiply(S[...,None,None,:], XpXn[...,:,None,:,:,None], axis=(-3,-2,-1), sumaxis=False) A_XpXn = misc.sum_multiply(B[...,:,:,None,:], S_XpXn[...,:,:,:], axis=(-4,-2), sumaxis=False) # Compute: \sum_n ^T # Axes: (N, D, D, D, K, D, K) SS_XpXp = misc.sum_multiply(SS[...,None,:,None,:], XpXp[...,None,:,None,:,None], axis=(-4,-3,-2,-1), sumaxis=False) B_SS_XpXp = misc.sum_multiply(B[...,:,:,:,None,None], SS_XpXp[...,:,:,:,:], axis=(-4,-3), sumaxis=True) A_XpXp_A = misc.sum_multiply(B_SS_XpXp[...,:,None,:,:], B[...,None,:,:,:], axis=(-4,-3), sumaxis=False) # Compute: \sum_n tr(CovA_n ) # Axes: (D,D,K,D,K) CovA_XpXp = misc.sum_multiply(CovB, SS_XpXp, axis=(-5,), sumaxis=False) return (A_XpXn, A_XpXp_A, CovA_XpXp) class RotateSwitchingMarkovChain(RotateGaussianMarkovChain): """ Rotation for :class:`bayespy.nodes.VaryingGaussianMarkovChain` Assume the following model. Constant, unit isotropic innovation noise. :math:`A_n = B_{z_n}` Gaussian B: (..., K, D) x (D) Categorical Z: (..., N-1) x (K) GaussianMarkovChain X: (...) x (N,D) No plates for X. """ def __init__(self, X, B, Z, B_rotator): self.X_node = X self.B_node = B self.Z_node = Z._convert(CategoricalMoments) self.B_rotator = B_rotator (N,D) = self.X_node.dims[0] K = self.Z_node.dims[0][0] if len(self.Z_node.plates) == 0 and self.Z_node.plates[-1] != N-1: raise ValueError("Incorrect plate length in Z") if self.B_node.plates[-2:] != (K,D): raise ValueError("Incorrect plates in B") if len(self.Z_node.dims[0]) != 1: raise ValueError("Z should have exactly one variable axis") if len(self.B_node.dims[0]) != 1: raise ValueError("B should have exactly one variable axes") super().__init__(X, B_rotator) def _computations_for_A_and_X(self, XpXn, XpXp): # Get moments of B and Z (B, BB) = self.B_node.get_moments() CovB = BB - B[...,:,None]*B[...,None,:] u_Z = self.Z_node.get_moments() Z = u_Z[0] # # Expectations with respect to A and X # # Compute: \sum_n Z_XpXn = np.einsum('...nij,...nk->...kij', XpXn, Z) A_XpXn = np.einsum('...kil,...klj->...ij', B, Z_XpXn) A_XpXn = sum_to_plates(A_XpXn, (), ndim=2, plates_from=self.X_node.plates) # Compute: \sum_n ^T Z_XpXp = np.einsum('...nij,...nk->...kij', XpXp, Z) B_Z_XpXp = np.einsum('...kil,...klj->...kij', B, Z_XpXp) A_XpXp_A = np.einsum('...kil,...kjl->...ij', B_Z_XpXp, B) A_XpXp_A = sum_to_plates(A_XpXp_A, (), ndim=2, plates_from=self.X_node.plates) # Compute: \sum_n tr(CovA_n ) CovA_XpXp = np.einsum('...kij,...kdij->...d', Z_XpXp, CovB) CovA_XpXp = sum_to_plates(CovA_XpXp, (), ndim=1, plates_from=self.X_node.plates) return (A_XpXn, A_XpXp_A, CovA_XpXp) class RotateMultiple(): r""" Identical parameter expansion for several nodes simultaneously Performs the same rotation for multiple nodes and combines the cost effect. """ def __init__(self, *rotators): self.rotators = rotators def nodes(self): return [node for node in rotator.nodes() for rotator in self.rotators] def rotate(self, R, inv=None, logdet=None): for rotator in self.rotators: rotator.rotate(R, inv=inv, logdet=logdet) def setup(self): for rotator in self.rotators: rotator.setup() def bound(self, R, logdet=None, inv=None): bound = 0 dbound = 0 for rotator in self.rotators: (b, db) = rotator.bound(R, logdet=logdet, inv=inv) bound = bound + b dbound = dbound + db return (bound, dbound) def get_bound_terms(self, R, logdet=None, inv=None): return {node: terms for (node, terms) in rotator.items() for rotator in self.rotators} bayespy-0.4.0/bayespy/inference/vmp/__init__.py0000664000175000017500000000000012174013072023217 0ustar jluttinejluttine00000000000000bayespy-0.4.0/bayespy/inference/vmp/nodes/0000755000175000017500000000000012615647103022236 5ustar jluttinejluttine00000000000000bayespy-0.4.0/bayespy/inference/vmp/nodes/tests/0000755000175000017500000000000012615647103023400 5ustar jluttinejluttine00000000000000bayespy-0.4.0/bayespy/inference/vmp/nodes/tests/test_poisson.py0000644000175000017500000000470412615643655026520 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for `poisson` module. """ import numpy as np import scipy from bayespy.nodes import Poisson from bayespy.nodes import Gamma from bayespy.utils import random from bayespy.utils.misc import TestCase class TestPoisson(TestCase): """ Unit tests for Poisson node """ def test_init(self): """ Test the creation of Poisson nodes. """ # Some simple initializations X = Poisson(12.8) X = Poisson(Gamma(43, 24)) # Check that plates are correct X = Poisson(np.ones((2,3))) self.assertEqual(X.plates, (2,3)) X = Poisson(Gamma(1, 1, plates=(2,3))) self.assertEqual(X.plates, (2,3)) # Invalid rate self.assertRaises(ValueError, Poisson, -0.1) # Inconsistent plates self.assertRaises(ValueError, Poisson, np.ones(3), plates=(2,)) # Explicit plates too small self.assertRaises(ValueError, Poisson, np.ones(3), plates=(1,)) pass def test_moments(self): """ Test the moments of Poisson nodes. """ # Simple test X = Poisson(12.8) u = X._message_to_child() self.assertEqual(len(u), 1) self.assertAllClose(u[0], 12.8) # Test plates in rate X = Poisson(12.8*np.ones((2,3))) u = X._message_to_child() self.assertAllClose(u[0], 12.8*np.ones((2,3))) # Test with gamma prior alpha = Gamma(5, 2) r = np.exp(alpha._message_to_child()[1]) X = Poisson(alpha) u = X._message_to_child() self.assertAllClose(u[0], r) # Test with broadcasted plates in parents X = Poisson(Gamma(5, 2, plates=(2,3))) u = X._message_to_child() self.assertAllClose(u[0]*np.ones((2,3)), r*np.ones((2,3))) pass bayespy-0.4.0/bayespy/inference/vmp/nodes/tests/test_deterministic.py0000644000175000017500000002620412615643655027670 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for `deterministic` module. """ import unittest import numpy as np import scipy from numpy import testing from ..node import Node, Moments from ..deterministic import tile from ..stochastic import Stochastic class TestTile(unittest.TestCase): def check_message_to_children(self, tiles, u_parent, u_tiled, dims=None, plates=None): # Set up the dummy model class Dummy(Stochastic): _moments = Moments() def __init__(self, u, dims=dims, plates=plates): super().__init__(dims=dims, plates=plates, initialize=False) self.u = u X = Dummy(u_parent, dims=dims, plates=plates) Y = tile(X, tiles) u_Y = Y._compute_moments(u_parent) for (x,y) in zip(u_Y, u_tiled): self.assertEqual(np.shape(x), np.shape(y), msg="Incorrect shape.") testing.assert_allclose(x, y, err_msg="Incorrect moments.") def test_message_to_children(self): """ Test the moments of Tile node. """ # Define th check function check_message_to_children = self.check_message_to_children # Check scalar (and broadcasting) check_message_to_children(2, (5,), (5,), dims=[()], plates=()), # Check 1-D check_message_to_children(2, ([1,2],), ([1,2,1,2],), dims=[()], plates=(2,)) # Check N-D check_message_to_children(2, ([[1,2], [3,4], [5,6]],), ([[1,2,1,2], [3,4,3,4], [5,6,5,6]],), dims=[()], plates=(3,2)) # Check not-last plate check_message_to_children([2,1], ([[1,2], [3,4]],), ([[1,2], [3,4], [1,2], [3,4]],), dims=[()], plates=(2,2)) # Check several plates check_message_to_children([2,3], ([[1,2], [3,4]],), ([[1,2,1,2,1,2], [3,4,3,4,3,4], [1,2,1,2,1,2], [3,4,3,4,3,4]],), dims=[()], plates=(2,2)) # Check non-zero dimensional variables check_message_to_children(2, ([[1,2], [3,4]],), ([[1,2], [3,4], [1,2], [3,4]],), dims=[(2,)], plates=(2,)) # Check several moments check_message_to_children(2, ([[1,2], [3,4]], [1,2]), ([[1,2], [3,4], [1,2], [3,4]], [1,2,1,2]), dims=[(2,),()], plates=(2,)) # Check broadcasting of tiled plate check_message_to_children(2, ([[1,], [2,]],), ([[1,], [2,]],), dims=[()], plates=(2,2)) # Check broadcasting of non-tiled plate check_message_to_children(2, ([[1,2]],), ([[1,2,1,2]],), dims=[()], plates=(2,2)) # Check broadcasting of leading plates that are not in parent check_message_to_children([2,1], ([1,2],), ([1,2],), dims=[()], plates=(2,)) def check_message_to_parent(self, tiles, m_children, m_true, dims=None, plates_parent=None, plates_children=None): # Set up the dummy model class Dummy(Stochastic): _moments = Moments() X = Dummy(dims=dims, plates=plates_parent, initialize=False) Y = tile(X, tiles) m = Y._compute_message_to_parent(0, m_children, None) for (x,y) in zip(m, m_true): self.assertEqual(np.shape(x), np.shape(y), msg="Incorrect shape.") testing.assert_allclose(x, y, err_msg="Incorrect message.") def test_message_to_parent(self): """ Test the parent message of Tile node. """ # Define th check function check = self.check_message_to_parent # Check scalar check(2, ([5,5],), (10,), dims=[()], plates_parent=(), plates_children=(2,)), # Check 1-D check(2, ([1,2,3,4],), ([4,6],), dims=[()], plates_parent=(2,), plates_children=(4,)) # Check N-D check(2, ([[1,2,7,8], [3,4,9,0], [5,6,1,2]],), ([[8,10], [12,4], [6,8]],), dims=[()], plates_parent=(3,2), plates_children=(3,4)) # Check not-last plate check([2,1], ([[1,2], [3,4], [5,6], [7,8]],), ([[6,8], [10,12]],), dims=[()], plates_parent=(2,2), plates_children=(4,2)) # Check several plates check([2,3], ([[1,2,1,2,1,2], [3,4,3,4,3,4], [1,2,1,2,1,2], [3,4,3,4,3,4]],), ([[6,12], [18,24]],), dims=[()], plates_parent=(2,2), plates_children=(4,6)) # Check broadcasting if message has unit axis for tiled plate check(2, ([[1,], [2,], [3,]],), ([[2,], [4,], [6,]],), dims=[()], plates_parent=(3,2), plates_children=(3,4)) # Check broadcasting if message has unit axis for non-tiled plate check(2, ([[1,2,3,4]],), ([[4,6]],), dims=[()], plates_parent=(3,2), plates_children=(3,4)) # Check non-zero dimensional variables check(2, ([[1,2], [3,4], [5,6], [7,8]],), ([[6,8], [10,12]],), dims=[(2,)], plates_parent=(2,), plates_children=(4,)) # Check several moments check(2, ([[1,2], [3,4], [5,6], [7,8]], [1,2,3,4]), ([[6,8], [10,12]], [4,6]), dims=[(2,),()], plates_parent=(2,), plates_children=(4,)) def check_mask_to_parent(self, tiles, mask_child, mask_true, plates_parent=None, plates_children=None): # Set up the dummy model class Dummy(Stochastic): _moments = Moments() X = Dummy(dims=[()], plates=plates_parent, initialize=False) Y = tile(X, tiles) mask = Y._compute_mask_to_parent(0, mask_child) self.assertEqual(np.shape(mask), np.shape(mask_true), msg="Incorrect shape.") testing.assert_equal(mask, mask_true, err_msg="Incorrect mask.") def test_mask_to_parent(self): """ Test the mask message to parent of Tile node. """ # Define th check function check = self.check_mask_to_parent # Check scalar parent check(2, [True,False], True, plates_parent=(), plates_children=(2,)) check(2, [False,False], False, plates_parent=(), plates_children=(2,)) # Check 1-D check(2, [True,False,False,False], [True,False], plates_parent=(2,), plates_children=(4,)) # Check N-D check(2, [[True,False,True,False], [False,True,False,False]], [[True,False], [False,True]], plates_parent=(2,2), plates_children=(2,4)) # Check not-last plate check([2,1], [[True,False], [False,True], [True,False], [False,False]], [[True,False], [False,True]], plates_parent=(2,2), plates_children=(4,2)) # Check several plates check([2,3], [[False,False,True,False,False,False], [False,False,False,False,False,False], [False,False,False,False,False,False], [False,True,False,False,False,False]], [[True,False], [False,True]], plates_parent=(2,2), plates_children=(4,6)) # Check broadcasting if message has unit axis for tiled plate check(2, [[True,], [False,], [True,]], [[True,], [False,], [True,]], plates_parent=(3,2), plates_children=(3,4)) # Check broadcasting if message has unit axis for non-tiled plate check(2, [[False,False,False,True]], [[False,True]], plates_parent=(3,2), plates_children=(3,4)) bayespy-0.4.0/bayespy/inference/vmp/nodes/tests/test_wishart.py0000644000175000017500000000730512615643655026507 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2015 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for `wishart` module. """ import numpy as np from scipy import special from .. import gaussian from bayespy.nodes import (Gaussian, Wishart) from ...vmp import VB from bayespy.utils import misc from bayespy.utils import linalg from bayespy.utils import random from bayespy.utils.misc import TestCase def _student_logpdf(y, mu, Cov, nu): D = np.shape(y)[-1] return (special.gammaln((nu+D)/2) - special.gammaln(nu/2) - 0.5 * D * np.log(nu) - 0.5 * D * np.log(np.pi) - 0.5 * np.linalg.slogdet(Cov)[1] - 0.5 * (nu+D) * np.log(1+1/nu*np.einsum('...i,...ij,...j->...', y-mu, np.linalg.inv(Cov), y-mu))) class TestWishart(TestCase): def test_lower_bound(self): """ Test the Wishart VB lower bound """ # # By having the Wishart node as the only latent node, VB will give exact # results, thus the VB lower bound is the true marginal log likelihood. # Thus, check that they are equal. The true marginal likelihood is the # multivariate Student-t distribution. # np.random.seed(42) D = 3 n = (D-1) + np.random.uniform(0.1, 0.5) V = random.covariance(D) Lambda = Wishart(n, V) mu = np.random.randn(D) Y = Gaussian(mu, Lambda) y = np.random.randn(D) Y.observe(y) Lambda.update() L = Y.lower_bound_contribution() + Lambda.lower_bound_contribution() mu = mu nu = n + 1 - D Cov = V / nu self.assertAllClose(L, _student_logpdf(y, mu, Cov, nu)) pass def test_moments(self): """ Test the moments of Wishart node """ np.random.seed(42) # Test prior moments D = 3 n = (D-1) + np.random.uniform(0.1,2) V = random.covariance(D) Lambda = Wishart(n, V) Lambda.update() u = Lambda.get_moments() self.assertAllClose(u[0], n*np.linalg.inv(V), msg='Mean incorrect') self.assertAllClose(u[1], (np.sum(special.digamma((n - np.arange(D))/2)) + D*np.log(2) - np.linalg.slogdet(V)[1]), msg='Log determinant incorrect') # Test posterior moments D = 3 n = (D-1) + np.random.uniform(0.1,2) V = random.covariance(D) Lambda = Wishart(n, V) mu = np.random.randn(D) Y = Gaussian(mu, Lambda) y = np.random.randn(D) Y.observe(y) Lambda.update() u = Lambda.get_moments() n = n + 1 V = V + np.outer(y-mu, y-mu) self.assertAllClose(u[0], n*np.linalg.inv(V), msg='Mean incorrect') self.assertAllClose(u[1], (np.sum(special.digamma((n - np.arange(D))/2)) + D*np.log(2) - np.linalg.slogdet(V)[1]), msg='Log determinant incorrect') pass bayespy-0.4.0/bayespy/inference/vmp/nodes/tests/test_bernoulli.py0000644000175000017500000000673612615643655027030 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for `bernoulli` module. """ import numpy as np import scipy from bayespy.nodes import (Bernoulli, Beta, Mixture) from bayespy.utils import random from bayespy.utils.misc import TestCase class TestBernoulli(TestCase): """ Unit tests for Bernoulli node """ def test_init(self): """ Test the creation of Bernoulli nodes. """ # Some simple initializations X = Bernoulli(0.5) X = Bernoulli(Beta([2,3])) # Check that plates are correct X = Bernoulli(0.7, plates=(4,3)) self.assertEqual(X.plates, (4,3)) X = Bernoulli(0.7*np.ones((4,3))) self.assertEqual(X.plates, (4,3)) X = Bernoulli(Beta([4,3], plates=(4,3))) self.assertEqual(X.plates, (4,3)) # Invalid probability self.assertRaises(ValueError, Bernoulli, -0.5) self.assertRaises(ValueError, Bernoulli, 1.5) # Inconsistent plates self.assertRaises(ValueError, Bernoulli, 0.5*np.ones(4), plates=(3,)) # Explicit plates too small self.assertRaises(ValueError, Bernoulli, 0.5*np.ones(4), plates=(1,)) pass def test_moments(self): """ Test the moments of Bernoulli nodes. """ # Simple test X = Bernoulli(0.7) u = X._message_to_child() self.assertEqual(len(u), 1) self.assertAllClose(u[0], 0.7) # Test plates in p p = np.random.rand(3) X = Bernoulli(p) u = X._message_to_child() self.assertAllClose(u[0], p) # Test with beta prior P = Beta([7, 3]) logp = P._message_to_child()[0] p0 = np.exp(logp[0]) / (np.exp(logp[0]) + np.exp(logp[1])) X = Bernoulli(P) u = X._message_to_child() self.assertAllClose(u[0], p0) # Test with broadcasted plates P = Beta([7, 3], plates=(10,)) X = Bernoulli(P) u = X._message_to_child() self.assertAllClose(u[0] * np.ones(X.get_shape(0)), p0*np.ones(10)) pass def test_mixture(self): """ Test mixture of Bernoulli """ P = Mixture([2,0,0], Bernoulli, [0.1, 0.2, 0.3]) u = P._message_to_child() self.assertEqual(len(u), 1) self.assertAllClose(u[0], [0.3, 0.1, 0.1]) pass def test_observed(self): """ Test observation of Bernoulli node """ Z = Bernoulli(0.3) Z.observe(2 < 3) pass def test_random(self): """ Test random sampling in Bernoulli node """ p = [1.0, 0.0] with np.errstate(divide='ignore'): Z = Bernoulli(p, plates=(3,2)).random() self.assertArrayEqual(Z, np.ones((3,2))*p) bayespy-0.4.0/bayespy/inference/vmp/nodes/tests/test_node.py0000644000175000017500000010050712615643655025751 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2013-2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for `dot` module. """ import unittest import numpy as np import scipy from numpy import testing from ..node import Node, Moments from ...vmp import VB from bayespy.utils import misc class TestMoments(unittest.TestCase): def test_converter(self): """ Tests complex conversions for moment classes """ # Simple one step conversions class A(Moments): pass class B(Moments): _converters = {A: lambda x: x+1} f = B().get_converter(B) self.assertEqual(f(3), 3) f = B().get_converter(Moments) self.assertEqual(f(3), 3) f = B().get_converter(A) self.assertEqual(f(3), 4) f = A().get_converter(A) self.assertEqual(f(3), 3) f = A().get_converter(Moments) self.assertEqual(f(3), 3) self.assertRaises(Moments.NoConverterError, A().get_converter, B) # Convert via parent class C(B): pass f = C().get_converter(A) self.assertEqual(f(3), 4) # Convert via grand parent class D(C): pass class E(D): pass f = E().get_converter(A) self.assertEqual(f(3), 4) # Can't convert to child self.assertRaises(Moments.NoConverterError, Moments().get_converter, A) # Convert to grand child class F(Moments): _converters = {E: lambda x: 2*x} f = F().get_converter(B) self.assertEqual(f(3), 6) # Use two conversions f = F().get_converter(A) self.assertEqual(f(3), 2*3+1) # Can't use child's converter class H(Moments): pass class I(Moments): _converters = {A: lambda x: x+1} self.assertRaises(Moments.NoConverterError, H().get_converter, A) # Conversion to parent is not success class J(A): pass self.assertRaises(Moments.NoConverterError, I().get_converter, J) # Infinite loop class X(Moments): pass class Y(Moments): pass X.add_converter(Y, lambda x: x+1) Y.add_converter(X, lambda x: x+1) self.assertRaises(Moments.NoConverterError, X().get_converter, A) # Test that add_converter function does not change the converters of # parent classes class Z(Moments): pass class W(Z): pass W.add_converter(Y, lambda x: x) self.assertRaises(Moments.NoConverterError, Z().get_converter, Y) # Test that after using add_converter for a child class and then for the # parent class, the child class is still able to use the parent's # converters class X(Moments): pass class Y(Moments): pass class A(Moments): pass class B(A): pass B.add_converter(Y, lambda x: x+1) A.add_converter(X, lambda x: 2*x) f = B().get_converter(X) self.assertEqual(f(3), 6) pass class TestNode(misc.TestCase): def check_message_to_parent(self, plates_child, plates_message, plates_mask, plates_parent, dims=(2,)): # Dummy message msg = np.random.randn(*(plates_message+dims)) # Mask with every other True and every other False mask = np.mod(np.arange(np.prod(plates_mask)).reshape(plates_mask), 2) == 0 # Set up the dummy model class Dummy(Node): _moments = Moments() def __init__(self, *args, **kwargs): self._parent_moments = len(args)*(Moments(),) super().__init__(*args, **kwargs) def _get_message_and_mask_to_parent(self, index): return ([msg], mask) def _get_id_list(self): return [] parent = Dummy(dims=[dims], plates=plates_parent) child = Dummy(parent, dims=[dims], plates=plates_child) m = child._message_to_parent(0)[0] * np.ones(plates_parent+dims) # Brute-force computation of the message without too much checking m_true = msg * misc.squeeze(mask[...,np.newaxis]) * np.ones(plates_child+dims) for ind in range(len(plates_child)): axis = -ind - 2 if ind >= len(plates_parent): m_true = np.sum(m_true, axis=axis, keepdims=False) elif plates_parent[-ind-1] == 1: m_true = np.sum(m_true, axis=axis, keepdims=True) testing.assert_allclose(m, m_true, err_msg="Incorrect message.") def test_message_to_parent(self): """ Test plate handling in _message_to_parent. """ # Test empty plates with scalar messages self.check_message_to_parent((), (), (), (), dims=()) # Test singular plates self.check_message_to_parent((2,3,4), (2,3,4), (2,3,4), (2,3,4)) self.check_message_to_parent((2,3,4), (2,1,4), (2,3,4), (2,3,4)) self.check_message_to_parent((2,3,4), (2,3,4), (2,1,4), (2,3,4)) self.check_message_to_parent((2,3,4), (2,3,4), (2,3,4), (2,1,4)) self.check_message_to_parent((2,3,4), (2,1,4), (2,1,4), (2,3,4)) self.check_message_to_parent((2,3,4), (2,3,4), (2,1,4), (2,1,4)) self.check_message_to_parent((2,3,4), (2,1,4), (2,3,4), (2,1,4)) self.check_message_to_parent((2,3,4), (2,1,4), (2,1,4), (2,1,4)) # Test missing plates self.check_message_to_parent((4,3), (4,3), (4,3), (4,3)) self.check_message_to_parent((4,3), ( 3,), (4,3), (4,3)) self.check_message_to_parent((4,3), (4,3), ( 3,), (4,3)) self.check_message_to_parent((4,3), (4,3), (4,3), ( 3,)) self.check_message_to_parent((4,3), ( 3,), ( 3,), (4,3)) self.check_message_to_parent((4,3), ( 3,), (4,3), ( 3,)) self.check_message_to_parent((4,3), (4,3), ( 3,), ( 3,)) self.check_message_to_parent((4,3), ( 3,), ( 3,), ( 3,)) # A complex test self.check_message_to_parent((7,6,5,4,3), ( 6,1,4,3), (1,1,5,4,1), ( 6,5,1,3)) # Test errors for inconsistent shapes self.assertRaises(ValueError, self.check_message_to_parent, (3,), (1,3,), (3,), (3,)) ## self.assertRaises(ValueError, ## self.check_message_to_parent, ## (3,), ## (3,), ## (1,3,), ## (3,)) self.assertRaises(ValueError, self.check_message_to_parent, (3,), (1,3,), (1,3,), (3,)) self.assertRaises(ValueError, self.check_message_to_parent, (3,), (4,), (3,), (3,)) self.assertRaises(ValueError, self.check_message_to_parent, (3,), (3,), (4,), (3,)) self.assertRaises(ValueError, self.check_message_to_parent, (3,), (4,), (4,), (3,)) self.assertRaises(ValueError, self.check_message_to_parent, (3,), (4,), (3,), (1,)) self.assertRaises(ValueError, self.check_message_to_parent, (3,), (3,), (4,), (1,)) self.assertRaises(ValueError, self.check_message_to_parent, (3,), (4,), (4,), (1,)) self.assertRaises(ValueError, self.check_message_to_parent, (1,), (4,), (3,), (1,)) self.assertRaises(ValueError, self.check_message_to_parent, (1,), (3,), (4,), (1,)) self.assertRaises(ValueError, self.check_message_to_parent, (1,), (4,), (4,), (1,)) def test_compute_message(self): """ Test the general sum-multiply function for message computations """ self.assertAllClose(Node._compute_message(3, plates_from=(), plates_to=(), ndim=0), 3) # Sum over one array self.assertAllClose(Node._compute_message([1, 2, 3], plates_from=(3,), plates_to=(), ndim=0), 6) # Sum plates self.assertAllClose(Node._compute_message([1, 2, 3], [4, 4, 4], [5, 5, 5], plates_from=(3,), plates_to=(), ndim=0), 20+40+60) # Do not sum plates self.assertAllClose(Node._compute_message([1, 2, 3], [4, 4, 4], [5, 5, 5], plates_from=(3,), plates_to=(3,), ndim=0), [20, 40, 60]) # Give ndim self.assertAllClose(Node._compute_message([1, 2, 3], [4, 4, 4], [5, 5, 5], plates_from=(), plates_to=(), ndim=1), [20, 40, 60]) # Broadcast plates_from self.assertAllClose(Node._compute_message(3, 4, 5, plates_from=(3,), plates_to=(), ndim=0), 3 * (3*4*5)) # Broadcast plates_to self.assertAllClose(Node._compute_message(3, 4, 5, plates_from=(3,), plates_to=(3,), ndim=0), 3*4*5) # Different ndims self.assertAllClose(Node._compute_message([1, 2, 3], [4], 5, plates_from=(3,), plates_to=(3,), ndim=0), [1*4*5, 2*4*5, 3*4*5]) # Broadcasting dims for some arrays self.assertAllClose(Node._compute_message([1, 2, 3], [4], 5, plates_from=(), plates_to=(), ndim=1), [1*4*5, 2*4*5, 3*4*5]) # Complex example x1 = np.random.randn(5,4,1,2,1) x2 = np.random.randn( 1,2,1) x3 = np.random.randn(5,1,1,1,1) self.assertAllClose(Node._compute_message(x1, x2, x3, plates_from=(6,5,4,3), plates_to=(5,1,3), ndim=2), 6*np.sum(x1*x2*x3, axis=(-4,), keepdims=True)) pass class TestSlice(misc.TestCase): def test_init(self): """ Test the constructor of the X[..] node operator. """ class MyNode(Node): _moments = Moments() def _get_id_list(self): return [] # Integer index X = MyNode(plates=(3,4), dims=((),)) Y = X[2,-4] self.assertEqual(Y.plates, ()) X = MyNode(plates=(3,4,5), dims=((),)) Y = X[2,1] self.assertEqual(Y.plates, (5,)) # Full slices X = MyNode(plates=(3,4,5), dims=((),)) Y = X[:,1,:] self.assertEqual(Y.plates, (3,5,)) X = MyNode(plates=(3,4,5), dims=((),)) Y = X[1,:,:] self.assertEqual(Y.plates, (4,5,)) X = MyNode(plates=(3,4,5), dims=((),)) Y = X[:,:,1] self.assertEqual(Y.plates, (3,4,)) # Slice with step X = MyNode(plates=(9,), dims=((),)) Y = X[::3] self.assertEqual(Y.plates, (3,)) X = MyNode(plates=(10,), dims=((),)) Y = X[::3] self.assertEqual(Y.plates, (4,)) X = MyNode(plates=(11,), dims=((),)) Y = X[::3] self.assertEqual(Y.plates, (4,)) # Slice with a start value X = MyNode(plates=(10,), dims=((),)) Y = X[3:] self.assertEqual(Y.plates, (7,)) # Slice with an end value X = MyNode(plates=(10,), dims=((),)) Y = X[:7] self.assertEqual(Y.plates, (7,)) # Slice with only one element X = MyNode(plates=(10,), dims=((),)) Y = X[6:7] self.assertEqual(Y.plates, (1,)) # Slice starts out of range X = MyNode(plates=(10,), dims=((),)) Y = X[-20:] self.assertEqual(Y.plates, (10,)) # Slice ends out of range X = MyNode(plates=(10,), dims=((),)) Y = X[:20] self.assertEqual(Y.plates, (10,)) # Counter-intuitive: This slice is not empty X = MyNode(plates=(3,), dims=((),)) Y = X[-4::4] self.assertEqual(Y.plates, (1,)) # One ellipsis X = MyNode(plates=(3,4,5,6), dims=((),)) Y = X[...,2,1] self.assertEqual(Y.plates, (3,4)) X = MyNode(plates=(3,4,5,6), dims=((),)) Y = X[2,...,1] self.assertEqual(Y.plates, (4,5)) X = MyNode(plates=(3,4,5,6), dims=((),)) Y = X[2,1,...] self.assertEqual(Y.plates, (5,6)) # Multiple ellipsis X = MyNode(plates=(3,4,5), dims=((),)) Y = X[...,2,...] self.assertEqual(Y.plates, (3,5)) X = MyNode(plates=(3,4,5), dims=((),)) Y = X[...,2,...,...] self.assertEqual(Y.plates, (4,5)) X = MyNode(plates=(3,4,5), dims=((),)) Y = X[...,...,...,...] self.assertEqual(Y.plates, (3,4,5)) # New axis X = MyNode(plates=(3,), dims=((),)) Y = X[None] self.assertEqual(Y.plates, (1,3)) X = MyNode(plates=(3,), dims=((),)) Y = X[:,None] self.assertEqual(Y.plates, (3,1)) X = MyNode(plates=(3,4), dims=((),)) Y = X[None,:,None,:] self.assertEqual(Y.plates, (1,3,1,4)) # # Test errors # class Z: def __getitem__(self, obj): return obj # Invalid argument self.assertRaises(TypeError, MyNode(plates=(3,), dims=((),)).__getitem__, Z()['a']) self.assertRaises(TypeError, MyNode(plates=(3,), dims=((),)).__getitem__, Z()[[2,1]]) # Too many indices self.assertRaises(IndexError, MyNode(plates=(3,), dims=((),)).__getitem__, Z()[:,:]) self.assertRaises(IndexError, MyNode(plates=(3,), dims=((),)).__getitem__, Z()[...,...,...]) # Index out of range self.assertRaises(IndexError, MyNode(plates=(3,), dims=((),)).__getitem__, Z()[3]) self.assertRaises(IndexError, MyNode(plates=(3,), dims=((),)).__getitem__, Z()[-4]) # Empty slice self.assertRaises(IndexError, MyNode(plates=(3,), dims=((),)).__getitem__, Z()[3:]) self.assertRaises(IndexError, MyNode(plates=(3,), dims=((),)).__getitem__, Z()[:-3]) pass def test_message_to_child(self): """ Test message to child of X[..] node operator. """ class DummyNode(Node): _moments = Moments() _parent_moments = (Moments(),) def __init__(self, u, **kwargs): self.u = u super().__init__(**kwargs) def _message_to_child(self): return self.u def _get_id_list(self): return [] # Message not a reference to X.u but a copy of it X = DummyNode([np.random.randn(3)], plates=(3,), dims=((),)) Y = X[2] self.assertTrue(Y._message_to_child() is not X.u, msg="Slice node operator sends a reference to the " "node's moment list as a message instead of a copy " "of the list.") # Integer indices X = DummyNode([np.random.randn(3,4)], plates=(3,4), dims=((),)) Y = X[2,1] self.assertMessageToChild(Y, [X.u[0][2,1]]) # Too few integer indices X = DummyNode([np.random.randn(3,4)], plates=(3,4), dims=((),)) Y = X[2] self.assertMessageToChild(Y, [X.u[0][2]]) # Integer for broadcasted moment X = DummyNode([np.random.randn(4)], plates=(3,4), dims=((),)) Y = X[2,1] self.assertMessageToChild(Y, [X.u[0][1]]) X = DummyNode([np.random.randn(4,1)], plates=(3,4), dims=((),)) Y = X[2,1] self.assertMessageToChild(Y, [X.u[0][2,0]]) # Ignore leading new axes X = DummyNode([np.random.randn(3)], plates=(3,), dims=((),)) Y = X[None,None,2] self.assertMessageToChild(Y, [X.u[0][2]]) # Ignore new axes before missing+broadcasted plate axes X = DummyNode([np.random.randn(3)], plates=(4,3,), dims=((),)) Y = X[1,None,None,2] self.assertMessageToChild(Y, [X.u[0][2]]) # New axes X = DummyNode([np.random.randn(3,4)], plates=(3,4), dims=((),)) Y = X[2,None,None,1] self.assertMessageToChild(Y, [X.u[0][2,None,None,1]]) # New axes for broadcasted axes X = DummyNode([np.random.randn(4)], plates=(3,4), dims=((),)) Y = X[2,1,None,None] self.assertMessageToChild(Y, [X.u[0][1,None,None]]) # Full slice X = DummyNode([np.random.randn(3,4)], plates=(3,4), dims=((),)) Y = X[:,2] self.assertMessageToChild(Y, [X.u[0][:,2]]) # Slice with start X = DummyNode([np.random.randn(3,4)], plates=(3,4), dims=((),)) Y = X[1:,2] self.assertMessageToChild(Y, [X.u[0][1:,2]]) # Slice with end X = DummyNode([np.random.randn(3,4)], plates=(3,4), dims=((),)) Y = X[:2,2] self.assertMessageToChild(Y, [X.u[0][:2,2]]) # Slice with step X = DummyNode([np.random.randn(3,4)], plates=(3,4), dims=((),)) Y = X[::2,2] self.assertMessageToChild(Y, [X.u[0][::2,2]]) # Slice for broadcasted axes X = DummyNode([np.random.randn(4)], plates=(3,4), dims=((),)) Y = X[0:2:2,2] self.assertMessageToChild(Y, [X.u[0][2]]) X = DummyNode([np.random.randn(1,4)], plates=(3,4), dims=((),)) Y = X[0:2:2,2] self.assertMessageToChild(Y, [X.u[0][0:1,2]]) # One ellipsis X = DummyNode([np.random.randn(3,4)], plates=(3,4), dims=((),)) Y = X[...,2] self.assertMessageToChild(Y, [X.u[0][...,2]]) # Ellipsis over broadcasted axes X = DummyNode([np.random.randn(5,6)], plates=(3,4,5,6), dims=((),)) Y = X[1,...,2] self.assertMessageToChild(Y, [X.u[0][:,2]]) X = DummyNode([np.random.randn(3,1,5,6)], plates=(3,4,5,6), dims=((),)) Y = X[1,...,2] self.assertMessageToChild(Y, [X.u[0][1,:,:,2]]) # Multiple ellipsis X = DummyNode([np.random.randn(2,3,4,5)], plates=(2,3,4,5), dims=((),)) Y = X[...,2,...] self.assertMessageToChild(Y, [X.u[0][:,:,2,:]]) # Ellipsis when dimensions X = DummyNode([np.random.randn(2,3,4)], plates=(2,3), dims=((4,),)) Y = X[...,2] self.assertMessageToChild(Y, [X.u[0][:,2,:]]) # Indexing for multiple moments X = DummyNode([np.random.randn(2,3,4), np.random.randn(2,3)], plates=(2,3), dims=((4,),())) Y = X[1,1] self.assertMessageToChild(Y, [X.u[0][1,1], X.u[1][1,1]]) pass def test_message_to_parent(self): """ Test message to parent of X[..] node operator. """ class ParentNode(Node): _moments = Moments() def _get_id_list(self): return [] class ChildNode(Node): _moments = Moments() _parent_moments = (Moments(),) def __init__(self, X, m, mask, **kwargs): super().__init__(X, **kwargs) self.m = m self.mask2 = mask def _message_to_parent(self, index): return self.m def _mask_to_parent(self, index): return self.mask2 def _get_id_list(self): return [] # General broadcasting V = ParentNode(plates=(3,3,3), dims=((),)) X = V[...] m = [ np.random.randn(3,1) ] msg = [ np.zeros((1,3,1)) ] msg[0][:,:,:] = m[0] Y = ChildNode(X, m, True, dims=((),)) X._update_mask() self.assertMessage(X._message_to_parent(0), msg) # Integer indices V = ParentNode(plates=(3,4), dims=((),)) X = V[2,1] m = [np.random.randn()] msg = [ np.zeros((3,4)) ] msg[0][2,1] = m[0] Y = ChildNode(X, m, True, dims=((),)) X._update_mask() self.assertMessage(X._message_to_parent(0), msg) # Integer indices with broadcasting V = ParentNode(plates=(3,3), dims=((),)) X = V[2,2] m = [ np.random.randn(1) ] msg = [ np.zeros((3,3)) ] msg[0][2,2] = m[0] Y = ChildNode(X, m, True, dims=((),)) X._update_mask() self.assertMessage(X._message_to_parent(0), msg) # Slice indices V = ParentNode(plates=(2,3,4,5), dims=((),)) X = V[:,:2,1:,::2] m = [np.random.randn(2,2,3,3)] msg = [ np.zeros((2,3,4,5)) ] msg[0][:,:2,1:,::2] = m[0] Y = ChildNode(X, m, True, dims=((),)) X._update_mask() self.assertMessage(X._message_to_parent(0), msg) # Full slice with broadcasting V = ParentNode(plates=(2,3), dims=((),)) X = V[:,:] m = [np.random.randn(1)] msg = [ np.zeros((1,1)) ] msg[0][:] = m[0] Y = ChildNode(X, m, True, dims=((),)) X._update_mask() self.assertMessage(X._message_to_parent(0), msg) # Start slice with broadcasting V = ParentNode(plates=(3,3,3,3), dims=((),)) X = V[0:,1:,-2:,-3:] m = [np.random.randn(1,1)] msg = [ np.zeros((1,3,3,1)) ] msg[0][:,1:,-2:,:] = m[0] Y = ChildNode(X, m, True, dims=((),)) X._update_mask() self.assertMessage(X._message_to_parent(0), msg) # End slice with broadcasting V = ParentNode(plates=(3,3,3,3), dims=((),)) X = V[:2,:3,:4,:-1] m = [np.random.randn(1,1)] msg = [ np.zeros((3,1,1,3)) ] msg[0][:2,:,:,:-1] = m[0] Y = ChildNode(X, m, True, dims=((),)) X._update_mask() self.assertMessage(X._message_to_parent(0), msg) # Step slice with broadcasting V = ParentNode(plates=(3,3,1), dims=((),)) X = V[::1,::2,::2] m = [np.random.randn(1)] msg = [ np.zeros((1,3,1)) ] msg[0][:,::2,:] = m[0] Y = ChildNode(X, m, True, dims=((),)) X._update_mask() self.assertMessage(X._message_to_parent(0), msg) # Ellipsis V = ParentNode(plates=(3,3,3), dims=((),)) X = V[...,0] m = [np.random.randn(3,3)] msg = [ np.zeros((3,3,3)) ] msg[0][:,:,0] = m[0] Y = ChildNode(X, m, True, dims=((),)) X._update_mask() self.assertMessage(X._message_to_parent(0), msg) # New axes V = ParentNode(plates=(3,3), dims=((),)) X = V[None,:,None,None,:] m = [np.random.randn(1,3,1,1,3)] msg = [ np.zeros((3,3)) ] msg[0][:,:] = m[0][0,:,0,0,:] Y = ChildNode(X, m, True, dims=((),)) X._update_mask() self.assertMessage(X._message_to_parent(0), msg) # New axes with broadcasting V = ParentNode(plates=(3,3), dims=((),)) X = V[None,:,None,:,None] m = [np.random.randn(1,3,1)] msg = [ np.zeros((1,3)) ] msg[0][:,:] = m[0][0,:,0] Y = ChildNode(X, m, True, dims=((),)) X._update_mask() self.assertMessage(X._message_to_parent(0), msg) # Multiple messages V = ParentNode(plates=(3,), dims=((),())) X = V[:] m = [np.random.randn(3), np.random.randn(3)] msg = [ np.zeros((3)), np.zeros((3)) ] msg[0][:] = m[0][:] msg[1][:] = m[1][:] Y = ChildNode(X, m, True, dims=((),())) X._update_mask() self.assertMessage(X._message_to_parent(0), msg) # Non-scalar variables V = ParentNode(plates=(2,3), dims=((4,),)) X = V[...] m = [np.random.randn(2,3,4)] msg = [ np.zeros((2,3,4)) ] msg[0][:,:,:] = m[0][:,:,:] Y = ChildNode(X, m, True, dims=((4,),)) X._update_mask() self.assertMessage(X._message_to_parent(0), msg) # Missing values V = ParentNode(plates=(3,3,3), dims=((3,),)) X = V[:,0,::2,None] m = [np.random.randn(3,2,1,3)] # mask shape: (3, 2, 1) mask = np.array([ [[True], [False]], [[False], [False]], [[False], [True]] ]) msg = [ np.zeros((3,3,3,3)) ] msg[0][:,0,::2,:] = (m[0] * mask[...,None])[:,:,0,:] Y = ChildNode(X, m, mask, dims=((3,),)) X._update_mask() self.assertMessage(X._message_to_parent(0), msg) # Found bug: int index after slice V = ParentNode(plates=(3,3), dims=((),)) X = V[:,0] m = [np.random.randn(3)] msg = [ np.zeros((3,3)) ] msg[0][:,0] = m[0] Y = ChildNode(X, m, True, dims=((),)) X._update_mask() self.assertMessage(X._message_to_parent(0), msg) # Found bug: message requires reshaping after reverse indexing pass bayespy-0.4.0/bayespy/inference/vmp/nodes/tests/test_dirichlet.py0000644000175000017500000000436312615643655026776 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for `dirichlet` module. """ import numpy as np from scipy import special from bayespy.nodes import Dirichlet from bayespy.utils import random from bayespy.utils.misc import TestCase class TestDirichlet(TestCase): """ Unit tests for Dirichlet node """ def test_init(self): """ Test the creation of Dirichlet nodes. """ # Some simple initializations p = Dirichlet([1.5, 4.2, 3.5]) # Check that plates are correct p = Dirichlet([2, 3, 4], plates=(4,3)) self.assertEqual(p.plates, (4,3)) p = Dirichlet(np.ones((4,3,5))) self.assertEqual(p.plates, (4,3)) # Parent not a vector self.assertRaises(ValueError, Dirichlet, 4) # Parent vector has invalid values self.assertRaises(ValueError, Dirichlet, [-2,3,1]) # Plates inconsistent self.assertRaises(ValueError, Dirichlet, np.ones((4,3)), plates=(3,)) # Explicit plates too small self.assertRaises(ValueError, Dirichlet, np.ones((4,3)), plates=(1,)) pass def test_moments(self): """ Test the moments of Dirichlet nodes. """ p = Dirichlet([2, 3, 4]) u = p._message_to_child() self.assertAllClose(u[0], special.psi([2,3,4]) - special.psi(2+3+4)) pass def test_constant(self): """ Test the constant moments of Dirichlet nodes. """ p = Dirichlet([1, 1, 1]) p.initialize_from_value([0.5, 0.4, 0.1]) u = p._message_to_child() self.assertAllClose(u[0], np.log([0.5, 0.4, 0.1])) pass bayespy-0.4.0/bayespy/inference/vmp/nodes/tests/test_gate.py0000644000175000017500000002501512615643655025744 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for `gate` module. """ import numpy as np from bayespy.nodes import (Gate, GaussianARD, Gamma, Categorical, Bernoulli, Multinomial) from bayespy.inference.vmp.nodes.gaussian import GaussianMoments from bayespy.utils import random from bayespy.utils import linalg from bayespy.utils.misc import TestCase class TestGate(TestCase): """ Unit tests for Gate node. """ def test_init(self): """ Test the creation of Gate node """ # Gating scalar node Z = Categorical(np.ones(3)/3) X = GaussianARD(0, 1, shape=(), plates=(3,)) Y = Gate(Z, X) self.assertEqual(Y.plates, ()) self.assertEqual(Y.dims, ( (), () )) # Gating non-scalar node Z = Categorical(np.ones(3)/3) X = GaussianARD(0, 1, shape=(2,), plates=(3,)) Y = Gate(Z, X) self.assertEqual(Y.plates, ()) self.assertEqual(Y.dims, ( (2,), (2,2) )) # Plates from Z Z = Categorical(np.ones(3)/3, plates=(4,)) X = GaussianARD(0, 1, shape=(2,), plates=(3,)) Y = Gate(Z, X) self.assertEqual(Y.plates, (4,)) self.assertEqual(Y.dims, ( (2,), (2,2) )) # Plates from X Z = Categorical(np.ones(3)/3) X = GaussianARD(0, 1, shape=(2,), plates=(4,3)) Y = Gate(Z, X) self.assertEqual(Y.plates, (4,)) self.assertEqual(Y.dims, ( (2,), (2,2) )) # Plates from Z and X Z = Categorical(np.ones(3)/3, plates=(5,)) X = GaussianARD(0, 1, shape=(2,), plates=(4,1,3)) Y = Gate(Z, X) self.assertEqual(Y.plates, (4,5)) self.assertEqual(Y.dims, ( (2,), (2,2) )) # Gating non-default plate Z = Categorical(np.ones(3)/3) X = GaussianARD(0, 1, shape=(), plates=(3,4)) Y = Gate(Z, X, gated_plate=-2) self.assertEqual(Y.plates, (4,)) self.assertEqual(Y.dims, ( (), () )) # Fixed gating Z = 2 X = GaussianARD(0, 1, shape=(2,), plates=(3,)) Y = Gate(Z, X) self.assertEqual(Y.plates, ()) self.assertEqual(Y.dims, ( (2,), (2,2) )) # Fixed X Z = Categorical(np.ones(3)/3) X = [1, 2, 3] Y = Gate(Z, X, moments=GaussianMoments(0)) self.assertEqual(Y.plates, ()) self.assertEqual(Y.dims, ( (), () )) # Do not accept non-negative cluster plates Z = Categorical(np.ones(3)/3) X = GaussianARD(0, 1, plates=(3,)) self.assertRaises(ValueError, Gate, Z, X, gated_plate=0) # None of the parents have the cluster plate axis Z = Categorical(np.ones(3)/3) X = GaussianARD(0, 1) self.assertRaises(ValueError, Gate, Z, X) # Inconsistent cluster plate Z = Categorical(np.ones(3)/3) X = GaussianARD(0, 1, plates=(2,)) self.assertRaises(ValueError, Gate, Z, X) pass def test_message_to_child(self): """ Test the message to child of Gate node. """ # Gating scalar node Z = 2 X = GaussianARD([1,2,3], 1, shape=(), plates=(3,)) Y = Gate(Z, X) u = Y._message_to_child() self.assertEqual(len(u), 2) self.assertAllClose(u[0], 3) self.assertAllClose(u[1], 3**2+1) # Fixed X Z = 2 X = [1, 2, 3] Y = Gate(Z, X, moments=GaussianMoments(0)) u = Y._message_to_child() self.assertEqual(len(u), 2) self.assertAllClose(u[0], 3) self.assertAllClose(u[1], 3**2) # Uncertain gating Z = Categorical([0.2,0.3,0.5]) X = GaussianARD([1,2,3], 1, shape=(), plates=(3,)) Y = Gate(Z, X) u = Y._message_to_child() self.assertAllClose(u[0], 0.2*1 + 0.3*2 + 0.5*3) self.assertAllClose(u[1], 0.2*2 + 0.3*5 + 0.5*10) # Plates in Z Z = [2, 0] X = GaussianARD([1,2,3], 1, shape=(), plates=(3,)) Y = Gate(Z, X) u = Y._message_to_child() self.assertAllClose(u[0], [3, 1]) self.assertAllClose(u[1], [10, 2]) # Plates in X Z = 2 X = GaussianARD([1,2,3], 1, shape=(), plates=(4,3,)) Y = Gate(Z, X) u = Y._message_to_child() self.assertAllClose(np.ones(4)*u[0], np.ones(4)*3) self.assertAllClose(np.ones(4)*u[1], np.ones(4)*10) # Gating non-default plate Z = 2 X = GaussianARD([[1],[2],[3]], 1, shape=(), plates=(3,4)) Y = Gate(Z, X, gated_plate=-2) u = Y._message_to_child() self.assertAllClose(np.ones(4)*u[0], np.ones(4)*3) self.assertAllClose(np.ones(4)*u[1], np.ones(4)*10) # Gating non-scalar node Z = 2 X = GaussianARD([1*np.ones(4), 2*np.ones(4), 3*np.ones(4)], 1, shape=(4,), plates=(3,)) Y = Gate(Z, X) u = Y._message_to_child() self.assertAllClose(u[0], 3*np.ones(4)) self.assertAllClose(u[1], 9*np.ones((4,4)) + 1*np.identity(4)) # Broadcasting the moments on the cluster axis Z = 2 X = GaussianARD(1, 1, shape=(), plates=(3,)) Y = Gate(Z, X) u = Y._message_to_child() self.assertEqual(len(u), 2) self.assertAllClose(u[0], 1) self.assertAllClose(u[1], 1**2+1) pass def test_message_to_parent(self): """ Test the message to parents of Gate node. """ # Unobserved and broadcasting Z = 2 X = GaussianARD(0, 1, shape=(), plates=(3,)) F = Gate(Z, X) Y = GaussianARD(F, 1) m = F._message_to_parent(0) self.assertEqual(len(m), 1) self.assertAllClose(m[0], 0*np.ones(3)) m = F._message_to_parent(1) self.assertEqual(len(m), 2) self.assertAllClose(m[0]*np.ones(3), [0, 0, 0]) self.assertAllClose(m[1]*np.ones(3), [0, 0, 0]) # Gating scalar node Z = 2 X = GaussianARD([1,2,3], 1, shape=(), plates=(3,)) F = Gate(Z, X) Y = GaussianARD(F, 1) Y.observe(10) m = F._message_to_parent(0) self.assertAllClose(m[0], [10*1-0.5*2, 10*2-0.5*5, 10*3-0.5*10]) m = F._message_to_parent(1) self.assertAllClose(m[0], [0, 0, 10]) self.assertAllClose(m[1], [0, 0, -0.5]) # Fixed X Z = 2 X = [1,2,3] F = Gate(Z, X, moments=GaussianMoments(0)) Y = GaussianARD(F, 1) Y.observe(10) m = F._message_to_parent(0) self.assertAllClose(m[0], [10*1-0.5*1, 10*2-0.5*4, 10*3-0.5*9]) m = F._message_to_parent(1) self.assertAllClose(m[0], [0, 0, 10]) self.assertAllClose(m[1], [0, 0, -0.5]) # Uncertain gating Z = Categorical([0.2, 0.3, 0.5]) X = GaussianARD([1,2,3], 1, shape=(), plates=(3,)) F = Gate(Z, X) Y = GaussianARD(F, 1) Y.observe(10) m = F._message_to_parent(0) self.assertAllClose(m[0], [10*1-0.5*2, 10*2-0.5*5, 10*3-0.5*10]) m = F._message_to_parent(1) self.assertAllClose(m[0], [0.2*10, 0.3*10, 0.5*10]) self.assertAllClose(m[1], [-0.5*0.2, -0.5*0.3, -0.5*0.5]) # Plates in Z Z = [2, 0] X = GaussianARD([1,2,3], 1, shape=(), plates=(3,)) F = Gate(Z, X) Y = GaussianARD(F, 1) Y.observe([10, 20]) m = F._message_to_parent(0) self.assertAllClose(m[0], [[10*1-0.5*2, 10*2-0.5*5, 10*3-0.5*10], [20*1-0.5*2, 20*2-0.5*5, 20*3-0.5*10]]) m = F._message_to_parent(1) self.assertAllClose(m[0], [20, 0, 10]) self.assertAllClose(m[1], [-0.5, 0, -0.5]) # Plates in X Z = 2 X = GaussianARD([[1,2,3], [4,5,6]], 1, shape=(), plates=(2,3,)) F = Gate(Z, X) Y = GaussianARD(F, 1) Y.observe([10, 20]) m = F._message_to_parent(0) self.assertAllClose(m[0], [10*1-0.5*2 + 20*4-0.5*17, 10*2-0.5*5 + 20*5-0.5*26, 10*3-0.5*10 + 20*6-0.5*37]) m = F._message_to_parent(1) self.assertAllClose(m[0], [[0, 0, 10], [0, 0, 20]]) self.assertAllClose(m[1]*np.ones((2,3)), [[0, 0, -0.5], [0, 0, -0.5]]) # Gating non-default plate Z = 2 X = GaussianARD([[1],[2],[3]], 1, shape=(), plates=(3,1)) F = Gate(Z, X, gated_plate=-2) Y = GaussianARD(F, 1) Y.observe([10]) m = F._message_to_parent(0) self.assertAllClose(m[0], [10*1-0.5*2, 10*2-0.5*5, 10*3-0.5*10]) m = F._message_to_parent(1) self.assertAllClose(m[0], [[0], [0], [10]]) self.assertAllClose(m[1], [[0], [0], [-0.5]]) # Gating non-scalar node Z = 2 X = GaussianARD([[1,4],[2,5],[3,6]], 1, shape=(2,), plates=(3,)) F = Gate(Z, X) Y = GaussianARD(F, 1) Y.observe([10,20]) m = F._message_to_parent(0) self.assertAllClose(m[0], [10*1-0.5*2 + 20*4-0.5*17, 10*2-0.5*5 + 20*5-0.5*26, 10*3-0.5*10 + 20*6-0.5*37]) m = F._message_to_parent(1) I = np.identity(2) self.assertAllClose(m[0], [[0,0], [0,0], [10,20]]) self.assertAllClose(m[1], [0*I, 0*I, -0.5*I]) # Broadcasting the moments on the cluster axis Z = 2 X = GaussianARD(2, 1, shape=(), plates=(3,)) F = Gate(Z, X) Y = GaussianARD(F, 1) Y.observe(10) m = F._message_to_parent(0) self.assertAllClose(m[0], [10*2-0.5*5, 10*2-0.5*5, 10*2-0.5*5]) m = F._message_to_parent(1) self.assertAllClose(m[0], [0, 0, 10]) self.assertAllClose(m[1], [0, 0, -0.5]) pass def test_mask_to_parent(self): """ Test the mask handling in Gate node """ pass bayespy-0.4.0/bayespy/inference/vmp/nodes/tests/test_categorical_markov_chain.py0000644000175000017500000001627112615643655032026 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for bayespy.inference.vmp.nodes.categorical_markov_chain module. """ import warnings import numpy as np from bayespy.utils import misc from bayespy.inference.vmp.nodes import CategoricalMarkovChain, \ Dirichlet class TestCategoricalMarkovChain(misc.TestCase): def test_init(self): """ Test the creation of CategoricalMarkovChain object """ # # Plates # p0 = np.random.dirichlet([1, 1]) P = np.random.dirichlet([1, 1], size=(3,2)) Z = CategoricalMarkovChain(p0, P) self.assertEqual((), Z.plates, msg="Incorrect plates") self.assertEqual(((2,),(3,2,2)), Z.dims, msg="Incorrect dimensions") p0 = np.random.dirichlet([1, 1], size=(4,)) P = np.random.dirichlet([1, 1], size=(3,2)) Z = CategoricalMarkovChain(p0, P) self.assertEqual((4,), Z.plates, msg="Incorrect plates") self.assertEqual(((2,),(3,2,2)), Z.dims, msg="Incorrect dimensions") p0 = np.random.dirichlet([1, 1]) P = np.random.dirichlet([1, 1], size=(4,3,2)) Z = CategoricalMarkovChain(p0, P) self.assertEqual((4,), Z.plates, msg="Incorrect plates") self.assertEqual(((2,),(3,2,2)), Z.dims, msg="Incorrect dimensions") # Test some overflow bugs p0 = np.array([0.5, 0.5]) P = Dirichlet(1e-3*np.ones(2), plates=(2,)) Z = CategoricalMarkovChain(p0, P, states=2000) u = Z._message_to_child() self.assertTrue(np.all(~np.isnan(u[0])), msg="Nans in moments") self.assertTrue(np.all(~np.isnan(u[1])), msg="Nans in moments") pass def test_message_to_child(self): """ Test the message of CategoricalMarkovChain to child """ with warnings.catch_warnings(): warnings.simplefilter("ignore", RuntimeWarning) # Deterministic oscillator p0 = np.array([1.0, 0.0]) P = np.array(3*[[[0.0, 1.0], [1.0, 0.0]]]) Z = CategoricalMarkovChain(p0, P) u = Z._message_to_child() self.assertAllClose(u[0], [1.0, 0]) self.assertAllClose(u[1], [ [[0.0, 1.0], [0.0, 0.0]], [[0.0, 0.0], [1.0, 0.0]], [[0.0, 1.0], [0.0, 0.0]] ]) # Maximum randomness p0 = np.array([0.5, 0.5]) P = np.array(3*[[[0.5, 0.5], [0.5, 0.5]]]) Z = CategoricalMarkovChain(p0, P) u = Z._message_to_child() self.assertAllClose(u[0], [0.5, 0.5]) self.assertAllClose(u[1], [ [[0.25, 0.25], [0.25, 0.25]], [[0.25, 0.25], [0.25, 0.25]], [[0.25, 0.25], [0.25, 0.25]] ]) # Random init, deterministic dynamics p0 = np.array([0.5, 0.5]) P = np.array(3*[[[0, 1], [1, 0]]]) Z = CategoricalMarkovChain(p0, P) u = Z._message_to_child() self.assertAllClose(u[0], [0.5, 0.5]) self.assertAllClose(u[1], [ [[0.0, 0.5], [0.5, 0.0]], [[0.0, 0.5], [0.5, 0.0]], [[0.0, 0.5], [0.5, 0.0]] ]) # Test plates p0 = np.array([ [1.0, 0.0], [0.5, 0.5] ]) P = np.array([ [ [[0.0, 1.0], [1.0, 0.0]] ], [ [[0.5, 0.5], [0.5, 0.5]] ] ]) Z = CategoricalMarkovChain(p0, P) u = Z._message_to_child() self.assertAllClose(u[0], [[1.0, 0.0], [0.5, 0.5]]) self.assertAllClose(u[1], [ [ [[0.0, 1.0], [0.0, 0.0]] ], [ [[0.25, 0.25], [0.25, 0.25]] ] ]) # Test broadcasted state transition probabilities p0 = np.array([1.0, 0.0]) P = Dirichlet([1e-10, 1e10], plates=(3,2)) Z = CategoricalMarkovChain(p0, P) u = Z._message_to_child() self.assertAllClose(u[0], [1.0, 0]) self.assertAllClose(u[1], [ [[0.0, 1.0], [0.0, 0.0]], [[0.0, 0.0], [0.0, 1.0]], [[0.0, 0.0], [0.0, 1.0]] ]) pass def test_random(self): """ Test random sampling of categorical Markov chain """ with warnings.catch_warnings(): warnings.simplefilter("ignore", RuntimeWarning) # Simple random sample Z = CategoricalMarkovChain([1, 0], [[0, 1], [1, 0]], states=3) z = Z.random() self.assertAllClose(z, [0, 1, 0]) # Draw sample with plates p0 = [ [1,0], [0,1] ] P = [ [ [[0,1], [1,0]] ], [ [[1,0], [0,1]] ] ] Z = CategoricalMarkovChain(p0, P, states=3) z = Z.random() self.assertAllClose(z, [[0, 1, 0], [1, 1, 1]]) # Draw sample with plates, parameters broadcasted Z = CategoricalMarkovChain([1, 0], [[0, 1], [1, 0]], states=3, plates=(3,4)) z = Z.random() self.assertAllClose(z, np.ones((3,4,1))*[0, 1, 0]) # Draw sample with time-varying transition matrix p0 = [1, 0] P = [ [[0,1], [1,0]], [[1,0], [0,1]], [[1, 0], [1, 0]] ] Z = CategoricalMarkovChain(p0, P, states=4) z = Z.random() self.assertAllClose(z, [0, 1, 1, 0]) pass bayespy-0.4.0/bayespy/inference/vmp/nodes/tests/test_concatenate.py0000644000175000017500000002340012615643655027304 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2015 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for `concatenate` module. """ import warnings import numpy as np from bayespy.nodes import (Concatenate, GaussianARD, Gamma) from bayespy.utils import random from bayespy.utils.misc import TestCase class TestConcatenate(TestCase): """ Unit tests for Concatenate node. """ def test_init(self): """ Test the creation of Concatenate node """ # One parent only X = GaussianARD(0, 1, plates=(3,), shape=()) Y = Concatenate(X) self.assertEqual(Y.plates, (3,)) self.assertEqual(Y.dims, ( (), () )) X = GaussianARD(0, 1, plates=(3,), shape=(2,4)) Y = Concatenate(X) self.assertEqual(Y.plates, (3,)) self.assertEqual(Y.dims, ( (2,4), (2,4,2,4) )) # Two parents X1 = GaussianARD(0, 1, plates=(2,), shape=()) X2 = GaussianARD(0, 1, plates=(3,), shape=()) Y = Concatenate(X1, X2) self.assertEqual(Y.plates, (5,)) self.assertEqual(Y.dims, ( (), () )) # Two parents with shapes X1 = GaussianARD(0, 1, plates=(2,), shape=(4,6)) X2 = GaussianARD(0, 1, plates=(3,), shape=(4,6)) Y = Concatenate(X1, X2) self.assertEqual(Y.plates, (5,)) self.assertEqual(Y.dims, ( (4,6), (4,6,4,6) )) # Two parents with non-default axis X1 = GaussianARD(0, 1, plates=(2,4), shape=()) X2 = GaussianARD(0, 1, plates=(3,4), shape=()) Y = Concatenate(X1, X2, axis=-2) self.assertEqual(Y.plates, (5,4)) self.assertEqual(Y.dims, ( (), () )) # Three parents X1 = GaussianARD(0, 1, plates=(2,), shape=()) X2 = GaussianARD(0, 1, plates=(3,), shape=()) X3 = GaussianARD(0, 1, plates=(4,), shape=()) Y = Concatenate(X1, X2, X3) self.assertEqual(Y.plates, (9,)) self.assertEqual(Y.dims, ( (), () )) # Constant parent X1 = [7.2, 3.5] X2 = GaussianARD(0, 1, plates=(3,), shape=()) Y = Concatenate(X1, X2) self.assertEqual(Y.plates, (5,)) self.assertEqual(Y.dims, ( (), () )) # Different moments X1 = GaussianARD(0, 1, plates=(3,)) X2 = Gamma(1, 1, plates=(4,)) self.assertRaises(ValueError, Concatenate, X1, X2) # Incompatible shapes X1 = GaussianARD(0, 1, plates=(3,), shape=(2,)) X2 = GaussianARD(0, 1, plates=(2,), shape=()) self.assertRaises(ValueError, Concatenate, X1, X2) # Incompatible plates X1 = GaussianARD(0, 1, plates=(4,3), shape=()) X2 = GaussianARD(0, 1, plates=(5,2,), shape=()) self.assertRaises(ValueError, Concatenate, X1, X2) pass def test_message_to_child(self): """ Test the message to child of Concatenate node. """ # Two parents without shapes X1 = GaussianARD(0, 1, plates=(2,), shape=()) X2 = GaussianARD(0, 1, plates=(3,), shape=()) Y = Concatenate(X1, X2) u1 = X1.get_moments() u2 = X2.get_moments() u = Y.get_moments() self.assertAllClose((u[0]*np.ones((5,)))[:2], u1[0]*np.ones((2,))) self.assertAllClose((u[1]*np.ones((5,)))[:2], u1[1]*np.ones((2,))) self.assertAllClose((u[0]*np.ones((5,)))[2:], u2[0]*np.ones((3,))) self.assertAllClose((u[1]*np.ones((5,)))[2:], u2[1]*np.ones((3,))) # Two parents with shapes X1 = GaussianARD(0, 1, plates=(2,), shape=(4,)) X2 = GaussianARD(0, 1, plates=(3,), shape=(4,)) Y = Concatenate(X1, X2) u1 = X1.get_moments() u2 = X2.get_moments() u = Y.get_moments() self.assertAllClose((u[0]*np.ones((5,4)))[:2], u1[0]*np.ones((2,4))) self.assertAllClose((u[1]*np.ones((5,4,4)))[:2], u1[1]*np.ones((2,4,4))) self.assertAllClose((u[0]*np.ones((5,4)))[2:], u2[0]*np.ones((3,4))) self.assertAllClose((u[1]*np.ones((5,4,4)))[2:], u2[1]*np.ones((3,4,4))) # Test with non-constant axis X1 = GaussianARD(0, 1, plates=(2,4), shape=()) X2 = GaussianARD(0, 1, plates=(3,4), shape=()) Y = Concatenate(X1, X2, axis=-2) u1 = X1.get_moments() u2 = X2.get_moments() u = Y.get_moments() self.assertAllClose((u[0]*np.ones((5,4)))[:2], u1[0]*np.ones((2,4))) self.assertAllClose((u[1]*np.ones((5,4)))[:2], u1[1]*np.ones((2,4))) self.assertAllClose((u[0]*np.ones((5,4)))[2:], u2[0]*np.ones((3,4))) self.assertAllClose((u[1]*np.ones((5,4)))[2:], u2[1]*np.ones((3,4))) # Test with constant parent X1 = np.random.randn(2, 4) X2 = GaussianARD(0, 1, plates=(3,), shape=(4,)) Y = Concatenate(X1, X2) u1 = Y.parents[0].get_moments() u2 = X2.get_moments() u = Y.get_moments() self.assertAllClose((u[0]*np.ones((5,4)))[:2], u1[0]*np.ones((2,4))) self.assertAllClose((u[1]*np.ones((5,4,4)))[:2], u1[1]*np.ones((2,4,4))) self.assertAllClose((u[0]*np.ones((5,4)))[2:], u2[0]*np.ones((3,4))) self.assertAllClose((u[1]*np.ones((5,4,4)))[2:], u2[1]*np.ones((3,4,4))) pass def test_message_to_parent(self): """ Test the message to parents of Concatenate node. """ # Two parents without shapes X1 = GaussianARD(0, 1, plates=(2,), shape=()) X2 = GaussianARD(0, 1, plates=(3,), shape=()) Z = Concatenate(X1, X2) Y = GaussianARD(Z, 1) Y.observe(np.random.randn(*Y.get_shape(0))) m1 = X1._message_from_children() m2 = X2._message_from_children() m = Z._message_from_children() self.assertAllClose((m[0]*np.ones((5,)))[:2], m1[0]*np.ones((2,))) self.assertAllClose((m[1]*np.ones((5,)))[:2], m1[1]*np.ones((2,))) self.assertAllClose((m[0]*np.ones((5,)))[2:], m2[0]*np.ones((3,))) self.assertAllClose((m[1]*np.ones((5,)))[2:], m2[1]*np.ones((3,))) # Two parents with shapes with warnings.catch_warnings(): warnings.simplefilter("ignore", FutureWarning) X1 = GaussianARD(0, 1, plates=(2,), shape=(4,6)) X2 = GaussianARD(0, 1, plates=(3,), shape=(4,6)) Z = Concatenate(X1, X2) Y = GaussianARD(Z, 1) Y.observe(np.random.randn(*Y.get_shape(0))) m1 = X1._message_from_children() m2 = X2._message_from_children() m = Z._message_from_children() self.assertAllClose((m[0]*np.ones((5,4,6)))[:2], m1[0]*np.ones((2,4,6))) self.assertAllClose((m[1]*np.ones((5,4,6,4,6)))[:2], m1[1]*np.ones((2,4,6,4,6))) self.assertAllClose((m[0]*np.ones((5,4,6)))[2:], m2[0]*np.ones((3,4,6))) self.assertAllClose((m[1]*np.ones((5,4,6,4,6)))[2:], m2[1]*np.ones((3,4,6,4,6))) # Two parents with non-default concatenation axis X1 = GaussianARD(0, 1, plates=(2,4), shape=()) X2 = GaussianARD(0, 1, plates=(3,4), shape=()) Z = Concatenate(X1, X2, axis=-2) Y = GaussianARD(Z, 1) Y.observe(np.random.randn(*Y.get_shape(0))) m1 = X1._message_from_children() m2 = X2._message_from_children() m = Z._message_from_children() self.assertAllClose((m[0]*np.ones((5,4)))[:2], m1[0]*np.ones((2,4))) self.assertAllClose((m[1]*np.ones((5,4)))[:2], m1[1]*np.ones((2,4))) self.assertAllClose((m[0]*np.ones((5,4)))[2:], m2[0]*np.ones((3,4))) self.assertAllClose((m[1]*np.ones((5,4)))[2:], m2[1]*np.ones((3,4))) # Constant parent X1 = np.random.randn(2,4,6) X2 = GaussianARD(0, 1, plates=(3,), shape=(4,6)) Z = Concatenate(X1, X2) Y = GaussianARD(Z, 1) Y.observe(np.random.randn(*Y.get_shape(0))) m1 = Z._message_to_parent(0) m2 = X2._message_from_children() m = Z._message_from_children() self.assertAllClose((m[0]*np.ones((5,4,6)))[:2], m1[0]*np.ones((2,4,6))) self.assertAllClose((m[1]*np.ones((5,4,6,4,6)))[:2], m1[1]*np.ones((2,4,6,4,6))) self.assertAllClose((m[0]*np.ones((5,4,6)))[2:], m2[0]*np.ones((3,4,6))) self.assertAllClose((m[1]*np.ones((5,4,6,4,6)))[2:], m2[1]*np.ones((3,4,6,4,6))) pass def test_mask_to_parent(self): """ Test the mask handling in Concatenate node """ pass bayespy-0.4.0/bayespy/inference/vmp/nodes/tests/test_mixture.py0000644000175000017500000002342212615643655026521 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for mixture module. """ import warnings import numpy as np from bayespy.nodes import (GaussianARD, Gamma, Mixture, Categorical, Bernoulli, Multinomial) from bayespy.utils import random from bayespy.utils import linalg from bayespy.utils.misc import TestCase class TestMixture(TestCase): def test_init(self): """ Test the creation of Mixture node """ # Do not accept non-negative cluster plates z = Categorical(np.random.dirichlet([1,1])) self.assertRaises(ValueError, Mixture, z, GaussianARD, GaussianARD(0, 1, plates=(2,)), Gamma(1, 1, plates=(2,)), cluster_plate=0) # Try constructing a mixture without any of the parents having the # cluster plate axis z = Categorical(np.random.dirichlet([1,1])) self.assertRaises(ValueError, Mixture, z, GaussianARD, GaussianARD(0, 1, plates=()), Gamma(1, 1, plates=())) def test_message_to_child(self): """ Test the message to child of Mixture node. """ K = 3 # # Estimate moments from parents only # # Simple case mu = GaussianARD([0,2,4], 1, ndim=0, plates=(K,)) alpha = Gamma(1, 1, plates=(K,)) z = Categorical(np.ones(K)/K) X = Mixture(z, GaussianARD, mu, alpha) self.assertEqual(X.plates, ()) self.assertEqual(X.dims, ( (), () )) u = X._message_to_child() self.assertAllClose(u[0], 2) self.assertAllClose(u[1], 2**2+1) # Broadcasting the moments on the cluster axis mu = GaussianARD(2, 1, ndim=0, plates=(K,)) alpha = Gamma(1, 1, plates=(K,)) z = Categorical(np.ones(K)/K) X = Mixture(z, GaussianARD, mu, alpha) self.assertEqual(X.plates, ()) self.assertEqual(X.dims, ( (), () )) u = X._message_to_child() self.assertAllClose(u[0], 2) self.assertAllClose(u[1], 2**2+1) # # Estimate moments with observed children # pass def test_message_to_parent(self): """ Test the message to parents of Mixture node. """ K = 3 # Broadcasting the moments on the cluster axis Mu = GaussianARD(2, 1, ndim=0, plates=(K,)) (mu, mumu) = Mu._message_to_child() Alpha = Gamma(3, 1, plates=(K,)) (alpha, logalpha) = Alpha._message_to_child() z = Categorical(np.ones(K)/K) X = Mixture(z, GaussianARD, Mu, Alpha) tau = 4 Y = GaussianARD(X, tau) y = 5 Y.observe(y) (x, xx) = X._message_to_child() m = z._message_from_children() self.assertAllClose(m[0] * np.ones(K), random.gaussian_logpdf(xx*alpha, x*alpha*mu, mumu*alpha, logalpha, 0) * np.ones(K)) m = Mu._message_from_children() self.assertAllClose(m[0], 1/K * (alpha*x) * np.ones(3)) self.assertAllClose(m[1], -0.5 * 1/K * alpha * np.ones(3)) # Some parameters do not have cluster plate axis Mu = GaussianARD(2, 1, ndim=0, plates=(K,)) (mu, mumu) = Mu._message_to_child() Alpha = Gamma(3, 1) # Note: no cluster plate axis! (alpha, logalpha) = Alpha._message_to_child() z = Categorical(np.ones(K)/K) X = Mixture(z, GaussianARD, Mu, Alpha) tau = 4 Y = GaussianARD(X, tau) y = 5 Y.observe(y) (x, xx) = X._message_to_child() m = z._message_from_children() self.assertAllClose(m[0] * np.ones(K), random.gaussian_logpdf(xx*alpha, x*alpha*mu, mumu*alpha, logalpha, 0) * np.ones(K)) m = Mu._message_from_children() self.assertAllClose(m[0], 1/K * (alpha*x) * np.ones(3)) self.assertAllClose(m[1], -0.5 * 1/K * alpha * np.ones(3)) # Cluster assignments do not have as many plate axes as parameters. M = 2 Mu = GaussianARD(2, 1, ndim=0, plates=(K,M)) (mu, mumu) = Mu._message_to_child() Alpha = Gamma(3, 1, plates=(K,M)) (alpha, logalpha) = Alpha._message_to_child() z = Categorical(np.ones(K)/K) X = Mixture(z, GaussianARD, Mu, Alpha, cluster_plate=-2) tau = 4 Y = GaussianARD(X, tau) y = 5 * np.ones(M) Y.observe(y) (x, xx) = X._message_to_child() m = z._message_from_children() self.assertAllClose(m[0]*np.ones(K), np.sum(random.gaussian_logpdf(xx*alpha, x*alpha*mu, mumu*alpha, logalpha, 0) * np.ones((K,M)), axis=-1)) m = Mu._message_from_children() self.assertAllClose(m[0] * np.ones((K,M)), 1/K * (alpha*x) * np.ones((K,M))) self.assertAllClose(m[1] * np.ones((K,M)), -0.5 * 1/K * alpha * np.ones((K,M))) # Mixed distribution broadcasts g # This tests for a found bug. The bug caused an error. Z = Categorical([0.3, 0.5, 0.2]) X = Mixture(Z, Categorical, [[0.2,0.8], [0.1,0.9], [0.3,0.7]]) m = Z._message_from_children() pass def test_lowerbound(self): """ Test log likelihood lower bound for Mixture node """ # Mixed distribution broadcasts g # This tests for a found bug. The bug caused an error. Z = Categorical([0.3, 0.5, 0.2]) X = Mixture(Z, Categorical, [[0.2,0.8], [0.1,0.9], [0.3,0.7]]) X.lower_bound_contribution() pass def test_mask_to_parent(self): """ Test the mask handling in Mixture node """ K = 3 Z = Categorical(np.ones(K)/K, plates=(4,5)) Mu = GaussianARD(0, 1, shape=(2,), plates=(4,K,5)) Alpha = Gamma(1, 1, plates=(4,K,5,2)) X = Mixture(Z, GaussianARD, Mu, Alpha, cluster_plate=-2) Y = GaussianARD(X, 1) mask = np.reshape((np.mod(np.arange(4*5), 2) == 0), (4,5)) Y.observe(np.ones((4,5,2)), mask=mask) self.assertArrayEqual(Z.mask, mask) self.assertArrayEqual(Mu.mask, mask[:,None,:]) self.assertArrayEqual(Alpha.mask, mask[:,None,:,None]) pass def test_nans(self): """ Test multinomial mixture """ # The probabilities p1 cause problems p0 = [0.1, 0.9] p1 = [1.0-1e-50, 1e-50] Z = Categorical([1-1e-10, 1e-10]) X = Mixture(Z, Multinomial, 10, [p0, p1]) u = X._message_to_child() self.assertAllClose(u[0], [1, 9]) p0 = [0.1, 0.9] p1 = [1.0-1e-10, 1e-10] Z = Categorical([1-1e-50, 1e-50]) X = Mixture(Z, Multinomial, 10, [p0, p1]) u = X._message_to_child() self.assertAllClose(u[0], [1, 9]) with warnings.catch_warnings(): warnings.simplefilter("ignore", RuntimeWarning) warnings.simplefilter("ignore", UserWarning) p0 = [0.1, 0.9] p1 = [1.0, 0.0] X = Mixture(0, Multinomial, 10, [p0, p1]) u = X._message_to_child() self.assertAllClose(u[0], np.nan*np.ones(2)) pass def test_random(self): """ Test random sampling of mixture node """ o = 1e-20 X = Mixture([1, 0, 2], Categorical, [ [o, o, o, 1], [o, o, 1, o], [1, o, o, o] ]) x = X.random() self.assertAllClose(x, [2, 3, 0]) pass bayespy-0.4.0/bayespy/inference/vmp/nodes/tests/__init__.py0000664000175000017500000000000012331123460025467 0ustar jluttinejluttine00000000000000bayespy-0.4.0/bayespy/inference/vmp/nodes/tests/test_gaussian_markov_chain.py0000644000175000017500000007416212615643655031366 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2013-2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for gaussian_markov_chain module. """ import numpy as np from ..gaussian_markov_chain import GaussianMarkovChain from ..gaussian_markov_chain import VaryingGaussianMarkovChain from ..gaussian import Gaussian from ..gaussian import GaussianARD from ..wishart import Wishart from ..gamma import Gamma from bayespy.utils import random from bayespy.utils import linalg from bayespy.utils import misc from bayespy.utils.misc import TestCase class TestGaussianMarkovChain(TestCase): def create_model(self, N, D): # Construct the model Mu = Gaussian(np.random.randn(D), np.identity(D)) Lambda = Wishart(D, random.covariance(D)) A = Gaussian(np.random.randn(D,D), np.identity(D)) V = Gamma(D, np.random.rand(D)) X = GaussianMarkovChain(Mu, Lambda, A, V, n=N) Y = Gaussian(X, np.identity(D)) return (Y, X, Mu, Lambda, A, V) def test_plates(self): """ Test that plates are handled correctly. """ def test_message_to_mu0(self): pass def test_message_to_Lambda0(self): pass def test_message_to_A(self): pass def test_message_to_v(self): pass def test_message_to_child(self): """ Test the updating of GaussianMarkovChain. Check that the moments and the lower bound contribution are computed correctly. """ # TODO: Add plates and missing values! # Dimensionalities D = 3 N = 5 (Y, X, Mu, Lambda, A, V) = self.create_model(N, D) # Inference with arbitrary observations y = np.random.randn(N,D) Y.observe(y) X.update() (x_vb, xnxn_vb, xpxn_vb) = X.get_moments() # Get parameter moments (mu0, mumu0) = Mu.get_moments() (icov0, logdet0) = Lambda.get_moments() (a, aa) = A.get_moments() (icov_x, logdetx) = V.get_moments() icov_x = np.diag(icov_x) # Prior precision Z = np.einsum('...kij,...kk->...ij', aa, icov_x) U_diag = [icov0+Z] + (N-2)*[icov_x+Z] + [icov_x] U_super = (N-1) * [-np.dot(a.T, icov_x)] U = misc.block_banded(U_diag, U_super) # Prior mean mu_prior = np.zeros(D*N) mu_prior[:D] = np.dot(icov0,mu0) # Data Cov = np.linalg.inv(U + np.identity(D*N)) mu = np.dot(Cov, mu_prior + y.flatten()) # Moments xx = mu[:,np.newaxis]*mu[np.newaxis,:] + Cov mu = np.reshape(mu, (N,D)) xx = np.reshape(xx, (N,D,N,D)) # Check results self.assertAllClose(x_vb, mu, msg="Incorrect mean") for n in range(N): self.assertAllClose(xnxn_vb[n,:,:], xx[n,:,n,:], msg="Incorrect second moment") for n in range(N-1): self.assertAllClose(xpxn_vb[n,:,:], xx[n,:,n+1,:], msg="Incorrect lagged second moment") # Compute the entropy H(X) ldet = linalg.logdet_cov(Cov) H = random.gaussian_entropy(-ldet, N*D) # Compute xx = np.reshape(xx, (N*D, N*D)) mu = np.reshape(mu, (N*D,)) ldet = -logdet0 - np.sum(np.ones((N-1,D))*logdetx) P = random.gaussian_logpdf(np.einsum('...ij,...ij', xx, U), np.einsum('...i,...i', mu, mu_prior), np.einsum('...ij,...ij', mumu0, icov0), -ldet, N*D) # The VB bound from the net l = X.lower_bound_contribution() self.assertAllClose(l, H+P) # Compute the true bound + H(X) # # Simple tests # def check(N, D, plates=None, mu=None, Lambda=None, A=None, V=None): if mu is None: mu = np.random.randn(D) if Lambda is None: Lambda = random.covariance(D) if A is None: A = np.random.randn(D,D) if V is None: V = np.random.rand(D) X = GaussianMarkovChain(mu, Lambda, A, V, plates=plates, n=N) (u0, u1, u2) = X._message_to_child() (mu, mumu) = X.parents[0].get_moments() (Lambda, _) = X.parents[1].get_moments() (a, aa) = X.parents[2].get_moments() a = a * np.ones((N-1,D,D)) # explicit broadcasting for simplicity aa = aa * np.ones((N-1,D,D,D)) # explicit broadcasting for simplicity (v, _) = X.parents[3].get_moments() v = v * np.ones((N-1,D)) plates_C = X.plates plates_mu = X.plates C = np.zeros(plates_C + (N,D,N,D)) plates_mu = np.shape(mu)[:-1] m = np.zeros(plates_mu + (N,D)) m[...,0,:] = np.einsum('...ij,...j->...i', Lambda, mu) C[...,0,:,0,:] = Lambda + np.einsum('...dij,...d->...ij', aa[...,0,:,:,:], v[...,0,:]) for n in range(1,N-1): C[...,n,:,n,:] = (np.einsum('...dij,...d->...ij', aa[...,n,:,:,:], v[...,n,:]) + v[...,n,:,None] * np.identity(D)) for n in range(N-1): C[...,n,:,n+1,:] = -np.einsum('...di,...d->...id', a[...,n,:,:], v[...,n,:]) C[...,n+1,:,n,:] = -np.einsum('...di,...d->...di', a[...,n,:,:], v[...,n,:]) C[...,-1,:,-1,:] = v[...,-1,:,None]*np.identity(D) C = np.reshape(C, plates_C+(N*D,N*D)) Cov = np.linalg.inv(C) Cov = np.reshape(Cov, plates_C+(N,D,N,D)) m0 = np.einsum('...minj,...nj->...mi', Cov, m) m1 = np.zeros(plates_C+(N,D,D)) m2 = np.zeros(plates_C+(N-1,D,D)) for n in range(N): m1[...,n,:,:] = Cov[...,n,:,n,:] + np.einsum('...i,...j->...ij', m0[...,n,:], m0[...,n,:]) for n in range(N-1): m2[...,n,:,:] = Cov[...,n,:,n+1,:] + np.einsum('...i,...j->...ij', m0[...,n,:], m0[...,n+1,:]) self.assertAllClose(m0, u0*np.ones(np.shape(m0))) self.assertAllClose(m1, u1*np.ones(np.shape(m1))) self.assertAllClose(m2, u2*np.ones(np.shape(m2))) pass check(4,1) check(4,3) # # Test mu # # Simple check(4,3, mu=Gaussian(np.random.randn(3), random.covariance(3))) # Plates check(4,3, mu=Gaussian(np.random.randn(5,6,3), random.covariance(3), plates=(5,6))) # Plates with moments broadcasted over plates check(4,3, mu=Gaussian(np.random.randn(3), random.covariance(3), plates=(5,))) check(4,3, mu=Gaussian(np.random.randn(1,3), random.covariance(3), plates=(5,))) # Plates broadcasting check(4,3, plates=(5,), mu=Gaussian(np.random.randn(3), random.covariance(3), plates=())) check(4,3, plates=(5,), mu=Gaussian(np.random.randn(1,3), random.covariance(3), plates=(1,))) # # Test Lambda # # Simple check(4,3, Lambda=Wishart(10+np.random.rand(), random.covariance(3))) # Plates check(4,3, Lambda=Wishart(10+np.random.rand(), random.covariance(3), plates=(5,6))) # Plates with moments broadcasted over plates check(4,3, Lambda=Wishart(10+np.random.rand(), random.covariance(3), plates=(5,))) check(4,3, Lambda=Wishart(10+np.random.rand(1), random.covariance(3), plates=(5,))) # Plates broadcasting check(4,3, plates=(5,), Lambda=Wishart(10+np.random.rand(), random.covariance(3), plates=())) check(4,3, plates=(5,), Lambda=Wishart(10+np.random.rand(), random.covariance(3), plates=(1,))) # # Test A # # Simple check(4,3, A=GaussianARD(np.random.randn(3,3), np.random.rand(3,3), shape=(3,), plates=(3,))) # Plates on time axis check(5,3, A=GaussianARD(np.random.randn(4,3,3), np.random.rand(4,3,3), shape=(3,), plates=(4,3))) # Plates on time axis with broadcasted moments check(5,3, A=GaussianARD(np.random.randn(1,3,3), np.random.rand(1,3,3), shape=(3,), plates=(4,3))) check(5,3, A=GaussianARD(np.random.randn(3,3), np.random.rand(3,3), shape=(3,), plates=(4,3))) # Plates check(4,3, A=GaussianARD(np.random.randn(5,6,1,3,3), np.random.rand(5,6,1,3,3), shape=(3,), plates=(5,6,1,3))) # Plates with moments broadcasted over plates check(4,3, A=GaussianARD(np.random.randn(3,3), np.random.rand(3,3), shape=(3,), plates=(5,1,3))) check(4,3, A=GaussianARD(np.random.randn(1,1,3,3), np.random.rand(1,1,3,3), shape=(3,), plates=(5,1,3))) # Plates broadcasting check(4,3, plates=(5,), A=GaussianARD(np.random.randn(3,3), np.random.rand(3,3), shape=(3,), plates=(3,))) check(4,3, plates=(5,), A=GaussianARD(np.random.randn(3,3), np.random.rand(3,3), shape=(3,), plates=(1,1,3))) # # Test v # # Simple check(4,3, V=Gamma(np.random.rand(1,3), np.random.rand(1,3), plates=(1,3))) check(4,3, V=Gamma(np.random.rand(3), np.random.rand(3), plates=(3,))) # Plates check(4,3, V=Gamma(np.random.rand(5,6,1,3), np.random.rand(5,6,1,3), plates=(5,6,1,3))) # Plates with moments broadcasted over plates check(4,3, V=Gamma(np.random.rand(1,3), np.random.rand(1,3), plates=(5,1,3))) check(4,3, V=Gamma(np.random.rand(1,1,3), np.random.rand(1,1,3), plates=(5,1,3))) # Plates broadcasting check(4,3, plates=(5,), V=Gamma(np.random.rand(1,3), np.random.rand(1,3), plates=(1,3))) check(4,3, plates=(5,), V=Gamma(np.random.rand(1,1,3), np.random.rand(1,1,3), plates=(1,1,3))) # # Check with input signals # mu = 2 Lambda = 3 A = 4 B = 5 v = 6 inputs = [[-2], [3]] X = GaussianMarkovChain([mu], [[Lambda]], [[A,B]], [v], inputs=inputs) V = (np.array([[v*A**2, -v*A, 0], [-v*A, v*A**2, -v*A], [0, -v*A, 0]]) + np.array([[Lambda, 0, 0], [0, v, 0], [0, 0, v]])) m = (np.array([Lambda*mu, 0, 0]) + np.array([0, v*B*inputs[0][0], v*B*inputs[1][0]]) - np.array([v*A*B*inputs[0][0], v*A*B*inputs[1][0], 0])) Cov = np.linalg.inv(V) mean = np.dot(Cov, m) X.update() u = X.get_moments() self.assertAllClose(u[0], mean[:,None]) self.assertAllClose(u[1] - u[0][...,None,:]*u[0][...,:,None], Cov[(0,1,2),(0,1,2),None,None]) self.assertAllClose(u[2] - u[0][...,:-1,:,None]*u[0][...,1:,None,:], Cov[(0,1),(1,2),None,None]) pass def test_smoothing(self): """ Test the posterior estimation of GaussianMarkovChain. Create time-variant dynamics and compare the results of BayesPy VB inference and standard Kalman filtering & smoothing. This is not that useful anymore, because the moments are checked much better in another test method. """ # # Set up an artificial system # # Dimensions N = 500 D = 2 # Dynamics (time varying) A0 = np.array([[.9, -.4], [.4, .9]]) A1 = np.array([[.98, -.1], [.1, .98]]) l = np.linspace(0, 1, N-1).reshape((-1,1,1)) A = (1-l)*A0 + l*A1 # Innovation covariance matrix (time varying) v = np.random.rand(D) V = np.diag(v) # Observation noise covariance matrix C = np.identity(D) # # Simulate data # X = np.empty((N,D)) Y = np.empty((N,D)) x = np.array([0.5, -0.5]) X[0,:] = x Y[0,:] = x + np.random.multivariate_normal(np.zeros(D), C) for n in range(N-1): x = np.dot(A[n,:,:],x) + np.random.multivariate_normal(np.zeros(D), V) X[n+1,:] = x Y[n+1,:] = x + np.random.multivariate_normal(np.zeros(D), C) # # BayesPy inference # # Construct VB model Xh = GaussianMarkovChain(np.zeros(D), np.identity(D), A, 1/v, n=N) Yh = Gaussian(Xh, np.identity(D), plates=(N,)) # Put data Yh.observe(Y) # Run inference Xh.update() # Store results Xh_vb = Xh.u[0] CovXh_vb = Xh.u[1] - Xh_vb[...,np.newaxis,:] * Xh_vb[...,:,np.newaxis] # # "The ground truth" using standard Kalman filter and RTS smoother # V = N*(V,) UY = Y U = N*(C,) (Xh, CovXh) = misc.kalman_filter(UY, U, A, V, np.zeros(D), np.identity(D)) (Xh, CovXh) = misc.rts_smoother(Xh, CovXh, A, V) # # Check results # self.assertTrue(np.allclose(Xh_vb, Xh)) self.assertTrue(np.allclose(CovXh_vb, CovXh)) class TestVaryingGaussianMarkovChain(TestCase): def test_plates_from_parents(self): """ Test that VaryingGaussianMarkovChain deduces plates correctly """ def check(plates_X, plates_mu=(), plates_Lambda=(), plates_B=(), plates_S=(), plates_v=()): D = 3 K = 2 N = 4 np.random.seed(42) mu = Gaussian(np.random.randn(*(plates_mu+(D,))), random.covariance(D)) Lambda = Wishart(D+np.ones(plates_Lambda), random.covariance(D)) B = GaussianARD(np.random.randn(*(plates_B+(D,D,K))), 1+np.random.rand(*(plates_B+(D,D,K))), shape=(D,K), plates=plates_B+(D,)) S = GaussianARD(np.random.randn(*(plates_S+(N,K))), 1+np.random.rand(*(plates_S+(N,K))), shape=(K,), plates=plates_S+(N,)) v = Gamma(1+np.random.rand(*(plates_v+(1,D))), 1+np.random.rand(*(plates_v+(1,D)))) X = VaryingGaussianMarkovChain(mu, Lambda, B, S, v, name="X") self.assertEqual(plates_X, X.plates, msg="Incorrect plates deduced") pass check(()) check((2,3), plates_mu=(2,3)) check((2,3), plates_Lambda=(2,3)) check((2,3), plates_B=(2,3)) check((2,3), plates_S=(2,3)) check((2,3), plates_v=(2,3)) pass def test_message_to_child(self): # A very simple check before the more complex ones: # 1-D process, k=1, fixed constant parameters m = 1.0 l = 4.0 b = 2.0 s = [3.0, 8.0] v = 5.0 X = VaryingGaussianMarkovChain([m], [[l]], [[[b]]], [[s[0]],[s[1]]], [v]) (u0, u1, u2) = X._message_to_child() C = np.array([[l+b**2*s[0]**2*v, -b*s[0]*v, 0], [ -b*s[0]*v, v+b**2*s[1]**2*v, -b*s[1]*v], [ 0, -b*s[1]*v, v]]) Cov = np.linalg.inv(C) m0 = np.dot(Cov, [[l*m], [0], [0]]) m1 = np.diag(Cov)[:,None,None] + m0[:,:,None]**2 m2 = np.diag(Cov, k=1)[:,None,None] + m0[1:,:,None]*m0[:-1,:,None] self.assertAllClose(m0, u0) self.assertAllClose(m1, u1) self.assertAllClose(m2, u2) def check(N, D, K, plates=None, mu=None, Lambda=None, B=None, S=None, V=None): if mu is None: mu = np.random.randn(D) if Lambda is None: Lambda = random.covariance(D) if B is None: B = np.random.randn(D,D,K) if S is None: S = np.random.randn(N-1,K) if V is None: V = np.random.rand(D) X = VaryingGaussianMarkovChain(mu, Lambda, B, S, V, plates=plates, n=N) (u0, u1, u2) = X._message_to_child() (mu, mumu) = X.parents[0].get_moments() (Lambda, _) = X.parents[1].get_moments() (b, bb) = X.parents[2].get_moments() (s, ss) = X.parents[3].get_moments() (v, _) = X.parents[4].get_moments() v = v * np.ones((N-1,D)) #V = np.atleast_3d(v)[...,-1,:,None]*np.identity(D) plates_C = X.plates plates_mu = X.plates C = np.zeros(plates_C + (N,D,N,D)) plates_mu = np.shape(mu)[:-1] m = np.zeros(plates_mu + (N,D)) m[...,0,:] = np.einsum('...ij,...j->...i', Lambda, mu) #m = np.reshape(m, plates_mu + (N*D,)) A = np.einsum('...dik,...nk->...ndi', b, s) AA = np.einsum('...dikjl,...nkl->...ndij', bb, ss) C[...,0,:,0,:] = Lambda + np.einsum('...dij,...d->...ij', AA[...,0,:,:,:], v[...,0,:]) for n in range(1,N-1): C[...,n,:,n,:] = (np.einsum('...dij,...d->...ij', AA[...,n,:,:,:], v[...,n,:]) + v[...,n,:,None] * np.identity(D)) for n in range(N-1): C[...,n,:,n+1,:] = -np.einsum('...di,...d->...id', A[...,n,:,:], v[...,n,:]) C[...,n+1,:,n,:] = -np.einsum('...di,...d->...di', A[...,n,:,:], v[...,n,:]) C[...,-1,:,-1,:] = v[...,-1,:,None]*np.identity(D) C = np.reshape(C, plates_C+(N*D,N*D)) Cov = np.linalg.inv(C) Cov = np.reshape(Cov, plates_C+(N,D,N,D)) m0 = np.einsum('...minj,...nj->...mi', Cov, m) m1 = np.zeros(plates_C+(N,D,D)) m2 = np.zeros(plates_C+(N-1,D,D)) for n in range(N): m1[...,n,:,:] = Cov[...,n,:,n,:] + np.einsum('...i,...j->...ij', m0[...,n,:], m0[...,n,:]) for n in range(N-1): m2[...,n,:,:] = Cov[...,n,:,n+1,:] + np.einsum('...i,...j->...ij', m0[...,n,:], m0[...,n+1,:]) self.assertAllClose(m0, u0*np.ones(np.shape(m0))) self.assertAllClose(m1, u1*np.ones(np.shape(m1))) self.assertAllClose(m2, u2*np.ones(np.shape(m2))) pass check(2,1,1) check(2,3,1) check(2,1,3) check(4,3,2) # # Test mu # # Simple check(4,3,2, mu=Gaussian(np.random.randn(3), random.covariance(3))) # Plates check(4,3,2, mu=Gaussian(np.random.randn(5,6,3), random.covariance(3), plates=(5,6))) # Plates with moments broadcasted over plates check(4,3,2, mu=Gaussian(np.random.randn(3), random.covariance(3), plates=(5,))) check(4,3,2, mu=Gaussian(np.random.randn(1,3), random.covariance(3), plates=(5,))) # Plates broadcasting check(4,3,2, plates=(5,), mu=Gaussian(np.random.randn(3), random.covariance(3), plates=())) check(4,3,2, plates=(5,), mu=Gaussian(np.random.randn(1,3), random.covariance(3), plates=(1,))) # # Test Lambda # # Simple check(4,3,2, Lambda=Wishart(10+np.random.rand(), random.covariance(3))) # Plates check(4,3,2, Lambda=Wishart(10+np.random.rand(), random.covariance(3), plates=(5,6))) # Plates with moments broadcasted over plates check(4,3,2, Lambda=Wishart(10+np.random.rand(), random.covariance(3), plates=(5,))) check(4,3,2, Lambda=Wishart(10+np.random.rand(1), random.covariance(3), plates=(5,))) # Plates broadcasting check(4,3,2, plates=(5,), Lambda=Wishart(10+np.random.rand(), random.covariance(3), plates=())) check(4,3,2, plates=(5,), Lambda=Wishart(10+np.random.rand(), random.covariance(3), plates=(1,))) # # Test B # # Simple check(4,3,2, B=GaussianARD(np.random.randn(3,3,2), np.random.rand(3,3,2), shape=(3,2), plates=(3,))) # Plates check(4,3,2, B=GaussianARD(np.random.randn(5,6,3,3,2), np.random.rand(5,6,3,3,2), shape=(3,2), plates=(5,6,3))) # Plates with moments broadcasted over plates check(4,3,2, B=GaussianARD(np.random.randn(3,3,2), np.random.rand(3,3,2), shape=(3,2), plates=(5,3))) check(4,3,2, B=GaussianARD(np.random.randn(1,3,3,2), np.random.rand(1,3,3,2), shape=(3,2), plates=(5,3))) # Plates broadcasting check(4,3,2, plates=(5,), B=GaussianARD(np.random.randn(3,3,2), np.random.rand(3,3,2), shape=(3,2), plates=(3,))) check(4,3,2, plates=(5,), B=GaussianARD(np.random.randn(3,3,2), np.random.rand(3,3,2), shape=(3,2), plates=(1,3))) # # Test S # # Simple check(4,3,2, S=GaussianARD(np.random.randn(4-1,2), np.random.rand(4-1,2), shape=(2,), plates=(4-1,))) # Plates check(4,3,2, S=GaussianARD(np.random.randn(5,6,4-1,2), np.random.rand(5,6,4-1,2), shape=(2,), plates=(5,6,4-1,))) # Plates with moments broadcasted over plates check(4,3,2, S=GaussianARD(np.random.randn(4-1,2), np.random.rand(4-1,2), shape=(2,), plates=(5,4-1,))) check(4,3,2, S=GaussianARD(np.random.randn(1,4-1,2), np.random.rand(1,4-1,2), shape=(2,), plates=(5,4-1,))) # Plates broadcasting check(4,3,2, plates=(5,), S=GaussianARD(np.random.randn(4-1,2), np.random.rand(4-1,2), shape=(2,), plates=(4-1,))) check(4,3,2, plates=(5,), S=GaussianARD(np.random.randn(4-1,2), np.random.rand(4-1,2), shape=(2,), plates=(1,4-1,))) # # Test v # # Simple check(4,3,2, V=Gamma(np.random.rand(1,3), np.random.rand(1,3), plates=(1,3))) check(4,3,2, V=Gamma(np.random.rand(3), np.random.rand(3), plates=(3,))) # Plates check(4,3,2, V=Gamma(np.random.rand(5,6,1,3), np.random.rand(5,6,1,3), plates=(5,6,1,3))) # Plates with moments broadcasted over plates check(4,3,2, V=Gamma(np.random.rand(1,3), np.random.rand(1,3), plates=(5,1,3))) check(4,3,2, V=Gamma(np.random.rand(1,1,3), np.random.rand(1,1,3), plates=(5,1,3))) # Plates broadcasting check(4,3,2, plates=(5,), V=Gamma(np.random.rand(1,3), np.random.rand(1,3), plates=(1,3))) check(4,3,2, plates=(5,), V=Gamma(np.random.rand(1,1,3), np.random.rand(1,1,3), plates=(1,1,3))) # # Uncertainty in both B and S # check(4,3,2, B=GaussianARD(np.random.randn(3,3,2), np.random.rand(3,3,2), shape=(3,2), plates=(3,)), S=GaussianARD(np.random.randn(4-1,2), np.random.rand(4-1,2), shape=(2,), plates=(4-1,))) pass def test_message_to_mu(self): # TODO pass def test_message_to_Lambda(self): # TODO pass def test_message_to_B(self): # TODO pass def test_message_to_S(self): # TODO pass def test_message_to_v(self): # TODO pass bayespy-0.4.0/bayespy/inference/vmp/nodes/tests/test_gaussian.py0000644000175000017500000013205312615643655026637 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2013-2015 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for `gaussian` module. """ import numpy as np from scipy import special from numpy import testing from .. import gaussian from bayespy.nodes import (Gaussian, GaussianARD, GaussianGammaISO, Gamma, Wishart) from ...vmp import VB from bayespy.utils import misc from bayespy.utils import linalg from bayespy.utils import random from bayespy.utils.misc import TestCase class TestGaussianFunctions(TestCase): def test_rotate_covariance(self): """ Test the Gaussian array covariance rotation. """ # Check matrix R = np.random.randn(2,2) Cov = np.random.randn(2,2) self.assertAllClose(gaussian.rotate_covariance(Cov, R), np.einsum('ik,kl,lj', R, Cov, R.T)) # Check matrix with plates R = np.random.randn(2,2) Cov = np.random.randn(4,3,2,2) self.assertAllClose(gaussian.rotate_covariance(Cov, R), np.einsum('...ik,...kl,...lj', R, Cov, R.T)) # Check array, first axis R = np.random.randn(2,2) Cov = np.random.randn(2,3,3,2,3,3) self.assertAllClose(gaussian.rotate_covariance(Cov, R, ndim=3, axis=-3), np.einsum('...ik,...kablcd,...lj->...iabjcd', R, Cov, R.T)) self.assertAllClose(gaussian.rotate_covariance(Cov, R, ndim=3, axis=0), np.einsum('...ik,...kablcd,...lj->...iabjcd', R, Cov, R.T)) # Check array, middle axis R = np.random.randn(2,2) Cov = np.random.randn(3,2,3,3,2,3) self.assertAllClose(gaussian.rotate_covariance(Cov, R, ndim=3, axis=-2), np.einsum('...ik,...akbcld,...lj->...aibcjd', R, Cov, R.T)) self.assertAllClose(gaussian.rotate_covariance(Cov, R, ndim=3, axis=1), np.einsum('...ik,...akbcld,...lj->...aibcjd', R, Cov, R.T)) # Check array, last axis R = np.random.randn(2,2) Cov = np.random.randn(3,3,2,3,3,2) self.assertAllClose(gaussian.rotate_covariance(Cov, R, ndim=3, axis=-1), np.einsum('...ik,...abkcdl,...lj->...abicdj', R, Cov, R.T)) self.assertAllClose(gaussian.rotate_covariance(Cov, R, ndim=3, axis=2), np.einsum('...ik,...abkcdl,...lj->...abicdj', R, Cov, R.T)) # Check array, middle axis with plates R = np.random.randn(2,2) Cov = np.random.randn(4,4,3,2,3,3,2,3) self.assertAllClose(gaussian.rotate_covariance(Cov, R, ndim=3, axis=-2), np.einsum('...ik,...akbcld,...lj->...aibcjd', R, Cov, R.T)) self.assertAllClose(gaussian.rotate_covariance(Cov, R, ndim=3, axis=1), np.einsum('...ik,...akbcld,...lj->...aibcjd', R, Cov, R.T)) pass class TestGaussianARD(TestCase): def test_init(self): """ Test the constructor of GaussianARD """ def check_init(true_plates, true_shape, mu, alpha, **kwargs): X = GaussianARD(mu, alpha, **kwargs) self.assertEqual(X.dims, (true_shape, true_shape+true_shape), msg="Constructed incorrect dimensionality") self.assertEqual(X.plates, true_plates, msg="Constructed incorrect plates") # # Create from constant parents # # Use ndim=0 for constant mu check_init((), (), 0, 1) check_init((3,2), (), np.zeros((3,2,)), np.ones((2,))) check_init((4,2,2,3), (), np.zeros((2,1,3,)), np.ones((4,1,2,3))) # Use ndim check_init((4,2), (2,3), np.zeros((2,1,3,)), np.ones((4,1,2,3)), ndim=2) # Use shape check_init((4,2), (2,3), np.zeros((2,1,3,)), np.ones((4,1,2,3)), shape=(2,3)) # Use ndim and shape check_init((4,2), (2,3), np.zeros((2,1,3,)), np.ones((4,1,2,3)), ndim=2, shape=(2,3)) # # Create from node parents # # Infer ndim from parent mu check_init((3,), (), GaussianARD(0, 1, plates=(3,)), Gamma(1, 1, plates=(3,))) # Infer ndim from mu, take broadcasted shape check_init((4,), (2,2,3), GaussianARD(np.zeros((2,1,3)), np.ones((2,1,3)), ndim=3), Gamma(np.ones((4,1,2,3)), np.ones((4,1,2,3)))) # Use ndim check_init((4,), (2,2,3), GaussianARD(np.zeros((4,1,2,3)), np.ones((4,1,2,3)), ndim=2), Gamma(np.ones((4,2,1,3)), np.ones((4,2,1,3))), ndim=3) # Use shape check_init((4,), (2,2,3), GaussianARD(np.zeros((4,1,2,3)), np.ones((4,1,2,3)), ndim=2), Gamma(np.ones((4,2,1,3)), np.ones((4,2,1,3))), shape=(2,2,3)) # Use ndim and shape check_init((4,2), (2,3), GaussianARD(np.zeros((2,1,3)), np.ones((2,1,3)), ndim=2), Gamma(np.ones((4,1,2,3)), np.ones((4,1,2,3))), ndim=2, shape=(2,3)) # Test for a found bug check_init((), (3,), np.ones(3), 1, ndim=1) # Add axes if necessary check_init((), (1,2,3), GaussianARD(np.zeros((2,3)), np.ones((2,3)), ndim=2), 1, ndim=3) # # Errors # # Inconsistent shapes self.assertRaises(ValueError, GaussianARD, GaussianARD(np.zeros((2,3)), np.ones((2,3)), ndim=1), np.ones((4,3)), ndim=2) # Inconsistent dims of mu and alpha self.assertRaises(ValueError, GaussianARD, np.zeros((2,3)), np.ones((2,))) # Inconsistent plates of mu and alpha self.assertRaises(ValueError, GaussianARD, GaussianARD(np.zeros((3,2,3)), np.ones((3,2,3)), ndim=2), np.ones((3,4,2,3)), ndim=3) # Inconsistent ndim and shape self.assertRaises(ValueError, GaussianARD, np.zeros((2,3)), np.ones((2,)), shape=(2,3), ndim=1) # Parent mu has more axes self.assertRaises(ValueError, GaussianARD, GaussianARD(np.zeros((2,3)), np.ones((2,3)), ndim=2), np.ones((2,3)), ndim=1) # Incorrect shape self.assertRaises(ValueError, GaussianARD, GaussianARD(np.zeros((2,3)), np.ones((2,3)), ndim=2), np.ones((2,3)), shape=(2,2)) pass def test_message_to_child(self): """ Test moments of GaussianARD. """ # Check that moments have full shape when broadcasting X = GaussianARD(np.zeros((2,)), np.ones((3,2)), shape=(4,3,2)) (u0, u1) = X._message_to_child() self.assertEqual(np.shape(u0), (4,3,2)) self.assertEqual(np.shape(u1), (4,3,2,4,3,2)) # Check the formula X = GaussianARD(2, 3) (u0, u1) = X._message_to_child() self.assertAllClose(u0, 2) self.assertAllClose(u1, 2**2 + 1/3) # Check the formula for multidimensional arrays X = GaussianARD(2*np.ones((2,1,4)), 3*np.ones((2,3,1)), ndim=3) (u0, u1) = X._message_to_child() self.assertAllClose(u0, 2*np.ones((2,3,4))) self.assertAllClose(u1, 2**2 * np.ones((2,3,4,2,3,4)) + 1/3 * misc.identity(2,3,4)) # Check the formula for dim-broadcasted mu X = GaussianARD(2*np.ones((3,1)), 3*np.ones((2,3,4)), ndim=3) (u0, u1) = X._message_to_child() self.assertAllClose(u0, 2*np.ones((2,3,4))) self.assertAllClose(u1, 2**2 * np.ones((2,3,4,2,3,4)) + 1/3 * misc.identity(2,3,4)) # Check the formula for dim-broadcasted alpha X = GaussianARD(2*np.ones((2,3,4)), 3*np.ones((3,1)), ndim=3) (u0, u1) = X._message_to_child() self.assertAllClose(u0, 2*np.ones((2,3,4))) self.assertAllClose(u1, 2**2 * np.ones((2,3,4,2,3,4)) + 1/3 * misc.identity(2,3,4)) # Check the formula for dim-broadcasted mu and alpha X = GaussianARD(2*np.ones((3,1)), 3*np.ones((3,1)), shape=(2,3,4)) (u0, u1) = X._message_to_child() self.assertAllClose(u0, 2*np.ones((2,3,4))) self.assertAllClose(u1, 2**2 * np.ones((2,3,4,2,3,4)) + 1/3 * misc.identity(2,3,4)) # Check the formula for dim-broadcasted mu with plates mu = GaussianARD(2*np.ones((5,1,3,4)), np.ones((5,1,3,4)), shape=(3,4), plates=(5,1)) X = GaussianARD(mu, 3*np.ones((5,2,3,4)), shape=(2,3,4), plates=(5,)) (u0, u1) = X._message_to_child() self.assertAllClose(u0, 2*np.ones((5,2,3,4))) self.assertAllClose(u1, 2**2 * np.ones((5,2,3,4,2,3,4)) + 1/3 * misc.identity(2,3,4)) # Check posterior X = GaussianARD(2, 3) Y = GaussianARD(X, 1) Y.observe(10) X.update() (u0, u1) = X._message_to_child() self.assertAllClose(u0, 1/(3+1) * (3*2 + 1*10)) self.assertAllClose(u1, (1/(3+1) * (3*2 + 1*10))**2 + 1/(3+1)) pass def test_message_to_parent_mu(self): """ Test that GaussianARD computes the message to the 1st parent correctly. """ # Check formula with uncertain parent alpha mu = GaussianARD(0, 1) alpha = Gamma(2,1) X = GaussianARD(mu, alpha) X.observe(3) (m0, m1) = mu._message_from_children() #(m0, m1) = X._message_to_parent(0) self.assertAllClose(m0, 2*3) self.assertAllClose(m1, -0.5*2) # Check formula with uncertain node mu = GaussianARD(1, 1e10) X = GaussianARD(mu, 2) Y = GaussianARD(X, 1) Y.observe(5) X.update() (m0, m1) = mu._message_from_children() self.assertAllClose(m0, 2 * 1/(2+1)*(2*1+1*5)) self.assertAllClose(m1, -0.5*2) # Check alpha larger than mu mu = GaussianARD(np.zeros((2,3)), 1e10, shape=(2,3)) X = GaussianARD(mu, 2*np.ones((3,2,3))) X.observe(3*np.ones((3,2,3))) (m0, m1) = mu._message_from_children() self.assertAllClose(m0, 2*3 * 3 * np.ones((2,3))) self.assertAllClose(m1, -0.5 * 3 * 2*misc.identity(2,3)) # Check mu larger than alpha mu = GaussianARD(np.zeros((3,2,3)), 1e10, shape=(3,2,3)) X = GaussianARD(mu, 2*np.ones((2,3))) X.observe(3*np.ones((3,2,3))) (m0, m1) = mu._message_from_children() self.assertAllClose(m0, 2 * 3 * np.ones((3,2,3))) self.assertAllClose(m1, -0.5 * 2*misc.identity(3,2,3)) # Check node larger than mu and alpha mu = GaussianARD(np.zeros((2,3)), 1e10, shape=(2,3)) X = GaussianARD(mu, 2*np.ones((3,)), shape=(3,2,3)) X.observe(3*np.ones((3,2,3))) (m0, m1) = mu._message_from_children() self.assertAllClose(m0, 2*3 * 3*np.ones((2,3))) self.assertAllClose(m1, -0.5 * 2 * 3*misc.identity(2,3)) # Check broadcasting of dimensions mu = GaussianARD(np.zeros((2,1)), 1e10, shape=(2,1)) X = GaussianARD(mu, 2*np.ones((2,3)), shape=(2,3)) X.observe(3*np.ones((2,3))) (m0, m1) = mu._message_from_children() self.assertAllClose(m0, 2*3 * 3*np.ones((2,1))) self.assertAllClose(m1, -0.5 * 2 * 3*misc.identity(2,1)) # Check plates for smaller mu than node mu = GaussianARD(0,1, shape=(3,), plates=(4,1,1)) X = GaussianARD(mu, 2*np.ones((3,)), shape=(2,3), plates=(4,5)) X.observe(3*np.ones((4,5,2,3))) (m0, m1) = mu._message_from_children() self.assertAllClose(m0 * np.ones((4,1,1,3)), 2*3 * 5*2*np.ones((4,1,1,3))) self.assertAllClose(m1 * np.ones((4,1,1,3,3)), -0.5*2 * 5*2*misc.identity(3) * np.ones((4,1,1,3,3))) # Check mask mu = GaussianARD(np.zeros((2,1,3)), 1e10, shape=(3,)) X = GaussianARD(mu, 2*np.ones((2,4,3)), shape=(3,), plates=(2,4,)) X.observe(3*np.ones((2,4,3)), mask=[[True, True, True, False], [False, True, False, True]]) (m0, m1) = mu._message_from_children() self.assertAllClose(m0, (2*3 * np.ones((2,1,3)) * np.array([[[3]], [[2]]]))) self.assertAllClose(m1, (-0.5*2 * misc.identity(3) * np.ones((2,1,1,1)) * np.array([[[[3]]], [[[2]]]]))) # Check mask with different shapes mu = GaussianARD(np.zeros((2,1,3)), 1e10, shape=()) X = GaussianARD(mu, 2*np.ones((2,4,3)), shape=(3,), plates=(2,4,)) mask = np.array([[True, True, True, False], [False, True, False, True]]) X.observe(3*np.ones((2,4,3)), mask=mask) (m0, m1) = mu._message_from_children() self.assertAllClose(m0, 2*3 * np.sum(np.ones((2,4,3))*mask[...,None], axis=-2, keepdims=True)) self.assertAllClose(m1, (-0.5*2 * np.sum(np.ones((2,4,3))*mask[...,None], axis=-2, keepdims=True))) # Check non-ARD Gaussian child mu = np.array([1,2]) Mu = GaussianARD(mu, 1e10, shape=(2,)) alpha = np.array([3,4]) Lambda = np.array([[1, 0.5], [0.5, 1]]) X = GaussianARD(Mu, alpha) Y = Gaussian(X, Lambda) y = np.array([5,6]) Y.observe(y) X.update() (m0, m1) = Mu._message_from_children() mean = np.dot(np.linalg.inv(np.diag(alpha)+Lambda), np.dot(np.diag(alpha), mu) + np.dot(Lambda, y)) self.assertAllClose(m0, np.dot(np.diag(alpha), mean)) self.assertAllClose(m1, -0.5*np.diag(alpha)) # Check broadcasted variable axes mu = GaussianARD(np.zeros(1), 1e10, shape=(1,)) X = GaussianARD(mu, 2, shape=(3,)) X.observe(3*np.ones(3)) (m0, m1) = mu._message_from_children() self.assertAllClose(m0, 2*3 * np.sum(np.ones(3), axis=-1, keepdims=True)) self.assertAllClose(m1, -0.5*2 * np.sum(np.identity(3), axis=(-1,-2), keepdims=True)) pass def test_message_to_parent_alpha(self): """ Test the message from GaussianARD the 2nd parent (alpha). """ # Check formula with uncertain parent mu mu = GaussianARD(1,1) tau = Gamma(0.5*1e10, 1e10) X = GaussianARD(mu, tau) X.observe(3) (m0, m1) = tau._message_from_children() self.assertAllClose(m0, -0.5*(3**2 - 2*3*1 + 1**2+1)) self.assertAllClose(m1, 0.5) # Check formula with uncertain node tau = Gamma(1e10, 1e10) X = GaussianARD(2, tau) Y = GaussianARD(X, 1) Y.observe(5) X.update() (m0, m1) = tau._message_from_children() self.assertAllClose(m0, -0.5*(1/(1+1)+3.5**2 - 2*3.5*2 + 2**2)) self.assertAllClose(m1, 0.5) # Check alpha larger than mu alpha = Gamma(np.ones((3,2,3))*1e10, 1e10) X = GaussianARD(np.ones((2,3)), alpha, ndim=3) X.observe(2*np.ones((3,2,3))) (m0, m1) = alpha._message_from_children() self.assertAllClose(m0 * np.ones((3,2,3)), -0.5*(2**2 - 2*2*1 + 1**2) * np.ones((3,2,3))) self.assertAllClose(m1*np.ones((3,2,3)), 0.5*np.ones((3,2,3))) # Check mu larger than alpha tau = Gamma(np.ones((2,3))*1e10, 1e10) X = GaussianARD(np.ones((3,2,3)), tau, ndim=3) X.observe(2*np.ones((3,2,3))) (m0, m1) = tau._message_from_children() self.assertAllClose(m0, -0.5*(2**2 - 2*2*1 + 1**2) * 3 * np.ones((2,3))) self.assertAllClose(m1 * np.ones((2,3)), 0.5 * 3 * np.ones((2,3))) # Check node larger than mu and alpha tau = Gamma(np.ones((3,))*1e10, 1e10) X = GaussianARD(np.ones((2,3)), tau, shape=(3,2,3)) X.observe(2*np.ones((3,2,3))) (m0, m1) = tau._message_from_children() self.assertAllClose(m0 * np.ones(3), -0.5*(2**2 - 2*2*1 + 1**2) * 6 * np.ones((3,))) self.assertAllClose(m1 * np.ones(3), 0.5 * 6 * np.ones(3)) # Check plates for smaller mu than node tau = Gamma(np.ones((4,1,2,3))*1e10, 1e10) X = GaussianARD(GaussianARD(1, 1, shape=(3,), plates=(4,1,1)), tau, shape=(2,3), plates=(4,5)) X.observe(2*np.ones((4,5,2,3))) (m0, m1) = tau._message_from_children() self.assertAllClose(m0 * np.ones((4,1,2,3)), (-0.5 * (2**2 - 2*2*1 + 1**2+1) * 5*np.ones((4,1,2,3)))) self.assertAllClose(m1 * np.ones((4,1,2,3)), 5*0.5 * np.ones((4,1,2,3))) # Check mask tau = Gamma(np.ones((4,3))*1e10, 1e10) X = GaussianARD(np.ones(3), tau, shape=(3,), plates=(2,4,)) X.observe(2*np.ones((2,4,3)), mask=[[True, False, True, False], [False, True, True, False]]) (m0, m1) = tau._message_from_children() self.assertAllClose(m0 * np.ones((4,3)), (-0.5 * (2**2 - 2*2*1 + 1**2) * np.ones((4,3)) * np.array([[1], [1], [2], [0]]))) self.assertAllClose(m1 * np.ones((4,3)), 0.5 * np.array([[1], [1], [2], [0]]) * np.ones((4,3))) # Check non-ARD Gaussian child mu = np.array([1,2]) alpha = np.array([3,4]) Alpha = Gamma(alpha*1e10, 1e10) Lambda = np.array([[1, 0.5], [0.5, 1]]) X = GaussianARD(mu, Alpha, ndim=1) Y = Gaussian(X, Lambda) y = np.array([5,6]) Y.observe(y) X.update() (m0, m1) = Alpha._message_from_children() Cov = np.linalg.inv(np.diag(alpha)+Lambda) mean = np.dot(Cov, np.dot(np.diag(alpha), mu) + np.dot(Lambda, y)) self.assertAllClose(m0 * np.ones(2), -0.5 * np.diag( np.outer(mean, mean) + Cov - np.outer(mean, mu) - np.outer(mu, mean) + np.outer(mu, mu))) self.assertAllClose(m1 * np.ones(2), 0.5 * np.ones(2)) pass def test_lowerbound(self): """ Test the variational Bayesian lower bound term for GaussianARD. """ # Test vector formula with full noise covariance m = np.random.randn(2) alpha = np.random.rand(2) y = np.random.randn(2) X = GaussianARD(m, alpha, ndim=1) V = np.array([[3,1],[1,3]]) Y = Gaussian(X, V) Y.observe(y) X.update() Cov = np.linalg.inv(np.diag(alpha) + V) mu = np.dot(Cov, np.dot(V, y) + alpha*m) x2 = np.outer(mu, mu) + Cov logH_X = (+ 2*0.5*(1+np.log(2*np.pi)) + 0.5*np.log(np.linalg.det(Cov))) logp_X = (- 2*0.5*np.log(2*np.pi) + 0.5*np.log(np.linalg.det(np.diag(alpha))) - 0.5*np.sum(np.diag(alpha) * (x2 - np.outer(mu,m) - np.outer(m,mu) + np.outer(m,m)))) self.assertAllClose(logp_X + logH_X, X.lower_bound_contribution()) def check_lower_bound(shape_mu, shape_alpha, plates_mu=(), **kwargs): M = GaussianARD(np.ones(plates_mu + shape_mu), np.ones(plates_mu + shape_mu), shape=shape_mu, plates=plates_mu) X = GaussianARD(M, 2*np.ones(shape_alpha), **kwargs) Y = GaussianARD(X, 3*np.ones(X.get_shape(0)), **kwargs) Y.observe(4*np.ones(Y.get_shape(0))) X.update() Cov = 1/(2+3) mu = Cov * (2*1 + 3*4) x2 = mu**2 + Cov logH_X = (+ 0.5*(1+np.log(2*np.pi)) + 0.5*np.log(Cov)) logp_X = (- 0.5*np.log(2*np.pi) + 0.5*np.log(2) - 0.5*2*(x2 - 2*mu*1 + 1**2+1)) r = np.prod(X.get_shape(0)) self.assertAllClose(r * (logp_X + logH_X), X.lower_bound_contribution()) # Test scalar formula check_lower_bound((), ()) # Test array formula check_lower_bound((2,3), (2,3)) # Test dim-broadcasting of mu check_lower_bound((3,1), (2,3,4)) # Test dim-broadcasting of alpha check_lower_bound((2,3,4), (3,1)) # Test dim-broadcasting of mu and alpha check_lower_bound((3,1), (3,1), shape=(2,3,4)) # Test dim-broadcasting of mu with plates check_lower_bound((), (), plates_mu=(), shape=(), plates=(5,)) # BUG: Scalar parents for array variable caused einsum error check_lower_bound((), (), shape=(3,)) # BUG: Log-det was summed over plates check_lower_bound((), (), shape=(3,), plates=(4,)) pass def test_rotate(self): """ Test the rotation of Gaussian ARD arrays. """ def check(shape, plates, einsum_x, einsum_xx, axis=-1): # TODO/FIXME: Improve by having non-diagonal precision/covariance # parameter for the Gaussian X D = shape[axis] X = GaussianARD(np.random.randn(*(plates+shape)), np.random.rand(*(plates+shape)), shape=shape, plates=plates) (x, xx) = X.get_moments() R = np.random.randn(D,D) X.rotate(R, axis=axis) (rx, rxxr) = X.get_moments() self.assertAllClose(rx, np.einsum(einsum_x, R, x)) self.assertAllClose(rxxr, np.einsum(einsum_xx, R, xx, R)) pass # Rotate vector check((3,), (), '...jk,...k->...j', '...mk,...kl,...nl->...mn') check((3,), (2,4), '...jk,...k->...j', '...mk,...kl,...nl->...mn') # Rotate array check((2,3,4), (), '...jc,...abc->...abj', '...mc,...abcdef,...nf->...abmden', axis=-1) check((2,3,4), (5,6), '...jc,...abc->...abj', '...mc,...abcdef,...nf->...abmden', axis=-1) check((2,3,4), (), '...jb,...abc->...ajc', '...mb,...abcdef,...ne->...amcdnf', axis=-2) check((2,3,4), (5,6), '...jb,...abc->...ajc', '...mb,...abcdef,...ne->...amcdnf', axis=-2) check((2,3,4), (), '...ja,...abc->...jbc', '...ma,...abcdef,...nd->...mbcnef', axis=-3) check((2,3,4), (5,6), '...ja,...abc->...jbc', '...ma,...abcdef,...nd->...mbcnef', axis=-3) pass def test_rotate_plates(self): # Basic test for Gaussian vectors X = GaussianARD(np.random.randn(3,2), np.random.rand(3,2), shape=(2,), plates=(3,)) (u0, u1) = X.get_moments() Cov = u1 - linalg.outer(u0, u0, ndim=1) Q = np.random.randn(3,3) Qu0 = np.einsum('ik,kj->ij', Q, u0) QCov = np.einsum('k,kij->kij', np.sum(Q, axis=0)**2, Cov) Qu1 = QCov + linalg.outer(Qu0, Qu0, ndim=1) X.rotate_plates(Q, plate_axis=-1) (u0, u1) = X.get_moments() self.assertAllClose(u0, Qu0) self.assertAllClose(u1, Qu1) # Test full covariance, that is, with observations X = GaussianARD(np.random.randn(3,2), np.random.rand(3,2), shape=(2,), plates=(3,)) Y = Gaussian(X, [[2.0, 1.5], [1.5, 3.0]], plates=(3,)) Y.observe(np.random.randn(3,2)) X.update() (u0, u1) = X.get_moments() Cov = u1 - linalg.outer(u0, u0, ndim=1) Q = np.random.randn(3,3) Qu0 = np.einsum('ik,kj->ij', Q, u0) QCov = np.einsum('k,kij->kij', np.sum(Q, axis=0)**2, Cov) Qu1 = QCov + linalg.outer(Qu0, Qu0, ndim=1) X.rotate_plates(Q, plate_axis=-1) (u0, u1) = X.get_moments() self.assertAllClose(u0, Qu0) self.assertAllClose(u1, Qu1) pass def test_initialization(self): """ Test initialization methods of GaussianARD """ X = GaussianARD(1, 2, shape=(2,), plates=(3,)) # Prior initialization mu = 1 * np.ones((3, 2)) alpha = 2 * np.ones((3, 2)) X.initialize_from_prior() u = X._message_to_child() self.assertAllClose(u[0]*np.ones((3,2)), mu) self.assertAllClose(u[1]*np.ones((3,2,2)), linalg.outer(mu, mu, ndim=1) + misc.diag(1/alpha, ndim=1)) # Parameter initialization mu = np.random.randn(3, 2) alpha = np.random.rand(3, 2) X.initialize_from_parameters(mu, alpha) u = X._message_to_child() self.assertAllClose(u[0], mu) self.assertAllClose(u[1], linalg.outer(mu, mu, ndim=1) + misc.diag(1/alpha, ndim=1)) # Value initialization x = np.random.randn(3, 2) X.initialize_from_value(x) u = X._message_to_child() self.assertAllClose(u[0], x) self.assertAllClose(u[1], linalg.outer(x, x, ndim=1)) # Random initialization X.initialize_from_random() pass class TestGaussianGammaISO(TestCase): """ Unit tests for GaussianGammaISO node. """ def test_init(self): """ Test the creation of GaussianGammaISO node """ # Simple construction X_alpha = GaussianGammaISO([1,2,3], np.identity(3), 2, 10) self.assertEqual(X_alpha.plates, ()) self.assertEqual(X_alpha.dims, ( (3,), (3,3), (), () )) # Plates X_alpha = GaussianGammaISO([1,2,3], np.identity(3), 2, 10, plates=(4,)) self.assertEqual(X_alpha.plates, (4,)) self.assertEqual(X_alpha.dims, ( (3,), (3,3), (), () )) # Plates in mu X_alpha = GaussianGammaISO(np.ones((4,3)), np.identity(3), 2, 10) self.assertEqual(X_alpha.plates, (4,)) self.assertEqual(X_alpha.dims, ( (3,), (3,3), (), () )) # Plates in Lambda X_alpha = GaussianGammaISO(np.ones(3), np.ones((4,3,3))*np.identity(3), 2, 10) self.assertEqual(X_alpha.plates, (4,)) self.assertEqual(X_alpha.dims, ( (3,), (3,3), (), () )) # Plates in a X_alpha = GaussianGammaISO(np.ones(3), np.identity(3), np.ones(4), 10) self.assertEqual(X_alpha.plates, (4,)) self.assertEqual(X_alpha.dims, ( (3,), (3,3), (), () )) # Plates in Lambda X_alpha = GaussianGammaISO(np.ones(3), np.identity(3), 2, np.ones(4)) self.assertEqual(X_alpha.plates, (4,)) self.assertEqual(X_alpha.dims, ( (3,), (3,3), (), () )) # Inconsistent plates self.assertRaises(ValueError, GaussianGammaISO, np.ones((4,3)), np.identity(3), 2, 10, plates=()) # Inconsistent plates self.assertRaises(ValueError, GaussianGammaISO, np.ones((4,3)), np.identity(3), 2, 10, plates=(5,)) # Unknown parameters mu = Gaussian(np.zeros(3), np.identity(3)) Lambda = Wishart(10, np.identity(3)) b = Gamma(1, 1) X_alpha = GaussianGammaISO(mu, Lambda, 2, b) self.assertEqual(X_alpha.plates, ()) self.assertEqual(X_alpha.dims, ( (3,), (3,3), (), () )) # mu is Gaussian-gamma mu_tau = GaussianGammaISO(np.ones(3), np.identity(3), 5, 5) X_alpha = GaussianGammaISO(mu_tau, np.identity(3), 5, 5) self.assertEqual(X_alpha.plates, ()) self.assertEqual(X_alpha.dims, ( (3,), (3,3), (), () )) pass def test_message_to_child(self): """ Test the message to child of GaussianGammaISO node. """ # Simple test mu = np.array([1,2,3]) Lambda = np.identity(3) a = 2 b = 10 X_alpha = GaussianGammaISO(mu, Lambda, a, b) u = X_alpha._message_to_child() self.assertEqual(len(u), 4) tau = np.array(a/b) self.assertAllClose(u[0], tau[...,None] * mu) self.assertAllClose(u[1], (linalg.inv(Lambda) + tau[...,None,None] * linalg.outer(mu, mu))) self.assertAllClose(u[2], tau) self.assertAllClose(u[3], -np.log(b) + special.psi(a)) # Test with unknown parents mu = Gaussian(np.arange(3), 10*np.identity(3)) Lambda = Wishart(10, np.identity(3)) a = 2 b = Gamma(3, 15) X_alpha = GaussianGammaISO(mu, Lambda, a, b) u = X_alpha._message_to_child() (mu, mumu) = mu._message_to_child() Cov_mu = mumu - linalg.outer(mu, mu) (Lambda, _) = Lambda._message_to_child() (b, _) = b._message_to_child() (tau, logtau) = Gamma(a, b + 0.5*np.sum(Lambda*Cov_mu))._message_to_child() self.assertAllClose(u[0], tau[...,None] * mu) self.assertAllClose(u[1], (linalg.inv(Lambda) + tau[...,None,None] * linalg.outer(mu, mu))) self.assertAllClose(u[2], tau) self.assertAllClose(u[3], logtau) # Test with plates mu = Gaussian(np.reshape(np.arange(3*4), (4,3)), 10*np.identity(3), plates=(4,)) Lambda = Wishart(10, np.identity(3)) a = 2 b = Gamma(3, 15) X_alpha = GaussianGammaISO(mu, Lambda, a, b, plates=(4,)) u = X_alpha._message_to_child() (mu, mumu) = mu._message_to_child() Cov_mu = mumu - linalg.outer(mu, mu) (Lambda, _) = Lambda._message_to_child() (b, _) = b._message_to_child() (tau, logtau) = Gamma(a, b + 0.5*np.sum(Lambda*Cov_mu, axis=(-1,-2)))._message_to_child() self.assertAllClose(u[0] * np.ones((4,1)), np.ones((4,1)) * tau[...,None] * mu) self.assertAllClose(u[1] * np.ones((4,1,1)), np.ones((4,1,1)) * (linalg.inv(Lambda) + tau[...,None,None] * linalg.outer(mu, mu))) self.assertAllClose(u[2] * np.ones(4), np.ones(4) * tau) self.assertAllClose(u[3] * np.ones(4), np.ones(4) * logtau) pass def test_mask_to_parent(self): """ Test the mask handling in GaussianGammaISO node """ pass class TestGaussianGradient(TestCase): """Numerically check Riemannian gradient of several nodes. Using VB-EM update equations will take a unit length step to the Riemannian gradient direction. Thus, the change caused by a VB-EM update and the Riemannian gradient should be equal. """ def test_riemannian_gradient(self): """Test Riemannian gradient of a Gaussian node.""" D = 3 # # Without observations # # Construct model mu = np.random.randn(D) Lambda = random.covariance(D) X = Gaussian(mu, Lambda) # Random initialization mu0 = np.random.randn(D) Lambda0 = random.covariance(D) X.initialize_from_parameters(mu0, Lambda0) # Initial parameters phi0 = X.phi # Gradient g = X.get_riemannian_gradient() # Parameters after VB-EM update X.update() phi1 = X.phi # Check self.assertAllClose(g[0], phi1[0] - phi0[0]) self.assertAllClose(g[1], phi1[1] - phi0[1]) # TODO/FIXME: Actually, gradient should be zero because cost function # is zero without observations! Use the mask! # # With observations # # Construct model mu = np.random.randn(D) Lambda = random.covariance(D) X = Gaussian(mu, Lambda) V = random.covariance(D) Y = Gaussian(X, V) Y.observe(np.random.randn(D)) # Random initialization mu0 = np.random.randn(D) Lambda0 = random.covariance(D) X.initialize_from_parameters(mu0, Lambda0) # Initial parameters phi0 = X.phi # Gradient g = X.get_riemannian_gradient() # Parameters after VB-EM update X.update() phi1 = X.phi # Check self.assertAllClose(g[0], phi1[0] - phi0[0]) self.assertAllClose(g[1], phi1[1] - phi0[1]) pass def test_gradient(self): """Test standard gradient of a Gaussian node.""" D = 3 np.random.seed(42) # # Without observations # # Construct model mu = np.random.randn(D) Lambda = random.covariance(D) X = Gaussian(mu, Lambda) # Random initialization mu0 = np.random.randn(D) Lambda0 = random.covariance(D) X.initialize_from_parameters(mu0, Lambda0) Q = VB(X) # Initial parameters phi0 = X.phi # Gradient rg = X.get_riemannian_gradient() g = X.get_gradient(rg) # Numerical gradient eps = 1e-6 p0 = X.get_parameters() l0 = Q.compute_lowerbound(ignore_masked=False) g_num = [np.zeros(D), np.zeros((D,D))] for i in range(D): e = np.zeros(D) e[i] = eps p1 = p0[0] + e X.set_parameters([p1, p0[1]]) l1 = Q.compute_lowerbound(ignore_masked=False) g_num[0][i] = (l1 - l0) / eps for i in range(D): for j in range(i+1): e = np.zeros((D,D)) e[i,j] += eps e[j,i] += eps p1 = p0[1] + e X.set_parameters([p0[0], p1]) l1 = Q.compute_lowerbound(ignore_masked=False) g_num[1][i,j] = (l1 - l0) / (2*eps) g_num[1][j,i] = (l1 - l0) / (2*eps) # Check self.assertAllClose(g[0], g_num[0]) self.assertAllClose(g[1], g_num[1]) # # With observations # # Construct model mu = np.random.randn(D) Lambda = random.covariance(D) X = Gaussian(mu, Lambda) # Random initialization mu0 = np.random.randn(D) Lambda0 = random.covariance(D) X.initialize_from_parameters(mu0, Lambda0) V = random.covariance(D) Y = Gaussian(X, V) Y.observe(np.random.randn(D)) Q = VB(Y, X) # Initial parameters phi0 = X.phi # Gradient rg = X.get_riemannian_gradient() g = X.get_gradient(rg) # Numerical gradient eps = 1e-6 p0 = X.get_parameters() l0 = Q.compute_lowerbound() g_num = [np.zeros(D), np.zeros((D,D))] for i in range(D): e = np.zeros(D) e[i] = eps p1 = p0[0] + e X.set_parameters([p1, p0[1]]) l1 = Q.compute_lowerbound() g_num[0][i] = (l1 - l0) / eps for i in range(D): for j in range(i+1): e = np.zeros((D,D)) e[i,j] += eps e[j,i] += eps p1 = p0[1] + e X.set_parameters([p0[0], p1]) l1 = Q.compute_lowerbound() g_num[1][i,j] = (l1 - l0) / (2*eps) g_num[1][j,i] = (l1 - l0) / (2*eps) # Check self.assertAllClose(g[0], g_num[0]) self.assertAllClose(g[1], g_num[1]) # # With plates # # Construct model K = D+1 mu = np.random.randn(D) Lambda = random.covariance(D) X = Gaussian(mu, Lambda, plates=(K,)) V = random.covariance(D, size=(K,)) Y = Gaussian(X, V) Y.observe(np.random.randn(K,D)) Q = VB(Y, X) # Random initialization mu0 = np.random.randn(*(X.get_shape(0))) Lambda0 = random.covariance(D, size=X.plates) X.initialize_from_parameters(mu0, Lambda0) # Initial parameters phi0 = X.phi # Gradient rg = X.get_riemannian_gradient() g = X.get_gradient(rg) # Numerical gradient eps = 1e-6 p0 = X.get_parameters() l0 = Q.compute_lowerbound() g_num = [np.zeros(X.get_shape(0)), np.zeros(X.get_shape(1))] for k in range(K): for i in range(D): e = np.zeros(X.get_shape(0)) e[k,i] = eps p1 = p0[0] + e X.set_parameters([p1, p0[1]]) l1 = Q.compute_lowerbound() g_num[0][k,i] = (l1 - l0) / eps for i in range(D): for j in range(i+1): e = np.zeros(X.get_shape(1)) e[k,i,j] += eps e[k,j,i] += eps p1 = p0[1] + e X.set_parameters([p0[0], p1]) l1 = Q.compute_lowerbound() g_num[1][k,i,j] = (l1 - l0) / (2*eps) g_num[1][k,j,i] = (l1 - l0) / (2*eps) # Check self.assertAllClose(g[0], g_num[0]) self.assertAllClose(g[1], g_num[1]) pass bayespy-0.4.0/bayespy/inference/vmp/nodes/tests/test_binomial.py0000644000175000017500000001171512615643655026620 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for `binomial` module. """ import numpy as np import scipy from bayespy.nodes import (Binomial, Beta, Mixture) from bayespy.utils import random from bayespy.utils.misc import TestCase class TestBinomial(TestCase): """ Unit tests for Binomial node """ def test_init(self): """ Test the creation of binomial nodes. """ # Some simple initializations X = Binomial(10, 0.5) X = Binomial(10, Beta([2,3])) # Check that plates are correct X = Binomial(10, 0.7, plates=(4,3)) self.assertEqual(X.plates, (4,3)) X = Binomial(10, 0.7*np.ones((4,3))) self.assertEqual(X.plates, (4,3)) n = np.ones((4,3), dtype=np.int) X = Binomial(n, 0.7) self.assertEqual(X.plates, (4,3)) X = Binomial(10, Beta([4,3], plates=(4,3))) self.assertEqual(X.plates, (4,3)) # Invalid probability self.assertRaises(ValueError, Binomial, 10, -0.5) self.assertRaises(ValueError, Binomial, 10, 1.5) # Invalid number of trials self.assertRaises(ValueError, Binomial, -1, 0.5) self.assertRaises(ValueError, Binomial, 8.5, 0.5) # Inconsistent plates self.assertRaises(ValueError, Binomial, 10, 0.5*np.ones(4), plates=(3,)) # Explicit plates too small self.assertRaises(ValueError, Binomial, 10, 0.5*np.ones(4), plates=(1,)) pass def test_moments(self): """ Test the moments of binomial nodes. """ # Simple test X = Binomial(1, 0.7) u = X._message_to_child() self.assertEqual(len(u), 1) self.assertAllClose(u[0], 0.7) # Test n X = Binomial(10, 0.7) u = X._message_to_child() self.assertAllClose(u[0], 10*0.7) # Test plates in p n = np.random.randint(1, 10) p = np.random.rand(3) X = Binomial(n, p) u = X._message_to_child() self.assertAllClose(u[0], p*n) # Test plates in n n = np.random.randint(1, 10, size=(3,)) p = np.random.rand() X = Binomial(n, p) u = X._message_to_child() self.assertAllClose(u[0], p*n) # Test plates in p and n n = np.random.randint(1, 10, size=(4,1)) p = np.random.rand(3) X = Binomial(n, p) u = X._message_to_child() self.assertAllClose(u[0], p*n) # Test with beta prior P = Beta([7, 3]) logp = P._message_to_child()[0] p0 = np.exp(logp[0]) / (np.exp(logp[0]) + np.exp(logp[1])) X = Binomial(1, P) u = X._message_to_child() self.assertAllClose(u[0], p0) # Test with broadcasted plates P = Beta([7, 3], plates=(10,)) X = Binomial(5, P) u = X._message_to_child() self.assertAllClose(u[0] * np.ones(X.get_shape(0)), 5*p0*np.ones(10)) pass def test_mixture(self): """ Test binomial mixture """ X = Mixture(2, Binomial, 10, [0.1, 0.2, 0.3, 0.4]) u = X._message_to_child() self.assertAllClose(u[0], 3.0) pass def test_observed(self): """ Test observation of Bernoulli node """ Z = Binomial(10, 0.3) Z.observe(10) u = Z._message_to_child() self.assertAllClose(u[0], 10) Z = Binomial(10, 0.9) Z.observe(2) u = Z._message_to_child() self.assertAllClose(u[0], 2) pass def test_random(self): """ Test random sampling in Binomial node """ N = [ [5], [50] ] p = [1.0, 0.0] with np.errstate(divide='ignore'): Z = Binomial(N, p, plates=(3,2,2)).random() self.assertArrayEqual(Z, np.ones((3,2,2))*N*p) bayespy-0.4.0/bayespy/inference/vmp/nodes/tests/test_beta.py0000644000175000017500000000515312615643655025740 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for `beta` module. """ import numpy as np from scipy import special from bayespy.nodes import Beta from bayespy.utils import random from bayespy.utils.misc import TestCase class TestBeta(TestCase): """ Unit tests for Beta node """ def test_init(self): """ Test the creation of beta nodes. """ # Some simple initializations p = Beta([1.5, 4.2]) # Check that plates are correct p = Beta([2, 3], plates=(4,3)) self.assertEqual(p.plates, (4,3)) p = Beta(np.ones((4,3,2))) self.assertEqual(p.plates, (4,3)) # Parent not a vector self.assertRaises(ValueError, Beta, 4) # Parent vector has wrong shape self.assertRaises(ValueError, Beta, [4]) self.assertRaises(ValueError, Beta, [4,4,4]) # Parent vector has invalid values self.assertRaises(ValueError, Beta, [-2,3]) # Plates inconsistent self.assertRaises(ValueError, Beta, np.ones((4,2)), plates=(3,)) # Explicit plates too small self.assertRaises(ValueError, Beta, np.ones((4,2)), plates=(1,)) pass def test_moments(self): """ Test the moments of beta nodes. """ p = Beta([2, 3]) u = p._message_to_child() self.assertAllClose(u[0], special.psi([2,3]) - special.psi(2+3)) pass def test_random(self): """ Test random sampling of beta nodes. """ p = Beta([1e20, 3e20]) x = p.random() self.assertAllClose(x, 0.25) p = Beta([[1e20, 3e20], [1e20, 1e20]]) x = p.random() self.assertAllClose(x, [0.25, 0.5]) p = Beta([1e20, 3e20], plates=(3,)) x = p.random() self.assertAllClose(x, [0.25, 0.25, 0.25]) pass bayespy-0.4.0/bayespy/inference/vmp/nodes/tests/test_categorical.py0000644000175000017500000001423512615643655027303 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for `categorical` module. """ import warnings import numpy as np import scipy from bayespy.nodes import (Categorical, Dirichlet, Mixture, Gamma) from bayespy.utils import random from bayespy.utils.misc import TestCase class TestCategorical(TestCase): """ Unit tests for Categorical node """ def test_init(self): """ Test the creation of categorical nodes. """ # Some simple initializations X = Categorical([0.1, 0.3, 0.6]) X = Categorical(Dirichlet([5,4,3])) # Check that plates are correct X = Categorical([0.1, 0.3, 0.6], plates=(3,4)) self.assertEqual(X.plates, (3,4)) X = Categorical(0.25*np.ones((2,3,4))) self.assertEqual(X.plates, (2,3)) X = Categorical(Dirichlet([2,1,9], plates=(3,4))) self.assertEqual(X.plates, (3,4)) # Probabilities not a vector self.assertRaises(ValueError, Categorical, 0.5) # Invalid probability self.assertRaises(ValueError, Categorical, [-0.5, 1.5], n=10) self.assertRaises(ValueError, Categorical, [0.5, 1.5], n=10) # Inconsistent plates self.assertRaises(ValueError, Categorical, 0.25*np.ones((2,4)), plates=(3,), n=10) # Explicit plates too small self.assertRaises(ValueError, Categorical, 0.25*np.ones((2,4)), plates=(1,), n=10) pass def test_moments(self): """ Test the moments of categorical nodes. """ # Simple test X = Categorical([0.7,0.2,0.1]) u = X._message_to_child() self.assertEqual(len(u), 1) self.assertAllClose(u[0], [0.7,0.2,0.1]) # Test plates in p p = np.random.dirichlet([1,1], size=3) X = Categorical(p) u = X._message_to_child() self.assertAllClose(u[0], p) # Test with Dirichlet prior P = Dirichlet([7, 3]) logp = P._message_to_child()[0] p0 = np.exp(logp[0]) / (np.exp(logp[0]) + np.exp(logp[1])) p1 = np.exp(logp[1]) / (np.exp(logp[0]) + np.exp(logp[1])) X = Categorical(P) u = X._message_to_child() p = np.array([p0, p1]) self.assertAllClose(u[0], p) # Test with broadcasted plates P = Dirichlet([7, 3], plates=(10,)) X = Categorical(P) u = X._message_to_child() self.assertAllClose(u[0] * np.ones(X.get_shape(0)), p*np.ones((10,1))) pass def test_observed(self): """ Test observed categorical nodes """ # Single observation X = Categorical([0.7,0.2,0.1]) X.observe(2) u = X._message_to_child() self.assertAllClose(u[0], [0,0,1]) # One plate axis X = Categorical([0.7,0.2,0.1], plates=(2,)) X.observe([2,1]) u = X._message_to_child() self.assertAllClose(u[0], [[0,0,1], [0,1,0]]) # Several plate axes X = Categorical([0.7,0.1,0.1,0.1], plates=(2,3,)) X.observe([[2,1,1], [0,2,3]]) u = X._message_to_child() self.assertAllClose(u[0], [ [[0,0,1,0], [0,1,0,0], [0,1,0,0]], [[1,0,0,0], [0,0,1,0], [0,0,0,1]] ]) # Check invalid observations X = Categorical([0.7,0.2,0.1]) self.assertRaises(ValueError, X.observe, -1) self.assertRaises(ValueError, X.observe, 3) self.assertRaises(ValueError, X.observe, 1.5) pass def test_constant(self): """ Test constant categorical nodes """ # Basic test Y = Mixture(2, Gamma, [1, 2, 3], [1, 1, 1]) u = Y._message_to_child() self.assertAllClose(u[0], 3/1) # Test with one plate axis alpha = [[1, 2, 3], [4, 5, 6]] Y = Mixture([2, 1], Gamma, alpha, 1) u = Y._message_to_child() self.assertAllClose(u[0], [3, 5]) # Test with two plate axes alpha = [ [[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]] ] Y = Mixture([[2, 1], [0, 2]], Gamma, alpha, 1) u = Y._message_to_child() self.assertAllClose(u[0], [[3, 5], [7, 12]]) pass def test_initialization(self): """ Test initialization of categorical nodes """ # Test initialization from random with warnings.catch_warnings(): warnings.simplefilter("ignore", RuntimeWarning) Z = Categorical([[0.0, 1.0, 0.0], [0.0, 0.0, 1.0]]) Z.initialize_from_random() u = Z._message_to_child() self.assertAllClose(u[0], [[0, 1, 0], [0, 0, 1]]) pass bayespy-0.4.0/bayespy/inference/vmp/nodes/tests/test_gamma.py0000644000175000017500000001274212615643655026111 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2015 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for `gamma` module. """ import numpy as np from scipy import special from numpy import testing from .. import gaussian from bayespy.nodes import (Gaussian, GaussianARD, GaussianGammaISO, Gamma, Wishart) from ...vmp import VB from bayespy.utils import misc from bayespy.utils import linalg from bayespy.utils import random from bayespy.utils.misc import TestCase class TestGammaGradient(TestCase): """Numerically check Riemannian gradient of several nodes. Using VB-EM update equations will take a unit length step to the Riemannian gradient direction. Thus, the change caused by a VB-EM update and the Riemannian gradient should be equal. """ def test_riemannian_gradient(self): """Test Riemannian gradient of a Gamma node.""" # # Without observations # # Construct model a = np.random.rand() b = np.random.rand() tau = Gamma(a, b) # Random initialization tau.initialize_from_parameters(np.random.rand(), np.random.rand()) # Initial parameters phi0 = tau.phi # Gradient g = tau.get_riemannian_gradient() # Parameters after VB-EM update tau.update() phi1 = tau.phi # Check self.assertAllClose(g[0], phi1[0] - phi0[0]) self.assertAllClose(g[1], phi1[1] - phi0[1]) # # With observations # # Construct model a = np.random.rand() b = np.random.rand() tau = Gamma(a, b) mu = np.random.randn() Y = GaussianARD(mu, tau) Y.observe(np.random.randn()) # Random initialization tau.initialize_from_parameters(np.random.rand(), np.random.rand()) # Initial parameters phi0 = tau.phi # Gradient g = tau.get_riemannian_gradient() # Parameters after VB-EM update tau.update() phi1 = tau.phi # Check self.assertAllClose(g[0], phi1[0] - phi0[0]) self.assertAllClose(g[1], phi1[1] - phi0[1]) pass def test_gradient(self): """Test standard gradient of a Gamma node.""" D = 3 np.random.seed(42) # # Without observations # # Construct model a = np.random.rand(D) b = np.random.rand(D) tau = Gamma(a, b) Q = VB(tau) # Random initialization tau.initialize_from_parameters(np.random.rand(D), np.random.rand(D)) # Initial parameters phi0 = tau.phi # Gradient rg = tau.get_riemannian_gradient() g = tau.get_gradient(rg) # Numerical gradient eps = 1e-8 p0 = tau.get_parameters() l0 = Q.compute_lowerbound(ignore_masked=False) g_num = [np.zeros(D), np.zeros(D)] for i in range(D): e = np.zeros(D) e[i] = eps p1 = p0[0] + e tau.set_parameters([p1, p0[1]]) l1 = Q.compute_lowerbound(ignore_masked=False) g_num[0][i] = (l1 - l0) / eps for i in range(D): e = np.zeros(D) e[i] = eps p1 = p0[1] + e tau.set_parameters([p0[0], p1]) l1 = Q.compute_lowerbound(ignore_masked=False) g_num[1][i] = (l1 - l0) / eps # Check self.assertAllClose(g[0], g_num[0]) self.assertAllClose(g[1], g_num[1]) # # With observations # # Construct model a = np.random.rand(D) b = np.random.rand(D) tau = Gamma(a, b) mu = np.random.randn(D) Y = GaussianARD(mu, tau) Y.observe(np.random.randn(D)) Q = VB(Y, tau) # Random initialization tau.initialize_from_parameters(np.random.rand(D), np.random.rand(D)) # Initial parameters phi0 = tau.phi # Gradient rg = tau.get_riemannian_gradient() g = tau.get_gradient(rg) # Numerical gradient eps = 1e-8 p0 = tau.get_parameters() l0 = Q.compute_lowerbound(ignore_masked=False) g_num = [np.zeros(D), np.zeros(D)] for i in range(D): e = np.zeros(D) e[i] = eps p1 = p0[0] + e tau.set_parameters([p1, p0[1]]) l1 = Q.compute_lowerbound(ignore_masked=False) g_num[0][i] = (l1 - l0) / eps for i in range(D): e = np.zeros(D) e[i] = eps p1 = p0[1] + e tau.set_parameters([p0[0], p1]) l1 = Q.compute_lowerbound(ignore_masked=False) g_num[1][i] = (l1 - l0) / eps # Check self.assertAllClose(g[0], g_num[0]) self.assertAllClose(g[1], g_num[1]) pass bayespy-0.4.0/bayespy/inference/vmp/nodes/tests/test_dot.py0000644000175000017500000007567112615643655025627 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2013-2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for `dot` module. """ import unittest import numpy as np import scipy from numpy import testing from ..dot import Dot, SumMultiply from ..gaussian import Gaussian, GaussianARD from bayespy.nodes import GaussianGammaISO from ...vmp import VB from bayespy.utils import misc from bayespy.utils import linalg from bayespy.utils import random from bayespy.utils.misc import TestCase class TestSumMultiply(TestCase): def test_parent_validity(self): """ Test that the parent nodes are validated properly in SumMultiply """ V = GaussianARD(1, 1) X = Gaussian(np.ones(1), np.identity(1)) Y = Gaussian(np.ones(3), np.identity(3)) Z = Gaussian(np.ones(5), np.identity(5)) A = SumMultiply(X, ['i']) self.assertEqual(A.dims, ((), ())) A = SumMultiply('i', X) self.assertEqual(A.dims, ((), ())) A = SumMultiply(X, ['i'], ['i']) self.assertEqual(A.dims, ((1,), (1,1))) A = SumMultiply('i->i', X) self.assertEqual(A.dims, ((1,), (1,1))) A = SumMultiply(X, ['i'], Y, ['j'], ['i','j']) self.assertEqual(A.dims, ((1,3), (1,3,1,3))) A = SumMultiply('i,j->ij', X, Y) self.assertEqual(A.dims, ((1,3), (1,3,1,3))) A = SumMultiply(V, [], X, ['i'], Y, ['i'], []) self.assertEqual(A.dims, ((), ())) A = SumMultiply(',i,i->', V, X, Y) self.assertEqual(A.dims, ((), ())) # Gaussian-gamma parents C = GaussianGammaISO(np.ones(3), np.identity(3), 1, 1) A = SumMultiply(Y, ['i'], C, ['i'], ['i']) self.assertEqual(A.dims, ((3,), (3,3), (), ())) A = SumMultiply('i,i->i', Y, C) self.assertEqual(A.dims, ((3,), (3,3), (), ())) C = GaussianGammaISO(np.ones(3), np.identity(3), 1, 1) A = SumMultiply(Y, ['i'], C, ['i'], []) self.assertEqual(A.dims, ((), (), (), ())) A = SumMultiply('i,i->', Y, C) self.assertEqual(A.dims, ((), (), (), ())) # Error: not enough inputs self.assertRaises(ValueError, SumMultiply) self.assertRaises(ValueError, SumMultiply, X) # Error: too many keys self.assertRaises(ValueError, SumMultiply, Y, ['i', 'j']) self.assertRaises(ValueError, SumMultiply, 'ij', Y) # Error: not broadcastable self.assertRaises(ValueError, SumMultiply, Y, ['i'], Z, ['i']) self.assertRaises(ValueError, SumMultiply, 'i,i', Y, Z) # Error: output key not in inputs self.assertRaises(ValueError, SumMultiply, X, ['i'], ['j']) self.assertRaises(ValueError, SumMultiply, 'i->j', X) # Error: non-unique input keys self.assertRaises(ValueError, SumMultiply, X, ['i','i']) self.assertRaises(ValueError, SumMultiply, 'ii', X) # Error: non-unique output keys self.assertRaises(ValueError, SumMultiply, X, ['i'], ['i','i']) self.assertRaises(ValueError, SumMultiply, 'i->ii', X) # String has too many '->' self.assertRaises(ValueError, SumMultiply, 'i->i->i', X) # String has too many input nodes self.assertRaises(ValueError, SumMultiply, 'i,i->i', X) # Same parent several times self.assertRaises(ValueError, SumMultiply, 'i,i->i', X, X) # Same parent several times via deterministic node Xh = SumMultiply('i->i', X) self.assertRaises(ValueError, SumMultiply, 'i,i->i', X, Xh) def test_message_to_child(self): """ Test the message from SumMultiply to its children. """ def compare_moments(u0, u1, *args): Y = SumMultiply(*args) u_Y = Y.get_moments() self.assertAllClose(u_Y[0], u0) self.assertAllClose(u_Y[1], u1) # Test constant parent y = np.random.randn(2,3,4) compare_moments(y, linalg.outer(y, y, ndim=2), 'ij->ij', y) # Do nothing for 2-D array Y = GaussianARD(np.random.randn(5,2,3), np.random.rand(5,2,3), plates=(5,), shape=(2,3)) y = Y.get_moments() compare_moments(y[0], y[1], 'ij->ij', Y) compare_moments(y[0], y[1], Y, [0,1], [0,1]) # Sum over the rows of a matrix Y = GaussianARD(np.random.randn(5,2,3), np.random.rand(5,2,3), plates=(5,), shape=(2,3)) y = Y.get_moments() mu = np.einsum('...ij->...j', y[0]) cov = np.einsum('...ijkl->...jl', y[1]) compare_moments(mu, cov, 'ij->j', Y) compare_moments(mu, cov, Y, [0,1], [1]) # Inner product of three vectors X1 = GaussianARD(np.random.randn(2), np.random.rand(2), plates=(), shape=(2,)) x1 = X1.get_moments() X2 = GaussianARD(np.random.randn(6,1,2), np.random.rand(6,1,2), plates=(6,1), shape=(2,)) x2 = X2.get_moments() X3 = GaussianARD(np.random.randn(7,6,5,2), np.random.rand(7,6,5,2), plates=(7,6,5), shape=(2,)) x3 = X3.get_moments() mu = np.einsum('...i,...i,...i->...', x1[0], x2[0], x3[0]) cov = np.einsum('...ij,...ij,...ij->...', x1[1], x2[1], x3[1]) compare_moments(mu, cov, 'i,i,i', X1, X2, X3) compare_moments(mu, cov, 'i,i,i->', X1, X2, X3) compare_moments(mu, cov, X1, [9], X2, [9], X3, [9]) compare_moments(mu, cov, X1, [9], X2, [9], X3, [9], []) # Outer product of two vectors X1 = GaussianARD(np.random.randn(2), np.random.rand(2), plates=(5,), shape=(2,)) x1 = X1.get_moments() X2 = GaussianARD(np.random.randn(6,1,2), np.random.rand(6,1,2), plates=(6,1), shape=(2,)) x2 = X2.get_moments() mu = np.einsum('...i,...j->...ij', x1[0], x2[0]) cov = np.einsum('...ik,...jl->...ijkl', x1[1], x2[1]) compare_moments(mu, cov, 'i,j->ij', X1, X2) compare_moments(mu, cov, X1, [9], X2, [7], [9,7]) # Matrix product Y1 = GaussianARD(np.random.randn(3,2), np.random.rand(3,2), plates=(), shape=(3,2)) y1 = Y1.get_moments() Y2 = GaussianARD(np.random.randn(5,2,3), np.random.rand(5,2,3), plates=(5,), shape=(2,3)) y2 = Y2.get_moments() mu = np.einsum('...ik,...kj->...ij', y1[0], y2[0]) cov = np.einsum('...ikjl,...kmln->...imjn', y1[1], y2[1]) compare_moments(mu, cov, 'ik,kj->ij', Y1, Y2) compare_moments(mu, cov, Y1, ['i','k'], Y2, ['k','j'], ['i','j']) # Trace of a matrix product Y1 = GaussianARD(np.random.randn(3,2), np.random.rand(3,2), plates=(), shape=(3,2)) y1 = Y1.get_moments() Y2 = GaussianARD(np.random.randn(5,2,3), np.random.rand(5,2,3), plates=(5,), shape=(2,3)) y2 = Y2.get_moments() mu = np.einsum('...ij,...ji->...', y1[0], y2[0]) cov = np.einsum('...ikjl,...kilj->...', y1[1], y2[1]) compare_moments(mu, cov, 'ij,ji', Y1, Y2) compare_moments(mu, cov, 'ij,ji->', Y1, Y2) compare_moments(mu, cov, Y1, ['i','j'], Y2, ['j','i']) compare_moments(mu, cov, Y1, ['i','j'], Y2, ['j','i'], []) # Vector-matrix-vector product X1 = GaussianARD(np.random.randn(3), np.random.rand(3), plates=(), shape=(3,)) x1 = X1.get_moments() X2 = GaussianARD(np.random.randn(6,1,2), np.random.rand(6,1,2), plates=(6,1), shape=(2,)) x2 = X2.get_moments() Y = GaussianARD(np.random.randn(3,2), np.random.rand(3,2), plates=(), shape=(3,2)) y = Y.get_moments() mu = np.einsum('...i,...ij,...j->...', x1[0], y[0], x2[0]) cov = np.einsum('...ia,...ijab,...jb->...', x1[1], y[1], x2[1]) compare_moments(mu, cov, 'i,ij,j', X1, Y, X2) compare_moments(mu, cov, X1, [1], Y, [1,2], X2, [2]) # Complex sum-product of 0-D, 1-D, 2-D and 3-D arrays V = GaussianARD(np.random.randn(7,6,5), np.random.rand(7,6,5), plates=(7,6,5), shape=()) v = V.get_moments() X = GaussianARD(np.random.randn(6,1,2), np.random.rand(6,1,2), plates=(6,1), shape=(2,)) x = X.get_moments() Y = GaussianARD(np.random.randn(3,4), np.random.rand(3,4), plates=(5,), shape=(3,4)) y = Y.get_moments() Z = GaussianARD(np.random.randn(4,2,3), np.random.rand(4,2,3), plates=(6,5), shape=(4,2,3)) z = Z.get_moments() mu = np.einsum('...,...i,...kj,...jik->...k', v[0], x[0], y[0], z[0]) cov = np.einsum('...,...ia,...kjcb,...jikbac->...kc', v[1], x[1], y[1], z[1]) compare_moments(mu, cov, ',i,kj,jik->k', V, X, Y, Z) compare_moments(mu, cov, V, [], X, ['i'], Y, ['k','j'], Z, ['j','i','k'], ['k']) # # Gaussian-gamma parents # # Outer product of vectors X1 = GaussianARD(np.random.randn(2), np.random.rand(2), shape=(2,)) x1 = X1.get_moments() X2 = GaussianGammaISO(np.random.randn(6,1,2), random.covariance(2), np.random.rand(6,1), np.random.rand(6,1), plates=(6,1)) x2 = X2.get_moments() Y = SumMultiply('i,j->ij', X1, X2) u = Y._message_to_child() y = np.einsum('...i,...j->...ij', x1[0], x2[0]) yy = np.einsum('...ik,...jl->...ijkl', x1[1], x2[1]) self.assertAllClose(u[0], y) self.assertAllClose(u[1], yy) self.assertAllClose(u[2], x2[2]) self.assertAllClose(u[3], x2[3]) pass def test_message_to_parent(self): """ Test the message from SumMultiply node to its parents. """ data = 2 tau = 3 def check_message(true_m0, true_m1, parent, *args, F=None): if F is None: A = SumMultiply(*args) B = GaussianARD(A, tau) B.observe(data*np.ones(A.plates + A.dims[0])) else: A = F (A_m0, A_m1) = A._message_to_parent(parent) self.assertAllClose(true_m0, A_m0) self.assertAllClose(true_m1, A_m1) pass # Check: different message to each of multiple parents X1 = GaussianARD(np.random.randn(2), np.random.rand(2), ndim=1) x1 = X1.get_moments() X2 = GaussianARD(np.random.randn(2), np.random.rand(2), ndim=1) x2 = X2.get_moments() m0 = tau * data * x2[0] m1 = -0.5 * tau * x2[1] * np.identity(2) check_message(m0, m1, 0, 'i,i->i', X1, X2) check_message(m0, m1, 0, X1, [9], X2, [9], [9]) m0 = tau * data * x1[0] m1 = -0.5 * tau * x1[1] * np.identity(2) check_message(m0, m1, 1, 'i,i->i', X1, X2) check_message(m0, m1, 1, X1, [9], X2, [9], [9]) # Check: key not in output X1 = GaussianARD(np.random.randn(2), np.random.rand(2), ndim=1) x1 = X1.get_moments() m0 = tau * data * np.ones(2) m1 = -0.5 * tau * np.ones((2,2)) check_message(m0, m1, 0, 'i', X1) check_message(m0, m1, 0, 'i->', X1) check_message(m0, m1, 0, X1, [9]) check_message(m0, m1, 0, X1, [9], []) # Check: key not in some input X1 = GaussianARD(np.random.randn(), np.random.rand()) x1 = X1.get_moments() X2 = GaussianARD(np.random.randn(2), np.random.rand(2), ndim=1) x2 = X2.get_moments() m0 = tau * data * np.sum(x2[0], axis=-1) m1 = -0.5 * tau * np.sum(x2[1] * np.identity(2), axis=(-1,-2)) check_message(m0, m1, 0, ',i->i', X1, X2) check_message(m0, m1, 0, X1, [], X2, [9], [9]) m0 = tau * data * x1[0] * np.ones(2) m1 = -0.5 * tau * x1[1] * np.identity(2) check_message(m0, m1, 1, ',i->i', X1, X2) check_message(m0, m1, 1, X1, [], X2, [9], [9]) # Check: keys in different order Y1 = GaussianARD(np.random.randn(3,2), np.random.rand(3,2), ndim=2) y1 = Y1.get_moments() Y2 = GaussianARD(np.random.randn(2,3), np.random.rand(2,3), ndim=2) y2 = Y2.get_moments() m0 = tau * data * y2[0].T m1 = -0.5 * tau * np.einsum('ijlk->jikl', y2[1] * misc.identity(2,3)) check_message(m0, m1, 0, 'ij,ji->ij', Y1, Y2) check_message(m0, m1, 0, Y1, ['i','j'], Y2, ['j','i'], ['i','j']) m0 = tau * data * y1[0].T m1 = -0.5 * tau * np.einsum('ijlk->jikl', y1[1] * misc.identity(3,2)) check_message(m0, m1, 1, 'ij,ji->ij', Y1, Y2) check_message(m0, m1, 1, Y1, ['i','j'], Y2, ['j','i'], ['i','j']) # Check: plates when different dimensionality X1 = GaussianARD(np.random.randn(5), np.random.rand(5), shape=(), plates=(5,)) x1 = X1.get_moments() X2 = GaussianARD(np.random.randn(5,3), np.random.rand(5,3), shape=(3,), plates=(5,)) x2 = X2.get_moments() m0 = tau * data * np.sum(np.ones((5,3)) * x2[0], axis=-1) m1 = -0.5 * tau * np.sum(x2[1] * misc.identity(3), axis=(-1,-2)) check_message(m0, m1, 0, ',i->i', X1, X2) check_message(m0, m1, 0, X1, [], X2, ['i'], ['i']) m0 = tau * data * x1[0][:,np.newaxis] * np.ones((5,3)) m1 = -0.5 * tau * x1[1][:,np.newaxis,np.newaxis] * misc.identity(3) check_message(m0, m1, 1, ',i->i', X1, X2) check_message(m0, m1, 1, X1, [], X2, ['i'], ['i']) # Check: other parent's moments broadcasts over plates when node has the # same plates X1 = GaussianARD(np.random.randn(5,4,3), np.random.rand(5,4,3), shape=(3,), plates=(5,4)) x1 = X1.get_moments() X2 = GaussianARD(np.random.randn(3), np.random.rand(3), shape=(3,), plates=(5,4)) x2 = X2.get_moments() m0 = tau * data * np.ones((5,4,3)) * x2[0] m1 = -0.5 * tau * x2[1] * misc.identity(3) check_message(m0, m1, 0, 'i,i->i', X1, X2) check_message(m0, m1, 0, X1, ['i'], X2, ['i'], ['i']) # Check: other parent's moments broadcasts over plates when node does # not have that plate X1 = GaussianARD(np.random.randn(3), np.random.rand(3), shape=(3,), plates=()) x1 = X1.get_moments() X2 = GaussianARD(np.random.randn(3), np.random.rand(3), shape=(3,), plates=(5,4)) x2 = X2.get_moments() m0 = tau * data * np.sum(np.ones((5,4,3)) * x2[0], axis=(0,1)) m1 = -0.5 * tau * np.sum(np.ones((5,4,1,1)) * misc.identity(3) * x2[1], axis=(0,1)) check_message(m0, m1, 0, 'i,i->i', X1, X2) check_message(m0, m1, 0, X1, ['i'], X2, ['i'], ['i']) # Check: other parent's moments broadcasts over plates when the node # only broadcasts that plate X1 = GaussianARD(np.random.randn(3), np.random.rand(3), shape=(3,), plates=(1,1)) x1 = X1.get_moments() X2 = GaussianARD(np.random.randn(3), np.random.rand(3), shape=(3,), plates=(5,4)) x2 = X2.get_moments() m0 = tau * data * np.sum(np.ones((5,4,3)) * x2[0], axis=(0,1), keepdims=True) m1 = -0.5 * tau * np.sum(np.ones((5,4,1,1)) * misc.identity(3) * x2[1], axis=(0,1), keepdims=True) check_message(m0, m1, 0, 'i,i->i', X1, X2) check_message(m0, m1, 0, X1, ['i'], X2, ['i'], ['i']) # Check: broadcasted dimensions X1 = GaussianARD(np.random.randn(1,1), np.random.rand(1,1), ndim=2) x1 = X1.get_moments() X2 = GaussianARD(np.random.randn(3,2), np.random.rand(3,2), ndim=2) x2 = X2.get_moments() m0 = tau * data * np.sum(np.ones((3,2)) * x2[0], keepdims=True) m1 = -0.5 * tau * np.sum(misc.identity(3,2) * x2[1], keepdims=True) check_message(m0, m1, 0, 'ij,ij->ij', X1, X2) check_message(m0, m1, 0, X1, [0,1], X2, [0,1], [0,1]) m0 = tau * data * np.ones((3,2)) * x1[0] m1 = -0.5 * tau * misc.identity(3,2) * x1[1] check_message(m0, m1, 1, 'ij,ij->ij', X1, X2) check_message(m0, m1, 1, X1, [0,1], X2, [0,1], [0,1]) # Check: non-ARD observations X1 = GaussianARD(np.random.randn(2), np.random.rand(2), ndim=1) x1 = X1.get_moments() Lambda = np.array([[2, 1.5], [1.5, 2]]) F = SumMultiply('i->i', X1) Y = Gaussian(F, Lambda) y = np.random.randn(2) Y.observe(y) m0 = np.dot(Lambda, y) m1 = -0.5 * Lambda check_message(m0, m1, 0, 'i->i', X1, F=F) check_message(m0, m1, 0, X1, ['i'], ['i'], F=F) # Check: mask with same shape X1 = GaussianARD(np.random.randn(3,2), np.random.rand(3,2), shape=(2,), plates=(3,)) x1 = X1.get_moments() mask = np.array([True, False, True]) F = SumMultiply('i->i', X1) Y = GaussianARD(F, tau) Y.observe(data*np.ones((3,2)), mask=mask) m0 = tau * data * mask[:,np.newaxis] * np.ones(2) m1 = -0.5 * tau * mask[:,np.newaxis,np.newaxis] * np.identity(2) check_message(m0, m1, 0, 'i->i', X1, F=F) check_message(m0, m1, 0, X1, ['i'], ['i'], F=F) # Check: mask larger X1 = GaussianARD(np.random.randn(2), np.random.rand(2), shape=(2,), plates=()) x1 = X1.get_moments() X2 = GaussianARD(np.random.randn(3,2), np.random.rand(3,2), shape=(2,), plates=(3,)) x2 = X2.get_moments() mask = np.array([True, False, True]) F = SumMultiply('i,i->i', X1, X2) Y = GaussianARD(F, tau, plates=(3,)) Y.observe(data*np.ones((3,2)), mask=mask) m0 = tau * data * np.sum(mask[:,np.newaxis] * x2[0], axis=0) m1 = -0.5 * tau * np.sum(mask[:,np.newaxis,np.newaxis] * x2[1] * np.identity(2), axis=0) check_message(m0, m1, 0, 'i,i->i', X1, X2, F=F) check_message(m0, m1, 0, X1, ['i'], X2, ['i'], ['i'], F=F) # Check: mask for broadcasted plate X1 = GaussianARD(np.random.randn(2), np.random.rand(2), ndim=1, plates=(1,)) x1 = X1.get_moments() X2 = GaussianARD(np.random.randn(2), np.random.rand(2), ndim=1, plates=(3,)) x2 = X2.get_moments() mask = np.array([True, False, True]) F = SumMultiply('i,i->i', X1, X2) Y = GaussianARD(F, tau, plates=(3,)) Y.observe(data*np.ones((3,2)), mask=mask) m0 = tau * data * np.sum(mask[:,np.newaxis] * x2[0], axis=0, keepdims=True) m1 = -0.5 * tau * np.sum(mask[:,np.newaxis,np.newaxis] * x2[1] * np.identity(2), axis=0, keepdims=True) check_message(m0, m1, 0, 'i->i', X1, F=F) check_message(m0, m1, 0, X1, ['i'], ['i'], F=F) # Check: Gaussian-gamma parents X1 = GaussianGammaISO(np.random.randn(2), random.covariance(2), np.random.rand(), np.random.rand()) x1 = X1.get_moments() X2 = GaussianGammaISO(np.random.randn(2), random.covariance(2), np.random.rand(), np.random.rand()) x2 = X2.get_moments() F = SumMultiply('i,i->i', X1, X2) V = random.covariance(2) y = np.random.randn(2) Y = Gaussian(F, V) Y.observe(y) m0 = np.dot(V, y) * x2[0] m1 = -0.5 * V * x2[1] m2 = -0.5 * np.einsum('i,ij,j', y, V, y) * x2[2]#linalg.inner(V, x2[2], ndim=2) m3 = 0.5 * 2 #linalg.chol_logdet(linalg.chol(V)) + 2*x2[3] m = F._message_to_parent(0) self.assertAllClose(m[0], m0) self.assertAllClose(m[1], m1) self.assertAllClose(m[2], m2) self.assertAllClose(m[3], m3) pass def check_performance(scale=1e2): """ Tests that the implementation of SumMultiply is efficient. This is not a unit test (not run automatically), but rather a performance test, which you may run to test the performance of the node. A naive implementation of SumMultiply will run out of memory in some cases and this method checks that the implementation is not naive but good. """ # Check: Broadcasted plates are computed efficiently # (bad implementation will take a long time to run) s = scale X1 = GaussianARD(np.random.randn(s,s), np.random.rand(s,s), shape=(s,), plates=(s,)) X2 = GaussianARD(np.random.randn(s,1,s), np.random.rand(s,1,s), shape=(s,), plates=(s,1)) F = SumMultiply('i,i', X1, X2) Y = GaussianARD(F, 1) Y.observe(np.ones((s,s))) try: F._message_to_parent(1) except e: print(e) print('SOMETHING BAD HAPPENED') # Check: Broadcasted dimensions are computed efficiently # (bad implementation will run out of memory) pass bayespy-0.4.0/bayespy/inference/vmp/nodes/tests/test_multinomial.py0000644000175000017500000001230712615643655027356 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Unit tests for `multinomial` module. """ import numpy as np import scipy from bayespy.nodes import (Multinomial, Dirichlet, Mixture) from bayespy.utils import random from bayespy.utils.misc import TestCase class TestMultinomial(TestCase): """ Unit tests for Multinomial node """ def test_init(self): """ Test the creation of multinomial nodes. """ # Some simple initializations X = Multinomial(10, [0.1, 0.3, 0.6]) X = Multinomial(10, Dirichlet([5,4,3])) # Check that plates are correct X = Multinomial(10, [0.1, 0.3, 0.6], plates=(3,4)) self.assertEqual(X.plates, (3,4)) X = Multinomial(10, 0.25*np.ones((2,3,4))) self.assertEqual(X.plates, (2,3)) n = 10 * np.ones((3,4), dtype=np.int) X = Multinomial(n, [0.1, 0.3, 0.6]) self.assertEqual(X.plates, (3,4)) X = Multinomial(n, Dirichlet([2,1,9], plates=(3,4))) self.assertEqual(X.plates, (3,4)) # Probabilities not a vector self.assertRaises(ValueError, Multinomial, 10, 0.5) # Invalid probability self.assertRaises(ValueError, Multinomial, 10, [-0.5, 1.5]) self.assertRaises(ValueError, Multinomial, 10, [0.5, 1.5]) # Invalid number of trials self.assertRaises(ValueError, Multinomial, -1, [0.5, 0.5]) self.assertRaises(ValueError, Multinomial, 8.5, [0.5, 0.5]) # Inconsistent plates self.assertRaises(ValueError, Multinomial, 10, 0.25*np.ones((2,4)), plates=(3,)) # Explicit plates too small self.assertRaises(ValueError, Multinomial, 10, 0.25*np.ones((2,4)), plates=(1,)) pass def test_moments(self): """ Test the moments of multinomial nodes. """ # Simple test X = Multinomial(1, [0.7,0.2,0.1]) u = X._message_to_child() self.assertEqual(len(u), 1) self.assertAllClose(u[0], [0.7,0.2,0.1]) # Test n X = Multinomial(10, [0.7,0.2,0.1]) u = X._message_to_child() self.assertAllClose(u[0], [7,2,1]) # Test plates in p n = np.random.randint(1, 10) p = np.random.dirichlet([1,1], size=3) X = Multinomial(n, p) u = X._message_to_child() self.assertAllClose(u[0], p*n) # Test plates in n n = np.random.randint(1, 10, size=(3,)) p = np.random.dirichlet([1,1,1,1]) X = Multinomial(n, p) u = X._message_to_child() self.assertAllClose(u[0], p*n[:,None]) # Test plates in p and n n = np.random.randint(1, 10, size=(4,1)) p = np.random.dirichlet([1,1], size=3) X = Multinomial(n, p) u = X._message_to_child() self.assertAllClose(u[0], p*n[...,None]) # Test with Dirichlet prior P = Dirichlet([7, 3]) logp = P._message_to_child()[0] p0 = np.exp(logp[0]) / (np.exp(logp[0]) + np.exp(logp[1])) p1 = np.exp(logp[1]) / (np.exp(logp[0]) + np.exp(logp[1])) X = Multinomial(1, P) u = X._message_to_child() p = np.array([p0, p1]) self.assertAllClose(u[0], p) # Test with broadcasted plates P = Dirichlet([7, 3], plates=(10,)) X = Multinomial(5, P) u = X._message_to_child() self.assertAllClose(u[0] * np.ones(X.get_shape(0)), 5*p*np.ones((10,1))) pass def test_lower_bound(self): """ Test lower bound for multinomial node. """ # Test for a bug found in multinomial X = Multinomial(10, [0.3, 0.5, 0.2]) l = X.lower_bound_contribution() self.assertAllClose(l, 0.0) pass def test_mixture(self): """ Test multinomial mixture """ p0 = [0.1, 0.5, 0.2, 0.2] p1 = [0.5, 0.1, 0.1, 0.3] p2 = [0.3, 0.2, 0.1, 0.4] X = Mixture(2, Multinomial, 10, [p0, p1, p2]) u = X._message_to_child() self.assertAllClose(u[0], 10*np.array(p2)) pass bayespy-0.4.0/bayespy/inference/vmp/nodes/categorical_markov_chain.py0000644000175000017500000003214412533543154027612 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Module for the categorical Markov chain node. """ import numpy as np from .deterministic import Deterministic from .expfamily import (ExponentialFamily, ExponentialFamilyDistribution, useconstructor) from .node import (Moments, ensureparents) from .categorical import CategoricalMoments from .dirichlet import (Dirichlet, DirichletMoments) from bayespy.utils import misc, random class CategoricalMarkovChainMoments(Moments): """ Class for the moments of categorical Markov chain variables. """ def __init__(self, categories): """ Create moments object for categorical Markov chain variables. """ self.D = categories def compute_fixed_moments(self, x): """ Compute the moments for a fixed value """ raise NotImplementedError("compute_fixed_moments not implemented for " "%s" % (self.__class__.__name__)) def compute_dims_from_values(self, x): """ Return the shape of the moments for a fixed value. """ raise NotImplementedError("compute_dims_from_values not implemented " "for %s" % (self.__class__.__name__)) class CategoricalMarkovChainDistribution(ExponentialFamilyDistribution): """ Class for the VMP formulas of categorical Markov chain variables. """ def __init__(self, categories, states): """ Create VMP formula node for a categorical variable `categories` is the total number of categories. `states` is the length of the chain. """ self.K = categories self.N = states def compute_message_to_parent(self, parent, index, u, u_p0, u_P): """ Compute the message to a parent node. """ if index == 0: return [ u[0] ] elif index == 1: return [ u[1] ] else: raise ValueError("Parent index out of bounds") def compute_mask_to_parent(self, index, mask): """ Maps the mask to the plates of a parent. """ if index == 0: return mask elif index == 1: # Add plate axis for the time axis and row axis of the transition # matrix return np.asanyarray(mask)[...,None,None] else: raise ValueError("Parent index out of bounds") def compute_phi_from_parents(self, u_p0, u_P, mask=True): """ Compute the natural parameter vector given parent moments. """ phi0 = u_p0[0] phi1 = u_P[0] * np.ones((self.N-1,self.K,self.K)) return [phi0, phi1] def compute_moments_and_cgf(self, phi, mask=True): """ Compute the moments and :math:`g(\phi)`. """ logp0 = phi[0] logP = phi[1] (z0, zz, cgf) = random.alpha_beta_recursion(logp0, logP) u = [z0, zz] return (u, cgf) def compute_cgf_from_parents(self, u_p0, u_P): """ Compute :math:`\mathrm{E}_{q(p)}[g(p)]` """ return 0 def compute_fixed_moments_and_f(self, x, mask=True): """ Compute the moments and :math:`f(x)` for a fixed value. """ raise NotImplementedError() def plates_to_parent(self, index, plates): """ Resolves the plate mapping to a parent. Given the plates of the node's moments, this method returns the plates that the message to a parent has for the parent's distribution. """ if index == 0: return plates elif index == 1: return plates + (self.N-1, self.K) else: raise ValueError("Parent index out of bounds") def plates_from_parent(self, index, plates): """ Resolve the plate mapping from a parent. Given the plates of a parent's moments, this method returns the plates that the moments has for this distribution. """ if index == 0: return plates elif index == 1: return plates[:-2] else: raise ValueError("Parent index out of bounds") def random(self, *phi, plates=None): """ Draw a random sample from the distribution. """ # Convert natural parameters to transition probabilities p0 = np.exp(phi[0] - misc.logsumexp(phi[0], axis=-1, keepdims=True)) P = np.exp(phi[1] - misc.logsumexp(phi[1], axis=-1, keepdims=True)) # Explicit broadcasting P = P * np.ones(plates)[...,None,None,None] # Allocate memory Z = np.zeros(plates + (self.N,), dtype=np.int) # Draw initial state Z[...,0] = random.categorical(p0, size=plates) # Create [0,1,2,...,len(plate_axis)] indices for each plate axis and # make them broadcast properly nplates = len(plates) plates_ind = [np.arange(plate)[(Ellipsis,)+(nplates-i-1)*(None,)] for (i, plate) in enumerate(plates)] plates_ind = tuple(plates_ind) # Draw next states iteratively for n in range(self.N-1): # Select the transition probabilities for the current state but take # into account the plates. This leads to complex NumPy # indexing.. :) time_ind = min(n, np.shape(P)[-3]-1) ind = plates_ind + (time_ind, Z[...,n], Ellipsis) p = P[ind] # Draw next state z = random.categorical(P[ind]) Z[...,n+1] = z return Z class CategoricalMarkovChain(ExponentialFamily): r""" Node for categorical Markov chain random variables. The node models a Markov chain which has a discrete set of K possible states and the next state depends only on the previous state and the state transition probabilities. The graphical model is shown below: .. bayesnet:: \tikzstyle{latent} += [minimum size=30pt]; \node[latent] (x0) {$x_0$}; \node[latent, right=of x0] (x1) {$x_1$}; \node[right=of x1] (dots) {$\cdots$}; \node[latent, right=of dots] (xn) {$x_{N-1}$}; \edge {x0}{x1}; \edge {x1}{dots}; \edge {dots}{xn}; \node[latent, above=of x0] (pi) {$\boldsymbol{\pi}$}; \node[latent, above=of dots] (A) {$\mathbf{A}$}; \edge {pi} {x0}; \edge {A} {x1,dots,xn}; where :math:`\boldsymbol{\pi}` contains the probabilities for the initial state and :math:`\mathbf{A}` is the state transition probability matrix. It is possible to have :math:`\mathbf{A}` varying in time. .. math:: p(x_0, \ldots, x_{N-1}) &= p(x_0) \prod^{N-1}_{n=1} p(x_n|x_{n-1}), where .. math:: p(x_0=k) &= \pi_k, \quad \text{for } k \in \{0,\ldots,K-1\}, \\ p(x_n=j|x_{n-1}=i) &= a_{ij}^{(n-1)} \quad \text{for } n=1,\ldots,N-1,\, i\in\{1,\ldots,K-1\},\, j\in\{1,\ldots,K-1\} \\ a_{ij}^{(n)} &= [\mathbf{A}_n]_{ij} This node can be used to construct hidden Markov models by using :class:`Mixture` for the emission distribution. Parameters ---------- pi : Dirichlet-like node or (...,K)-array :math:`\boldsymbol{\pi}`, probabilities for the first state. :math:`K`-dimensional Dirichlet. A : Dirichlet-like node or (K,K)-array or (...,1,K,K)-array or (...,N-1,K,K)-array :math:`\mathbf{A}`, probabilities for state transitions. :math:`K`-dimensional Dirichlet with plates (K,) or (...,1,K) or (...,N-1,K). states : int, optional :math:`N`, the length of the chain. See also -------- Categorical, Dirichlet, GaussianMarkovChain, Mixture, SwitchingGaussianMarkovChain """ _parent_moments = (DirichletMoments(), DirichletMoments()) def __init__(self, pi, A, states=None, **kwargs): """ Create categorical Markov chain """ super().__init__(pi, A, states=states, **kwargs) @classmethod @ensureparents def _constructor(cls, p0, P, states=None, **kwargs): """ Constructs distribution and moments objects. This method is called if useconstructor decorator is used for __init__. Becase the distribution and moments object depend on the number of categories, that is, they depend on the parent node, this method can be used to construct those objects. """ # Number of categories D = p0.dims[0][0] # Number of states if len(P.plates) < 2: if states is None: raise ValueError("Could not infer the length of the Markov " "chain") N = int(states) else: if P.plates[-2] == 1: if states is None: N = 2 else: N = int(states) else: if states is not None and P.plates[-2]+1 != states: raise ValueError("Given length of the Markov chain is " "inconsistent with the transition " "probability matrix") N = P.plates[-2] + 1 if p0.dims != P.dims: raise ValueError("Initial state probability vector and state " "transition probability matrix have different " "size") if len(P.plates) < 1 or P.plates[-1] != D: raise ValueError("Transition probability matrix is not square") dims = ( (D,), (N-1,D,D) ) parents = [p0, P] distribution = CategoricalMarkovChainDistribution(D, N) moments = CategoricalMarkovChainMoments(D) parent_moments = cls._parent_moments return (parents, kwargs, dims, cls._total_plates(kwargs.get('plates'), distribution.plates_from_parent(0, p0.plates), distribution.plates_from_parent(1, P.plates)), distribution, moments, parent_moments) class CategoricalMarkovChainToCategorical(Deterministic): """ A node for converting categorical MC moments to categorical moments. """ def __init__(self, Z, **kwargs): """ Create a categorical MC moments to categorical moments conversion node. """ # Convert parent to proper type. Z must be a node. Z = Z._convert(CategoricalMarkovChainMoments) K = Z.dims[0][-1] dims = ( (K,), ) self._moments = CategoricalMoments(K) self._parent_moments = (CategoricalMarkovChainMoments(K),) super().__init__(Z, dims=dims, **kwargs) def _compute_moments(self, u_Z): """ Compute the moments given the moments of the parents. """ # Add time axis to p0 p0 = u_Z[0][...,None,:] # Sum joint probability arrays to marginal probability vectors zz = u_Z[1] p = np.sum(zz, axis=-2) # Broadcast p0 and p to same shape, except the time axis plates_p0 = np.shape(p0)[:-2] plates_p = np.shape(p)[:-2] shape = misc.broadcasted_shape(plates_p0, plates_p) + (1,1) p0 = p0 * np.ones(shape) p = p * np.ones(shape) # Concatenate P = np.concatenate((p0,p), axis=-2) return [P] def _compute_message_to_parent(self, index, m, u_Z): """ Compute the message to a parent. """ m0 = m[0][...,0,:] m1 = m[0][...,1:,None,:] return [m0, m1] def _compute_mask_to_parent(self, index, mask): """ Compute the mask used for messages sent to a parent. """ if index == 0: # "Sum" over the last axis # TODO/FIXME: Check this. BUG I THINK. return np.any(mask, axis=-1) else: raise ValueError("Parent index out of bounds") def _plates_to_parent(self, index): if index == 0: return self.plates[:-1] else: raise ValueError("Parent index out of bounds") def _plates_from_parent(self, index): if index == 0: N = self.parents[0].dims[1][0] return self.parents[0].plates + (N+1,) else: raise ValueError("Parent index out of bounds") # Make use of the conversion node CategoricalMarkovChainMoments.add_converter(CategoricalMoments, CategoricalMarkovChainToCategorical) bayespy-0.4.0/bayespy/inference/vmp/nodes/CovarianceFunctions.py0000644000175000017500000006774512533543154026576 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2012 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import itertools import numpy as np #import scipy as sp import scipy.sparse as sp # prefer CSC format #import scipy.linalg.decomp_cholesky as decomp #import scipy.linalg as linalg #import scipy.special as special #import matplotlib.pyplot as plt #import time #import profile import scipy.spatial.distance as dist #import scikits.sparse.distance as spdist from . import node as ef from bayespy.utils import misc as utils # Covariance matrices can be either arrays or matrices so be careful # with products and powers! Use explicit multiply or dot instead of # *-operator. def gp_cov_se(D2, overwrite=False): if overwrite: K = D2 K *= -0.5 np.exp(K, out=K) else: K = np.exp(-0.5*D2) return K def gp_cov_pp2_new(r, d, derivative=False): # Dimension dependent parameter q = 2 j = np.floor(d/2) + q + 1 # Polynomial coefficients a2 = j**2 + 4*j + 3 a1 = 3*j + 6 a0 = 3 # Two parts of the covariance function k1 = (1-r) ** (j+2) k2 = (a2*r**2 + a1*r + 3) # TODO: Check that derivative is 0, 1 or 2! if derivative == 0: # Return covariance return k1 * k2 / 3 dk1 = - (j+2) * (1-r)**(j+1) dk2 = 2*a2*r + a1 if derivative == 1: # Return first derivative of the covariance return (k1 * dk2 + dk1 * k2) / 3 ddk1 = (j+2) * (j+1) * (1-r)**j ddk2 = 2*a2 if derivative == 2: # Return second derivative of the covariance return (ddk1*k2 + 2*dk1*dk2 + k1*ddk2) / 3 def gp_cov_pp2(r, d, gradient=False): # Dimension dependent parameter j = np.floor(d/2) + 2 + 1 # Polynomial coefficients a2 = j**2 + 4*j + 3 a1 = 3*j + 6 a0 = 3 # Two parts of the covariance function k1 = (1-r) ** (j+2) k2 = (a2*r**2 + a1*r + 3) # The covariance function k = k1 * k2 / 3 if gradient: # The gradient w.r.t. r dk = k * (j+2) / (r-1) + k1 * (2*a2*r + a1) / 3 return (k, dk) else: return k def gp_cov_delta(N): # TODO: Use sparse matrices here! if N > 0: #print('in gpcovdelta', N, sp.identity(N).shape) return sp.identity(N) else: # Sparse matrices do not allow zero-length dimensions return np.identity(N) #return np.identity(N) #return np.asmatrix(np.identity(N)) def squared_distance(x1, x2): ## # Reshape arrays to 2-D arrays ## sh1 = np.shape(x1)[:-1] ## sh2 = np.shape(x2)[:-1] ## d = np.shape(x1)[-1] ## x1 = np.reshape(x1, (-1,d)) ## x2 = np.reshape(x2, (-1,d)) (m1,n1) = x1.shape (m2,n2) = x2.shape if m1 == 0 or m2 == 0: D2 = np.empty((m1,m2)) else: # Compute squared Euclidean distance D2 = dist.cdist(x1, x2, metric='sqeuclidean') #D2 = np.asmatrix(D2) # Reshape the result #D2 = np.reshape(D2, sh1 + sh2) return D2 # General rule for the parameters for covariance functions: # # (value, [ [dvalue1, ...], [dvalue2, ...], [dvalue3, ...], ...]) # # For instance, # # k = covfunc_se((1.0, []), (15, [ [1,update_grad] ])) # K = k((x1, [ [dx1,update_grad] ]), (x2, [])) # # Plain values are converted as: # value -> (value, []) def gp_standardize_input(x): if np.size(x) == 0: x = np.reshape(x, (0,0)) elif np.ndim(x) == 0: x = np.reshape(x, (1,1)) elif np.ndim(x) == 1: x = np.reshape(x, (-1,1)) elif np.ndim(x) == 2: x = np.atleast_2d(x) else: raise Exception("Standard GP inputs must be 2-dimensional") return x def gp_preprocess_inputs(x1,x2=None): #args = list(args) #if len(args) < 1 or len(args) > 2: #raise Exception("Number of inputs must be one or two") if x2 is None: x1 = gp_standardize_input(x1) return x1 else: if x1 is x2: x1 = gp_standardize_input(x1) x2 = x1 else: x1 = gp_standardize_input(x1) x2 = gp_standardize_input(x2) return (x1, x2) #return args ## def gp_preprocess_inputs(x1,x2=None): ## #args = list(args) ## #if len(args) < 1 or len(args) > 2: ## #raise Exception("Number of inputs must be one or two") ## if x2 is not None: len(args) == 2: ## if args[0] is args[1]: ## args[0] = gp_standardize_input(args[0]) ## args[1] = args[0] ## else: ## args[1] = gp_standardize_input(args[1]) ## args[0] = gp_standardize_input(args[0]) ## else: ## args[0] = gp_standardize_input(args[0]) ## return args # TODO: # General syntax for these covariance functions: # covfunc(hyper1, # hyper2, # ... # hyperN, # x1, # x2=None, # gradient=list_of_booleans_for_each_hyperparameter) def covfunc_zeros(x1, x2=None, gradient=False): inputs = gp_preprocess_inputs(*inputs) # Compute distance and covariance matrix if x2 is None: x1 = gp_preprocess_inputs(x1) # Only variance vector asked N = np.shape(x1)[0] # TODO: Use sparse matrices! K = np.zeros(N) #K = np.asmatrix(np.zeros((N,1))) else: (x1,x2) = gp_preprocess_inputs(x1,x2) # Full covariance matrix asked #x1 = inputs[0] #x2 = inputs[1] # Number of inputs x1 N1 = np.shape(x1)[0] N2 = np.shape(x2)[0] # TODO: Use sparse matrices! K = np.zeros((N1,N2)) #K = np.asmatrix(np.zeros((N1,N2))) if gradient is not False: return (K, []) else: return K def covfunc_delta(amplitude, x1, x2=None, gradient=False): # Make sure that amplitude is a scalar, not an array object amplitude = utils.array_to_scalar(amplitude) ## if gradient: ## gradient_amplitude = gradient[0] ## else: ## gradient_amplitude = [] ## inputs = gp_preprocess_inputs(*inputs) # Compute distance and covariance matrix if x2 is None: x1 = gp_preprocess_inputs(x1) # Only variance vector asked #x = inputs[0] N = np.shape(x1)[0] K = np.ones(N) * amplitude**2 else: (x1,x2) = gp_preprocess_inputs(x1,x2) # Full covariance matrix asked #x1 = inputs[0] #x2 = inputs[1] # Number of inputs x1 N1 = np.shape(x1)[0] # x1 == x2? if x1 is x2: delta = True # Delta covariance # # FIXME: Broadcasting doesn't work with sparse matrices, # so must use scalar multiplication K = gp_cov_delta(N1) * amplitude**2 #K = gp_cov_delta(N1).multiply(amplitude**2) else: delta = False # Number of inputs x2 N2 = np.shape(x2)[0] # Zero covariance if N1 > 0 and N2 > 0: K = sp.csc_matrix((N1,N2)) else: K = np.zeros((N1,N2)) # Gradient w.r.t. amplitude if gradient: # FIXME: Broadcasting doesn't work with sparse matrices, # so must use scalar multiplication gradient_amplitude = K*(2/amplitude) print("noise grad", gradient_amplitude) return (K, (gradient_amplitude,)) else: return K def covfunc_pp2(amplitude, lengthscale, x1, x2, gradient=False): # Make sure that hyperparameters are scalars, not an array objects amplitude = utils.array_to_scalar(amplitude) lengthscale = utils.array_to_scalar(lengthscale) #amplitude = theta[0] #lengthscale = theta[1] ## if gradient: ## gradient_amplitude = gradient[0] ## gradient_lengthscale = gradient[1] ## else: ## gradient_amplitude = [] ## gradient_lengthscale = [] ## inputs = gp_preprocess_inputs(*inputs) # Compute covariance matrix if x2 is None: x1 = gp_preprocess_inputs(x1) # Compute variance vector K = np.ones(np.shape(x)[:-1]) K *= amplitude**2 # Compute gradient w.r.t. lengthscale if gradient: gradient_lengthscale = np.zeros(np.shape(x1)[:-1]) else: (x1,x2) = gp_preprocess_inputs(x1,x2) # Compute (sparse) distance matrix if x1 is x2: x1 = inputs[0] / (lengthscale) x2 = x1 D2 = spdist.pdist(x1, 1.0, form="full", format="csc") else: x1 = inputs[0] / (lengthscale) x2 = inputs[1] / (lengthscale) D2 = spdist.cdist(x1, x2, 1.0, format="csc") r = np.sqrt(D2.data) N1 = np.shape(x1)[0] N2 = np.shape(x2)[0] # Compute the covariances if gradient: (k, dk) = gp_cov_pp2(r, np.shape(x1)[-1], gradient=True) else: k = gp_cov_pp2(r, np.shape(x1)[-1]) k *= amplitude**2 # Compute gradient w.r.t. lengthscale if gradient: if N1 >= 1 and N2 >= 1: dk *= r * (-amplitude**2 / lengthscale) gradient_lengthscale = sp.csc_matrix((dk, D2.indices, D2.indptr), shape=(N1,N2)) else: gradient_lengthscale = np.empty((N1,N2)) # Form sparse covariance matrix if N1 >= 1 and N2 >= 1: ## K = sp.csc_matrix((k, ij), shape=(N1,N2)) K = sp.csc_matrix((k, D2.indices, D2.indptr), shape=(N1,N2)) else: K = np.empty((N1, N2)) #print(K.__class__) # Gradient w.r.t. amplitude if gradient: gradient_amplitude = K * (2 / amplitude) # Return values if gradient: print("pp2 grad", gradient_lengthscale) return (K, (gradient_amplitude, gradient_lengthscale)) else: return K def covfunc_se(amplitude, lengthscale, x1, x2=None, gradient=False): # Make sure that hyperparameters are scalars, not an array objects amplitude = utils.array_to_scalar(amplitude) lengthscale = utils.array_to_scalar(lengthscale) # Compute covariance matrix if x2 is None: x1 = gp_preprocess_inputs(x1) #x = inputs[0] # Compute variance vector N = np.shape(x1)[0] K = np.ones(N) np.multiply(K, amplitude**2, out=K) # Compute gradient w.r.t. lengthscale if gradient: # TODO: Use sparse matrices? gradient_lengthscale = np.zeros(N) else: (x1,x2) = gp_preprocess_inputs(x1,x2) x1 = x1 / (lengthscale) x2 = x2 / (lengthscale) # Compute distance matrix K = squared_distance(x1, x2) # Compute gradient partly if gradient: gradient_lengthscale = np.divide(K, lengthscale) # Compute covariance matrix gp_cov_se(K, overwrite=True) np.multiply(K, amplitude**2, out=K) # Compute gradient w.r.t. lengthscale if gradient: gradient_lengthscale *= K # Gradient w.r.t. amplitude if gradient: gradient_amplitude = K * (2 / amplitude) # Return values if gradient: print("se grad", gradient_amplitude, gradient_lengthscale) return (K, (gradient_amplitude, gradient_lengthscale)) else: return K class CovarianceFunctionWrapper(): def __init__(self, covfunc, *params): # Parse parameter values and their gradients to separate lists self.covfunc = covfunc self.params = list(params) self.gradient_params = list() ## print(params) for ind in range(len(params)): if isinstance(params[ind], tuple): # Parse the value and the list of gradients from the # form: # ([value, ...], [ [grad1, ...], [grad2, ...], ... ]) self.gradient_params.append(params[ind][1]) self.params[ind] = params[ind][0][0] else: # No gradients, parse from the form: # [value, ...] self.gradient_params.append([]) self.params[ind] = params[ind][0] def fixed_covariance_function(self, *inputs, gradient=False): # What if this is called several times?? if gradient: ## grads = [[grad[0] for grad in self.gradient_params[ind]] ## for ind in range(len(self.gradient_params))] ## (K, dK) = self.covfunc(self.params, ## *inputs, ## gradient=self.gradient_params) arguments = tuple(self.params) + tuple(inputs) (K, dK) = self.covfunc(*arguments, gradient=True) ## (K, dK) = self.covfunc(self.params, ## *inputs, ## gradient=grads) DK = [] for ind in range(len(dK)): # Gradient w.r.t. covariance function's ind-th # hyperparameter dk = dK[ind] # Chain rule: Multiply by the gradient of the # hyperparameter w.r.t. parent node and append the # list DK: # DK = [ (dx1_1, callback), ..., (dx1_n, callback) ] for grad in self.gradient_params[ind]: #print(grad[0]) #print(grad[1:]) #print(dk) if sp.issparse(dk): print(dk.shape) print(grad[0].shape) DK += [ [dk.multiply(grad[0])] + grad[1:] ] else: DK += [ [np.multiply(dk,grad[0])] + grad[1:] ] #DK += [ [np.multiply(grad[0], dk)] + grad[1:] ] ## DK += [ (np.multiply(grad, dk),) + grad[1:] ## for grad in self.gradient_params[ind] ] ## for grad in self.gradient_params[ind]: ## DK += ( (np.multiply(grad, dk),) + grad[1:] ) ## DK = [] ## for ind in range(len(dK)): ## for (grad, dk) in zip(self.gradient_params[ind], dK[ind]): ## DK += [ [dk] + grad[1:] ] K = [K] return (K, DK) else: arguments = tuple(self.params) + tuple(inputs) #print(arguments) K = self.covfunc(*arguments, gradient=False) return [K] class CovarianceFunction(ef.Node): def __init__(self, covfunc, *args, **kwargs): self.covfunc = covfunc params = list(args) for i in range(len(args)): # Check constant parameters if utils.is_numeric(args[i]): params[i] = ef.NodeConstant([np.asanyarray(args[i])], dims=[np.shape(args[i])]) # TODO: Parameters could be constant functions? :) ef.Node.__init__(self, *params, dims=[(np.inf, np.inf)], **kwargs) def __call__(self, x1, x2): """ Compute covariance matrix for inputs x1 and x2. """ covfunc = self.message_to_child() return covfunc(x1, x2)[0] def message_to_child(self, gradient=False): params = [parent.message_to_child(gradient=gradient) for parent in self.parents] covfunc = self.get_fixed_covariance_function(*params) return covfunc def get_fixed_covariance_function(self, *params): get_cov_func = CovarianceFunctionWrapper(self.covfunc, *params) return get_cov_func.fixed_covariance_function ## def covariance_function(self, *params): ## # Parse parameter values and their gradients to separate lists ## params = list(params) ## gradient_params = list() ## print(params) ## for ind in range(len(params)): ## if isinstance(params[ind], tuple): ## # Parse the value and the list of gradients from the ## # form: ## # ([value, ...], [ [grad1, ...], [grad2, ...], ... ]) ## gradient_params.append(params[ind][1]) ## params[ind] = params[ind][0][0] ## else: ## # No gradients, parse from the form: ## # [value, ...] ## gradient_params.append([]) ## params[ind] = params[ind][0] ## # This gradient_params changes mysteriously.. ## print('grad_params before') ## if isinstance(self, SquaredExponential): ## print(gradient_params) ## def cov(*inputs, gradient=False): ## if gradient: ## print('grad_params after') ## print(gradient_params) ## grads = [[grad[0] for grad in gradient_params[ind]] ## for ind in range(len(gradient_params))] ## print('CovarianceFunction.cov') ## #if isinstance(self, SquaredExponential): ## #print(self.__class__) ## #print(grads) ## (K, dK) = self.covfunc(params, ## *inputs, ## gradient=grads) ## for ind in range(len(dK)): ## for (grad, dk) in zip(gradient_params[ind], dK[ind]): ## grad[0] = dk ## K = [K] ## dK = [] ## for grad in gradient_params: ## dK += grad ## return (K, dK) ## else: ## K = self.covfunc(params, ## *inputs, ## gradient=False) ## return [K] ## return cov class Sum(CovarianceFunction): def __init__(self, *args, **kwargs): CovarianceFunction.__init__(self, None, *args, **kwargs) def get_fixed_covariance_function(self, *covfunc_parents): def covfunc(*inputs, gradient=False): K_sum = None if gradient: dK_sum = list() for k in covfunc_parents: if gradient: (K, dK) = k(*inputs, gradient=gradient) print("dK in sum", dK) dK_sum += dK #print("dK_sum in sum", dK_sum) else: K = k(*inputs, gradient=gradient) if K_sum is None: K_sum = K[0] else: try: K_sum += K[0] except: # You have to do this way, for instance, if # K_sum is sparse and K[0] is dense. K_sum = K_sum + K[0] if gradient: #print("dK_sum on: ", dK_sum) #print('covsum', dK_sum) return ([K_sum], dK_sum) else: return [K_sum] return covfunc class Delta(CovarianceFunction): def __init__(self, amplitude, **kwargs): CovarianceFunction.__init__(self, covfunc_delta, amplitude, **kwargs) class Zeros(CovarianceFunction): def __init__(self, **kwargs): CovarianceFunction.__init__(self, covfunc_zeros, **kwargs) class SquaredExponential(CovarianceFunction): def __init__(self, amplitude, lengthscale, **kwargs): CovarianceFunction.__init__(self, covfunc_se, amplitude, lengthscale, **kwargs) class PiecewisePolynomial2(CovarianceFunction): def __init__(self, amplitude, lengthscale, **kwargs): CovarianceFunction.__init__(self, covfunc_pp2, amplitude, lengthscale, **kwargs) # TODO: Rename to Blocks or Joint ? class Multiple(CovarianceFunction): def __init__(self, covfuncs, **kwargs): self.d = len(covfuncs) #self.sparse = sparse parents = [covfunc for row in covfuncs for covfunc in row] CovarianceFunction.__init__(self, None, *parents, **kwargs) def get_fixed_covariance_function(self, *covfuncs): def cov(*inputs, gradient=False): # Computes the covariance matrix from blocks which all # have their corresponding covariance functions if len(inputs) < 2: # For one input, return the variance vector instead of # the covariance matrix x1 = inputs[0] # Collect variance vectors from the covariance # functions corresponding to the diagonal blocks K = [covfuncs[i*self.d+i](x1[i], gradient=gradient)[0] for i in range(self.d)] # Form the variance vector from the collected vectors if gradient: raise Exception('Gradient not yet implemented.') else: ## print("in cov multiple") ## for (k,kf) in zip(K,covfuncs): ## print(np.shape(k), k.__class__, kf) #K = np.vstack(K) K = np.concatenate(K) else: x1 = inputs[0] x2 = inputs[1] # Collect the covariance matrix (and possibly # gradients) from each block. #print('cov mat collection begins') K = [[covfuncs[i*self.d+j](x1[i], x2[j], gradient=gradient) for j in range(self.d)] for i in range(self.d)] #print('cov mat collection ends') # Remove matrices that have zero length dimensions? if gradient: K = [[K[i][j] for j in range(self.d) if np.shape(K[i][j][0][0])[1] != 0] for i in range(self.d) if np.shape(K[i][0][0][0])[0] != 0] else: K = [[K[i][j] for j in range(self.d) if np.shape(K[i][j][0])[1] != 0] for i in range(self.d) if np.shape(K[i][0][0])[0] != 0] n_blocks = len(K) #print("nblocks", n_blocks) #print("K", K) # Check whether all blocks are sparse is_sparse = True for i in range(n_blocks): for j in range(n_blocks): if gradient: A = K[i][j][0][0] else: A = K[i][j][0] if not sp.issparse(A): is_sparse = False if gradient: ## Compute the covariance matrix and the gradients # Create block matrices of zeros. This helps in # computing the gradient. if is_sparse: # Empty sparse matrices. Some weird stuff here # because sparse matrices can't have zero # length dimensions. Z = [[sp.csc_matrix(np.shape(K[i][j][0][0])) for j in range(n_blocks)] for i in range(n_blocks)] else: # Empty dense matrices Z = [[np.zeros(np.shape(K[i][j][0][0])) for j in range(n_blocks)] for i in range(n_blocks)] ## for j in range(self.d)] ## for i in range(self.d)] # Compute gradients block by block dK = list() for i in range(n_blocks): for j in range(n_blocks): # Store the zero block z_old = Z[i][j] # Go through the gradients for the (i,j) # block for dk in K[i][j][1]: # Keep other blocks at zero and set # the gradient to (i,j) block. Form # the matrix from blocks if is_sparse: Z[i][j] = dk[0] dk[0] = sp.bmat(Z).tocsc() else: if sp.issparse(dk[0]): Z[i][j] = dk[0].toarray() else: Z[i][j] = dk[0] #print("Z on:", Z) dk[0] = np.asarray(np.bmat(Z)) # Append the computed gradient matrix # to the list of gradients dK.append(dk) # Restore the zero block Z[i][j] = z_old ## Compute the covariance matrix but not the ## gradients if is_sparse: # Form the full sparse covariance matrix from # blocks. Ignore blocks having a zero-length # axis because sparse matrices consider zero # length as an invalid shape (BUG IN SCIPY?). K = [[K[i][j][0][0] for j in range(n_blocks)] for i in range(n_blocks)] K = sp.bmat(K).tocsc() else: # Form the full dense covariance matrix from # blocks. Transform sparse blocks to dense # blocks. K = [[K[i][j][0][0] if not sp.issparse(K[i][j][0][0]) else K[i][j][0][0].toarray() for j in range(n_blocks)] for i in range(n_blocks)] K = np.asarray(np.bmat(K)) else: ## Compute the covariance matrix but not the ## gradients if is_sparse: # Form the full sparse covariance matrix from # blocks. Ignore blocks having a zero-length # axis because sparse matrices consider zero # length as an invalid shape (BUG IN SCIPY?). K = [[K[i][j][0] for j in range(n_blocks)] for i in range(n_blocks)] K = sp.bmat(K).tocsc() else: # Form the full dense covariance matrix from # blocks. Transform sparse blocks to dense # blocks. K = [[K[i][j][0] if not sp.issparse(K[i][j][0]) else K[i][j][0].toarray() for j in range(n_blocks)] for i in range(n_blocks)] K = np.asarray(np.bmat(K)) if gradient: return ([K], dK) else: return [K] return cov bayespy-0.4.0/bayespy/inference/vmp/nodes/logistic.py0000644000175000017500000001536512615643655024447 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2015 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### """ Module for Bernoulli using the logistic function for Gaussian """ import numpy as np from .node import ensureparents from .expfamily import (ExponentialFamily, useconstructor) from .multinomial import (MultinomialMoments, MultinomialDistribution, Multinomial) from .dirichlet import DirichletMoments from .gaussian import GaussianMoments from bayespy.utils import random from bayespy.utils import misc class CategoricalDistribution(MultinomialDistribution): """ Class for the VMP formulas of categorical variables. """ def __init__(self, categories): """ Create VMP formula node for a categorical variable `categories` is the total number of categories. """ if not isinstance(categories, int): raise ValueError("Number of categories must be integer") if categories < 0: raise ValueError("Number of categoriess must be non-negative") self.D = categories super().__init__(1) def compute_message_to_parent(self, parent, index, u, u_p): """ Compute the message to a parent node. """ return super().compute_message_to_parent(parent, index, u, u_p) def compute_phi_from_parents(self, u_p, mask=True): """ Compute the natural parameter vector given parent moments. """ return super().compute_phi_from_parents(u_p, mask=mask) def compute_moments_and_cgf(self, phi, mask=True): """ Compute the moments and :math:`g(\phi)`. """ return super().compute_moments_and_cgf(phi, mask=mask) def compute_cgf_from_parents(self, u_p): """ Compute :math:`\mathrm{E}_{q(p)}[g(p)]` """ return super().compute_cgf_from_parents(u_p) def compute_fixed_moments_and_f(self, x, mask=True): """ Compute the moments and :math:`f(x)` for a fixed value. """ # Check the validity of x x = np.asanyarray(x) if not misc.isinteger(x): raise ValueError("Values must be integers") if np.any(x < 0) or np.any(x >= self.D): raise ValueError("Invalid category index") # Form a binary matrix with only one non-zero (1) in the last axis u0 = np.zeros((np.size(x), self.D)) u0[[np.arange(np.size(x)), np.ravel(x)]] = 1 u0 = np.reshape(u0, np.shape(x) + (self.D,)) u = [u0] # f(x) is zero f = 0 return (u, f) def random(self, *phi, plates=None): """ Draw a random sample from the distribution. """ logp = phi[0] logp -= np.amax(logp, axis=-1, keepdims=True) p = np.exp(logp) return random.categorical(p, size=plates) class Logistic(ExponentialFamily): r""" :cite:`Jaakkola:2000` The true probability density function: .. math:: p(z=1|x) = g(x) \\ p(z=0|x) = g(-x) which can be written as: .. math:: p(z|x) = g(H_z) where :math:`H_z=(2z-1)x` and :math:`g(x)` is the logistic function: .. math:: g(x) = \frac{1} {1 + e^{-x}} The log of the logistic function: .. math:: \log g(x) = -\log(1 + e^{-x}) = \frac{x}{2} - \log(e^{x/2} + e^{-x/2}) The latter term: .. math:: f(x) = -\log(e^{x/2} + e^{-x/2}) This is a convex function in the variable :math:`x^2`, thus it can be bounded globally with a first order Taylor expansion in the variable x^2: .. math:: f(x) &\geq f(\xi) + \frac {\partial f(\xi)}{\partial(\xi^2)} (x^2 - \xi^2) \\ &= -\frac{\xi}{2} + \log g(\xi) + \frac{1}{4\xi}\tanh(\frac{\xi}{2}) (x^2 - \xi^2) Thus, the variational lower bound for the probability density function is: .. math:: p(z|x) \geq g(xi) \exp( \frac{H_z-\xi}{2} - \lambda(\xi) (H_z^2 - \xi^2)) and in log form: .. math:: \log p(z|x) \geq \log g(xi) + ( \frac{H_z-\xi}{2} - \lambda(\xi) (H_z^2 - \xi^2) ) where .. math:: \lambda(\xi) = \frac {\tanh(\xi/2)} {4\xi} Now, this log lower bound is quadratic with respect to :math:`H_z`, thus it is quadratic with respect to :math:`x` and it is conjugate with the Gaussian distribution. Re-organize terms: .. math:: \log p(z|x) &\geq -\lambda(\xi)(2z-1)^2 x^2 + zx - \frac{1}{2}x - \frac{1}{2}\xi + \lambda(\xi) \xi^2 + \log g(\xi) \\ &= -\lambda(\xi)(2z+1) x^2 + zx - \frac{1}{2}x - \frac{1}{2}\xi + \lambda(\xi) \xi^2 + \log g(\xi) \\ &= z (-2\lambda(\xi) x^2 + x) - \lambda(\xi) x^2 - \frac{1}{2}x - \frac{1}{2}\xi + \lambda(\xi) \xi^2 + \log g(\xi) where we have used :math:`z^2=z`. See also -------- Bernoulli, GaussianARD """ _parent_moments = [GaussianMoments(0)] def __init__(self, x, **kwargs): """ """ super().__init__(x, **kwargs) @classmethod @ensureparents def _constructor(cls, x, **kwargs): """ Constructs distribution and moments objects. """ # Get the number of categories D = p.dims[0][0] parents = [p] moments = CategoricalMoments(D) distribution = CategoricalDistribution(D) return (parents, kwargs, ( (D,), ), cls._total_plates(kwargs.get('plates'), distribution.plates_from_parent(0, p.plates)), distribution, moments, cls._parent_moments) def __str__(self): """ Print the distribution using standard parameterization. """ raise NotImplementedError() bayespy-0.4.0/bayespy/inference/vmp/nodes/gaussian_markov_chain.py0000644000175000017500000020755212615643655027166 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2012-2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ This module contains VMP nodes for Gaussian Markov chains. """ import numpy as np import scipy from bayespy.utils import misc from bayespy.utils import linalg from .node import Node, message_sum_multiply from .deterministic import Deterministic from .expfamily import ExponentialFamily from .expfamily import ExponentialFamilyDistribution from .expfamily import useconstructor from .gaussian import Gaussian, GaussianMoments from .wishart import Wishart, WishartMoments from .gamma import Gamma, GammaMoments from .categorical import CategoricalMoments from .node import Moments, ensureparents class GaussianMarkovChainMoments(Moments): def compute_fixed_moments(self, x): u0 = x u1 = x[...,:,np.newaxis] * x[...,np.newaxis,:] u2 = x[...,:-1,:,np.newaxis] * x[...,1:,np.newaxis,:] return [u0, u1, u2] class TemplateGaussianMarkovChainDistribution(ExponentialFamilyDistribution): """ Sub-classes implement distribution specific computations. """ def __init__(self, N, D): self.N = N self.D = D super().__init__() def compute_message_to_parent(self, parent, index, u_self, *u_parents): raise NotImplementedError() def compute_mask_to_parent(self, index, mask): raise NotImplementedError() def compute_phi_from_parents(self, *u_parents, mask=True): raise NotImplementedError() def compute_moments_and_cgf(self, phi, mask=True): """ Compute the moments and the cumulant-generating function. This basically performs the filtering and smoothing for the variable. Parameters ---------- phi Returns ------- u g """ # Solve the Kalman filtering and smoothing problem y = phi[0] A = -2*phi[1] # Don't multiply phi[2] by two because it is a sum of the super- and # sub-diagonal blocks so we would need to divide by two anyway. B = -phi[2] (CovXnXn, CovXpXn, Xn, ldet) = linalg.block_banded_solve(A, B, y) # Compute moments u0 = Xn u1 = CovXnXn + Xn[...,:,np.newaxis] * Xn[...,np.newaxis,:] u2 = CovXpXn + Xn[...,:-1,:,np.newaxis] * Xn[...,1:,np.newaxis,:] u = [u0, u1, u2] # Compute cumulant-generating function g = -0.5 * np.einsum('...ij,...ij', u[0], phi[0]) + 0.5*ldet return (u, g) def compute_cgf_from_parents(self, *u_parents): raise NotImplementedError() def compute_fixed_moments_and_f(self, x, mask=True): """ Compute u(x) and f(x) for given x. """ u0 = x u1 = x[...,:,np.newaxis] * x[...,np.newaxis,:] u2 = x[...,:-1,:,np.newaxis] * x[...,1:,np.newaxis,:] u = [u0, u1, u2] f = -0.5 * np.shape(x)[-2] * np.shape(x)[-1] * np.log(2*np.pi) return (u, f) def plates_to_parent(self, index, plates): """ Computes the plates of this node with respect to a parent. Child classes must implement this. Parameters ----------- index : int The index of the parent node to use. """ raise NotImplementedError() def plates_from_parent(self, index, plates): """ Compute the plates using information of a parent node. Child classes must implement this. Parameters ---------- index : int Index of the parent to use. """ raise NotImplementedError() class _TemplateGaussianMarkovChain(ExponentialFamily): r""" VMP abstract node for Gaussian Markov chain. This is a general base class for different Gaussian Markov chain nodes. Output is Gaussian variables with mean, covariance and one-step cross-covariance. self.phi and self.u are defined in a particular way but otherwise the parent nodes may vary. Child classes must implement the following methods: _plates_to_parent _plates_from_parent See also -------- bayespy.inference.vmp.nodes.gaussian.Gaussian bayespy.inference.vmp.nodes.wishart.Wishart """ _moments = GaussianMarkovChainMoments() def random(self, *phi, plates=None): raise NotImplementedError() def rotate(self, R, inv=None, logdet=None): if inv is not None: invR = inv else: invR = np.linalg.inv(R) if logdet is not None: logdetR = logdet else: logdetR = np.linalg.slogdet(R)[1] # It would be more efficient and simpler, if you just rotated the # moments and didn't touch phi. However, then you would need to call # update() before lower_bound_contribution. This is more error-safe. # Transform parameters self.phi[0] = linalg.mvdot(invR.T, self.phi[0]) self.phi[1] = linalg.dot(invR.T, self.phi[1], invR) self.phi[2] = linalg.dot(invR.T, self.phi[2], invR) N = self.dims[0][0] if False: self._update_moments_and_cgf() else: # Transform moments and g u0 = linalg.mvdot(R, self.u[0]) u1 = linalg.dot(R, self.u[1], R.T) u2 = linalg.dot(R, self.u[2], R.T) self.u = [u0, u1, u2] self.g -= N*logdetR def _compute_cgf_for_gaussian_markov_chain(mumu, Lambda, logdet_Lambda, logdet_v, N): """ Compute CGF using the moments of the parents. """ g0 = -0.5 * np.einsum('...ij,...ij->...', mumu, Lambda) g1 = 0.5 * logdet_Lambda if np.ndim(logdet_v) == 1: g1 = g1 + 0.5 * (N-1) * np.sum(logdet_v, axis=-1) elif np.shape(logdet_v)[-2] == 1: g1 = g1 + 0.5 * (N-1) * np.sum(logdet_v, axis=(-1,-2)) else: g1 = g1 + 0.5 * np.sum(logdet_v, axis=(-1,-2)) return g0 + g1 class GaussianMarkovChainDistribution(TemplateGaussianMarkovChainDistribution): """ Sub-classes implement distribution specific computations. """ def compute_message_to_parent(self, parent, index, u, u_mu, u_Lambda, u_A, u_v, *u_inputs): """ Compute a message to a parent. Parameters ---------- index : int Index of the parent requesting the message. u : list of ndarrays Moments of this node. u_mu : list of ndarrays Moments of parent `mu`. u_Lambda : list of ndarrays Moments of parent `Lambda`. u_A : list of ndarrays Moments of parent `A`. u_v : list of ndarrays Moments of parent `v`. u_inputs : list of ndarrays Moments of input signals. """ D = np.shape(u[0])[-1] if index == 0: # mu Lambda = u_Lambda[0] x0 = u[0][...,0,:] m0 = np.einsum('...ik,...k->...i', Lambda, x0) m1 = -0.5*Lambda elif index == 1: # Lambda x0 = u[0][...,0,:] x0x0 = u[1][...,0,:,:] mu = u_mu[0] mumu = u_mu[1] x0mu = np.einsum('...i,...j->...ij', x0, mu) mux0 = np.swapaxes(x0mu, -1, -2) m0 = -0.5*x0x0 + 0.5*x0mu + 0.5*mux0 - 0.5*mumu m1 = 0.5 elif index == 2: # A XnXn = u[1] XpXn = u[2] v = u_v[0] m0 = v[...,np.newaxis] * XpXn.swapaxes(-1,-2) # The following message matrix could be huge, so let's use a help # function which computes sum(v*XnXn) without computing the huge # v*XnXn explicitly. m1 = -0.5 * message_sum_multiply(parent.plates, (D, D), #parent.dims[1], v[...,np.newaxis,np.newaxis], XnXn[..., :-1, np.newaxis, :, :]) if len(u_inputs): Xn = u[0] z = u_inputs[0][0] zz = u_inputs[0][1] D_inputs = np.shape(z)[-1] m0_B = v[...,None] * Xn[...,1:,:,None] * z[...,None,:] m1_BB = -0.5 * message_sum_multiply(parent.plates, (D_inputs, D_inputs), zz[..., None, :, :], v[ ..., :, None, None]) Xp_z = Xn[...,:-1,:,None] * z[...,None,:] m1_AB = -0.5 * message_sum_multiply(parent.plates, (D, D_inputs), Xp_z[..., None, :, :], v[ ..., :, None, None]) # Construct full message arrays from blocks m0 = np.concatenate([m0, m0_B], axis=-1) row1 = np.concatenate([m1, m1_AB], axis=-1) row2 = np.concatenate([m1_AB.swapaxes(-1,-2), m1_BB], axis=-1) m1 = np.concatenate([row1, row2], axis=-2) #m1 = -0.5 * v[...,np.newaxis,np.newaxis] * XnXn[..., :-1, np.newaxis, :, :] elif index == 3: # v ## if len(u_inputs): ## raise NotImplementedError("Message to innovation not yet implemented " ## "if using input signals") XnXn = u[1] # (...,N,D,D) XpXn = u[2] # (...,N-1,D,D) A = u_A[0][...,:D] # (..., N-1, D, D) AA = u_A[1][...,:D,:D] # (..., N-1, D, D, D) m0 = (- 0.5*np.einsum('...ii->...i', XnXn[...,1:,:,:]) + np.einsum('...ik,...ki->...i', A, XpXn) - 0.5*np.einsum('...ikl,...kl->...i', AA, XnXn[...,:-1,:,:])) if len(u_inputs): Xn = u[0] # (..., N, D) B = u_A[0][...,D:] # (..., N-1, D, inputs) BB = u_A[1][...,D:,D:] # (..., N-1, D, inputs, inputs) AB = u_A[1][...,:D,D:] # (..., N-1, D, D, inputs) Un = u_inputs[0][0] # (..., N-1, inputs) UnUn = u_inputs[0][1] # (..., N-1, inputs, inputs) BUn = np.einsum('...dk,...k->...d', B, Un) m0 = m0 + (- 0.5*np.einsum('...ikl,...kl->...i', BB, UnUn) + BUn * Xn[...,1:,:] - np.einsum('...ijk,...j,...k', AB, Xn[...,:-1,:], Un)) m1 = 0.5 elif index == 4: # input signals raise NotImplementedError() return [m0, m1] def compute_mask_to_parent(self, index, mask): if index == 0: # mu return mask elif index == 1: # Lambda return mask elif index == 2: # A return mask[...,np.newaxis,np.newaxis] elif index == 3: # v return mask[...,np.newaxis,np.newaxis] elif index == 4: # input signals return mask[...,np.newaxis] else: raise ValueError("Index out of bounds") def compute_phi_from_parents(self, u_mu, u_Lambda, u_A, u_v, *u_inputs, mask=True): """ Compute the natural parameters using parents' moments. Parameters ---------- u_parents : list of list of arrays List of parents' lists of moments. Returns ------- phi : list of arrays Natural parameters. dims : tuple Shape of the variable part of phi. """ # Dimensionality of the Gaussian states D = np.shape(u_mu[0])[-1] # Number of time instances in the process N = self.N # Helpful variables (show shapes in comments) mu = u_mu[0] # (..., D) Lambda = u_Lambda[0] # (..., D, D) A = u_A[0][...,:D] # (..., N-1, D, D) AA = u_A[1][...,:D,:D] # (..., N-1, D, D, D) B = u_A[0][...,D:] # (..., N-1, D, inputs) BB = u_A[1][...,D:,D:] # (..., N-1, D, inputs, inputs) AB = u_A[1][...,:D,D:] # (..., N-1, D, D, inputs) v = u_v[0] # (..., N-1, D) if len(u_inputs): inputs = u_inputs[0][0] else: inputs = None # Allocate memory (take into account effective plates) if inputs is not None: plates_phi0 = misc.broadcasted_shape(np.shape(mu)[:-1], np.shape(Lambda)[:-2], np.shape(B)[:-3], np.shape(v)[:-2], np.shape(AB)[:-4]) else: plates_phi0 = misc.broadcasted_shape(np.shape(mu)[:-1], np.shape(Lambda)[:-2]) plates_phi1 = misc.broadcasted_shape(np.shape(Lambda)[:-2], np.shape(v)[:-2], np.shape(AA)[:-4]) plates_phi2 = misc.broadcasted_shape(np.shape(v)[:-2], np.shape(A)[:-3]) phi0 = np.zeros(plates_phi0+(N,D)) phi1 = np.zeros(plates_phi1+(N,D,D)) phi2 = np.zeros(plates_phi2+(N-1,D,D)) # Parameters for x0 phi0[...,0,:] = np.einsum('...ik,...k->...i', Lambda, mu) phi1[...,0,:,:] = Lambda # Effect of the input signals if inputs is not None: phi0[...,1:,:] += np.einsum('...i,...ij,...j->...i', v, B, inputs) AB_v = np.einsum('...dij,...d->...ij', AB, v) phi0[...,:-1,:] -= np.einsum('...ij,...j->...i', AB_v, inputs) # Diagonal blocks: -0.5 * (V_i + A_{i+1}' * V_{i+1} * A_{i+1}) phi1[..., 1:, :, :] = v[...,np.newaxis]*np.identity(D) phi1[..., :-1, :, :] += np.einsum('...kij,...k->...ij', AA, v) phi1 *= -0.5 # Super-diagonal blocks: 0.5 * A.T * V # However, don't multiply by 0.5 because there are both super- and # sub-diagonal blocks (sum them together) phi2[..., :, :, :] = np.einsum('...ji,...j->...ij', A, v) return (phi0, phi1, phi2) def compute_cgf_from_parents(self, u_mu, u_Lambda, u_A, u_v, *u_inputs): """ Compute CGF using the moments of the parents. """ g = _compute_cgf_for_gaussian_markov_chain(u_mu[1], u_Lambda[0], u_Lambda[1], u_v[1], self.N) if len(u_inputs): D = np.shape(u_mu[0])[-1] uu = u_inputs[0][1] BB = u_A[1][...,D:,D:] v = u_v[0] BB_v = np.einsum('...d,...dij->...ij', v, BB) g_inputs = -0.5 * np.einsum('...ij,...ij->...', uu, BB_v) # Sum over time axis if np.ndim(g_inputs) == 0 or np.shape(g_inputs)[-1] == 1: g_inputs *= self.N else: g_inputs = np.sum(g_inputs, axis=-1) g = g + g_inputs return g def plates_to_parent(self, index, plates): """ Computes the plates of this node with respect to a parent. If this node has plates (...), the latent dimensionality is D and the number of time instances is N, the plates with respect to the parents are: mu: (...) Lambda: (...) A: (...,N-1,D) v: (...,N-1,D) Parameters ---------- index : int The index of the parent node to use. """ if index == 0: # mu return plates elif index == 1: # Lambda return plates elif index == 2: # A return plates + (self.N-1, self.D) elif index == 3: # v return plates + (self.N-1, self.D) elif index == 4: # input signals return plates + (self.N-1,) else: raise ValueError("Invalid parent index.") def plates_from_parent(self, index, plates): """ Compute the plates using information of a parent node. If the plates of the parents are: mu: (...) Lambda: (...) A: (...,N-1,D) v: (...,N-1,D) N: () the resulting plates of this node are (...) Parameters ---------- index : int Index of the parent to use. """ if index == 0: # mu return plates elif index == 1: # Lambda return plates elif index == 2: # A return plates[:-2] elif index == 3: # v return plates[:-2] elif index == 4: # input signals return plates[:-1] else: raise ValueError("Invalid parent index.") class GaussianMarkovChain(_TemplateGaussianMarkovChain): r""" Node for Gaussian Markov chain random variables. In a simple case, the graphical model can be presented as: .. bayesnet:: \tikzstyle{latent} += [minimum size=30pt]; \node[latent] (x0) {$\mathbf{x}_0$}; \node[latent, right=of x0] (x1) {$\mathbf{x}_1$}; \node[right=of x1] (dots) {$\cdots$}; \node[latent, right=of dots] (xn) {$\mathbf{x}_{N-1}$}; \edge {x0}{x1}; \edge {x1}{dots}; \edge {dots}{xn}; \node[latent, above left=1 and 0.1 of x0] (mu) {$\boldsymbol{\mu}$}; \node[latent, above right=1 and 0.1 of x0] (Lambda) {$\mathbf{\Lambda}$}; \node[latent, above left=1 and 0.1 of dots] (A) {$\mathbf{A}$}; \node[latent, above right=1 and 0.1 of dots] (nu) {$\boldsymbol{\nu}$}; \edge {mu,Lambda} {x0}; \edge {A,nu} {x1,dots,xn}; where :math:`\boldsymbol{\mu}` and :math:`\mathbf{\Lambda}` are the mean and the precision matrix of the initial state, :math:`\mathbf{A}` is the state dynamics matrix and :math:`\boldsymbol{\nu}` is the precision of the innovation noise. It is possible that :math:`\mathbf{A}` and/or :math:`\boldsymbol{\nu}` are different for each transition instead of being constant. The probability distribution is .. math:: p(\mathbf{x}_0, \ldots, \mathbf{x}_{N-1}) = p(\mathbf{x}_0) \prod^{N-1}_{n=1} p(\mathbf{x}_n | \mathbf{x}_{n-1}) where .. math:: p(\mathbf{x}_0) &= \mathcal{N}(\mathbf{x}_0 | \boldsymbol{\mu}, \mathbf{\Lambda}) \\ p(\mathbf{x}_n|\mathbf{x}_{n-1}) &= \mathcal{N}(\mathbf{x}_n | \mathbf{A}_{n-1}\mathbf{x}_{n-1}, \mathrm{diag}(\boldsymbol{\nu}_{n-1})). Parameters ---------- mu : Gaussian-like node or (...,D)-array :math:`\boldsymbol{\mu}`, mean of :math:`x_0`, :math:`D`-dimensional with plates (...) Lambda : Wishart-like node or (...,D,D)-array :math:`\mathbf{\Lambda}`, precision matrix of :math:`x_0`, :math:`D\times D` -dimensional with plates (...) A : Gaussian-like node or (D,D)-array or (...,1,D,D)-array or (...,N-1,D,D)-array :math:`\mathbf{A}`, state dynamics matrix, :math:`D`-dimensional with plates (D,) or (...,1,D) or (...,N-1,D) nu : gamma-like node or (D,)-array or (...,1,D)-array or (...,N-1,D)-array :math:`\boldsymbol{\nu}`, diagonal elements of the precision of the innovation process, plates (D,) or (...,1,D) or (...,N-1,D) n : int, optional :math:`N`, the length of the chain. Must be given if :math:`\mathbf{A}` and :math:`\boldsymbol{\nu}` are constant over time. See also -------- Gaussian, GaussianARD, Wishart, Gamma, SwitchingGaussianMarkovChain, VaryingGaussianMarkovChain, CategoricalMarkovChain """ def __init__(self, mu, Lambda, A, nu, n=None, inputs=None, **kwargs): """ Create GaussianMarkovChain node. """ super().__init__(mu, Lambda, A, nu, n=n, inputs=inputs, **kwargs) @classmethod def _constructor(cls, mu, Lambda, A, v, n=None, inputs=None, **kwargs): """ Constructs distribution and moments objects. Compute the dimensions of phi and u. The plates and dimensions of the parents should be: mu: (...) and D-dimensional Lambda: (...) and D-dimensional A: (...,1,D) or (...,N-1,D) and D-dimensional v: (...,1,D) or (...,N-1,D) and 0-dimensional N: () and 0-dimensional (dummy parent) Check that the dimensionalities of the parents are proper. For instance, A should be a collection of DxD matrices, thus the dimensionality and the last plate should both equal D. Similarly, `v` should be a collection of diagonal innovation matrix elements, thus the last plate should equal D. """ # Check whether to use input signals or not if inputs is None: _parent_moments = (GaussianMoments(1), WishartMoments(), GaussianMoments(1), GammaMoments()) else: _parent_moments = (GaussianMoments(1), WishartMoments(), GaussianMoments(1), GammaMoments(), GaussianMoments(1)) # Ensure that parent nodes are of proper type mu = cls._ensure_moments(mu, _parent_moments[0]) Lambda = cls._ensure_moments(Lambda, _parent_moments[1]) A = cls._ensure_moments(A, _parent_moments[2]) v = cls._ensure_moments(v, _parent_moments[3]) if inputs is not None: inputs = cls._ensure_moments(inputs, _parent_moments[4]) # Time instances from input signals if inputs is not None and len(inputs.plates) >= 1: n_inputs = inputs.plates[-1] else: n_inputs = 1 # Time instances from state dynamics matrix if len(A.plates) >= 2: n_A = A.plates[-2] else: n_A = 1 # Time instances from innovation noise if len(v.plates) >= 2: n_v = v.plates[-2] else: n_v = 1 # Check consistency of the number of time instances if ( (n_v != n_A and n_v != 1 and n_A != 1) or (n_inputs != n_A and n_inputs != 1 and n_A != 1) or (n_inputs != n_v and n_inputs != 1 and n_v != 1) ): raise Exception("Plates of parents are giving different number of time instances") n_parents = max(n_v, n_A, n_inputs) if n is None: if n_parents == 1: raise Exception("The number of time instances could not be " "determined automatically. Give the number of " "time instances.") n = n_parents + 1 elif n_parents != 1 and n_parents+1 != n: raise Exception("The number of time instances must match " "the number of last plates of parents: " "%d != %d+1" % (n, n_parents)) # Dimensionality of the states D = mu.dims[0][0] # Number of states M = n # Dimensionality of the inputs if inputs is None: D_inputs = 0 else: D_inputs = inputs.dims[0][0] # Check mu if mu.dims != ( (D,), (D,D) ): raise Exception("First parent has wrong dimensionality") # Check Lambda if Lambda.dims != ( (D,D), () ): raise Exception("Second parent has wrong dimensionality") # Check A if A.dims != ( (D+D_inputs,), (D+D_inputs,D+D_inputs) ): raise Exception("Third parent has wrong dimensionality") if len(A.plates) == 0 or A.plates[-1] != D: raise Exception("Third parent should have a last plate " "equal to the dimensionality of the " "system.") if (len(A.plates) >= 2 and A.plates[-2] != 1 and A.plates[-2] != M-1): raise ValueError("The second last plate of the third " "parent should have length equal to one or " "N-1, where N is the number of time " "instances.") # Check v if v.dims != ( (), () ): raise Exception("Fourth parent has wrong dimensionality") if len(v.plates) == 0 or v.plates[-1] != D: raise Exception("Fourth parent should have a last plate " "equal to the dimensionality of the " "system.") if (len(v.plates) >= 2 and v.plates[-2] != 1 and v.plates[-2] != M-1): raise ValueError("The second last plate of the fourth " "parent should have length equal to one or " "N-1 where N is the number of time " "instances.") # Check input signals if inputs is not None: if inputs.dims != ( (D_inputs,), (D_inputs, D_inputs) ): raise ValueError("Input signals have wrong dimensionality") dims = ( (M,D), (M,D,D), (M-1,D,D) ) distribution = GaussianMarkovChainDistribution(M, D) if inputs is None: parents = [mu, Lambda, A, v] else: parents = [mu, Lambda, A, v, inputs] return ( parents, kwargs, dims, cls._total_plates(kwargs.get('plates'), distribution.plates_from_parent(0, mu.plates), distribution.plates_from_parent(1, Lambda.plates), distribution.plates_from_parent(2, A.plates), distribution.plates_from_parent(3, v.plates)), distribution, cls._moments, _parent_moments) class VaryingGaussianMarkovChainDistribution(TemplateGaussianMarkovChainDistribution): """ Sub-classes implement distribution specific computations. """ def compute_message_to_parent(self, parent, index, u, u_mu, u_Lambda, u_B, u_S, u_v): """ Compute a message to a parent. Parameters ----------- index : int Index of the parent requesting the message. u : list of ndarrays Moments of this node. u_mu : list of ndarrays Moments of parent `mu`. u_Lambda : list of ndarrays Moments of parent `Lambda`. u_B : list of ndarrays Moments of parent `B`. u_S : list of ndarrays Moments of parent `S`. u_v : list of ndarrays Moments of parent `v`. """ if index == 0: # mu raise NotImplementedError() elif index == 1: # Lambda raise NotImplementedError() elif index == 2: # B, (...,D)x(D,K) XnXn = u[1] # (...,N,D,D) XpXn = u[2] # (...,N,D,D) S = misc.atleast_nd(u_S[0], 2) # (...,N,K) SS = misc.atleast_nd(u_S[1], 3) # (...,N,K,K) v = misc.atleast_nd(u_v[0], 2) # (...,N,D) # m0: (...,D,D,K) m0 = np.einsum('...nji,...nk,...ni->...ijk', XpXn, S, v) # m1: (...,D,D,K,D,K) if np.ndim(v) >= 2 and np.shape(v)[-2] > 1: raise ValueError("Innovation noise is time dependent") m1 = np.einsum('...nij,...nkl->...ikjl', XnXn[...,:-1,:,:], SS) m1 = -0.5 * np.einsum('...ikjl,...d->...dikjl', m1, v[...,0,:]) elif index == 3: # S, (...,N-1)x(K) XnXn = u[1] # (...,N,D,D) XpXn = u[2] # (...,N,D,D) B = u_B[0] # (...,D,D,K) BB = u_B[1] # (...,D,D,K,D,K) v = u_v[0] # (...,N,D) # m0: (...,N,K) m0 = np.einsum('...nji,...ijk,...ni->...nk', XpXn, B, np.atleast_2d(v)) # m1: (...,N,K,K) if np.ndim(v) >= 2 and np.shape(v)[-2] > 1: raise ValueError("Innovation noise is time dependent") m1 = np.einsum('...dikjl,...d->...ikjl', BB, np.atleast_2d(v)[...,0,:]) m1 = -0.5 * np.einsum('...nij,...ikjl->...nkl', XnXn[...,:-1,:,:], m1) elif index == 4: # v raise NotImplementedError() elif index == 5: # N raise NotImplementedError() return [m0, m1] def compute_mask_to_parent(self, index, mask): if index == 0: # mu return mask elif index == 1: # Lambda return mask elif index == 2: # B return mask[...,np.newaxis] # new plate axis for D elif index == 3: # S return mask[...,np.newaxis] # new plate axis for N elif index == 4: # v return mask[...,np.newaxis,np.newaxis] # new plate axis for N and D elif index == 5: # N return mask else: raise ValueError("Invalid index") def compute_phi_from_parents(self, u_mu, u_Lambda, u_B, u_S, u_v, mask=True): """ Compute the natural parameters using parents' moments. Parameters ---------- u_parents : list of list of arrays List of parents' lists of moments. Returns ------- phi : list of arrays Natural parameters. dims : tuple Shape of the variable part of phi. """ # Dimensionality of the Gaussian states D = np.shape(u_mu[0])[-1] # Number of time instances in the process N = self.N # Helpful variables (show shapes in comments) mu = u_mu[0] # (..., D) Lambda = u_Lambda[0] # (..., D, D) B = u_B[0] # (..., D, D, K) BB = u_B[1] # (..., D, D, K, D, K) S = u_S[0] # (..., N-1, K) or (..., 1, K) SS = u_S[1] # (..., N-1, K, K) v = u_v[0] # (..., N-1, D) or (..., 1, D) # TODO/FIXME: Take into account plates! plates_phi0 = misc.broadcasted_shape(np.shape(mu)[:-1], np.shape(Lambda)[:-2]) plates_phi1 = misc.broadcasted_shape(np.shape(Lambda)[:-2], np.shape(v)[:-2], np.shape(BB)[:-5], np.shape(SS)[:-3]) plates_phi2 = misc.broadcasted_shape(np.shape(B)[:-3], np.shape(S)[:-2], np.shape(v)[:-2]) phi0 = np.zeros(plates_phi0 + (N,D)) phi1 = np.zeros(plates_phi1 + (N,D,D)) phi2 = np.zeros(plates_phi2 + (N-1,D,D)) # Parameters for x0 phi0[...,0,:] = np.einsum('...ik,...k->...i', Lambda, mu) phi1[...,0,:,:] = Lambda # Diagonal blocks: -0.5 * (V_i + A_{i+1}' * V_{i+1} * A_{i+1}) phi1[..., 1:, :, :] = v[...,np.newaxis]*np.identity(D) if np.ndim(v) >= 2 and np.shape(v)[-2] > 1: raise Exception("This implementation is not efficient if " "innovation noise is time-dependent.") phi1[..., :-1, :, :] += np.einsum('...dikjl,...kl,...d->...ij', BB[...,None,:,:,:,:,:], SS, v) else: # We know that S does not have the D plate so we can sum that plate # axis out v_BB = np.einsum('...dikjl,...d->...ikjl', BB[...,None,:,:,:,:,:], v) phi1[..., :-1, :, :] += np.einsum('...ikjl,...kl->...ij', v_BB, SS) #phi1[..., :-1, :, :] += np.einsum('...kij,...k->...ij', AA, v) phi1 *= -0.5 # Super-diagonal blocks: 0.5 * A.T * V # However, don't multiply by 0.5 because there are both super- and # sub-diagonal blocks (sum them together) phi2[..., :, :, :] = np.einsum('...jik,...k,...j->...ij', B[...,None,:,:,:], S, v) #phi2[..., :, :, :] = np.einsum('...ji,...j->...ij', A, v) return (phi0, phi1, phi2) def compute_cgf_from_parents(self, u_mu, u_Lambda, u_B, u_S, u_v): """ Compute CGF using the moments of the parents. """ return _compute_cgf_for_gaussian_markov_chain(u_mu[1], u_Lambda[0], u_Lambda[1], u_v[1], self.N) def plates_to_parent(self, index, plates): """ Computes the plates of this node with respect to a parent. If this node has plates (...), the latent dimensionality is D and the number of time instances is N, the plates with respect to the parents are: mu: (...) Lambda: (...) A: (...,N-1,D) v: (...,N-1,D) Parameters ----------- index : int The index of the parent node to use. """ if index == 0: # mu return plates elif index == 1: # Lambda return plates elif index == 2: # B return plates + (self.D,) elif index == 3: # S return plates + (self.N-1,) elif index == 4: # v return plates + (self.N-1,self.D) else: raise ValueError("Invalid parent index.") def plates_from_parent(self, index, plates): """ Compute the plates using information of a parent node. If the plates of the parents are: mu: (...) Lambda: (...) B: (...,D) S: (...,N-1) v: (...,N-1,D) N: () the resulting plates of this node are (...) Parameters ---------- index : int Index of the parent to use. """ if index == 0: # mu return plates elif index == 1: # Lambda return plates elif index == 2: # B, remove last plate D return plates[:-1] elif index == 3: # S, remove last plate N-1 return plates[:-1] elif index == 4: # v, remove last plates N-1,D return plates[:-2] else: raise ValueError("Invalid parent index.") class VaryingGaussianMarkovChain(_TemplateGaussianMarkovChain): r""" Node for Gaussian Markov chain random variables with time-varying dynamics. The node models a sequence of Gaussian variables :math:`\mathbf{x}_0,\ldots,\mathbf{x}_{N-1}` with linear Markovian dynamics. The time variability of the dynamics is obtained by modelling the state dynamics matrix as a linear combination of a set of matrices with time-varying linear combination weights. The graphical model can be presented as: .. bayesnet:: \tikzstyle{latent} += [minimum size=40pt]; \node[latent] (x0) {$\mathbf{x}_0$}; \node[latent, right=of x0] (x1) {$\mathbf{x}_1$}; \node[right=of x1] (dots) {$\cdots$}; \node[latent, right=of dots] (xn) {$\mathbf{x}_{N-1}$}; \edge {x0}{x1}; \edge {x1}{dots}; \edge {dots}{xn}; \node[latent, above left=1 and 0.1 of x0] (mu) {$\boldsymbol{\mu}$}; \node[latent, above right=1 and 0.1 of x0] (Lambda) {$\mathbf{\Lambda}$}; \node[det, below=of x1] (A0) {$\mathbf{A}_0$}; \node[right=of A0] (Adots) {$\cdots$}; \node[det, right=of Adots] (An) {$\mathbf{A}_{N-2}$}; \node[latent, above=of dots] (nu) {$\boldsymbol{\nu}$}; \edge {mu,Lambda} {x0}; \edge {nu} {x1,dots,xn}; \edge {A0} {x1}; \edge {Adots} {dots}; \edge {An} {xn}; \node[latent, below=of A0] (s0) {$s_{0,k}$}; \node[right=of s0] (sdots) {$\cdots$}; \node[latent, right=of sdots] (sn) {$\mathbf{s}_{N-2,k}$}; \node[latent, left=of s0] (B) {$\mathbf{B}_k$}; \edge {B} {A0, Adots, An}; \edge {s0} {A0}; \edge {sdots} {Adots}; \edge {sn} {An}; \plate {K} {(B)(s0)(sdots)(sn)} {$k=0,\ldots,K-1$}; where :math:`\boldsymbol{\mu}` and :math:`\mathbf{\Lambda}` are the mean and the precision matrix of the initial state, :math:`\boldsymbol{\nu}` is the precision of the innovation noise, and :math:`\mathbf{A}_n` are the state dynamics matrix obtained by mixing matrices :math:`\mathbf{B}_k` with weights :math:`s_{n,k}`. The probability distribution is .. math:: p(\mathbf{x}_0, \ldots, \mathbf{x}_{N-1}) = p(\mathbf{x}_0) \prod^{N-1}_{n=1} p(\mathbf{x}_n | \mathbf{x}_{n-1}) where .. math:: p(\mathbf{x}_0) &= \mathcal{N}(\mathbf{x}_0 | \boldsymbol{\mu}, \mathbf{\Lambda}) \\ p(\mathbf{x}_n|\mathbf{x}_{n-1}) &= \mathcal{N}(\mathbf{x}_n | \mathbf{A}_{n-1}\mathbf{x}_{n-1}, \mathrm{diag}(\boldsymbol{\nu})), \quad \text{for } n=1,\ldots,N-1, \\ \mathbf{A}_n & = \sum^{K-1}_{k=0} s_{n,k} \mathbf{B}_k, \quad \text{for } n=0,\ldots,N-2. Parameters ---------- mu : Gaussian-like node or (...,D)-array :math:`\boldsymbol{\mu}`, mean of :math:`x_0`, :math:`D`-dimensional with plates (...) Lambda : Wishart-like node or (...,D,D)-array :math:`\mathbf{\Lambda}`, precision matrix of :math:`x_0`, :math:`D\times D` -dimensional with plates (...) B : Gaussian-like node or (...,D,D,K)-array :math:`\{\mathbf{B}_k\}_{k=0}^{K-1}`, a set of state dynamics matrix, :math:`D \times K`-dimensional with plates (...,D) S : Gaussian-like node or (...,N-1,K)-array :math:`\{\mathbf{s}_0,\ldots,\mathbf{s}_{N-2}\}`, time-varying weights of the linear combination, :math:`K`-dimensional with plates (...,N-1) nu : gamma-like node or (...,D)-array :math:`\boldsymbol{\nu}`, diagonal elements of the precision of the innovation process, plates (...,D) n : int, optional :math:`N`, the length of the chain. Must be given if :math:`\mathbf{S}` does not have plates over the time domain (which would not make sense). See also -------- Gaussian, GaussianARD, Wishart, Gamma, GaussianMarkovChain, SwitchingGaussianMarkovChain Notes ----- Equivalent model block can be constructed with :class:`GaussianMarkovChain` by explicitly using :class:`SumMultiply` to compute the linear combination. However, that approach is not very efficient for large datasets because it does not utilize the structure of :math:`\mathbf{A}_n`, thus it explicitly computes huge moment arrays. References ---------- :cite:`Luttinen:2014` """ _parent_moments = (GaussianMoments(1), WishartMoments(), GaussianMoments(2), GaussianMoments(1), GammaMoments()) def __init__(self, mu, Lambda, B, S, nu, n=None, **kwargs): """ Create VaryingGaussianMarkovChain node. """ super().__init__(mu, Lambda, B, S, nu, n=n, **kwargs) @classmethod @ensureparents def _constructor(cls, mu, Lambda, B, S, v, n=None, **kwargs): """ Constructs distribution and moments objects. Compute the dimensions of phi and u. The plates and dimensions of the parents should be: mu: (...) and D-dimensional Lambda: (...) and D-dimensional B: (...,D) and (D,K)-dimensional S: (...,N-1) and K-dimensional v: (...,1,D) or (...,N-1,D) and 0-dimensional N: () and 0-dimensional (dummy parent) Check that the dimensionalities of the parents are proper. """ # A dummy wrapper for the number of time instances. n_S = 1 if len(S.plates) >= 1: n_S = S.plates[-1] n_v = 1 if len(v.plates) >= 2: n_v = v.plates[-2] if n_v != n_S and n_v != 1 and n_S != 1: raise Exception( "Plates of A and v are giving different number of time " "instances") n_S = max(n_v, n_S) if n is None: if n_S == 1: raise Exception( "The number of time instances could not be determined " "automatically. Give the number of time instances.") n = n_S + 1 elif n_S != 1 and n_S+1 != n: raise Exception( "The number of time instances must match the number of last " "plates of parents:" "%d != %d+1" % (n, n_S)) D = mu.dims[0][0] K = B.dims[0][-1] M = n #N.get_moments()[0] # Check mu if mu.dims != ( (D,), (D,D) ): raise ValueError("First parent has wrong dimensionality") # Check Lambda if Lambda.dims != ( (D,D), () ): raise ValueError("Second parent has wrong dimensionality") # Check B if B.dims != ( (D,K), (D,K,D,K) ): raise ValueError("Third parent has wrong dimensionality {0}. Should be {1}.".format(B.dims[0], (D,K))) if len(B.plates) == 0 or B.plates[-1] != D: raise ValueError("Third parent should have a last plate " "equal to the dimensionality of the " "system.") if S.dims != ( (K,), (K,K) ): raise ValueError("Fourth parent has wrong dimensionality %s, " "should be %s" % (S.dims, ( (K,), (K,K) ))) if (len(S.plates) >= 1 and S.plates[-1] != 1 and S.plates[-1] != M-1): raise ValueError("The last plate of the fourth " "parent should have length equal to one or " "N-1, where N is the number of time " "instances.") # Check v if v.dims != ( (), () ): raise Exception("Fifth parent has wrong dimensionality") if len(v.plates) == 0 or v.plates[-1] != D: raise Exception("Fifth parent should have a last plate " "equal to the dimensionality of the " "system.") if (len(v.plates) >= 2 and v.plates[-2] != 1 and v.plates[-2] != M-1): raise ValueError("The second last plate of the fifth " "parent should have length equal to one or " "N-1 where N is the number of time " "instances.") dims = ( (M,D), (M,D,D), (M-1,D,D) ) distribution = VaryingGaussianMarkovChainDistribution(M, D) parents = [mu, Lambda, B, S, v] return (parents, kwargs, dims, cls._total_plates(kwargs.get('plates'), distribution.plates_from_parent(0, mu.plates), distribution.plates_from_parent(1, Lambda.plates), distribution.plates_from_parent(2, B.plates), distribution.plates_from_parent(3, S.plates), distribution.plates_from_parent(4, v.plates)), distribution, cls._moments, cls._parent_moments) class SwitchingGaussianMarkovChainDistribution(TemplateGaussianMarkovChainDistribution): """ Sub-classes implement distribution specific computations. """ def __init__(self, N, D, K): self.K = K super().__init__(N, D) def compute_message_to_parent(self, parent, index, u, u_mu, u_Lambda, u_B, u_Z, u_v): """ Compute a message to a parent. Parameters ---------- index : int Index of the parent requesting the message. u : list of ndarrays Moments of this node. u_mu : list of ndarrays Moments of parent `mu`. u_Lambda : list of ndarrays Moments of parent `Lambda`. u_B : list of ndarrays Moments of parent `B`. u_Z : list of ndarrays Moments of parent `Z`. u_v : list of ndarrays Moments of parent `v`. """ if index == 0: # mu raise NotImplementedError() elif index == 1: # Lambda raise NotImplementedError() elif index == 2: # B, (...,K,D)x(D) XnXn = u[1] # (...,N,D,D) XpXn = u[2] # (...,N-1,D,D) Z = u_Z[0] # (...,N-1,K) v = misc.atleast_nd(u_v[0], 2) # (...,N-1,D) # Check that there is no time-dependency in v and remove the axis if np.ndim(v) >= 2 and np.shape(v)[-2] > 1: raise ValueError("Innovation noise is time dependent") v = np.squeeze(v, axis=-2) # m0: (...,K,D,D) m0 = np.einsum('...nji,...nk,...i->...kij', XpXn, Z, v) # m1: (...,K,D,D,D) m1 = np.einsum('...nij,...nk->...kij', XnXn[...,:-1,:,:], Z) m1 = -0.5 * np.einsum('...kij,...d->...kdij', m1, v) return [m0, m1] elif index == 3: # Z, (...,N-1)x(K) XnXn = u[1] # (...,N,D,D) XpXn = u[2] # (...,N-1,D,D) B = u_B[0] # (...,K,D,D) BB = u_B[1] # (...,K,D,D,D) v = misc.atleast_nd(u_v[0], 2) # (...,N-1,D) logv = misc.atleast_nd(u_v[1], 2) # (...,N-1,D) # Check that there is no time-dependency in v and remove the axis if np.ndim(v) >= 2 and np.shape(v)[-2] > 1: raise ValueError("Innovation noise is time dependent") v = np.squeeze(v, axis=-2) if np.ndim(logv) >= 2 and np.shape(logv)[-2] > 1: raise ValueError("Innovation noise is time dependent") logv = np.squeeze(logv, axis=-2) XnXn_v = np.einsum('...nii,...i->...n', XnXn[...,1:,:,:], v) XpXn_v_B = np.einsum('...nil,...l,...kli->...nk', XpXn, v, B) BvB = np.einsum('...kdij,...d->...kij', BB, v) XpXp_BvB = np.einsum('...nij,...kij->...nk', XnXn[...,:-1,:,:], BvB) m0 = ( -0.5 * XnXn_v[...,None] + XpXn_v_B -0.5 * XpXp_BvB +0.5 * np.sum(logv, axis=-1)[...,None,None] -0.5 * self.D * np.log(2*np.pi) ) return [m0] elif index == 4: # v raise NotImplementedError() elif index == 5: # N raise NotImplementedError() def compute_mask_to_parent(self, index, mask): if index == 0: # mu: (...)x(N,D) -> (...)x(D) return mask elif index == 1: # Lambda: (...)x(N,D) -> (...)x(D,D) return mask elif index == 2: # B: (...)x(N,D) -> (...,K,D)x(D) return mask[...,None,None] elif index == 3: # Z: (...)x(N,D) -> (...,N-1)x(K) return mask[...,None] elif index == 4: # v: (...)x(N,D) -> (...,N-1,D)x() return mask[...,None,None] else: raise ValueError("Invalid index") def compute_phi_from_parents(self, u_mu, u_Lambda, u_B, u_Z, u_v, mask=True): """ Compute the natural parameters using parents' moments. Parameters ---------- u_parents : list of list of arrays List of parents' lists of moments. Returns ------- phi : list of arrays Natural parameters. dims : tuple Shape of the variable part of phi. """ # Dimensionality of the Gaussian states D = np.shape(u_mu[0])[-1] # Number of time instances in the process N = self.N # Helpful variables (show shapes in comments) mu = u_mu[0] # (..., D) Lambda = u_Lambda[0] # (..., D, D) B = u_B[0] # (..., K, D, D) BB = u_B[1] # (..., K, D, D, D) Z = u_Z[0] # (..., N-1, K) v = misc.atleast_nd(u_v[0], 2) # (..., N-1, D) or (..., 1, D) # TODO/FIXME: Take into account plates! plates_phi0 = misc.broadcasted_shape(np.shape(mu)[:-1], np.shape(Lambda)[:-2]) plates_phi1 = misc.broadcasted_shape(np.shape(Lambda)[:-2], np.shape(v)[:-2], np.shape(BB)[:-4], np.shape(Z)[:-2]) plates_phi2 = misc.broadcasted_shape(np.shape(B)[:-3], np.shape(Z)[:-2], np.shape(v)[:-2]) phi0 = np.zeros(plates_phi0 + (N,D)) phi1 = np.zeros(plates_phi1 + (N,D,D)) phi2 = np.zeros(plates_phi2 + (N-1,D,D)) # Parameters for x0 phi0[...,0,:] = np.einsum('...ik,...k->...i', Lambda, mu) phi1[...,0,:,:] = Lambda # Diagonal blocks: -0.5 * (V_i + A_{i+1}' * V_{i+1} * A_{i+1}) phi1[..., 1:, :, :] = v[...,None]*np.identity(D) if np.shape(v)[-2] > 1: raise Exception("This implementation is not efficient if " "innovation noise is time-dependent.") phi1[..., :-1, :, :] += np.einsum('...kdij,...nk,...nd->...nij', BB[...,:,:,:,:], Z, v) else: # We know that S does not have the D plate so we can sum that plate # axis out v_BB = np.einsum('...kdij,...nd->...nkij', BB[...,:,:,:,:], v) phi1[..., :-1, :, :] += np.einsum('...nkij,...nk->...nij', v_BB, Z) phi1 *= -0.5 # Super-diagonal blocks: 0.5 * A.T * V # However, don't multiply by 0.5 because there are both super- and # sub-diagonal blocks (sum them together) phi2[..., :, :, :] = np.einsum('...kji,...nk,...nj->...nij', B[...,:,:,:], Z, v) return (phi0, phi1, phi2) def compute_cgf_from_parents(self, u_mu, u_Lambda, u_B, u_Z, u_v): """ Compute CGF using the moments of the parents. """ return _compute_cgf_for_gaussian_markov_chain(u_mu[1], u_Lambda[0], u_Lambda[1], u_v[1], self.N) def plates_to_parent(self, index, plates): """ Computes the plates of this node with respect to a parent. If this node has plates (...), the latent dimensionality is D and the number of time instances is N, the plates with respect to the parents are: mu: (...) Lambda: (...) A: (...,N-1,D) v: (...,N-1,D) Parameters ---------- index : int The index of the parent node to use. """ if index == 0: # mu: (...)x(N,D) -> (...)x(D) return plates elif index == 1: # Lambda: (...)x(N,D) -> (...)x(D,D) return plates elif index == 2: # B: (...)x(N,D) -> (...,K,D)x(D) return plates + (self.K,self.D) elif index == 3: # Z: (...)x(N,D) -> (...,N-1)x(K) return plates + (self.N-1,) elif index == 4: # v: (...)x(N,D) -> (...,N-1,D)x() return plates + (self.N-1,self.D) else: raise ValueError("Invalid parent index.") def plates_from_parent(self, index, plates): """ Compute the plates using information of a parent node. If the plates of the parents are: mu: (...) Lambda: (...) B: (...,D) S: (...,N-1) v: (...,N-1,D) N: () the resulting plates of this node are (...) Parameters ---------- index : int Index of the parent to use. """ if index == 0: # mu: (...)x(D) -> (...)x(N,D) return plates elif index == 1: # Lambda: (...)x(D,D) -> (...)x(N,D) return plates elif index == 2: # B: (...,K,D)x(D) -> (...)x(N,D) return plates[:-2] elif index == 3: # Z: (...,N-1)x(K) -> (...)x(N,D) return plates[:-1] elif index == 4: # v: (...,N-1,D)x() -> (...)x(N,D) return plates[:-2] else: raise ValueError("Invalid parent index.") class SwitchingGaussianMarkovChain(_TemplateGaussianMarkovChain): r""" Node for Gaussian Markov chain random variables with switching dynamics. The node models a sequence of Gaussian variables :math:`\mathbf{x}_0,\ldots,\mathbf{x}_{N-1}$ with linear Markovian dynamics. The dynamics may change in time, which is obtained by having a set of matrices and at each time selecting one of them as the state dynamics matrix. The graphical model can be presented as: .. bayesnet:: \tikzstyle{latent} += [minimum size=40pt]; \node[latent] (x0) {$\mathbf{x}_0$}; \node[latent, right=of x0] (x1) {$\mathbf{x}_1$}; \node[right=of x1] (dots) {$\cdots$}; \node[latent, right=of dots] (xn) {$\mathbf{x}_{N-1}$}; \edge {x0}{x1}; \edge {x1}{dots}; \edge {dots}{xn}; \node[latent, above left=1 and 0.1 of x0] (mu) {$\boldsymbol{\mu}$}; \node[latent, above right=1 and 0.1 of x0] (Lambda) {$\mathbf{\Lambda}$}; \node[det, below=of x1] (A0) {$\mathbf{A}_0$}; \node[right=of A0] (Adots) {$\cdots$}; \node[det, right=of Adots] (An) {$\mathbf{A}_{N-2}$}; \node[latent, above=of dots] (nu) {$\boldsymbol{\nu}$}; \edge {mu,Lambda} {x0}; \edge {nu} {x1,dots,xn}; \edge {A0} {x1}; \edge {Adots} {dots}; \edge {An} {xn}; \node[latent, below=of A0] (z0) {$z_0$}; \node[right=of z0] (zdots) {$\cdots$}; \node[latent, right=of zdots] (zn) {$z_{N-2}$}; \node[latent, left=of z0] (B) {$\mathbf{B}_k$}; \edge {B} {A0, Adots, An}; \edge {z0} {A0}; \edge {zdots} {Adots}; \edge {zn} {An}; \plate {K} {(B)} {$k=0,\ldots,K-1$}; where :math:`\boldsymbol{\mu}` and :math:`\mathbf{\Lambda}` are the mean and the precision matrix of the initial state, :math:`\boldsymbol{\nu}` is the precision of the innovation noise, and :math:`\mathbf{A}_n` are the state dynamics matrix obtained by selecting one of the matrices :math:`\{\mathbf{B}_k\}^{K-1}_{k=0}` at each time. The selections are provided by :math:`z_n\in\{0,\ldots,K-1\}`. The probability distribution is .. math:: p(\mathbf{x}_0, \ldots, \mathbf{x}_{N-1}) = p(\mathbf{x}_0) \prod^{N-1}_{n=1} p(\mathbf{x}_n | \mathbf{x}_{n-1}) where .. math:: p(\mathbf{x}_0) &= \mathcal{N}(\mathbf{x}_0 | \boldsymbol{\mu}, \mathbf{\Lambda}) \\ p(\mathbf{x}_n|\mathbf{x}_{n-1}) &= \mathcal{N}(\mathbf{x}_n | \mathbf{A}_{n-1}\mathbf{x}_{n-1}, \mathrm{diag}(\boldsymbol{\nu})), \quad \text{for } n=1,\ldots,N-1, \\ \mathbf{A}_n &= \mathbf{B}_{z_n}, \quad \text{for } n=0,\ldots,N-2. Parameters ---------- mu : Gaussian-like node or (...,D)-array :math:`\boldsymbol{\mu}`, mean of :math:`x_0`, :math:`D`-dimensional with plates (...) Lambda : Wishart-like node or (...,D,D)-array :math:`\mathbf{\Lambda}`, precision matrix of :math:`x_0`, :math:`D\times D` -dimensional with plates (...) B : Gaussian-like node or (...,D,D,K)-array :math:`\{\mathbf{B}_k\}_{k=0}^{K-1}`, a set of state dynamics matrix, :math:`D \times K`-dimensional with plates (...,D) Z : categorical-like node or (...,N-1)-array :math:`\{z_0,\ldots,z_{N-2}\}`, time-dependent selection, :math:`K`-categorical with plates (...,N-1) nu : gamma-like node or (...,D)-array :math:`\boldsymbol{\nu}`, diagonal elements of the precision of the innovation process, plates (...,D) n : int, optional :math:`N`, the length of the chain. Must be given if :math:`\mathbf{Z}` does not have plates over the time domain (which would not make sense). See also -------- Gaussian, GaussianARD, Wishart, Gamma, GaussianMarkovChain, VaryingGaussianMarkovChain, Categorical, CategoricalMarkovChain Notes ----- Equivalent model block can be constructed with :class:`GaussianMarkovChain` by explicitly using :class:`Gate` to select the state dynamics matrix. However, that approach is not very efficient for large datasets because it does not utilize the structure of :math:`\mathbf{A}_n`, thus it explicitly computes huge moment arrays. """ def __init__(self, mu, Lambda, B, Z, nu, n=None, **kwargs): """ Create SwitchingGaussianMarkovChain node. """ super().__init__(mu, Lambda, B, Z, nu, n=n, **kwargs) @classmethod def _constructor(cls, mu, Lambda, B, Z, v, n=None, **kwargs): """ Constructs distribution and moments objects. Compute the dimensions of phi and u. The plates and dimensions of the parents should be: mu: (...) and D-dimensional Lambda: (...) and D-dimensional B: (...,K,D) and D-dimensional Z: (...,N-1) and K-dimensional v: (...,1,D) or (...,N-1,D) and 0-dimensional Check that the dimensionalities of the parents are proper. """ # Infer the number of dynamic matrices B = cls._ensure_moments(B, GaussianMoments(2)) K = B.plates[-2] parent_moments = (GaussianMoments(1), WishartMoments(), GaussianMoments(1), CategoricalMoments(K), GammaMoments()) # Infer the length of the chain Z = cls._ensure_moments(Z, parent_moments[3]) v = cls._ensure_moments(v, parent_moments[4]) n_Z = 1 if len(Z.plates) == 0: raise ValueError("Z must have temporal axis on plates") n_Z = Z.plates[-1] n_v = 1 if len(v.plates) >= 2: n_v = v.plates[-2] if n_v != n_Z and n_v != 1 and n_Z != 1: raise Exception( "Plates of Z and v are giving different number of time " "instances") n_Z = max(n_v, n_Z) if n is None: if n_Z == 1: raise Exception( "The number of time instances could not be determined " "automatically. Give the number of time instances.") n = n_Z + 1 elif n_Z != 1 and n_Z+1 != n: raise Exception( "The number of time instances must match the number of last " "plates of parents:" "%d != %d+1" % (n, n_Z)) mu = cls._ensure_moments(mu, parent_moments[0]) D = mu.dims[0][0] K = Z.dims[0][0] M = n #N.get_moments()[0] # Check mu if mu.dims != ( (D,), (D,D) ): raise ValueError("First parent has wrong dimensionality") # Check Lambda Lambda = cls._ensure_moments(Lambda, parent_moments[1]) if Lambda.dims != ( (D,D), () ): raise ValueError("Second parent has wrong dimensionality") # Check B if B.dims != ( (D,), (D,D) ): raise ValueError("Third parent has wrong dimensionality") if len(B.plates) < 2 or B.plates[-2:] != (K,D): raise ValueError("Third parent should have a last plate " "equal to the dimensionality of the " "system.") if Z.dims != ( (K,), ): raise ValueError("Fourth parent has wrong dimensionality %s, " "should be %s" % (Z.dims, ( (K,), ))) if Z.plates[-1] != M-1: raise ValueError("The last plate of the fourth " "parent should have length equal to one or " "N-1, where N is the number of time " "instances.") # Check v if v.dims != ( (), () ): raise Exception("Fifth parent has wrong dimensionality") if len(v.plates) == 0 or v.plates[-1] != D: raise Exception("Fifth parent should have a last plate " "equal to the dimensionality of the " "system.") if (len(v.plates) >= 2 and v.plates[-2] != 1 and v.plates[-2] != M-1): raise ValueError("The second last plate of the fifth " "parent should have length equal to one or " "N-1 where N is the number of time " "instances.") dims = ( (M,D), (M,D,D), (M-1,D,D) ) distribution = SwitchingGaussianMarkovChainDistribution(M, D, K) parents = [mu, Lambda, B, Z, v] return (parents, kwargs, dims, cls._total_plates(kwargs.get('plates'), distribution.plates_from_parent(0, mu.plates), distribution.plates_from_parent(1, Lambda.plates), distribution.plates_from_parent(2, B.plates), distribution.plates_from_parent(3, Z.plates), distribution.plates_from_parent(4, v.plates)), distribution, cls._moments, parent_moments) class _MarkovChainToGaussian(Deterministic): """ Transform a Gaussian Markov chain node into a Gaussian node. This node is deterministic. """ _moments = GaussianMoments(1) _parent_moments = (GaussianMarkovChainMoments(),) def __init__(self, X, **kwargs): # Check for constant n if misc.is_numeric(X): X = Constant(GaussianMarkovChain)(X) # Make the time dimension a plate dimension... #plates = X.plates + (X.dims[0][0],) # ... and remove it from the variable dimensions dims = ( X.dims[0][-1:], X.dims[1][-2:] ) super().__init__(X, #plates=plates, dims=dims, **kwargs) def _plates_to_parent(self, index): """ Return the number of plates to the parent node. Normally, the parent sees the same number of plates as the node itself. However, now that one of the variable dimensions of the parents corresponds to a plate in this node, it is necessary to fix it here: the last plate is ignored when calculating plates with respect to the parent. Parent: Plates = (...) Dims = (N, ...) This node: Plates = (..., N) Dims = (...) """ return self.plates[:-1] def _plates_from_parent(self, index): # Sub-classes may want to overwrite this if they manipulate plates if index != 0: raise ValueError("Invalid parent index.") parent = self.parents[0] plates = parent.plates + (parent.dims[0][0],) return plates def _compute_moments(self, u): """ Transform the moments of a GMC to moments of a Gaussian. There is no need to worry about the plates and variable dimensions because the child node is free to interpret the axes as it pleases. However, the Gaussian moments contain only and but not , thus the last moment is discarded. """ # Get the moments from the parent Gaussian Markov Chain #u = self.parents[0].get_moments() #message_to_child() # Send only moments and but not return u[:2] def _compute_mask_to_parent(self, index, mask): # Remove the last axis of the mask if np.ndim(mask) >= 1: mask = np.any(mask, axis=-1) return mask @staticmethod def _compute_message_to_parent(index, m_children, *u_parents): """ Transform a message to a Gaussian into a message to a GMC. The messages to a Gaussian are almost correct, there are only two minor things to be done: 1) The last plate is changed into a variable/time dimension. Because a message mask is applied for plates only, the last axis of the mask must be applied to the message because the last plate is changed to a variable/time dimension. 2) Because the message does not contain part, we'll put the last/third message to None meaning that it is empty. Parameters ---------- index : int Index of the parent requesting the message. u_parents : list of list of ndarrays List of parents' moments. Returns ------- m : list of ndarrays Message as a list of arrays. mask : boolean ndarray Mask telling which plates should be taken into account. """ # Add the third empty message return [m_children[0], m_children[1], None] # Make use of the converter GaussianMarkovChainMoments.add_converter(GaussianMoments, _MarkovChainToGaussian) bayespy-0.4.0/bayespy/inference/vmp/nodes/beta.py0000644000175000017500000001042412535521223023517 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ A module for the beta distribution node """ import numpy as np import scipy.special as special from .dirichlet import (DirichletMoments, DirichletDistribution, Dirichlet) from .node import Moments, ensureparents class BetaMoments(DirichletMoments): """ Class for the moments of beta variables. """ def compute_fixed_moments(self, p): """ Compute the moments for a fixed value """ p = np.asanyarray(p)[...,None] * [1,-1] + [0,1] return super().compute_fixed_moments(p) def compute_dims_from_values(self, p): """ Return the shape of the moments for a fixed value. """ return ( (2,), ) class BetaDistribution(DirichletDistribution): """ Class for the VMP formulas of beta variables. Although the realizations are scalars (probability p), the moments is a two-dimensional vector: [log(p), log(1-p)]. """ def compute_message_to_parent(self, parent, index, u_self, u_alpha): """ Compute the message to a parent node. """ return super().compute_message_to_parent(parent, index, u_self, u_alpha) def compute_phi_from_parents(self, u_alpha, mask=True): """ Compute the natural parameter vector given parent moments. """ return super().compute_phi_from_parents(u_alpha, mask=mask) def compute_moments_and_cgf(self, phi, mask=True): """ Compute the moments and :math:`g(\phi)`. """ return super().compute_moments_and_cgf(phi, mask) def compute_cgf_from_parents(self, u_alpha): """ Compute :math:`\mathrm{E}_{q(p)}[g(p)]` """ return super().compute_cgf_from_parents(u_alpha) def compute_fixed_moments_and_f(self, p, mask=True): """ Compute the moments and :math:`f(x)` for a fixed value. """ p = np.asanyarray(p)[...,None] * [1,-1] + [0,1] return super().compute_fixed_moments_and_f(p, mask=mask) def random(self, *phi, plates=None): """ Draw a random sample from the distribution. """ p = super().random(*phi, plates=plates) return p[...,0] class Beta(Dirichlet): r""" Node for beta random variables. The node models a probability variable :math:`p \in [0,1]` as .. math:: p \sim \mathrm{Beta}(a, b) where :math:`a` and :math:`b` are prior counts for success and failure, respectively. Parameters ---------- alpha : (...,2)-shaped array Two-element vector containing :math:`a` and :math:`b` Examples -------- >>> import warnings >>> warnings.filterwarnings('ignore', category=RuntimeWarning) >>> from bayespy.nodes import Bernoulli, Beta >>> p = Beta([1e-3, 1e-3]) >>> z = Bernoulli(p, plates=(10,)) >>> z.observe([0, 1, 1, 1, 0, 1, 1, 1, 0, 1]) >>> p.update() >>> import bayespy.plot as bpplt >>> import numpy as np >>> bpplt.pdf(p, np.linspace(0, 1, num=100)) [] """ _moments = BetaMoments() _distribution = BetaDistribution() def __init__(self, alpha, **kwargs): """ Create beta node """ super().__init__(alpha, **kwargs) @classmethod @ensureparents def _constructor(cls, alpha, **kwargs): """ Constructs distribution and moments objects. """ D = alpha.dims[0][0] if D != 2: raise ValueError("Parent has wrong dimensionality. Must be a " "two-dimensional vector.") return super()._constructor(alpha, **kwargs) def __str__(self): """ Print the distribution using standard parameterization. """ a = self.phi[0][...,0] b = self.phi[0][...,1] return ("%s ~ Beta(a, b)\n" " a = \n" "%s\n" " b = \n" "%s\n" % (self.name, a, b)) bayespy-0.4.0/bayespy/inference/vmp/nodes/bernoulli.py0000644000175000017500000001041112535521260024574 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ A module for the Bernoulli distribution node """ import numpy as np from .binomial import (BinomialMoments, BinomialDistribution) from .expfamily import ExponentialFamily from .beta import BetaMoments from .node import Moments class BernoulliMoments(BinomialMoments): """ Class for the moments of Bernoulli variables. """ def __init__(self): super().__init__(1) class BernoulliDistribution(BinomialDistribution): """ Class for the VMP formulas of Bernoulli variables. """ def __init__(self): super().__init__(1) class Bernoulli(ExponentialFamily): r""" Node for Bernoulli random variables. The node models a binary random variable :math:`z \in \{0,1\}` with prior probability :math:`p \in [0,1]` for value one: .. math:: z \sim \mathrm{Bernoulli}(p). Parameters ---------- p : beta-like node Probability of a successful trial Examples -------- >>> import warnings >>> warnings.filterwarnings('ignore', category=RuntimeWarning) >>> from bayespy.nodes import Bernoulli, Beta >>> p = Beta([1e-3, 1e-3]) >>> z = Bernoulli(p, plates=(10,)) >>> z.observe([0, 1, 1, 1, 0, 1, 1, 1, 0, 1]) >>> p.update() >>> import bayespy.plot as bpplt >>> import numpy as np >>> bpplt.pdf(p, np.linspace(0, 1, num=100)) [] """ _moments = BernoulliMoments() _parent_moments = (BetaMoments(),) _distribution = BernoulliDistribution() def __init__(self, p, **kwargs): """ Create Bernoulli node. """ super().__init__(p, **kwargs) @classmethod def _constructor(cls, p, **kwargs): """ Constructs distribution and moments objects. """ p = cls._ensure_moments(p, cls._parent_moments[0]) parents = [p] return ( parents, kwargs, ( (), ), cls._total_plates(kwargs.get('plates'), cls._distribution.plates_from_parent(0, p.plates)), cls._distribution, cls._moments, cls._parent_moments) def __str__(self): """ Print the distribution using standard parameterization. """ p = 1 / (1 + np.exp(-self.phi[0])) return ("%s ~ Bernoulli(p)\n" " p = \n" "%s\n" % (self.name, p)) from .deterministic import Deterministic from .categorical import Categorical, CategoricalMoments class CategoricalToBernoulli(Deterministic): """ A node for converting 2-class categorical moments to Bernoulli moments. """ def __init__(self, Z, **kwargs): """ Create a categorical MC moments to categorical moments conversion node. """ # Convert parent to proper type. Z must be a node. Z = Z._convert(CategoricalMoments) K = Z.dims[0][-1] if K != 2: raise Moments.NoConverterError("Only 2-class categorical can be converted to " "Bernoulli") dims = ( (), ) self._moments = BernoulliMoments() self._parent_moments = (CategoricalMoments(2),) super().__init__(Z, dims=dims, **kwargs) def _compute_moments(self, u_Z): """ Compute the moments given the moments of the parents. """ u0 = u_Z[0][...,0] u = [u0] return u def _compute_message_to_parent(self, index, m, u_Z): """ Compute the message to a parent. """ if index == 0: m0 = np.concatenate([m[0][...,None], np.zeros(np.shape(m[0]))[...,None]], axis=-1) return [m0] else: raise ValueError("Incorrect parent index") # Make use of the conversion node CategoricalMoments.add_converter(BernoulliMoments, CategoricalToBernoulli) bayespy-0.4.0/bayespy/inference/vmp/nodes/gp.py0000644000175000017500000005560712533543154023233 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2012 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import itertools import numpy as np import scipy as sp import scipy.linalg.decomp_cholesky as decomp import scipy.linalg as linalg import scipy.special as special import matplotlib.pyplot as plt import time import profile import scipy.spatial.distance as distance from .node import Node from .stochastic import Stochastic from bayespy.utils.misc import * # Computes log probability density function of the Gaussian # distribution def gaussian_logpdf(y_invcov_y, y_invcov_mu, mu_invcov_mu, logdetcov, D): return (-0.5*D*np.log(2*np.pi) -0.5*logdetcov -0.5*y_invcov_y +y_invcov_mu -0.5*mu_invcov_mu) # m prior mean function # k prior covariance function # x data inputs # z processed data outputs (z = inv(Cov) * (y-m(x))) # U data covariance Cholesky factor def gp_posterior_moment_function(m, k, x, y, noise=None): # Prior mu = m(x)[0] K = k(x,x)[0] if noise != None: K += noise #print('hereiamagain') #print(K) # Compute posterior GP N = len(y) if N == 0: U = None z = None else: U = chol(K) z = chol_solve(U, y-mu) def get_moments(xh, covariance=1, mean=True): (kh,) = k(x, xh) # Function for computing posterior moments if mean: # Mean vector mh = m(xh) if z != None: mh += np.dot(kh.T, z) else: mh = None if covariance: if covariance == 1: # Variance vector khh = k(xh) if U != None: khh -= np.einsum('i...,i...', kh, chol_solve(U, kh)) elif covariance == 2: # Full covariance matrix khh = k(xh,xh) if U != None: khh -= np.dot(kh.T, chol_solve(U,kh)) else: khh = None return [mh, khh] return get_moments # m prior mean function # k prior covariance function # x data inputs # z processed data outputs (z = inv(Cov) * (y-m(x))) # U data covariance Cholesky factor ## def gp_multi_posterior_moment_function(m, k, x, y, noise=None): ## # Prior ## mu = m(x)[0] ## K = k(x,x)[0] ## if noise != None: ## K += noise ## #print('hereiamagain') ## #print(K) ## # Compute posterior GP ## N = len(y) ## if N == 0: ## U = None ## z = None ## else: ## U = chol(K) ## z = chol_solve(U, y-mu) ## def get_moments(xh, covariance=1, mean=True): ## (kh,) = k(x, xh) ## # Function for computing posterior moments ## if mean: ## # Mean vector ## mh = m(xh) ## if z != None: ## mh += np.dot(kh.T, z) ## else: ## mh = None ## if covariance: ## if covariance == 1: ## # Variance vector ## khh = k(xh) ## if U != None: ## khh -= np.einsum('i...,i...', kh, chol_solve(U, kh)) ## elif covariance == 2: ## # Full covariance matrix ## khh = k(xh,xh) ## if U != None: ## khh -= np.dot(kh.T, chol_solve(U,kh)) ## else: ## khh = None ## return [mh, khh] ## return get_moments def gp_cov_se(D2, overwrite=False): if overwrite: K = D2 K *= -0.5 np.exp(K, out=K) else: K = np.exp(-0.5*D2) return K def gp_cov_delta(N): return np.identity(N) def squared_distance(x1, x2): # Reshape arrays to 2-D arrays sh1 = np.shape(x1)[:-1] sh2 = np.shape(x2)[:-1] d = np.shape(x1)[-1] x1 = np.reshape(x1, (-1,d)) x2 = np.reshape(x2, (-1,d)) # Compute squared Euclidean distance D2 = distance.cdist(x1, x2, metric='sqeuclidean') # Reshape the result D2 = np.reshape(D2, sh1 + sh2) return D2 # General rule for the parameters for covariance functions: # # (value, [ [dvalue1, ...], [dvalue2, ...], [dvalue3, ...], ...]) # # For instance, # # k = covfunc_se((1.0, []), (15, [ [1,update_grad] ])) # K = k((x1, [ [dx1,update_grad] ]), (x2, [])) # # Plain values are converted as: # value -> (value, []) def gp_standardize_input(x): if np.ndim(x) == 0: x = add_trailing_axes(x, 2) elif np.ndim(x) == 1: x = add_trailing_axes(x, 1) return x def gp_preprocess_inputs(*args): args = list(args) if len(args) < 1 or len(args) > 2: raise Exception("Number of inputs must be one or two") if len(args) == 2: if args[0] is args[1]: args[0] = gp_standardize_input(args[0]) args[1] = args[0] else: args[1] = gp_standardize_input(args[1]) args[0] = gp_standardize_input(args[0]) else: args[0] = gp_standardize_input(args[0]) return args def covfunc_delta(theta, *inputs, gradient=False): amplitude = theta[0] if gradient: gradient_amplitude = gradient[0] else: gradient_amplitude = [] inputs = gp_preprocess_inputs(*inputs) # Compute distance and covariance matrix if len(inputs) == 1: # Only variance vector asked x = inputs[0] K = np.ones(np.shape(x)[:-1]) * amplitude**2 else: # Full covariance matrix asked x1 = inputs[0] x2 = inputs[1] # Number of inputs x1 N1 = np.shape(x1)[-2] # x1 == x2? if x1 is x2: delta = True # Delta covariance K = gp_cov_delta(N1) * amplitude**2 else: delta = False # Number of inputs x2 N2 = np.shape(x2)[-2] # Zero covariance K = np.zeros((N1,N2)) # Gradient w.r.t. amplitude if gradient: for ind in range(len(gradient_amplitude)): gradient_amplitude[ind] = K * (2 * gradient_amplitude[ind] / amplitude) if gradient: return (K, gradient) else: return K def covfunc_se(theta, *inputs, gradient=False): amplitude = theta[0] lengthscale = theta[1] ## print('in se') ## print(amplitude) ## print(lengthscale) if gradient: gradient_amplitude = gradient[0] gradient_lengthscale = gradient[1] else: gradient_amplitude = [] gradient_lengthscale = [] inputs = gp_preprocess_inputs(*inputs) # Compute covariance matrix if len(inputs) == 1: x = inputs[0] # Compute variance vector K = np.ones(np.shape(x)[:-1]) K *= amplitude**2 # Compute gradient w.r.t. lengthscale for ind in range(len(gradient_lengthscale)): gradient_lengthscale[ind] = np.zeros(np.shape(x)[:-1]) else: x1 = inputs[0] / (lengthscale) x2 = inputs[1] / (lengthscale) # Compute distance matrix K = squared_distance(x1, x2) # Compute gradient partly if gradient: for ind in range(len(gradient_lengthscale)): gradient_lengthscale[ind] = K * ((lengthscale**-1) * gradient_lengthscale[ind]) # Compute covariance matrix gp_cov_se(K, overwrite=True) K *= amplitude**2 # Compute gradient w.r.t. lengthscale if gradient: for ind in range(len(gradient_lengthscale)): gradient_lengthscale[ind] *= K # Gradient w.r.t. amplitude if gradient: for ind in range(len(gradient_amplitude)): gradient_amplitude[ind] = K * (2 * gradient_amplitude[ind] / amplitude) # Return values if gradient: return (K, gradient) else: return K class NodeCovarianceFunction(Node): def __init__(self, covfunc, *args, **kwargs): self.covfunc = covfunc params = list(args) for i in range(len(args)): # Check constant parameters if is_numeric(args[i]): params[i] = NodeConstant([np.asanyarray(args[i])], dims=[np.shape(args[i])]) # TODO: Parameters could be constant functions? :) Node.__init__(self, *params, dims=[(np.inf, np.inf)], **kwargs) def message_to_child(self, gradient=False): params = [parent.message_to_child(gradient=gradient) for parent in self.parents] return self.covariance_function(*params) def covariance_function(self, *params): params = list(params) gradient_params = list() for ind in range(len(params)): if isinstance(params[ind], tuple): gradient_params.append(params[ind][1]) params[ind] = params[ind][0][0] else: gradient_params.append([]) params[ind] = params[ind][0] def cov(*inputs, gradient=False): if gradient: grads = [[grad[0] for grad in gradient_params[ind]] for ind in range(len(gradient_params))] (K, dK) = self.covfunc(params, *inputs, gradient=grads) for ind in range(len(dK)): for (grad, dk) in zip(gradient_params[ind], dK[ind]): grad[0] = dk K = [K] dK = [] for grad in gradient_params: dK += grad return (K, dK) else: K = self.covfunc(params, *inputs, gradient=False) return [K] return cov class NodeCovarianceFunctionSum(NodeCovarianceFunction): def __init__(self, *args, **kwargs): NodeCovarianceFunction.__init__(self, None, *args, **kwargs) def covariance_function(self, *covfuncs): def cov(*inputs, gradient=False): K_sum = 0 if gradient: dK_sum = list() for k in covfuncs: if gradient: (K, dK) = k(*inputs, gradient=gradient) dK_sum += dK else: K = k(*inputs, gradient=gradient) K_sum += K[0] if gradient: return ([K_sum], dK_sum) else: return [K_sum] return cov class NodeCovarianceFunctionDelta(NodeCovarianceFunction): def __init__(self, amplitude, **kwargs): NodeCovarianceFunction.__init__(self, covfunc_delta, amplitude, **kwargs) class NodeCovarianceFunctionSquaredExponential(NodeCovarianceFunction): def __init__(self, amplitude, lengthscale, **kwargs): NodeCovarianceFunction.__init__(self, covfunc_se, amplitude, lengthscale, **kwargs) class NodeMultiCovarianceFunction(NodeCovarianceFunction): def __init__(self, *args, **kwargs): NodeCovarianceFunction.__init__(self, None, *args, **kwargs) def covfunc(self, *covfuncs): def cov(*inputs, gradient=False): K_sum = 0 if gradient: dK_sum = list() for k in covfuncs: if gradient: (K, dK) = k(*inputs, gradient=gradient) dK_sum += dK else: K = k(*inputs, gradient=gradient) K_sum += K[0] if gradient: return ([K_sum], dK_sum) else: return [K_sum] return cov class NodeConstantGaussianProcess(Node): def __init__(self, f, **kwargs): self.f = f Node.__init__(self, dims=[(np.inf,)], **kwargs) def message_to_child(self, gradient=False): # Wrapper def func(x, gradient=False): if gradient: return ([self.f(x)], []) else: return [self.f(x)] return func # At least for now, simplify this GP node such that a GP is either # observed or latent. If it is observed, it doesn't take messages from # children, actually, it should not even have children! #class NodeMultiGaussianProcess(NodeVariable): class NodeMultiGaussianProcess(Stochastic): def __init__(self, m, k, **kwargs): self.x = [] self.f = [] # By default, posterior == prior self.m = m self.k = k # Ignore plates NodeVariable.__init__(self, m, k, plates=(), dims=[(np.inf,), (np.inf,np.inf)], **kwargs) def message_to_parent(self, index): if index == 0: k = self.parents[1].message_to_child()[0] K = k(self.x, self.x) return [self.x, self.mu, K] if index == 1: raise Exception("not implemented yet") def message_to_child(self): if self.observed: raise Exception("Observable GP should not have children.") return self.u def get_parameters(self): return self.u def observe(self, x, f): if np.ndim(x) == 1: if np.shape(f) != np.shape(x): print(np.shape(f)) print(np.shape(x)) raise Exception("Number of inputs and function values do not match") elif np.shape(f) != np.shape(x)[:-1]: print(np.shape(f)) print(np.shape(x)) raise Exception("Number of inputs and function values do not match") self.observed = True self.x = x self.f = f ## self.x_obs = x ## self.f_obs = f # You might want: # - mean for x # - covariance (and mean) for x # - variance (and mean) for x # - i.e., mean and/or (co)variance for x # - covariance for x1 and x2 def lower_bound_contribution(self, gradient=False): m = self.parents[0].message_to_child(gradient=gradient) k = self.parents[1].message_to_child(gradient=gradient) ## m = self.parents[0].message_to_child(gradient=gradient)[0] ## k = self.parents[1].message_to_child(gradient=gradient)[0] # Prior if gradient: (mu, dmus) = m(self.x, gradient=True) (K, dKs) = k(self.x, self.x, gradient=True) else: mu = m(self.x) K = k(self.x, self.x) dmus = [] dKs = [] mu = mu[0] K = K[0] # Log pdf if self.observed: # Vector of f-mu f0 = np.vstack([(f-m) for (f,m) in zip(self.f,mu)]) # Full covariance matrix K_full = np.bmat(K) try: U = chol(K_full) except linalg.LinAlgError: print('non positive definite, return -inf') return -np.inf z = chol_solve(U, f0) #print(K) L = gaussian_logpdf(np.dot(f0, z), 0, 0, logdet_chol(U), np.size(self.f)) for (dmu, func) in dmus: # Derivative w.r.t. mean vector d = -np.sum(z) # Send the derivative message func += d #func(d) for (dK, func) in dKs: # Compute derivative w.r.t. covariance matrix d = 0.5 * (np.dot(z, np.dot(dK, z)) - np.trace(chol_solve(U, dK))) # Send the derivative message #print('add gradient') #func += d func(d) else: raise Exception('Not implemented yet') return L ## Let f1 be observed and f2 latent function values. # Compute #L = gaussian_logpdf(sum_product(np.outer(self.f,self.f) + self.Cov, # Compute def update(self): # Messages from parents m = self.parents[0].message_to_child() k = self.parents[1].message_to_child() ## m = self.parents[0].message_to_child()[0] ## k = self.parents[1].message_to_child()[0] if self.observed: # Observations of this node self.u = gp_posterior_moment_function(m, k, self.x, self.f) else: x = np.array([]) y = np.array([]) # Messages from children for (child,index) in self.children: (msg, mask) = child.message_to_parent(index) # Ignoring masks and plates.. # m[0] is the inputs x = np.concatenate((x, msg[0]), axis=-2) # m[1] is the observations y = np.concatenate((y, msg[1])) # m[2] is the covariance matrix V = linalg.block_diag(V, msg[2]) self.u = gp_posterior_moment_function(m, k, x, y, covariance=V) self.x = x self.f = y class NodeGaussianProcess(Stochastic): #class NodeGaussianProcess(NodeVariable): def __init__(self, m, k, **kwargs): self.x = np.array([]) self.f = np.array([]) ## self.x_obs = np.zeros((0,1)) ## self.f_obs = np.zeros((0,)) # By default, posterior == prior self.m = m self.k = k # Ignore plates NodeVariable.__init__(self, m, k, plates=(), dims=[(np.inf,), (np.inf,np.inf)], **kwargs) def message_to_parent(self, index): if index == 0: k = self.parents[1].message_to_child()[0] K = k(self.x, self.x) return [self.x, self.mu, K] if index == 1: raise Exception("not implemented yet") def message_to_child(self): if self.observed: raise Exception("Observable GP should not have children.") return self.u def get_parameters(self): return self.u def observe(self, x, f): if np.ndim(x) == 1: if np.shape(f) != np.shape(x): print(np.shape(f)) print(np.shape(x)) raise Exception("Number of inputs and function values do not match") elif np.shape(f) != np.shape(x)[:-1]: print(np.shape(f)) print(np.shape(x)) raise Exception("Number of inputs and function values do not match") self.observed = True self.x = x self.f = f ## self.x_obs = x ## self.f_obs = f # You might want: # - mean for x # - covariance (and mean) for x # - variance (and mean) for x # - i.e., mean and/or (co)variance for x # - covariance for x1 and x2 def lower_bound_contribution(self, gradient=False): m = self.parents[0].message_to_child(gradient=gradient) k = self.parents[1].message_to_child(gradient=gradient) ## m = self.parents[0].message_to_child(gradient=gradient)[0] ## k = self.parents[1].message_to_child(gradient=gradient)[0] # Prior if gradient: (mu, dmus) = m(self.x, gradient=True) (K, dKs) = k(self.x, self.x, gradient=True) else: mu = m(self.x) K = k(self.x, self.x) dmus = [] dKs = [] mu = mu[0] K = K[0] # Log pdf if self.observed: f0 = self.f - mu #print('hereiam') #print(K) try: U = chol(K) except linalg.LinAlgError: print('non positive definite, return -inf') return -np.inf z = chol_solve(U, f0) #print(K) L = gaussian_logpdf(np.dot(f0, z), 0, 0, logdet_chol(U), np.size(self.f)) for (dmu, func) in dmus: # Derivative w.r.t. mean vector d = -np.sum(z) # Send the derivative message func += d #func(d) for (dK, func) in dKs: # Compute derivative w.r.t. covariance matrix d = 0.5 * (np.dot(z, np.dot(dK, z)) - np.trace(chol_solve(U, dK))) # Send the derivative message #print('add gradient') #func += d func(d) else: raise Exception('Not implemented yet') return L ## Let f1 be observed and f2 latent function values. # Compute #L = gaussian_logpdf(sum_product(np.outer(self.f,self.f) + self.Cov, # Compute def update(self): # Messages from parents m = self.parents[0].message_to_child() k = self.parents[1].message_to_child() ## m = self.parents[0].message_to_child()[0] ## k = self.parents[1].message_to_child()[0] if self.observed: # Observations of this node self.u = gp_posterior_moment_function(m, k, self.x, self.f) else: x = np.array([]) y = np.array([]) # Messages from children for (child,index) in self.children: (msg, mask) = child.message_to_parent(index) # Ignoring masks and plates.. # m[0] is the inputs x = np.concatenate((x, msg[0]), axis=-2) # m[1] is the observations y = np.concatenate((y, msg[1])) # m[2] is the covariance matrix V = linalg.block_diag(V, msg[2]) self.u = gp_posterior_moment_function(m, k, x, y, covariance=V) self.x = x self.f = y bayespy-0.4.0/bayespy/inference/vmp/nodes/multinomial.py0000644000175000017500000001506412533543154025150 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Module for the multinomial distribution node. """ import numpy as np from scipy import special from .expfamily import ExponentialFamily from .expfamily import ExponentialFamilyDistribution from .expfamily import useconstructor from .dirichlet import Dirichlet, DirichletMoments from .node import Moments, ensureparents from bayespy.utils import random from bayespy.utils import misc class MultinomialMoments(Moments): """ Class for the moments of multinomial variables. """ def compute_fixed_moments(self, x): """ Compute the moments for a fixed value `x` must be a vector of counts. """ # Check that counts are valid x = np.asanyarray(x) if not misc.isinteger(x): raise ValueError("Counts must be integer") if np.any(x < 0): raise ValueError("Counts must be non-negative") # Moments is just the counts vector u0 = x.copy() return [u0] def compute_dims_from_values(self, x): """ Return the shape of the moments for a fixed value. """ D = np.shape(x)[-1] return ( (D,), ) class MultinomialDistribution(ExponentialFamilyDistribution): """ Class for the VMP formulas of multinomial variables. """ def __init__(self, trials): """ Create VMP formula node for a multinomial variable `trials` is the total number of trials. """ if not misc.isinteger(trials): raise ValueError("Number of trials must be integer") if np.any(trials < 0): raise ValueError("Number of trials must be non-negative") self.N = trials super().__init__() def compute_message_to_parent(self, parent, index, u, u_p): """ Compute the message to a parent node. """ if index == 0: return [ u[0].copy() ] else: raise ValueError("Index out of bounds") def compute_phi_from_parents(self, u_p, mask=True): """ Compute the natural parameter vector given parent moments. """ logp = u_p[0] return [logp] def compute_moments_and_cgf(self, phi, mask=True): """ Compute the moments and :math:`g(\phi)`. """ # Compute the normalized probabilities in a numerically stable way logsum_p = misc.logsumexp(phi[0], axis=-1, keepdims=True) logp = phi[0] - logsum_p p = np.exp(logp) # Because of small numerical inaccuracy, normalize the probabilities # again for more accurate results N = np.expand_dims(self.N, -1) u0 = N * p / np.sum(p, axis=-1, keepdims=True) u = [u0] g = -np.squeeze(N * logsum_p, axis=-1) return (u, g) def compute_cgf_from_parents(self, u_p): """ Compute :math:`\mathrm{E}_{q(p)}[g(p)]` """ return 0 def compute_fixed_moments_and_f(self, x, mask=True): """ Compute the moments and :math:`f(x)` for a fixed value. """ # Check that counts are valid x = np.asanyarray(x) if not misc.isinteger(x): raise ValueError("Counts must be integers") if np.any(x < 0): raise ValueError("Counts must be non-negative") if np.any(np.sum(x, axis=-1) != self.N): raise ValueError("Counts must sum to the number of trials") # Moments is just the counts vector u0 = x.copy() u = [u0] f = special.gammaln(self.N+1) - np.sum(special.gammaln(x+1), axis=-1) return (u, f) def random(self, *phi): """ Draw a random sample from the distribution. """ raise NotImplementedError() class Multinomial(ExponentialFamily): r""" Node for multinomial random variables. Assume there are :math:`K` categories and :math:`N` trials each of which leads a success for exactly one of the categories. Given the probabilities :math:`p_0,\ldots,p_{K-1}` for the categories, multinomial distribution is gives the probability of any combination of numbers of successes for the categories. The node models the number of successes :math:`x_k \in \{0, \ldots, n\}` in :math:`n` trials with probability :math:`p_k` for success in :math:`K` categories. .. math:: \mathrm{Multinomial}(\mathbf{x}| N, \mathbf{p}) = \frac{N!}{x_0!\cdots x_{K-1}!} p_0^{x_0} \cdots p_{K-1}^{x_{K-1}} Parameters ---------- n : scalar or array :math:`N`, number of trials p : Dirichlet-like node or (...,K)-array :math:`\mathbf{p}`, probabilities of successes for the categories See also -------- Dirichlet, Binomial, Categorical """ _moments = MultinomialMoments() _parent_moments = (DirichletMoments(),) def __init__(self, n, p, **kwargs): """ Create Multinomial node. """ super().__init__(n, p, **kwargs) @classmethod def _constructor(cls, n, p, **kwargs): """ Constructs distribution and moments objects. This method is called if useconstructor decorator is used for __init__. Becase the distribution and moments object depend on the number of categories, that is, they depend on the parent node, this method can be used to construct those objects. """ # Get the number of categories p = cls._ensure_moments(p, cls._parent_moments[0]) D = p.dims[0][0] parents = [p] distribution = MultinomialDistribution(n) return (parents, kwargs, ( (D,), ), cls._total_plates(kwargs.get('plates'), distribution.plates_from_parent(0, p.plates), np.shape(n)), distribution, cls._moments, cls._parent_moments) def __str__(self): """ Print the distribution using standard parameterization. """ logsum_p = misc.logsumexp(self.phi[0], axis=-1, keepdims=True) p = np.exp(self.phi[0] - logsum_p) p /= np.sum(p, axis=-1, keepdims=True) return ("%s ~ Multinomial(p)\n" " p = \n" "%s\n" % (self.name, p)) bayespy-0.4.0/bayespy/inference/vmp/nodes/constant.py0000644000175000017500000000457612555653143024461 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2012,2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import numpy as np from .node import Node, Moments class Constant(Node): r""" Node for presenting constant values. The node wraps arrays into proper node type. """ def __init__(self, moments, x, **kwargs): if not isinstance(moments, Moments) and issubclass(moments, Moments): raise ValueError("Give moments as an object instance instead of a class") self._moments = moments x = np.asanyarray(x) # Compute moments self.u = self._moments.compute_fixed_moments(x) # Dimensions of the moments dims = self._moments.compute_dims_from_values(x) # Resolve plates D = len(dims[0]) if D > 0: plates = np.shape(self.u[0])[:-D] else: plates = np.shape(self.u[0]) # Parent constructor super().__init__(dims=dims, plates=plates, **kwargs) def _get_id_list(self): """ Returns the stochastic ID list. This method is used to check that same stochastic nodes are not direct parents of a node several times. It is only valid if there are intermediate stochastic nodes. To put it another way: each ID corresponds to one factor q(..) in the posterior approximation. Different IDs mean different factors, thus they mean independence. The parents must have independent factors. Stochastic nodes should return their unique ID. Deterministic nodes should return the IDs of their parents. Constant nodes should return empty list of IDs. """ return [] def get_moments(self): return self.u def set_value(self, x): x = np.asanyarray(x) shapes = [np.shape(ui) for ui in self.u] self.u = self._moments.compute_fixed_moments(x) for (i, shape) in enumerate(shapes): if np.shape(self.u[i]) != shape: raise ValueError("Incorrect shape for the array") def lower_bound_contribution(self, gradient=False, **kwargs): # Deterministic functions are delta distributions so the lower bound # contribuion is zero. return 0 bayespy-0.4.0/bayespy/inference/vmp/nodes/GaussianProcesses.py0000644000175000017500000006254112533543154026261 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2012 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import itertools import numpy as np #import scipy as sp #import scipy.linalg.decomp_cholesky as decomp import scipy.linalg as linalg #import scipy.special as special #import matplotlib.pyplot as plt #import time #import profile #import scipy.spatial.distance as distance import scipy.sparse as sp from bayespy.utils import misc as utils from . import node as EF from . import CovarianceFunctions as CF class CovarianceMatrix: def cholesky(self): pass def multiply(A, B): return np.multiply(A,B) # m prior mean function # k prior covariance function # x data inputs # z processed data outputs (z = inv(Cov) * (y-m(x))) # U data covariance Cholesky factor def gp_posterior_moment_function(m, k, x, y, k_sparse=None, pseudoinputs=None, noise=None): # Prior # FIXME: We are ignoring the covariance of mu now.. mu = m(x)[0] ## if np.ndim(mu) == 1: ## mu = np.asmatrix(mu).T ## else: ## mu = np.asmatrix(mu) K_noise = None if noise != None: if K_noise is None: K_noise = noise else: K_noise += noise if k_sparse != None: if K_noise is None: K_noise = k_sparse(x,x)[0] else: K_noise += k_sparse(x,x)[0] if pseudoinputs != None: p = pseudoinputs #print('in pseudostuff') #print(K_noise) #print(np.shape(K_noise)) K_pp = k(p,p)[0] K_xp = k(x,p)[0] U = utils.chol(K_noise) # Compute Lambda Lambda = K_pp + np.dot(K_xp.T, utils.chol_solve(U, K_xp)) U_lambda = utils.chol(Lambda) # Compute statistics for posterior predictions #print(np.shape(U_lambda)) #print(np.shape(y)) z = utils.chol_solve(U_lambda, np.dot(K_xp.T, utils.chol_solve(U, y - mu))) U = utils.chol(K_pp) # Now we can forget the location of the observations and # consider only the pseudoinputs when predicting. x = p else: K = K_noise if K is None: K = k(x,x)[0] else: try: K += k(x,x)[0] except: K = K + k(x,x)[0] # Compute posterior GP N = len(y) U = None z = None if N > 0: U = utils.chol(K) z = utils.chol_solve(U, y-mu) def get_moments(h, covariance=1, mean=True): K_xh = k(x, h)[0] if k_sparse != None: try: # This may not work, for instance, if either one is a # sparse matrix. K_xh += k_sparse(x, h)[0] except: K_xh = K_xh + k_sparse(x, h)[0] # NumPy has problems when mixing matrices and arrays. # Matrices may appear, for instance, when you sum an array and # a sparse matrix. Make sure the result is either an array or # a sparse matrix (not dense matrix!), because matrix objects # cause lots of problems: # # array.dot(array) = array # matrix.dot(array) = matrix # sparse.dot(array) = array if not sp.issparse(K_xh): K_xh = np.asarray(K_xh) # Function for computing posterior moments if mean: # Mean vector # FIXME: Ignoring the covariance of prior mu m_h = m(h)[0] if z != None: m_h += K_xh.T.dot(z) else: m_h = None # Compute (co)variance matrix/vector if covariance: if covariance == 1: ## Compute variance vector k_h = k(h)[0] if k_sparse != None: k_h += k_sparse(h)[0] if U != None: if isinstance(K_xh, np.ndarray): k_h -= np.einsum('i...,i...', K_xh, utils.chol_solve(U, K_xh)) else: # TODO: This isn't very efficient way, but # einsum doesn't work for sparse matrices.. # This may consume A LOT of memory for sparse # matrices. k_h -= np.asarray(K_xh.multiply(utils.chol_solve(U, K_xh))).sum(axis=0) if pseudoinputs != None: if isinstance(K_xh, np.ndarray): k_h += np.einsum('i...,i...', K_xh, utils.chol_solve(U_lambda, K_xh)) else: # TODO: This isn't very efficient way, but # einsum doesn't work for sparse matrices.. # This may consume A LOT of memory for sparse # matrices. k_h += np.asarray(K_xh.multiply(utils.chol_solve(U_lambda, K_xh))).sum(axis=0) # Ensure non-negative variances k_h[k_h<0] = 0 return (m_h, k_h) elif covariance == 2: ## Compute full covariance matrix K_hh = k(h,h)[0] if k_sparse != None: K_hh += k_sparse(h)[0] if U != None: K_hh -= K_xh.T.dot(utils.chol_solve(U,K_xh)) #K_hh -= np.dot(K_xh.T, utils.chol_solve(U,K_xh)) if pseudoinputs != None: K_hh += K_xh.T.dot(utils.chol_solve(U_lambda, K_xh)) #K_hh += np.dot(K_xh.T, utils.chol_solve(U_lambda, K_xh)) return (m_h, K_hh) else: return (m_h, None) return get_moments # Constant function using GP mean protocol class Constant(EF.Node): def __init__(self, f, **kwargs): self.f = f EF.Node.__init__(self, dims=[(np.inf,)], **kwargs) def message_to_child(self, gradient=False): # Wrapper def func(x, gradient=False): if gradient: return ([self.f(x), None], []) else: return [self.f(x), None] return func #class MultiDimensional(EF.NodeVariable): # """ A multi-dimensional Gaussian process f(x). """ ## class ToGaussian(EF.NodeVariable): ## """ Deterministic node which transform a Gaussian process into ## finite-dimensional Gaussian variable. """ ## def __init__(self, f, x, **kwargs): ## EF.NodeVariable.__init__(self, ## f, ## x, ## plates= ## dims= # Deterministic node for creating a set of GPs which can be used as a # mean function to a general GP node. class Multiple(EF.Node): def __init__(self, GPs, **kwargs): # Ignore plates EF.NodeVariable.__init__(self, *GPs, plates=(), dims=[(np.inf,), (np.inf,np.inf)], **kwargs) def message_to_parent(self, index): raise Exception("not implemented yet") def message_to_child(self, gradient=False): u = [parent.message_to_child() for parent in self.parents] def get_moments(xh, **kwargs): mh_all = [] khh_all = [] for i in range(len(self.parents)): xi = np.array(xh[i]) #print(xi) #print(np.shape(xi)) #print(xi) # FIXME: We are ignoring the covariance of mu now.. if gradient: ((mh, khh), dm) = u[i](xi, **kwargs) else: (mh, khh) = u[i](xi, **kwargs) #mh = u[i](xi, **kwargs)[0] #print(mh) #print(mh_all) ## print(mh) ## print(khh) ## print(np.shape(mh)) mh_all = np.concatenate([mh_all, mh]) #print(np.shape(mh_all)) if khh != None: print(khh) raise Exception('Not implemented yet for covariances') #khh_all = np.concatenate([khh_all, khh]) # FIXME: Compute gradients! if gradient: return ([mh_all, khh_all], []) else: return [mh_all, khh_all] #return [mh_all, khh_all] return get_moments # Gaussian process distribution class GaussianProcess(EF.Node): def __init__(self, m, k, k_sparse=None, pseudoinputs=None, **kwargs): self.x = np.array([]) self.f = np.array([]) ## self.x_obs = np.zeros((0,1)) ## self.f_obs = np.zeros((0,)) if pseudoinputs != None: pseudoinputs = EF.NodeConstant([pseudoinputs], dims=[np.shape(pseudoinputs)]) # By default, posterior == prior self.m = None #m self.k = None #k if isinstance(k, list) and isinstance(m, list): if len(k) != len(m): raise Exception('The number of mean and covariance functions must be equal.') k = CF.Multiple(k) m = Multiple(m) elif isinstance(k, list): D = len(k) k = CF.Multiple(k) m = Multiple(D*[m]) elif isinstance(m, list): D = len(m) k = CF.Multiple(D*[k]) m = Multiple(m) # Ignore plates EF.NodeVariable.__init__(self, m, k, k_sparse, pseudoinputs, plates=(), dims=[(np.inf,), (np.inf,np.inf)], **kwargs) def __call__(self, x, covariance=None): if not covariance: return self.u(x, covariance=False)[0] elif covariance.lower() == 'vector': return self.u(x, covariance=1) elif covariance.lower() == 'matrix': return self.u(x, covariance=2) else: raise Exception("Unknown covariance type requested") def message_to_parent(self, index): if index == 0: k = self.parents[1].message_to_child()[0] K = k(self.x, self.x) return [self.x, self.mu, K] if index == 1: raise Exception("not implemented yet") def message_to_child(self): if self.observed: raise Exception("Observable GP should not have children.") return self.u def get_parameters(self): return self.u def observe(self, x, f): self.observed = True self.x = x self.f = f ## if np.ndim(f) == 1: ## self.f = np.asmatrix(f).T ## else: ## self.f = np.asmatrix(f) # You might want: # - mean for x # - covariance (and mean) for x # - variance (and mean) for x # - i.e., mean and/or (co)variance for x # - covariance for x1 and x2 def lower_bound_contribution(self, gradient=False): # Get moment functions from parents m = self.parents[0].message_to_child(gradient=gradient) k = self.parents[1].message_to_child(gradient=gradient) if self.parents[2]: k_sparse = self.parents[2].message_to_child(gradient=gradient) else: k_sparse = None if self.parents[3]: pseudoinputs = self.parents[3].message_to_child(gradient=gradient) #pseudoinputs = self.parents[3].message_to_child(gradient=gradient)[0] else: pseudoinputs = None ## m = self.parents[0].message_to_child(gradient=gradient)[0] ## k = self.parents[1].message_to_child(gradient=gradient)[0] # Compute the parameters (covariance matrices etc) using # parents' moment functions DKs_xx = [] DKd_xx = [] DKd_xp = [] DKd_pp = [] Dxp = [] Dmu = [] if gradient: # FIXME: We are ignoring the covariance of mu now.. ((mu, _), Dmu) = m(self.x, gradient=True) ## if k_sparse: ## ((Ks_xx,), DKs_xx) = k_sparse(self.x, self.x, gradient=True) if pseudoinputs: ((Ks_xx,), DKs_xx) = k_sparse(self.x, self.x, gradient=True) ((xp,), Dxp) = pseudoinputs ((Kd_pp,), DKd_pp) = k(xp,xp, gradient=True) ((Kd_xp,), DKd_xp) = k(self.x, xp, gradient=True) else: ((K_xx,), DKd_xx) = k(self.x, self.x, gradient=True) if k_sparse: ((Ks_xx,), DKs_xx) = k_sparse(self.x, self.x, gradient=True) try: K_xx += Ks_xx except: K_xx = K_xx + Ks_xx else: # FIXME: We are ignoring the covariance of mu now.. (mu, _) = m(self.x) ## if k_sparse: ## (Ks_xx,) = k_sparse(self.x, self.x) if pseudoinputs: (Ks_xx,) = k_sparse(self.x, self.x) (xp,) = pseudoinputs (Kd_pp,) = k(xp, xp) (Kd_xp,) = k(self.x, xp) else: (K_xx,) = k(self.x, self.x) if k_sparse: (Ks_xx,) = k_sparse(self.x, self.x) try: K_xx += Ks_xx except: K_xx = K_xx + Ks_xx mu = mu[0] #K = K[0] # Log pdf if self.observed: ## Log pdf for directly observed GP f0 = self.f - mu #print('hereiam') #print(K) if pseudoinputs: ## Pseudo-input approximation # Decompose the full-rank sparse/noise covariance matrix try: Us_xx = utils.cholesky(Ks_xx) except linalg.LinAlgError: print('Noise/sparse covariance not positive definite') return -np.inf # Use Woodbury-Sherman-Morrison formula with the # following notation: # # y2 = f0' * inv(Kd_xp*inv(Kd_pp)*Kd_xp' + Ks_xx) * f0 # # z = Ks_xx \ f0 # Lambda = Kd_pp + Kd_xp'*inv(Ks_xx)*Kd_xp # nu = inv(Lambda) * (Kd_xp' * (Ks_xx \ f0)) # rho = Kd_xp * inv(Lambda) * (Kd_xp' * (Ks_xx \ f0)) # # y2 = f0' * z - z' * rho z = Us_xx.solve(f0) Lambda = Kd_pp + np.dot(Kd_xp.T, Us_xx.solve(Kd_xp)) ## z = utils.chol_solve(Us_xx, f0) ## Lambda = Kd_pp + np.dot(Kd_xp.T, ## utils.chol_solve(Us_xx, Kd_xp)) try: U_Lambda = utils.cholesky(Lambda) #U_Lambda = utils.chol(Lambda) except linalg.LinAlgError: print('Lambda not positive definite') return -np.inf nu = U_Lambda.solve(np.dot(Kd_xp.T, z)) #nu = utils.chol_solve(U_Lambda, np.dot(Kd_xp.T, z)) rho = np.dot(Kd_xp, nu) y2 = np.dot(f0, z) - np.dot(z, rho) # Use matrix determinant lemma # # det(Kd_xp*inv(Kd_pp)*Kd_xp' + Ks_xx) # = det(Kd_pp + Kd_xp'*inv(Ks_xx)*Kd_xp) # * det(inv(Kd_pp)) * det(Ks_xx) # = det(Lambda) * det(Ks_xx) / det(Kd_pp) try: Ud_pp = utils.cholesky(Kd_pp) #Ud_pp = utils.chol(Kd_pp) except linalg.LinAlgError: print('Covariance of pseudo inputs not positive definite') return -np.inf logdet = (U_Lambda.logdet() + Us_xx.logdet() - Ud_pp.logdet()) ## logdet = (utils.logdet_chol(U_Lambda) ## + utils.logdet_chol(Us_xx) ## - utils.logdet_chol(Ud_pp)) # Compute the log pdf L = gaussian_logpdf(y2, 0, 0, logdet, np.size(self.f)) # Add the variational cost of the pseudo-input # approximation # Compute gradients for (dmu, func) in Dmu: # Derivative w.r.t. mean vector d = np.nan # Send the derivative message func(d) for (dKs_xx, func) in DKs_xx: # Compute derivative w.r.t. covariance matrix d = np.nan # Send the derivative message func(d) for (dKd_xp, func) in DKd_xp: # Compute derivative w.r.t. covariance matrix d = np.nan # Send the derivative message func(d) V = Ud_pp.solve(Kd_xp.T) Z = Us_xx.solve(V.T) ## V = utils.chol_solve(Ud_pp, Kd_xp.T) ## Z = utils.chol_solve(Us_xx, V.T) for (dKd_pp, func) in DKd_pp: # Compute derivative w.r.t. covariance matrix d = (0.5 * np.trace(Ud_pp.solve(dKd_pp)) - 0.5 * np.trace(U_Lambda.solve(dKd_pp)) + np.dot(nu, np.dot(dKd_pp, nu)) + np.trace(np.dot(dKd_pp, np.dot(V,Z)))) ## d = (0.5 * np.trace(utils.chol_solve(Ud_pp, dKd_pp)) ## - 0.5 * np.trace(utils.chol_solve(U_Lambda, dKd_pp)) ## + np.dot(nu, np.dot(dKd_pp, nu)) ## + np.trace(np.dot(dKd_pp, ## np.dot(V,Z)))) # Send the derivative message func(d) for (dxp, func) in Dxp: # Compute derivative w.r.t. covariance matrix d = np.nan # Send the derivative message func(d) else: ## Full exact (no pseudo approximations) try: U = utils.cholesky(K_xx) #U = utils.chol(K_xx) except linalg.LinAlgError: print('non positive definite, return -inf') return -np.inf z = U.solve(f0) #z = utils.chol_solve(U, f0) #print(K) L = utils.gaussian_logpdf(np.dot(f0, z), 0, 0, U.logdet(), ## utils.logdet_chol(U), np.size(self.f)) for (dmu, func) in Dmu: # Derivative w.r.t. mean vector d = -np.sum(z) # Send the derivative message func(d) for (dK, func) in DKd_xx: # Compute derivative w.r.t. covariance matrix # # TODO: trace+chol_solve should be handled better # for sparse matrices. Use sparse-inverse! d = 0.5 * (dK.dot(z).dot(z) - U.trace_solve_gradient(dK)) ## - np.trace(U.solve(dK))) ## d = 0.5 * (dK.dot(z).dot(z) ## - np.trace(utils.chol_solve(U, dK))) #print('derivate', d, dK) ## d = 0.5 * (np.dot(z, np.dot(dK, z)) ## - np.trace(utils.chol_solve(U, dK))) # # Send the derivative message func(d) for (dK, func) in DKs_xx: # Compute derivative w.r.t. covariance matrix d = 0.5 * (dK.dot(z).dot(z) - U.trace_solve_gradient(dK)) ## - np.trace(U.solve(dK))) ## d = 0.5 * (dK.dot(z).dot(z) ## - np.trace(utils.chol_solve(U, dK))) ## d = 0.5 * (np.dot(z, np.dot(dK, z)) ## - np.trace(utils.chol_solve(U, dK))) # Send the derivative message func(d) else: ## Log pdf for latent GP raise Exception('Not implemented yet') return L ## Let f1 be observed and f2 latent function values. # Compute #L = gaussian_logpdf(sum_product(np.outer(self.f,self.f) + self.Cov, # Compute def update(self): # Messages from parents m = self.parents[0].message_to_child() k = self.parents[1].message_to_child() if self.parents[2]: k_sparse = self.parents[2].message_to_child() else: k_sparse = None if self.parents[3]: pseudoinputs = self.parents[3].message_to_child()[0] else: pseudoinputs = None ## m = self.parents[0].message_to_child()[0] ## k = self.parents[1].message_to_child()[0] if self.observed: # Observations of this node self.u = gp_posterior_moment_function(m, k, self.x, self.f, k_sparse=k_sparse, pseudoinputs=pseudoinputs) else: x = np.array([]) y = np.array([]) # Messages from children for (child,index) in self.children: (msg, mask) = child.message_to_parent(index) # Ignoring masks and plates.. # m[0] is the inputs x = np.concatenate((x, msg[0]), axis=-2) # m[1] is the observations y = np.concatenate((y, msg[1])) # m[2] is the covariance matrix V = linalg.block_diag(V, msg[2]) self.u = gp_posterior_moment_function(m, k, x, y, covariance=V) self.x = x self.f = y # At least for now, simplify this GP node such that a GP is either # observed or latent. If it is observed, it doesn't take messages from # children, actually, it should not even have children! ## # Pseudo for GPFA: ## k1 = gp_cov_se(magnitude=theta1, lengthscale=theta2) ## k2 = gp_cov_periodic(magnitude=.., lengthscale=.., period=..) ## k3 = gp_cov_rq(magnitude=.., lengthscale=.., alpha=..) ## f = NodeGPSet(0, [k1,k2,k3]) # assumes block diagonality ## # f = NodeGPSet(0, [[k11,k12,k13],[k21,k22,k23],[k31,k32,k33]]) ## X = GaussianFromGP(f, [ [[t0,0],[t0,1],[t0,2]], [t1,0],[t1,1],[t1,2], ..]) ## ... ## # Construct a sum of GPs if interested only in the sum term ## k1 = gp_cov_se(magnitude=theta1, lengthscale=theta2) ## k2 = gp_cov_periodic(magnitude=.., lengthscale=.., period=..) ## k = gp_cov_sum(k1, k2) ## f = NodeGP(0, k) ## f.observe(x, y) ## f.update() ## (mp, kp) = f.get_parameters() ## # Construct a sum of GPs when interested also in the individual ## # GPs: ## k1 = gp_cov_se(magnitude=theta1, lengthscale=theta2) ## k2 = gp_cov_periodic(magnitude=.., lengthscale=.., period=..) ## k3 = gp_cov_delta(magnitude=theta3) ## f = NodeGPSum(0, [k1,k2,k3]) ## x = np.array([1,2,3,4,5,6,7,8,9,10]) ## y = np.sin(x[0]) + np.random.normal(0, 0.1, (10,)) ## # Observe the sum (index 0) ## f.observe((0,x), y) ## # Inference ## f.update() ## (mp, kp) = f.get_parameters() ## # Mean of the sum ## mp[0](...) ## # Mean of the individual terms ## mp[1](...) ## mp[2](...) ## mp[3](...) ## # Covariance of the sum ## kp[0][0](..., ...) ## # Other covariances ## kp[1][1](..., ...) ## kp[2][2](..., ...) ## kp[3][3](..., ...) ## kp[1][2](..., ...) ## kp[1][3](..., ...) ## kp[2][3](..., ...) bayespy-0.4.0/bayespy/inference/vmp/nodes/logpdf.py0000644000175000017500000000576212533543154024075 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2015 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import numpy as np from .expfamily import ExponentialFamily, useconstructor from .stochastic import Distribution from .node import Moments class LogPDFDistribution(Distribution): pass class LogPDF(ExponentialFamily): """ General node with arbitrary probability density function """ def __init__(self, logpdf, *parents, **kwargs): self._logpdf = logpdf super().__init__(logpdf, *parents, initialize=False, **kwargs) @classmethod def _constructor(cls, logpdf, *parents, approximation=None, shape=None, samples=10, **kwargs): r""" Constructs distribution and moments objects. """ if approximation is not None: raise NotImplementedError() #self._distribution = approximation._constructor dims = ( shape, ) _distribution = LogPDFDistribution() _moments = np.nan _parent_moments = [Moments()] * len(parents) parent_plates = [_distribution.plates_from_parent(i, parent.plates) for (i, parent) in enumerate(parents)] return (parents, kwargs, dims, cls._total_plates(kwargs.get('plates'), *parent_plates), _distribution, _moments, _parent_moments) def _get_message_and_mask_to_parent(self, index): def logpdf_sampler(x): inputs = [self.parents[j].random() if j != index else x for j in range(len(self.parents))] return self._logpdf(self.random(), *inputs) mask = self._distribution.compute_mask_to_parent(index, self.mask) return (logpdf_sampler, mask) def observe(self, x, *args, mask=True): """ Fix moments, compute f and propagate mask. """ # Compute fixed moments if not np.isnan(self._moments): u = self._moments.compute_fixed_moments(x, *args, mask=mask) else: u = (x,) + args # Check the dimensionality of the observations for (i,v) in enumerate(u): # This is what the dimensionality "should" be s = self.plates + self.dims[i] t = np.shape(v) if s != t: msg = "Dimensionality of the observations incorrect." msg += "\nShape of input: " + str(t) msg += "\nExpected shape: " + str(s) msg += "\nCheck plates." raise Exception(msg) # Set the moments self._set_moments(u, mask=mask) # Observed nodes should not be ignored self.observed = mask self._update_mask() bayespy-0.4.0/bayespy/inference/vmp/nodes/dot.py0000644000175000017500000005147712535521744023417 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import numpy as np from bayespy.utils import misc from .node import Node from .deterministic import Deterministic from .gaussian import Gaussian, GaussianMoments from .gaussian import GaussianGammaISOMoments class SumMultiply(Deterministic): r""" Node for computing general products and sums of Gaussian nodes. The node is similar to `numpy.einsum`, which is a very general function for computing dot products, sums, products and other sums of products of arrays. For instance, consider the following arrays: >>> import numpy as np >>> X = np.random.randn(2, 3, 4) >>> Y = np.random.randn(3, 5) >>> Z = np.random.randn(4, 2) Then, the Einstein summation can be used as: >>> np.einsum('abc,bd,ca->da', X, Y, Z) array([[...]]) SumMultiply node can be used similarly for Gaussian nodes. For instance, consider the following Gaussian nodes: >>> from bayespy.nodes import GaussianARD >>> X = GaussianARD(0, 1, shape=(2, 3, 4)) >>> Y = GaussianARD(0, 1, shape=(3, 5)) >>> Z = GaussianARD(0, 1, shape=(4, 2)) Then, similarly to `numpy.einsum`, SumMultiply could be used as: >>> from bayespy.nodes import SumMultiply >>> SumMultiply('abc,bd,ca->da', X, Y, Z) or >>> SumMultiply(X, [0,1,2], Y, [1,3], Z, [2,0], [3,0]) which is similar to the alternative syntax of numpy.einsum. This node operates similarly as numpy.einsum. However, you must use all the elements of each node, that is, an operation like np.einsum('ii->i',X) is not allowed. Thus, for each node, each axis must be given unique id. The id identifies which axes correspond to which axes between the different nodes. Also, Ellipsis ('...') is not yet supported for simplicity. It would also have some problems with constant inputs (because how to determine ndim), so let us just forget it for now. Each output axis must appear in the input mappings. The keys must refer to variable dimension axes only, not plate axes. The input nodes may be Gaussian-gamma (isotropic) nodes. The output message is Gaussian-gamma (isotropic) if any of the input nodes is Gaussian-gamma. Examples -------- Sum over the rows: 'ij->j' Inner product of three vectors: 'i,i,i' Matrix-vector product: 'ij,j->i' Matrix-matrix product: 'ik,kj->ij' Outer product: 'i,j->ij' Vector-matrix-vector product: 'i,ij,j' Notes ----- This operation can be extremely slow if not used wisely. For large and complex operations, it is sometimes more efficient to split the operation into multiple nodes. For instance, the example above could probably be computed faster by >>> XZ = SumMultiply(X, [0,1,2], Z, [2,0], [0,1]) >>> F = SumMultiply(XZ, [0,1], Y, [1,2], [2,0]) because the third axis ('c') could be summed out already in the first operation. This same effect applies also to numpy.einsum in general. """ def __init__(self, *args, iterator_axis=None, **kwargs): """ SumMultiply(Node1, map1, Node2, map2, ..., NodeN, mapN [, map_out]) """ args = list(args) if len(args) < 2: raise ValueError("Not enough inputs") if iterator_axis is not None: raise NotImplementedError("Iterator axis not implemented yet") if iterator_axis is not None and not isinstance(iterator_axis, int): raise ValueError("Iterator axis must be integer") # Two different parsing methods, depends on how the arguments are given if misc.is_string(args[0]): # This is the format: # SumMultiply('ik,k,kj->ij', X, Y, Z) strings = args[0] nodes = args[1:] # Remove whitespace strings = misc.remove_whitespace(strings) # Split on '->' (should contain only one '->' or none) strings = strings.split('->') if len(strings) > 2: raise ValueError('The string contains too many ->') strings_in = strings[0] if len(strings) == 2: string_out = strings[1] else: string_out = '' # Split former part on ',' (the number of parts should be equal to # nodes) strings_in = strings_in.split(',') if len(strings_in) != len(nodes): raise ValueError('Number of given input nodes is different ' 'from the input keys in the string') # Split strings into key lists using single character keys keysets = [list(string_in) for string_in in strings_in] keys_out = list(string_out) else: # This is the format: # SumMultiply(X, [0,2], Y, [2], Z, [2,1], [0,1]) # If given, the output mapping is the last argument if len(args) % 2 == 0: keys_out = [] else: keys_out = args.pop(-1) # Node and axis mapping are given in turns nodes = args[::2] keysets = args[1::2] # Find all the keys (store only once each) full_keyset = [] for keyset in keysets: full_keyset += keyset #full_keyset += list(keyset.keys()) full_keyset = list(set(full_keyset)) # Input and output messages are Gaussian unless there is at least one # Gaussian-gamma message from the parents self._moments = GaussianMoments(len(keys_out)) self._parent_moments = [GaussianMoments(len(keyset)) for keyset in keysets] self.gaussian_gamma = False for n in range(len(nodes)): try: # Try to convert into Gaussian moments nodes[n] = self._ensure_moments(nodes[n], self._parent_moments[n]) except GaussianMoments.NoConverterError: # Failure. It is probably a Gaussian-gamma node self.gaussian_gamma = True if self.gaussian_gamma: self._moments = GaussianGammaISOMoments(len(keys_out)) self._parent_moments = [GaussianGammaISOMoments(len(keyset)) for keyset in keysets] for n in range(len(nodes)): # Convert into Gaussian-gamma nodes nodes[n] = self._ensure_moments(nodes[n], self._parent_moments[n]) # # Check the validity of each node # for n in range(len(nodes)): # Check that the maps and the size of the variable are consistent if len(nodes[n].dims[0]) != len(keysets[n]): raise ValueError("Wrong number of keys (%d) for the node " "number %d with %d dimensions" % (len(keysets[n]), n, len(nodes[n].dims[0]))) # Check that the keys are unique if len(set(keysets[n])) != len(keysets[n]): raise ValueError("Axis keys for node number %d are not unique" % n) # Check the validity of output keys: each output key must be included in # the input keys if len(keys_out) != len(set(keys_out)): raise ValueError("Output keys are not unique") for key in keys_out: if key not in full_keyset: raise ValueError("Output key %s does not appear in any input" % key) # Check the validity of the nodes with respect to the key mapping. # Check that the node dimensions map and broadcast properly, that is, # all the nodes using the same key for axes must have equal size for # those axes (or size 1). broadcasted_size = {} for key in full_keyset: broadcasted_size[key] = 1 for (node, keyset) in zip(nodes, keysets): try: # Find the axis for the key index = keyset.index(key) except ValueError: # OK, this node doesn't use this key for any axis pass else: # Length of the axis for that key node_size = node.dims[0][index] if node_size != broadcasted_size[key]: if broadcasted_size[key] == 1: # Apply broadcasting broadcasted_size[key] = node_size elif node_size != 1: # Different sizes and neither has size 1 raise ValueError("Axes using key %s do not " "broadcast properly" % key) # Compute the shape of the output dim0 = [broadcasted_size[key] for key in keys_out] dim1 = dim0 + dim0 dims = ( tuple(dim0), tuple(dim1) ) if self.gaussian_gamma: dims = dims + ( (), () ) # Rename the keys to [0,1,...,N-1] where N is the total number of keys self.N_keys = len(full_keyset) self.out_keys = [full_keyset.index(key) for key in keys_out] self.in_keys = [ [full_keyset.index(key) for key in keyset] for keyset in keysets ] super().__init__(*nodes, dims=dims, **kwargs) def _compute_moments(self, *u_parents): # Compute the number of plate axes for each node plate_counts0 = [(np.ndim(u_parent[0]) - len(keys)) for (keys,u_parent) in zip(self.in_keys, u_parents)] plate_counts1 = [(np.ndim(u_parent[1]) - 2*len(keys)) for (keys,u_parent) in zip(self.in_keys, u_parents)] # The number of plate axes for the output N0 = max(plate_counts0) N1 = max(plate_counts1) # The total number of unique keys used (keys are 0,1,...,N_keys-1) D = self.N_keys # # Compute the mean # out_all_keys = list(range(D+N0-1, D-1, -1)) + self.out_keys #nodes_dim_keys = self.nodes_dim_keys in_all_keys = [list(range(D+plate_count-1, D-1, -1)) + keys for (plate_count, keys) in zip(plate_counts0, self.in_keys)] u0 = [u[0] for u in u_parents] args = misc.zipper_merge(u0, in_all_keys) + [out_all_keys] x0 = np.einsum(*args) # # Compute the covariance # out_all_keys = (list(range(2*D+N1-1, 2*D-1, -1)) + [D+key for key in self.out_keys] + self.out_keys) in_all_keys = [list(range(2*D+plate_count-1, 2*D-1, -1)) + [D+key for key in node_keys] + node_keys for (plate_count, node_keys) in zip(plate_counts1, self.in_keys)] u1 = [u[1] for u in u_parents] args = misc.zipper_merge(u1, in_all_keys) + [out_all_keys] x1 = np.einsum(*args) if not self.gaussian_gamma: return [x0, x1] # Compute Gaussian-gamma specific moments x2 = 1 x3 = 0 for i in range(len(u_parents)): x2 = x2 * u_parents[i][2] x3 = x3 + u_parents[i][3] return [x0, x1, x2, x3] def get_parameters(self): # Compute mean and variance u = self.get_moments() u[1] -= u[0]**2 return u def _message_to_parent(self, index): """ Compute the message and mask to a parent node. """ # Check index if index >= len(self.parents): raise ValueError("Parent index larger than the number of parents") # Get messages from other parents and children u_parents = self._message_from_parents(exclude=index) m = self._message_from_children() mask = self.mask # Normally we don't need to care about masks when computing the # message. However, in this node we want to avoid computing huge message # arrays so we sum some axes already here. Thus, we need to apply the # mask. # # Actually, we don't need to care about masks because the message from # children has already been masked. parent = self.parents[index] # # Compute the first message # msg = [None, None] # Compute the two messages for ind in range(2): # The total number of keys for the non-plate dimensions N = (ind+1) * self.N_keys parent_num_dims = len(parent.dims[ind]) parent_num_plates = len(parent.plates) parent_plate_keys = list(range(N + parent_num_plates, N, -1)) parent_dim_keys = self.in_keys[index] if ind == 1: parent_dim_keys = ([key + self.N_keys for key in self.in_keys[index]] + parent_dim_keys) args = [] # This variable counts the maximum number of plates of the # arguments, thus it will tell the number of plates in the result # (if the artificially added plates above were ignored). result_num_plates = 0 result_plates = () # Mask and its keysr mask_num_plates = np.ndim(mask) mask_plates = np.shape(mask) mask_plate_keys = list(range(N + mask_num_plates, N, -1)) result_num_plates = max(result_num_plates, mask_num_plates) result_plates = misc.broadcasted_shape(result_plates, mask_plates) # Moments and keys of other parents for (k, u) in enumerate(u_parents): if k != index: num_dims = (ind+1) * len(self.in_keys[k]) num_plates = np.ndim(u[ind]) - num_dims plates = np.shape(u[ind])[:num_plates] plate_keys = list(range(N + num_plates, N, -1)) dim_keys = self.in_keys[k] if ind == 1: dim_keys = ([key + self.N_keys for key in self.in_keys[k]] + dim_keys) args.append(u[ind]) args.append(plate_keys + dim_keys) result_num_plates = max(result_num_plates, num_plates) result_plates = misc.broadcasted_shape(result_plates, plates) # Message and keys from children child_num_dims = (ind+1) * len(self.out_keys) child_num_plates = np.ndim(m[ind]) - child_num_dims child_plates = np.shape(m[ind])[:child_num_plates] child_plate_keys = list(range(N + child_num_plates, N, -1)) child_dim_keys = self.out_keys if ind == 1: child_dim_keys = ([key + self.N_keys for key in self.out_keys] + child_dim_keys) args.append(m[ind]) args.append(child_plate_keys + child_dim_keys) result_num_plates = max(result_num_plates, child_num_plates) result_plates = misc.broadcasted_shape(result_plates, child_plates) # Output keys, that is, the keys of the parent[index] parent_keys = parent_plate_keys + parent_dim_keys # Performance trick: Check which axes can be summed because they # have length 1 or are non-existing in parent[index]. Thus, remove # keys corresponding to unit length axes in parent[index] so that # einsum sums over those axes. After computations, these axes must # be added back in order to get the correct shape for the message. # Also, remove axes/keys that are in output (parent[index]) but not in # any inputs (children and other parents). parent_shape = parent.get_shape(ind) removed_axes = [] for j in range(len(parent_keys)): if parent_shape[j] == 1: # Remove the key (take into account the number of keys that # have already been removed) del parent_keys[j-len(removed_axes)] removed_axes.append(j) else: # Remove the key if it doesn't appear in any of the # messages from children or other parents. if not np.any([parent_keys[j-len(removed_axes)] in keys for keys in args[1::2]]): del parent_keys[j-len(removed_axes)] removed_axes.append(j) args.append(parent_keys) # THE BEEF: Compute the message msg[ind] = np.einsum(*args) # Find the correct shape for the message array message_shape = list(np.shape(msg[ind])) # First, add back the axes with length 1 for ax in removed_axes: message_shape.insert(ax, 1) # Second, remove leading axes for plates that were not present in # the child nor other parents' messages. This is not really # necessary, but it is just elegant to remove the leading unit # length axes that we added artificially at the beginning just # because we wanted the key mapping to be simple. if parent_num_plates > result_num_plates: del message_shape[:(parent_num_plates-result_num_plates)] # Then, the actual reshaping msg[ind] = np.reshape(msg[ind], message_shape) # Broadcasting is not supported for variable dimensions, thus force # explicit correct shape for variable dimensions var_dims = parent.dims[ind] msg[ind] = msg[ind] * np.ones(var_dims) # Apply plate multiplier: If this node has non-unit plates that are # unit plates in the parent, those plates are summed. However, if # the message has unit axis for that plate, it should be first # broadcasted to the plates of this node and then summed to the # plates of the parent. In order to avoid this broadcasting and # summing, it is more efficient to just multiply by the correct # factor. r = self.broadcasting_multiplier(self.plates, result_plates, parent.plates) if r != 1: msg[ind] *= r if self.gaussian_gamma: alphas = [u_parents[i][2] for i in range(len(u_parents)) if i != index] ## logalphas = [u_parents[i][3] ## for i in range(len(u_parents)) if i != index] m2 = self._compute_message(m[2], *alphas, ndim=0, plates_from=self.plates, plates_to=parent.plates) m3 = self._compute_message(m[3],#*logalphas, ndim=0, plates_from=self.plates, plates_to=parent.plates) msg = msg + [m2, m3] return msg def Dot(*args, **kwargs): """ Node for computing inner product of several Gaussian vectors. This is a simple wrapper of the much more general SumMultiply. For now, it is here for backward compatibility. """ einsum = 'i' + ',i'*(len(args)-1) return SumMultiply(einsum, *args, **kwargs) bayespy-0.4.0/bayespy/inference/vmp/nodes/categorical.py0000644000175000017500000001360012533543154025065 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2012,2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Module for the categorical distribution node. """ import numpy as np from .node import ensureparents from .expfamily import (ExponentialFamily, useconstructor) from .multinomial import (MultinomialMoments, MultinomialDistribution, Multinomial) from .dirichlet import DirichletMoments from bayespy.utils import random from bayespy.utils import misc class CategoricalMoments(MultinomialMoments): """ Class for the moments of categorical variables. """ def __init__(self, categories): """ Create moments object for categorical variables """ self.D = categories super().__init__() def compute_fixed_moments(self, x): """ Compute the moments for a fixed value """ # Check that x is valid x = np.asanyarray(x) if not misc.isinteger(x): raise ValueError("Values must be integers") if np.any(x < 0) or np.any(x >= self.D): raise ValueError("Invalid category index") u0 = np.zeros((np.size(x), self.D)) u0[[np.arange(np.size(x)), np.ravel(x)]] = 1 u0 = np.reshape(u0, np.shape(x) + (self.D,)) return [u0] def compute_dims_from_values(self, x): """ Return the shape of the moments for a fixed value. The observations are scalar. """ return ( (self.D,), ) class CategoricalDistribution(MultinomialDistribution): """ Class for the VMP formulas of categorical variables. """ def __init__(self, categories): """ Create VMP formula node for a categorical variable `categories` is the total number of categories. """ if not isinstance(categories, int): raise ValueError("Number of categories must be integer") if categories < 0: raise ValueError("Number of categoriess must be non-negative") self.D = categories super().__init__(1) def compute_message_to_parent(self, parent, index, u, u_p): """ Compute the message to a parent node. """ return super().compute_message_to_parent(parent, index, u, u_p) def compute_phi_from_parents(self, u_p, mask=True): """ Compute the natural parameter vector given parent moments. """ return super().compute_phi_from_parents(u_p, mask=mask) def compute_moments_and_cgf(self, phi, mask=True): """ Compute the moments and :math:`g(\phi)`. """ return super().compute_moments_and_cgf(phi, mask=mask) def compute_cgf_from_parents(self, u_p): """ Compute :math:`\mathrm{E}_{q(p)}[g(p)]` """ return super().compute_cgf_from_parents(u_p) def compute_fixed_moments_and_f(self, x, mask=True): """ Compute the moments and :math:`f(x)` for a fixed value. """ # Check the validity of x x = np.asanyarray(x) if not misc.isinteger(x): raise ValueError("Values must be integers") if np.any(x < 0) or np.any(x >= self.D): raise ValueError("Invalid category index") # Form a binary matrix with only one non-zero (1) in the last axis u0 = np.zeros((np.size(x), self.D)) u0[[np.arange(np.size(x)), np.ravel(x)]] = 1 u0 = np.reshape(u0, np.shape(x) + (self.D,)) u = [u0] # f(x) is zero f = 0 return (u, f) def random(self, *phi, plates=None): """ Draw a random sample from the distribution. """ logp = phi[0] logp -= np.amax(logp, axis=-1, keepdims=True) p = np.exp(logp) return random.categorical(p, size=plates) class Categorical(ExponentialFamily): r""" Node for categorical random variables. The node models a categorical random variable :math:`x \in \{0,\ldots,K-1\}` with prior probabilities :math:`\{p_0, \ldots, p_{K-1}\}` for each category: .. math:: p(x=k) = p_k \quad \text{for } k\in \{0,\ldots,K-1\}. Parameters ---------- p : Dirichlet-like node or (...,K)-array Probabilities for each category See also -------- Bernoulli, Multinomial, Dirichlet """ _parent_moments = [DirichletMoments()] def __init__(self, p, **kwargs): """ Create Categorical node. """ super().__init__(p, **kwargs) @classmethod @ensureparents def _constructor(cls, p, **kwargs): """ Constructs distribution and moments objects. This method is called if useconstructor decorator is used for __init__. Becase the distribution and moments object depend on the number of categories, that is, they depend on the parent node, this method can be used to construct those objects. """ # Get the number of categories D = p.dims[0][0] parents = [p] moments = CategoricalMoments(D) distribution = CategoricalDistribution(D) return (parents, kwargs, ( (D,), ), cls._total_plates(kwargs.get('plates'), distribution.plates_from_parent(0, p.plates)), distribution, moments, cls._parent_moments) def __str__(self): """ Print the distribution using standard parameterization. """ p = self.u[0] return ("%s ~ Categorical(p)\n" " p = \n" "%s\n" % (self.name, p)) bayespy-0.4.0/bayespy/inference/vmp/nodes/mixture.py0000644000175000017500000004721712535506420024315 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2012,2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Module for the mixture distribution node. """ import warnings import numpy as np from bayespy.utils import misc from .expfamily import ExponentialFamily, \ ExponentialFamilyDistribution, \ useconstructor from .categorical import Categorical, \ CategoricalMoments class MixtureDistribution(ExponentialFamilyDistribution): """ Class for the VMP formulas of mixture variables. """ def __init__(self, distribution, cluster_plate, n_clusters, ndims, ndims_parents): """ Create VMP formula node for a mixture variable """ self.distribution = distribution self.cluster_plate = cluster_plate self.ndims = ndims self.ndims_parents = ndims_parents self.K = n_clusters def compute_message_to_parent(self, parent, index, u, *u_parents): """ Compute the message to a parent node. """ if index == 0: # Shape(phi) = [Nn,..,K,..,N0,Dd,..,D0] # Shape(L) = [Nn,..,K,..,N0] # Shape(u) = [Nn,..,N0,Dd,..,D0] # Shape(result) = [Nn,..,N0,K] # Compute g: # Shape(g) = [Nn,..,K,..,N0] g = self.distribution.compute_cgf_from_parents(*(u_parents[1:])) # Reshape(g): # Shape(g) = [Nn,..,N0,K] if np.ndim(g) < abs(self.cluster_plate): # Not enough axes, just add the cluster plate axis g = np.expand_dims(g, -1) else: # Move the cluster plate axis g = misc.moveaxis(g, self.cluster_plate, -1) # Compute phi: # Shape(phi) = [Nn,..,K,..,N0,Dd,..,D0] phi = self.distribution.compute_phi_from_parents(*(u_parents[1:])) # Move phi axis: # Shape(phi) = [Nn,..,N0,K,Dd,..,D0] for ind in range(len(phi)): if self.cluster_plate < 0: axis_from = self.cluster_plate-self.ndims[ind] else: raise RuntimeError("Cluster plate axis must be negative") axis_to = -1-self.ndims[ind] if np.ndim(phi[ind]) >= abs(axis_from): # Cluster plate axis exists, move it to the correct position phi[ind] = misc.moveaxis(phi[ind], axis_from, axis_to) else: # No cluster plate axis, just add a new axis to the correct # position, if phi has something on that axis if np.ndim(phi[ind]) >= abs(axis_to): phi[ind] = np.expand_dims(phi[ind], axis=axis_to) # Reshape u: # Shape(u) = [Nn,..,N0,1,Dd,..,D0] u_self = list() for ind in range(len(u)): u_self.append(np.expand_dims(u[ind], axis=(-1-self.ndims[ind]))) # Compute logpdf: # Shape(L) = [Nn,..,N0,K] L = self.distribution.compute_logpdf(u_self, phi, g, 0, self.ndims) # Sum over other than the cluster dimensions? No! # Hmm.. I think the message passing method will do # that automatically m = [L] return m elif index >= 1: # Parent index for the distribution used for the # mixture. index = index - 1 # Reshape u: # Shape(u) = [Nn,..1,..,N0,Dd,..,D0] u_self = list() for ind in range(len(u)): if self.cluster_plate < 0: cluster_axis = self.cluster_plate - self.ndims[ind] else: cluster_axis = self.cluster_plate u_self.append(np.expand_dims(u[ind], axis=cluster_axis)) # Message from the mixed distribution m = self.distribution.compute_message_to_parent(parent, index, u_self, *(u_parents[1:])) # Weigh the messages with the responsibilities for i in range(len(m)): # Shape(m) = [Nn,..,K,..,N0,Dd,..,D0] # Shape(p) = [Nn,..,N0,K] # Shape(result) = [Nn,..,K,..,N0,Dd,..,D0] # Number of axes for the variable dimensions for # the parent message. D = self.ndims_parents[index][i] # Responsibilities for clusters are the first # parent's first moment: # Shape(p) = [Nn,..,N0,K] p = u_parents[0][0] # Move the cluster axis to the proper place: # Shape(p) = [Nn,..,K,..,N0] p = misc.atleast_nd(p, abs(self.cluster_plate)) p = misc.moveaxis(p, -1, self.cluster_plate) # Add axes for variable dimensions to the contributions # Shape(p) = [Nn,..,K,..,N0,1,..,1] p = misc.add_trailing_axes(p, D) if self.cluster_plate < 0: # Add the variable dimensions cluster_axis = self.cluster_plate - D # Add axis for clusters: # Shape(m) = [Nn,..,1,..,N0,Dd,..,D0] #m[i] = np.expand_dims(m[i], axis=cluster_axis) # # TODO: You could do summing here already so that # you wouldn't compute huge matrices as # intermediate result. Use einsum. # Compute the message contributions for each # cluster: # Shape(result) = [Nn,..,K,..,N0,Dd,..,D0] m[i] = m[i] * p return m def compute_mask_to_parent(self, index, mask): """ Maps the mask to the plates of a parent. """ if index == 0: return mask else: if self.cluster_plate >= 0: raise ValueError("Cluster plate axis must be negative") if np.ndim(mask) >= abs(self.cluster_plate): mask = np.expand_dims(mask, axis=self.cluster_plate) return self.distribution.compute_mask_to_parent(index-1, mask) def compute_phi_from_parents(self, *u_parents, mask=True): """ Compute the natural parameter vector given parent moments. """ # Compute weighted average of the parameters # Cluster parameters Phi = self.distribution.compute_phi_from_parents(*(u_parents[1:])) # Contributions/weights/probabilities P = u_parents[0][0] phi = list() nans = False for ind in range(len(Phi)): # Compute element-wise product and then sum over K clusters. # Note that the dimensions aren't perfectly aligned because # the cluster dimension (K) may be arbitrary for phi, and phi # also has dimensions (Dd,..,D0) of the parameters. # Shape(phi) = [Nn,..,K,..,N0,Dd,..,D0] # Shape(p) = [Nn,..,N0,K] # Shape(result) = [Nn,..,N0,Dd,..,D0] # General broadcasting rules apply for Nn,..,N0, that is, # preceding dimensions may be missing or dimension may be # equal to one. Probably, shape(phi) has lots of missing # dimensions and/or dimensions that are one. if self.cluster_plate < 0: cluster_axis = self.cluster_plate - self.ndims[ind] else: raise RuntimeError("Cluster plate should be negative") # Move cluster axis to the last: # Shape(phi) = [Nn,..,N0,Dd,..,D0,K] if np.ndim(Phi[ind]) >= abs(cluster_axis): phi.append(misc.moveaxis(Phi[ind], cluster_axis, -1)) else: phi.append(Phi[ind][...,None]) # Add axes to p: # Shape(p) = [Nn,..,N0,K,1,..,1] p = misc.add_trailing_axes(P, self.ndims[ind]) # Move cluster axis to the last: # Shape(p) = [Nn,..,N0,1,..,1,K] p = misc.moveaxis(p, -(self.ndims[ind]+1), -1) # Now the shapes broadcast perfectly and we can sum # p*phi over the last axis: # Shape(result) = [Nn,..,N0,Dd,..,D0] phi[ind] = misc.sum_product(p, phi[ind], axes_to_sum=-1) if np.any(np.isnan(phi[ind])): nans = True if nans: warnings.warn("The natural parameters of mixture distribution " "contain nans. This may happen if you use fixed " "parameters in your model. Technically, one possible " "reason is that the cluster assignment probability " "for some element is zero (p=0) and the natural " "parameter of that cluster is -inf, thus " "0*(-inf)=nan. Solution: Use parameters that assign " "non-zero probabilities for the whole domain.") return phi def compute_moments_and_cgf(self, phi, mask=True): """ Compute the moments and :math:`g(\phi)`. """ return self.distribution.compute_moments_and_cgf(phi, mask=mask) def compute_cgf_from_parents(self, *u_parents): """ Compute :math:`\mathrm{E}_{q(p)}[g(p)]` """ # Compute weighted average of g over the clusters. # Shape(g) = [Nn,..,K,..,N0] # Shape(p) = [Nn,..,N0,K] # Shape(result) = [Nn,..,N0] # Compute g for clusters: # Shape(g) = [Nn,..,K,..,N0] g = self.distribution.compute_cgf_from_parents(*(u_parents[1:])) # Move cluster axis to last: # Shape(g) = [Nn,..,N0,K] if np.ndim(g) < abs(self.cluster_plate): # Not enough axes, just add the cluster plate axis g = np.expand_dims(g, -1) else: # Move the cluster plate axis g = misc.moveaxis(g, self.cluster_plate, -1) # Cluster assignments/contributions/probabilities/weights: # Shape(p) = [Nn,..,N0,K] p = u_parents[0][0] # Weighted average of g over the clusters. As p and g are # properly aligned, you can just sum p*g over the last # axis and utilize broadcasting: # Shape(result) = [Nn,..,N0] g = misc.sum_product(p, g, axes_to_sum=-1) return g def compute_fixed_moments_and_f(self, x, mask=True): """ Compute the moments and :math:`f(x)` for a fixed value. """ return self.distribution.compute_fixed_moments_and_f(x, mask=True) def plates_to_parent(self, index, plates): """ Resolves the plate mapping to a parent. Given the plates of the node's moments, this method returns the plates that the message to a parent has for the parent's distribution. """ if index == 0: return plates else: # Add the cluster plate axis plates = list(plates) if self.cluster_plate < 0: knd = len(plates) + self.cluster_plate + 1 else: raise RuntimeError("Cluster plate axis must be negative") plates.insert(knd, self.K) plates = tuple(plates) return self.distribution.plates_to_parent(index-1, plates) def plates_from_parent(self, index, plates): """ Resolve the plate mapping from a parent. Given the plates of a parent's moments, this method returns the plates that the moments has for this distribution. """ if index == 0: return plates else: plates = self.distribution.plates_from_parent(index-1, plates) # Remove the cluster plate, if the parent has it plates = list(plates) if len(plates) >= abs(self.cluster_plate): plates.pop(self.cluster_plate) return tuple(plates) def random(self, *phi, plates=None): """ Draw a random sample from the distribution. """ return self.distribution.random(*phi, plates=plates) class Mixture(ExponentialFamily): r""" Node for exponential family mixture variables. The node represents a random variable which is sampled from a mixture distribution. It is possible to mix any exponential family distribution. The probability density function is .. math:: p(x|z=k,\boldsymbol{\theta}_0,\ldots,\boldsymbol{\theta}_{K-1}) = \phi(x|\boldsymbol{\theta}_k), where :math:`\phi` is the probability density function of the mixed exponential family distribution and :math:`\boldsymbol{\theta}_0, \ldots, \boldsymbol{\theta}_{K-1}` are the parameters of each cluster. For instance, :math:`\phi` could be the Gaussian probability density function :math:`\mathcal{N}` and :math:`\boldsymbol{\theta}_k = \{\boldsymbol{\mu}_k, \mathbf{\Lambda}_k\}` where :math:`\boldsymbol{\mu}_k` and :math:`\mathbf{\Lambda}_k` are the mean vector and precision matrix for cluster :math:`k`. Parameters ---------- z : categorical-like node or array :math:`z`, cluster assignment node_class : stochastic exponential family node class Mixed distribution params : types specified by the mixed distribution Parameters of the mixed distribution. If some parameters should vary between clusters, those parameters' plate axis `cluster_plate` should have a size which equals the number of clusters. For parameters with shared values, that plate axis should have length 1. At least one parameter should vary between clusters. cluster_plate : int, optional Negative integer defining which plate axis is used for the clusters in the parameters. That plate axis is ignored from the parameters when considering the plates for this node. By default, mix over the last plate axis. See also -------- Categorical, CategoricalMarkovChain Examples -------- A simple 2-dimensional Gaussian mixture model with three clusters for 100 samples can be constructed, for instance, as: >>> from bayespy.nodes import (Dirichlet, Categorical, Mixture, ... Gaussian, Wishart) >>> alpha = Dirichlet([1e-3, 1e-3, 1e-3]) >>> Z = Categorical(alpha, plates=(100,)) >>> mu = Gaussian(np.zeros(2), 1e-6*np.identity(2), plates=(3,)) >>> Lambda = Wishart(2, 1e-6*np.identity(2), plates=(3,)) >>> X = Mixture(Z, Gaussian, mu, Lambda) """ def __init__(self, z, node_class, *params, cluster_plate=-1, **kwargs): self.cluster_plate = cluster_plate super().__init__(z, node_class, *params, cluster_plate=cluster_plate, **kwargs) @classmethod def _constructor(cls, z, node_class, *args, cluster_plate=-1, **kwargs): """ Constructs distribution and moments objects. """ if cluster_plate >= 0: raise ValueError("Cluster plate axis must be negative") # Get the stuff for the mixed distribution (parents, _, dims, mixture_plates, distribution, moments, parent_moments) = \ node_class._constructor(*args) # Check that at least one of the parents has the cluster plate axis if len(mixture_plates) < abs(cluster_plate): raise ValueError("The mixed distribution does not have a plates " "axis for the cluster plate axis") # Resolve the number of clusters mixture_plates = list(mixture_plates) K = mixture_plates.pop(cluster_plate) # Convert a node to get the number of clusters z = cls._ensure_moments(z, CategoricalMoments(K)) if z.dims[0][0] != K: raise ValueError("Inconsistent number of clusters") plates = cls._total_plates(kwargs.get('plates'), mixture_plates, z.plates) ndims = [len(dim) for dim in dims] parents = [cls._ensure_moments(p_i, m_i) for (p_i, m_i) in zip(parents, parent_moments)] ndims_parents = [[len(dims_i) for dims_i in parent.dims] for parent in parents] # Convert the distribution to a mixture distribution = MixtureDistribution(distribution, cluster_plate, K, ndims, ndims_parents) # Add cluster assignments to parents parent_moments = [CategoricalMoments(K)] + list(parent_moments) parents = [z] + list(parents) return (parents, kwargs, dims, plates, distribution, moments, parent_moments) def integrated_logpdf_from_parents(self, x, index): """ Approximates the posterior predictive pdf \int p(x|parents) q(parents) dparents in log-scale as \int q(parents_i) exp( \int q(parents_\i) \log p(x|parents) dparents_\i ) dparents_i.""" if index == 0: # Integrate out the cluster assignments # First, integrate the cluster parameters in log-scale # compute_logpdf(cls, u, phi, g, f): # Shape(x) = [M1,..,Mm,N1,..,Nn,D1,..,Dd] u_parents = self._message_from_parents() # Shape(u) = [M1,..,Mm,N1,..,1,..,Nn,D1,..,Dd] # Shape(f) = [M1,..,Mm,N1,..,1,..,Nn] (u, f) = self._distribution.distribution.compute_fixed_moments_and_f(x) f = np.expand_dims(f, axis=self.cluster_plate) for i in range(len(u)): ndim_i = len(self.dims[i]) cluster_axis = self.cluster_plate - ndim_i u[i] = np.expand_dims(u[i], axis=cluster_axis) # Shape(phi) = [N1,..,K,..,Nn,D1,..,Dd] phi = self._distribution.distribution.compute_phi_from_parents(*(u_parents[1:])) # Shape(g) = [N1,..,K,..,Nn] g = self._distribution.distribution.compute_cgf_from_parents(*(u_parents[1:])) # Shape(lpdf) = [M1,..,Mm,N1,..,K,..,Nn] lpdf = self._distribution.distribution.compute_logpdf(u, phi, g, f, self.ndims) # From logpdf to pdf, but avoid over/underflow lpdf_max = np.max(lpdf, axis=self.cluster_plate, keepdims=True) pdf = np.exp(lpdf-lpdf_max) # Move cluster axis to be the last: # Shape(pdf) = [M1,..,Mm,N1,..,Nn,K] pdf = misc.moveaxis(pdf, self.cluster_plate, -1) # Cluster assignments/probabilities/weights # Shape(p) = [N1,..,Nn,K] p = u_parents[0][0] # Weighted average. TODO/FIXME: Use einsum! # Shape(pdf) = [M1,..,Mm,N1,..,Nn] pdf = np.sum(pdf * p, axis=self.cluster_plate) # Back to log-scale (add the overflow fix!) lpdf_max = np.squeeze(lpdf_max, axis=self.cluster_plate) lpdf = np.log(pdf) + lpdf_max return lpdf raise NotImplementedError() bayespy-0.4.0/bayespy/inference/vmp/nodes/__init__.py0000644000175000017500000000237412615643655024365 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2012 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ # Import some most commonly used nodes from . import * from .bernoulli import Bernoulli from .binomial import Binomial from .categorical import Categorical from .multinomial import Multinomial from .poisson import Poisson from .beta import Beta from .dirichlet import Dirichlet from .exponential import Exponential from .gaussian import Gaussian, GaussianARD from .wishart import Wishart from .gamma import Gamma, GammaShape from .gaussian import (GaussianGammaISO, GaussianGammaARD, GaussianWishart) from .gaussian_markov_chain import GaussianMarkovChain from .gaussian_markov_chain import VaryingGaussianMarkovChain from .gaussian_markov_chain import SwitchingGaussianMarkovChain from .categorical_markov_chain import CategoricalMarkovChain from .mixture import Mixture from .gate import Gate from .concatenate import Concatenate from .dot import Dot from .dot import SumMultiply from .add import Add from .logpdf import LogPDF from .constant import Constant bayespy-0.4.0/bayespy/inference/vmp/nodes/deterministic.py0000644000175000017500000002675312533543154025470 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2013-2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import functools import numpy as np from bayespy.utils import misc from .node import Node, Moments class Deterministic(Node): """ Base class for deterministic nodes. Sub-classes must implement: 1. For implementing the deterministic function: _compute_moments(self, *u) 2. One of the following options: a) Simple methods: _compute_message_to_parent(self, index, m, *u) not? _compute_mask_to_parent(self, index, mask) b) More control with: _compute_message_and_mask_to_parent(self, index, m, *u) Sub-classes may need to re-implement: 1. If they manipulate plates: _compute_mask_to_parent(index, mask) _compute_plates_to_parent(self, index, plates) _compute_plates_from_parent(self, index, plates) """ def __init__(self, *args, **kwargs): super().__init__(*args, plates=None, notify_parents=False, **kwargs) def _get_id_list(self): """ Returns the stochastic ID list. This method is used to check that same stochastic nodes are not direct parents of a node several times. It is only valid if there are intermediate stochastic nodes. To put it another way: each ID corresponds to one factor q(..) in the posterior approximation. Different IDs mean different factors, thus they mean independence. The parents must have independent factors. Stochastic nodes should return their unique ID. Deterministic nodes should return the IDs of their parents. Constant nodes should return empty list of IDs. """ id_list = [] for parent in self.parents: id_list = id_list + parent._get_id_list() return id_list def get_moments(self): u_parents = self._message_from_parents() return self._compute_moments(*u_parents) def _compute_message_and_mask_to_parent(self, index, m_children, *u_parents): # The following methods should be implemented by sub-classes. m = self._compute_message_to_parent(index, m_children, *u_parents) mask = self._compute_mask_to_parent(index, self.mask) return (m, mask) def _get_message_and_mask_to_parent(self, index): u_parents = self._message_from_parents(exclude=index) m_children = self._message_from_children() return self._compute_message_and_mask_to_parent(index, m_children, *u_parents) def _compute_moments(self, *u_parents): """ Compute the moments given the moments of the parents. """ raise NotImplementedError() def _compute_message_to_parent(self, index, m_children, *u_parents): """ Compute the message to a parent. """ raise NotImplementedError() def _add_child(self, child, index): """ Add a child node. Only child nodes that are stochastic (or have stochastic children recursively) are counted as children because deterministic nodes without stochastic children do not have any messages to send so the parents do not need to know about the deterministic node. A deterministic node does not notify its parents when created, but if it gets a stochastic child node, then notify parents. This method is called only if a stochastic child (recursively) node is added, thus there is at least one stochastic node below this deterministic node. Parameters ---------- child : node index : int The parent index of this node for the child node. The child node recognizes its parents by their index number. """ super()._add_child(child, index) # Now that this deterministic node has non-deterministic children, # notify parents for (ind,parent) in enumerate(self.parents): parent._add_child(self, ind) def _remove_child(self, child, index): """ Remove a child node. Only child nodes that are stochastic (or have stochastic children recursively) are counted as children because deterministic nodes without stochastic children do not have any messages to send so the parents do not need to know about the deterministic node. So, if the deterministic node does not have any stochastic children left after removal, remove it from its parents. """ super()._remove_child(child, index) # Check whether there are any children left. If not, remove from parents if len(self.children) == 0: for (ind, parent) in enumerate(self.parents): parent._remove_child(self, ind) def lower_bound_contribution(self, gradient=False, **kwargs): # Deterministic functions are delta distributions so the lower bound # contribuion is zero. return 0 def tile(X, tiles): """ Tile the plates of the input node. x = [a,b,c] y = tile(x, 2) = [a,b,c,a,b,c] There should be no need to tile plates that have unit length because they are handled properly by the broadcasting rules already. Parameters ---------- X : Node Input node to be tiled. tiles : int, tuple Tiling of the plates (broadcasting rules for plates apply). See also -------- numpy.tile """ # Make sure `tiles` is tuple (even if an integer is given) tiles = tuple(np.ravel(tiles)) class _Tile(Deterministic): _parent_moments = (Moments(),) def __init__(self, X, **kwargs): self._moments = X._moments super().__init__(X, dims=X.dims, **kwargs) def _compute_plates_to_parent(self, index, plates): plates = list(plates) for i in range(-len(tiles), 0): plates[i] = plates[i] // tiles[i] return tuple(plates) def _compute_plates_from_parent(self, index, plates): return tuple(misc.multiply_shapes(plates, tiles)) def _compute_mask_to_parent(self, index, mask): # Idea: Reshape the message array such that every other axis # will be summed and every other kept. # Make plates equal length plates = self._plates_to_parent(index) shape_m = np.shape(mask) (plates, tiles_m, shape_m) = misc.make_equal_length(plates, tiles, shape_m) # Handle broadcasting rules for axes that have unit length in # the message (although the plate may be non-unit length). Also, # compute the corresponding broadcasting_multiplier. plates = list(plates) tiles_m = list(tiles_m) for j in range(len(plates)): if shape_m[j] == 1: plates[j] = 1 tiles_m[j] = 1 # Combine the tuples by picking every other from tiles_ind and # every other from shape shape = functools.reduce(lambda x,y: x+y, zip(tiles_m, plates)) # ..and reshape the array, that is, every other axis corresponds # to tiles and every other to plates/dimensions in parents mask = np.reshape(mask, shape) # Sum over every other axis axes = tuple(range(0,len(shape),2)) mask = np.any(mask, axis=axes) # Remove extra leading axes ndim_parent = len(self.parents[index].plates) mask = misc.squeeze_to_dim(mask, ndim_parent) return mask def _compute_message_to_parent(self, index, m, u_X): m = list(m) for ind in range(len(m)): # Idea: Reshape the message array such that every other axis # will be summed and every other kept. shape_ind = self._plates_to_parent(index) + self.dims[ind] # Add variable dimensions to tiles tiles_ind = tiles + (1,)*len(self.dims[ind]) # Make shape tuples equal length shape_m = np.shape(m[ind]) (tiles_ind, shape, shape_m) = misc.make_equal_length(tiles_ind, shape_ind, shape_m) # Handle broadcasting rules for axes that have unit length in # the message (although the plate may be non-unit length). Also, # compute the corresponding broadcasting multiplier. r = 1 shape = list(shape) tiles_ind = list(tiles_ind) for j in range(len(shape)): if shape_m[j] == 1: r *= tiles_ind[j] shape[j] = 1 tiles_ind[j] = 1 # Combine the tuples by picking every other from tiles_ind and # every other from shape shape = functools.reduce(lambda x,y: x+y, zip(tiles_ind, shape)) # ..and reshape the array, that is, every other axis corresponds # to tiles and every other to plates/dimensions in parents m[ind] = np.reshape(m[ind], shape) # Sum over every other axis axes = tuple(range(0,len(shape),2)) m[ind] = r * np.sum(m[ind], axis=axes) # Remove extra leading axes ndim_parent = len(self.parents[index].get_shape(ind)) m[ind] = misc.squeeze_to_dim(m[ind], ndim_parent) return m def _compute_moments(self, u_X): """ Tile the plates of the parent's moments. """ # Utilize broadcasting: If a tiled axis is unit length in u_X, there # is no need to tile it. u = list() for ind in range(len(u_X)): ui = u_X[ind] shape_u = np.shape(ui) if np.ndim(ui) > 0: # Add variable dimensions tiles_ind = tiles + (1,)*len(self.dims[ind]) # Utilize broadcasting: Do not tile leading empty axes nd = min(len(tiles_ind), np.ndim(ui)) tiles_ind = tiles_ind[(-nd):] # For simplicity, make tiles and shape equal length (tiles_ind, shape_u) = misc.make_equal_length(tiles_ind, shape_u) # Utilize broadcasting: Use tiling only if the parent's # moment has non-unit axis length. tiles_ind = [tile if sh > 1 else 1 for (tile, sh) in zip(tiles_ind, shape_u)] # Tile ui = np.tile(ui, tiles_ind) u.append(ui) return u return _Tile(X, name="tile(%s, %s)" % (X.name, tiles)) bayespy-0.4.0/bayespy/inference/vmp/nodes/point_estimate.py0000644000175000017500000001214612615643655025650 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2015 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import numpy as np from .node import Node, Moments from .stochastic import Stochastic from .deterministic import Deterministic class DeltaMoments(Moments): r""" Class for the moments of constants or delta distributed variables """ def compute_fixed_moments(self, x): r""" Compute the moments for a fixed value """ return [x] def compute_dims_from_values(self, x): r""" Return the shape of the moments for a fixed value. """ return ((),) class DeltaToAny(Deterministic): r""" Special converter of delta moments to any moments """ def __init__(self, X, moments): r""" """ self._moments = moments self._parent_moments = [DeltaMoments()] #(plates, dims) = moments.compute_plates_and_dims(X.get_shape(0)) dims = moments.compute_dims_from_shape(X.get_shape(0)) super().__init__(X, dims=dims, **kwargs) def _compute_moments(self, u_X): r""" """ x = u_X[0] return self._moments.compute_fixed_moments(x) def _compute_message_to_parent(self, index, m_child, u_X): r""" """ # Convert child message array to a gradient function raise NotImplementedError() if index == 0: m = m_child[:2] return m else: raise ValueError("Invalid parent index") def _compute_mask_to_parent(self, index, mask): r""" """ raise NotImplementedError() if index == 0: raise NotImplementedError() else: raise ValueError("Invalid parent index") def _plates_to_parent(self, index): r""" """ raise NotImplementedError() if index == 0: self.get_shape(0) raise NotImplementedError() else: raise ValueError("Invalid parent index") def _plates_from_parent(self, index): r""" """ raise NotImplementedError() if index == 0: return self.__cached_plates raise NotImplementedError() else: raise ValueError("Invalid parent index") class Scalar(Stochastic): def __init__(self, plates=None): dims = [()] raise NotImplementedError() def get_riemannian_gradient(self): m_children = self._message_from_children() g = self.annealing * m_children[0] return g def get_gradient(self, rg): return rg def get_parameters(self): return self.u def set_parameters(self, x): if len(x) != 1: raise Exception("Wrong number of parameters. Should be 1, is {0}".format(len(x))) self.u = [x[0]] return class PositiveScalar(Stochastic): pass class Constant(Node): r""" Node for presenting constant values. The node wraps arrays into proper node type. """ def __init__(self, moments, x, **kwargs): if not isinstance(moments, Moments) and issubclass(moments, Moments): raise ValueError("Give moments as an object instance instead of a class") self._moments = moments x = np.asanyarray(x) # Compute moments self.u = self._moments.compute_fixed_moments(x) # Dimensions of the moments dims = self._moments.compute_dims_from_values(x) # Resolve plates D = len(dims[0]) if D > 0: plates = np.shape(self.u[0])[:-D] else: plates = np.shape(self.u[0]) # Parent constructor super().__init__(dims=dims, plates=plates, **kwargs) def _get_id_list(self): """ Returns the stochastic ID list. This method is used to check that same stochastic nodes are not direct parents of a node several times. It is only valid if there are intermediate stochastic nodes. To put it another way: each ID corresponds to one factor q(..) in the posterior approximation. Different IDs mean different factors, thus they mean independence. The parents must have independent factors. Stochastic nodes should return their unique ID. Deterministic nodes should return the IDs of their parents. Constant nodes should return empty list of IDs. """ return [] def get_moments(self): return self.u def set_value(self, x): x = np.asanyarray(x) shapes = [np.shape(ui) for ui in self.u] self.u = self._moments.compute_fixed_moments(x) for (i, shape) in enumerate(shapes): if np.shape(self.u[i]) != shape: raise ValueError("Incorrect shape for the array") def lower_bound_contribution(self, gradient=False, **kwargs): # Deterministic functions are delta distributions so the lower bound # contribuion is zero. return 0 bayespy-0.4.0/bayespy/inference/vmp/nodes/wishart.py0000644000175000017500000001511012556144617024275 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2012,2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import numpy as np import scipy.special as special from bayespy.utils import misc, linalg from .expfamily import ExponentialFamily from .expfamily import ExponentialFamilyDistribution from .expfamily import useconstructor from .constant import Constant from .node import Moments, Node class WishartPriorMoments(Moments): def __init__(self, k): self.k = k return def compute_fixed_moments(self, n): """ Compute moments for fixed x. """ u0 = np.asanyarray(n) u1 = special.multigammaln(0.5*u0, self.k) return [u0, u1] def compute_dims_from_values(self, n): """ Compute the dimensions of phi or u. """ return ( (), () ) class WishartMoments(Moments): def compute_fixed_moments(self, Lambda): """ Compute moments for fixed x. """ ldet = linalg.chol_logdet(linalg.chol(Lambda)) u = [Lambda, ldet] return u def compute_dims_from_values(self, x): """ Compute the dimensions of phi and u. """ if np.ndim(x) < 2: raise ValueError("Values for Wishart distribution must be at least " "2-D arrays.") if np.shape(x)[-1] != np.shape(x)[-2]: raise ValueError("Values for Wishart distribution must be square " "matrices, thus the two last axes must have equal " "length.") d = np.shape(x)[-1] return ( (d,d), () ) class WishartDistribution(ExponentialFamilyDistribution): """ Sub-classes implement distribution specific computations. Distribution for :math:`k \times k` symmetric positive definite matrix. .. math:: \Lambda \sim \mathcal{W}(n, V) Note: :math:`V` is inverse scale matrix. .. math:: p(\Lambda | n, V) = .. """ def compute_message_to_parent(self, parent, index, u_self, u_n, u_V): if index == 0: raise NotImplementedError("Message from Wishart to degrees of " "freedom parameter (first parent) " "not yet implemented") elif index == 1: Lambda = u_self[0] n = u_n[0] return [-0.5 * Lambda, 0.5 * n] else: raise ValueError("Invalid parent index {0}".format(index)) def compute_phi_from_parents(self, u_n, u_V, mask=True): r""" Compute natural parameters .. math:: \phi(n, V) = \begin{bmatrix} -\frac{1}{2} V \\ \frac{1}{2} n \end{bmatrix} """ return [-0.5 * u_V[0], 0.5 * u_n[0]] def compute_moments_and_cgf(self, phi, mask=True): r""" Return moments and cgf for given natural parameters .. math:: \langle u \rangle = \begin{bmatrix} \phi_2 (-\phi_1)^{-1} \\ -\log|-\phi_1| + \psi_k(\phi_2) \end{bmatrix} \\ g(\phi) = \phi_2 \log|-\phi_1| - \log \Gamma_k(\phi_2) """ U = linalg.chol(-phi[0]) k = np.shape(phi[0])[-1] #k = self.dims[0][0] logdet_phi0 = linalg.chol_logdet(U) u0 = phi[1][...,np.newaxis,np.newaxis] * linalg.chol_inv(U) u1 = -logdet_phi0 + misc.multidigamma(phi[1], k) u = [u0, u1] g = phi[1] * logdet_phi0 - special.multigammaln(phi[1], k) return (u, g) def compute_cgf_from_parents(self, u_n, u_V): r""" CGF from parents .. math:: g(n, V) = \frac{n}{2} \log|V| - \frac{nk}{2} \log 2 - \log \Gamma_k(\frac{n}{2}) """ n = u_n[0] gammaln_n = u_n[1] V = u_V[0] logdet_V = u_V[1] k = np.shape(V)[-1] g = 0.5*n*logdet_V - 0.5*k*n*np.log(2) - gammaln_n return g def compute_fixed_moments_and_f(self, Lambda, mask=True): r""" Compute u(x) and f(x) for given x. .. math: u(\Lambda) = \begin{bmatrix} \Lambda \\ \log |\Lambda| \end{bmatrix} """ k = np.shape(Lambda)[-1] ldet = linalg.chol_logdet(linalg.chol(Lambda)) u = [Lambda, ldet] f = -(k+1)/2 * ldet return (u, f) class Wishart(ExponentialFamily): r""" Node for Wishart random variables. The random variable :math:`\mathbf{\Lambda}` is a :math:`D\times{}D` positive-definite symmetric matrix. .. math:: p(\mathbf{\Lambda}) = \mathrm{Wishart}(\mathbf{\Lambda} | N, \mathbf{V}) Parameters ---------- n : scalar or array :math:`N`, degrees of freedom, :math:`N>D-1`. V : Wishart-like node or (...,D,D)-array :math:`\mathbf{V}`, scale matrix. """ _distribution = WishartDistribution() _moments = WishartMoments() def __init__(self, n, V, **kwargs): """ Create Wishart node. """ super().__init__(n, V, **kwargs) @classmethod def _constructor(cls, n, V, **kwargs): """ Constructs distribution and moments objects. """ # Make V a proper parent node and get the dimensionality of the matrix V = cls._ensure_moments(V, WishartMoments()) k = V.dims[0][-1] # Parent node message types parent_moments = (WishartPriorMoments(k), WishartMoments()) # Dimensionality of the natural parameters dims = ( (k,k), () ) n = cls._ensure_moments(n, parent_moments[0]) parents = [n, V] return (parents, kwargs, dims, cls._total_plates(kwargs.get('plates'), cls._distribution.plates_from_parent(0, n.plates), cls._distribution.plates_from_parent(1, V.plates)), cls._distribution, cls._moments, parent_moments) def __str__(self): n = 2*self.phi[1] A = 0.5 * self.u[0] / self.phi[1][...,np.newaxis,np.newaxis] return ("%s ~ Wishart(n, A)\n" " n =\n" "%s\n" " A =\n" "%s\n" % (self.name, n, A)) bayespy-0.4.0/bayespy/inference/vmp/nodes/gaussian.py0000644000175000017500000024142212610223512024414 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Module for the Gaussian distribution and similar distributions. """ import numpy as np from scipy import special from bayespy.utils import (random, misc, linalg) from bayespy.utils.linalg import dot, mvdot from .expfamily import (ExponentialFamily, ExponentialFamilyDistribution, useconstructor) from .wishart import (WishartMoments, WishartPriorMoments) from .gamma import (GammaMoments, GammaPriorMoments) from .deterministic import Deterministic from .node import (Moments, ensureparents) # # MOMENTS # ## class GaussianISOMoments(Moments): ## """ ## Class for the moments of Gaussian ISO variables. ## """ ## def __init__(self, ndim): ## self.ndim = ndim ## super().__init__() ## def compute_fixed_moments(self, x): ## """ ## Compute the moments for a fixed value ## """ ## if np.ndim(x) < self.ndim: ## raise ValueError("Not enough dimensions in x") ## u0 = x ## u1 = linalg.sum_multiply(x, x, axis=tuple(range(-self.ndim,0))) ## u = [u0, u1] ## return u ## def compute_dims_from_values(self, x): ## """ ## Return the shape of the moments for a fixed value. ## """ ## x = misc.atleast_nd(x, self.ndim) ## if self.ndim == 0: ## shape = () ## else: ## shape = np.shape(x)[-self.ndim:] ## return (shape, ()) ## class GaussianARDMoments(Moments): ## """ ## Class for the moments of Gaussian ARD variables. ## """ ## def __init__(self): ## super().__init__() ## def compute_fixed_moments(self, x): ## """ ## Compute the moments for a fixed value ## """ ## u0 = x ## u1 = x**2 ## u = [u0, u1] ## return u ## def compute_dims_from_values(self, x): ## """ ## Return the shape of the moments for a fixed value. ## """ ## return ((), ()) class GaussianMoments(Moments): r""" Class for the moments of Gaussian variables. """ def __init__(self, ndim): self.ndim = ndim super().__init__() def compute_fixed_moments(self, x): r""" Compute the moments for a fixed value """ x = misc.atleast_nd(x, self.ndim) return [x, linalg.outer(x, x, ndim=self.ndim)] def compute_dims_from_values(self, x): r""" Return the shape of the moments for a fixed value. """ x = misc.atleast_nd(x, self.ndim) if self.ndim == 0: shape = () else: shape = np.shape(x)[-self.ndim:] return (shape, shape+shape) class GaussianGammaISOMoments(Moments): r""" Class for the moments of Gaussian-gamma-ISO variables. """ def __init__(self, ndim): r""" Create moments object for Gaussian-gamma isotropic variables ndim=0: scalar ndim=1: vector ndim=2: matrix ... """ self.ndim = ndim super().__init__() def compute_fixed_moments(self, x, alpha): r""" Compute the moments for a fixed value `x` is a mean vector. `alpha` is a precision scale """ x = np.asanyarray(x) alpha = np.asanyarray(alpha) u0 = x * misc.add_trailing_axes(alpha, self.ndim) u1 = (linalg.outer(x, x, ndim=self.ndim) * misc.addtrailing_axes(alpha, 2*self.ndim)) u2 = np.copy(alpha) u3 = np.log(alpha) u = [u0, u1, u2, u3] return u def compute_dims_from_values(self, x, alpha): r""" Return the shape of the moments for a fixed value. """ if np.ndim(x) < 1: raise ValueError("Mean must be a vector") if ndim == 0: return ( (), (), (), () ) dims = np.shape(x)[-ndim:] return ( dims, 2*dims, (), () ) class GaussianGammaARDMoments(Moments): r""" Class for the moments of Gaussian-gamma-ARD variables. """ def __init__(self, ndim): r""" Create moments object for Gaussian-gamma isotropic variables ndim=0: scalar ndim=1: vector ndim=2: matrix ... """ self.ndim = ndim super().__init__() def compute_fixed_moments(self, x, alpha): r""" Compute the moments for a fixed value `x` is a mean vector. `alpha` is a precision scale """ x = np.asanyarray(x) alpha = np.asanyarray(alpha) if np.ndim(x) < self.ndim: raise ValueError("Not enough dimensions in x") if np.ndim(alpha) < self.ndim: raise ValueError("Not enough dimensions in alpha") if np.shape(x) != np.shape(alpha): raise ValueError("Mean and ARD scales have inconsistent shapes") u0 = alpha * x u1 = u0 * x u2 = np.copy(alpha) u3 = np.log(alpha) u = [u0, u1, u2, u3] return u def compute_dims_from_values(self, x, alpha): r""" Return the shape of the moments for a fixed value. """ if np.ndim(x) < self.ndim: raise ValueError("Not enough dimensions in x") if np.ndim(alpha) < self.ndim: raise ValueError("Not enough dimensions in alpha") if np.shape(x) != np.shape(alpha): raise ValueError("Mean and ARD scales have inconsistent shapes") if ndim > 0: shape = np.shape(x)[-self.ndim:] else: shape = () return ( shape, shape, shape, shape ) class GaussianWishartMoments(Moments): r""" Class for the moments of Gaussian-Wishart variables. """ def compute_fixed_moments(self, x, Lambda): r""" Compute the moments for a fixed value `x` is a vector. `Lambda` is a precision matrix """ x = np.asanyarray(x) Lambda = np.asanyarray(Lambda) u0 = np.einsum('...ik,...k->...i', Lambda, x) u1 = np.einsum('...i,...ij,...j->...', x, Lambda, x) u2 = np.copy(Lambda) u3 = linalg.logdet_cov(Lambda) return [u0, u1, u2, u3] def compute_dims_from_values(self, x, Lambda): r""" Return the shape of the moments for a fixed value. """ if np.ndim(x) < 1: raise ValueError("Mean must be a vector") if np.ndim(Lambda) < 2: raise ValueError("Precision must be a matrix") D = np.shape(x)[-1] if np.shape(Lambda)[-2:] != (D,D): raise ValueError("Mean vector and precision matrix have " "inconsistent shapes") return ( (D,), (), (D,D), () ) # # DISTRIBUTIONS # class GaussianDistribution(ExponentialFamilyDistribution): r""" Class for the VMP formulas of Gaussian variables. Currently, supports only vector variables. Notes ----- Message passing equations: .. math:: \mathbf{x} &\sim \mathcal{N}(\boldsymbol{\mu}, \mathbf{\Lambda}), .. math:: \mathbf{x},\boldsymbol{\mu} \in \mathbb{R}^{D}, \quad \mathbf{\Lambda} \in \mathbb{R}^{D \times D}, \quad \mathbf{\Lambda} \text{ symmetric positive definite} .. math:: \log\mathcal{N}( \mathbf{x} | \boldsymbol{\mu}, \mathbf{\Lambda} ) &= - \frac{1}{2} \mathbf{x}^{\mathrm{T}} \mathbf{\Lambda} \mathbf{x} + \mathbf{x}^{\mathrm{T}} \mathbf{\Lambda} \boldsymbol{\mu} - \frac{1}{2} \boldsymbol{\mu}^{\mathrm{T}} \mathbf{\Lambda} \boldsymbol{\mu} + \frac{1}{2} \log |\mathbf{\Lambda}| - \frac{D}{2} \log (2\pi) """ def compute_message_to_parent(self, parent, index, u, u_mu_Lambda): r""" Compute the message to a parent node. .. math:: \boldsymbol{\phi}_{\boldsymbol{\mu}} (\mathbf{x}, \mathbf{\Lambda}) &= \left[ \begin{matrix} \mathbf{\Lambda} \mathbf{x} \\ - \frac{1}{2} \mathbf{\Lambda} \end{matrix} \right] \\ \boldsymbol{\phi}_{\mathbf{\Lambda}} (\mathbf{x}, \boldsymbol{\mu}) &= \left[ \begin{matrix} - \frac{1}{2} \mathbf{xx}^{\mathrm{T}} + \frac{1}{2} \mathbf{x}\boldsymbol{\mu}^{\mathrm{T}} + \frac{1}{2} \boldsymbol{\mu}\mathbf{x}^{\mathrm{T}} - \frac{1}{2} \boldsymbol{\mu\mu}^{\mathrm{T}} \\ \frac{1}{2} \end{matrix} \right] """ if index == 0: x = u[0] xx = u[1] m0 = x m1 = -0.5 m2 = -0.5*xx m3 = 0.5 return [m0, m1, m2, m3] else: raise ValueError("Index out of bounds") def compute_phi_from_parents(self, u_mu_Lambda, mask=True): r""" Compute the natural parameter vector given parent moments. .. math:: \boldsymbol{\phi} (\boldsymbol{\mu}, \mathbf{\Lambda}) &= \left[ \begin{matrix} \mathbf{\Lambda} \boldsymbol{\mu} \\ - \frac{1}{2} \mathbf{\Lambda} \end{matrix} \right] """ Lambda_mu = u_mu_Lambda[0] Lambda = u_mu_Lambda[2] return [Lambda_mu, -0.5 * Lambda] def compute_moments_and_cgf(self, phi, mask=True): r""" Compute the moments and :math:`g(\phi)`. .. math:: \overline{\mathbf{u}} (\boldsymbol{\phi}) &= \left[ \begin{matrix} - \frac{1}{2} \boldsymbol{\phi}^{-1}_2 \boldsymbol{\phi}_1 \\ \frac{1}{4} \boldsymbol{\phi}^{-1}_2 \boldsymbol{\phi}_1 \boldsymbol{\phi}^{\mathrm{T}}_1 \boldsymbol{\phi}^{-1}_2 - \frac{1}{2} \boldsymbol{\phi}^{-1}_2 \end{matrix} \right] \\ g_{\boldsymbol{\phi}} (\boldsymbol{\phi}) &= \frac{1}{4} \boldsymbol{\phi}^{\mathrm{T}}_1 \boldsymbol{\phi}^{-1}_2 \boldsymbol{\phi}_1 + \frac{1}{2} \log | -2 \boldsymbol{\phi}_2 | """ # TODO: Compute -2*phi[1] and simplify the formulas L = linalg.chol(-2*phi[1]) k = np.shape(phi[0])[-1] # Moments u0 = linalg.chol_solve(L, phi[0]) u1 = misc.m_outer(u0, u0) + linalg.chol_inv(L) u = [u0, u1] # G g = (-0.5 * np.einsum('...i,...i', u[0], phi[0]) + 0.5 * linalg.chol_logdet(L)) #+ 0.5 * np.log(2) * self.dims[0][0]) return (u, g) def compute_cgf_from_parents(self, u_mu_Lambda): r""" Compute :math:`\mathrm{E}_{q(p)}[g(p)]` .. math:: g (\boldsymbol{\mu}, \mathbf{\Lambda}) &= - \frac{1}{2} \operatorname{tr}(\boldsymbol{\mu\mu}^{\mathrm{T}} \mathbf{\Lambda} ) + \frac{1}{2} \log |\mathbf{\Lambda}| """ mu_Lambda_mu = u_mu_Lambda[1] logdet_Lambda = u_mu_Lambda[3] g = -0.5*mu_Lambda_mu + 0.5*logdet_Lambda return g def compute_fixed_moments_and_f(self, x, mask=True): r""" Compute the moments and :math:`f(x)` for a fixed value. .. math:: \mathbf{u} (\mathbf{x}) &= \left[ \begin{matrix} \mathbf{x} \\ \mathbf{xx}^{\mathrm{T}} \end{matrix} \right] \\ f(\mathbf{x}) &= - \frac{D}{2} \log(2\pi) """ k = np.shape(x)[-1] u = [x, misc.m_outer(x,x)] f = -k/2*np.log(2*np.pi) return (u, f) def compute_gradient(self, g, u, phi): r""" Compute the standard gradient with respect to the natural parameters. Gradient of the moments: .. math:: \mathrm{d}\overline{\mathbf{u}} &= \begin{bmatrix} \frac{1}{2} \phi_2^{-1} \mathrm{d}\phi_2 \phi_2^{-1} \phi_1 - \frac{1}{2} \phi_2^{-1} \mathrm{d}\phi_1 \\ - \frac{1}{4} \phi_2^{-1} \mathrm{d}\phi_2 \phi_2^{-1} \phi_1 \phi_1^{\mathrm{T}} \phi_2^{-1} - \frac{1}{4} \phi_2^{-1} \phi_1 \phi_1^{\mathrm{T}} \phi_2^{-1} \mathrm{d}\phi_2 \phi_2^{-1} + \frac{1}{2} \phi_2^{-1} \mathrm{d}\phi_2 \phi_2^{-1} + \frac{1}{4} \phi_2^{-1} \mathrm{d}\phi_1 \phi_1^{\mathrm{T}} \phi_2^{-1} + \frac{1}{4} \phi_2^{-1} \phi_1 \mathrm{d}\phi_1^{\mathrm{T}} \phi_2^{-1} \end{bmatrix} \\ &= \begin{bmatrix} 2 (\overline{u}_2 - \overline{u}_1 \overline{u}_1^{\mathrm{T}}) \mathrm{d}\phi_2 \overline{u}_1 + (\overline{u}_2 - \overline{u}_1 \overline{u}_1^{\mathrm{T}}) \mathrm{d}\phi_1 \\ u_2 d\phi_2 u_2 - 2 u_1 u_1^T d\phi_2 u_1 u_1^T + 2 (u_2 - u_1 u_1^T) d\phi_1 u_1^T \end{bmatrix} Standard gradient given the gradient with respect to the moments, that is, given the Riemannian gradient :math:`\tilde{\nabla}`: .. math:: \nabla = \begin{bmatrix} (\overline{u}_2 - \overline{u}_1 \overline{u}_1^{\mathrm{T}}) \tilde{\nabla}_1 + 2 (u_2 - u_1 u_1^T) \tilde{\nabla}_2 u_1 \\ (u_2 - u_1 u_1^T) \tilde{\nabla}_1 u_1^T + u_1 \tilde{\nabla}_1^T (u_2 - u_1 u_1^T) + 2 u_2 \tilde{\nabla}_2 u_2 - 2 u_1 u_1^T \tilde{\nabla}_2 u_1 u_1^T \end{bmatrix} """ ndim = 1 x = u[0] xx = u[1] # Some helpful variables x_x = linalg.outer(x, x, ndim=ndim) Cov = xx - x_x cov_g0 = linalg.mvdot(Cov, g[0], ndim=ndim) cov_g0_x = linalg.outer(cov_g0, x, ndim=ndim) g1_x = linalg.mvdot(g[1], x, ndim=ndim) # Compute gradient terms d0 = cov_g0 + 2 * linalg.mvdot(Cov, g1_x, ndim=ndim) d1 = (cov_g0_x + linalg.transpose(cov_g0_x, ndim=ndim) + 2 * linalg.mmdot(xx, linalg.mmdot(g[1], xx, ndim=ndim), ndim=ndim) - 2 * x_x * misc.add_trailing_axes(linalg.inner(g1_x, x, ndim=ndim), 2*ndim)) return [d0, d1] def random(self, *phi, plates=None): r""" Draw a random sample from the distribution. """ # TODO/FIXME: You shouldn't draw random values for # observed/fixed elements! # Note that phi[1] is -0.5*inv(Cov) U = linalg.chol(-2*phi[1]) mu = linalg.chol_solve(U, phi[0]) shape = plates + np.shape(mu)[-1:] z = np.random.randn(*shape) # Denote Lambda = -2*phi[1] # Then, Cov = inv(Lambda) = inv(U'*U) = inv(U) * inv(U') # Thus, compute mu + U\z z = misc.m_solve_triangular(U, z, trans='N', lower=False) return mu + z class GaussianARDDistribution(ExponentialFamilyDistribution): r""" ... Log probability density function: .. math:: \log p(x|\mu, \alpha) = -\frac{1}{2} x^T \mathrm{diag}(\alpha) x + x^T \mathrm{diag}(\alpha) \mu - \frac{1}{2} \mu^T \mathrm{diag}(\alpha) \mu + \frac{1}{2} \sum_i \log \alpha_i - \frac{D}{2} \log(2\pi) Parent has moments: .. math:: \begin{bmatrix} \alpha \circ \mu \\ \alpha \circ \mu \circ \mu \\ \alpha \\ \log(\alpha) \end{bmatrix} """ def __init__(self, shape, ndim_mu): self.shape = shape self.ndim_mu = ndim_mu self.ndim = len(shape) super().__init__() def compute_message_to_parent(self, parent, index, u, u_mu_alpha): r""" ... .. math:: m = \begin{bmatrix} x \\ [-\frac{1}{2}, \ldots, -\frac{1}{2}] \\ -\frac{1}{2} \mathrm{diag}(xx^T) \\ [\frac{1}{2}, \ldots, \frac{1}{2}] \end{bmatrix} """ if index == 0: x = u[0] x2 = misc.get_diag(u[1], ndim=self.ndim) m0 = x m1 = -0.5 * np.ones(self.shape) m2 = -0.5 * x2 m3 = 0.5 * np.ones(self.shape) m = [m0, m1, m2, m3] return m ## if index == 0: ## x = u[0] ## alpha = u_alpha[0] ## axes0 = list(range(-self.ndim, -self.ndim_mu)) ## m0 = misc.sum_multiply(alpha, x, axis=axes0) ## Alpha = misc.diag(alpha, ndim=self.ndim) ## axes1 = [axis+self.ndim for axis in axes0] + axes0 ## m1 = -0.5 * misc.sum_multiply(Alpha, ## misc.identity(*self.shape), ## axis=axes1) ## return [m0, m1] ## elif index == 1: ## x = u[0] ## x2 = misc.get_diag(u[1], ndim=self.ndim) ## mu = u_mu[0] ## mu2 = misc.get_diag(u_mu[1], ndim=self.ndim_mu) ## if self.ndim_mu == 0: ## mu_shape = np.shape(mu) + (1,)*self.ndim ## else: ## mu_shape = (np.shape(mu)[:-self.ndim_mu] ## + (1,)*(self.ndim-self.ndim_mu) ## + np.shape(mu)[-self.ndim_mu:]) ## mu = np.reshape(mu, mu_shape) ## mu2 = np.reshape(mu2, mu_shape) ## m0 = -0.5*x2 + x*mu - 0.5*mu2 ## m1 = 0.5 ## return [m0, m1] else: raise ValueError("Invalid parent index") def compute_mask_to_parent(self, index, mask): r""" Maps the mask to the plates of a parent. """ if index == 0: if self.ndim_mu == self.ndim: return mask elif self.ndim_mu < self.ndim: diff = self.ndim - self.ndim_mu return misc.add_trailing_axes(mask, diff) else: raise RuntimeError("Parent's ndim is larger") else: raise ValueError("Invalid parent index") def compute_phi_from_parents(self, u_mu_alpha, mask=True): alpha_mu = u_mu_alpha[0] alpha = u_mu_alpha[2] #mu = u_mu[0] #alpha = u_alpha[0] ## if np.ndim(mu) < self.ndim_mu: ## raise ValueError("Moment of mu does not have enough dimensions") ## mu = misc.add_axes(mu, ## axis=np.ndim(mu)-self.ndim_mu, ## num=self.ndim-self.ndim_mu) phi0 = alpha_mu phi1 = -0.5 * alpha if self.ndim > 0: # Ensure that phi is not using broadcasting for variable # dimension axes ones = np.ones(self.shape) phi0 = ones * phi0 phi1 = ones * phi1 # Make a diagonal matrix phi1 = misc.diag(phi1, ndim=self.ndim) return [phi0, phi1] def compute_moments_and_cgf(self, phi, mask=True): if self.ndim == 0: # Use scalar equations u0 = -phi[0] / (2*phi[1]) u1 = u0**2 - 1 / (2*phi[1]) u = [u0, u1] g = (-0.5 * u[0] * phi[0] + 0.5 * np.log(-2*phi[1])) # TODO/FIXME: You could use these equations if phi is a scalar # in practice although ndim>0 (because the shape can be, e.g., # (1,1,1,1) for ndim=4). else: # Reshape to standard vector and matrix D = np.prod(self.shape) phi0 = np.reshape(phi[0], phi[0].shape[:-self.ndim] + (D,)) phi1 = np.reshape(phi[1], phi[1].shape[:-2*self.ndim] + (D,D)) # Compute the moments L = linalg.chol(-2*phi1) Cov = linalg.chol_inv(L) u0 = linalg.chol_solve(L, phi0) u1 = linalg.outer(u0, u0) + Cov # Compute CGF g = (- 0.5 * np.einsum('...i,...i', u0, phi0) + 0.5 * linalg.chol_logdet(L)) # Reshape to arrays u0 = np.reshape(u0, u0.shape[:-1] + self.shape) u1 = np.reshape(u1, u1.shape[:-2] + self.shape + self.shape) u = [u0, u1] return (u, g) def compute_cgf_from_parents(self, u_mu_alpha): r""" Compute the value of the cumulant generating function. """ # Compute sum(mu^2 * alpha) correctly for broadcasted shapes alpha_mu2 = u_mu_alpha[1] logdet_alpha = u_mu_alpha[3] axes = tuple(range(-self.ndim, 0)) # TODO/FIXME: You could use plate multiplier type of correction instead # of explicitly broadcasting with ones. if self.ndim > 0: alpha_mu2 = misc.sum_multiply(alpha_mu2, np.ones(self.shape), axis=axes) if self.ndim > 0: logdet_alpha = misc.sum_multiply(logdet_alpha, np.ones(self.shape), axis=axes) # Compute g g = -0.5*alpha_mu2 + 0.5*logdet_alpha return g def compute_fixed_moments_and_f(self, x, mask=True): r""" Compute u(x) and f(x) for given x. """ if self.ndim > 0 and np.shape(x)[-self.ndim:] != self.shape: raise ValueError("Invalid shape") k = np.prod(self.shape) u = [x, linalg.outer(x, x, ndim=self.ndim)] f = -k/2*np.log(2*np.pi) return (u, f) def plates_to_parent(self, index, plates): r""" Resolves the plate mapping to a parent. Given the plates of the node's moments, this method returns the plates that the message to a parent has for the parent's distribution. """ if index == 0: if self.ndim_mu == self.ndim: return plates elif self.ndim_mu < self.ndim: diff = self.ndim - self.ndim_mu return plates + self.shape[:diff] else: raise RuntimeError("Parent's ndim is larger") else: raise ValueError("Invalid parent index") def plates_from_parent(self, index, plates): r""" Resolve the plate mapping from a parent. Given the plates of a parent's moments, this method returns the plates that the moments has for this distribution. """ if index == 0: if self.ndim_mu == self.ndim: return plates elif self.ndim_mu < self.ndim: diff = self.ndim - self.ndim_mu return plates[:-diff] else: raise RuntimeError("Parent's ndim is larger") else: raise ValueError("Invalid parent index") def random(self, *phi, plates=None): r""" Draw a random sample from the Gaussian distribution. """ # TODO/FIXME: You shouldn't draw random values for # observed/fixed elements! D = self.ndim if D == 0: dims = () else: dims = np.shape(phi[0])[-D:] if np.prod(dims) == 1.0: # Scalar Gaussian phi1 = phi[1] if D > 0: # Because the covariance matrix has shape (1,1,...,1,1), # that is 2*D number of ones, remove the extra half of the # shape phi1 = np.reshape(phi1, np.shape(phi1)[:-2*D] + D*(1,)) var = -0.5 / phi1 std = np.sqrt(var) mu = var * phi[0] shape = plates + dims z = np.random.randn(*shape) x = mu + std * z else: N = np.prod(dims) dims_cov = dims + dims # Reshape precision matrix plates_cov = np.shape(phi[1])[:-2*D] V = -2 * np.reshape(phi[1], plates_cov + (N,N)) # Compute Cholesky U = linalg.chol(V) # Reshape mean vector plates_phi0 = np.shape(phi[0])[:-D] phi0 = np.reshape(phi[0], plates_phi0 + (N,)) mu = linalg.chol_solve(U, phi0) # Compute mu + U\z shape = plates + (N,) z = np.random.randn(*shape) # Denote Lambda = -2*phi[1] # Then, Cov = inv(Lambda) = inv(U'*U) = inv(U) * inv(U') # Thus, compute mu + U\z x = mu + linalg.solve_triangular(U, z, trans='N', lower=False) x = np.reshape(x, plates + dims) return x def compute_gradient(self, g, u, phi): r""" Compute the standard gradient with respect to the natural parameters. Gradient of the moments: .. math:: \mathrm{d}\overline{\mathbf{u}} &= \begin{bmatrix} \frac{1}{2} \phi_2^{-1} \mathrm{d}\phi_2 \phi_2^{-1} \phi_1 - \frac{1}{2} \phi_2^{-1} \mathrm{d}\phi_1 \\ - \frac{1}{4} \phi_2^{-1} \mathrm{d}\phi_2 \phi_2^{-1} \phi_1 \phi_1^{\mathrm{T}} \phi_2^{-1} - \frac{1}{4} \phi_2^{-1} \phi_1 \phi_1^{\mathrm{T}} \phi_2^{-1} \mathrm{d}\phi_2 \phi_2^{-1} + \frac{1}{2} \phi_2^{-1} \mathrm{d}\phi_2 \phi_2^{-1} + \frac{1}{4} \phi_2^{-1} \mathrm{d}\phi_1 \phi_1^{\mathrm{T}} \phi_2^{-1} + \frac{1}{4} \phi_2^{-1} \phi_1 \mathrm{d}\phi_1^{\mathrm{T}} \phi_2^{-1} \end{bmatrix} \\ &= \begin{bmatrix} 2 (\overline{u}_2 - \overline{u}_1 \overline{u}_1^{\mathrm{T}}) \mathrm{d}\phi_2 \overline{u}_1 + (\overline{u}_2 - \overline{u}_1 \overline{u}_1^{\mathrm{T}}) \mathrm{d}\phi_1 \\ u_2 d\phi_2 u_2 - 2 u_1 u_1^T d\phi_2 u_1 u_1^T + 2 (u_2 - u_1 u_1^T) d\phi_1 u_1^T \end{bmatrix} Standard gradient given the gradient with respect to the moments, that is, given the Riemannian gradient :math:`\tilde{\nabla}`: .. math:: \nabla = \begin{bmatrix} (\overline{u}_2 - \overline{u}_1 \overline{u}_1^{\mathrm{T}}) \tilde{\nabla}_1 + 2 (u_2 - u_1 u_1^T) \tilde{\nabla}_2 u_1 \\ (u_2 - u_1 u_1^T) \tilde{\nabla}_1 u_1^T + u_1 \tilde{\nabla}_1^T (u_2 - u_1 u_1^T) + 2 u_2 \tilde{\nabla}_2 u_2 - 2 u_1 u_1^T \tilde{\nabla}_2 u_1 u_1^T \end{bmatrix} """ ndim = self.ndim x = u[0] xx = u[1] # Some helpful variables x_x = linalg.outer(x, x, ndim=ndim) Cov = xx - x_x cov_g0 = linalg.mvdot(Cov, g[0], ndim=ndim) cov_g0_x = linalg.outer(cov_g0, x, ndim=ndim) g1_x = linalg.mvdot(g[1], x, ndim=ndim) # Compute gradient terms d0 = cov_g0 + 2 * linalg.mvdot(Cov, g1_x, ndim=ndim) d1 = (cov_g0_x + linalg.transpose(cov_g0_x, ndim=ndim) + 2 * linalg.mmdot(xx, linalg.mmdot(g[1], xx, ndim=ndim), ndim=ndim) - 2 * x_x * misc.add_trailing_axes(linalg.inner(g1_x, x, ndim=ndim), 2*ndim)) return [d0, d1] class GaussianGammaISODistribution(ExponentialFamilyDistribution): r""" Class for the VMP formulas of Gaussian-Gamma-ISO variables. Currently, supports only vector variables. """ def compute_message_to_parent(self, parent, index, u, u_mu_Lambda, u_a, u_b): r""" Compute the message to a parent node. """ if index == 0: raise NotImplementedError() elif index == 1: raise NotImplementedError() elif index == 2: raise NotImplementedError() else: raise ValueError("Index out of bounds") def compute_phi_from_parents(self, u_mu_Lambda, u_a, u_b, mask=True): r""" Compute the natural parameter vector given parent moments. """ Lambda_mu = u_mu_Lambda[0] mu_Lambda_mu = u_mu_Lambda[1] Lambda = u_mu_Lambda[2] a = u_a[0] b = u_b[0] phi = [Lambda_mu, -0.5*Lambda, -0.5*mu_Lambda_mu - b, a] return phi def compute_moments_and_cgf(self, phi, mask=True): r""" Compute the moments and :math:`g(\phi)`. """ # Compute helpful variables V = -2*phi[1] L_V = linalg.chol(V) logdet_V = linalg.chol_logdet(L_V) mu = linalg.chol_solve(L_V, phi[0]) Cov = linalg.chol_inv(L_V) a = phi[3] b = -phi[2] - 0.5 * linalg.inner(mu, phi[0]) log_b = np.log(b) # Compute moments u2 = a / b u3 = -log_b + special.psi(a) u0 = u2[...,None] * mu u1 = Cov + u2[...,None,None] * linalg.outer(mu, mu) u = [u0, u1, u2, u3] # Compute g g = 0.5*logdet_V + a*log_b - special.gammaln(a) return (u, g) def compute_cgf_from_parents(self, u_mu_Lambda, u_a, u_b): r""" Compute :math:`\mathrm{E}_{q(p)}[g(p)]` """ logdet_Lambda = u_mu_Lambda[3] a = u_a[0] gammaln_a = u_a[1] log_b = u_b[1] g = 0.5*logdet_Lambda + a*log_b - gammaln_a return g def compute_fixed_moments_and_f(self, x, alpha, mask=True): r""" Compute the moments and :math:`f(x)` for a fixed value. """ logalpha = np.log(alpha) u0 = x * misc.add_trailing_axes(alpha, 1) u1 = linalg.outer(x, x, ndim=1) * misc.add_trailing_axes(alpha, 2) u2 = alpha u3 = logalpha u = [u0, u1, u2, u3] D = np.shape(x)[-1] f = (D/2 - 1) * logalpha - D/2 * np.log(2*np.pi) return (u, f) def random(self, *params, plates=None): r""" Draw a random sample from the distribution. """ raise NotImplementedError() class GaussianGammaARDDistribution(ExponentialFamilyDistribution): r""" """ def __init__(self): raise NotImplementedError() class GaussianWishartDistribution(ExponentialFamilyDistribution): r""" Class for the VMP formulas of Gaussian-Wishart variables. Currently, supports only vector variables. """ def compute_message_to_parent(self, parent, index, u, u_mu_alpha, u_V, u_n): r""" Compute the message to a parent node. """ if index == 0: raise NotImplementedError() elif index == 1: raise NotImplementedError() elif index == 2: raise NotImplementedError() else: raise ValueError("Index out of bounds") def compute_phi_from_parents(self, u_mu_alpha, u_V, u_n, mask=True): r""" Compute the natural parameter vector given parent moments. """ raise NotImplementedError() def compute_moments_and_cgf(self, phi, mask=True): r""" Compute the moments and :math:`g(\phi)`. """ raise NotImplementedError() return (u, g) def compute_cgf_from_parents(self, u_mu_alpha, u_V, u_n): r""" Compute :math:`\mathrm{E}_{q(p)}[g(p)]` """ raise NotImplementedError() return g def compute_fixed_moments_and_f(self, x, Lambda, mask=True): r""" Compute the moments and :math:`f(x)` for a fixed value. """ raise NotImplementedError() return (u, f) def random(self, *params, plates=None): r""" Draw a random sample from the distribution. """ raise NotImplementedError() # # NODES # class Gaussian(ExponentialFamily): r""" Node for Gaussian variables. The node represents a :math:`D`-dimensional vector from the Gaussian distribution: .. math:: \mathbf{x} &\sim \mathcal{N}(\boldsymbol{\mu}, \mathbf{\Lambda}), where :math:`\boldsymbol{\mu}` is the mean vector and :math:`\mathbf{\Lambda}` is the precision matrix (i.e., inverse of the covariance matrix). .. math:: \mathbf{x},\boldsymbol{\mu} \in \mathbb{R}^{D}, \quad \mathbf{\Lambda} \in \mathbb{R}^{D \times D}, \quad \mathbf{\Lambda} \text{ symmetric positive definite} Parameters ---------- mu : Gaussian-like node or GaussianGammaISO-like node or GaussianWishart-like node or array Mean vector Lambda : Wishart-like node or array Precision matrix See also -------- Wishart, GaussianARD, GaussianWishart, GaussianGammaARD, GaussianGammaISO """ _distribution = GaussianDistribution() _moments = GaussianMoments(1) _parent_moments = [GaussianWishartMoments()] def __init__(self, mu, Lambda, **kwargs): r""" Create Gaussian node """ super().__init__(mu, Lambda, **kwargs) @classmethod def _constructor(cls, mu, Lambda, **kwargs): r""" Constructs distribution and moments objects. """ mu_Lambda = WrapToGaussianWishart(mu, Lambda) D = mu_Lambda.dims[0][0] if mu_Lambda.dims != ( (D,), (), (D,D), () ): raise Exception("Parents have wrong dimensionality") parents = [mu_Lambda] dims = ( (D,), (D,D) ) return (parents, kwargs, dims, cls._total_plates(kwargs.get('plates'), cls._distribution.plates_from_parent(0, mu_Lambda.plates)), cls._distribution, cls._moments, cls._parent_moments) def initialize_from_parameters(self, mu, Lambda): u = self._parent_moments[0].compute_fixed_moments(mu, Lambda) self._initialize_from_parent_moments(u) def __str__(self): mu = self.u[0] Cov = self.u[1] - misc.m_outer(mu, mu) return ("%s ~ Gaussian(mu, Cov)\n" " mu = \n" "%s\n" " Cov = \n" "%s\n" % (self.name, mu, Cov)) def rotate(self, R, inv=None, logdet=None, Q=None): if inv is not None: invR = inv else: invR = np.linalg.inv(R) if logdet is not None: logdetR = logdet else: logdetR = np.linalg.slogdet(R)[1] # It would be more efficient and simpler, if you just rotated the # moments and didn't touch phi. However, then you would need to call # update() before lower_bound_contribution. This is more error-safe. # Rotate plates, if plate rotation matrix is given. Assume that there's # only one plate-axis if Q is not None: # Rotate moments using Q self.u[0] = np.einsum('ik,kj->ij', Q, self.u[0]) sumQ = np.sum(Q, axis=0) # Rotate natural parameters using Q self.phi[1] = np.einsum('d,dij->dij', sumQ**(-2), self.phi[1]) self.phi[0] = np.einsum('dij,dj->di', -2*self.phi[1], self.u[0]) # Transform parameters using R self.phi[0] = mvdot(invR.T, self.phi[0]) self.phi[1] = dot(invR.T, self.phi[1], invR) if Q is not None: self._update_moments_and_cgf() else: # Transform moments and g using R self.u[0] = mvdot(R, self.u[0]) self.u[1] = dot(R, self.u[1], R.T) self.g -= logdetR def rotate_matrix(self, R1, R2, inv1=None, logdet1=None, inv2=None, logdet2=None, Q=None): r""" The vector is reshaped into a matrix by stacking the row vectors. Computes R1*X*R2', which is identical to kron(R1,R2)*x (??) Note that this is slightly different from the standard Kronecker product definition because Numpy stacks row vectors instead of column vectors. Parameters ---------- R1 : ndarray A matrix from the left R2 : ndarray A matrix from the right """ if Q is not None: # Rotate moments using Q self.u[0] = np.einsum('ik,kj->ij', Q, self.u[0]) sumQ = np.sum(Q, axis=0) # Rotate natural parameters using Q self.phi[1] = np.einsum('d,dij->dij', sumQ**(-2), self.phi[1]) self.phi[0] = np.einsum('dij,dj->di', -2*self.phi[1], self.u[0]) if inv1 is None: inv1 = np.linalg.inv(R1) if logdet1 is None: logdet1 = np.linalg.slogdet(R1)[1] if inv2 is None: inv2 = np.linalg.inv(R2) if logdet2 is None: logdet2 = np.linalg.slogdet(R2)[1] D1 = np.shape(R1)[0] D2 = np.shape(R2)[0] # Reshape into matrices sh0 = np.shape(self.phi[0])[:-1] + (D1,D2) sh1 = np.shape(self.phi[1])[:-2] + (D1,D2,D1,D2) phi0 = np.reshape(self.phi[0], sh0) phi1 = np.reshape(self.phi[1], sh1) # Apply rotations to phi #phi0 = dot(inv1, phi0, inv2.T) phi0 = dot(inv1.T, phi0, inv2) phi1 = np.einsum('...ia,...abcd->...ibcd', inv1.T, phi1) phi1 = np.einsum('...ic,...abcd->...abid', inv1.T, phi1) phi1 = np.einsum('...ib,...abcd->...aicd', inv2.T, phi1) phi1 = np.einsum('...id,...abcd->...abci', inv2.T, phi1) # Reshape back into vectors self.phi[0] = np.reshape(phi0, self.phi[0].shape) self.phi[1] = np.reshape(phi1, self.phi[1].shape) # It'd be better to rotate the moments too.. self._update_moments_and_cgf() class GaussianARD(ExponentialFamily): r""" Node for Gaussian variables with ARD prior. The node represents a :math:`D`-dimensional vector from the Gaussian distribution: .. math:: \mathbf{x} &\sim \mathcal{N}(\boldsymbol{\mu}, \mathrm{diag}(\boldsymbol{\alpha})), where :math:`\boldsymbol{\mu}` is the mean vector and :math:`\mathrm{diag}(\boldsymbol{\alpha})` is the diagonal precision matrix (i.e., inverse of the covariance matrix). .. math:: \mathbf{x},\boldsymbol{\mu} \in \mathbb{R}^{D}, \quad \alpha_d > 0 \text{ for } d=0,\ldots,D-1 *Note:* The form of the posterior approximation is a Gaussian distribution with full covariance matrix instead of a diagonal matrix. Parameters ---------- mu : Gaussian-like node or GaussianGammaISO-like node or GaussianGammaARD-like node or array Mean vector alpha : gamma-like node or array Diagonal elements of the precision matrix See also -------- Gamma, Gaussian, GaussianGammaARD, GaussianGammaISO, GaussianWishart """ def __init__(self, mu, alpha, ndim=None, shape=None, **kwargs): r""" Create GaussianARD node. """ super().__init__(mu, alpha, ndim=ndim, shape=shape, **kwargs) @classmethod def _constructor(cls, mu, alpha, ndim=None, shape=None, **kwargs): r""" Constructs distribution and moments objects. If __init__ uses useconstructor decorator, this method is called to construct distribution and moments objects. The method is given the same inputs as __init__. For some nodes, some of these can't be "static" class attributes, then the node class must overwrite this method to construct the objects manually. The point of distribution class is to move general distribution but not-node specific code. The point of moments class is to define the messaging protocols. """ # Check consistency if ndim is not None and shape is not None and ndim != len(shape): raise ValueError("Given shape and ndim inconsistent") if ndim is None and shape is not None: ndim = len(shape) # Infer shape of mu try: # Case: mu is a node mu = mu._convert(GaussianGammaARDMoments) except AttributeError: # Case: mu is constant, we can use it as a scalar shape_mu = () mu = cls._ensure_moments(mu, GaussianMoments(0)) mu = mu._convert(GaussianGammaARDMoments) else: shape_mu = mu.dims[0] ndim_mu = len(shape_mu) # Infer dimensionality if ndim is None: ndim = ndim_mu #max(ndim_mu, ndim_alpha) elif ndim < ndim_mu: # or ndim < ndim_alpha: raise ValueError("Parent mu has more axes") # Infer shape of alpha alpha = cls._ensure_moments(alpha, GammaMoments()) if ndim == 0: shape_alpha = () else: shape_alpha = alpha.plates[-ndim:] # Infer shape of the node #shape_bc = misc.broadcasted_shape(shape_mu, shape_alpha) try: shape_bc = misc.broadcasted_shape(mu.plates+shape_mu, alpha.plates) except ValueError: raise ValueError("Parent nodes have incompatible shapes") if ndim == 0: shape_bc = () elif ndim > len(shape_bc): shape_bc = (ndim-len(shape_bc))*(1,) + shape_bc else: shape_bc = shape_bc[-ndim:] # By default, use the broadcasted shape if shape is None: shape = shape_bc if not misc.is_shape_subset(shape_bc, shape): raise ValueError("Broadcasted shape of the parents %s does not " "broadcast to the given shape %s" % (shape_bc, shape)) mu_alpha = WrapToGaussianGammaARD(mu, alpha) # Check shape consistency shape_cov = shape[-ndim_mu:] + shape[-ndim_mu:] moments = GaussianMoments(ndim) parent_moments = [mu_alpha._moments] distribution = GaussianARDDistribution(shape, ndim_mu) dims = (shape, shape+shape) plates = cls._total_plates(kwargs.get('plates'), distribution.plates_from_parent(0, mu_alpha.plates)) parents = [mu_alpha] return (parents, kwargs, dims, plates, distribution, moments, parent_moments) def initialize_from_parameters(self, mu, alpha): # Explicit broadcasting so the shapes match mu = mu * np.ones(np.shape(alpha)) alpha = alpha * np.ones(np.shape(mu)) # Compute parent moments u = self._parent_moments[0].compute_fixed_moments(mu, alpha) # Initialize distribution self._initialize_from_parent_moments(u) def initialize_from_mean_and_covariance(self, mu, Cov): ndim = len(self._distribution.shape) u = [mu, Cov + linalg.outer(mu, mu, ndim=ndim)] mask = np.logical_not(self.observed) # TODO: You could compute the CGF but it requires Cholesky of # Cov. Do it later. self._set_moments_and_cgf(u, np.nan, mask=mask) return def __str__(self): mu = self.u[0] Cov = self.u[1] - misc.m_outer(mu, mu) return ("%s ~ Gaussian(mu, Cov)\n" " mu = \n" "%s\n" " Cov = \n" "%s\n" % (self.name, mu, Cov)) def rotate(self, R, inv=None, logdet=None, axis=-1, Q=None, subset=None): ndim = len(self._distribution.shape) if inv is not None: invR = inv else: invR = np.linalg.inv(R) if logdet is not None: logdetR = logdet else: logdetR = np.linalg.slogdet(R)[1] self.phi[0] = rotate_mean(self.phi[0], invR.T, axis=axis, ndim=ndim) self.phi[1] = rotate_covariance(self.phi[1], invR.T, axis=axis, ndim=ndim) self.u[0] = rotate_mean(self.u[0], R, axis=axis, ndim=ndim) self.u[1] = rotate_covariance(self.u[1], R, axis=axis, ndim=ndim) s = list(self.dims[0]) s.pop(axis) self.g -= logdetR * np.prod(s) return def rotate_plates(self, Q, plate_axis=-1): r""" Approximate rotation of a plate axis. Mean is rotated exactly but covariance/precision matrix is rotated approximately. """ ndim = len(self._distribution.shape) # Rotate moments using Q if not isinstance(plate_axis, int): raise ValueError("Plate axis must be integer") if plate_axis >= 0: plate_axis -= len(self.plates) if plate_axis < -len(self.plates) or plate_axis >= 0: raise ValueError("Axis out of bounds") u0 = rotate_mean(self.u[0], Q, ndim=ndim+(-plate_axis), axis=0) sumQ = misc.add_trailing_axes(np.sum(Q, axis=0), 2*ndim-plate_axis-1) phi1 = sumQ**(-2) * self.phi[1] phi0 = -2 * matrix_dot_vector(phi1, u0, ndim=ndim) self.phi[0] = phi0 self.phi[1] = phi1 self._update_moments_and_cgf() return class GaussianGammaISO(ExponentialFamily): r""" Node for Gaussian-gamma (isotropic) random variables. The prior: .. math:: p(x, \alpha| \mu, \Lambda, a, b) p(x|\alpha, \mu, \Lambda) = \mathcal{N}(x | \mu, \alpha Lambda) p(\alpha|a, b) = \mathcal{G}(\alpha | a, b) The posterior approximation :math:`q(x, \alpha)` has the same Gaussian-gamma form. Currently, supports only vector variables. """ _moments = GaussianGammaISOMoments(1) _parent_moments = (GaussianWishartMoments(), GammaPriorMoments(), GammaMoments()) _distribution = GaussianGammaISODistribution() @classmethod def _constructor(cls, mu, Lambda, a, b, **kwargs): r""" Constructs distribution and moments objects. This method is called if useconstructor decorator is used for __init__. `mu` is the mean/location vector `alpha` is the scale `V` is the scale matrix `n` is the degrees of freedom """ # Convert parent nodes mu_Lambda = WrapToGaussianWishart(mu, Lambda) a = cls._ensure_moments(a, cls._parent_moments[1]) b = cls._ensure_moments(b, cls._parent_moments[2]) D = mu_Lambda.dims[0][0] # Check shapes if mu_Lambda.dims != ( (D,), (), (D,D), () ): raise ValueError("mu and Lambda have wrong shape") if a.dims != ( (), () ): raise ValueError("a has wrong shape") if b.dims != ( (), () ): raise ValueError("b has wrong shape") # Shapes of the moments / natural parameters dims = ( (D,), (D,D), (), () ) # List of parent nodes parents = [mu_Lambda, a, b] return (parents, kwargs, dims, cls._total_plates(kwargs.get('plates'), cls._distribution.plates_from_parent(0, mu_Lambda.plates), cls._distribution.plates_from_parent(1, a.plates), cls._distribution.plates_from_parent(2, b.plates)), cls._distribution, cls._moments, cls._parent_moments) def plotmatrix(self): r""" Creates a matrix of marginal plots. On diagonal, are marginal plots of each variable. Off-diagonal plot (i,j) shows the joint marginal density of x_i and x_j. """ import bayespy.plot as bpplt if np.prod(self.plates) != 1: raise ValueError("Currently, does not support plates in the node.") if len(self.dims[0]) != 1: raise ValueError("Currently, supports only vector variables") # Dimensionality of the Gaussian D = self.dims[0][0] # Compute standard parameters tau = self.u[2] mu = self.u[0] mu = mu / misc.add_trailing_axes(tau, 1) Cov = self.u[1] - linalg.outer(self.u[0], mu, ndim=1) Cov = Cov / misc.add_trailing_axes(tau, 2) a = self.phi[3] b = -self.phi[2] - 0.5*linalg.inner(self.phi[0], mu, ndim=1) # Create subplots (fig, axes) = bpplt.pyplot.subplots(D+1, D+1) # Plot marginal Student t distributions for i in range(D): for j in range(i+1): if i == j: bpplt._pdf_t(*(random.gaussian_gamma_to_t(mu[i], Cov[i,i], a, b, ndim=0)), axes=axes[i,i]) else: S = Cov[np.ix_([i,j],[i,j])] (m, S, nu) = random.gaussian_gamma_to_t(mu[[i,j]], S, a, b) bpplt._contour_t(m, S, nu, axes=axes[i,j]) bpplt._contour_t(m, S, nu, axes=axes[j,i], transpose=True) # Plot Gaussian-gamma marginal distributions for k in range(D): bpplt._contour_gaussian_gamma(mu[k], Cov[k,k], a, b, axes=axes[D,k]) bpplt._contour_gaussian_gamma(mu[k], Cov[k,k], a, b, axes=axes[k,D], transpose=True) # Plot gamma marginal distribution bpplt._pdf_gamma(a, b, axes=axes[D,D]) return axes def get_gaussian_mean_and_variance(self): r""" Return the mean and variance of the distribution """ a = self.phi[3] nu = 2*a if nu <= 1: raise ValueError("Mean not defined for degrees of freedom <= 1") if nu <= 2: raise ValueError("Variance not defined if degrees of freedom <= 2") tau = self.u[2] tau_mu = self.u[0] mu = tau_mu / misc.add_trailing_axes(tau, 1) var = misc.get_diag(self.u[1], ndim=1) - tau_mu*mu var = var / misc.add_trailing_axes(tau, 1) var = nu / (nu-2) * var return (mu, var) def get_marginal_logpdf(self, gaussian=None, gamma=None): r""" Get the (marginal) log pdf of a subset of the variables Parameters ---------- gaussian : list or None Indices of the Gaussian variables to keep or None gamma : bool or None True if keep the gamma variable, otherwise False or None Returns ------- function A function which computes log-pdf """ if gaussian is None and not gamma: raise ValueError("Must give some variables") # Compute standard parameters tau = self.u[2] mu = self.u[0] mu = mu / misc.add_trailing_axes(tau, 1) Cov = np.linalg.inv(-2*self.phi[1]) if not np.allclose(Cov, self.u[1] - linalg.outer(self.u[0], mu, ndim=1)): raise Exception("WAAAT") #Cov = Cov / misc.add_trailing_axes(tau, 2) a = self.phi[3] b = -self.phi[2] - 0.5*linalg.inner(self.phi[0], mu, ndim=1) if not gamma: # Student t distributions inds = list(gaussian) mu = mu[inds] Cov = Cov[np.ix_(inds, inds)] (mu, Cov, nu) = random.gaussian_gamma_to_t(mu, Cov, a, b, ndim=1) L = linalg.chol(Cov) logdet_Cov = linalg.chol_logdet(L) D = len(inds) def logpdf(x): y = x - mu v = linalg.chol_solve(L, y) z2 = linalg.inner(y, v, ndim=1) return random.t_logpdf(z2, logdet_Cov, nu, D) return logpdf elif gaussian is None: # Gamma distribution def logpdf(x): logx = np.log(x) return random.gamma_logpdf(b*x, logx, a*logx, a*np.log(b), special.gammaln(a)) return logpdf else: # Gaussian-gamma distribution inds = list(gaussian) mu = mu[inds] Cov = Cov[np.ix_(inds, inds)] D = len(inds) L = linalg.chol(Cov) logdet_Cov = linalg.chol_logdet(L) def logpdf(x): tau = x[...,-1] logtau = np.log(tau) x = x[...,:-1] y = x - mu v = linalg.chol_solve(L, y) * tau[...,None] z2 = linalg.inner(y, v, ndim=1) return (random.gaussian_logpdf(z2, 0, 0, logdet_Cov + D*logtau, D) + random.gamma_logpdf(b*tau, logtau, a*logtau, a*np.log(b), special.gammaln(a))) return logpdf class GaussianGammaARD(ExponentialFamily): r""" Node for Gaussian and gamma random variables with ARD form. The prior: .. math:: p(x, \tau| \mu, \alpha, a, b) = p(x|\tau, \mu, \alpha) p(\tau|a, b) p(x|\alpha, \mu, \alpha) = \mathcal{N}(x | \mu, \mathrm{diag}( \boldsymbol{\alpha} \boldsymbol{\tau} )) p(\tau|a, b) = \mathcal{G}(\tau | a, b) The posterior approximation :math:`q(x, \tau)` has the same Gaussian-gamma form. .. warning:: Not yet implemented. See also -------- Gaussian, GaussianARD, Gamma, GaussianGammaISO, GaussianWishart """ def __init__(self, mu, alpha, a, b, **kwargs): r""" """ raise NotImplementedError() class GaussianWishart(ExponentialFamily): r""" Node for Gaussian-Wishart random variables. The prior: .. math:: p(x, \Lambda| \mu, \alpha, V, n) p(x|\Lambda, \mu, \alpha) = \mathcal(N)(x | \mu, \alpha^{-1} Lambda^{-1}) p(\Lambda|V, n) = \mathcal(W)(\Lambda | n, V) The posterior approximation :math:`q(x, \Lambda)` has the same Gaussian-Wishart form. Currently, supports only vector variables. """ _moments = GaussianWishartMoments() _distribution = GaussianWishartDistribution() @classmethod def _constructor(cls, mu, alpha, n, V, **kwargs): r""" Constructs distribution and moments objects. This method is called if useconstructor decorator is used for __init__. `mu` is the mean/location vector `alpha` is the scale `n` is the degrees of freedom `V` is the scale matrix """ # Convert parent nodes mu_alpha = WrapToGaussianGammaISO(mu, alpha) D = mu_alpha.dims[0][0] parent_moments = (GaussianGammaISOMoments(1), WishartMoments(), WishartPriorMoments(D)) n = cls._ensure_moments(n, parent_moments[1]) V = cls._ensure_moments(V, parent_moments[2]) # Check shapes if mu_alpha.dims != ( (D,), (D,D), (), () ): raise ValueError("mu and alpha have wrong shape") if V.dims != ( (D,D), () ): raise ValueError("Precision matrix has wrong shape") if n.dims != ( (), () ): raise ValueError("Degrees of freedom has wrong shape") dims = ( (D,), (), (D,D), () ) parents = [mu_alpha, n, V] return (parents, kwargs, dims, cls._total_plates(kwargs.get('plates'), cls._distribution.plates_from_parent(0, mu_alpha.plates), cls._distribution.plates_from_parent(1, n.plates), cls._distribution.plates_from_parent(2, V.plates)), cls._distribution, cls._moments, parent_moments) # # CONVERTERS # ## class GaussianToGaussianARD(Deterministic): ## """ ## Converter for Gaussian moments to Gaussian ARD moments ## """ ## def __init__(self, X, **kwargs): ## """ ## """ ## self.ndim = X._moments.ndim ## self._moments = GaussianARDMoments() ## self._parent_moments = [GaussianMoments(self.ndim)] ## dims = ( (), () ) ## super().__init__(X, dims=dims, **kwargs) ## def _compute_moments(self, u_X): ## """ ## """ ## x = u_X[0] ## xx = u_X[1] ## u0 = x ## u1 = misc.get_diag(xx, ndim=self.ndim) ## u = [u0, u1] ## return u ## def _compute_message_to_parent(self, index, m_child, u_X): ## """ ## """ ## if index == 0: ## m0 = m_child[0] ## m1 = misc.diag(m_child[1], ndim=self.ndim) ## m = [m0, m1] ## return m ## else: ## raise ValueError("Invalid parent index") ## def _compute_mask_to_parent(self, index, mask): ## """ ## """ ## if index == 0: ## if ndim == 0: ## return mask ## else: ## axes = tuple(range(-self.ndim, 0)) ## return np.any(mask, axis=axes) ## else: ## raise ValueError("Invalid parent index") ## def _plates_to_parent(self, index): ## """ ## """ ## if index == 0: ## if ndim == 0: ## return self.plates ## else: ## return self.plates[:-self.ndim] ## else: ## raise ValueError("Invalid parent index") ## def _plates_from_parent(self, index): ## """ ## """ ## if index == 0: ## shape = self.parents[0].dims[0] ## return self.parents[0].plates + shape ## else: ## raise ValueError("Invalid parent index") ## GaussianMoments.add_converter(GaussianARDMoments, ## GaussianToGaussianARD) ## class GaussianARDToGaussianISO(Deterministic): ## """ ## Converter for Gaussian moments to Gaussian-gamma isotropic moments ## Combines the Gaussian moments with gamma moments for a fixed value 1. ## """ ## def __init__(self, X, **kwargs): ## """ ## """ ## self.ndim = X._moments.ndim ## self._moments = GaussianISOMoments(self.ndim) ## self._parent_moments = [GaussianARDMoments()] ## shape = X.dims[0] ## dims = ( shape, 2*shape, (), () ) ## super().__init__(X, dims=dims, **kwargs) ## def _compute_moments(self, u_X): ## """ ## """ ## x = u_X[0] ## xx = u_X[1] ## u = [x, xx, 1, 0] ## return u ## def _compute_message_to_parent(self, index, m_child, u_X): ## """ ## """ ## if index == 0: ## m = m_child[:2] ## return m ## else: ## raise ValueError("Invalid parent index") ## GaussianARDMoments.add_converter(GaussianISOMoments, ## GaussianARDToGaussianISO) class GaussianToGaussianGammaISO(Deterministic): r""" Converter for Gaussian moments to Gaussian-gamma isotropic moments Combines the Gaussian moments with gamma moments for a fixed value 1. """ def __init__(self, X, **kwargs): r""" """ self.ndim = X._moments.ndim self._moments = GaussianGammaISOMoments(self.ndim) self._parent_moments = [GaussianMoments(self.ndim)] shape = X.dims[0] dims = ( shape, 2*shape, (), () ) super().__init__(X, dims=dims, **kwargs) def _compute_moments(self, u_X): r""" """ x = u_X[0] xx = u_X[1] u = [x, xx, 1, 0] return u def _compute_message_to_parent(self, index, m_child, u_X): r""" """ if index == 0: m = m_child[:2] return m else: raise ValueError("Invalid parent index") GaussianMoments.add_converter(GaussianGammaISOMoments, GaussianToGaussianGammaISO) class GaussianGammaISOToGaussianGammaARD(Deterministic): r""" Converter for Gaussian-gamma ISO moments to Gaussian-gamma ARD moments """ def __init__(self, X, **kwargs): r""" """ self.ndim = X._moments.ndim self._moments = GaussianGammaARDMoments(self.ndim) self._parent_moments = [GaussianGammaISOMoments(self.ndim)] shape = X.dims[0] dims = ( shape, shape, shape, shape ) super().__init__(X, dims=dims, **kwargs) def _compute_moments(self, u_X_alpha): r""" ... .. math:: u_0 &= \alpha x \\ u_1 &= \alpha \mathrm{diag}(xx^T) \\ u_2 &= [\alpha, \ldots, \alpha] \\ u_3 &= [\log(\alpha), \ldots, \log(\alpha)] """ shape = self.dims[0] alpha_x = u_X_alpha[0] alpha_xx = misc.get_diag(u_X_alpha[1], ndim=self.ndim) alpha = misc.add_trailing_axes(u_X_alpha[2], self.ndim) * np.ones(shape) logalpha = misc.add_trailing_axes(u_X_alpha[3], self.ndim) * np.ones(shape) u = [alpha_x, alpha_xx, alpha, logalpha] return u def _compute_message_to_parent(self, index, m_child, u_X_alpha): r""" ... Message from the child is :math:`[m_0, m_1, m_2, m_3]`: .. math:: \alpha m_0^T x + m_1 \alpha m_1^T \mathrm{diag}(xx^T) + \alpha\mathrm{sum}(m_2) + \mathrm{sum}(m_3) \log|\alpha| Thus, message to the first parent is (in case of Gaussian-gamma and Wishart parents): .. math:: \tilde{m_0} &= m_0 \\ \tilde{m_1} &= \mathrm{diag}(m_1) \\ \tilde{m_2} &= \mathrm{sum}(m_2) \\ \tilde{m_3} &= \mathrm{sum}(m_3) """ if index == 0: m0 = m_child[0] m1 = misc.diag(m_child[1], ndim=self.ndim) m2 = np.sum(m_child[2], axis=tuple(range(-self.ndim,0))) m3 = np.sum(m_child[3], axis=tuple(range(-self.ndim,0))) m = [m0, m1, m2, m3] return m else: raise ValueError("Invalid parent index") GaussianGammaISOMoments.add_converter(GaussianGammaARDMoments, GaussianGammaISOToGaussianGammaARD) class GaussianGammaARDToGaussianWishart(Deterministic): r""" """ def __init__(self, X_alpha, **kwargs): raise NotImplementedError() GaussianGammaARDMoments.add_converter(GaussianWishartMoments, GaussianGammaARDToGaussianWishart) ## class GaussianGammaISOToGamma(Deterministic): ## """ ## """ ## def __init__(self): ## raise NotImplementedError() ## class GaussianGammaARDToGamma(Deterministic): ## """ ## """ ## def __init__(self): ## raise NotImplementedError() ## class GaussianWishartToWishart(Deterministic): ## """ ## """ ## def __init__(self): ## raise NotImplementedError() # # WRAPPERS # # These wrappers form a single node from two nodes for messaging purposes. # class WrapToGaussianGammaISO(Deterministic): r""" """ _moments = GaussianGammaISOMoments(1) _parent_moments = [GaussianGammaISOMoments(1), GammaMoments()] @ensureparents def __init__(self, X, alpha, **kwargs): r""" """ D = X.dims[0][0] dims = ( (D,), (D,D), (), () ) super().__init__(X, alpha, dims=dims, **kwargs) def _compute_moments(self, u_X, u_alpha): r""" """ raise NotImplementedError() def _compute_message_to_parent(self, index, m_child, u_X, u_alpha): r""" """ if index == 0: raise NotImplementedError() elif index == 1: raise NotImplementedError() else: raise ValueError("Invalid parent index") def _compute_mask_to_parent(self, index, mask): r""" """ if index == 0: raise NotImplementedError() elif index == 1: raise NotImplementedError() else: raise ValueError("Invalid parent index") def _plates_to_parent(self, index): r""" """ if index == 0: raise NotImplementedError() elif index == 1: raise NotImplementedError() else: raise ValueError("Invalid parent index") def _plates_from_parent(self, index): r""" """ if index == 0: raise NotImplementedError() elif index == 1: raise NotImplementedError() else: raise ValueError("Invalid parent index") class WrapToGaussianGammaARD(Deterministic): r""" """ def __init__(self, mu_alpha, tau, **kwargs): r""" """ # First, just in case mu_alpha is a numeric array, convert mu_alpha to # (constant) Gaussian. ## if not isinstance(mu_alpha, Node): ## raise ValueError("Mu must be a node") ## if ndim is None: ## raise ValueError("For non-node mu, provide ndim") ## try: ## mu_alpha = self._ensure_moments(mu_alpha, GaussianMoments(ndim)) ## except Moments.NoConverterError: ## pass # Ensure proper moments from parents try: mu_alpha = mu_alpha._convert(GaussianGammaARDMoments) except AttributeError: raise ValueError("Mu must be a node") # Parent moments self._parent_moments = [mu_alpha._moments, GammaMoments()] tau = self._ensure_moments(tau, self._parent_moments[1]) ndim = len(mu_alpha.dims[0]) if ndim == 0: shape = () else: shape_mu = mu_alpha.dims[0] shape_tau = tau.plates[-ndim:] shape = misc.broadcasted_shape(shape_mu, shape_tau) self.ndim = len(shape) dims = ( shape, shape, shape, shape ) self._moments = GaussianGammaARDMoments(self.ndim) super().__init__(mu_alpha, tau, dims=dims, **kwargs) def _compute_moments(self, u_mu_alpha, u_tau): r""" """ mu_alpha = u_mu_alpha[0] mu2_alpha = u_mu_alpha[1] alpha = u_mu_alpha[2] logalpha = u_mu_alpha[3] tau = u_tau[0] logtau = u_tau[1] u0 = mu_alpha * tau u1 = mu2_alpha * tau u2 = alpha * tau u3 = logalpha + logtau u = [u0, u1, u2, u3] return u def _compute_message_to_parent(self, index, m_child, u_mu_alpha, u_tau): r""" ... Message from the child is :math:`[m_0, m_1, m_2, m_3]`: .. math:: m_0^T \mathrm{diag}(\alpha \circ \tau) \mu + m_1^T \mathrm{diag}(\alpha \circ \tau) (\mu \circ \mu) + m_2^T (\alpha \circ \tau) + m_3^T \circ (\log \alpha + \log \tau) Thus, message to the first parent is: .. math:: \tilde{m_0} &= m_0 \circ \tau \\ \tilde{m_1} &= m_1 \circ \tau \\ \tilde{m_2} &= m_2 \circ \tau \\ \tilde{m_3} &= m_3 Sum those to proper shape of mu-alpha. The message to the second parent is: .. math:: \tilde{m_0} &= \alpha \circ (m_0 \circ \mu + m_1 \circ \mu \circ \mu + m_2) \\ \tilde{m_1} &= m_3 """ if index == 0: tau = u_tau[0] m0 = m_child[0] * tau m1 = m_child[1] * tau m2 = m_child[2] * tau m3 = m_child[3] # Sum the broadcasted variable axes. Plate axes are handled by # default. shape_mu = self.parents[0].dims[0] ndim_mu = len(shape_mu) if ndim_mu > 0 and shape_mu != self.dims[0]: plates_m0 = np.shape(m0)[:-ndim_mu] plates_m1 = np.shape(m1)[:-ndim_mu] plates_m2 = np.shape(m2)[:-ndim_mu] plates_m3 = np.shape(m3)[:-ndim_mu] m0 = misc.sum_to_shape(m0, plates_m0 + shape_mu) m1 = misc.sum_to_shape(m1, plates_m1 + shape_mu) m2 = misc.sum_to_shape(m2, plates_m2 + shape_mu) m3 = misc.sum_to_shape(m3, plates_m3 + shape_mu) m = [m0, m1, m2, m3] return m elif index == 1: ndim_mu = len(self.parents[0].dims[0]) alpha_mu = u_mu_alpha[0] alpha_mu2 = u_mu_alpha[1] alpha = u_mu_alpha[2] m0 = m_child[0]*alpha_mu + m_child[1]*alpha_mu2 + m_child[2]*alpha m1 = m_child[3] m = [m0, m1] return m else: raise ValueError("Invalid parent index") def _compute_mask_to_parent(self, index, mask): r""" """ if index == 0: return mask elif index == 1: return misc.add_trailing_axes(mask, self.ndim) else: raise ValueError("Invalid parent index") def _plates_to_parent(self, index): r""" """ if index == 0: return self.plates elif index == 1: shape = self.dims[0] return self.plates + shape else: raise ValueError("Invalid parent index") def _plates_from_parent(self, index): r""" """ if index == 0: return self.parents[0].plates elif index == 1: if self.ndim == 0: return self.parents[1].plates else: return self.parents[1].plates[:-self.ndim] else: raise ValueError("Invalid parent index") class WrapToGaussianWishart(Deterministic): r""" Wraps Gaussian and Wishart nodes into a Gaussian-Wishart node. The following node combinations can be wrapped: * Gaussian and Wishart * Gaussian-gamma and Wishart * Gaussian-Wishart and gamma """ _moments = GaussianWishartMoments() def __init__(self, X, Lambda, **kwargs): r""" """ # Just in case X is an array, convert it to a Gaussian node first. try: X = self._ensure_moments(X, GaussianMoments(1)) except Moments.NoConverterError: pass try: # Try combo Gaussian-Gamma and Wishart X = self._ensure_moments(X, GaussianGammaISOMoments(1)) except ValueError: # Have to use Gaussian-Wishart and Gamma self._parent_moments = [GaussianWishartMoments(), GammaMoments()] X = self._ensure_moments(X, GaussianWishartMoments()) self.wishart = False else: self._parent_moments = [GaussianGammaISOMoments(1), WishartMoments()] self.wishart = True D = X.dims[0][0] dims = ( (D,), (), (D,D), () ) super().__init__(X, Lambda, dims=dims, **kwargs) def _compute_moments(self, u_X_alpha, u_Lambda): r""" """ if self.wishart: alpha_x = u_X_alpha[0] alpha_xx = u_X_alpha[1] alpha = u_X_alpha[2] log_alpha = u_X_alpha[3] Lambda = u_Lambda[0] logdet_Lambda = u_Lambda[1] D = np.prod(self.dims[0]) u0 = linalg.mvdot(Lambda, alpha_x) u1 = linalg.inner(Lambda, alpha_xx, ndim=2) u2 = Lambda * misc.add_trailing_axes(alpha, 2) u3 = logdet_Lambda + D * log_alpha u = [u0, u1, u2, u3] return u else: raise NotImplementedError() def _compute_message_to_parent(self, index, m_child, u_X_alpha, u_Lambda): r""" ... Message from the child is :math:`[m_0, m_1, m_2, m_3]`: .. math:: \alpha m_0^T \Lambda x + m_1 \alpha x^T \Lambda x + \mathrm{tr}(\alpha m_2 \Lambda) + m_3 (\log | \alpha \Lambda |) In case of Gaussian-gamma and Wishart parents: Message to the first parent (x, alpha): .. math:: \tilde{m_0} &= \Lambda m_0 \\ \tilde{m_1} &= m_1 \Lambda \\ \tilde{m_2} &= \mathrm{tr}(m_2 \Lambda) \\ \tilde{m_3} &= m_3 \cdot D Message to the second parent (Lambda): .. math:: \tilde{m_0} &= \alpha (\frac{1}{2} m_0 x^T + \frac{1}{2} x m_0^T + m_1 xx^T + m_2) \\ \tilde{m_1} &= m_3 """ if index == 0: if self.wishart: # Message to Gaussian-gamma (isotropic) Lambda = u_Lambda[0] D = np.prod(self.dims[0]) m0 = linalg.mvdot(Lambda, m_child[0]) m1 = Lambda * misc.add_trailing_axes(m_child[1], 2) m2 = linalg.inner(Lambda, m_child[2], ndim=2) m3 = D * m_child[3] m = [m0, m1, m2, m3] return m else: # Message to Gaussian-Wishart raise NotImplementedError() elif index == 1: if self.wishart: # Message to Wishart alpha_x = u_X_alpha[0] alpha_xx = u_X_alpha[1] alpha = u_X_alpha[2] m0 = (0.5*linalg.outer(alpha_x, m_child[0], ndim=1) + 0.5*linalg.outer(m_child[0], alpha_x, ndim=1) + alpha_xx * misc.add_trailing_axes(m_child[1], 2) + misc.add_trailing_axes(alpha, 2) * m_child[2]) m1 = m_child[3] m = [m0, m1] return m else: # Message to gamma (isotropic) raise NotImplementedError() else: raise ValueError("Invalid parent index") def reshape_gaussian_array(dims_from, dims_to, x0, x1): r""" Reshape the moments Gaussian array variable. The plates remain unaffected. """ num_dims_from = len(dims_from) num_dims_to = len(dims_to) # Reshape the first moment / mean num_plates_from = np.ndim(x0) - num_dims_from plates_from = np.shape(x0)[:num_plates_from] shape = ( plates_from + (1,)*(num_dims_to-num_dims_from) + dims_from ) x0 = np.ones(dims_to) * np.reshape(x0, shape) # Reshape the second moment / covariance / precision num_plates_from = np.ndim(x1) - 2*num_dims_from plates_from = np.shape(x1)[:num_plates_from] shape = ( plates_from + (1,)*(num_dims_to-num_dims_from) + dims_from + (1,)*(num_dims_to-num_dims_from) + dims_from ) x1 = np.ones(dims_to+dims_to) * np.reshape(x1, shape) return (x0, x1) def transpose_covariance(Cov, ndim=1): r""" Transpose the covariance array of Gaussian array variable. That is, swap the last ndim axes with the ndim axes before them. This makes transposing easy for array variables when the covariance is not a matrix but a multidimensional array. """ axes_in = [Ellipsis] + list(range(2*ndim,0,-1)) axes_out = [Ellipsis] + list(range(ndim,0,-1)) + list(range(2*ndim,ndim,-1)) return np.einsum(Cov, axes_in, axes_out) def left_rotate_covariance(Cov, R, axis=-1, ndim=1): r""" Rotate the covariance array of Gaussian array variable. ndim is the number of axes for the Gaussian variable. For vector variable, ndim=1 and covariance is a matrix. """ if not isinstance(axis, int): raise ValueError("Axis must be an integer") if axis < -ndim or axis >= ndim: raise ValueError("Axis out of range") # Force negative axis if axis >= 0: axis -= ndim # Rotation from left axes_R = [Ellipsis, ndim+abs(axis)+1, ndim+abs(axis)] axes_Cov = [Ellipsis] + list(range(ndim+abs(axis), 0, -1)) axes_out = [Ellipsis, ndim+abs(axis)+1] + list(range(ndim+abs(axis)-1, 0, -1)) Cov = np.einsum(R, axes_R, Cov, axes_Cov, axes_out) return Cov def right_rotate_covariance(Cov, R, axis=-1, ndim=1): r""" Rotate the covariance array of Gaussian array variable. ndim is the number of axes for the Gaussian variable. For vector variable, ndim=1 and covariance is a matrix. """ if not isinstance(axis, int): raise ValueError("Axis must be an integer") if axis < -ndim or axis >= ndim: raise ValueError("Axis out of range") # Force negative axis if axis >= 0: axis -= ndim # Rotation from right axes_R = [Ellipsis, abs(axis)+1, abs(axis)] axes_Cov = [Ellipsis] + list(range(abs(axis), 0, -1)) axes_out = [Ellipsis, abs(axis)+1] + list(range(abs(axis)-1, 0, -1)) Cov = np.einsum(R, axes_R, Cov, axes_Cov, axes_out) return Cov def rotate_covariance(Cov, R, axis=-1, ndim=1): r""" Rotate the covariance array of Gaussian array variable. ndim is the number of axes for the Gaussian variable. For vector variable, ndim=1 and covariance is a matrix. """ # Rotate from left and right Cov = left_rotate_covariance(Cov, R, ndim=ndim, axis=axis) Cov = right_rotate_covariance(Cov, R, ndim=ndim, axis=axis) return Cov def rotate_mean(mu, R, axis=-1, ndim=1): r""" Rotate the mean array of Gaussian array variable. ndim is the number of axes for the Gaussian variable. For vector variable, ndim=1 and mu is a vector. """ if not isinstance(axis, int): raise ValueError("Axis must be an integer") if axis < -ndim or axis >= ndim: raise ValueError("Axis out of range") # Force negative axis if axis >= 0: axis -= ndim # Rotation from right axes_R = [Ellipsis, abs(axis)+1, abs(axis)] axes_mu = [Ellipsis] + list(range(abs(axis), 0, -1)) axes_out = [Ellipsis, abs(axis)+1] + list(range(abs(axis)-1, 0, -1)) mu = np.einsum(R, axes_R, mu, axes_mu, axes_out) return mu def array_to_vector(x, ndim=1): if ndim == 0: return x shape_x = np.shape(x) D = np.prod(shape_x[-ndim:]) return np.reshape(x, shape_x[:-ndim] + (D,)) def array_to_matrix(A, ndim=1): if ndim == 0: return A shape_A = np.shape(A) D = np.prod(shape_A[-ndim:]) return np.reshape(A, shape_A[:-2*ndim] + (D,D)) def vector_to_array(x, shape): shape_x = np.shape(x) return np.reshape(x, np.shape(x)[:-1] + tuple(shape)) def matrix_dot_vector(A, x, ndim=1): if ndim < 0: raise ValueError("ndim must be non-negative integer") if ndim == 0: return A*x dims_x = np.shape(x)[-ndim:] A = array_to_matrix(A, ndim=ndim) x = array_to_vector(x, ndim=ndim) y = np.einsum('...ik,...k->...i', A, x) return vector_to_array(y, dims_x) bayespy-0.4.0/bayespy/inference/vmp/nodes/add.py0000644000175000017500000000763312615643655023361 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2015 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import numpy as np import functools from .deterministic import Deterministic from .gaussian import Gaussian, GaussianMoments from bayespy.utils import linalg class Add(Deterministic): r""" Node for computing sums of Gaussian nodes: :math:`X+Y+Z`. Examples -------- >>> from bayespy import nodes >>> X = nodes.Gaussian(np.zeros(2), np.identity(2), plates=(3,)) >>> Y = nodes.Gaussian(np.ones(2), np.identity(2)) >>> Z = nodes.Add(X, Y) >>> print("Mean:\n", Z.get_moments()[0]) Mean: [[ 1. 1.]] >>> print("Second moment:\n", Z.get_moments()[1]) Second moment: [[[ 3. 1.] [ 1. 3.]]] Notes ----- Shapes of the nodes must be identical. Plates are broadcasted. This node sums nodes that are independent in the posterior approximation. However, summing variables puts a strong coupling among the variables, which is lost in this construction. Thus, it is usually better to use a single Gaussian node to represent the set of the summed variables and use SumMultiply node to compute the sum. In that way, the correlation between the variables is not lost. However, in some cases it is necessary or useful to use Add node. See also -------- Dot, SumMultiply """ def __init__(self, *nodes, **kwargs): """ Add(X1, X2, ...) """ N = len(nodes) if N < 2: raise ValueError("Give at least two parents") nodes = list(nodes) for n in range(N-1): if nodes[n].dims != nodes[n+1].dims: raise ValueError("Nodes do not have identical shapes") ndim = len(nodes[0].dims[0]) dims = tuple(nodes[0].dims) self._moments = GaussianMoments(ndim) self._parent_moments = N * [GaussianMoments(ndim)] self.ndim = ndim self.N = N super().__init__(*nodes, dims=dims, **kwargs) def _compute_moments(self, *u_parents): """ Compute the moments of the sum """ u0 = functools.reduce(np.add, (u_parent[0] for u_parent in u_parents)) u1 = functools.reduce(np.add, (u_parent[1] for u_parent in u_parents)) for i in range(self.N): for j in range(i+1, self.N): xi_xj = linalg.outer(u_parents[i][0], u_parents[j][0], ndim=self.ndim) xj_xi = linalg.transpose(xi_xj, ndim=self.ndim) u1 = u1 + xi_xj + xj_xi return [u0, u1] def _compute_message_to_parent(self, index, m, *u_parents): """ Compute the message to a parent node. .. math:: (\sum_i \mathbf{x}_i)^T \mathbf{M}_2 (\sum_j \mathbf{x}_j) + (\sum_i \mathbf{x}_i)^T \mathbf{m}_1 Moments of the parents are .. math:: u_1^{(i)} = \langle \mathbf{x}_i \rangle \\ u_2^{(i)} = \langle \mathbf{x}_i \mathbf{x}_i^T \rangle Thus, the message for :math:`i`-th parent is .. math:: \phi_{x_i}^{(1)} = \mathbf{m}_1 + 2 \mathbf{M}_2 \sum_{j\neq i} \mathbf{x}_j \\ \phi_{x_i}^{(2)} = \mathbf{M}_2 """ # Remove the moments of the parent that receives the message u_parents = u_parents[:index] + u_parents[(index+1):] m0 = (m[0] + linalg.mvdot( 2*m[1], functools.reduce(np.add, (u_parent[0] for u_parent in u_parents)), ndim=self.ndim)) m1 = m[1] return [m0, m1] bayespy-0.4.0/bayespy/inference/vmp/nodes/binomial.py0000644000175000017500000001362712535521231024405 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ A module for the binomial distribution node """ import numpy as np import scipy.special as special from .expfamily import (ExponentialFamily, ExponentialFamilyDistribution, useconstructor) from .beta import BetaMoments from .poisson import PoissonMoments from .node import (Moments, ensureparents) from bayespy.utils import misc, random class BinomialMoments(PoissonMoments): """ Class for the moments of binomial variables """ def __init__(self, N): self.N = N super().__init__() def compute_fixed_moments(self, x): """ Compute the moments for a fixed value """ # Make sure the values are integers in valid range x = np.asanyarray(x) if np.any(x > self.N): raise ValueError("Invalid count") return super().compute_fixed_moments() def compute_dims_from_values(self, x): """ Return the shape of the moments for a fixed value. The realizations are scalars, thus the shape of the moment is (). """ return super().compute_dims_from_values() class BinomialDistribution(ExponentialFamilyDistribution): """ Class for the VMP formulas of binomial variables. """ def __init__(self, N): N = np.asanyarray(N) if not misc.isinteger(N): raise ValueError("Number of trials must be integer") if np.any(N < 0): raise ValueError("Number of trials must be non-negative") self.N = np.asanyarray(N) super().__init__() def compute_message_to_parent(self, parent, index, u_self, u_p): """ Compute the message to a parent node. """ if index == 0: x = u_self[0][...,None] n = self.N[...,None] m0 = x*[1, -1] + n*[0, 1] m = [m0] return m else: raise ValueError("Incorrect parent index") def compute_phi_from_parents(self, u_p, mask=True): """ Compute the natural parameter vector given parent moments. """ logp0 = u_p[0][...,0] logp1 = u_p[0][...,1] phi0 = logp0 - logp1 return [phi0] def compute_moments_and_cgf(self, phi, mask=True): """ Compute the moments and :math:`g(\phi)`. """ u0 = self.N / (1 + np.exp(-phi[0])) g = -self.N * np.log1p(np.exp(phi[0])) return ( [u0], g ) def compute_cgf_from_parents(self, u_p): """ Compute :math:`\mathrm{E}_{q(p)}[g(p)]` """ logp0 = u_p[0][...,0] logp1 = u_p[0][...,1] return self.N * logp1 def compute_fixed_moments_and_f(self, x, mask=True): """ Compute the moments and :math:`f(x)` for a fixed value. """ # Make sure the values are integers in valid range x = np.asanyarray(x) if not misc.isinteger(x): raise ValueError("Counts must be integer") if np.any(x < 0) or np.any(x > self.N): raise ValueError("Invalid count") # Now, the moments are just the counts u = [x] f = (special.gammaln(self.N+1) - special.gammaln(x+1) - special.gammaln(self.N-x+1)) return (u, f) def random(self, *phi, plates=None): """ Draw a random sample from the distribution. """ p = random.logodds_to_probability(phi[0]) return np.random.binomial(self.N, p, size=plates) class Binomial(ExponentialFamily): r""" Node for binomial random variables. The node models the number of successes :math:`x \in \{0, \ldots, n\}` in :math:`n` trials with probability :math:`p` for success: .. math:: x \sim \mathrm{Binomial}(n, p). Parameters ---------- n : scalar or array Number of trials p : beta-like node or scalar or array Probability of a success in a trial Examples -------- >>> import warnings >>> warnings.filterwarnings('ignore', category=RuntimeWarning) >>> from bayespy.nodes import Binomial, Beta >>> p = Beta([1e-3, 1e-3]) >>> x = Binomial(10, p) >>> x.observe(7) >>> p.update() >>> import bayespy.plot as bpplt >>> import numpy as np >>> bpplt.pdf(p, np.linspace(0, 1, num=100)) [] See also -------- Bernoulli, Multinomial, Beta """ _parent_moments = (BetaMoments(),) def __init__(self, n, p, **kwargs): """ Create binomial node """ super().__init__(n, p, **kwargs) @classmethod def _constructor(cls, n, p, **kwargs): """ Constructs distribution and moments objects. """ p = cls._ensure_moments(p, cls._parent_moments[0]) parents = [p] moments = BinomialMoments(n) distribution = BinomialDistribution(n) return ( parents, kwargs, ( (), ), cls._total_plates(kwargs.get('plates'), distribution.plates_from_parent(0, p.plates), np.shape(n)), distribution, moments, cls._parent_moments) def __str__(self): """ Print the distribution using standard parameterization. """ p = 1 / (1 + np.exp(-self.phi[0])) n = self._distribution.N return ("%s ~ Binomial(n, p)\n" " n = \n" "%s\n" " p = \n" "%s\n" % (self.name, n, p)) bayespy-0.4.0/bayespy/inference/vmp/nodes/dirichlet.py0000644000175000017500000001666712535517341024600 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2012,2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Module for the Dirichlet distribution node. """ import numpy as np import scipy.special as special from bayespy.utils import random from .expfamily import ExponentialFamily, ExponentialFamilyDistribution from .constant import Constant from .node import Node, Moments, ensureparents class DirichletPriorMoments(Moments): """ Class for the moments of Dirichlet conjugate-prior variables. """ def compute_fixed_moments(self, alpha): """ Compute the moments for a fixed value """ alpha = np.asanyarray(alpha) if np.ndim(alpha) < 1: raise ValueError("The prior sample sizes must be a vector") if np.any(alpha < 0): raise ValueError("The prior sample sizes must be non-negative") gammaln_sum = special.gammaln(np.sum(alpha, axis=-1)) sum_gammaln = np.sum(special.gammaln(alpha), axis=-1) z = gammaln_sum - sum_gammaln return [alpha, z] def compute_dims_from_values(self, alpha): """ Return the shape of the moments for a fixed value. """ if np.ndim(alpha) < 1: raise ValueError("The array must be at least 1-dimensional array.") d = np.shape(alpha)[-1] return [(d,), ()] class DirichletMoments(Moments): """ Class for the moments of Dirichlet variables. """ def compute_fixed_moments(self, p): """ Compute the moments for a fixed value """ # Check that probabilities are non-negative p = np.asanyarray(p) if np.ndim(p) < 1: raise ValueError("Probabilities must be given as a vector") if np.any(p < 0) or np.any(p > 1): raise ValueError("Probabilities must be in range [0,1]") if not np.allclose(np.sum(p, axis=-1), 1.0): raise ValueError("Probabilities must sum to one") # Normalize probabilities p = p / np.sum(p, axis=-1, keepdims=True) # Message is log-probabilities logp = np.log(p) u = [logp] return u def compute_dims_from_values(self, x): """ Return the shape of the moments for a fixed value. """ if np.ndim(x) < 1: raise ValueError("Probabilities must be given as a vector") D = np.shape(x)[-1] return ( (D,), ) class DirichletDistribution(ExponentialFamilyDistribution): """ Class for the VMP formulas of Dirichlet variables. """ def compute_message_to_parent(self, parent, index, u_self, u_alpha): r""" Compute the message to a parent node. """ raise NotImplementedError() def compute_phi_from_parents(self, u_alpha, mask=True): r""" Compute the natural parameter vector given parent moments. """ return [u_alpha[0]] def compute_moments_and_cgf(self, phi, mask=True): r""" Compute the moments and :math:`g(\phi)`. .. math:: \overline{\mathbf{u}} (\boldsymbol{\phi}) &= \begin{bmatrix} \psi(\phi_1) - \psi(\sum_d \phi_{1,d}) \end{bmatrix} \\ g_{\boldsymbol{\phi}} (\boldsymbol{\phi}) &= TODO """ sum_gammaln = np.sum(special.gammaln(phi[0]), axis=-1) gammaln_sum = special.gammaln(np.sum(phi[0], axis=-1)) psi_sum = special.psi(np.sum(phi[0], axis=-1, keepdims=True)) # Moments u0 = special.psi(phi[0]) - psi_sum u = [u0] # G g = gammaln_sum - sum_gammaln return (u, g) def compute_cgf_from_parents(self, u_alpha): r""" Compute :math:`\mathrm{E}_{q(p)}[g(p)]` """ return u_alpha[1] def compute_fixed_moments_and_f(self, p, mask=True): r""" Compute the moments and :math:`f(x)` for a fixed value. .. math:: u(p) = \begin{bmatrix} \log(p_1) \\ \vdots \\ \log(p_D) \end{bmatrix} .. math:: f(p) = - \sum_d \log(p_d) """ # Check that probabilities are non-negative p = np.asanyarray(p) if np.ndim(p) < 1: raise ValueError("Probabilities must be given as a vector") if np.any(p < 0) or np.any(p > 1): raise ValueError("Probabilities must be in range [0,1]") if not np.allclose(np.sum(p, axis=-1), 1.0): raise ValueError("Probabilities must sum to one") # Normalize probabilities p = p / np.sum(p, axis=-1, keepdims=True) # Message is log-probabilities logp = np.log(p) u = [logp] f = - np.sum(logp, axis=-1) return (u, f) def random(self, *phi, plates=None): r""" Draw a random sample from the distribution. """ return random.dirichlet(phi[0], size=plates) def compute_gradient(self, g, u, phi): r""" Compute the moments and :math:`g(\phi)`. \psi(\phi_1) - \psi(\sum_d \phi_{1,d}) Standard gradient given the gradient with respect to the moments, that is, given the Riemannian gradient :math:`\tilde{\nabla}`: .. math:: \nabla &= \begin{bmatrix} (\psi^{(1)}(\phi_1) - \psi^{(1)}(\sum_d \phi_{1,d}) \nabla_1 \end{bmatrix} """ sum_phi = np.sum(phi[0], axis=-1, keepdims=True) d0 = g[0] * (special.polygamma(1, phi[0]) - special.polygamma(1, sum_phi)) return [d0] class Dirichlet(ExponentialFamily): r""" Node for Dirichlet random variables. The node models a set of probabilities :math:`\{\pi_0, \ldots, \pi_{K-1}\}` which satisfy :math:`\sum_{k=0}^{K-1} \pi_k = 1` and :math:`\pi_k \in [0,1] \ \forall k=0,\ldots,K-1`. .. math:: p(\pi_0, \ldots, \pi_{K-1}) = \mathrm{Dirichlet}(\alpha_0, \ldots, \alpha_{K-1}) where :math:`\alpha_k` are concentration parameters. The posterior approximation has the same functional form but with different concentration parameters. Parameters ---------- alpha : (...,K)-shaped array Prior counts :math:`\alpha_k` See also -------- Beta, Categorical, Multinomial, CategoricalMarkovChain """ _moments = DirichletMoments() _parent_moments = (DirichletPriorMoments(),) _distribution = DirichletDistribution() @classmethod @ensureparents def _constructor(cls, alpha, **kwargs): """ Constructs distribution and moments objects. """ # Number of categories D = alpha.dims[0][0] parents = [alpha] return ( parents, kwargs, ( (D,), ), cls._total_plates(kwargs.get('plates'), alpha.plates), cls._distribution, cls._moments, cls._parent_moments) def __str__(self): """ Show distribution as a string """ alpha = self.phi[0] return ("%s ~ Dirichlet(alpha)\n" " alpha =\n" "%s" % (self.name, alpha)) bayespy-0.4.0/bayespy/inference/vmp/nodes/concatenate.py0000644000175000017500000001236612533543154025104 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2015 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import numpy as np from bayespy.utils import misc from .deterministic import Deterministic from .node import Moments class Concatenate(Deterministic): """ Concatenate similar nodes along a plate axis. Nodes must be of same type and dimensionality. Also, plates must be identical except for the plate axis along which the concatenation is performed. See also -------- numpy.concatenate """ def __init__(self, *nodes, axis=-1, **kwargs): if axis >= 0: raise ValueError("Currently, only negative axis indeces " "are allowed.") self._axis = axis parent_moments = None for node in nodes: try: parent_moments = node._moments except: pass else: break # All parents must have same moments self._parent_moments = (parent_moments,) * len(nodes) self._moments = parent_moments # Convert nodes try: nodes = [self._ensure_moments(node, self._parent_moments[0]) for node in nodes] except Moments.NoConverterError: raise ValueError("Parents have different moments") # Dimensionality of the node dims = tuple([dim for dim in nodes[0].dims]) for node in nodes: if node.dims != dims: raise ValueError("Parents have different dimensionalities") super().__init__(*nodes, dims=dims, **kwargs) # Compute start indices for each parent on the concatenated plate axis self._indices = np.zeros(len(nodes)+1, dtype=np.int) self._indices[1:] = np.cumsum([int(parent.plates[axis]) for parent in self.parents]) self._lengths = [parent.plates[axis] for parent in self.parents] return def _compute_plates_to_parent(self, index, plates): plates = list(plates) plates[self._axis] = self.parents[index].plates[self._axis] return tuple(plates) def _compute_plates_from_parent(self, index, plates): plates = list(plates) plates[self._axis] = 0 for parent in self.parents: plates[self._axis] += parent.plates[self._axis] return tuple(plates) def _plates_multiplier_from_parent(self, index): multipliers = [parent.plates_multiplier for parent in self.parents] for m in multipliers: if np.any(np.array(m) != 1): raise ValueError("Concatenation node does not support plate " "multipliers.") return () def _compute_mask_to_parent(self, index, mask): axis = self._axis indices = self._indices[index:(index+1)] if np.ndim(mask) >= abs(axis) and np.shape(mask)[axis] > 1: # Take the middle one of the returned three arrays return np.split(mask, indices, axis=axis)[1] else: return mask def _compute_message_to_parent(self, index, m, *u_parents): msg = [] indices = self._indices[index:(index+2)] for i in range(len(m)): # Fix plate axis to array axis axis = self._axis - len(self.dims[i]) # Find the slice from the message if np.ndim(m[i]) >= abs(axis) and np.shape(m[i])[axis] > 1: mi = np.split(m[i], indices, axis=axis)[1] else: mi = m[i] msg.append(mi) return msg def _compute_moments(self, *u_parents): # TODO/FIXME: Unfortunately, np.concatenate doesn't support # broadcasting but moment messages may use broadcasting. # # WORKAROUND: Broadcast the arrays explcitly to have same shape # except for the concatenated axis. u = [] for i in range(len(self.dims)): # Fix plate axis to array axis axis = self._axis - len(self.dims[i]) # Find broadcasted shape ui_parents = [u_parent[i] for u_parent in u_parents] shapes = [list(np.shape(uip)) for uip in ui_parents] for i in range(len(shapes)): if len(shapes[i]) >= abs(axis): shapes[i][axis] = 1 ## shapes = [np.shape(uip[:axis]) + (1,) + np.shape(uip[(axis+1)]) ## if np.ndim(uip) >= abs(self._axis) else ## np.shape(uip) ## for uip in ui_parents] bc_shape = misc.broadcasted_shape(*shapes) # Concatenated axis must be broadcasted explicitly bc_shapes = [misc.broadcasted_shape(bc_shape, (length,) + (1,)*(abs(axis)-1)) for length in self._lengths] # Broadcast explicitly ui_parents = [uip * np.ones(shape) for (uip, shape) in zip(ui_parents, bc_shapes)] # Concatenate ui = np.concatenate(ui_parents, axis=axis) u.append(ui) return u bayespy-0.4.0/bayespy/inference/vmp/nodes/gamma.py0000644000175000017500000002346712556655517023722 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2012,2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Module for the gamma distribution node. """ import numpy as np import scipy.special as special from .node import Node from .deterministic import Deterministic from .stochastic import Stochastic from .expfamily import ExponentialFamily, ExponentialFamilyDistribution from .constant import Constant from .node import Moments from .wishart import WishartMoments from bayespy.utils import misc def diagonal(alpha): """ Create a diagonal Wishart node from a Gamma node. """ return _GammaToDiagonalWishart(alpha, name=alpha.name + " as Wishart") class GammaPriorMoments(Moments): """ Class for the moments of the shape parameter in gamma distributions. """ def compute_fixed_moments(self, a): """ Compute the moments for a fixed value """ u0 = a u1 = special.gammaln(a) return [u0, u1] def compute_dims_from_values(self, a): """ Return the shape of the moments for a fixed value. """ return ( (), () ) class GammaMoments(Moments): """ Class for the moments of gamma variables. """ def compute_fixed_moments(self, x): """ Compute the moments for a fixed value """ if np.any(x < 0): raise ValueError("Values must be positive") u0 = x u1 = np.log(x) return [u0, u1] def compute_dims_from_values(self, x): """ Return the shape of the moments for a fixed value. """ return ( (), () ) class GammaDistribution(ExponentialFamilyDistribution): """ Class for the VMP formulas of gamma variables. """ def compute_message_to_parent(self, parent, index, u_self, u_a, u_b): r""" Compute the message to a parent node. """ x = u_self[0] logx = u_self[1] if index == 0: b = u_b[0] logb = u_b[1] return [logx + logb, -1] elif index == 1: a = u_a[0] return [-x, a] else: raise ValueError("Index out of bounds") def compute_phi_from_parents(self, *u_parents, mask=True): r""" Compute the natural parameter vector given parent moments. """ return [-u_parents[1][0], 1*u_parents[0][0]] def compute_moments_and_cgf(self, phi, mask=True): r""" Compute the moments and :math:`g(\phi)`. .. math:: \overline{\mathbf{u}} (\boldsymbol{\phi}) &= \begin{bmatrix} - \frac{\phi_2} {\phi_1} \\ \psi(\phi_2) - \log(-\phi_1) \end{bmatrix} \\ g_{\boldsymbol{\phi}} (\boldsymbol{\phi}) &= TODO """ with np.errstate(invalid='raise', divide='raise'): log_b = np.log(-phi[0]) u0 = phi[1] / (-phi[0]) u1 = special.digamma(phi[1]) - log_b u = [u0, u1] g = phi[1] * log_b - special.gammaln(phi[1]) return (u, g) def compute_cgf_from_parents(self, *u_parents): r""" Compute :math:`\mathrm{E}_{q(p)}[g(p)]` """ a = u_parents[0][0] gammaln_a = u_parents[0][1] #special.gammaln(a) b = u_parents[1][0] log_b = u_parents[1][1] g = a * log_b - gammaln_a return g def compute_fixed_moments_and_f(self, x, mask=True): r""" Compute the moments and :math:`f(x)` for a fixed value. """ if np.any(x < 0): raise ValueError("Values must be positive") logx = np.log(x) u = [x, logx] f = -logx return (u, f) def random(self, *phi, plates=None): r""" Draw a random sample from the distribution. """ return np.random.gamma(phi[1], -1/phi[0], size=plates) def compute_gradient(self, g, u, phi): r""" Compute the moments and :math:`g(\phi)`. .. math:: \mathrm{d}\overline{\mathbf{u}} &= \begin{bmatrix} - \frac{\mathrm{d}\phi_2} {phi_1} + \frac{\phi_2}{\phi_1^2} \mathrm{d}\phi_1 \\ \psi^{(1)}(\phi_2) \mathrm{d}\phi_2 - \frac{1}{\phi_1} \mathrm{d}\phi_1 \end{bmatrix} Standard gradient given the gradient with respect to the moments, that is, given the Riemannian gradient :math:`\tilde{\nabla}`: .. math:: \nabla = \begin{bmatrix} \nabla_1 \frac{\phi_2}{\phi_1^2} - \nabla_2 \frac{1}{\phi_1} \\ \nabla_2 \psi^{(1)}(\phi_2) - \nabla_1 \frac {1} {\phi_1} \end{bmatrix} """ d0 = g[0] * phi[1] / phi[0]**2 - g[1] / phi[0] d1 = g[1] * special.polygamma(1, phi[1]) - g[0] / phi[0] return [d0, d1] class Gamma(ExponentialFamily): """ Node for gamma random variables. Parameters ---------- a : scalar or array Shape parameter b : gamma-like node or scalar or array Rate parameter """ dims = ( (), () ) _distribution = GammaDistribution() _moments = GammaMoments() _parent_moments = (GammaPriorMoments(), GammaMoments()) def __init__(self, a, b, **kwargs): """ Create gamma random variable node """ super().__init__(a, b, **kwargs) def __str__(self): """ Print the distribution using standard parameterization. """ a = self.phi[1] b = -self.phi[0] return ("%s ~ Gamma(a, b)\n" " a =\n" "%s\n" " b =\n" "%s\n" % (self.name, a, b)) def as_diagonal_wishart(self): return _GammaToDiagonalWishart(self, name=self.name + " as Wishart") class GammaShape(Stochastic): """ ML point estimator for the shape parameter of the gamma distribution """ dims = ( (), () ) _moments = GammaPriorMoments() _parent_moments = () def __init__(self, **kwargs): """ Create gamma random variable node """ super().__init__(dims=self.dims, initialize=False, **kwargs) def _update_distribution_and_lowerbound(self, m): r""" Find maximum likelihood estimate for the shape parameter Messages from children appear in the lower bound as .. math:: m_0 \cdot x + m_1 \cdot \log(\Gamma(x)) Take derivative, put it zero and solve: .. math:: m_0 + m_1 \cdot d\log(\Gamma(x)) &= 0 \\ m_0 + m_1 \cdot \psi(x) &= 0 \\ x &= \psi^{-1}(-\frac{m_0}{m_1}) where :math:`\psi^{-1}` is the inverse digamma function. """ # Maximum likelihood estimate x = misc.invpsi(-m[0]/m[1]) # Compute moments self.u = self._moments.compute_fixed_moments(x) return def initialize_from_value(self, x): self.u = self._moments.compute_fixed_moments(x) return def lower_bound_contribution(self): return 0 class _GammaToDiagonalWishart(Deterministic): """ Transform a set of gamma scalars into a diagonal Wishart matrix. The last plate is used as the diagonal dimension. """ _moments = WishartMoments() _parent_moments = [GammaMoments()] def __init__(self, alpha, **kwargs): # Check for constant if misc.is_numeric(alpha): alpha = Constant(Gamma)(alpha) # Remove the last plate... #plates = alpha.plates[:-1] # ... and use it as the dimensionality of the Wishart # distribution if len(alpha.plates) == 0: raise Exception("Gamma variable needs to have plates in " "order to be used as a diagonal Wishart.") D = alpha.plates[-1] dims = ( (D,D), () ) # Construct the node super().__init__(alpha, #plates=plates, dims=dims, **kwargs) def _plates_to_parent(self, index): D = self.dims[0][0] return self.plates + (D,) def _plates_from_parent(self, index): return self.parents[index].plates[:-1] @staticmethod def _compute_mask_to_parent(index, mask): return mask[..., np.newaxis] def get_moments(self): u = self.parents[0].get_moments() # Form a diagonal matrix from the gamma variables return [np.identity(self.dims[0][0]) * u[0][...,np.newaxis], np.sum(u[1], axis=(-1))] @staticmethod def _compute_message_to_parent(index, m_children, *u_parents): # Take the diagonal m0 = np.einsum('...ii->...i', m_children[0]) m1 = np.reshape(m_children[1], np.shape(m_children[1]) + (1,)) return [m0, m1] ## def _compute_message_and_mask_to_parent(self, index, m_children, *u_parents): ## m = self._message_from_children() ## #(m, mask) = self.message_from_children() ## # Take the diagonal ## m[0] = np.einsum('...ii->...i', m[0]) ## # TODO/FIXME: I think m[1] is wrong.. ## m[1] = np.reshape(m[1], np.shape(m[1]) + (1,)) ## # m[1] is ok ## mask = self._compute_mask_to_parent(index, self.mask) ## #mask = mask[...,np.newaxis] ## return (m, mask) ## #return m bayespy-0.4.0/bayespy/inference/vmp/nodes/expfamily.py0000644000175000017500000004322412556612757024626 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2013-2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import warnings import numpy as np from bayespy.utils import misc from .node import ensureparents from .stochastic import Stochastic, Distribution class ExponentialFamilyDistribution(Distribution): """ Sub-classes implement distribution specific computations. """ # # The following methods are for ExponentialFamily distributions # def compute_message_to_parent(self, parent, index, u_self, *u_parents): raise NotImplementedError() def compute_phi_from_parents(self, *u_parents, mask=True): raise NotImplementedError() def compute_moments_and_cgf(self, phi, mask=True): raise NotImplementedError() # # The following methods are for Mixture class # def compute_cgf_from_parents(self, *u_parents): raise NotImplementedError() def compute_fixed_moments_and_f(self, x, mask=True): raise NotImplementedError() def compute_logpdf(self, u, phi, g, f, ndims): """ Compute E[log p(X)] given E[u], E[phi], E[g] and E[f]. Does not sum over plates.""" # TODO/FIXME: Should I take into account what is latent or # observed, or what is even totally ignored (by the mask). L = g + f for (phi_i, u_i, ndims_i) in zip(phi, u, ndims): # Axes to sum (dimensions of the variable, not the plates) axis_sum = tuple(range(-ndims_i,0)) # Compute the term # TODO/FIXME: Use einsum! L = L + np.sum(phi_i * u_i, axis=axis_sum) return L def compute_gradient(self, g, u, phi): r""" Compute the standard gradient with respect to the natural parameters. """ raise NotImplementedError("Standard gradient not yet implemented for %s" % (self.__class__.__name__)) def useconstructor(__init__): def constructor_decorator(self, *args, **kwargs): if (self.dims is None or self._distribution is None or self._moments is None or self._parent_moments is None): (args, kwargs, dims, plates, dist, stats, pstats) = \ self._constructor(*args, **kwargs) self.dims = dims self._distribution = dist self._moments = stats self._parent_moments = pstats self.plates = plates __init__(self, *args, **kwargs) return constructor_decorator class ExponentialFamily(Stochastic): """ A base class for nodes using natural parameterization `phi`. phi Sub-classes must implement the following static methods: _compute_message_to_parent(index, u_self, *u_parents) _compute_phi_from_parents(*u_parents, mask) _compute_moments_and_cgf(phi, mask) _compute_fixed_moments_and_f(x, mask=True) Sub-classes may need to re-implement: 1. If they manipulate plates: _compute_mask_to_parent(index, mask) _compute_plates_to_parent(self, index, plates) _compute_plates_from_parent(self, index, plates) """ # Sub-classes should overwrite this (possibly using _constructor) dims = None # Sub-classes should overwrite this _distribution = None @useconstructor def __init__(self, *parents, initialize=True, **kwargs): self.annealing = 1.0 # Terms for the lower bound (G for latent and F for observed) self.g = np.array(np.nan) self.f = np.array(np.nan) super().__init__(*parents, initialize=initialize, dims=self.dims, **kwargs) if not initialize: axes = len(self.plates)*(1,) self.phi = [misc.nans(axes+dim) for dim in self.dims] @classmethod @ensureparents def _constructor(cls, *parents, **kwargs): """ Constructs distribution and moments objects. If __init__ uses useconstructor decorator, this method is called to construct distribution and moments objects. The method is given the same inputs as __init__. For some nodes, some of these can't be "static" class attributes, then the node class must overwrite this method to construct the objects manually. The point of distribution class is to move general distribution but not-node specific code. The point of moments class is to define the messaging protocols. """ parent_plates = [cls._distribution.plates_from_parent(ind, parent.plates) for (ind, parent) in enumerate(parents)] return (parents, kwargs, cls.dims, cls._total_plates(kwargs.get('plates'), *parent_plates), cls._distribution, cls._moments, cls._parent_moments) def _initialize_from_parent_moments(self, *u_parents): if not np.all(self.observed): # Update natural parameters using parents self._update_phi_from_parents(*u_parents) # Update moments mask = np.logical_not(self.observed) (u, g) = self._distribution.compute_moments_and_cgf(self.phi, mask=mask) # TODO/FIXME/BUG: You should use observation mask in order to not # overwrite them! self._set_moments_and_cgf(u, g, mask=mask) def initialize_from_prior(self): u_parents = self._message_from_parents() self._initialize_from_parent_moments(*u_parents) def initialize_from_parameters(self, *args): u_parents = [p_mom.compute_fixed_moments(x) for (p_mom, x) in zip(self._parent_moments, args)] self._initialize_from_parent_moments(*u_parents) def initialize_from_value(self, x, *args): # Update moments from value mask = np.logical_not(self.observed) u = self._moments.compute_fixed_moments(x, *args) # Check that the shape is correct for i in range(len(u)): ndim = len(self.dims[i]) if ndim > 0: if np.shape(u[i])[-ndim:] != self.dims[i]: raise ValueError("The initial value for node %s has invalid shape %s." % (np.shape(x))) self._set_moments_and_cgf(u, np.inf, mask=mask) def initialize_from_random(self): """ Set the variable to a random sample from the current distribution. """ #self.initialize_from_prior() X = self.random() self.initialize_from_value(X) def _update_phi_from_parents(self, *u_parents): # TODO/FIXME: Could this be combined to the function # _update_distribution_and_lowerbound ? # No, because some initialization methods may want to use this. # This makes correct broadcasting self.phi = self._distribution.compute_phi_from_parents(*u_parents) #self.phi = self._compute_phi_from_parents(*u_parents) self.phi = list(self.phi) # Make sure phi has the correct number of axes. It makes life # a bit easier elsewhere. for i in range(len(self.phi)): axes = len(self.plates) + self.ndims[i] - np.ndim(self.phi[i]) if axes > 0: # Add axes self.phi[i] = misc.add_leading_axes(self.phi[i], axes) elif axes < 0: # Remove extra leading axes first = -(len(self.plates)+self.ndims[i]) sh = np.shape(self.phi[i])[first:] self.phi[i] = np.reshape(self.phi[i], sh) # Check that the shape is correct if not misc.is_shape_subset(np.shape(self.phi[i]), self.get_shape(i)): raise ValueError("Incorrect shape of phi[%d] in node class %s. " "Shape is %s but it should be broadcastable " "to shape %s." % (i, self.__class__.__name__, np.shape(self.phi[i]), self.get_shape(i))) def _set_moments_and_cgf(self, u, g, mask=True): self._set_moments(u, mask=mask) self.g = np.where(mask, g, self.g) return def get_riemannian_gradient(self): r""" Computes the Riemannian/natural gradient. """ u_parents = self._message_from_parents() m_children = self._message_from_children() # TODO/FIXME: Put observed plates to zero? # Compute the gradient phi = self._distribution.compute_phi_from_parents(*u_parents) for i in range(len(self.phi)): phi[i] = self.annealing * (phi[i] + m_children[i]) - self.phi[i] phi[i] = phi[i] * np.ones(self.get_shape(i)) return phi def get_gradient(self, rg): r""" Computes gradient with respect to the natural parameters. The function takes the Riemannian gradient as an input. This is for three reasons: 1) You probably want to use the Riemannian gradient anyway so this helps avoiding accidental use of this function. 2) The gradient is computed by using the Riemannian gradient and chain rules. 3) Probably you need both Riemannian and normal gradients anyway so you can provide it to this function to avoid re-computing it.""" g = self._distribution.compute_gradient(rg, self.u, self.phi) for i in range(len(g)): g[i] /= self.annealing return g ## def update_parameters(self, d, scale=1.0): ## r""" ## Update the parameters of the VB distribution given a change. ## The parameters should be such that they can be used for ## optimization, that is, use log transformation for positive ## parameters. ## """ ## phi = self.get_parameters() ## for i in range(len(phi)): ## phi[i] = phi[i] + scale*d[i] ## self.set_parameters(phi) ## return def get_parameters(self): r""" Return parameters of the VB distribution. The parameters should be such that they can be used for optimization, that is, use log transformation for positive parameters. """ return [np.copy(p) for p in self.phi] def _decode_parameters(self, x): return [np.copy(p) for p in x] def set_parameters(self, x): r""" Set the parameters of the VB distribution. The parameters should be such that they can be used for optimization, that is, use log transformation for positive parameters. """ self.phi = self._decode_parameters(x) self._update_moments_and_cgf() return def _update_distribution_and_lowerbound(self, m_children, *u_parents): # Update phi first from parents.. self._update_phi_from_parents(*u_parents) # .. then just add children's message self.phi = [self.annealing * (phi + m) for (phi, m) in zip(self.phi, m_children)] # Update u and g self._update_moments_and_cgf() def _update_moments_and_cgf(self): """ Update moments and cgf based on current phi. """ # Mask for plates to update (i.e., unobserved plates) update_mask = np.logical_not(self.observed) # Compute the moments (u) and CGF (g)... (u, g) = self._distribution.compute_moments_and_cgf(self.phi, mask=update_mask) # ... and store them self._set_moments_and_cgf(u, g, mask=update_mask) def observe(self, x, *args, mask=True): """ Fix moments, compute f and propagate mask. """ # Compute fixed moments (u, f) = self._distribution.compute_fixed_moments_and_f(x, *args, mask=mask) # Check the dimensionality of the observations for (i,v) in enumerate(u): # This is what the dimensionality "should" be s = self.plates + self.dims[i] t = np.shape(v) if s != t: msg = "Dimensionality of the observations incorrect." msg += "\nShape of input: " + str(t) msg += "\nExpected shape: " + str(s) msg += "\nCheck plates." raise Exception(msg) # Set the moments self._set_moments(u, mask=mask) self.f = np.where(mask, f, self.f) # Observed nodes should not be ignored self.observed = mask self._update_mask() def lower_bound_contribution(self, gradient=False, ignore_masked=True): r"""Compute E[ log p(X|parents) - log q(X) ] If deterministic annealing is used, the term E[ -log q(X) ] is divided by the anneling coefficient. That is, phi and cgf of q are multiplied by the temperature (inverse annealing coefficient). """ # Annealing temperature T = 1 / self.annealing # Messages from parents u_parents = self._message_from_parents() phi = self._distribution.compute_phi_from_parents(*u_parents) # G from parents L = self._distribution.compute_cgf_from_parents(*u_parents) # G for unobserved variables (ignored variables are handled properly # automatically) latent_mask = np.logical_not(self.observed) # G and F if np.all(self.observed): z = np.nan elif T == 1: z = -self.g else: z = -T * self.g ## TRIED THIS BUT IT WAS WRONG: ## z = -T * self.g + (1-T) * self.f ## if np.any(np.isnan(self.f)): ## warnings.warn("F(x) not implemented for node %s. This " ## "is required for annealed lower bound " ## "computation." % self.__class__.__name__) ## ## It was wrong because the optimal q distribution has f which is ## weighted by 1/T and here the f of q is weighted by T so the ## total weight is 1, thus it cancels out with f of p. L = L + np.where(self.observed, self.f, z) for (phi_p, phi_q, u_q, dims) in zip(phi, self.phi, self.u, self.dims): # Form a mask which puts observed variables to zero and # broadcasts properly latent_mask_i = misc.add_trailing_axes( misc.add_leading_axes( latent_mask, len(self.plates) - np.ndim(latent_mask)), len(dims)) axis_sum = tuple(range(-len(dims),0)) # Compute the term phi_q = np.where(latent_mask_i, phi_q, 0) # Apply annealing # TODO/FIXME: Use einsum here? Z = np.sum((phi_p-T*phi_q) * u_q, axis=axis_sum) L = L + Z if ignore_masked: return (np.sum(np.where(self.mask, L, 0)) * self.broadcasting_multiplier(self.plates, np.shape(L), np.shape(self.mask)) * np.prod(self.plates_multiplier)) else: return (np.sum(L) * self.broadcasting_multiplier(self.plates, np.shape(L)) * np.prod(self.plates_multiplier)) def logpdf(self, X, mask=True): """ Compute the log probability density function Q(X) of this node. """ if mask is not True: raise NotImplementedError('Mask not yet implemented') (u, f) = self._distribution.compute_fixed_moments_and_f(X, mask=mask) Z = 0 for (phi_d, u_d, dims) in zip(self.phi, u, self.dims): axis_sum = tuple(range(-len(dims),0)) # TODO/FIXME: Use einsum here? Z = Z + np.sum(phi_d * u_d, axis=axis_sum) #Z = Z + misc.sum_multiply(phi_d, u_d, axis=axis_sum) return (self.g + f + Z) def pdf(self, X, mask=True): """ Compute the probability density function of this node. """ return np.exp(self.logpdf(X, mask=mask)) def _save(self, group): """ Save the state of the node into a HDF5 file. group can be the root """ ## if name is None: ## name = self.name ## subgroup = group.create_group(name) for i in range(len(self.phi)): misc.write_to_hdf5(group, self.phi[i], 'phi%d' % i) misc.write_to_hdf5(group, self.f, 'f') misc.write_to_hdf5(group, self.g, 'g') super()._save(group) def _load(self, group): """ Load the state of the node from a HDF5 file. """ # TODO/FIXME: Check that the shapes are correct! for i in range(len(self.phi)): phii = group['phi%d' % i][...] self.phi[i] = phii self.f = group['f'][...] self.g = group['g'][...] super()._load(group) def random(self): """ Draw a random sample from the distribution. """ return self._distribution.random(*(self.phi), plates=self.plates) bayespy-0.4.0/bayespy/inference/vmp/nodes/exponential.py0000644000175000017500000000413112533543154025135 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Module for the exponential distribution node. """ from .gamma import (GammaMoments, Gamma) from .expfamily import ExponentialFamily ExponentialMoments = GammaMoments class Exponential(Gamma): r""" Node for exponential random variables. .. warning:: Use :class:`Gamma` instead of this. `Exponential(l)` is equivalent to `Gamma(1, l)`. Parameters ---------- l : gamma-like node or scalar or array Rate parameter See also -------- Gamma, Poisson Notes ----- For simplicity, this is just a gamma node with the first parent fixed to one. Note that this is a bit inconsistent with the BayesPy philosophy which states that the node does not only define the form of the prior distribution but more importantly the form of the posterior approximation. Thus, one might expect that this node would have exponential posterior distribution approximation. However, it has a gamma distribution. Also, the moments are gamma moments although only E[x] would be the moment of a exponential random variable. All this was done because: a) gamma was already implemented, so there was no need to implement anything, and b) people might easily use Exponential node as a prior definition and expect to get gamma posterior (which is what happens now). Maybe some day a pure Exponential node is implemented and the users are advised to use Gamma(1,b) if they want to use an exponential prior distribution but gamma posterior approximation. """ def __init__(self, l, **kwargs): raise NotImplementedError("Not yet implemented. Use Gamma(1, lambda)") super().__init__(1, l, **kwargs) @classmethod def _constructor(cls, l, **kwargs): raise NotImplementedError("Not yet implemented. Use Gamma(1, lambda)") bayespy-0.4.0/bayespy/inference/vmp/nodes/stochastic.py0000644000175000017500000002365312533543154024765 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2013-2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import numpy as np from bayespy.utils import misc import h5py from .node import Node class Distribution(): """ A base class for the VMP formulas of variables. Sub-classes implement distribution specific computations. If a sub-class maps the plates differently, it needs to overload the following methods: * compute_mask_to_parent * plates_to_parent * plates_from_parent """ def compute_message_to_parent(self, parent, index, u_self, *u_parents): """ Compute the message to a parent node. """ raise NotImplementedError() def compute_mask_to_parent(self, index, mask): """ Maps the mask to the plates of a parent. """ # Sub-classes may need to overwrite this method return mask def plates_to_parent(self, index, plates): """ Resolves the plate mapping to a parent. Given the plates of the node's moments, this method returns the plates that the message to a parent has for the parent's distribution. """ return plates def plates_from_parent(self, index, plates): """ Resolve the plate mapping from a parent. Given the plates of a parent's moments, this method returns the plates that the moments has for this distribution. """ return plates def random(self, *params, plates=None): """ Draw a random sample from the distribution. """ raise NotImplementedError() class Stochastic(Node): """ Base class for nodes that are stochastic. u observed Sub-classes must implement: _compute_message_to_parent(parent, index, u_self, *u_parents) _update_distribution_and_lowerbound(self, m, *u) lowerbound(self) _compute_dims initialize_from_prior() If you want to be able to observe the variable: _compute_fixed_moments_and_f Sub-classes may need to re-implement: 1. If they manipulate plates: _compute_mask_to_parent(index, mask) _compute_plates_to_parent(self, index, plates) _compute_plates_from_parent(self, index, plates) """ # Sub-classes must over-write this _distribution = None def __init__(self, *args, initialize=True, dims=None, **kwargs): self._id = Node._id_counter Node._id_counter += 1 super().__init__(*args, dims=dims, **kwargs) # Initialize moment array axes = len(self.plates)*(1,) self.u = [misc.nans(axes+dim) for dim in dims] # Not observed self.observed = False self.ndims = [len(dim) for dim in self.dims] if initialize: self.initialize_from_prior() def _get_id_list(self): """ Returns the stochastic ID list. This method is used to check that same stochastic nodes are not direct parents of a node several times. It is only valid if there are intermediate stochastic nodes. To put it another way: each ID corresponds to one factor q(..) in the posterior approximation. Different IDs mean different factors, thus they mean independence. The parents must have independent factors. Stochastic nodes should return their unique ID. Deterministic nodes should return the IDs of their parents. Constant nodes should return empty list of IDs. """ return [self._id] def _compute_plates_to_parent(self, index, plates): return self._distribution.plates_to_parent(index, plates) def _compute_plates_from_parent(self, index, plates): return self._distribution.plates_from_parent(index, plates) def _compute_mask_to_parent(self, index, mask): return self._distribution.compute_mask_to_parent(index, mask) def get_moments(self): # Just for safety, do not return a reference to the moment list of this # node but instead create a copy of the list. return [ui for ui in self.u] def _get_message_and_mask_to_parent(self, index): u_parents = self._message_from_parents(exclude=index) m = self._distribution.compute_message_to_parent(self.parents[index], index, self.u, *u_parents) mask = self._distribution.compute_mask_to_parent(index, self.mask) ## m = self._compute_message_to_parent(self.parents[index], index, self.u, *u_parents) ## mask = self._compute_mask_to_parent(index, self.mask) return (m, mask) def _set_mask(self, mask): self.mask = np.logical_or(mask, self.observed) def _set_moments(self, u, mask=True): # Store the computed moments u but do not change moments for # observations, i.e., utilize the mask. for ind in range(len(u)): # Add axes to the mask for the variable dimensions (mask # contains only axes for the plates). u_mask = misc.add_trailing_axes(mask, self.ndims[ind]) # Enlarge self.u[ind] as necessary so that it can store the # broadcasted result. sh = misc.broadcasted_shape_from_arrays(self.u[ind], u[ind], u_mask) self.u[ind] = misc.repeat_to_shape(self.u[ind], sh) # TODO/FIXME/BUG: The mask of observations is not used, observations # may be overwritten!!! ??? # Hah, this function is used to set the observations! The caller # should be careful what mask he uses! If you want to set only # latent variables, then use such a mask. # Use mask to update only unobserved plates and keep the # observed as before np.copyto(self.u[ind], u[ind], where=u_mask) # Make sure u has the correct number of dimensions: # TODO/FIXME: Maybe it would be good to also check that u has a # shape that is a sub-shape of get_shape. shape = self.get_shape(ind) ndim = len(shape) ndim_u = np.ndim(self.u[ind]) if ndim > ndim_u: self.u[ind] = misc.add_leading_axes(u[ind], ndim - ndim_u) elif ndim < ndim_u: raise RuntimeError( "The size of the variable %s's %s-th moment " "array is %s which is larger than it should " "be, that is, %s, based on the plates %s and " "dimension %s. Check that you have provided " "plates properly." % (self.name, ind, np.shape(self.u[ind]), shape, self.plates, self.dims[ind])) def update(self, annealing=1.0): if not np.all(self.observed): u_parents = self._message_from_parents() m_children = self._message_from_children() if annealing != 1.0: m_children = [annealing * m for m in m_children] self._update_distribution_and_lowerbound(m_children, *u_parents) def observe(self, x, mask=True): """ Fix moments, compute f and propagate mask. """ raise NotImplementedError() def unobserve(self): # Update mask self.observed = False self._update_mask() def lowerbound(self): # Sub-class should implement this raise NotImplementedError() def _update_distribution_and_lowerbound(self, m_children, *u_parents): # Sub-classes should implement this raise NotImplementedError() def save(self, filename): # Open HDF5 file h5f = h5py.File(filename, 'w') try: # Write each node nodegroup = h5f.create_group('nodes') if self.name == '': raise Exception("In order to save nodes, they must have " "(unique) names.") self._save(nodegroup.create_group(node.name)) finally: # Close file h5f.close() def _save(self, group): """ Save the state of the node into a HDF5 file. group can be the root """ for i in range(len(self.u)): misc.write_to_hdf5(group, self.u[i], 'u%d' % i) misc.write_to_hdf5(group, self.observed, 'observed') return def load(self, filename): h5f = h5py.File(filename, 'r') try: self._load(h5f['nodes'][self.name]) finally: h5f.close() return def _load(self, group): """ Load the state of the node from a HDF5 file. """ # TODO/FIXME: Check that the shapes are correct! for i in range(len(self.u)): ui = group['u%d' % i][...] self.u[i] = ui old_observed = self.observed self.observed = group['observed'][...] # Update masks if necessary if np.any(old_observed != self.observed): self._update_mask() def random(self): """ Draw a random sample from the distribution. """ raise NotImplementedError() def show(self): """ Print the distribution using standard parameterization. """ print(str(self)) def __str__(self): """ """ raise NotImplementedError("String representation not yet implemented for " "node class %s" % (self.__class__.__name__)) bayespy-0.4.0/bayespy/inference/vmp/nodes/gate.py0000644000175000017500000001553012533543154023534 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ """ import numpy as np from bayespy.utils import misc from .node import Node, Moments from .deterministic import Deterministic from .categorical import CategoricalMoments class Gate(Deterministic): """ Deterministic gating of one node. Gating is performed over one plate axis. Note: You should not use gating for several variables which parents of a same node if the gates use the same gate assignments. In such case, the results will be wrong. The reason is a general one: A stochastic node may not be a parent of another node via several paths unless at most one path has no other stochastic nodes between them. """ def __init__(self, Z, X, gated_plate=-1, moments=None, **kwargs): """ Constructor for the gating node. Parameters ---------- Z : Categorical-like node A variable which chooses the index along the gated plate axis X : node The node whose plate axis is gated gated_plate : int (optional) The index of the plate axis to be gated (by default, -1, that is, the last axis). """ if gated_plate >= 0: raise ValueError("Cluster plate must be negative integer") self.gated_plate = gated_plate if moments is not None: X = self._ensure_moments(X, moments) if not isinstance(X, Node): raise ValueError("X must be a node or moments should be provided") X_moments = X._moments self._moments = X_moments dims = X.dims if len(X.plates) < abs(gated_plate): raise ValueError("The gated node does not have a plate axis is " "gated") K = X.plates[gated_plate] self._parent_moments = [CategoricalMoments(K), X_moments] Z = self._ensure_moments(Z, self._parent_moments[0]) if Z.dims != ( (K,), ): raise ValueError("Inconsistent number of clusters") self.K = K super().__init__(Z, X, dims=dims, **kwargs) def _compute_moments(self, u_Z, u_X): """ """ u = [] for i in range(len(u_X)): # Make the moments of Z and X broadcastable and move the gated plate # to be the last axis in the moments, then sum-product over that # axis ndim = len(self.dims[i]) z = misc.add_trailing_axes(u_Z[0], ndim) z = misc.moveaxis(z, -ndim-1, -1) gated_axis = self.gated_plate - ndim if np.ndim(u_X[i]) < abs(gated_axis): x = misc.add_trailing_axes(u_X[i], 1) else: x = misc.moveaxis(u_X[i], gated_axis, -1) ui = misc.sum_product(z, x, axes_to_sum=-1) u.append(ui) return u def _compute_message_to_parent(self, index, m_child, u_Z, u_X): """ """ if index == 0: m0 = 0 # Compute Child * X, sum over variable axes and move the gated axis # to be the last. Need to do some shape changing in order to make # Child and X to broadcast properly. for i in range(len(m_child)): ndim = len(self.dims[i]) c = m_child[i][...,None] c = misc.moveaxis(c, -1, -ndim-1) gated_axis = self.gated_plate - ndim x = u_X[i] if np.ndim(x) < abs(gated_axis): x = np.expand_dims(x, -ndim-1) else: x = misc.moveaxis(x, gated_axis, -ndim-1) axes = tuple(range(-ndim, 0)) m0 = m0 + misc.sum_product(c, x, axes_to_sum=axes) # Make sure the variable axis does not use broadcasting m0 = m0 * np.ones(self.K) # Send the message m = [m0] return m elif index == 1: m = [] for i in range(len(m_child)): # Make the moments of Z and the message from children # broadcastable. The gated plate is handled as the last axis in # the arrays and moved to the correct position at the end. # Add variable axes to Z moments ndim = len(self.dims[i]) z = misc.add_trailing_axes(u_Z[0], ndim) z = misc.moveaxis(z, -ndim-1, -1) # Axis index of the gated plate gated_axis = self.gated_plate - ndim # Add the gate axis to the message from the children c = misc.add_trailing_axes(m_child[i], 1) # Compute the message to parent mi = z * c # Add extra axes if necessary if np.ndim(mi) < abs(gated_axis): mi = misc.add_leading_axes(mi, abs(gated_axis) - np.ndim(mi)) # Move the axis to the correct position mi = misc.moveaxis(mi, -1, gated_axis) m.append(mi) return m else: raise ValueError("Invalid parent index") def _compute_mask_to_parent(self, index, mask): """ """ if index == 0: return mask elif index == 1: if self.gated_plate >= 0: raise ValueError("Gated plate axis must be negative") if np.ndim(mask) >= abs(self.gated_plate): mask = np.expand_dims(mask, axis=self.gated_plate) return mask else: raise ValueError("Invalid parent index") def _compute_plates_to_parent(self, index, plates): """ """ if index == 0: return plates elif index == 1: plates = list(plates) # Add the cluster plate axis if self.gated_plate < 0: knd = len(plates) + self.gated_plate + 1 else: raise RuntimeError("Cluster plate axis must be negative") plates.insert(knd, self.K) return tuple(plates) else: raise ValueError("Invalid parent index") def _compute_plates_from_parent(self, index, plates): """ """ if index == 0: return plates elif index == 1: plates = list(plates) # Remove the cluster plate, if the parent has it if len(plates) >= abs(self.gated_plate): plates.pop(self.gated_plate) return tuple(plates) else: raise ValueError("Invalid parent index") bayespy-0.4.0/bayespy/inference/vmp/nodes/node.py0000644000175000017500000013037212556653765023562 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2013-2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import numpy as np import matplotlib.pyplot as plt from bayespy.utils import misc """ This module contains a sketch of a new implementation of the framework. """ def message_sum_multiply(plates_parent, dims_parent, *arrays): """ Compute message to parent and sum over plates. Divide by the plate multiplier. """ # The shape of the full message shapes = [np.shape(array) for array in arrays] shape_full = misc.broadcasted_shape(*shapes) # Find axes that should be summed shape_parent = plates_parent + dims_parent sum_axes = misc.axes_to_collapse(shape_full, shape_parent) # Compute the multiplier for cancelling the # plate-multiplier. Because we are summing over the # dimensions already in this function (for efficiency), we # need to cancel the effect of the plate-multiplier # applied in the message_to_parent function. r = 1 for j in sum_axes: if j >= 0 and j < len(plates_parent): r *= shape_full[j] elif j < 0 and j < -len(dims_parent): r *= shape_full[j] # Compute the sum-product m = misc.sum_multiply(*arrays, axis=sum_axes, sumaxis=True, keepdims=True) / r # Remove extra axes m = misc.squeeze_to_dim(m, len(shape_parent)) return m class Moments(): """ Base class for defining the expectation of the sufficient statistics. The benefits: * Write statistic-specific features in one place only. For instance, covariance from Gaussian message. * Different nodes may have identically defined statistic so you need to implement related features only once. For instance, Gaussian and GaussianARD differ on the prior but the moments are the same. * General processing nodes which do not change the type of the moments may "inherit" the features from the parent node. For instance, slicing operator. * Conversions can be done easily in both of the above cases if the message conversion is defined in the moments class. For instance, GaussianMarkovChain to Gaussian and VaryingGaussianMarkovChain to Gaussian. """ _converters = {} class NoConverterError(Exception): pass @classmethod def add_converter(cls, moments_to, converter): cls._converters = cls._converters.copy() cls._converters[moments_to] = converter return def get_converter(self, moments_to): """ Finds conversion to another moments type if possible. Note that a conversion from moments A to moments B may require intermediate conversions. For instance: A->C->D->B. This method finds the path which uses the least amount of conversions and returns that path as a single conversion. If no conversion path is available, an error is raised. The search algorithm starts from the original moments class and applies all possible converters to get a new list of moments classes. This list is extended by adding recursively all parent classes because their converters are applicable. Then, all possible converters are applied to this list to get a new list of current moments classes. This is iterated until the algorithm hits the target moments class or its subclass. """ # Check if there is no need for a conversion # # TODO/FIXME: This isn't sufficient. Moments can have attributes that # make them incompatible (e.g., ndim in GaussianMoments). if isinstance(self, moments_to): return lambda X: X # Initialize variables visited = set() visited.add(self.__class__) converted_list = [(self.__class__, [])] # Each iteration step consists of two parts: # 1) form a set of the current classes and all their parent classes # recursively # 2) from the current set, apply possible conversions to get a new set # of classes # Repeat these two steps until in step (1) you hit the target class. while len(converted_list) > 0: # Go through all parents recursively so we can then use all # converters that are available current_list = [] for (moments_class, converter_path) in converted_list: if issubclass(moments_class, moments_to): # Shortest conversion path found, return the resulting total # conversion function return misc.composite_function(converter_path) current_list.append((moments_class, converter_path)) parents = list(moments_class.__bases__) for parent in parents: # Recursively add parents for p in parent.__bases__: if isinstance(p, Moments): parents.append(p) # Add un-visited parents if issubclass(parent, Moments) and parent not in visited: visited.add(parent) current_list.append((parent, converter_path)) # Find all converters and extend the converter paths converted_list = [] for (moments_class, converter_path) in current_list: for (conv_mom_cls, conv) in moments_class._converters.items(): if conv_mom_cls not in visited: visited.add(conv_mom_cls) converted_list.append((conv_mom_cls, converter_path + [conv])) raise self.NoConverterError("No conversion defined from %s to %s" % (self.__class__.__name__, moments_to.__name__)) def compute_fixed_moments(self, x): # This method can't be static because the computation of the moments may # depend on, for instance, ndim in Gaussian arrays. raise NotImplementedError("compute_fixed_moments not implemented for " "%s" % (self.__class__.__name__)) def compute_dims_from_values(self, x): # This method can't be static because the computation of the moments may # depend on, for instance, ndim in Gaussian arrays. raise NotImplementedError("compute_dims_from_values not implemented " "for %s" % (self.__class__.__name__)) def ensureparents(func): def new_func(self, *parents, **kwargs): # Convert parents to proper nodes parents = list(parents) for (ind, parent) in enumerate(parents): parents[ind] = self._ensure_moments(parent, self._parent_moments[ind]) # Run the function return func(self, *parents, **kwargs) return new_func class Node(): """ Base class for all nodes. mask dims plates parents children name Sub-classes must implement: 1. For computing the message to children: get_moments(self): 2. For computing the message to parents: _get_message_and_mask_to_parent(self, index) Sub-classes may need to re-implement: 1. If they manipulate plates: _compute_mask_to_parent(index, mask) _plates_to_parent(self, index) _plates_from_parent(self, index) """ # These are objects of the _parent_moments_class. If the default way of # creating them is not correct, write your own creation code. _moments = None _parent_moments = None plates = None _id_counter = 0 @ensureparents def __init__(self, *parents, dims=None, plates=None, name="", notify_parents=True, plotter=None, plates_multiplier=None): self.parents = parents self.dims = dims self.name = name self._plotter = plotter parent_id_list = [] for parent in parents: parent_id_list = parent_id_list + list(parent._get_id_list()) if len(parent_id_list) != len(set(parent_id_list)): raise ValueError("Parent nodes are not independent") # Inform parent nodes if notify_parents: for (index,parent) in enumerate(self.parents): parent._add_child(self, index) # Check plates parent_plates = [self._plates_from_parent(index) for index in range(len(self.parents))] if any(p is None for p in parent_plates): raise ValueError("Method _plates_from_parent returned None") # Get and validate the plates for this node plates = self._total_plates(plates, *parent_plates) if self.plates is None: self.plates = plates # By default, ignore all plates self.mask = np.array(False) # Children self.children = set() # Get and validate the plate multiplier parent_plates_multiplier = [self._plates_multiplier_from_parent(index) for index in range(len(self.parents))] #if plates_multiplier is None: # plates_multiplier = parent_plates_multiplier plates_multiplier = self._total_plates(plates_multiplier, *parent_plates_multiplier) self.plates_multiplier = plates_multiplier def _get_id_list(self): """ Returns the stochastic ID list. This method is used to check that same stochastic nodes are not direct parents of a node several times. It is only valid if there are intermediate stochastic nodes. To put it another way: each ID corresponds to one factor q(..) in the posterior approximation. Different IDs mean different factors, thus they mean independence. The parents must have independent factors. Stochastic nodes should return their unique ID. Deterministic nodes should return the IDs of their parents. Constant nodes should return empty list of IDs. """ raise NotImplementedError() @classmethod def _total_plates(cls, plates, *parent_plates): if plates is None: # By default, use the minimum number of plates determined # from the parent nodes try: return misc.broadcasted_shape(*parent_plates) except ValueError: raise ValueError("The plates of the parents do not broadcast.") else: # Check that the parent_plates are a subset of plates. for (ind, p) in enumerate(parent_plates): if not misc.is_shape_subset(p, plates): raise ValueError("The plates %s of the parents " "are not broadcastable to the given " "plates %s." % (p, plates)) return plates @staticmethod def _ensure_moments(node, moments): if isinstance(node, Node): # Convert to correct type return node._convert(moments.__class__) # Convert to constant node from .constant import Constant return Constant(moments, node) def _compute_plates_to_parent(self, index, plates): # Sub-classes may want to overwrite this if they manipulate plates return plates def _compute_plates_from_parent(self, index, plates): # Sub-classes may want to overwrite this if they manipulate plates return plates def _plates_to_parent(self, index): return self._compute_plates_to_parent(index, self.plates) def _plates_from_parent(self, index): return self._compute_plates_from_parent(index, self.parents[index].plates) def _plates_multiplier_from_parent(self, index): return self._compute_plates_from_parent(index, self.parents[index].plates_multiplier) @property def plates_multiplier(self): """ Plate multiplier is applied to messages to parents """ return self.__plates_multiplier @plates_multiplier.setter def plates_multiplier(self, value): # TODO/FIXME: Check that multiplier is consistent with plates self.__plates_multiplier = value return def get_shape(self, ind): return self.plates + self.dims[ind] def _add_child(self, child, index): """ Add a child node. Parameters ---------- child : node index : int The parent index of this node for the child node. The child node recognizes its parents by their index number. """ self.children.add((child, index)) def _remove_child(self, child, index): """ Remove a child node. """ self.children.remove((child, index)) def get_mask(self): return self.mask ## def _get_message_mask(self): ## return self.mask def _set_mask(self, mask): # Sub-classes may overwrite this method if they have some other masks to # be combined (for instance, observation mask) self.mask = mask def _update_mask(self): # Combine masks from children mask = np.array(False) for (child, index) in self.children: mask = np.logical_or(mask, child._mask_to_parent(index)) # Set the mask of this node self._set_mask(mask) if not misc.is_shape_subset(np.shape(self.mask), self.plates): raise ValueError("The mask of the node %s has updated " "incorrectly. The plates in the mask %s are not a " "subset of the plates of the node %s." % (self.name, np.shape(self.mask), self.plates)) # Tell parents to update their masks for parent in self.parents: parent._update_mask() ## @staticmethod ## def _compute_mask_to_parent(index, mask): ## # Sub-classes may want to overwrite this if they do something to plates. ## return mask # TODO: Rename to _compute_message_mask_to_parent(index, mask) def _compute_mask_to_parent(self, index, mask): """ Compute the mask used for messages sent to parent[index]. The mask tells which plates in the messages are active. This method is used for obtaining the mask which is used to set plates in the messages to parent to zero. Sub-classes may want to overwrite this method if they do something to plates so that the mask is somehow altered. """ return mask def _mask_to_parent(self, index): """ Get the mask with respect to parent[index]. The mask tells which plate connections are active. The mask is "summed" (logical or) and reshaped into the plate shape of the parent. Thus, it can't be used for masking messages, because some plates have been summed already. This method is used for propagating the mask to parents. """ mask = self._compute_mask_to_parent(index, self.mask) # Check the shape of the mask plates_to_parent = self._plates_to_parent(index) if not misc.is_shape_subset(np.shape(mask), plates_to_parent): raise ValueError("In node %s, the mask being sent to " "parent[%d] (%s) has invalid shape: The shape of " "the mask %s is not a sub-shape of the plates of " "the node with respect to the parent %s. It could " "be that this node (%s) is manipulating plates " "but has not overwritten the method " "_compute_mask_to_parent." % (self.name, index, self.parents[index].name, np.shape(mask), plates_to_parent, self.__class__.__name__)) # "Sum" (i.e., logical or) over the plates that have unit length in # the parent node. parent_plates = self.parents[index].plates s = misc.axes_to_collapse(np.shape(mask), parent_plates) mask = np.any(mask, axis=s, keepdims=True) mask = misc.squeeze_to_dim(mask, len(parent_plates)) return mask #return self._compute_mask_to_parent(index, self.get_mask()) def _message_to_child(self): u = self.get_moments() # Debug: Check that the message has appropriate shape for (ui, dim) in zip(u, self.dims): ndim = len(dim) if ndim > 0: if np.shape(ui)[-ndim:] != dim: raise RuntimeError( "A bug found by _message_to_child for %s: " "The variable axes of the moments %s are not equal to " "the axes %s defined by the node %s. A possible reason " "is that the plates of the node are inferred " "incorrectly from the parents, and the method " "_plates_from_parents should be implemented." % (self.__class__.__name__, np.shape(ui)[-ndim:], dim, self.name)) if not misc.is_shape_subset(np.shape(ui)[:-ndim], self.plates): raise RuntimeError( "A bug found by _message_to_child for %s: " "The plate axes of the moments %s are not a subset of " "the plate axes %s defined by the node %s." % (self.__class__.__name__, np.shape(ui)[:-ndim], self.plates, self.name)) else: if not misc.is_shape_subset(np.shape(ui), self.plates): raise RuntimeError( "A bug found by _message_to_child for %s: " "The plate axes of the moments %s are not a subset of " "the plate axes %s defined by the node %s." % (self.__class__.__name__, np.shape(ui), self.plates, self.name)) return u def _message_to_parent(self, index): # Compute the message, check plates, apply mask and sum over some plates if index >= len(self.parents): raise ValueError("Parent index larger than the number of parents") # Compute the message and mask (m, mask) = self._get_message_and_mask_to_parent(index) mask = misc.squeeze(mask) # Plates in the mask plates_mask = np.shape(mask) # The parent we're sending the message to parent = self.parents[index] # Plates with respect to the parent plates_self = self._plates_to_parent(index) # Plate multiplier of the parent multiplier_parent = self._plates_multiplier_from_parent(index) # Check if m is a logpdf function (for black-box variational inference) if callable(m): def m_function(*args): lpdf = m(*args) # Log pdf only contains plate axes! plates_m = np.shape(lpdf) r = (self.broadcasting_multiplier(plates_self, plates_m, plates_mask, parent.plates) * self.broadcasting_multiplier(self.plates_multiplier, multiplier_parent)) axes_msg = misc.axes_to_collapse(plates_m, parent.plates) m[i] = misc.sum_multiply(mask_i, m[i], r, axis=axes_msg, keepdims=True) # Remove leading singular plates if the parent does not have # those plate axes. m[i] = misc.squeeze_to_dim(m[i], len(shape_parent)) return m_function raise NotImplementedError() # Compact the message to a proper shape for i in range(len(m)): # Empty messages are given as None. We can ignore those. if m[i] is not None: try: r = self.broadcasting_multiplier(self.plates_multiplier, multiplier_parent) except: raise ValueError("The plate multipliers are incompatible. " "This node (%s) has %s and parent[%d] " "(%s) has %s" % (self.name, self.plates_multiplier, index, parent.name, multiplier_parent)) ndim = len(parent.dims[i]) # Source and target shapes if ndim > 0: dims = misc.broadcasted_shape(np.shape(m[i])[-ndim:], parent.dims[i]) from_shape = plates_self + dims else: from_shape = plates_self to_shape = parent.get_shape(i) # Add variable axes to the mask mask_i = misc.add_trailing_axes(mask, ndim) # Apply mask and sum plate axes as necessary (and apply plate # multiplier) m[i] = r * misc.sum_multiply_to_plates(m[i], mask_i, to_plates=to_shape, from_plates=from_shape, ndim=0) return m def _message_from_children(self): msg = [np.zeros(shape) for shape in self.dims] #msg = [np.array(0.0) for i in range(len(self.dims))] for (child,index) in self.children: m = child._message_to_parent(index) for i in range(len(self.dims)): if m[i] is not None: # Check broadcasting shapes sh = misc.broadcasted_shape(self.get_shape(i), np.shape(m[i])) try: # Try exploiting broadcasting rules msg[i] += m[i] except ValueError: msg[i] = msg[i] + m[i] return msg def _message_from_parents(self, exclude=None): return [list(parent._message_to_child()) if ind != exclude else None for (ind,parent) in enumerate(self.parents)] def get_moments(self): raise NotImplementedError() def delete(self): """ Delete this node and the children """ for child in self.children: child.delete() for (ind, parent) in enumerate(self.parents): parent._remove_child(self, ind) @staticmethod def broadcasting_multiplier(plates, *args): return misc.broadcasting_multiplier(plates, *args) ## """ ## Compute the plate multiplier for given shapes. ## The first shape is compared to all other shapes (using NumPy ## broadcasting rules). All the elements which are non-unit in the first ## shape but 1 in all other shapes are multiplied together. ## This method is used, for instance, for computing a correction factor for ## messages to parents: If this node has non-unit plates that are unit ## plates in the parent, those plates are summed. However, if the message ## has unit axis for that plate, it should be first broadcasted to the ## plates of this node and then summed to the plates of the parent. In ## order to avoid this broadcasting and summing, it is more efficient to ## just multiply by the correct factor. This method computes that ## factor. The first argument is the full plate shape of this node (with ## respect to the parent). The other arguments are the shape of the message ## array and the plates of the parent (with respect to this node). ## """ ## # Check broadcasting of the shapes ## for arg in args: ## misc.broadcasted_shape(plates, arg) ## # Check that each arg-plates are a subset of plates? ## for arg in args: ## if not misc.is_shape_subset(arg, plates): ## raise ValueError("The shapes in args are not a sub-shape of " ## "plates.") ## r = 1 ## for j in range(-len(plates),0): ## mult = True ## for arg in args: ## # if -j <= len(arg) and arg[j] != 1: ## if not (-j > len(arg) or arg[j] == 1): ## mult = False ## if mult: ## r *= plates[j] ## return r def move_plates(self, from_plate, to_plate): return _MovePlate(self, from_plate, to_plate, name=self.name + ".move_plates") def add_plate_axis(self, to_plate): return AddPlateAxis(to_plate)(self, name=self.name+".add_plate_axis") def __getitem__(self, index): return Slice(self, index, name=(self.name+".__getitem__")) def has_plotter(self): """ Return True if the node has a plotter """ return callable(self._plotter) def set_plotter(self, plotter): self._plotter = plotter def plot(self, fig=None, **kwargs): """ Plot the node distribution using the plotter of the node Because the distributions are in general very difficult to plot, the user must specify some functions which performs the plotting as wanted. See, for instance, bayespy.plot.plotting for available plotters, that is, functions that perform plotting for a node. """ if fig is None: fig = plt.gcf() if callable(self._plotter): ax = self._plotter(self, fig=fig, **kwargs) fig.suptitle('q(%s)' % self.name) return ax else: raise Exception("No plotter defined, can not plot") def _convert(self, moments_class): converter = self._moments.get_converter(moments_class) return converter(self) ## def convert(self, node_class): ## return self._convert(node_class._moments_class) @staticmethod def _compute_message(*arrays, plates_from=(), plates_to=(), ndim=0): """ A general function for computing messages by sum-multiply The function computes the product of the input arrays and then sums to the requested plates. """ # Check that the plates broadcast properly if not misc.is_shape_subset(plates_to, plates_from): raise ValueError("plates_to must be broadcastable to plates_from") # Compute the explicit shape of the product shapes = [np.shape(array) for array in arrays] arrays_shape = misc.broadcasted_shape(*shapes) # Compute plates and dims that are present if ndim == 0: arrays_plates = arrays_shape dims = () else: arrays_plates = arrays_shape[:-ndim] dims = arrays_shape[-ndim:] # Compute the correction term. If some of the plates that should be # summed are actually broadcasted, one must multiply by the size of the # corresponding plate r = Node.broadcasting_multiplier(plates_from, arrays_plates, plates_to) # For simplicity, make the arrays equal ndim arrays = misc.make_equal_ndim(*arrays) # Keys for the input plates: (N-1, N-2, ..., 0) nplates = len(arrays_plates) in_plate_keys = list(range(nplates)) # Keys for the output plates out_plate_keys = [key for key in in_plate_keys if key < len(plates_to) and plates_to[-key-1] != 1] # Keys for the dims dim_keys = list(range(nplates, nplates+ndim)) # Total input and output keys in_keys = len(arrays) * [in_plate_keys + dim_keys] out_keys = out_plate_keys + dim_keys # Compute the sum-product with correction einsum_args = misc.zipper_merge(arrays, in_keys) + [out_keys] y = r * np.einsum(*einsum_args) # Reshape the result and apply correction nplates_result = min(len(plates_to), len(arrays_plates)) if nplates_result == 0: plates_result = [] else: plates_result = [min(plates_to[ind], arrays_plates[ind]) for ind in range(-nplates_result, 0)] y = np.reshape(y, plates_result + list(dims)) return y from .deterministic import Deterministic def slicelen(s, length=None): if length is not None: s = slice(*(s.indices(length))) return max(0, misc.ceildiv(s.stop - s.start, s.step)) class Slice(Deterministic): """ Basic slicing for plates. Slicing occurs when index is a slice object (constructed by start:stop:step notation inside of brackets), an integer, or a tuple of slice objects and integers. Currently, accept slices, newaxis, ellipsis and integers. For instance, does not accept lists/tuples to pick multiple indices of the same axis. Ellipsis expand to the number of : objects needed to make a selection tuple of the same length as x.ndim. Only the first ellipsis is expanded, any others are interpreted as :. Similar to: http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#basic-slicing """ _parent_moments = (Moments(),) def __init__(self, X, slices, **kwargs): self._moments = X._moments # Force a list if not isinstance(slices, tuple): slices = [slices] else: slices = list(slices) # # Expand Ellipsis # # Compute the number of required axes and how Ellipsis is expanded num_axis = 0 ellipsis_index = None for (k, s) in enumerate(slices): if isinstance(s, int) or isinstance(s, slice): num_axis += 1 elif s is None: pass elif s is Ellipsis: # Index is an ellipsis, e.g., [...] if ellipsis_index is None: # Expand ... ellipsis_index = k else: # Interpret ... as : num_axis += 1 slices[k] = slice(None) else: raise TypeError("Invalid argument type.") if num_axis > len(X.plates): raise IndexError("Too many indices") # The number of plates that were not given explicit slicing (either # Ellipsis was used or the number of slices was smaller than the number # of plate axes) expand_len = len(X.plates) - num_axis if ellipsis_index is not None: # Replace Ellipsis with correct number of : k = ellipsis_index del slices[k] slices = slices[:k] + [slice(None)] * expand_len + slices[k:] else: # Add trailing : so that each plate has explicit slicing slices = slices + [slice(None)] * expand_len # # Preprocess indexing: # - integer indices to non-negative values # - slice start/stop values to non-negative # - slice start/stop values based on the size of the plate # # Index for parent plates j = 0 for (k, s) in enumerate(slices): if isinstance(s, int): # Index is an integer, e.g., [3] if s < 0: # Handle negative index s += X.plates[j] if s < 0 or s >= X.plates[j]: raise IndexError("Index out of range") # Store the preprocessed integer index slices[k] = s j += 1 elif isinstance(s, slice): # Index is a slice, e.g., [2:6] # Normalize the slice s = slice(*(s.indices(X.plates[j]))) if slicelen(s) <= 0: raise IndexError("Slicing leads to empty plates") slices[k] = s j += 1 self.slices = slices super().__init__(X, dims=X.dims, **kwargs) def _plates_to_parent(self, index): return self.parents[index].plates def _plates_from_parent(self, index): plates = list(self.parents[index].plates) # Compute the plates. Note that Ellipsis has already been preprocessed # to a proper number of : k = 0 for s in self.slices: # Then, each case separately: slice, newaxis, integer if isinstance(s, slice): # Slice, e.g., [2:5] N = slicelen(s) if N <= 0: raise IndexError("Slicing leads to empty plates") plates[k] = N k += 1 elif s is None: # [np.newaxis] plates = plates[:k] + [1] + plates[k:] k += 1 elif isinstance(s, int): # Integer, e.g., [3] del plates[k] return tuple(plates) @staticmethod def __reverse_indexing(slices, m_child, plates, dims): """ A helpful function for performing reverse indexing/slicing """ j = -1 # plate index for parent i = -1 # plate index for child child_slices = () parent_slices = () msg_plates = () # Compute plate axes in the message from children ndim = len(dims) if ndim > 0: m_plates = np.shape(m_child)[:-ndim] else: m_plates = np.shape(m_child) for s in reversed(slices): if isinstance(s, int): # Case: integer parent_slices = (s,) + parent_slices msg_plates = (plates[j],) + msg_plates j -= 1 elif s is None: # Case: newaxis if -i <= len(m_plates): child_slices = (0,) + child_slices i -= 1 else: # Case: slice if -i <= len(m_plates): child_slices = (slice(None),) + child_slices parent_slices = (s,) + parent_slices if ((-i > len(m_plates) or m_plates[i] == 1) and slicelen(s) == plates[j]): # Broadcasting can be applied. The message does not need # to be explicitly shaped to the full size msg_plates = (1,) + msg_plates else: # No broadcasting. Must explicitly form the full size # axis msg_plates = (plates[j],) + msg_plates j -= 1 i -= 1 # Set the elements of the message m_parent = np.zeros(msg_plates + dims) if np.ndim(m_parent) == 0 and np.ndim(m_child) == 0: m_parent = m_child elif np.ndim(m_parent) == 0: m_parent = m_child[child_slices] elif np.ndim(m_child) == 0: m_parent[parent_slices] = m_child else: m_parent[parent_slices] = m_child[child_slices] return m_parent def _compute_mask_to_parent(self, index, mask): """ Compute the mask to the parent node. """ if index != 0: raise ValueError("Invalid index") parent = self.parents[0] return self.__reverse_indexing(self.slices, self.mask, parent.plates, ()) def _compute_message_and_mask_to_parent(self, index, m, u): """ Compute the message to a parent node. """ if index != 0: raise ValueError("Invalid index") parent = self.parents[0] # Apply reverse indexing for the message arrays msg = [self.__reverse_indexing(self.slices, m_child, parent.plates, dims) for (m_child, dims) in zip(m, parent.dims)] # Apply reverse indexing for the mask mask = self._compute_mask_to_parent(0, self.mask) return (msg, mask) def _compute_moments(self, u): """ Get the moments with an added plate axis. """ # Process each moment for n in range(len(u)): # Compute the effective plates in the message/moment ndim = len(self.dims[n]) if ndim > 0: shape = np.shape(u[n])[:-ndim] else: shape = np.shape(u[n]) # Construct a list of slice objects u_slices = [] # Index for the shape j = -len(self.parents[0].plates) for (k, s) in enumerate(self.slices): if s is None: # [np.newaxis] if -j < len(shape): # Only add newaxis if there are some axes before # this. It does not make any difference if you added # leading unit axes u_slices.append(s) else: # slice or integer index if -j <= len(shape): # The moment has this axis, so it is not broadcasting it if shape[j] != 1: # Use the slice as it is u_slices.append(s) elif isinstance(s, slice): # Slice. # The moment is using broadcasting, just pick the # first element but use slice in order to keep the # axis u_slices.append(slice(0,1,1)) else: # Integer. # The moment is using broadcasting, just pick the # first element u_slices.append(0) j += 1 # Slice the message/moment u[n] = u[n][tuple(u_slices)] return u def AddPlateAxis(to_plate): if to_plate >= 0: raise Exception("Give negative value for axis index to_plate.") class _AddPlateAxis(Deterministic): def __init__(self, X, **kwargs): nonlocal to_plate N = len(X.plates) + 1 # Check the parameters if to_plate >= 0 or to_plate < -N: raise ValueError("Invalid plate position to add.") # Use positive indexing only ## if to_plate < 0: ## to_plate += N # Use negative indexing only if to_plate >= 0: to_plate -= N #self.to_plate = to_plate super().__init__(X, dims=X.dims, **kwargs) def _plates_to_parent(self, index): plates = list(self.plates) plates.pop(to_plate) return tuple(plates) #return self.plates[:to_plate] + self.plates[(to_plate+1):] def _plates_from_parent(self, index): plates = list(self.parents[index].plates) plates.insert(len(plates)-to_plate+1, 1) return tuple(plates) def _compute_mask_to_parent(self, index, mask): # Remove the added mask plate if abs(to_plate) <= np.ndim(mask): sh_mask = list(np.shape(mask)) sh_mask.pop(to_plate) mask = np.reshape(mask, sh_mask) return mask def _compute_message_and_mask_to_parent(self, index, m, *u_parents): """ Compute the message to a parent node. """ # Get the message from children #(m, mask) = self.message_from_children() # Remove the added message plate for i in range(len(m)): # Make sure the message has all the axes #diff = len(self.plates) + len(self.dims[i]) - np.ndim(m[i]) #m[i] = misc.add_leading_axes(m[i], diff) # Remove the axis if np.ndim(m[i]) >= abs(to_plate) + len(self.dims[i]): axis = to_plate - len(self.dims[i]) sh_m = list(np.shape(m[i])) sh_m.pop(axis) m[i] = np.reshape(m[i], sh_m) mask = self._compute_mask_to_parent(index, self.mask) return (m, mask) def _compute_moments(self, u): """ Get the moments with an added plate axis. """ # Get parents' moments #u = self.parents[0].message_to_child() # Move a plate axis u = list(u) for i in range(len(u)): # Make sure the moments have all the axes #diff = len(self.plates) + len(self.dims[i]) - np.ndim(u[i]) - 1 #u[i] = misc.add_leading_axes(u[i], diff) # The location of the new axis/plate: axis = np.ndim(u[i]) - abs(to_plate) - len(self.dims[i]) + 1 if axis > 0: # Add one axes to the correct position sh_u = list(np.shape(u[i])) sh_u.insert(axis, 1) u[i] = np.reshape(u[i], sh_u) return u return _AddPlateAxis class NodeConstantScalar(Node): @staticmethod def compute_fixed_u_and_f(x): """ Compute u(x) and f(x) for given x. """ return ([x], 0) def __init__(self, a, **kwargs): self.u = [a] super().__init__(self, plates=np.shape(a), dims=[()], **kwargs) def start_optimization(self): # FIXME: Set the plate sizes appropriately!! x0 = self.u[0] #self.gradient = np.zeros(np.shape(x0)) def transform(x): # E.g., for positive scalars you could have exp here. self.gradient = np.zeros(np.shape(x0)) self.u[0] = x def gradient(): # This would need to apply the gradient of the # transformation to the computed gradient return self.gradient return (x0, transform, gradient) def add_to_gradient(self, d): self.gradient += d def message_to_child(self, gradient=False): if gradient: return (self.u, [ [np.ones(np.shape(self.u[0])), #self.gradient] ]) self.add_to_gradient] ]) else: return self.u def stop_optimization(self): #raise Exception("Not implemented for " + str(self.__class__)) pass bayespy-0.4.0/bayespy/inference/vmp/nodes/poisson.py0000644000175000017500000000755212533543154024313 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2014 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Module for the Poisson distribution node. """ import numpy as np from scipy import special from .expfamily import ExponentialFamily from .expfamily import ExponentialFamilyDistribution from .node import Moments from .gamma import GammaMoments from bayespy.utils import misc class PoissonMoments(Moments): """ Class for the moments of Poisson variables """ def compute_fixed_moments(self, x): """ Compute the moments for a fixed value """ # Make sure the values are integers in valid range x = np.asanyarray(x) if not misc.isinteger(x): raise ValueError("Count not integer") # Now, the moments are just the counts return [x] def compute_dims_from_values(self, x): """ Return the shape of the moments for a fixed value. The realizations are scalars, thus the shape of the moment is (). """ return ( (), ) class PoissonDistribution(ExponentialFamilyDistribution): """ Class for the VMP formulas of Poisson variables. """ def compute_message_to_parent(self, parent, index, u, u_lambda): """ Compute the message to a parent node. """ if index == 0: m0 = -1 m1 = np.copy(u[0]) return [m0, m1] else: raise ValueError("Index out of bounds") def compute_phi_from_parents(self, u_lambda, mask=True): """ Compute the natural parameter vector given parent moments. """ l = u_lambda[0] logl = u_lambda[1] phi0 = logl return [phi0] def compute_moments_and_cgf(self, phi, mask=True): """ Compute the moments and :math:`g(\phi)`. """ u0 = np.exp(phi[0]) u = [u0] g = -u0 return (u, g) def compute_cgf_from_parents(self, u_lambda): """ Compute :math:`\mathrm{E}_{q(p)}[g(p)]` """ l = u_lambda[0] g = -l return g def compute_fixed_moments_and_f(self, x, mask=True): """ Compute the moments and :math:`f(x)` for a fixed value. """ # Check the validity of x x = np.asanyarray(x) if not misc.isinteger(x): raise ValueError("Values must be integers") if np.any(x < 0): raise ValueError("Values must be positive") # Compute moments u0 = np.copy(x) u = [u0] # Compute f(x) f = -special.gammaln(x+1) return (u, f) def random(self, *phi): """ Draw a random sample from the distribution. """ raise NotImplementedError() class Poisson(ExponentialFamily): """ Node for Poisson random variables. The node uses Poisson distribution: .. math:: p(x) = \mathrm{Poisson}(x|\lambda) where :math:`\lambda` is the rate parameter. Parameters ---------- l : gamma-like node or scalar or array :math:`\lambda`, rate parameter See also -------- Gamma, Exponential """ dims = ( (), ) _moments = PoissonMoments() _parent_moments = [GammaMoments()] _distribution = PoissonDistribution() def __init__(self, l, **kwargs): """ Create Poisson random variable node """ super().__init__(l, **kwargs) def __str__(self): """ Print the distribution using standard parameterization. """ l = self.u[0] return ("%s ~ Categorical(lambda)\n" " lambda =\n" "%s\n" % (self.name, l)) bayespy-0.4.0/bayespy/inference/vmp/vmp.py0000644000175000017500000005154612600472544022314 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2011-2015 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ import numpy as np import matplotlib.pyplot as plt import warnings import time import h5py import datetime import tempfile import scipy import logging from bayespy.utils import misc from bayespy.inference.vmp.nodes.node import Node class VB(): r""" Variational Bayesian (VB) inference engine Parameters ---------- nodes : nodes Nodes that form the model. Must include all at least all stochastic nodes of the model. tol : double, optional Convergence criterion. Tolerance for the relative change in the VB lower bound. autosave_filename : string, optional Filename for automatic saving autosave_iterations : int, optional Iteration interval between each automatic saving callback : callable, optional Function which is called after each update iteration step """ def __init__(self, *nodes, tol=1e-5, autosave_filename=None, autosave_iterations=0, use_logging=False, callback=None): for (ind, node) in enumerate(nodes): if not isinstance(node, Node): raise ValueError("Argument number %d is not a node" % (ind+1)) if use_logging: logger = logging.getLogger(__name__) self.print = logger.info else: # By default, don't use logging, just print stuff self.print = print # Remove duplicate nodes self.model = misc.unique(nodes) self.ignore_bound_checks = False self._figures = {} self.iter = 0 self.annealing_changed = False self.converged = False self.L = np.array(()) self.cputime = np.array(()) self.l = dict(zip(self.model, len(self.model)*[np.array([])])) self.autosave_iterations = autosave_iterations self.autosave_nodes = None if not autosave_filename: date = datetime.datetime.today().strftime('%Y%m%d%H%M%S') prefix = 'vb_autosave_%s_' % date tmpfile = tempfile.NamedTemporaryFile(prefix=prefix, suffix='.hdf5') self.autosave_filename = tmpfile.name self.filename = None else: self.autosave_filename = autosave_filename self.filename = autosave_filename # Check uniqueness of the node names names = [node.name for node in self.model] if len(names) != len(self.model): raise Exception("Use unique names for nodes.") self.callback = callback self.callback_output = None self.tol = tol def use_logging(self, use): if use_logging: logger = logging.getLogger(__name__) self.print = logger.info else: # By default, don't use logging, just print stuff self.print = print return def set_autosave(self, filename, iterations=None, nodes=None): self.autosave_filename = filename self.filename = filename self.autosave_nodes = nodes if iterations is not None: self.autosave_iterations = iterations def set_callback(self, callback): self.callback = callback def update(self, *nodes, repeat=1, plot=False, tol=None, verbose=True): # TODO/FIXME: # # If no nodes are given and thus everything is updated, the update order # should be from down to bottom. Or something similar.. # By default, update all nodes if len(nodes) == 0: nodes = self.model converged = False for i in range(repeat): t = time.clock() # Update nodes for node in nodes: X = self[node] if hasattr(X, 'update') and callable(X.update): X.update() if plot: self.plot(X) cputime = time.clock() - t if self._end_iteration_step(None, cputime, tol=tol, verbose=verbose): return def has_converged(self, tol=None): return self.converged def compute_lowerbound(self, ignore_masked=True): L = 0 for node in self.model: L += node.lower_bound_contribution(ignore_masked=ignore_masked) return L def compute_lowerbound_terms(self, *nodes): if len(nodes) == 0: nodes = self.model return {node: node.lower_bound_contribution() for node in nodes} def loglikelihood_lowerbound(self): L = 0 for node in self.model: lp = node.lower_bound_contribution() L += lp self.l[node][self.iter] = lp return L def plot_iteration_by_nodes(self, axes=None, diff=False): """ Plot the cost function per node during the iteration. Handy tool for debugging. """ if axes is None: axes = plt.gca() D = len(self.l) N = self.iter + 1 if diff: L = np.empty((N-1,D)) x = np.arange(N-1) + 2 else: L = np.empty((N,D)) x = np.arange(N) + 1 legends = [] for (d, node) in enumerate(self.l): if diff: L[:,d] = np.diff(self.l[node][:N]) else: L[:,d] = self.l[node][:N] legends += [node.name] axes.plot(x, L) axes.legend(legends, loc='lower right') axes.set_title('Lower bound contributions by nodes') axes.set_xlabel('Iteration') def get_iteration_by_nodes(self): return self.l def save(self, *nodes, filename=None): if len(nodes) == 0: nodes = self.model else: nodes = [self[node] for node in nodes if node is not None] if self.iter == 0: # Check HDF5 version. if h5py.version.hdf5_version_tuple < (1,8,7): warnings.warn("WARNING! Your HDF5 version is %s. HDF5 versions " "<1.8.7 are not able to save empty arrays, thus " "you may experience problems if you for instance " "try to save before running any iteration steps." % str(h5py.version.hdf5_version_tuple)) # By default, use the same file as for auto-saving if not filename: if self.autosave_filename: filename = self.autosave_filename else: raise Exception("Filename must be given.") # Open HDF5 file h5f = h5py.File(filename, 'w') try: # Write each node nodegroup = h5f.create_group('nodes') for node in nodes: if node.name == '': raise Exception("In order to save nodes, they must have " "(unique) names.") if hasattr(node, '_save') and callable(node._save): node._save(nodegroup.create_group(node.name)) # Write iteration statistics misc.write_to_hdf5(h5f, self.L, 'L') misc.write_to_hdf5(h5f, self.cputime, 'cputime') misc.write_to_hdf5(h5f, self.iter, 'iter') misc.write_to_hdf5(h5f, self.converged, 'converged') if self.callback_output is not None: misc.write_to_hdf5(h5f, self.callback_output, 'callback_output') boundgroup = h5f.create_group('boundterms') for node in nodes: misc.write_to_hdf5(boundgroup, self.l[node], node.name) finally: # Close file h5f.close() def load(self, *nodes, filename=None, nodes_only=False): # By default, use the same file as for auto-saving if not filename: if self.autosave_filename: filename = self.autosave_filename else: raise Exception("Filename must be given.") # Open HDF5 file h5f = h5py.File(filename, 'r') try: # Get nodes to load if len(nodes) == 0: nodes = self.model else: nodes = [self[node] for node in nodes if node is not None] # Read each node for node_id in nodes: node = self[node_id] if node.name == '': h5f.close() raise Exception("In order to load nodes, they must have " "(unique) names.") if hasattr(node, 'load') and callable(node.load): try: node._load(h5f['nodes'][node.name]) except KeyError: h5f.close() raise Exception("File does not contain variable %s" % node.name) # Read iteration statistics if not nodes_only: self.L = h5f['L'][...] self.cputime = h5f['cputime'][...] self.iter = h5f['iter'][...] self.converged = h5f['converged'][...] for node in nodes: self.l[node] = h5f['boundterms'][node.name][...] try: self.callback_output = h5f['callback_output'][...] except KeyError: pass finally: # Close file h5f.close() def __getitem__(self, name): if name in self.model: return name else: # Dictionary for mapping node names to nodes dictionary = {node.name: node for node in self.model} return dictionary[name] def plot(self, *nodes, **kwargs): """ Plot the distribution of the given nodes (or all nodes) """ if len(nodes) == 0: nodes = self.model for node in nodes: node = self[node] if node.has_plotter(): try: fignum = self._figures[node] except KeyError: fig = plt.figure() self._figures[node] = fig.number else: fig = plt.figure(num=fignum) fig.clf() node.plot(fig=fig, **kwargs) fig.canvas.draw() @property def ignore_bound_checks(self): return self.__ignore_bound_checks @ignore_bound_checks.setter def ignore_bound_checks(self, ignore): self.__ignore_bound_checks = ignore def get_gradients(self, *nodes, euclidian=False): """ Computes gradients (both Riemannian and normal) """ rg = [self[node].get_riemannian_gradient() for node in nodes] if euclidian: g = [self[node].get_gradient(rg_x) for (node, rg_x) in zip(nodes, rg)] return (rg, g) else: return rg def get_parameters(self, *nodes): """ Get parameters of the nodes """ return [self[node].get_parameters() for node in nodes] def set_parameters(self, x, *nodes): """ Set parameters of the nodes """ for (node, xi) in zip(nodes, x): self[node].set_parameters(xi) return def gradient_step(self, *nodes, scale=1.0): """ Update nodes by taking a gradient ascent step """ p = self.add(self.get_parameters(*nodes), self.get_gradients(*nodes), scale=scale) self.set_parameters(p, *nodes) return def dot(self, x1, x2): """ Computes dot products of given vectors (in parameter format) """ v = 0 # Loop over nodes for (y1, y2) in zip(x1, x2): # Loop over parameters for (z1, z2) in zip(y1, y2): v += np.dot(np.ravel(z1), np.ravel(z2)) return v def add(self, x1, x2, scale=1): """ Add two vectors (in parameter format) """ v = [] # Loop over nodes for (y1, y2) in zip(x1, x2): v.append([]) # Loop over parameters for (z1, z2) in zip(y1, y2): v[-1].append(z1 + scale*z2) return v def optimize(self, *nodes, maxiter=10, verbose=True, method='fletcher-reeves', riemannian=True, collapsed=None, tol=None): """ Optimize nodes using Riemannian conjugate gradient """ method = method.lower() if collapsed is None: collapsed = [] t = time.clock() # Current parameters p = self.get_parameters(*nodes) # Get gradients if riemannian and method == 'gradient': rg = self.get_gradients(*nodes, euclidian=False) else: (rg, g) = self.get_gradients(*nodes, euclidian=True) if riemannian: g1 = g g2 = rg else: g1 = g g2 = g if method == 'gradient': pass elif method == 'fletcher-reeves': dd_prev = self.dot(g1, g2) else: raise Exception("Unknown optimization method: %s" % (method)) # Take a gradient ascending step p_new = self.add(p, g2) self.set_parameters(p_new, *nodes) p = p_new # Update collapsed variables for node in collapsed: self[node].update() L = self.compute_lowerbound() s = g2 cputime = time.clock() - t self._end_iteration_step('OPT', cputime, tol=tol) for i in range(maxiter-1): t = time.clock() # Get gradients if riemannian and method == 'gradient': rg = self.get_gradients(*nodes, euclidian=False) else: (rg, g) = self.get_gradients(*nodes, euclidian=True) if riemannian: g1 = g g2 = rg else: g1 = g g2 = g if method == 'gradient': b = 0 elif method == 'fletcher-reeves': dd_curr = self.dot(g1, g2) if dd_prev == 0: b = 0 else: b = dd_curr / dd_prev dd_prev = dd_curr else: raise Exception("Unknown optimization method: %s" % (method)) if b: s = self.add(g2, s, scale=b) else: s = g2 success = False while not success: p_new = self.add(p, s) try: self.set_parameters(p_new, *nodes) except: self.print("CG update was unsuccessful, using gradient and resetting CG") s = g2 continue # Update collapsed variables for node in collapsed: self[node].update() L = self.compute_lowerbound() if L < self.L[self.iter-1] and not np.allclose(L, self.L[self.iter-1], rtol=1e-8): self.print("CG decreased lower bound to %e, using gradient and resetting CG" % L) s = g2 continue success = True p = p_new cputime = time.clock() - t if self._end_iteration_step('OPT', cputime, tol=tol): break def pattern_search(self, *nodes, collapsed=None, maxiter=3): """Perform simple pattern search :cite:`Honkela:2003`. Some of the variables can be collapsed. """ if collapsed is None: collapsed = [] t = time.clock() # Update all nodes for x in nodes: self[x].update() for x in collapsed: self[x].update() # Current parameter values p0 = self.get_parameters(*nodes) # Update optimized nodes for x in nodes: self[x].update() # New parameter values p1 = self.get_parameters(*nodes) # Search direction dp = self.add(p1, p0, scale=-1) # Cost function for pattern search def cost(alpha): p_new = self.add(p1, dp, scale=alpha) try: self.set_parameters(p_new, *nodes) except: return np.inf # Update collapsed nodes for x in collapsed: self[x].update() return -self.compute_lowerbound() # Optimize step length res = scipy.optimize.minimize_scalar(cost, bracket=[0, 3], options={'maxiter':maxiter}) # Set found parameter values p_new = self.add(p1, dp, scale=res.x) self.set_parameters(p_new, *nodes) # Update collapsed nodes for x in collapsed: self[x].update() cputime = time.clock() - t self._end_iteration_step('PS', cputime) def set_annealing(self, annealing): """ Set deterministic annealing from range (0, 1]. With 1, no annealing, standard updates. With smaller values, entropy has more weight and model probability equations less. With 0, one would obtain improper uniform distributions. """ for node in self.model: node.annealing = annealing self.annealing_changed = True self.converged = False return def _append_iterations(self, iters): """ Append some arrays for more iterations """ self.L = np.append(self.L, misc.nans(iters)) self.cputime = np.append(self.cputime, misc.nans(iters)) for (node, l) in self.l.items(): self.l[node] = np.append(l, misc.nans(iters)) return def _end_iteration_step(self, method, cputime, tol=None, verbose=True, bound_cpu_time=True): """ Do some routines after each iteration step """ if self.iter >= len(self.L): self._append_iterations(100) # Call the custom function provided by the user if callable(self.callback): z = self.callback() if z is not None: z = np.array(z)[...,np.newaxis] if self.callback_output is None: self.callback_output = z else: self.callback_output = np.concatenate((self.callback_output,z), axis=-1) t = time.clock() L = self.loglikelihood_lowerbound() if bound_cpu_time: cputime += time.clock() - t self.cputime[self.iter] = cputime self.L[self.iter] = L if verbose: if method: self.print("Iteration %d (%s): loglike=%e (%.3f seconds)" % (self.iter+1, method, L, cputime)) else: self.print("Iteration %d: loglike=%e (%.3f seconds)" % (self.iter+1, L, cputime)) # Check the progress of the iteration self.converged = False if not self.ignore_bound_checks and not self.annealing_changed and self.iter > 0: # Check for errors if self.L[self.iter-1] - L > 1e-6: L_diff = (self.L[self.iter-1] - L) warnings.warn("Lower bound decreased %e! Bug somewhere or " "numerical inaccuracy?" % L_diff) # Check for convergence L0 = self.L[self.iter-1] L1 = self.L[self.iter] if tol is None: tol = self.tol div = 0.5 * (abs(L0) + abs(L1)) if (L1 - L0) / div < tol: #if (L1 - L0) / div < tol or L1 - L0 <= 0: if verbose: self.print("Converged at iteration %d." % (self.iter+1)) self.converged = True # Auto-save, if requested if (self.autosave_iterations > 0 and np.mod(self.iter+1, self.autosave_iterations) == 0): if self.autosave_nodes is not None: self.save(*self.autosave_nodes, filename=self.autosave_filename) else: self.save(filename=self.autosave_filename) if verbose: self.print('Auto-saved to %s' % self.autosave_filename) self.annealing_changed = False self.iter += 1 return self.converged bayespy-0.4.0/bayespy/inference/__init__.py0000644000175000017500000000143712533543154022442 0ustar jluttinejluttine00000000000000################################################################################ # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under the MIT License. ################################################################################ """ Package for Bayesian inference engines Inference engines ----------------- .. autosummary:: :toctree: generated/ VB Parameter expansions -------------------- .. autosummary:: :toctree: generated/ vmp.transformations.RotationOptimizer vmp.transformations.RotateGaussian vmp.transformations.RotateGaussianARD vmp.transformations.RotateGaussianMarkovChain vmp.transformations.RotateSwitchingMarkovChain vmp.transformations.RotateVaryingMarkovChain vmp.transformations.RotateMultiple """ from .vmp.vmp import VB bayespy-0.4.0/MANIFEST.in0000664000175000017500000000041012174013072016435 0ustar jluttinejluttine00000000000000include README* include INSTALL* include LICENSE # Include header files recursive-include bayespy *.h # Add documentation to sdist-generated tarballs recursive-include doc/source *.rst recursive-include doc/source *.py include doc/Makefile #include doc/make.bat bayespy-0.4.0/INSTALL.rst0000644000175000017500000001353312535510603016552 0ustar jluttinejluttine00000000000000.. Copyright (C) 2011-2012,2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. Installation ============ BayesPy is a Python 3 package and it can be installed from PyPI or the latest development version from GitHub. The instructions below explain how to set up the system by installing required packages, how to install BayesPy and how to compile this documentation yourself. However, if these instructions contain errors or some relevant details are missing, please file a bug report at https://github.com/bayespy/bayespy/issues. Installing BayesPy ------------------ BayesPy can be installed easily by using Pip if the system has been properly set up. If you have problems with the following methods, see the following section for some help on installing the requirements. For instance, a bug in recent versions of h5py and pip may require you to install some of the requirements manually. For users +++++++++ First, you may want to set up a virtual environment. Using virtual environment is optional but recommended. To create and activate a new virtual environment, run (in the folder in which you want to create the environment): .. code-block:: console virtualenv -p python3 --system-site-packages ENV source ENV/bin/activate The latest release of BayesPy can be installed from PyPI simply as .. code-block:: console pip install bayespy If you want to install the latest development version of BayesPy, use GitHub instead: .. code-block:: console pip install https://github.com/bayespy/bayespy/archive/master.zip For developers ++++++++++++++ If you want to install the development version of BayesPy in such a way that you can easily edit the package, follow these instructions. Get the git repository: .. code-block:: console git clone https://github.com/bayespy/bayespy.git cd bayespy Create and activate a new virtual environment (optional but recommended): .. code-block:: console virtualenv -p python3 --system-site-packages ENV source ENV/bin/activate Install BayesPy in editable mode: .. code-block:: console pip install -e . Checking installation +++++++++++++++++++++ If you have problems installing BayesPy, read the next section for more details. It is recommended to run the unit tests in order to check that BayesPy is working properly. Thus, install Nose and run the unit tests: .. code-block:: console pip install nose nosetests bayespy Installing requirements ----------------------- BayesPy requires Python 3.3 (or later) and the following packages: * NumPy (>=1.8.0), * SciPy (>=0.13.0) * matplotlib (>=1.2) * h5py Ideally, Pip should install the necessary requirements and a manual installation of these dependencies is not required. However, there are several reasons why the installation of these dependencies needs to be done manually in some cases. Thus, this section tries to give some details on how to set up your system. A proper installation of the dependencies for Python 3 can be a bit tricky and you may refer to http://www.scipy.org/install.html for more detailed instructions about the SciPy stack. Detailed instructions on installing recent SciPy stack for various platforms is out of the scope of these instructions, but we provide some general guidance here. There are basically three ways to install the dependencies: 1. Install a Python distribution which includes the packages. For Windows, Mac and Linux, there are several Python distributions which include all the necessary packages: http://www.scipy.org/install.html#scientific-python-distributions. For instance, you may try `Anaconda `_ or `Enthought `_. 2. Install the packages using the system package manager. On Linux, the packages might be called something like ``python-scipy`` or ``scipy``. However, it is possible that these system packages are not recent enough for BayesPy. 3. Install the packages using Pip: .. code-block:: console pip install "distribute>=0.6.28" pip install "numpy>=1.8.0" "scipy>=0.13.0" "matplotlib>=1.2" h5py This also makes sure you have recent enough version of Distribute (required by Matplotlib). However, this installation method may require that the system has some libraries needed for compiling (e.g., C compiler, Python development files, BLAS/LAPACK). For instance, on Ubuntu (>= 12.10), you may install the required system libraries for each package as: .. code-block:: console sudo apt-get build-dep python3-numpy sudo apt-get build-dep python3-scipy sudo apt-get build-dep python3-matplotlib sudo apt-get build-dep python-h5py Then installation using Pip should work. Compiling documentation ----------------------- This documentation can be found at http://bayespy.org/ in HTML and PDF formats. The documentation source files are also readable as such in reStructuredText format in ``doc/source/`` directory. It is possible to compile the documentation into HTML or PDF yourself. In order to compile the documentation, Sphinx is required and a few extensions for it. Those can be installed as: .. code-block:: console pip install "sphinx>=1.2.3" sphinxcontrib-tikz sphinxcontrib-bayesnet sphinxcontrib-bibtex "numpydoc>=0.5" In order to visualize graphical models in HTML, you need to have ``ImageMagick`` or ``Netpbm`` installed. The documentation can be compiled to HTML and PDF by running the following commands in the ``doc`` directory: .. code-block:: console make html make latexpdf You can also run doctest to test code snippets in the documentation: .. code-block:: console make doctest or in the docstrings: .. code-block:: console nosetests --with-doctest --doctest-options="+ELLIPSIS" bayespy bayespy-0.4.0/doc/0000755000175000017500000000000012615647103015457 5ustar jluttinejluttine00000000000000bayespy-0.4.0/doc/source/0000755000175000017500000000000012615647103016757 5ustar jluttinejluttine00000000000000bayespy-0.4.0/doc/source/dev_guide/0000755000175000017500000000000012615647103020712 5ustar jluttinejluttine00000000000000bayespy-0.4.0/doc/source/dev_guide/engine.rst0000644000175000017500000000127512533543154022716 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. Implementing inference engines ============================== Currently, only variational Bayesian inference engine is implemented. This implementation is not very modular, that is, the inference engine is not well separated from the model construction. Thus, it is not straightforward to implement other inference engines at the moment. Improving the modularity of the inference engine and model construction is future work with high priority. In any case, BayesPy aims to be an efficient, simple and modular Bayesian package for variational inference at least. bayespy-0.4.0/doc/source/dev_guide/vmp/0000755000175000017500000000000012615647103021514 5ustar jluttinejluttine00000000000000bayespy-0.4.0/doc/source/dev_guide/vmp/vmp_gaussian.rst0000664000175000017500000000464112174013072024741 0ustar jluttinejluttine00000000000000Multivariate normal distribution -------------------------------- .. math:: \mathbf{x} &\sim \mathcal{N}(\boldsymbol{\mu}, \mathbf{\Lambda}), .. math:: \mathbf{x},\boldsymbol{\mu} \in \mathbb{R}^{D}, \quad \mathbf{\Lambda} \in \mathbb{R}^{D \times D}, \quad \mathbf{\Lambda} \text{ symmetric positive definite} .. math:: \log\mathcal{N}( \mathbf{x} | \boldsymbol{\mu}, \mathbf{\Lambda} ) &= - \frac{1}{2} \mathbf{x}^{\mathrm{T}} \mathbf{\Lambda} \mathbf{x} + \mathbf{x}^{\mathrm{T}} \mathbf{\Lambda} \boldsymbol{\mu} - \frac{1}{2} \boldsymbol{\mu}^{\mathrm{T}} \mathbf{\Lambda} \boldsymbol{\mu} + \frac{1}{2} \log |\mathbf{\Lambda}| - \frac{D}{2} \log (2\pi) .. math:: \mathbf{u} (\mathbf{x}) &= \left[ \begin{matrix} \mathbf{x} \\ \mathbf{xx}^{\mathrm{T}} \end{matrix} \right] \\ \boldsymbol{\phi} (\boldsymbol{\mu}, \mathbf{\Lambda}) &= \left[ \begin{matrix} \mathbf{\Lambda} \boldsymbol{\mu} \\ - \frac{1}{2} \mathbf{\Lambda} \end{matrix} \right] \\ \boldsymbol{\phi}_{\boldsymbol{\mu}} (\mathbf{x}, \mathbf{\Lambda}) &= \left[ \begin{matrix} \mathbf{\Lambda} \mathbf{x} \\ - \frac{1}{2} \mathbf{\Lambda} \end{matrix} \right] \\ \boldsymbol{\phi}_{\mathbf{\Lambda}} (\mathbf{x}, \boldsymbol{\mu}) &= \left[ \begin{matrix} - \frac{1}{2} \mathbf{xx}^{\mathrm{T}} + \frac{1}{2} \mathbf{x}\boldsymbol{\mu}^{\mathrm{T}} + \frac{1}{2} \boldsymbol{\mu}\mathbf{x}^{\mathrm{T}} - \frac{1}{2} \boldsymbol{\mu\mu}^{\mathrm{T}} \\ \frac{1}{2} \end{matrix} \right] \\ g (\boldsymbol{\mu}, \mathbf{\Lambda}) &= - \frac{1}{2} \operatorname{tr}(\boldsymbol{\mu\mu}^{\mathrm{T}} \mathbf{\Lambda} ) + \frac{1}{2} \log |\mathbf{\Lambda}| \\ g_{\boldsymbol{\phi}} (\boldsymbol{\phi}) &= \frac{1}{4} \boldsymbol{\phi}^{\mathrm{T}}_1 \boldsymbol{\phi}^{-1}_2 \boldsymbol{\phi}_1 + \frac{1}{2} \log | -2 \boldsymbol{\phi}_2 | \\ f(\mathbf{x}) &= - \frac{D}{2} \log(2\pi) \\ \overline{\mathbf{u}} (\boldsymbol{\phi}) &= \left[ \begin{matrix} - \frac{1}{2} \boldsymbol{\phi}^{-1}_2 \boldsymbol{\phi}_1 \\ \frac{1}{4} \boldsymbol{\phi}^{-1}_2 \boldsymbol{\phi}_1 \boldsymbol{\phi}^{\mathrm{T}}_1 \boldsymbol{\phi}^{-1}_2 - \frac{1}{2} \boldsymbol{\phi}^{-1}_2 \end{matrix} \right] bayespy-0.4.0/doc/source/dev_guide/vmp/vmp_wishart.rst0000664000175000017500000000360212174013072024604 0ustar jluttinejluttine00000000000000Wishart distribution -------------------- .. math:: \mathbf{\Lambda} &\sim \mathcal{W}(n, \mathbf{V}), .. math:: n > D-1, \quad \mathbf{\Lambda}, \mathbf{V} \in \mathbb{R}^{D \times D}, \quad \mathbf{\Lambda}, \mathbf{V} \text{ symmetric positive definite} .. math:: \log\mathcal{W}( \mathbf{\Lambda} | n, \mathbf{V} ) &= - \frac{1}{2} \operatorname{tr} (\mathbf{\Lambda V}) + \frac{n}{2} \log |\mathbf{\Lambda}| + \frac{n}{2} \log |\mathbf{V}| - \frac{D+1}{2} \log |\mathbf{\Lambda}| - \frac{nD}{2} \log 2 - \log \Gamma_D \left(\frac{n}{2}\right) .. math:: \mathbf{u} (\mathbf{\Lambda}) &= \left[ \begin{matrix} \mathbf{\Lambda} \\ \log |\mathbf{\Lambda}| \end{matrix} \right] \\ \boldsymbol{\phi} (n, \mathbf{V}) &= \left[ \begin{matrix} - \frac{1}{2} \mathbf{V} \\ \frac{1}{2} n \end{matrix} \right] \\ \boldsymbol{\phi}_{n} (\mathbf{\Lambda}, \mathbf{V}) &= \left[ \begin{matrix} \frac{1}{2}\log|\mathbf{\Lambda}| + \frac{1}{2}\log|\mathbf{V}| + \frac{D}{2} \log 2 \\ -1 \end{matrix} \right] \\ \boldsymbol{\phi}_{\mathbf{V}} (\mathbf{\Lambda}, n) &= \left[ \begin{matrix} - \frac{1}{2} \mathbf{\Lambda} \\ \frac{1}{2} n \end{matrix} \right] \\ g (n, \mathbf{V}) &= \frac{n}{2} \log|\mathbf{V}| - \frac{nD}{2}\log 2 - \log \Gamma_D \left(\frac{n}{2}\right) \\ g_{\boldsymbol{\phi}} (\boldsymbol{\phi}) &= \boldsymbol{\phi}_2 \log|-\boldsymbol{\phi}_1| - \log \Gamma_D (\boldsymbol{\phi}_2) \\ f(\mathbf{\Lambda}) &= - \frac{D+1}{2} \log|\mathbf{\Lambda}| \\ \overline{\mathbf{u}} (\boldsymbol{\phi}) &= \left[ \begin{matrix} - \boldsymbol{\phi}_2 \boldsymbol{\phi}^{-1}_1 \\ - \log|-\boldsymbol{\phi}_1| + \psi_D(\boldsymbol{\phi}_2) \end{matrix} \right] bayespy-0.4.0/doc/source/dev_guide/vmp/vmp_normal.rst0000664000175000017500000000007612174013072024415 0ustar jluttinejluttine00000000000000Univariate normal distribution ------------------------------ bayespy-0.4.0/doc/source/dev_guide/vmp/vmp_gamma.rst0000664000175000017500000000004612174013072024204 0ustar jluttinejluttine00000000000000Gamma distribution ------------------ bayespy-0.4.0/doc/source/dev_guide/vmp/vmp_gaussian_gamma.rst0000664000175000017500000000007112174013072026074 0ustar jluttinejluttine00000000000000 Gaussian-Gamma distribution --------------------------- bayespy-0.4.0/doc/source/dev_guide/vmp/vmp_mixture.rst0000664000175000017500000000632612174013072024626 0ustar jluttinejluttine00000000000000Mixture distribution -------------------- .. math:: \mathbf{x} &\sim \mathrm{Mix}_{\mathcal{D}} \left( \lambda, \left\{ \mathbf{\Theta}^{(n)}_1, \ldots, \mathbf{\Theta}^{(n)}_K \right\}^N_{n=1} \right) .. math:: \lambda \in \{1, \ldots, N\}, \quad \mathcal{D} \text{ is an exp.fam. distribution}, \quad \mathbf{\Theta}^{(n)}_k \text{ are parameters of } \mathcal{D} .. math:: \log\mathrm{Mix}_{\mathcal{D}} \left( \mathbf{x} \left| \lambda, \left\{ \mathbf{\Theta}^{(n)}_1, \ldots, \mathbf{\Theta}^{(n)}_K \right\}^N_{n=1} \right. \right) &= \sum^N_{n=1} [\lambda=n] \mathbf{u}_{\mathcal{D}}(\mathbf{x})^{\mathrm{T}} \boldsymbol{\phi}_{\mathcal{D}} \left( \mathbf{\Theta}^{(n)}_1, \ldots, \mathbf{\Theta}^{(n)}_K \right) \\ & \quad + \sum^N_{n=1} [\lambda=n] g_{\mathcal{D}} \left( \mathbf{\Theta}^{(n)}_1, \ldots, \mathbf{\Theta}^{(n)}_K \right) + f_{\mathcal{D}} (\mathbf{x}) .. math:: \mathbf{u} (\mathbf{x}) &= \mathbf{u}_{\mathcal{D}} (\mathbf{x}) \\ \boldsymbol{\phi} \left( \lambda, \left\{ \mathbf{\Theta}^{(n)}_1, \ldots, \mathbf{\Theta}^{(n)}_K \right\}^N_{n=1} \right) &= \sum^N_{n=1} [\lambda=n] \boldsymbol{\phi}_{\mathcal{D}} \left( \mathbf{\Theta}^{(n)}_1, \ldots, \mathbf{\Theta}^{(n)}_K \right) % \\ % \boldsymbol{\phi}_{\lambda} \left( \mathbf{x}, \left\{ \mathbf{\Theta}^{(n)}_1, \ldots, \mathbf{\Theta}^{(n)}_K \right\}^N_{n=1} \right) &= \left[\begin{matrix} \mathbf{u}_{\mathcal{D}} (\mathbf{x})^{\mathrm{T}} \boldsymbol{\phi}_{\mathcal{D}} \left( \mathbf{\Theta}^{(1)}_1, \ldots, \mathbf{\Theta}^{(1)}_K \right) + g_{\mathcal{D}} \left( \mathbf{\Theta}^{(1)}_1, \ldots, \mathbf{\Theta}^{(1)}_K \right) \\ \vdots \\ \mathbf{u}_{\mathcal{D}} (\mathbf{x})^{\mathrm{T}} \boldsymbol{\phi}_{\mathcal{D}} \left( \mathbf{\Theta}^{(N)}_1, \ldots, \mathbf{\Theta}^{(N)}_K \right) + g_{\mathcal{D}} \left( \mathbf{\Theta}^{(N)}_1, \ldots, \mathbf{\Theta}^{(N)}_K \right) \end{matrix}\right] % \\ % \boldsymbol{\phi}_{\mathbf{\Theta}^{(m)}_l} \left( \mathbf{x}, \lambda, \left\{ \mathbf{\Theta}^{(n)}_1, \ldots, \mathbf{\Theta}^{(n)}_K \right\}^N_{n=1} \setminus \left\{ \mathbf{\Theta}^{(m)}_l \right\} \right) &= [\lambda=m] \boldsymbol{\phi}_{\mathcal{D}\rightarrow\mathbf{\Theta}_l} \left( \mathbf{x}, \left\{ \mathbf{\Theta}^{(m)}_k \right\}_{k\neq l} \right) % \\ % g \left( \lambda, \left\{ \mathbf{\Theta}^{(n)}_1, \ldots, \mathbf{\Theta}^{(n)}_K \right\}^N_{n=1} \right) &= \sum^N_{n=1} [\lambda=n] g_{\mathcal{D}} \left( \mathbf{\Theta}^{(n)}_1, \ldots, \mathbf{\Theta}^{(n)}_K \right) \\ g (\boldsymbol{\phi}) &= g_{\mathcal{D}} (\boldsymbol{\phi}) \\ f(\mathbf{x}) &= f_{\mathcal{D}} (\mathbf{x}) \\ \overline{\mathbf{u}} (\boldsymbol{\phi}) &= \overline{\mathbf{u}}_{\mathcal{D}} (\boldsymbol{\phi}) bayespy-0.4.0/doc/source/dev_guide/vmp/vmp_normal_gamma.rst0000664000175000017500000000006412174013072025554 0ustar jluttinejluttine00000000000000Normal-Gamma distribution ------------------------- bayespy-0.4.0/doc/source/dev_guide/vmp/vmp_gaussian_wishart.rst0000664000175000017500000000007512174013072026477 0ustar jluttinejluttine00000000000000Gaussian-Wishart distribution ----------------------------- bayespy-0.4.0/doc/source/dev_guide/writingnodes.rst0000644000175000017500000003627612533543154024176 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. Implementing nodes ================== The main goal of BayesPy is to provide a package which enables easy and flexible construction of simple and complex models with efficient inference. However, users may sometimes be unable to construct their models because the built-in nodes do not implement some specific features. Thus, one may need to implement new nodes in order to construct the model. BayesPy aims to make the implementation of new nodes both simple and fast. Probably, a large complex model can be constructed almost completely with the built-in nodes and the user needs to implement only a few nodes. Moments ------- .. currentmodule:: bayespy.inference.vmp.nodes.node In order to implement nodes, it is important to understand the messaging framework of the nodes. A node is a unit of calculation which communicates to its parent and child nodes using messages. These messages have types that need to match between nodes, that is, the child node needs to understand the messages its parents are sending and vice versa. Thus, a node defines which message type it requires from each of its parents, and only nodes that have that type of output message (i.e., the message to a child node) are valid parent nodes for that node. The message type is defined by the moments of the parent node. The moments are a collection of expectations: :math:`\{ \langle f_1(X) \rangle, \ldots, \langle f_N(X) \rangle \}`. The functions :math:`f_1, \ldots, f_N` (and the number of the functions) define the message type and they are the sufficient statistic as discussed in the previous section. Different message types are represented by :class:`Moments` class hierarchy. For instance, :class:`GaussianMoments` represents a message type with parent moments :math:`\{\langle \mathbf{x} \rangle, \langle \mathbf{xx}^T \rangle \}` and :class:`WishartMoments` a message type with parent moments :math:`\{\langle \mathbf{\Lambda} \rangle, \langle \log |\mathbf{\Lambda}| \rangle\}`. .. currentmodule:: bayespy.nodes Let us give an example: :class:`Gaussian` node outputs :class:`GaussianMoments` messages and :class:`Wishart` node outputs :class:`WishartMoments` messages. :class:`Gaussian` node requires that it receives :class:`GaussianMoments` messages from the mean parent node and :class:`WishartMoments` messages from the precision parent node. Thus, :class:`Gaussian` and :class:`Wishart` are valid node classes as the mean and precision parent nodes of :class:`Gaussian` node. Note that several nodes may have the same output message type and some message types can be transformed to other message types using deterministic converter nodes. For instance, :class:`Gaussian` and :class:`GaussianARD` nodes both output :class:`GaussianMoments` messages, deterministic :class:`SumMultiply` also outputs :class:`GaussianMoments` messages, and deterministic converter :class:`_MarkovChainToGaussian` converts :class:`GaussianMarkovChainMoments` to :class:`GaussianMoments`. .. currentmodule:: bayespy.inference.vmp.nodes.node Each node specifies the message type requirements of its parents by :attr:`Node._parent_moments` attribute which is a list of :class:`Moments` sub-class instances. These moments objects have a few purpose when creating the node: 1) check that parents are sending proper messages; 2) if parents use different message type, try to add a converter which converts the messages to the correct type if possible; 3) if given parents are not nodes but numeric arrays, convert them to constant nodes with correct output message type. When implementing a new node, it is not always necessary to implement a new moments class. If another node has the same sufficient statistic vector, thus the same moments, that moments class can be used. Otherwise, one must implement a simple moments class which has the following methods: * :func:`Moments.compute_fixed_moments` Computes the moments for a known value. This is used to compute the moments of constant numeric arrays and wrap them into constant nodes. * :func:`Moments.compute_dims_from_values` Given a known value of the variable, return the shape of the variable dimensions in the moments. This is used to solve the shape of the moments array for constant nodes. Distributions ------------- .. currentmodule:: bayespy.inference.vmp.stochastic In order to implement a stochastic exponential family node, one must first write down the log probability density function of the node and derive the terms discussed in section :ref:`sec-vmp-terms`. These terms are implemented and collected as a class which is a subclass of :class:`Distribution`. The main reason to implement these methods in another class instead of the node class itself is that these methods can be used without creating a node, for instance, in :class:`Mixture` class. .. currentmodule:: bayespy.inference.vmp.nodes.expfamily For exponential family distributions, the distribution class is a subclass of :class:`ExponentialFamilyDistribution`, and the relation between the terms in section :ref:`sec-vmp-terms` and the methods is as follows: * :func:`ExponentialFamilyDistribution.compute_phi_from_parents` Computes the expectation of the natural parameters :math:`\langle \boldsymbol{\phi} \rangle` in the prior distribution given the moments of the parents. * :func:`ExponentialFamilyDistribution.compute_cgf_from_parents` Computes the expectation of the negative log normalizer :math:`\langle g \rangle` of the prior distribution given the moments of the parents. * :func:`ExponentialFamilyDistribution.compute_moments_and_cgf` Computes the moments :math:`\langle \mathbf{u} \rangle` and the negative log normalizer :math:`\tilde{g}` of the posterior distribution given the natural parameters :math:`\tilde{\boldsymbol{\phi}}`. * :func:`ExponentialFamilyDistribution.compute_message_to_parent` Computes the message :math:`\langle \boldsymbol{\phi}_{\mathbf{x}\rightarrow\boldsymbol{\theta}} \rangle` from the node :math:`\mathbf{x}` to its parent node :math:`\boldsymbol{\theta}` given the moments of the node and the other parents. * :func:`ExponentialFamilyDistribution.compute_fixed_moments_and_f` Computes :math:`\mathbf{u}(\mathbf{x})` and :math:`f(\mathbf{x})` for given observed value :math:`\mathbf{x}`. Without this method, variables from this distribution cannot be observed. For each stochastic exponential family node, one must write a distribution class which implements these methods. After that, the node class is basically a simple wrapper and it also stores the moments and the natural parameters of the current posterior approximation. Note that the distribution classes do not store node-specific information, they are more like static collections of methods. However, sometimes the implementations depend on some information, such as the dimensionality of the variable, and this information must be provided, if needed, when constructing the distribution object. In addition to the methods listed above, it is necessary to implement a few more methods in some cases. This happens when the plates of the parent do not map to the plates directly as discussed in section :ref:`sec-irregular-plates`. Then, one must write methods that implement this plate mapping and apply the same mapping to the mask array: * :func:`ExponentialFamilyDistribution.plates_from_parent` Given the plates of the parent, return the resulting plates of the child. * :func:`ExponentialFamilyDistribution.plates_to_parent` Given the plates of the child, return the plates of the parent that would have resulted them. * :func:`ExponentialFamilyDistribution.compute_mask_to_parent` Given the mask array of the child, apply the plate mapping. It is important to understand when one must implement these methods, because the default implementations in the base class will lead to errors or weird results. Stochastic exponential family nodes ----------------------------------- After implementing the distribution class, the next task is to implement the node class. First, we need to explain a few important attributes before we can explain how to implement a node class. Stochastic exponential family nodes have two attributes that store the state of the posterior distribution: * ``phi`` The natural parameter vector :math:`\tilde{\boldsymbol{\phi}}` of the posterior approximation. * ``u`` The moments :math:`\langle \mathbf{u} \rangle` of the posterior approximation. Instead of storing these two variables as vectors (as in the mathematical formulas), they are stored as lists of arrays with convenient shapes. For instance, :class:`Gaussian` node stores the moments as a list consisting of a vector :math:`\langle \mathbf{x} \rangle` and a matrix :math:`\langle \mathbf{xx}^T \rangle` instead of reshaping and concatenating these into a single vector. The same applies for the natural parameters ``phi`` because it has the same shape as ``u``. The shapes of the arrays in the lists ``u`` and ``phi`` consist of the shape caused by the plates and the shape caused by the variable itself. For instance, the moments of :class:`Gaussian` node have shape ``(D,)`` and ``(D, D)``, where ``D`` is the dimensionality of the Gaussian vector. In addition, if the node has plates, they are added to these shapes. Thus, for instance, if the :class:`Gaussian` node has plates ``(3, 7)`` and ``D`` is 5, the shape of ``u[0]`` and ``phi[0]`` would be ``(3, 7, 5)`` and the shape of ``u[1]`` and ``phi[1]`` would be ``(3, 7, 5, 5)``. This shape information is stored in the following attributes: * ``plates`` : a tuple The plates of the node. In our example, ``(3, 7)``. * ``dims`` : a list of tuples The shape of each of the moments arrays (or natural parameter arrays) without plates. In our example, ``[ (5,), (5, 5) ]``. Finally, three attributes define VMP for the node: * ``_moments`` : :class:`Moments` sub-class instance An object defining the moments of the node. * ``_parent_moments`` : list of :class:`Moments` sub-class instances A list defining the moments requirements for each parent. * ``_distribution`` : :class:`Distribution` sub-class instance An object implementing the VMP formulas. Basically, a node class is a collection of the above attributes. When a node is created, these attributes are defined. The base class for exponential family nodes, :class:`ExponentialFamily`, provides a simple default constructor which does not need to be overwritten if ``dims``, ``_moments``, ``_parent_moments`` and ``_distribution`` can be provided as static class attributes. For instance, :class:`Gamma` node defines these attributes statically. However, usually at least one of these attributes cannot be defined statically in the class. In that case, one must implement a class method which overloads :func:`ExponentialFamily._constructor`. The purpose of this method is to define all the attributes given the parent nodes. These are defined using a class method instead of ``__init__`` method in order to be able to use the class constructors statically, for instance, in :class:`Mixture` class. This construction allows users to create mixtures of any exponential family distribution with simple syntax. The parents of a node must be converted so that they have a correct message type, because the user may have provided numeric arrays or nodes with incorrect message type. Numeric arrays should be converted to constant nodes with correct message type. Incorrect message type nodes should be converted to correct message type nodes if possible. Thus, the constructor should use ``Node._ensure_moments`` method to make sure the parent is a node with correct message type. Instead of calling this method for each parent node in the constructor, one can use ``ensureparents`` decorator to do this automatically. However, the decorator requires that ``_parent_moments`` attribute has already been defined statically. If this is not possible, the parent nodes must be converted manually in the constructor, because one should never assume that the parent nodes given to the constructor are nodes with correct message type or even nodes at all. Deterministic nodes ------------------- Deterministic nodes are nodes that do not correspond to any probability distribution but rather a deterministic function. It does not have any moments or natural parameters to store. A deterministic node is implemented as a subclass of :class:`Deterministic` base class. The new node class must implement the following methods: * :func:`Deterministic._compute_moments` Computes the moments given the moments of the parents. * :func:`Deterministic._compute_message_to_parent` Computes the message to a parent node given the message from the children and the moments of the other parents. In some cases, one may want to implement :func:`Deterministic._compute_message_and_mask_to_parent` or :func:`Deterministic._message_to_parent` instead in order to gain more control over efficient computation. Similarly as in :class:`Distribution` class, if the node handles plates irregularly, it is important to implement the following methods: * :func:`Deterministic._plates_from_parent` Given the plates of the parent, return the resulting plates of the child. * :func:`Deterministic._plates_to_parent` Given the plates of the child, return the plates of the parent that would have resulted them. * :func:`Deterministic._compute_mask_to_parent` Given the mask array, convert it to a plate mask of the parent. Converter nodes +++++++++++++++ Sometimes a node has incorrect message type but the message can be converted into a correct type. For instance, :class:`GaussianMarkovChain` has :class:`GaussianMarkovChainMoments` message type, which means moments :math:`\{ \langle \mathbf{x}_n \rangle, \langle \mathbf{x}_n \mathbf{x}_n^T \rangle, \langle \mathbf{x}_n \mathbf{x}_{n-1}^T \rangle \}^N_{n=1}`. These moments can be converted to :class:`GaussianMoments` by ignoring the third element and considering the time axis as a plate axis. Thus, if a node requires :class:`GaussianMoments` message from its parent, :class:`GaussianMarkovChain` is a valid parent if its messages are modified properly. This conversion is implemented in :class:`_MarkovChainToGaussian` converter class. Converter nodes are simple deterministic nodes that have one parent node and they convert the messages to another message type. For the user, it is not convenient if the exact message type has to be known and an explicit converter node needs to be created. Thus, the conversions are done automatically and the user will be unaware of them. In order to enable this automatization, when writing a converter node, one should register the converter to the moments class using :func:`Moments.add_converter`. For instance, a class ``X`` which converts moments ``A`` to moments ``B`` is registered as ``A.add_conveter(B, X)``. After that, :func:`Node._ensure_moments` and :func:`Node._convert` methods are used to perform the conversions automatically. The conversion can consist of several consecutive converter nodes, and the least number of conversions is used. bayespy-0.4.0/doc/source/dev_guide/dev_guide.rst0000644000175000017500000000071312533543154023400 0ustar jluttinejluttine00000000000000.. Copyright (C) 2011,2012 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. Developer guide *************** This chapter provides basic information for developers about contributing, the theoretical background and the core structure. It is assumed that the reader has read and is familiar with :ref:`sec-user-guide`. .. toctree:: :maxdepth: 2 workflow vmp engine writingnodes bayespy-0.4.0/doc/source/dev_guide/vmp.rst0000644000175000017500000002503112533543154022247 0ustar jluttinejluttine00000000000000.. Copyright (C) 2012 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. Variational message passing =========================== This section briefly describes the variational message passing (VMP) framework, which is currently the only implemented inference engine in BayesPy. The variational Bayesian (VB) inference engine in BayesPy assumes that the posterior approximation factorizes with respect to nodes and plates. VMP is based on updating one node at a time (the plates in one node can be updated simultaneously) and iteratively updating all nodes in turns until convergence. Standard update equation ------------------------ The general update equation for the factorized approximation of node :math:`\boldsymbol{\theta}` is the following: .. math:: :label: vmp_general_update \log q(\boldsymbol{\theta}) &= \langle \log p\left( \boldsymbol{\theta} | \mathrm{pa}(\boldsymbol{\theta}) \right) \rangle + \sum_{\mathbf{x} \in \mathrm{ch}(\boldsymbol{\theta})} \langle \log p(\mathbf{x}|\mathrm{pa}(\mathbf{x})) \rangle + \mathrm{const}, where :math:`\mathrm{pa}(\boldsymbol{\theta})` and :math:`\mathrm{ch}(\boldsymbol{\theta})` are the set of parents and children of :math:`\boldsymbol{\theta}`, respectively. Thus, the posterior approximation of a node is updated by taking a sum of the expectations of all log densities in which the node variable appears. The expectations are over the approximate distribution of all other variables than :math:`\boldsymbol{\theta}`. Actually, not all the variables are needed, because the non-constant part depends only on the Markov blanket of :math:`\boldsymbol{\theta}`. This leads to a local optimization scheme, which uses messages from neighbouring nodes. The messages are simple for conjugate exponential family models. An exponential family distribution has the following log probability density function: .. math:: :label: likelihood \log p(\mathbf{x}|\mathbf{\Theta}) &= \mathbf{u}_{\mathbf{x}}(\mathbf{x})^{\mathrm{T}} \boldsymbol{\phi}_{\mathbf{x}}(\mathbf{\Theta}) + g_{\mathbf{x}}(\mathbf{\Theta}) + f_{\mathbf{x}}(\mathbf{x}), where :math:`\mathbf{\Theta}=\{\boldsymbol{\theta}_j\}` is the set of parents, :math:`\mathbf{u}` is the sufficient statistic vector, :math:`\boldsymbol{\phi}` is the natural parameter vector, :math:`g` is the negative log normalizer, and :math:`f` is the log base function. Note that the log density is linear with respect to the terms that are functions of :math:`\mathbf{x}`: :math:`\mathbf{u}` and :math:`f`. If a parent has a conjugate prior, :eq:`likelihood` is also linear with respect to the parent's sufficient statistic vector. Thus, :eq:`likelihood` can be re-organized with respect to a parent :math:`\boldsymbol{\theta}_j` as .. math:: \log p(\mathbf{x}|\mathbf{\Theta}) &= \mathbf{u}_{\boldsymbol{\theta}_j}(\boldsymbol{\theta}_j)^{\mathrm{T}} \boldsymbol{\phi}_{\mathbf{x}\rightarrow\boldsymbol{\theta}_j} (\mathbf{x}, \{\boldsymbol{\theta}_k\}_{k\neq j}) + \mathrm{const}, where :math:`\mathbf{u}_{\boldsymbol{\theta}_j}` is the sufficient statistic vector of :math:`\boldsymbol{\theta}_j` and the constant part is constant with respect to :math:`\boldsymbol{\theta}_j`. Thus, the update equation :eq:`vmp_general_update` for :math:`\boldsymbol{\theta}_j` can be written as .. math:: \log q(\boldsymbol{\theta}_j) &= \mathbf{u}_{\boldsymbol{\theta}_j}(\boldsymbol{\theta}_j)^{\mathrm{T}} \langle \boldsymbol{\phi}_{\boldsymbol{\theta}_j} \rangle + f_{\boldsymbol{\theta}_j}(\boldsymbol{\theta}_j) + \mathbf{u}_{\boldsymbol{\theta}_j}(\boldsymbol{\theta}_j)^{\mathrm{T}} \sum_{\mathbf{x} \in \mathrm{ch}(\boldsymbol{\theta}_j)} \langle \boldsymbol{\phi}_{\mathbf{x}\rightarrow\boldsymbol{\theta}_j} \rangle + \mathrm{const}, \\ &= \mathbf{u}_{\boldsymbol{\theta}_j}(\boldsymbol{\theta}_j)^{\mathrm{T}} \left( \langle \boldsymbol{\phi}_{\boldsymbol{\theta}_j} \rangle + \sum_{\mathbf{x} \in \mathrm{ch}(\boldsymbol{\theta}_j)} \langle \boldsymbol{\phi}_{\mathbf{x}\rightarrow\boldsymbol{\theta}_j} \rangle \right) + f_{\boldsymbol{\theta}_j}(\boldsymbol{\theta}_j) + \mathrm{const}, where the summation is over all the child nodes of :math:`\boldsymbol{\theta}_j`. Because of the conjugacy, :math:`\langle\boldsymbol{\phi}_{\boldsymbol{\theta}_j}\rangle` depends (multi)linearly on the parents' sufficient statistic vector. Similarly, :math:`\langle \boldsymbol{\phi}_{\mathbf{x}\rightarrow\boldsymbol{\theta}_j} \rangle` depends (multi)linearly on the expectations of the children's and co-parents' sufficient statistics. This gives the following update equation for the natural parameter vector of the posterior approximation :math:`q(\boldsymbol{\phi}_j)`: .. math:: :label: update_phi \tilde{\boldsymbol{\phi}}_j &= \langle \boldsymbol{\phi}_{\boldsymbol{\theta}_j} \rangle + \sum_{\mathbf{x} \in \mathrm{ch}(\boldsymbol{\theta}_j)} \langle \boldsymbol{\phi}_{\mathbf{x}\rightarrow\boldsymbol{\theta}_j} \rangle. Variational messages -------------------- The update equation :eq:`update_phi` leads to a message passing scheme: the term :math:`\langle \boldsymbol{\phi}_{\boldsymbol{\theta}_j} \rangle` is a function of the parents' sufficient statistic vector and the term :math:`\langle \boldsymbol{\phi}_{\mathbf{x}\rightarrow\boldsymbol{\theta}_j} \rangle` can be interpreted as a message from the child node :math:`\mathbf{x}`. Thus, the message from the child node :math:`\mathbf{x}` to the parent node :math:`\boldsymbol{\theta}` is .. math:: \mathbf{m}_{\mathbf{x}\rightarrow\boldsymbol{\theta}} &\equiv \langle \boldsymbol{\phi}_{\mathbf{x}\rightarrow\boldsymbol{\theta}} \rangle, which can be computed as a function of the sufficient statistic vector of the co-parent nodes of :math:`\boldsymbol{\theta}` and the sufficient statistic vector of the child node :math:`\mathbf{x}`. The message from the parent node :math:`\boldsymbol{\theta}` to the child node :math:`\mathbf{x}` is simply the expectation of the sufficient statistic vector: .. math:: \mathbf{m}_{\mathbf{\boldsymbol{\theta}}\rightarrow\mathbf{x}} &\equiv \langle \mathbf{u}_{\boldsymbol{\theta}} \rangle. In order to compute the expectation of the sufficient statistic vector we need to write :math:`q(\boldsymbol{\theta})` as .. math:: \log q(\boldsymbol{\theta}) &= \mathbf{u}(\boldsymbol{\theta})^{\mathrm{T}} \tilde{\boldsymbol{\phi}} + \tilde{g}(\tilde{\boldsymbol{\phi}}) + f(\boldsymbol{\theta}), where :math:`\tilde{\boldsymbol{\phi}}` is the natural parameter vector of :math:`q(\boldsymbol{\theta})`. Now, the expectation of the sufficient statistic vector is defined as .. math:: :label: moments \langle \mathbf{u}_{\boldsymbol{\theta}} \rangle &= - \frac{\partial \tilde{g}}{\partial \tilde{\boldsymbol{\phi}}_{\boldsymbol{\theta}}} (\tilde{\boldsymbol{\phi}}_{\boldsymbol{\theta}}). We call this expectation of the sufficient statistic vector as the moments vector. Lower bound ----------- Computing the VB lower bound is not necessary in order to find the posterior approximation, although it is extremely useful in monitoring convergence and possible bugs. The VB lower bound can be written as .. math:: \mathcal{L} = \langle \log p(\mathbf{Y}, \mathbf{X}) \rangle - \langle \log q(\mathbf{X}) \rangle, where :math:`\mathbf{Y}` is the set of all observed variables and :math:`\mathbf{X}` is the set of all latent variables. It can also be written as .. math:: \mathcal{L} = \sum_{\mathbf{y} \in \mathbf{Y}} \langle \log p(\mathbf{y} | \mathrm{pa}(\mathbf{y})) \rangle + \sum_{\mathbf{x} \in \mathbf{X}} \left[ \langle \log p(\mathbf{x} | \mathrm{pa}(\mathbf{x})) \rangle - \langle \log q(\mathbf{x}) \right], which shows that observed and latent variables contribute differently to the lower bound. These contributions have simple forms for exponential family nodes. Observed exponential family nodes contribute to the lower bound as follows: .. math:: \langle \log p(\mathbf{y}|\mathrm{pa}(\mathbf{y})) \rangle &= \mathbf{u}(\mathbf{y})^T \langle \boldsymbol{\phi} \rangle + \langle g \rangle + f(\mathbf{x}), where :math:`\mathbf{y}` is the observed data. On the other hand, latent exponential family nodes contribute to the lower bound as follows: .. math:: \langle \log p(\mathbf{x}|\boldsymbol{\theta}) \rangle - \langle \log q(\mathbf{x}) \rangle &= \langle \mathbf{u} \rangle^T (\langle \boldsymbol{\phi} \rangle - \tilde{\boldsymbol{\phi}} ) + \langle g \rangle - \tilde{g}. If a node is partially observed and partially unobserved, these formulas are applied plate-wise appropriately. .. _sec-vmp-terms: Terms ----- To summarize, implementing VMP requires one to write for each stochastic exponential family node: :math:`\langle \boldsymbol{\phi} \rangle` : the expectation of the prior natural parameter vector Computed as a function of the messages from parents. :math:`\tilde{\boldsymbol{\phi}}` : natural parameter vector of the posterior approximation Computed as a sum of :math:`\langle \boldsymbol{\phi} \rangle` and the messages from children. :math:`\langle \mathbf{u} \rangle` : the posterior moments vector Computed as a function of :math:`\tilde{\boldsymbol{\phi}}` as defined in :eq:`moments`. :math:`\mathbf{u}(\mathbf{x})` : the moments vector for given data Computed as a function of of the observed data :math:`\mathbf{x}`. :math:`\langle g \rangle` : the expectation of the negative log normalizer of the prior Computed as a function of parent moments. :math:`\tilde{g}` : the negative log normalizer of the posterior approximation Computed as a function of :math:`\tilde{\boldsymbol{\phi}}`. :math:`f(\mathbf{x})` : the log base measure for given data Computed as a function of the observed data :math:`\mathbf{x}`. :math:`\langle \boldsymbol{\phi}_{\mathbf{x}\rightarrow\boldsymbol{\theta}} \rangle` : the message to parent :math:`\boldsymbol{\theta}` Computed as a function of the moments of this node and the other parents. Deterministic nodes require only the following terms: :math:`\langle \mathbf{u} \rangle` : the posterior moments vector Computed as a function of the messages from the parents. :math:`\mathbf{m}` : the message to a parent Computed as a function of the messages from the other parents and all children. bayespy-0.4.0/doc/source/dev_guide/workflow.rst0000644000175000017500000000444112615643655023331 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. Workflow ======== The main forum for BayesPy development is `GitHub `_. Bugs and other issues can be reported at https://github.com/bayespy/bayespy/issues. Contributions to the code and documentation are welcome and should be given as pull requests at https://github.com/bayespy/bayespy/pulls. In order to create pull requests, it is recommended to fork the git repository, make local changes and submit these changes as a pull request. The style guide for writing docstrings follows the style guide of NumPy, available at https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt. Detailed instructions on development workflow can be read from NumPy guide, available at http://docs.scipy.org/doc/numpy/dev/gitwash/development_workflow.html. BayesPy uses the following acronyms to start the commit message: * API: an (incompatible) API change * BLD: change related to building numpy * BUG: bug fix * DEMO: modification in demo code * DEP: deprecate something, or remove a deprecated object * DEV: development tool or utility * DOC: documentation * ENH: enhancement * MAINT: maintenance commit (refactoring, typos, etc.) * REV: revert an earlier commit * STY: style fix (whitespace, PEP8) * TST: addition or modification of tests * REL: related to releasing Since version 0.3.7, we have started following `Vincent Driessen's branching model `_ in how git is used. Making releases --------------- * Commit any current changes to git. * Start a release branch: ``git flow release start x.y.z`` * Edit version number in setup.py and commit. * Add changes to CHANGELOG.rst and commit. * Publish the release branch: ``git flow release publish x.y.z`` * Finish the release: ``git flow release finish x.y.z``. Write the following commit message: ``REL: Version x.y.z``. * Push to GitHub: ``git push && git push --tags`` * Publish in PyPI: ``python setup.py release_pypi`` * Update the documentation web page: ``cd doc && make gh-pages`` * Publish in mloss.org. * Announcements to bayespy@googlegroups.com, scipy-user@scipy.org and numpy-discussion@scipy.org. bayespy-0.4.0/doc/source/dev_guide/advanced.rst0000644000175000017500000001407112533543154023214 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014-2015 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. .. testsetup:: import numpy as np np.random.seed(1) # This is the PCA model from the previous sections from bayespy.nodes import GaussianARD, Gamma, Dot D = 3 X = GaussianARD(0, 1, shape=(D,), plates=(1,100), name='X') alpha = Gamma(1e-3, 1e-3, plates=(D,), name='alpha') C = GaussianARD(0, alpha, shape=(D,), plates=(10,1), name='C') F = Dot(C, X) tau = Gamma(1e-3, 1e-3, name='tau') Y = GaussianARD(F, tau, name='Y') c = np.random.randn(10, 2) x = np.random.randn(2, 100) data = np.dot(c, x) + 0.1*np.random.randn(10, 100) Y.observe(data) from bayespy.inference import VB import bayespy.plot as bpplt Q = None Advanced topics =============== The VB lower bound and its gradients ------------------------------------ The VB lower bound: .. math:: \mathcal{L} &= \underbrace{\langle \log p(X,Z) \rangle}_{\equiv \mathcal{L}_p} - \underbrace{\langle \log q(Z) \rangle}_{\equiv \mathcal{L}_q} The child nodes pass the gradient to the parent node so the parent node can optimize its parameters. In general, :math:`\mathcal{L}_p` can be almost arbitrarily complex function of :math:`Z`: .. math:: \mathcal{L}_p = \langle \log p(X,Z) \rangle. The gradient is .. math:: \frac{\partial}{\partial \xi} \mathcal{L}_p &= \frac{\partial}{\partial \xi} \langle \log p(X,Z) \rangle \\ &= \langle \log p(X,Z) \frac{\partial}{\partial \xi} \log q(Z) \rangle which can be computed, for instance, by sampling from :math:`q(Z)`. Note that :math:`\xi` can represent, for instance, the expectation parameters :math:`\bar{u}` of :math:`q(Z)` in order to obtain the Riemannian gradient for an exponential family :math:`q(Z)`. Often, :math:`\mathcal{L}_p` has a simpler form (or it can be further lower bounded by a simpler form). If :math:`\mathcal{L}_p` can be written as a function of :math:`\bar{u}` as .. math:: \mathcal{L}_p = \bar{u}^T \psi + \mathrm{const}, the gradient with respect to the moments is .. math:: \frac{\partial}{\partial \bar{u}} \mathcal{L}_p = \psi. It can be that :math:`\psi` can be computed exactly by using moments of other nodes or it needs to be approximated by using sampling from the distribution of other nodes. To summarize, the gradient message can be a numerical gradient, an approximate stochastic gradient (by sampling other nodes) or a function which can be used to compute an approximate stochastic gradient by sampling the node itself (and possibly other nodes). Riemannian gradient ------------------- In principle, the VB lower bound can be maximized with respect to any parameterization of the approximate distribution. However, normal gradient can perform badly, because it doesn't take into account the geometry of the space of probability distributions. This can be fixed by using Riemannian (i.e., natural) gradient. In general, the Riemannian gradient is defined as .. math:: \tilde{\nabla}_\xi \mathcal{L} = G^{-1} \nabla_\xi \mathcal{L} where .. math:: [G]_{ij} = \left\langle \frac{\partial \log q(Z)}{\partial \xi_i} \frac{\partial \log q(Z)}{\partial \xi_j} \right\rangle = - \left\langle \frac{\partial^2 \log q(Z)}{\partial \xi_i \partial \xi_j} \right\rangle. For exponential family distributions, the Riemannian gradient with respect to the natural parameters :math:`\phi` can be computed easily by taking the gradient with respect to the moments :math:`\bar{u}`: .. math:: \tilde{\nabla}_\phi = G^{-1} \nabla_\phi \mathcal{L} = \nabla_{\bar{u}} \mathcal{L}. Note that :math:`G` depends only on the approximate distribution :math:`q(Z)`. Thus, the model itself does not need to be in the exponential family but only the approximation, in order to use this property. The Riemannian gradient of :math:`\mathcal{L}_q` for exponential family distributions :math:`q(Z)` is .. math:: \tilde{\nabla}_\phi \mathcal{L}_q = \nabla_{\bar{u}} \mathcal{L}_q = \nabla_{\bar{u}} [ \bar{u}^T \phi + \langle f(Z) \rangle + g(\phi) ] = \phi. Thus, the Riemannian gradient is .. math:: \tilde{\nabla}_\phi \mathcal{L} = \nabla_{\bar{u}} \mathcal{L}_p - \phi. .. todo:: Should f(Z) be taken into account? It cancels out if prior and q are in the same family. But if they are not, it doesn't cancel out. Does it affect the gradient? Nonlinear conjugate gradient methods :cite:`Hensman:2012`: * Fletcher-Reeves: .. math:: \beta_n = \frac { \langle \tilde{g}_n, \tilde{g}_n \rangle_n } { \langle \tilde{g}_{n-1}, \tilde{g}_{n-1} \rangle_{n-1} } = \frac { \langle g_n, \tilde{g}_n \rangle } { \langle g_{n-1}, \tilde{g}_{n-1} \rangle } * Polak-Ribiere: .. math:: \beta_n = \frac { \langle \tilde{g}_n, \tilde{g}_n - \tilde{g}_{n-1} \rangle_n } { \langle \tilde{g}_{n-1}, \tilde{g}_{n-1} \rangle_{n-1} } = \frac { \langle g_n, \tilde{g}_n - \tilde{g}_{n-1} \rangle } { \langle g_{n-1}, \tilde{g}_{n-1} \rangle } * Hestenes-Stiefel: .. math:: \beta_n = - \frac { \langle \tilde{g}_n, \tilde{g}_n - \tilde{g}_{n-1} \rangle_n } { \langle \tilde{g}_{n-1}, \tilde{g}_{n-1} \rangle_{n-1} } = - \frac { \langle g_n, \tilde{g}_n - \tilde{g}_{n-1} \rangle } { \langle g_{n-1}, \tilde{g}_{n-1} \rangle } where :math:`\langle \rangle_i` denotes the inner product in the Riemannian geometry, :math:`\langle \rangle` denotes the inner product in the Euclidean space, :math:`\tilde{g}` denotes the Riemannian gradient and :math:`g` denotes the gradient, and the following property has been used: .. math:: \langle \tilde{g}_n, \tilde{x} \rangle_n = \tilde{g}_n^T G_n \tilde{x} = g^T G^{-1}_n G_n \tilde{x} = g^T \tilde{x} = \langle g, \tilde{x} \rangle TODO ---- * simulated annealing * Riemannian (conjugate) gradient * black box variational inference * stochastic variational inference * pattern search * fast inference * parameter expansion bayespy-0.4.0/doc/source/dev_api/0000755000175000017500000000000012615647103020366 5ustar jluttinejluttine00000000000000bayespy-0.4.0/doc/source/dev_api/distributions.rst0000644000175000017500000000204312533543154024021 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. Distributions ============= .. currentmodule:: bayespy.inference.vmp.nodes .. autosummary:: :toctree: generated/ stochastic.Distribution expfamily.ExponentialFamilyDistribution gaussian.GaussianDistribution gaussian.GaussianARDDistribution gaussian.GaussianGammaISODistribution gaussian.GaussianGammaARDDistribution gaussian.GaussianWishartDistribution gaussian_markov_chain.GaussianMarkovChainDistribution gaussian_markov_chain.SwitchingGaussianMarkovChainDistribution gaussian_markov_chain.VaryingGaussianMarkovChainDistribution gamma.GammaDistribution wishart.WishartDistribution beta.BetaDistribution dirichlet.DirichletDistribution bernoulli.BernoulliDistribution binomial.BinomialDistribution categorical.CategoricalDistribution categorical_markov_chain.CategoricalMarkovChainDistribution multinomial.MultinomialDistribution poisson.PoissonDistribution bayespy-0.4.0/doc/source/dev_api/utils.rst0000644000175000017500000000043712533543154022264 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. Utility functions ================= .. currentmodule:: bayespy.utils .. autosummary:: :toctree: generated/ linalg random optimize misc bayespy-0.4.0/doc/source/dev_api/dev_api.rst0000644000175000017500000000054312533543154022531 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. Developer API ============= This chapter contains API specifications which are relevant to BayesPy developers and contributors. .. toctree:: :maxdepth: 1 nodes moments distributions utils bayespy-0.4.0/doc/source/dev_api/nodes.rst0000644000175000017500000000151212533543154022227 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. Developer nodes =============== .. currentmodule:: bayespy.inference.vmp.nodes The following base classes are useful if writing new nodes: .. autosummary:: :toctree: generated/ node.Node stochastic.Stochastic expfamily.ExponentialFamily deterministic.Deterministic The following nodes are examples of special nodes that remain hidden for the user although they are often implicitly used: .. autosummary:: :toctree: generated/ constant.Constant gaussian.GaussianToGaussianGammaISO gaussian.GaussianGammaISOToGaussianGammaARD gaussian.GaussianGammaARDToGaussianWishart gaussian.WrapToGaussianGammaISO gaussian.WrapToGaussianGammaARD gaussian.WrapToGaussianWishart bayespy-0.4.0/doc/source/dev_api/moments.rst0000644000175000017500000000136012533543154022602 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. Moments ======= .. currentmodule:: bayespy.inference.vmp.nodes .. autosummary:: :toctree: generated/ node.Moments gaussian.GaussianMoments gaussian_markov_chain.GaussianMarkovChainMoments gaussian.GaussianGammaISOMoments gaussian.GaussianGammaARDMoments gaussian.GaussianWishartMoments gamma.GammaMoments wishart.WishartMoments beta.BetaMoments dirichlet.DirichletMoments bernoulli.BernoulliMoments binomial.BinomialMoments categorical.CategoricalMoments categorical_markov_chain.CategoricalMarkovChainMoments multinomial.MultinomialMoments poisson.PoissonMoments bayespy-0.4.0/doc/source/references.rst0000644000175000017500000000033312533543154021631 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. .. rubric:: Bibliography .. bibliography:: references.bib :all: :style: plain bayespy-0.4.0/doc/source/index.rst0000644000175000017500000000070112533543154020616 0ustar jluttinejluttine00000000000000.. Copyright (C) 2011,2012 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. BayesPy -- Bayesian Python ========================== .. toctree:: :maxdepth: 2 intro user_guide/user_guide examples/examples dev_guide/dev_guide user_api/user_api dev_api/dev_api references * :doc:`Bibliography ` * :ref:`genindex` * :ref:`modindex` * :ref:`search` bayespy-0.4.0/doc/source/_templates/0000755000175000017500000000000012615647103021114 5ustar jluttinejluttine00000000000000bayespy-0.4.0/doc/source/_templates/autosummary/0000755000175000017500000000000012615647103023502 5ustar jluttinejluttine00000000000000bayespy-0.4.0/doc/source/_templates/autosummary/class.rst0000664000175000017500000000110512356455721025345 0ustar jluttinejluttine00000000000000{{ fullname }} {{ underline }} .. currentmodule:: {{ module }} .. autoclass:: {{ objname }} {% block methods %} .. automethod:: __init__ {% if methods %} .. rubric:: Methods .. autosummary:: :toctree: generated/ {% for item in methods %} ~{{ name }}.{{ item }} {%- endfor %} {% endif %} {% endblock %} {% block attributes %} {% if attributes %} .. rubric:: Attributes .. autosummary:: :toctree: generated/ {% for item in attributes %} ~{{ name }}.{{ item }} {%- endfor %} {% endif %} {% endblock %} bayespy-0.4.0/doc/source/_templates/autosummary/module.rst0000664000175000017500000000131412356455721025527 0ustar jluttinejluttine00000000000000{{ fullname }} {{ underline }} .. automodule:: {{ fullname }} {% block functions %} {% if functions %} .. rubric:: Functions .. autosummary:: :toctree: generated/ {% for item in functions %} {{ item }} {%- endfor %} {% endif %} {% endblock %} {% block classes %} {% if classes %} .. rubric:: Classes .. autosummary:: :toctree: generated/ {% for item in classes %} {{ item }} {%- endfor %} {% endif %} {% endblock %} {% block exceptions %} {% if exceptions %} .. rubric:: Exceptions .. autosummary:: :toctree: generated/ {% for item in exceptions %} {{ item }} {%- endfor %} {% endif %} {% endblock %} bayespy-0.4.0/doc/source/_templates/autosummary/short_module.rst0000664000175000017500000000007712356455721026753 0ustar jluttinejluttine00000000000000{{ fullname }} {{ underline }} .. automodule:: {{ fullname }} bayespy-0.4.0/doc/source/conf.py0000644000175000017500000002452112533543154020262 0ustar jluttinejluttine00000000000000# -*- coding: utf-8 -*- # # BayesPy documentation build configuration file, created by # sphinx-quickstart on Mon Aug 27 12:22:11 2012. # # This file is execfile()d with the current directory set to its containing dir. # # Note that not all possible configuration values are present in this # autogenerated file. # # All configuration values have a default; values that are commented out # serve to show the default. import sys, os # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. #sys.path.insert(0, os.path.abspath('.')) # Import some information from the setup.py script. sys.path.insert(0, os.path.abspath('../..')) import setup as setupfile # -- General configuration ----------------------------------------------------- # If your documentation needs a minimal Sphinx version, state it here. #needs_sphinx = '1.0' # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = [ 'sphinx.ext.autodoc', 'sphinx.ext.mathjax', 'sphinx.ext.todo', 'sphinx.ext.viewcode', 'sphinx.ext.doctest', 'numpydoc', 'matplotlib.sphinxext.plot_directive', 'sphinx.ext.autosummary', 'sphinxcontrib.tikz', 'sphinxcontrib.bayesnet', 'sphinxcontrib.bibtex', ] numpydoc_show_class_members = False # Include TODOs in the documentation? todo_include_todos = True # Add a path to binary files that are necessary for readthedocs.org to # build HTML with tikz extension if os.environ.get('READTHEDOCS', None) == 'True': os.environ["PATH"] += os.pathsep + os.path.abspath('bin') os.environ["LD_LIBRARY_PATH"] = os.path.abspath('bin') # Generate autosummary stub pages automatically # Or manually: sphinx-autogen -o source/generated source/*.rst #autosummary_generate = False import glob autosummary_generate = glob.glob("*.rst") + glob.glob("*/*.rst") + glob.glob("*/*/*.rst") + glob.glob("*/*/*/*.rst") # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # The suffix of source filenames. source_suffix = '.rst' # The encoding of source files. #source_encoding = 'utf-8-sig' # The master toctree document. master_doc = 'index' # General information about the project. project = setupfile.NAME copyright = u'2011-2015, Jaakko Luttinen, MIT' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The short X.Y version. version = setupfile.VERSION # The full version, including alpha/beta/rc tags. release = setupfile.VERSION # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. #language = None # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: #today = '' # Else, today_fmt is used as the format for a strftime call. #today_fmt = '%B %d, %Y' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. exclude_patterns = [] # The reST default role (used for this markup: `text`) to use for all documents. #default_role = None # If true, '()' will be appended to :func: etc. cross-reference text. #add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). #add_module_names = True # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. #show_authors = False # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' # A list of ignored prefixes for module index sorting. #modindex_common_prefix = [] # -- Options for HTML output --------------------------------------------------- # Sphinx-TikZ extension tikz_latex_preamble = """ \usepackage{amsmath} """ # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. html_theme = 'sphinxdoc' #html_theme = 'nature' #html_theme = 'default' # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. html_theme_options = { "sidebarwidth": 300 } # Add any paths that contain custom themes here, relative to this directory. #html_theme_path = [] # The name for this set of Sphinx documents. If None, it defaults to # " v documentation". html_title = "BayesPy v%s Documentation" % (version) # A shorter title for the navigation bar. Default is the same as html_title. #html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. #html_logo = None # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. #html_favicon = None # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['_static'] # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, # using the given strftime format. #html_last_updated_fmt = '%b %d, %Y' # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. #html_use_smartypants = True # Custom sidebar templates, maps document names to template names. #html_sidebars = {} # Additional templates that should be rendered to pages, maps page names to # template names. #html_additional_pages = {} # If false, no module index is generated. #html_domain_indices = True # If false, no index is generated. #html_use_index = True # If true, the index is split into individual pages for each letter. #html_split_index = False # If true, links to the reST sources are added to the pages. #html_show_sourcelink = True # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. #html_show_sphinx = True # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. #html_show_copyright = True # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. #html_use_opensearch = '' # This is the file name suffix for HTML files (e.g. ".xhtml"). #html_file_suffix = None # Output file base name for HTML help builder. htmlhelp_basename = 'BayesPydoc' # -- Options for LaTeX output -------------------------------------------------- latex_elements = { # The paper size ('letterpaper' or 'a4paper'). #'papersize': 'letterpaper', # The font size ('10pt', '11pt' or '12pt'). #'pointsize': '10pt', # Additional stuff for the LaTeX preamble. 'preamble': r''' \usepackage{tikz} \usepackage{amssymb} \usepackage{amsmath} \usetikzlibrary{shapes} \usetikzlibrary{fit} \usetikzlibrary{chains} \usetikzlibrary{arrows} ''', # Do not use [T1]{fontenc} because it does not work on libre systems 'fontenc': '' } #latex_additional_files = ['images/bayesnet.sty',] # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, author, documentclass [howto/manual]). latex_documents = [ ('index', 'BayesPy.tex', u'BayesPy Documentation', u'Jaakko Luttinen', 'manual'), ] # The name of an image file (relative to this directory) to place at the top of # the title page. #latex_logo = None # For "manual" documents, if this is true, then toplevel headings are parts, # not chapters. #latex_use_parts = False # If true, show page references after internal links. #latex_show_pagerefs = False # If true, show URL addresses after external links. #latex_show_urls = False # Documents to append as an appendix to all manuals. #latex_appendices = [] # If false, no module index is generated. #latex_domain_indices = True # -- Options for manual page output -------------------------------------------- # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ ('index', 'bayespy', u'BayesPy Documentation', [u'Jaakko Luttinen'], 1) ] # If true, show URL addresses after external links. #man_show_urls = False # -- Options for Texinfo output ------------------------------------------------ # Grouping the document tree into Texinfo files. List of tuples # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ ('index', 'BayesPy', u'BayesPy Documentation', u'Jaakko Luttinen', 'BayesPy', 'One line description of project.', 'Miscellaneous'), ] # Documents to append as an appendix to all manuals. #texinfo_appendices = [] # If false, no module index is generated. #texinfo_domain_indices = True # How to display URL addresses: 'footnote', 'no', or 'inline'. #texinfo_show_urls = 'footnote' # -- Options for Epub output --------------------------------------------------- # Bibliographic Dublin Core info. epub_title = u'BayesPy' epub_author = u'Jaakko Luttinen' epub_publisher = u'Jaakko Luttinen' epub_copyright = u'2012, Jaakko Luttinen' # The language of the text. It defaults to the language option # or en if the language is not set. #epub_language = '' # The scheme of the identifier. Typical schemes are ISBN or URL. #epub_scheme = '' # The unique identifier of the text. This can be a ISBN number # or the project homepage. #epub_identifier = '' # A unique identification for the text. #epub_uid = '' # A tuple containing the cover image and cover page html template filenames. #epub_cover = () # HTML files that should be inserted before the pages created by sphinx. # The format is a list of tuples containing the path and title. #epub_pre_files = [] # HTML files shat should be inserted after the pages created by sphinx. # The format is a list of tuples containing the path and title. #epub_post_files = [] # A list of files that should not be packed into the epub file. #epub_exclude_files = [] # The depth of the table of contents in toc.ncx. #epub_tocdepth = 3 # Allow duplicate toc entries. #epub_tocdup = True bayespy-0.4.0/doc/source/intro.rst0000644000175000017500000000043312533543154020644 0ustar jluttinejluttine00000000000000.. Copyright (C) 2011,2012 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. Introduction ============ .. include:: ../../README.rst :start-line: 2 Version history --------------- .. include:: ../../CHANGELOG.rst bayespy-0.4.0/doc/source/nodes.rst0000644000175000017500000000101612533543154020617 0ustar jluttinejluttine00000000000000.. Copyright (C) 2011,2012 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. Nodes ===== Stochastic nodes ---------------- Normal ++++++ Gaussian ++++++++ Gamma +++++ Wishart +++++++ Bernoulli +++++++++ Categorical +++++++++++ Binomial ++++++++ Multinomial +++++++++++ Beta ++++ Dirichlet +++++++++ Mixture +++++++ Constant ++++++++ Deterministic nodes ------------------- Dot product (of Gaussian variables) +++++++++++++++++++++++++++++++++++ bayespy-0.4.0/doc/source/user_api/0000755000175000017500000000000012615647103020566 5ustar jluttinejluttine00000000000000bayespy-0.4.0/doc/source/user_api/user_api.rst0000644000175000017500000000050312533543154023125 0ustar jluttinejluttine00000000000000.. Copyright (C) 2011-2013 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. .. _sec-user-api: User API ======== .. autosummary:: :toctree: generated/ :template: autosummary/short_module.rst bayespy.nodes bayespy.inference bayespy.plot bayespy-0.4.0/doc/source/demos.rst0000644000175000017500000000032512533543154020620 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. Examples ******** .. toctree:: _notebooks/mog.rst _notebooks/pca.rst bayespy-0.4.0/doc/source/examples/0000755000175000017500000000000012615647103020575 5ustar jluttinejluttine00000000000000bayespy-0.4.0/doc/source/examples/gmm.rst0000644000175000017500000001752712533543154022123 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. .. testsetup:: import numpy numpy.random.seed(1) Gaussian mixture model ====================== This example demonstrates the use of Gaussian mixture model for flexible density estimation, clustering or classification. Data ---- First, let us generate some artificial data for the analysis. The data are two-dimensional vectors from one of the four different Gaussian distributions: >>> import numpy as np >>> y0 = np.random.multivariate_normal([0, 0], [[2, 0], [0, 0.1]], size=50) >>> y1 = np.random.multivariate_normal([0, 0], [[0.1, 0], [0, 2]], size=50) >>> y2 = np.random.multivariate_normal([2, 2], [[2, -1.5], [-1.5, 2]], size=50) >>> y3 = np.random.multivariate_normal([-2, -2], [[0.5, 0], [0, 0.5]], size=50) >>> y = np.vstack([y0, y1, y2, y3]) Thus, there are 200 data vectors in total. The data looks as follows: >>> import bayespy.plot as bpplt >>> bpplt.pyplot.plot(y[:,0], y[:,1], 'rx') [] .. plot:: import numpy as np np.random.seed(1) y0 = np.random.multivariate_normal([0, 0], [[1, 0], [0, 0.02]], size=50) y1 = np.random.multivariate_normal([0, 0], [[0.02, 0], [0, 1]], size=50) y2 = np.random.multivariate_normal([2, 2], [[1, -0.9], [-0.9, 1]], size=50) y3 = np.random.multivariate_normal([-2, -2], [[0.1, 0], [0, 0.1]], size=50) y = np.vstack([y0, y1, y2, y3]) import bayespy.plot as bpplt bpplt.pyplot.plot(y[:,0], y[:,1], 'rx') bpplt.pyplot.show() Model ----- For clarity, let us denote the number of the data vectors with ``N`` >>> N = 200 and the dimensionality of the data vectors with ``D``: >>> D = 2 We will use a "large enough" number of Gaussian clusters in our model: >>> K = 10 Cluster assignments ``Z`` and the prior for the cluster assignment probabilities ``alpha``: >>> from bayespy.nodes import Dirichlet, Categorical >>> alpha = Dirichlet(1e-5*np.ones(K), ... name='alpha') >>> Z = Categorical(alpha, ... plates=(N,), ... name='z') The mean vectors and the precision matrices of the clusters: >>> from bayespy.nodes import Gaussian, Wishart >>> mu = Gaussian(np.zeros(D), 1e-5*np.identity(D), ... plates=(K,), ... name='mu') >>> Lambda = Wishart(D, 1e-5*np.identity(D), ... plates=(K,), ... name='Lambda') If either the mean or precision should be shared between clusters, then that node should not have plates, that is, ``plates=()``. The data vectors are from a Gaussian mixture with cluster assignments ``Z`` and Gaussian component parameters ``mu`` and ``Lambda``: >>> from bayespy.nodes import Mixture >>> Y = Mixture(Z, Gaussian, mu, Lambda, ... name='Y') >>> Z.initialize_from_random() >>> from bayespy.inference import VB >>> Q = VB(Y, mu, Lambda, Z, alpha) Inference --------- Before running the inference algorithm, we provide the data: >>> Y.observe(y) Then, run VB iteration until convergence: >>> Q.update(repeat=1000) Iteration 1: loglike=-1.402345e+03 (... seconds) ... Iteration 61: loglike=-8.888464e+02 (... seconds) Converged at iteration 61. The algorithm converges very quickly. Note that the default update order of the nodes was such that ``mu`` and ``Lambda`` were updated before ``Z``, which is what we wanted because ``Z`` was initialized randomly. Results ------- .. currentmodule:: bayespy.plot For two-dimensional Gaussian mixtures, the mixture components can be plotted using :func:`gaussian_mixture_2d`: >>> bpplt.gaussian_mixture_2d(Y, alpha=alpha, scale=2) .. plot:: import numpy numpy.random.seed(1) import numpy as np y0 = np.random.multivariate_normal([0, 0], [[1, 0], [0, 0.02]], size=50) y1 = np.random.multivariate_normal([0, 0], [[0.02, 0], [0, 1]], size=50) y2 = np.random.multivariate_normal([2, 2], [[1, -0.9], [-0.9, 1]], size=50) y3 = np.random.multivariate_normal([-2, -2], [[0.1, 0], [0, 0.1]], size=50) y = np.vstack([y0, y1, y2, y3]) import bayespy.plot as bpplt bpplt.pyplot.plot(y[:,0], y[:,1], 'rx') N = 200 D = 2 K = 10 from bayespy.nodes import Dirichlet, Categorical alpha = Dirichlet(1e-5*np.ones(K), name='alpha') Z = Categorical(alpha, plates=(N,), name='z') from bayespy.nodes import Gaussian, Wishart mu = Gaussian(np.zeros(D), 1e-5*np.identity(D), plates=(K,), name='mu') Lambda = Wishart(D, 1e-5*np.identity(D), plates=(K,), name='Lambda') from bayespy.nodes import Mixture Y = Mixture(Z, Gaussian, mu, Lambda, name='Y') Z.initialize_from_random() from bayespy.inference import VB Q = VB(Y, mu, Lambda, Z, alpha) Y.observe(y) Q.update(repeat=1000) bpplt.gaussian_mixture_2d(Y, alpha=alpha, scale=2) bpplt.pyplot.show() The function is called with ``scale=2`` which means that each ellipse shows two standard deviations. From the ten cluster components, the model uses effectively the correct number of clusters (4). These clusters capture the true density accurately. In addition to clustering and density estimation, this model could also be used for classification by setting the known class assignments as observed. Advanced next steps ------------------- Joint node for mean and precision +++++++++++++++++++++++++++++++++ .. currentmodule:: bayespy.nodes The next step for improving the results could be to use :class:`GaussianWishart` node for modelling the mean vectors ``mu`` and precision matrices ``Lambda`` jointly without factorization. This should improve the accuracy of the posterior approximation and the speed of the VB estimation. However, the implementation is a bit more complex. Fast collapsed inference ++++++++++++++++++++++++ .. MOVE THE FOLLOWING TO, FOR INSTANCE, MOG OR PCA EXAMPLE: >>> def reset(): ... alpha.initialize_from_prior() ... C.initialize_from_prior() ... X.initialize_from_random() ... tau.initialize_from_prior() ... return VB(Y, C, X, alpha, tau) >>> Q = reset() >>> Q.update(repeat=1000) ... >>> bpplt.pyplot.plot(Q.L, 'k-') >>> Q = reset() >>> Q.optimize(X, C, alpha, tau, maxiter=1000) ... >>> bpplt.pyplot.plot(Q.L, 'r--') .. plot:: import numpy as np np.random.seed(1) # This is the PCA model from the previous sections from bayespy.nodes import GaussianARD, Gamma, Dot D = 3 X = GaussianARD(0, 1, shape=(D,), plates=(1,100), name='X') alpha = Gamma(1e-3, 1e-3, plates=(D,), name='alpha') C = GaussianARD(0, alpha, shape=(D,), plates=(10,1), name='C') F = Dot(C, X) tau = Gamma(1e-3, 1e-3, name='tau') Y = GaussianARD(F, tau, name='Y') c = np.random.randn(10, 2) x = np.random.randn(2, 100) data = np.dot(c, x) + 0.1*np.random.randn(10, 100) Y.observe(data) from bayespy.inference import VB import bayespy.plot as bpplt Q = None def reset(): alpha.initialize_from_prior() C.initialize_from_prior() X.initialize_from_random() tau.initialize_from_prior() return VB(Y, C, X, alpha, tau) Q = reset() Q.update(repeat=1000) bpplt.pyplot.plot(np.cumsum(Q.cputime), Q.L, 'k-') Q = reset() Q.optimize(X, C, alpha, tau, maxiter=1000) bpplt.pyplot.plot(np.cumsum(Q.cputime), Q.L, 'b--') Q = reset() Q.optimize(C, tau, maxiter=1000, collapsed=[X, alpha]) bpplt.pyplot.plot(np.cumsum(Q.cputime), Q.L, 'r:') bpplt.pyplot.ylim(-100, 100) bayespy-0.4.0/doc/source/examples/additive_fhmm.rst0000644000175000017500000000704112533543154024131 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. Additive factorial hidden Markov model ====================================== .. code:: python import numpy as np from bayespy.nodes import (Dirichlet, CategoricalMarkovChain, GaussianARD, Gate, SumMultiply, Gamma) D = 4 # dimensionality of the data vectors (and mu vectors) N = 5 # number of chains K = 3 # number of states in each chain T = 100 # length of each chain # Markov chain parameters. # Use known values p0 = np.ones(K) / K P = np.ones((K,K)) / K # Or set Dirichlet prior. p0 = Dirichlet(np.ones(K), plates=(N,)) P = Dirichlet(np.ones(K), plates=(N,1,K)) # N Markov chains with K possible states, and length T X = CategoricalMarkovChain(p0, P, states=T, plates=(N,)) # For each of the N chains, have K different D-dimensional mu's # Unknown mu's mu = GaussianARD(0, 1e-3, plates=(D,1,1,K), shape=(N,)) # Gate/select mu's print(mu.plates, mu.dims[0], X.plates, X.dims[1]) Z = Gate(X, mu) # Sum the mu's of different chains F = SumMultiply('i->', Z) print(mu.plates, mu.dims[0], X.plates, X.dims[0], Z.plates, Z.dims[0], F.plates, F.dims[0]) # Known observation noise inverse covariance tau = np.ones(D) # or unknown observation noise inverse covariance tau = Gamma(1e-3, 1e-3, plates=(D,)) # Observed process Y = GaussianARD(F, tau) # Data data = np.random.randn(T, D) Y.observe(data) from bayespy.inference import VB Q = VB(Y, X, p0, P, mu, tau) Q.update(repeat=10) .. parsed-literal:: (4, 1, 1, 3) (5,) (5,) (99, 3, 3) (4, 1, 1, 3) (5,) (5,) (3,) (4, 5, 100) (5,) (4, 5, 100) () :: --------------------------------------------------------------------------- ValueError Traceback (most recent call last) in () 41 42 # Observed process ---> 43 Y = GaussianARD(F, tau) 44 45 # Data /home/jluttine/workspace/bayespy/bayespy/inference/vmp/nodes/expfamily.py in constructor_decorator(self, *args, **kwargs) 81 82 (args, kwargs, dims, plates, dist, stats, pstats) = \ ---> 83 self._constructor(*args, **kwargs) 84 85 self.dims = dims /home/jluttine/workspace/bayespy/bayespy/inference/vmp/nodes/gaussian.py in _constructor(cls, mu, alpha, ndim, shape, **kwargs) 890 plates = cls._total_plates(kwargs.get('plates'), 891 distribution.plates_from_parent(0, mu.plates), --> 892 distribution.plates_from_parent(1, alpha.plates)) 893 894 parents = [mu, alpha] /home/jluttine/workspace/bayespy/bayespy/inference/vmp/nodes/node.py in _total_plates(cls, plates, *parent_plates) 244 return utils.broadcasted_shape(*parent_plates) 245 except ValueError: --> 246 raise ValueError("The plates of the parents do not broadcast.") 247 else: 248 # Check that the parent_plates are a subset of plates. ValueError: The plates of the parents do not broadcast. bayespy-0.4.0/doc/source/examples/pca.rst0000644000175000017500000001371212535524744022104 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. .. testsetup:: import numpy numpy.random.seed(1) Principal component analysis ============================ This example uses a simple principal component analysis to find a two-dimensional latent subspace in a higher dimensional dataset. Data ---- Let us create a Gaussian dataset with latent space dimensionality two and some observation noise: >>> M = 20 >>> N = 100 >>> import numpy as np >>> x = np.random.randn(N, 2) >>> w = np.random.randn(M, 2) >>> f = np.einsum('ik,jk->ij', w, x) >>> y = f + 0.1*np.random.randn(M, N) Model ----- We will use 10-dimensional latent space in our model and let it learn the true dimensionality: >>> D = 10 Import relevant nodes: >>> from bayespy.nodes import GaussianARD, Gamma, SumMultiply The latent states: >>> X = GaussianARD(0, 1, plates=(1,N), shape=(D,)) The loading matrix with automatic relevance determination (ARD) prior: >>> alpha = Gamma(1e-5, 1e-5, plates=(D,)) >>> C = GaussianARD(0, alpha, plates=(M,1), shape=(D,)) Compute the dot product of the latent states and the loading matrix: >>> F = SumMultiply('d,d->', X, C) The observation noise: >>> tau = Gamma(1e-5, 1e-5) The observed variable: >>> Y = GaussianARD(F, tau) Inference --------- Observe the data: >>> Y.observe(y) We do not have missing data now, but they could be easily handled with ``mask`` keyword argument. Construct variational Bayesian (VB) inference engine: >>> from bayespy.inference import VB >>> Q = VB(Y, X, C, alpha, tau) Initialize the latent subspace randomly, otherwise both ``X`` and ``C`` would converge to zero: >>> C.initialize_from_random() .. currentmodule:: bayespy.inference Now we could use :func:`VB.update` to run the inference. However, let us first create a parameter expansion to speed up the inference. The expansion is based on rotating the latent subspace optimally. This is optional but will usually improve the speed of the inference significantly, especially in high-dimensional problems: >>> from bayespy.inference.vmp.transformations import RotateGaussianARD >>> rot_X = RotateGaussianARD(X) >>> rot_C = RotateGaussianARD(C, alpha) By giving ``alpha`` for ``rot_C``, the rotation will also optimize ``alpha`` jointly with ``C``. Now that we have defined the rotations for our variables, we need to construct an optimizer: >>> from bayespy.inference.vmp.transformations import RotationOptimizer >>> R = RotationOptimizer(rot_X, rot_C, D) In order to use the rotations automatically, we need to set it as a callback function: >>> Q.set_callback(R.rotate) For more information about the rotation parameter expansion, see :cite:`Luttinen:2010` and :cite:`Luttinen:2013`. Now we can run the actual inference until convergence: >>> Q.update(repeat=1000) Iteration 1: loglike=-2.339710e+03 (... seconds) ... Iteration 22: loglike=6.500...e+02 (... seconds) Converged at iteration 2... Results ------- The results can be visualized, for instance, by plotting the Hinton diagram of the loading matrix: >>> import bayespy.plot as bpplt >>> bpplt.pyplot.figure() >>> bpplt.hinton(C) .. plot:: import numpy numpy.random.seed(1) M = 20 N = 100 import numpy as np x = np.random.randn(N, 2) w = np.random.randn(M, 2) f = np.einsum('ik,jk->ij', w, x) y = f + 0.1*np.random.randn(M, N) D = 10 from bayespy.nodes import GaussianARD, Gamma, SumMultiply X = GaussianARD(0, 1, plates=(1,N), shape=(D,)) alpha = Gamma(1e-5, 1e-5, plates=(D,)) C = GaussianARD(0, alpha, plates=(M,1), shape=(D,)) F = SumMultiply('d,d->', X, C) tau = Gamma(1e-5, 1e-5) Y = GaussianARD(F, tau) Y.observe(y) from bayespy.inference import VB Q = VB(Y, X, C, alpha, tau) C.initialize_from_random() from bayespy.inference.vmp.transformations import RotateGaussianARD rot_X = RotateGaussianARD(X) rot_C = RotateGaussianARD(C, alpha) from bayespy.inference.vmp.transformations import RotationOptimizer R = RotationOptimizer(rot_X, rot_C, D) Q.set_callback(R.rotate) Q.update(repeat=1000) import bayespy.plot as bpplt bpplt.hinton(C) The method has been able to prune out unnecessary latent dimensions and keep two components, which is the true number of components. >>> bpplt.pyplot.figure() >>> bpplt.plot(F) >>> bpplt.plot(f, color='r', marker='x', linestyle='None') .. plot:: import numpy numpy.random.seed(1) M = 20 N = 100 import numpy as np x = np.random.randn(N, 2) w = np.random.randn(M, 2) f = np.einsum('ik,jk->ij', w, x) y = f + 0.1*np.random.randn(M, N) D = 10 from bayespy.nodes import GaussianARD, Gamma, SumMultiply X = GaussianARD(0, 1, plates=(1,N), shape=(D,)) alpha = Gamma(1e-5, 1e-5, plates=(D,)) C = GaussianARD(0, alpha, plates=(M,1), shape=(D,)) F = SumMultiply('d,d->', X, C) tau = Gamma(1e-5, 1e-5) Y = GaussianARD(F, tau) Y.observe(y) from bayespy.inference import VB Q = VB(Y, X, C, alpha, tau) C.initialize_from_random() from bayespy.inference.vmp.transformations import RotateGaussianARD rot_X = RotateGaussianARD(X) rot_C = RotateGaussianARD(C, alpha) from bayespy.inference.vmp.transformations import RotationOptimizer R = RotationOptimizer(rot_X, rot_C, D) Q.set_callback(R.rotate) Q.update(repeat=1000) import bayespy.plot as bpplt bpplt.plot(F) bpplt.plot(f, color='r', marker='x', linestyle='None') .. currentmodule:: bayespy.nodes The reconstruction of the noiseless function values are practically perfect in this simple example. Larger noise variance, more latent space dimensions and missing values would make this problem more difficult. The model construction could also be improved by having, for instance, ``C`` and ``tau`` in the same node without factorizing between them in the posterior approximation. This can be achieved by using :class:`GaussianGammaISO` node. bayespy-0.4.0/doc/source/examples/examples.rst0000644000175000017500000000036012600470630023135 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. Examples ******** .. toctree:: :maxdepth: 1 regression gmm bmm hmm pca lssm lda bayespy-0.4.0/doc/source/examples/bmm.rst0000644000175000017500000002016612533543154022107 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. .. testsetup:: import numpy numpy.random.seed(1) Bernoulli mixture model ======================= This example considers data generated from a Bernoulli mixture model. One simple example process could be a questionnaire for election candidates. We observe a set of binary vectors, where each vector represents a candidate in the election and each element in these vectors correspond to a candidate's answer to a yes-or-no question. The goal is to find groups of similar candidates and analyze the answer patterns of these groups. Data ---- First, we generate artificial data to analyze. Let us assume that the questionnaire contains ten yes-or-no questions. We assume that there are three groups with similar opinions. These groups could represent parties. These groups have the following answering patterns, which are represented by vectors with probabilities of a candidate answering yes to the questions: >>> p0 = [0.1, 0.9, 0.1, 0.9, 0.1, 0.9, 0.1, 0.9, 0.1, 0.9] >>> p1 = [0.1, 0.1, 0.1, 0.1, 0.1, 0.9, 0.9, 0.9, 0.9, 0.9] >>> p2 = [0.9, 0.9, 0.9, 0.9, 0.9, 0.1, 0.1, 0.1, 0.1, 0.1] Thus, the candidates in the first group are likely to answer no to questions 1, 3, 5, 7 and 9, and yes to questions 2, 4, 6, 8, 10. The candidates in the second group are likely to answer yes to the last five questions, whereas the candidates in the third group are likely to answer yes to the first five questions. For convenience, we form a NumPy array of these vectors: >>> import numpy as np >>> p = np.array([p0, p1, p2]) Next, we generate a hundred candidates. First, we randomly select the group for each candidate: >>> from bayespy.utils import random >>> z = random.categorical([1/3, 1/3, 1/3], size=100) Using the group patterns, we generate yes-or-no answers for the candidates: >>> x = random.bernoulli(p[z]) This is our simulated data to be analyzed. Model ----- Now, we construct a model for learning the structure in the data. We have a dataset of hundred 10-dimensional binary vectors: >>> N = 100 >>> D = 10 We will create a Bernoulli mixture model. We assume that the true number of groups is unknown to us, so we use a large enough number of clusters: >>> K = 10 We use the categorical distribution for the group assignments and give the group assignment probabilities an uninformative Dirichlet prior: >>> from bayespy.nodes import Categorical, Dirichlet >>> R = Dirichlet(K*[1e-5], ... name='R') >>> Z = Categorical(R, ... plates=(N,1), ... name='Z') Each group has a probability of a yes answer for each question. These probabilities are given beta priors: >>> from bayespy.nodes import Beta >>> P = Beta([0.5, 0.5], ... plates=(D,K), ... name='P') The answers of the candidates are modelled with the Bernoulli distribution: >>> from bayespy.nodes import Mixture, Bernoulli >>> X = Mixture(Z, Bernoulli, P) Here, ``Z`` defines the group assignments and ``P`` the answering probability patterns for each group. Note how the plates of the nodes are matched: ``Z`` has plates ``(N,1)`` and ``P`` has plates ``(D,K)``, but in the mixture node the last plate axis of ``P`` is discarded and thus the node broadcasts plates ``(N,1)`` and ``(D,)`` resulting in plates ``(N,D)`` for ``X``. Inference --------- In order to infer the variables in our model, we construct a variational Bayesian inference engine: >>> from bayespy.inference import VB >>> Q = VB(Z, R, X, P) This also gives the default update order of the nodes. In order to find different groups, they must be initialized differently, thus we use random initialization for the group probability patterns: >>> P.initialize_from_random() We provide our simulated data: >>> X.observe(x) Now, we can run inference: >>> Q.update(repeat=1000) Iteration 1: loglike=-6.872145e+02 (... seconds) ... Iteration 17: loglike=-5.236921e+02 (... seconds) Converged at iteration 17. The algorithm converges in 17 iterations. Results ------- Now we can examine the approximate posterior distribution. First, let us plot the group assignment probabilities: >>> import bayespy.plot as bpplt >>> bpplt.hinton(R) .. plot:: import numpy numpy.random.seed(1) p0 = [0.1, 0.9, 0.1, 0.9, 0.1, 0.9, 0.1, 0.9, 0.1, 0.9] p1 = [0.1, 0.1, 0.1, 0.1, 0.1, 0.9, 0.9, 0.9, 0.9, 0.9] p2 = [0.9, 0.9, 0.9, 0.9, 0.9, 0.1, 0.1, 0.1, 0.1, 0.1] import numpy as np p = np.array([p0, p1, p2]) from bayespy.utils import random z = random.categorical([1/3, 1/3, 1/3], size=100) x = random.bernoulli(p[z]) N = 100 D = 10 K = 10 from bayespy.nodes import Categorical, Dirichlet R = Dirichlet(K*[1e-5], name='R') Z = Categorical(R, plates=(N,1), name='Z') from bayespy.nodes import Beta P = Beta([0.5, 0.5], plates=(D,K), name='P') from bayespy.nodes import Mixture, Bernoulli X = Mixture(Z, Bernoulli, P) from bayespy.inference import VB Q = VB(Z, R, X, P) P.initialize_from_random() X.observe(x) Q.update(repeat=1000) import bayespy.plot as bpplt bpplt.hinton(R) bpplt.pyplot.show() This plot shows that there are three dominant groups, which is equal to the true number of groups used to generate the data. However, there are still two smaller groups as the data does not give enough evidence to prune them out. The yes-or-no answer probability patterns for the groups can be plotted as: >>> bpplt.hinton(P) .. plot:: import numpy numpy.random.seed(1) p0 = [0.1, 0.9, 0.1, 0.9, 0.1, 0.9, 0.1, 0.9, 0.1, 0.9] p1 = [0.1, 0.1, 0.1, 0.1, 0.1, 0.9, 0.9, 0.9, 0.9, 0.9] p2 = [0.9, 0.9, 0.9, 0.9, 0.9, 0.1, 0.1, 0.1, 0.1, 0.1] import numpy as np p = np.array([p0, p1, p2]) from bayespy.utils import random z = random.categorical([1/3, 1/3, 1/3], size=100) x = random.bernoulli(p[z]) N = 100 D = 10 K = 10 from bayespy.nodes import Categorical, Dirichlet R = Dirichlet(K*[1e-5], name='R') Z = Categorical(R, plates=(N,1), name='Z') from bayespy.nodes import Beta P = Beta([0.5, 0.5], plates=(D,K), name='P') from bayespy.nodes import Mixture, Bernoulli X = Mixture(Z, Bernoulli, P) from bayespy.inference import VB Q = VB(Z, R, X, P) P.initialize_from_random() X.observe(x) Q.update(repeat=1000) import bayespy.plot as bpplt bpplt.hinton(P) bpplt.pyplot.show() The three dominant groups have found the true patterns accurately. The patterns of the two minor groups some kind of mixtures of the three groups and they exist because the generated data happened to contain a few samples giving evidence for these groups. Finally, we can plot the group assignment probabilities for the candidates: >>> bpplt.hinton(Z) .. plot:: import numpy numpy.random.seed(1) p0 = [0.1, 0.9, 0.1, 0.9, 0.1, 0.9, 0.1, 0.9, 0.1, 0.9] p1 = [0.1, 0.1, 0.1, 0.1, 0.1, 0.9, 0.9, 0.9, 0.9, 0.9] p2 = [0.9, 0.9, 0.9, 0.9, 0.9, 0.1, 0.1, 0.1, 0.1, 0.1] import numpy as np p = np.array([p0, p1, p2]) from bayespy.utils import random z = random.categorical([1/3, 1/3, 1/3], size=100) x = random.bernoulli(p[z]) N = 100 D = 10 K = 10 from bayespy.nodes import Categorical, Dirichlet R = Dirichlet(K*[1e-5], name='R') Z = Categorical(R, plates=(N,1), name='Z') from bayespy.nodes import Beta P = Beta([0.5, 0.5], plates=(D,K), name='P') from bayespy.nodes import Mixture, Bernoulli X = Mixture(Z, Bernoulli, P) from bayespy.inference import VB Q = VB(Z, R, X, P) P.initialize_from_random() X.observe(x) Q.update(repeat=1000) import bayespy.plot as bpplt bpplt.hinton(Z) bpplt.pyplot.show() .. currentmodule:: bayespy.plot This plot shows the clustering of the candidates. It is possible to use :class:`HintonPlotter` to enable monitoring during the VB iteration by providing ``plotter=HintonPlotter()`` for ``Z``, ``P`` and ``R`` when creating the nodes. bayespy-0.4.0/doc/source/examples/lda.rst0000644000175000017500000002055612600471211022064 0ustar jluttinejluttine00000000000000.. Copyright (C) 2015 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. Latent Dirichlet allocation =========================== Latent Dirichlet allocation is a widely used topic model. The data is a collection of documents which contain words. The goal of the analysis is to find topics (distribution of words in topics) and document topics (distribution of topics in documents). Data ---- The data consists of two vectors of equal length. The elements in these vectors correspond to the words in all documents combined. If there were :math:`M` documents and each document had :math:`K` words, the vectors contain :math:`M \cdot K` elements. Let :math:`M` be the number of documents in total. The first vector gives each word a document index :math:`i\in \{0,\ldots,M-1\}` defining to which document the word belongs. Let :math:`N` be the size of the whole available vocabulary. The second vector gives each word a vocabulary index :math:`j\in \{0,\ldots,N-1\}` defining which word it is from the vocabulary. For this demo, we will just generate an artificial dataset for simplicity. We use the LDA model itself to generate the dataset. First, import relevant packages: >>> import numpy as np >>> from bayespy import nodes Let us decide the number of documents and the number of words in those documents: >>> n_documents = 10 >>> n_words = 10000 Randomly choose into which document each word belongs to: >>> word_documents = nodes.Categorical(np.ones(n_documents)/n_documents, ... plates=(n_words,)).random() Let us also decide the size of our vocabulary: >>> n_vocabulary = 100 Also, let us decide the true number of topics: >>> n_topics = 5 Generate some random distributions for the topics in each document: >>> p_topic = nodes.Dirichlet(1e-1*np.ones(n_topics), ... plates=(n_documents,)).random() Generate some random distributions for the words in each topic: >>> p_word = nodes.Dirichlet(1e-1*np.ones(n_vocabulary), ... plates=(n_topics,)).random() Sample topic assignments for each word in each document: >>> topic = nodes.Categorical(p_topic[word_documents], ... plates=(n_words,)).random() And finally, draw vocabulary indices for each word in all the documents: >>> corpus = nodes.Categorical(p_word[topic], ... plates=(n_words,)).random() Now, our dataset consists of ``word_documents`` and ``corpus``, which define the document and vocabulary indices for each word in our dataset. .. todo:: Use some large real-world dataset, for instance, Wikipedia. Model ----- Variable for learning the topic distribution for each document: >>> p_topic = nodes.Dirichlet(np.ones(n_topics), ... plates=(n_documents,), ... name='p_topic') Variable for learning the word distribution for each topic: >>> p_word = nodes.Dirichlet(np.ones(n_vocabulary), ... plates=(n_topics,), ... name='p_word') The document indices for each word in the corpus: >>> from bayespy.inference.vmp.nodes.categorical import CategoricalMoments >>> document_indices = nodes.Constant(CategoricalMoments(n_documents), word_documents, ... name='document_indices') Variable for learning the topic assignments of each word in the corpus: >>> topics = nodes.Categorical(nodes.Gate(document_indices, p_topic), ... plates=(len(corpus),), ... name='topics') The vocabulary indices for each word in the corpus: >>> words = nodes.Categorical(nodes.Gate(topics, p_word), ... name='words') Inference --------- Observe the corpus: >>> words.observe(corpus) Break symmetry by random initialization: >>> p_topic.initialize_from_random() >>> p_word.initialize_from_random() Construct inference engine: >>> from bayespy.inference import VB >>> Q = VB(words, topics, p_word, p_topic, document_indices) Run the VB learning algorithm: >>> Q.update(repeat=1000) Iteration ... Results ------- Use ``bayespy.plot`` to plot the results: >>> import bayespy.plot as bpplt Plot the topic distributions for each document: >>> bpplt.pyplot.figure() >>> bpplt.hinton(Q['p_topic']) >>> bpplt.pyplot.title("Posterior topic distribution for each document") >>> bpplt.pyplot.xlabel("Topics") >>> bpplt.pyplot.ylabel("Documents") Plot the word distributions for each topic: >>> bpplt.pyplot.figure() >>> bpplt.hinton(Q['p_word']) >>> bpplt.pyplot.title("Posterior word distributions for each topic") >>> bpplt.pyplot.xlabel("Words") >>> bpplt.pyplot.ylabel("Topics") .. todo:: Create more illustrative plots. Stochastic variational inference -------------------------------- LDA is a popular example for stochastic variational inference (SVI). Using SVI for LDA is quite simple in BayesPy. In SVI, only a subset of the dataset is used at each iteration step but this subset is "repeated" to get the same size as the original dataset. Let us define a size for the subset: >>> subset_size = 1000 Thus, our subset will be repeat this many times: >>> plates_multiplier = n_words / subset_size Note that this multiplier doesn't need to be an integer. Now, let us repeat the model construction with only one minor addition. The following variables are identical to previous: >>> p_topic = nodes.Dirichlet(np.ones(n_topics), ... plates=(n_documents,), ... name='p_topic') >>> p_word = nodes.Dirichlet(np.ones(n_vocabulary), ... plates=(n_topics,), ... name='p_word') The document indices vector is now a bit shorter, using only a subset: >>> document_indices = nodes.Constant(CategoricalMoments(n_documents), ... word_documents[:subset_size], ... name='document_indices') Note that at this point, it doesn't matter which elements we chose for the subset. For the topic assignments of each word in the corpus we need to use ``plates_multiplier`` because these topic assignments for the subset are "repeated" to recover the full dataset: >>> topics = nodes.Categorical(nodes.Gate(document_indices, p_topic), ... plates=(subset_size,), ... plates_multiplier=(plates_multiplier,), ... name='topics') Finally, the vocabulary indices for each word in the corpus are constructed as before: >>> words = nodes.Categorical(nodes.Gate(topics, p_word), ... name='words') This node inherits the plates and multipliers from its parent ``topics``, so there is no need to define them here. Again, break symmetry by random initialization: >>> p_topic.initialize_from_random() >>> p_word.initialize_from_random() Construct inference engine: >>> from bayespy.inference import VB >>> Q = VB(words, topics, p_word, p_topic, document_indices) In order to use SVI, we need to disable some lower bound checks, because the lower bound doesn't anymore necessarily increase at each iteration step: >>> Q.ignore_bound_checks = True For the stochastic gradient ascent, we'll define some learning parameters: >>> delay = 1 >>> forgetting_rate = 0.7 Run the inference: >>> for n in range(1000): ... # Observe a random mini-batch ... subset = np.random.choice(n_words, subset_size) ... Q['words'].observe(corpus[subset]) ... Q['document_indices'].set_value(word_documents[subset]) ... # Learn intermediate variables ... Q.update('topics') ... # Set step length ... step = (n + delay) ** (-forgetting_rate) ... # Stochastic gradient for the global variables ... Q.gradient_step('p_topic', 'p_word', scale=step) Iteration 1: ... If one is interested, the lower bound values during the SVI algorithm can be plotted as: >>> bpplt.pyplot.figure() >>> bpplt.pyplot.plot(Q.L) [] The other results can be plotted as before. bayespy-0.4.0/doc/source/examples/regression.rst0000644000175000017500000002021512533543154023507 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. .. testsetup:: import numpy numpy.random.seed(1) Linear regression ================= Data ---- The true parameters of the linear regression: >>> k = 2 # slope >>> c = 5 # bias >>> s = 2 # noise standard deviation Generate data: >>> import numpy as np >>> x = np.arange(10) >>> y = k*x + c + s*np.random.randn(10) Model ----- The regressors, that is, the input data: >>> X = np.vstack([x, np.ones(len(x))]).T Note that we added a column of ones to the regressor matrix for the bias term. We model the slope and the bias term in the same node so we do not factorize between them: >>> from bayespy.nodes import GaussianARD >>> B = GaussianARD(0, 1e-6, shape=(2,)) The first element is the slope which multiplies ``x`` and the second element is the bias term which multiplies the constant ones. Now we compute the dot product of ``X`` and ``B``: >>> from bayespy.nodes import SumMultiply >>> F = SumMultiply('i,i', B, X) The noise parameter: >>> from bayespy.nodes import Gamma >>> tau = Gamma(1e-3, 1e-3) The noisy observations: >>> Y = GaussianARD(F, tau) Inference --------- Observe the data: >>> Y.observe(y) Construct the variational Bayesian (VB) inference engine by giving all stochastic nodes: >>> from bayespy.inference import VB >>> Q = VB(Y, B, tau) Iterate until convergence: >>> Q.update(repeat=1000) Iteration 1: loglike=-4.595948e+01 (... seconds) ... Iteration 5: loglike=-4.495017e+01 (... seconds) Converged at iteration 5. Results ------- Create a simple predictive model for new inputs: >>> xh = np.linspace(-5, 15, 100) >>> Xh = np.vstack([xh, np.ones(len(xh))]).T >>> Fh = SumMultiply('i,i', B, Xh) Note that we use the learned node ``B`` but create a new regressor array for predictions. Plot the predictive distribution of noiseless function values: >>> import bayespy.plot as bpplt >>> bpplt.pyplot.figure() >>> bpplt.plot(Fh, x=xh, scale=2) >>> bpplt.plot(y, x=x, color='r', marker='x', linestyle='None') >>> bpplt.plot(k*xh+c, x=xh, color='r'); .. plot:: import numpy numpy.random.seed(1) k = 2 # slope c = 5 # bias s = 2 # noise standard deviation import numpy as np x = np.arange(10) y = k*x + c + s*np.random.randn(10) X = np.vstack([x, np.ones(len(x))]).T from bayespy.nodes import GaussianARD B = GaussianARD(0, 1e-6, shape=(2,)) from bayespy.nodes import SumMultiply F = SumMultiply('i,i', B, X) from bayespy.nodes import Gamma tau = Gamma(1e-3, 1e-3) Y = GaussianARD(F, tau) Y.observe(y) from bayespy.inference import VB Q = VB(Y, B, tau) Q.update(repeat=1000) xh = np.linspace(-5, 15, 100) Xh = np.vstack([xh, np.ones(len(xh))]).T Fh = SumMultiply('i,i', B, Xh) import bayespy.plot as bpplt bpplt.plot(Fh, x=xh, scale=2) bpplt.plot(y, x=x, color='r', marker='x', linestyle='None') bpplt.plot(k*xh+c, x=xh, color='r'); bpplt.pyplot.show() Note that the above plot shows two standard deviation of the posterior of the noiseless function, thus the data points may lie well outside this range. The red line shows the true linear function. Next, plot the distribution of the noise parameter and the true value, :math:`2^{-2}=0.25`: >>> bpplt.pyplot.figure() >>> bpplt.pdf(tau, np.linspace(1e-6,1,100), color='k') [] >>> bpplt.pyplot.axvline(s**(-2), color='r') .. plot:: import numpy numpy.random.seed(1) k = 2 # slope c = 5 # bias s = 2 # noise standard deviation import numpy as np x = np.arange(10) y = k*x + c + s*np.random.randn(10) X = np.vstack([x, np.ones(len(x))]).T from bayespy.nodes import GaussianARD B = GaussianARD(0, 1e-6, shape=(2,)) from bayespy.nodes import SumMultiply F = SumMultiply('i,i', B, X) from bayespy.nodes import Gamma tau = Gamma(1e-3, 1e-3) Y = GaussianARD(F, tau) Y.observe(y) from bayespy.inference import VB Q = VB(Y, B, tau) Q.update(repeat=1000) xh = np.linspace(-5, 15, 100) Xh = np.vstack([xh, np.ones(len(xh))]).T Fh = SumMultiply('i,i', B, Xh) import bayespy.plot as bpplt bpplt.pdf(tau, np.linspace(1e-6,1,100), color='k') bpplt.pyplot.axvline(s**(-2), color='r') bpplt.pyplot.show() The noise level is captured quite well, although the posterior has more mass on larger noise levels (smaller precision parameter values). Finally, plot the distribution of the regression parameters and mark the true value: >>> bpplt.pyplot.figure() >>> bpplt.contour(B, np.linspace(1,3,1000), np.linspace(1,9,1000), ... n=10, colors='k') >>> bpplt.plot(c, x=k, color='r', marker='x', linestyle='None', ... markersize=10, markeredgewidth=2) >>> bpplt.pyplot.xlabel(r'$k$') >>> bpplt.pyplot.ylabel(r'$c$'); .. plot:: import numpy numpy.random.seed(1) k = 2 # slope c = 5 # bias s = 2 # noise standard deviation import numpy as np x = np.arange(10) y = k*x + c + s*np.random.randn(10) X = np.vstack([x, np.ones(len(x))]).T from bayespy.nodes import GaussianARD B = GaussianARD(0, 1e-6, shape=(2,)) from bayespy.nodes import SumMultiply F = SumMultiply('i,i', B, X) from bayespy.nodes import Gamma tau = Gamma(1e-3, 1e-3) Y = GaussianARD(F, tau) Y.observe(y) from bayespy.inference import VB Q = VB(Y, B, tau) Q.update(repeat=1000) xh = np.linspace(-5, 15, 100) Xh = np.vstack([xh, np.ones(len(xh))]).T Fh = SumMultiply('i,i', B, Xh) import bayespy.plot as bpplt bpplt.contour(B, np.linspace(1,3,1000), np.linspace(1,9,1000), n=10, colors='k') bpplt.plot(c, x=k, color='r', marker='x', linestyle='None', markersize=10, markeredgewidth=2) bpplt.pyplot.xlabel(r'$k$') bpplt.pyplot.ylabel(r'$c$'); bpplt.pyplot.show() In this case, the true parameters are captured well by the posterior distribution. Improving accuracy ------------------ .. currentmodule:: bayespy.nodes The model can be improved by not factorizing between ``B`` and ``tau`` but learning their joint posterior distribution. This requires a slight modification to the model by using :class:`GaussianGammaISO` node: >>> from bayespy.nodes import GaussianGammaISO >>> B_tau = GaussianGammaISO(np.zeros(2), 1e-6*np.identity(2), 1e-3, 1e-3) This node contains both the regression parameter vector and the noise parameter. We compute the dot product similarly as before: >>> F_tau = SumMultiply('i,i', B_tau, X) However, ``Y`` is constructed as follows: >>> Y = GaussianARD(F_tau, 1) Because the noise parameter is already in ``F_tau`` we can give a constant one as the second argument. The total noise parameter for ``Y`` is the product of the noise parameter in ``F_tau`` and one. Now, inference is run similarly as before: >>> Y.observe(y) >>> Q = VB(Y, B_tau) >>> Q.update(repeat=1000) Iteration 1: loglike=-4.678478e+01 (... seconds) Iteration 2: loglike=-4.678478e+01 (... seconds) Converged at iteration 2. Note that the method converges immediately. This happens because there is only one unobserved stochastic node so there is no need for iteration and the result is actually the exact true posterior distribution, not an approximation. Currently, the main drawback of using this approach is that BayesPy does not yet contain any plotting utilities for nodes that contain both Gaussian and gamma variables jointly. Further extensions ------------------ The approach discussed in this example can easily be extended to non-linear regression and multivariate regression. For non-linear regression, the inputs are first transformed by some known non-linear functions and then linear regression is applied to this transformed data. For multivariate regression, ``X`` and ``B`` are concatenated appropriately: If there are more regressors, add more columns to both ``X`` and ``B``. If there are more output dimensions, add plates to ``B``. bayespy-0.4.0/doc/source/examples/my_test.rst0000644000175000017500000000011512564300215023001 0ustar jluttinejluttine00000000000000 .. code:: python 4+5 .. parsed-literal:: 9 Here some text. bayespy-0.4.0/doc/source/examples/hmm.rst0000644000175000017500000003000312533543154022104 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. .. testsetup:: import numpy numpy.random.seed(1) Hidden Markov model =================== In this example, we will demonstrate the use of hidden Markov model in the case of known and unknown parameters. We will also use two different emission distributions to demonstrate the flexibility of the model construction. Known parameters ---------------- This example follows the one presented in `Wikipedia `__. Model +++++ Each day, the state of the weather is either 'rainy' or 'sunny'. The weather follows a first-order discrete Markov process. It has the following initial state probabilities >>> a0 = [0.6, 0.4] # p(rainy)=0.6, p(sunny)=0.4 and state transition probabilities: >>> A = [[0.7, 0.3], # p(rainy->rainy)=0.7, p(rainy->sunny)=0.3 ... [0.4, 0.6]] # p(sunny->rainy)=0.4, p(sunny->sunny)=0.6 We will be observing one hundred samples: >>> N = 100 The discrete first-order Markov chain is constructed as: >>> from bayespy.nodes import CategoricalMarkovChain >>> Z = CategoricalMarkovChain(a0, A, states=N) However, instead of observing this process directly, we observe whether Bob is 'walking', 'shopping' or 'cleaning'. The probability of each activity depends on the current weather as follows: >>> P = [[0.1, 0.4, 0.5], ... [0.6, 0.3, 0.1]] where the first row contains activity probabilities on a rainy weather and the second row contains activity probabilities on a sunny weather. Using these emission probabilities, the observed process is constructed as: >>> from bayespy.nodes import Categorical, Mixture >>> Y = Mixture(Z, Categorical, P) Data ++++ In order to test our method, we'll generate artificial data from the model itself. First, draw realization of the weather process: >>> weather = Z.random() Then, using this weather, draw realizations of the activities: >>> activity = Mixture(weather, Categorical, P).random() Inference +++++++++ Now, using this data, we set our variable :math:`Y` to be observed: >>> Y.observe(activity) In order to run inference, we construct variational Bayesian inference engine: >>> from bayespy.inference import VB >>> Q = VB(Y, Z) Note that we need to give all random variables to ``VB``. In this case, the only random variables were ``Y`` and ``Z``. Next we run the inference, that is, compute our posterior distribution: >>> Q.update() Iteration 1: loglike=-1.095883e+02 (... seconds) In this case, because there is only one unobserved random variable, we recover the exact posterior distribution and there is no need to iterate more than one step. Results +++++++ .. currentmodule:: bayespy.plot One way to plot a 2-class categorical timeseries is to use the basic :func:`plot` function: >>> import bayespy.plot as bpplt >>> bpplt.plot(Z) >>> bpplt.plot(1-weather, color='r', marker='x') .. plot:: import numpy numpy.random.seed(1) from bayespy.nodes import CategoricalMarkovChain a0 = [0.6, 0.4] # p(rainy)=0.6, p(sunny)=0.4 A = [[0.7, 0.3], # p(rainy->rainy)=0.7, p(rainy->sunny)=0.3 [0.4, 0.6]] # p(sunny->rainy)=0.4, p(sunny->sunny)=0.6 N = 100 Z = CategoricalMarkovChain(a0, A, states=N) from bayespy.nodes import Categorical, Mixture P = [[0.1, 0.4, 0.5], [0.6, 0.3, 0.1]] Y = Mixture(Z, Categorical, P) weather = Z.random() activity = Mixture(weather, Categorical, P).random() Y.observe(activity) from bayespy.inference import VB Q = VB(Y, Z) Q.update() import bayespy.plot as bpplt bpplt.plot(Z) bpplt.plot(1-weather, color='r', marker='x') bpplt.pyplot.show() The black line shows the posterior probability of rain and the red line and crosses show the true state. Clearly, the method is not able to infer the weather very accurately in this case because the activies do not give that much information about the weather. Unknown parameters ------------------ In this example, we consider unknown parameters for the Markov process and different emission distribution. Data ++++ We generate data from three 2-dimensional Gaussian distributions with different mean vectors and common standard deviation: >>> import numpy as np >>> mu = np.array([ [0,0], [3,4], [6,0] ]) >>> std = 2.0 Thus, the number of clusters is three: >>> K = 3 And the number of samples is 200: >>> N = 200 Each initial state is equally probable: >>> p0 = np.ones(K) / K State transition matrix is such that with probability 0.9 the process stays in the same state. The probability to move one of the other two states is 0.05 for both of those states. >>> q = 0.9 >>> r = (1-q) / (K-1) >>> P = q*np.identity(K) + r*(np.ones((3,3))-np.identity(3)) Simulate the data: >>> y = np.zeros((N,2)) >>> z = np.zeros(N) >>> state = np.random.choice(K, p=p0) >>> for n in range(N): ... z[n] = state ... y[n,:] = std*np.random.randn(2) + mu[state] ... state = np.random.choice(K, p=P[state]) Then, let us visualize the data: >>> bpplt.pyplot.figure() >>> bpplt.pyplot.axis('equal') (...) >>> colors = [ [[1,0,0], [0,1,0], [0,0,1]][int(state)] for state in z ] >>> bpplt.pyplot.plot(y[:,0], y[:,1], 'k-', zorder=-10) [] >>> bpplt.pyplot.scatter(y[:,0], y[:,1], c=colors, s=40) .. plot:: import numpy numpy.random.seed(1) from bayespy.nodes import CategoricalMarkovChain a0 = [0.6, 0.4] # p(rainy)=0.6, p(sunny)=0.4 A = [[0.7, 0.3], # p(rainy->rainy)=0.7, p(rainy->sunny)=0.3 [0.4, 0.6]] # p(sunny->rainy)=0.4, p(sunny->sunny)=0.6 N = 100 Z = CategoricalMarkovChain(a0, A, states=N) from bayespy.nodes import Categorical, Mixture P = [[0.1, 0.4, 0.5], [0.6, 0.3, 0.1]] Y = Mixture(Z, Categorical, P) weather = Z.random() from bayespy.inference import VB import bayespy.plot as bpplt import numpy as np mu = np.array([ [0,0], [3,4], [6,0] ]) std = 2.0 K = 3 N = 200 p0 = np.ones(K) / K q = 0.9 r = (1-q)/(K-1) P = q*np.identity(K) + r*(np.ones((3,3))-np.identity(3)) y = np.zeros((N,2)) z = np.zeros(N) state = np.random.choice(K, p=p0) for n in range(N): z[n] = state y[n,:] = std*np.random.randn(2) + mu[state] state = np.random.choice(K, p=P[state]) bpplt.pyplot.figure() bpplt.pyplot.axis('equal') colors = [ [[1,0,0], [0,1,0], [0,0,1]][int(state)] for state in z ] bpplt.pyplot.plot(y[:,0], y[:,1], 'k-', zorder=-10) bpplt.pyplot.scatter(y[:,0], y[:,1], c=colors, s=40) bpplt.pyplot.show() Consecutive states are connected by a solid black line and the dot color shows the true class. Model +++++ Now, assume that we do not know the parameters of the process (initial state probability and state transition probabilities). We give these parameters quite non-informative priors, but it is possible to provide more informative priors if such information is available: >>> from bayespy.nodes import Dirichlet >>> a0 = Dirichlet(1e-3*np.ones(K)) >>> A = Dirichlet(1e-3*np.ones((K,K))) The discrete Markov chain is constructed as: >>> Z = CategoricalMarkovChain(a0, A, states=N) Now, instead of using categorical emission distribution as before, we'll use Gaussian distribution. For simplicity, we use the true parameters of the Gaussian distributions instead of giving priors and estimating them. The known standard deviation can be converted to a precision matrix as: >>> Lambda = std**(-2) * np.identity(2) Thus, the observed process is a Gaussian mixture with cluster assignments from the hidden Markov process ``Z``: >>> from bayespy.nodes import Gaussian >>> Y = Mixture(Z, Gaussian, mu, Lambda) Note that ``Lambda`` does not have cluster plate axis because it is shared between the clusters. Inference +++++++++ Let us use the simulated data: >>> Y.observe(y) Because ``VB`` takes all the random variables, we need to provide ``A`` and ``a0`` also: >>> Q = VB(Y, Z, A, a0) Then, run VB iteration until convergence: >>> Q.update(repeat=1000) Iteration 1: loglike=-9.963054e+02 (... seconds) ... Iteration 8: loglike=-9.235053e+02 (... seconds) Converged at iteration 8. Results +++++++ Plot the classification of the data similarly as the data: >>> bpplt.pyplot.figure() >>> bpplt.pyplot.axis('equal') (...) >>> colors = Y.parents[0].get_moments()[0] >>> bpplt.pyplot.plot(y[:,0], y[:,1], 'k-', zorder=-10) [] >>> bpplt.pyplot.scatter(y[:,0], y[:,1], c=colors, s=40) .. plot:: import numpy numpy.random.seed(1) from bayespy.nodes import CategoricalMarkovChain a0 = [0.6, 0.4] # p(rainy)=0.6, p(sunny)=0.4 A = [[0.7, 0.3], # p(rainy->rainy)=0.7, p(rainy->sunny)=0.3 [0.4, 0.6]] # p(sunny->rainy)=0.4, p(sunny->sunny)=0.6 N = 100 Z = CategoricalMarkovChain(a0, A, states=N) from bayespy.nodes import Categorical, Mixture P = [[0.1, 0.4, 0.5], [0.6, 0.3, 0.1]] Y = Mixture(Z, Categorical, P) weather = Z.random() from bayespy.inference import VB import bayespy.plot as bpplt import numpy as np mu = np.array([ [0,0], [3,4], [6,0] ]) std = 2.0 K = 3 N = 200 p0 = np.ones(K) / K q = 0.9 r = (1-q)/(K-1) P = q*np.identity(K) + r*(np.ones((3,3))-np.identity(3)) y = np.zeros((N,2)) z = np.zeros(N) state = np.random.choice(K, p=p0) for n in range(N): z[n] = state y[n,:] = std*np.random.randn(2) + mu[state] state = np.random.choice(K, p=P[state]) from bayespy.nodes import Dirichlet a0 = Dirichlet(1e-3*np.ones(K)) A = Dirichlet(1e-3*np.ones((K,K))) Z = CategoricalMarkovChain(a0, A, states=N) Lambda = std**(-2) * np.identity(2) from bayespy.nodes import Gaussian Y = Mixture(Z, Gaussian, mu, Lambda) Y.observe(y) Q = VB(Y, Z, A, a0) Q.update(repeat=1000) bpplt.pyplot.figure() bpplt.pyplot.axis('equal') colors = Y.parents[0].get_moments()[0] bpplt.pyplot.plot(y[:,0], y[:,1], 'k-', zorder=-10) bpplt.pyplot.scatter(y[:,0], y[:,1], c=colors, s=40) bpplt.pyplot.show() The data has been classified quite correctly. Even samples that are more in the region of another cluster are classified correctly if the previous and next sample provide enough evidence for the correct class. We can also plot the state transition matrix: >>> bpplt.hinton(A) .. plot:: import numpy numpy.random.seed(1) from bayespy.nodes import CategoricalMarkovChain a0 = [0.6, 0.4] # p(rainy)=0.6, p(sunny)=0.4 A = [[0.7, 0.3], # p(rainy->rainy)=0.7, p(rainy->sunny)=0.3 [0.4, 0.6]] # p(sunny->rainy)=0.4, p(sunny->sunny)=0.6 N = 100 Z = CategoricalMarkovChain(a0, A, states=N) from bayespy.nodes import Categorical, Mixture P = [[0.1, 0.4, 0.5], [0.6, 0.3, 0.1]] Y = Mixture(Z, Categorical, P) weather = Z.random() from bayespy.inference import VB import bayespy.plot as bpplt import numpy as np mu = np.array([ [0,0], [3,4], [6,0] ]) std = 2.0 K = 3 N = 200 p0 = np.ones(K) / K q = 0.9 r = (1-q)/(K-1) P = q*np.identity(K) + r*(np.ones((3,3))-np.identity(3)) y = np.zeros((N,2)) z = np.zeros(N) state = np.random.choice(K, p=p0) for n in range(N): z[n] = state y[n,:] = std*np.random.randn(2) + mu[state] state = np.random.choice(K, p=P[state]) from bayespy.nodes import Dirichlet a0 = Dirichlet(1e-3*np.ones(K)) A = Dirichlet(1e-3*np.ones((K,K))) Z = CategoricalMarkovChain(a0, A, states=N) Lambda = std**(-2) * np.identity(2) from bayespy.nodes import Gaussian Y = Mixture(Z, Gaussian, mu, Lambda) Y.observe(y) Q = VB(Y, Z, A, a0) Q.update(repeat=1000) bpplt.hinton(A) bpplt.pyplot.show() Clearly, the learned state transition matrix is close to the true matrix. The models described above could also be used for classification by providing the known class assignments as observed data to ``Z`` and the unknown class assignments as missing data. bayespy-0.4.0/doc/source/examples/lssm.rst0000644000175000017500000003346412615643655022327 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. .. testsetup:: import numpy as np np.random.seed(1) Linear state-space model ======================== Model ----- .. currentmodule:: bayespy.nodes In linear state-space models a sequence of :math:`M`-dimensional observations :math:`\mathbf{Y}=(\mathbf{y}_1,\ldots,\mathbf{y}_N)` is assumed to be generated from latent :math:`D`-dimensional states :math:`\mathbf{X}=(\mathbf{x}_1,\ldots,\mathbf{x}_N)` which follow a first-order Markov process: .. math:: \begin{aligned} \mathbf{x}_{n} &= \mathbf{A}\mathbf{x}_{n-1} + \text{noise} \,, \\ \mathbf{y}_{n} &= \mathbf{C}\mathbf{x}_{n} + \text{noise} \,, \end{aligned} where the noise is Gaussian, :math:`\mathbf{A}` is the :math:`D\times D` state dynamics matrix and :math:`\mathbf{C}` is the :math:`M\times D` loading matrix. Usually, the latent space dimensionality :math:`D` is assumed to be much smaller than the observation space dimensionality :math:`M` in order to model the dependencies of high-dimensional observations efficiently. In order to construct the model in BayesPy, first import relevant nodes: >>> from bayespy.nodes import GaussianARD, GaussianMarkovChain, Gamma, Dot The data vectors will be 30-dimensional: >>> M = 30 There will be 400 data vectors: >>> N = 400 Let us use 10-dimensional latent space: >>> D = 10 The state dynamics matrix :math:`\mathbf{A}` has ARD prior: >>> alpha = Gamma(1e-5, ... 1e-5, ... plates=(D,), ... name='alpha') >>> A = GaussianARD(0, ... alpha, ... shape=(D,), ... plates=(D,), ... name='A') Note that :math:`\mathbf{A}` is a :math:`D\times{}D`-dimensional matrix. However, in BayesPy it is modelled as a collection (``plates=(D,)``) of :math:`D`-dimensional vectors (``shape=(D,)``) because this is how the variables factorize in the posterior approximation of the state dynamics matrix in :class:`GaussianMarkovChain`. The latent states are constructed as >>> X = GaussianMarkovChain(np.zeros(D), ... 1e-3*np.identity(D), ... A, ... np.ones(D), ... n=N, ... name='X') where the first two arguments are the mean and precision matrix of the initial state, the third argument is the state dynamics matrix and the fourth argument is the diagonal elements of the precision matrix of the innovation noise. The node also needs the length of the chain given as the keyword argument ``n=N``. Thus, the shape of this node is ``(N,D)``. The linear mapping from the latent space to the observation space is modelled with the loading matrix which has ARD prior: >>> gamma = Gamma(1e-5, ... 1e-5, ... plates=(D,), ... name='gamma') >>> C = GaussianARD(0, ... gamma, ... shape=(D,), ... plates=(M,1), ... name='C') Note that the plates for ``C`` are ``(M,1)``, thus the full shape of the node is ``(M,1,D)``. The unit plate axis is added so that ``C`` broadcasts with ``X`` when computing the dot product: >>> F = Dot(C, ... X, ... name='F') This dot product is computed over the :math:`D`-dimensional latent space, thus the result is a :math:`M\times{}N`-dimensional matrix which is now represented with plates ``(M,N)`` in BayesPy: >>> F.plates (30, 400) We also need to use random initialization either for ``C`` or ``X`` in order to find non-zero latent space because by default both ``C`` and ``X`` are initialized to zero because of their prior distributions. We use random initialization for ``C`` and then we must update ``X`` the first time before updating ``C``: >>> C.initialize_from_random() The precision of the observation noise is given gamma prior: >>> tau = Gamma(1e-5, ... 1e-5, ... name='tau') The observations are noisy versions of the dot products: >>> Y = GaussianARD(F, ... tau, ... name='Y') The variational Bayesian inference engine is then construced as: >>> from bayespy.inference import VB >>> Q = VB(X, C, gamma, A, alpha, tau, Y) Note that ``X`` is given before ``C``, thus ``X`` is updated before ``C`` by default. Data ---- Now, let us generate some toy data for our model. Our true latent space is four dimensional with two noisy oscillator components, one random walk component and one white noise component. >>> w = 0.3 >>> a = np.array([[np.cos(w), -np.sin(w), 0, 0], ... [np.sin(w), np.cos(w), 0, 0], ... [0, 0, 1, 0], ... [0, 0, 0, 0]]) The true linear mapping is just random: >>> c = np.random.randn(M,4) Then, generate the latent states and the observations using the model equations: >>> x = np.empty((N,4)) >>> f = np.empty((M,N)) >>> y = np.empty((M,N)) >>> x[0] = 10*np.random.randn(4) >>> f[:,0] = np.dot(c,x[0]) >>> y[:,0] = f[:,0] + 3*np.random.randn(M) >>> for n in range(N-1): ... x[n+1] = np.dot(a,x[n]) + [1,1,10,10]*np.random.randn(4) ... f[:,n+1] = np.dot(c,x[n+1]) ... y[:,n+1] = f[:,n+1] + 3*np.random.randn(M) We want to simulate missing values, thus we create a mask which randomly removes 80% of the data: >>> from bayespy.utils import random >>> mask = random.mask(M, N, p=0.2) >>> Y.observe(y, mask=mask) Inference --------- As we did not define plotters for our nodes when creating the model, it is done now for some of the nodes: >>> import bayespy.plot as bpplt >>> X.set_plotter(bpplt.FunctionPlotter(center=True, axis=-2)) >>> A.set_plotter(bpplt.HintonPlotter()) >>> C.set_plotter(bpplt.HintonPlotter()) >>> tau.set_plotter(bpplt.PDFPlotter(np.linspace(0.02, 0.5, num=1000))) This enables plotting of the approximate posterior distributions during VB learning. The inference engine can be run using :func:`VB.update` method: >>> Q.update(repeat=10) Iteration 1: loglike=-1.439704e+05 (... seconds) ... Iteration 10: loglike=-1.051441e+04 (... seconds) The iteration progresses a bit slowly, thus we'll consider parameter expansion to speed it up. Parameter expansion +++++++++++++++++++ Section :ref:`sec-parameter-expansion` discusses parameter expansion for state-space models to speed up inference. It is based on a rotating the latent space such that the posterior in the observation space is not affected: .. math:: \mathbf{y}_n = \mathbf{C}\mathbf{x}_n = (\mathbf{C}\mathbf{R}^{-1}) (\mathbf{R}\mathbf{x}_n) \,. Thus, the transformation is :math:`\mathbf{C}\rightarrow\mathbf{C}\mathbf{R}^{-1}` and :math:`\mathbf{X}\rightarrow\mathbf{R}\mathbf{X}`. In order to keep the dynamics of the latent states unaffected by the transformation, the state dynamics matrix :math:`\mathbf{A}` must be transformed accordingly: .. math:: \mathbf{R}\mathbf{x}_n = \mathbf{R}\mathbf{A}\mathbf{R}^{-1} \mathbf{R}\mathbf{x}_{n-1} \,, resulting in a transformation :math:`\mathbf{A}\rightarrow\mathbf{R}\mathbf{A}\mathbf{R}^{-1}`. For more details, refer to :cite:`Luttinen:2013` and :cite:`Luttinen:2010`. In BayesPy, the transformations are available in :mod:`bayespy.inference.vmp.transformations`: >>> from bayespy.inference.vmp import transformations The rotation of the loading matrix along with the ARD parameters is defined as: >>> rotC = transformations.RotateGaussianARD(C, gamma) For rotating ``X``, we first need to define the rotation of the state dynamics matrix: >>> rotA = transformations.RotateGaussianARD(A, alpha) Now we can define the rotation of the latent states: >>> rotX = transformations.RotateGaussianMarkovChain(X, rotA) The optimal rotation for all these variables is found using rotation optimizer: >>> R = transformations.RotationOptimizer(rotX, rotC, D) Set the parameter expansion to be applied after each iteration: >>> Q.callback = R.rotate Now, run iterations until convergence: >>> Q.update(repeat=1000) Iteration 11: loglike=-1.010806e+04 (... seconds) ... Iteration 58: loglike=-8.906...e+03 (... seconds) Converged at iteration ... .. Iteration 60: loglike=-8.906259e+03 (... seconds) Converged at iteration 60. Results ------- Because we have set the plotters, we can plot those nodes as: >>> Q.plot(X, A, C, tau) .. plot:: import numpy as np np.random.seed(1) from bayespy.nodes import GaussianARD, GaussianMarkovChain, Gamma, Dot M = 30 N = 400 D = 10 alpha = Gamma(1e-5, 1e-5, plates=(D,), name='alpha') A = GaussianARD(0, alpha, shape=(D,), plates=(D,), name='A') X = GaussianMarkovChain(np.zeros(D), 1e-3*np.identity(D), A, np.ones(D), n=N, name='X') gamma = Gamma(1e-5, 1e-5, plates=(D,), name='gamma') C = GaussianARD(0, gamma, shape=(D,), plates=(M,1), name='C') F = Dot(C, X, name='F') C.initialize_from_random() tau = Gamma(1e-5, 1e-5, name='tau') Y = GaussianARD(F, tau, name='Y') from bayespy.inference import VB Q = VB(X, C, gamma, A, alpha, tau, Y) w = 0.3 a = np.array([[np.cos(w), -np.sin(w), 0, 0], [np.sin(w), np.cos(w), 0, 0], [0, 0, 1, 0], [0, 0, 0, 0]]) c = np.random.randn(M,4) x = np.empty((N,4)) f = np.empty((M,N)) y = np.empty((M,N)) x[0] = 10*np.random.randn(4) f[:,0] = np.dot(c,x[0]) y[:,0] = f[:,0] + 3*np.random.randn(M) for n in range(N-1): x[n+1] = np.dot(a,x[n]) + [1,1,10,10]*np.random.randn(4) f[:,n+1] = np.dot(c,x[n+1]) y[:,n+1] = f[:,n+1] + 3*np.random.randn(M) from bayespy.utils import random mask = random.mask(M, N, p=0.2) Y.observe(y, mask=mask) import bayespy.plot as bpplt X.set_plotter(bpplt.FunctionPlotter(center=True, axis=-2)) A.set_plotter(bpplt.HintonPlotter()) C.set_plotter(bpplt.HintonPlotter()) tau.set_plotter(bpplt.PDFPlotter(np.linspace(0.02, 0.5, num=1000))) Q.update(repeat=10) from bayespy.inference.vmp import transformations rotC = transformations.RotateGaussianARD(C, gamma) rotA = transformations.RotateGaussianARD(A, alpha) rotX = transformations.RotateGaussianMarkovChain(X, rotA) R = transformations.RotationOptimizer(rotX, rotC, D) Q.callback = R.rotate Q.update(repeat=1000) Q.plot(X, A, C, tau) bpplt.pyplot.show() There are clearly four effective components in ``X``: random walk (component number 1), random oscillation (7 and 10), and white noise (9). These dynamics are also visible in the state dynamics matrix Hinton diagram. Note that the white noise component does not have any dynamics. Also ``C`` shows only four effective components. The posterior of ``tau`` captures the true value :math:`3^{-2}\approx0.111` accurately. We can also plot predictions in the observation space: >>> bpplt.plot(F, center=True) .. plot:: import numpy as np np.random.seed(1) from bayespy.nodes import GaussianARD, GaussianMarkovChain, Gamma, Dot M = 30 N = 400 D = 10 alpha = Gamma(1e-5, 1e-5, plates=(D,), name='alpha') A = GaussianARD(0, alpha, shape=(D,), plates=(D,), name='A') X = GaussianMarkovChain(np.zeros(D), 1e-3*np.identity(D), A, np.ones(D), n=N, name='X') gamma = Gamma(1e-5, 1e-5, plates=(D,), name='gamma') C = GaussianARD(0, gamma, shape=(D,), plates=(M,1), name='C') F = Dot(C, X, name='F') C.initialize_from_random() tau = Gamma(1e-5, 1e-5, name='tau') Y = GaussianARD(F, tau, name='Y') from bayespy.inference import VB Q = VB(X, C, gamma, A, alpha, tau, Y) w = 0.3 a = np.array([[np.cos(w), -np.sin(w), 0, 0], [np.sin(w), np.cos(w), 0, 0], [0, 0, 1, 0], [0, 0, 0, 0]]) c = np.random.randn(M,4) x = np.empty((N,4)) f = np.empty((M,N)) y = np.empty((M,N)) x[0] = 10*np.random.randn(4) f[:,0] = np.dot(c,x[0]) y[:,0] = f[:,0] + 3*np.random.randn(M) for n in range(N-1): x[n+1] = np.dot(a,x[n]) + [1,1,10,10]*np.random.randn(4) f[:,n+1] = np.dot(c,x[n+1]) y[:,n+1] = f[:,n+1] + 3*np.random.randn(M) from bayespy.utils import random mask = random.mask(M, N, p=0.2) Y.observe(y, mask=mask) import bayespy.plot as bpplt Q.update(repeat=10) from bayespy.inference.vmp import transformations rotC = transformations.RotateGaussianARD(C, gamma) rotA = transformations.RotateGaussianARD(A, alpha) rotX = transformations.RotateGaussianMarkovChain(X, rotA) R = transformations.RotationOptimizer(rotX, rotC, D) Q.callback = R.rotate Q.update(repeat=1000) bpplt.plot(F, center=True) bpplt.pyplot.show() We can also measure the performance numerically by computing root-mean-square error (RMSE) of the missing values: >>> from bayespy.utils import misc >>> misc.rmse(y[~mask], F.get_moments()[0][~mask]) 5.18... This is relatively close to the standard deviation of the noise (3), so the predictions are quite good considering that only 20% of the data was used. bayespy-0.4.0/doc/source/user_guide/0000755000175000017500000000000012615647103021112 5ustar jluttinejluttine00000000000000bayespy-0.4.0/doc/source/user_guide/install.rst0000644000175000017500000000025512533543154023314 0ustar jluttinejluttine00000000000000.. Copyright (C) 2011,2012 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. .. include:: ../../../INSTALL.rst bayespy-0.4.0/doc/source/user_guide/user_guide.rst0000644000175000017500000000042712533543154024002 0ustar jluttinejluttine00000000000000.. Copyright (C) 2011,2012 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. .. _sec-user-guide: User guide ********** .. toctree:: install quickstart modelconstruct inference plot advanced bayespy-0.4.0/doc/source/user_guide/modelconstruct.rst0000644000175000017500000005066612533543154024726 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. .. currentmodule:: bayespy.nodes Constructing the model ====================== In BayesPy, the model is constructed by creating nodes which form a directed network. There are two types of nodes: stochastic and deterministic. A stochastic node corresponds to a random variable (or a set of random variables) from a specific probability distribution. A deterministic node corresponds to a deterministic function of its parents. For a list of built-in nodes, see the :ref:`sec-user-api`. Creating nodes -------------- Creating a node is basically like writing the conditional prior distribution of the variable in Python. The node is constructed by giving the parent nodes, that is, the conditioning variables as arguments. The number of parents and their meaning depend on the node. For instance, a :class:`Gaussian` node is created by giving the mean vector and the precision matrix. These parents can be constant numerical arrays if they are known: >>> from bayespy.nodes import Gaussian >>> X = Gaussian([2, 5], [[1.0, 0.3], [0.3, 1.0]]) or other nodes if they are unknown and given prior distributions: >>> from bayespy.nodes import Gaussian, Wishart >>> mu = Gaussian([0, 0], [[1e-6, 0],[0, 1e-6]]) >>> Lambda = Wishart(2, [[1, 0], [0, 1]]) >>> X = Gaussian(mu, Lambda) Nodes can also be named by providing ``name`` keyword argument: >>> X = Gaussian(mu, Lambda, name='x') The name may be useful when referring to the node using an inference engine. For the parent nodes, there are two main restrictions: non-constant parent nodes must be conjugate and the parent nodes must be mutually independent in the posterior approximation. Conjugacy of the parents ~~~~~~~~~~~~~~~~~~~~~~~~ In Bayesian framework in general, one can give quite arbitrary probability distributions for variables. However, one often uses distributions that are easy to handle in practice. Quite often this means that the parents are given conjugate priors. This is also one of the limitations in BayesPy: only conjugate family prior distributions are accepted currently. Thus, although in principle one could give, for instance, gamma prior for the mean parameter ``mu``, only Gaussian-family distributions are accepted because of the conjugacy. If the parent is not of a proper type, an error is raised. This conjugacy is checked automatically by BayesPy and ``NoConverterError`` is raised if a parent cannot be interpreted as being from a conjugate distribution. Independence of the parents ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Another a bit rarely encountered limitation is that the parents must be mutually independent (in the posterior factorization). Thus, a node cannot have the same stochastic node as several parents without intermediate stochastic nodes. For instance, the following leads to an error: >>> from bayespy.nodes import Dot >>> Y = Dot(X, X) Traceback (most recent call last): ... ValueError: Parent nodes are not independent The error is raised because ``X`` is given as two parents for ``Y``, and obviously ``X`` is not independent of ``X`` in the posterior approximation. Even if ``X`` is not given several times directly but there are some intermediate deterministic nodes, an error is raised because the deterministic nodes depend on their parents and thus the parents of ``Y`` would not be independent. However, it is valid that a node is a parent of another node via several paths if all the paths or all except one path has intermediate stochastic nodes. This is valid because the intermediate stochastic nodes have independent posterior approximations. Thus, for instance, the following construction does not raise errors: >>> from bayespy.nodes import Dot >>> Z = Gaussian(X, [[1,0], [0,1]]) >>> Y = Dot(X, Z) This works because there is now an intermediate stochastic node ``Z`` on the other path from ``X`` node to ``Y`` node. Effects of the nodes on inference --------------------------------- When constructing the network with nodes, the stochastic nodes actually define three important aspects: 1. the prior probability distribution for the variables, 2. the factorization of the posterior approximation, 3. the functional form of the posterior approximation for the variables. Prior probability distribution ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ First, the most intuitive feature of the nodes is that they define the prior distribution. In the previous example, ``mu`` was a stochastic :class:`GaussianARD` node corresponding to :math:`\mu` from the normal distribution, ``tau`` was a stochastic :class:`Gamma` node corresponding to :math:`\tau` from the gamma distribution, and ``y`` was a stochastic :class:`GaussianARD` node corresponding to :math:`y` from the normal distribution with mean :math:`\mu` and precision :math:`\tau`. If we denote the set of all stochastic nodes by :math:`\Omega`, and by :math:`\pi_X` the set of parents of a node :math:`X`, the model is defined as .. math:: p(\Omega) = \prod_{X \in \Omega} p(X|\pi_X), where nodes correspond to the terms :math:`p(X|\pi_X)`\ . Posterior factorization ~~~~~~~~~~~~~~~~~~~~~~~ Second, the nodes define the structure of the posterior approximation. The variational Bayesian approximation factorizes with respect to nodes, that is, each node corresponds to an independent probability distribution in the posterior approximation. In the previous example, ``mu`` and ``tau`` were separate nodes, thus the posterior approximation factorizes with respect to them: :math:`q(\mu)q(\tau)`\ . Thus, the posterior approximation can be written as: .. math:: p(\tilde{\Omega}|\hat{\Omega}) \approx \prod_{X \in \tilde{\Omega}} q(X), where :math:`\tilde{\Omega}` is the set of latent stochastic nodes and :math:`\hat{\Omega}` is the set of observed stochastic nodes. Sometimes one may want to avoid the factorization between some variables. For this purpose, there are some nodes which model several variables jointly without factorization. For instance, :class:`GaussianGammaISO` is a joint node for :math:`\mu` and :math:`\tau` variables from the normal-gamma distribution and the posterior approximation does not factorize between :math:`\mu` and :math:`\tau`, that is, the posterior approximation is :math:`q(\mu,\tau)`. Functional form of the posterior ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Last, the nodes define the functional form of the posterior approximation. Usually, the posterior approximation has the same or similar functional form as the prior. For instance, :class:`Gamma` uses gamma distribution to also approximate the posterior distribution. Similarly, :class:`GaussianARD` uses Gaussian distribution for the posterior. However, the posterior approximation of :class:`GaussianARD` uses a full covariance matrix although the prior assumes a diagonal covariance matrix. Thus, there can be slight differences in the exact functional form of the posterior approximation but the rule of thumb is that the functional form of the posterior approximation is the same as or more general than the functional form of the prior. Using plate notation -------------------- Defining plates ~~~~~~~~~~~~~~~ Stochastic nodes take the optional parameter ``plates``, which can be used to define plates of the variable. A plate defines the number of repetitions of a set of variables. For instance, a set of random variables :math:`\mathbf{y}_{mn}` could be defined as .. math:: \mathbf{y}_{mn} \sim \mathcal{N}(\boldsymbol{\mu}, \mathbf{\Lambda}),\qquad m=0,\ldots,9, \quad n=0,\ldots,29. This can also be visualized as a graphical model: .. bayesnet:: \node[latent] (y) {$\mathbf{y}_{mn}$} ; \node[latent, above left=1.8 and 0.4 of y] (mu) {$\boldsymbol{\mu}$} ; \node[latent, above right=1.8 and 0.4 of y] (Lambda) {$\mathbf{\Lambda}$} ; \factor[above=of y] {y-f} {left:$\mathcal{N}$} {mu,Lambda} {y}; \plate {m-plate} {(y)(y-f)(y-f-caption)} {$m=0,\ldots,9$} ; \plate {n-plate} {(m-plate)(m-plate-caption)} {$n=0,\ldots,29$} ; The variable has two plates: one for the index :math:`m` and one for the index :math:`n`\ . In BayesPy, this random variable can be constructed as: >>> y = Gaussian(mu, Lambda, plates=(10,30)) .. note:: The plates are always given as a tuple of positive integers. Plates also define indexing for the nodes, thus you can use simple NumPy-style slice indexing to obtain a subset of the plates: >>> y_0 = y[0] >>> y_0.plates (30,) >>> y_even = y[:,::2] >>> y_even.plates (10, 15) >>> y_complex = y[:5, 10:20:5] >>> y_complex.plates (5, 2) Note that this indexing is for the plates only, not for the random variable dimensions. Sharing and broadcasting plates ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Instead of having a common mean and precision matrix for all :math:`\mathbf{y}_{mn}`\ , it is also possible to share plates with parents. For instance, the mean could be different for each index :math:`m` and the precision for each index :math:`n`\ : .. math:: \mathbf{y}_{mn} \sim \mathcal{N}(\boldsymbol{\mu}_m, \mathbf{\Lambda}_n),\qquad m=0,\ldots,9, \quad n=0,\ldots,29. which has the following graphical representation: .. bayesnet:: \node[latent] (y) {$\mathbf{y}_{mn}$} ; \node[latent, above left=1 and 2 of y] (mu) {$\boldsymbol{\mu}_m$} ; \node[latent, above right=1 and 1 of y] (Lambda) {$\mathbf{\Lambda}_n$} ; \factor[above=of y] {y-f} {above:$\mathcal{N}$} {mu,Lambda} {y}; \plate {m-plate} {(mu)(y)(y-f)(y-f-caption)} {$m=0,\ldots,9$} ; \plate {n-plate} {(Lambda)(y)(y-f)(y-f-caption)(m-plate-caption)(m-plate.north east)} {$n=0,\ldots,29$} ; This can be constructed in BayesPy, for instance, as: >>> from bayespy.nodes import Gaussian, Wishart >>> mu = Gaussian([0, 0], [[1e-6, 0],[0, 1e-6]], plates=(10,1)) >>> Lambda = Wishart(2, [[1, 0], [0, 1]], plates=(1,30)) >>> X = Gaussian(mu, Lambda) There are a few things to notice here. First, the plates are defined similarly as shapes in NumPy, that is, they use similar broadcasting rules. For instance, the plates ``(10,1)`` and ``(1,30)`` broadcast to ``(10,30)``. In fact, one could use plates ``(10,1)`` and ``(30,)`` to get the broadcasted plates ``(10,30)`` because broadcasting compares the plates from right to left starting from the last axis. Second, ``X`` is not given ``plates`` keyword argument because the default plates are the plates broadcasted from the parents and that was what we wanted so it was not necessary to provide the keyword argument. If we wanted, for instance, plates ``(20,10,30)`` for ``X``, then we would have needed to provide ``plates=(20,10,30)``. The validity of the plates between a child and its parents is checked as follows. The plates are compared plate-wise starting from the last axis and working the way forward. A plate of the child is compatible with a plate of the parent if either of the following conditions is met: 1. The two plates have equal size 2. The parent has size 1 (or no plate) Table below shows an example of compatible plates for a child node and its two parent nodes: +---------+----------------------------+ | node | plates | +=========+===+===+===+===+===+===+====+ | parent1 | | 3 | 1 | 1 | 1 | 8 | 10 | +---------+---+---+---+---+---+---+----+ | parent2 | | | 1 | 1 | 5 | 1 | 10 | +---------+---+---+---+---+---+---+----+ | child | 5 | 3 | 1 | 7 | 5 | 8 | 10 | +---------+---+---+---+---+---+---+----+ Plates in deterministic nodes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Note that plates can be defined explicitly only for stochastic nodes. For deterministic nodes, the plates are defined implicitly by the plate broadcasting rules from the parents. Deterministic nodes do not need more plates than this because there is no randomness. The deterministic node would just have the same value over the extra plates, but it is not necessary to do this explicitly because the child nodes of the deterministic node can utilize broadcasting anyway. Thus, there is no point in having extra plates in deterministic nodes, and for this reason, deterministic nodes do not use ``plates`` keyword argument. Plates in constants ~~~~~~~~~~~~~~~~~~~ It is useful to understand how the plates and the shape of a random variable are connected. The shape of an array which contains all the plates of a random variable is the concatenation of the plates and the shape of the variable. For instance, consider a 2-dimensional Gaussian variable with plates ``(3,)``. If you want the value of the constant mean vector and constant precision matrix to vary between plates, they are given as ``(3,2)``-shape and ``(3,2,2)``-shape arrays, respectively: >>> import numpy as np >>> mu = [ [0,0], [1,1], [2,2] ] >>> Lambda = [ [[1.0, 0.0], ... [0.0, 1.0]], ... [[1.0, 0.9], ... [0.9, 1.0]], ... [[1.0, -0.3], ... [-0.3, 1.0]] ] >>> X = Gaussian(mu, Lambda) >>> np.shape(mu) (3, 2) >>> np.shape(Lambda) (3, 2, 2) >>> X.plates (3,) Thus, the leading axes of an array are the plate axes and the trailing axes are the random variable axes. In the example above, the mean vector has plates ``(3,)`` and shape ``(2,)``, and the precision matrix has plates ``(3,)`` and shape ``(2,2)``. Factorization of plates ~~~~~~~~~~~~~~~~~~~~~~~ It is important to undestand the independency structure the plates induce for the model. First, the repetitions defined by a plate are independent a priori given the parents. Second, the repetitions are independent in the posterior approximation, that is, the posterior approximation factorizes with respect to plates. Thus, the plates also have an effect on the independence structure of the posterior approximation, not only prior. If dependencies between a set of variables need to be handled, that set must be handled as a some kind of multi-dimensional variable. .. _sec-irregular-plates: Irregular plates ~~~~~~~~~~~~~~~~ The handling of plates is not always as simple as described above. There are cases in which the plates of the parents do not map directly to the plates of the child node. The user API should mention such irregularities. For instance, the parents of a mixture distribution have a plate which contains the different parameters for each cluster, but the variable from the mixture distribution does not have that plate: >>> from bayespy.nodes import Gaussian, Wishart, Categorical, Mixture >>> mu = Gaussian([[0], [0], [0]], [ [[1]], [[1]], [[1]] ]) >>> Lambda = Wishart(1, [ [[1]], [[1]], [[1]]]) >>> Z = Categorical([1/3, 1/3, 1/3], plates=(100,)) >>> X = Mixture(Z, Gaussian, mu, Lambda) >>> mu.plates (3,) >>> Lambda.plates (3,) >>> Z.plates (100,) >>> X.plates (100,) The plates ``(3,)`` and ``(100,)`` should not broadcast according to the rules mentioned above. However, when validating the plates, :class:`Mixture` removes the plate which corresponds to the clusters in ``mu`` and ``Lambda``. Thus, ``X`` has plates which are the result of broadcasting plates ``()`` and ``(100,)`` which equals ``(100,)``. Also, sometimes the plates of the parents may be mapped to the variable axes. For instance, an automatic relevance determination (ARD) prior for a Gaussian variable is constructed by giving the diagonal elements of the precision matrix (or tensor). The Gaussian variable itself can be a scalar, a vector, a matrix or a tensor. A set of five :math:`4 \times 3` -dimensional Gaussian matrices with ARD prior is constructed as: >>> from bayespy.nodes import GaussianARD, Gamma >>> tau = Gamma(1, 1, plates=(5,4,3)) >>> X = GaussianARD(0, tau, shape=(4,3)) >>> tau.plates (5, 4, 3) >>> X.plates (5,) Note how the last two plate axes of ``tau`` are mapped to the variable axes of ``X`` with shape ``(4,3)`` and the plates of ``X`` are obtained by taking the remaining leading plate axes of ``tau``. Example model: Principal component analysis ------------------------------------------- Now, we'll construct a bit more complex model which will be used in the following sections. The model is a probabilistic version of principal component analysis (PCA): .. math:: \mathbf{Y} = \mathbf{C}\mathbf{X}^T + \mathrm{noise} where :math:`\mathbf{Y}` is :math:`M\times N` data matrix, :math:`\mathbf{C}` is :math:`M\times D` loading matrix, :math:`\mathbf{X}` is :math:`N\times D` state matrix, and noise is isotropic Gaussian. The dimensionality :math:`D` is usually assumed to be much smaller than :math:`M` and :math:`N`. A probabilistic formulation can be written as: .. math:: p(\mathbf{Y}) &= \prod^{M-1}_{m=0} \prod^{N-1}_{n=0} \mathcal{N}(y_{mn} | \mathbf{c}_m^T \mathbf{x}_n, \tau) \\ p(\mathbf{X}) &= \prod^{N-1}_{n=0} \prod^{D-1}_{d=0} \mathcal{N}(x_{nd} | 0, 1) \\ p(\mathbf{C}) &= \prod^{M-1}_{m=0} \prod^{D-1}_{d=0} \mathcal{N}(c_{md} | 0, \alpha_d) \\ p(\boldsymbol{\alpha}) &= \prod^{D-1}_{d=0} \mathcal{G} (\alpha_d | 10^{-3}, 10^{-3}) \\ p(\tau) &= \mathcal{G} (\tau | 10^{-3}, 10^{-3}) where we have given automatic relevance determination (ARD) prior for :math:`\mathbf{C}`. This can be visualized as a graphical model: .. bayesnet:: \node[latent] (y) {$\mathbf{y}_{mn}$} ; \node[det, above=of y] (dot) {dot} ; \node[latent, right=2 of dot] (tau) {$\tau$} ; \node[latent, above left=1 and 2 of dot] (C) {$c_{md}$} ; \node[latent, above=of C] (alpha) {$\alpha_d$} ; \node[latent, above right=1 and 1 of dot] (X) {$x_{nd}$} ; \factor[above=of y] {y-f} {left:$\mathcal{N}$} {dot,tau} {y}; \factor[above=of C] {C-f} {left:$\mathcal{N}$} {alpha} {C}; \factor[above=of X] {X-f} {above:$\mathcal{N}$} {} {X}; \factor[above=of alpha] {alpha-f} {above:$\mathcal{G}$} {} {alpha}; \factor[above=of tau] {tau-f} {above:$\mathcal{G}$} {} {tau}; \edge {C,X} {dot}; \tikzstyle{plate caption} += [below left=0pt and 0pt of #1.north east] ; \plate {d-plate} {(X)(X-f)(X-f-caption)(C)(C-f)(C-f-caption)(alpha)(alpha-f)(alpha-f-caption)} {$d=0,\ldots,2$} ; \tikzstyle{plate caption} += [below left=5pt and 0pt of #1.south east] ; \plate {m-plate} {(y)(y-f)(y-f-caption)(C)(C-f)(C-f-caption)(d-plate.south west)} {$m=0,\ldots,9$} ; \plate {n-plate} {(y)(y-f)(y-f-caption)(X)(X-f)(X-f-caption)(m-plate-caption)(m-plate.north east)(d-plate.south east)} {$n=0,\ldots,99$} ; Now, let us construct this model in BayesPy. First, we'll define the dimensionality of the latent space in our model: >>> D = 3 Then the prior for the latent states :math:`\mathbf{X}`: >>> X = GaussianARD(0, 1, ... shape=(D,), ... plates=(1,100), ... name='X') Note that the shape of ``X`` is ``(D,)``, although the latent dimensions are marked with a plate in the graphical model and they are conditionally independent in the prior. However, we want to (and need to) model the posterior dependency of the latent dimensions, thus we cannot factorize them, which would happen if we used ``plates=(1,100,D)`` and ``shape=()``. The first plate axis with size 1 is given just for clarity. The prior for the ARD parameters :math:`\boldsymbol{\alpha}` of the loading matrix: >>> alpha = Gamma(1e-3, 1e-3, ... plates=(D,), ... name='alpha') The prior for the loading matrix :math:`\mathbf{C}`: >>> C = GaussianARD(0, alpha, ... shape=(D,), ... plates=(10,1), ... name='C') Again, note that the shape is the same as for ``X`` for the same reason. Also, the plates of ``alpha``, ``(D,)``, are mapped to the full shape of the node ``C``, ``(10,1,D)``, using standard broadcasting rules. The dot product is just a deterministic node: >>> F = Dot(C, X) However, note that ``Dot`` requires that the input Gaussian nodes have the same shape and that this shape has exactly one axis, that is, the variables are vectors. This the reason why we used shape ``(D,)`` for ``X`` and ``C`` but from a bit different perspective. The node computes the inner product of :math:`D`-dimensional vectors resulting in plates ``(10,100)`` broadcasted from the plates ``(1,100)`` and ``(10,1)``: >>> F.plates (10, 100) The prior for the observation noise :math:`\tau`: >>> tau = Gamma(1e-3, 1e-3, name='tau') Finally, the observations are conditionally independent Gaussian scalars: >>> Y = GaussianARD(F, tau, name='Y') Now we have defined our model and the next step is to observe some data and to perform inference. bayespy-0.4.0/doc/source/user_guide/inference.rst0000644000175000017500000003367512533543154023620 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. .. testsetup:: # This is the PCA model from the previous section import numpy as np np.random.seed(1) from bayespy.nodes import GaussianARD, Gamma, Dot D = 3 X = GaussianARD(0, 1, shape=(D,), plates=(1,100), name='X') alpha = Gamma(1e-3, 1e-3, plates=(D,), name='alpha') C = GaussianARD(0, alpha, shape=(D,), plates=(10,1), name='C') F = Dot(C, X) tau = Gamma(1e-3, 1e-3, name='tau') Y = GaussianARD(F, tau) Performing inference ==================== Approximation of the posterior distribution can be divided into several steps: - Observe some nodes - Choose the inference engine - Initialize the posterior approximation - Run the inference algorithm In order to illustrate these steps, we'll be using the PCA model constructed in the previous section. Observing nodes --------------- First, let us generate some toy data: >>> c = np.random.randn(10, 2) >>> x = np.random.randn(2, 100) >>> data = np.dot(c, x) + 0.1*np.random.randn(10, 100) The data is provided by simply calling ``observe`` method of a stochastic node: >>> Y.observe(data) It is important that the shape of the ``data`` array matches the plates and shape of the node ``Y``. For instance, if ``Y`` was :class:`Wishart` node for :math:`3\times 3` matrices with plates ``(5,1,10)``, the full shape of ``Y`` would be ``(5,1,10,3,3)``. The ``data`` array should have this shape exactly, that is, no broadcasting rules are applied. Missing values ++++++++++++++ It is possible to mark missing values by providing a mask which is a boolean array: >>> Y.observe(data, mask=[[True], [False], [False], [True], [True], ... [False], [True], [True], [True], [False]]) ``True`` means that the value is observed and ``False`` means that the value is missing. The shape of the above mask is ``(10,1)``, which broadcasts to the plates of Y, ``(10,100)``. Thus, the above mask means that the second, third, sixth and tenth rows of the :math:`10\times 100` data matrix are missing. The mask is applied to the *plates*, not to the data array directly. This means that it is not possible to observe a random variable partially, each repetition defined by the plates is either fully observed or fully missing. Thus, the mask is applied to the plates. It is often possible to circumvent this seemingly tight restriction by adding an observable child node which factorizes more. The shape of the mask is broadcasted to plates using standard NumPy broadcasting rules. So, if the variable has plates ``(5,1,10)``, the mask could have a shape ``()``, ``(1,)``, ``(1,1)``, ``(1,1,1)``, ``(10,)``, ``(1,10)``, ``(1,1,10)``, ``(5,1,1)`` or ``(5,1,10)``. In order to speed up the inference, missing values are automatically integrated out if they are not needed as latent variables to child nodes. This leads to faster convergence and more accurate approximations. Choosing the inference method ----------------------------- Inference methods can be found in :mod:`bayespy.inference` package. Currently, only variational Bayesian approximation is implemented (:class:`bayespy.inference.VB`). The inference engine is constructed by giving the stochastic nodes of the model. >>> from bayespy.inference import VB >>> Q = VB(Y, C, X, alpha, tau) There is no need to give any deterministic nodes. Currently, the inference engine does not automatically search for stochastic parents and children, thus it is important that all stochastic nodes of the model are given. This should be made more robust in future versions. A node of the model can be obtained by using the name of the node as a key: >>> Q['X'] Note that the returned object is the same as the node object itself: >>> Q['X'] is X True Thus, one may use the object ``X`` when it is available. However, if the model and the inference engine are constructed in another function or module, the node object may not be available directly and this feature becomes useful. Initializing the posterior approximation ---------------------------------------- The inference engines give some initialization to the stochastic nodes by default. However, the inference algorithms can be sensitive to the initialization, thus it is sometimes necessary to have better control over the initialization. For VB, the following initialization methods are available: - ``initialize_from_prior``: Use the current states of the parent nodes to update the node. This is the default initialization. - ``initialize_from_parameters``: Use the given parameter values for the distribution. - ``initialize_from_value``: Use the given value for the variable. - ``initialize_from_random``: Draw a random value for the variable. The random sample is drawn from the current state of the node's distribution. Note that ``initialize_from_value`` and ``initialize_from_random`` initialize the distribution with a value of the variable instead of parameters of the distribution. Thus, the distribution is actually a delta distribution with a peak on the value after the initialization. This state of the distribution does not have proper natural parameter values nor normalization, thus the VB lower bound terms are ``np.nan`` for this initial state. These initialization methods can be used to perform even a bit more complex initializations. For instance, a Gaussian distribution could be initialized with a random mean and variance 0.1. In our PCA model, this can be obtained by >>> X.initialize_from_parameters(np.random.randn(1, 100, D), 10) Note that the shape of the random mean is the sum of the plates ``(1, 100)`` and the variable shape ``(D,)``. In addition, instead of variance, :class:`GaussianARD` uses precision as the second parameter, thus we initialized the variance to :math:`\frac{1}{10}`. This random initialization is important in our PCA model because the default initialization gives ``C`` and ``X`` zero mean. If the mean of the other variable was zero when the other is updated, the other variable gets zero mean too. This would lead to an update algorithm where both means remain zeros and effectively no latent space is found. Thus, it is important to give non-zero random initialization for ``X`` if ``C`` is updated before ``X`` the first time. It is typical that at least some nodes need be initialized with some randomness. By default, nodes are initialized with the method ``initialize_from_prior``. The method is not very time consuming but if for any reason you want to avoid that default initialization computation, you can provide ``initialize=False`` when creating the stochastic node. However, the node does not have a proper state in that case, which leads to errors in VB learning unless the distribution is initialized using the above methods. Running the inference algorithm ------------------------------- The approximation methods are based on iterative algorithms, which can be run using ``update`` method. By default, it takes one iteration step updating all nodes once: >>> Q.update() Iteration 1: loglike=-9.305259e+02 (... seconds) The ``loglike`` tells the VB lower bound. The order in which the nodes are updated is the same as the order in which the nodes were given when creating ``Q``. If you want to change the order or update only some of the nodes, you can give as arguments the nodes you want to update and they are updated in the given order: >>> Q.update(C, X) Iteration 2: loglike=-8.818976e+02 (... seconds) It is also possible to give the same node several times: >>> Q.update(C, X, C, tau) Iteration 3: loglike=-8.071222e+02 (... seconds) Note that each call to ``update`` is counted as one iteration step although not variables are necessarily updated. Instead of doing one iteration step, ``repeat`` keyword argument can be used to perform several iteration steps: >>> Q.update(repeat=10) Iteration 4: loglike=-7.167588e+02 (... seconds) Iteration 5: loglike=-6.827873e+02 (... seconds) Iteration 6: loglike=-6.259477e+02 (... seconds) Iteration 7: loglike=-4.725400e+02 (... seconds) Iteration 8: loglike=-3.270816e+02 (... seconds) Iteration 9: loglike=-2.208865e+02 (... seconds) Iteration 10: loglike=-1.658761e+02 (... seconds) Iteration 11: loglike=-1.469468e+02 (... seconds) Iteration 12: loglike=-1.420311e+02 (... seconds) Iteration 13: loglike=-1.405139e+02 (... seconds) The VB algorithm stops automatically if it converges, that is, the relative change in the lower bound is below some threshold: >>> Q.update(repeat=1000) Iteration 14: loglike=-1.396481e+02 (... seconds) ... Iteration 488: loglike=-1.224106e+02 (... seconds) Converged at iteration 488. Now the algorithm stopped before taking 1000 iteration steps because it converged. The relative tolerance can be adjusted by providing ``tol`` keyword argument to the ``update`` method: >>> Q.update(repeat=10000, tol=1e-6) Iteration 489: loglike=-1.224094e+02 (... seconds) ... Iteration 847: loglike=-1.222506e+02 (... seconds) Converged at iteration 847. Making the tolerance smaller, may improve the result but it may also significantly increase the iteration steps until convergence. Instead of using ``update`` method of the inference engine ``VB``, it is possible to use the ``update`` methods of the nodes directly as >>> C.update() or >>> Q['C'].update() However, this is not recommended, because the ``update`` method of the inference engine ``VB`` is a wrapper which, in addition to calling the nodes' ``update`` methods, checks for convergence and does a few other useful minor things. But if for any reason these direct update methods are needed, they can be used. .. _sec-parameter-expansion: Parameter expansion +++++++++++++++++++ Sometimes the VB algorithm converges very slowly. This may happen when the variables are strongly coupled in the true posterior but factorized in the approximate posterior. This coupling leads to zigzagging of the variational parameters which progresses slowly. One solution to this problem is to use parameter expansion. The idea is to add an auxiliary variable which parameterizes the posterior approximation of several variables. Then optimizing this auxiliary variable actually optimizes several posterior approximations jointly leading to faster convergence. The parameter expansion is model specific. Currently in BayesPy, only state-space models have built-in parameter expansions available. These state-space models contain a variable which is a dot product of two variables (plus some noise): .. math:: y = \mathbf{c}^T\mathbf{x} + \mathrm{noise} The parameter expansion can be motivated by noticing that we can add an auxiliary variable which rotates the variables :math:`\mathbf{c}` and :math:`\mathbf{x}` so that the dot product is unaffected: .. math:: y &= \mathbf{c}^T\mathbf{x} + \mathrm{noise} = \mathbf{c}^T \mathbf{R} \mathbf{R}^{-1}\mathbf{x} + \mathrm{noise} = (\mathbf{R}^T\mathbf{c})^T(\mathbf{R}^{-1}\mathbf{x}) + \mathrm{noise} Now, applying this rotation to the posterior approximations :math:`q(\mathbf{c})` and :math:`q(\mathbf{x})`, and optimizing the VB lower bound with respect to the rotation leads to parameterized joint optimization of :math:`\mathbf{c}` and :math:`\mathbf{x}`. The available parameter expansion methods are in module ``transformations``: >>> from bayespy.inference.vmp import transformations First, you create the rotation transformations for the two variables: >>> rotX = transformations.RotateGaussianARD(X) >>> rotC = transformations.RotateGaussianARD(C, alpha) .. currentmodule:: bayespy.inference.vmp.transformations Here, the rotation for ``C`` provides the ARD parameters ``alpha`` so they are updated simultaneously. In addition to :class:`RotateGaussianARD`, there are a few other built-in rotations defined, for instance, :class:`RotateGaussian` and :class:`RotateGaussianMarkovChain`. It is extremely important that the model satisfies the assumptions made by the rotation class and the user is mostly responsible for this. The optimizer for the rotations is constructed by giving the two rotations and the dimensionality of the rotated space: .. currentmodule:: bayespy.nodes >>> R = transformations.RotationOptimizer(rotC, rotX, D) Now, calling ``rotate`` method will find optimal rotation and update the relevant nodes (``X``, ``C`` and ``alpha``) accordingly: >>> R.rotate() Let us see how our iteration would have gone if we had used this parameter expansion. First, let us re-initialize our nodes and VB algorithm: >>> alpha.initialize_from_prior() >>> C.initialize_from_prior() >>> X.initialize_from_parameters(np.random.randn(1, 100, D), 10) >>> tau.initialize_from_prior() >>> Q = VB(Y, C, X, alpha, tau) Then, the rotation is set to run after each iteration step: >>> Q.callback = R.rotate Now the iteration converges to the relative tolerance :math:`10^{-6}` much faster: >>> Q.update(repeat=1000, tol=1e-6) Iteration 1: loglike=-9.363500e+02 (... seconds) ... Iteration 18: loglike=-1.221354e+02 (... seconds) Converged at iteration 18. The convergence took 18 iterations with rotations and 488 or 847 iterations without the parameter expansion. In addition, the lower bound is improved slightly. One can compare the number of iteration steps in this case because the cost per iteration step with or without parameter expansion is approximately the same. Sometimes the parameter expansion can have the drawback that it converges to a bad local optimum. Usually, this can be solved by updating the nodes near the observations a few times before starting to update the hyperparameters and to use parameter expansion. In any case, the parameter expansion is practically necessary when using state-space models in order to converge to a proper solution in a reasonable time. bayespy-0.4.0/doc/source/user_guide/quickstartbackup.rst0000644000175000017500000001171412533543154025230 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. Quick start guide ================= This short guide shows the key steps in using BayesPy for variational Bayesian inference by applying BayesPy to a simple problem. The key steps in using BayesPy are the following: - Construct the model - Observe some of the variables by providing the data in a proper format - Run variational Bayesian inference - Examine the resulting posterior approximation To demonstrate BayesPy, we'll consider a very simple problem: we have a set of observations from a Gaussian distribution with unknown mean and variance, and we want to learn these parameters. In this case, we do not use any real-world data but generate some artificial data. The dataset consists of ten samples from a Gaussian distribution with mean 5 and standard deviation 10. This dataset can be generated with NumPy as follows: .. code:: python import numpy as np data = np.random.normal(5, 10, size=(10,)) Now, given this data we would like to estimate the mean and the standard deviation as if we didn't know their values. The model can be defined as follows: .. math:: \begin{split} p(\mathbf{y}|\mu,\tau) &= \prod^{9}_{n=0} \mathcal{N}(y_n|\mu,\tau) \\ p(\mu) &= \mathcal{N}(\mu|0,10^{-6}) \\ p(\tau) &= \mathcal{G}(\tau|10^{-6},10^{-6}) \end{split} where :math:`\mathcal{N}` is the Gaussian distribution parameterized by its mean and precision (i.e., inverse variance), and :math:`\mathcal{G}` is the gamma distribution parameterized by its shape and rate parameters. Note that we have given quite uninformative priors for the variables :math:`\mu` and :math:`\tau`\ . This simple model can also be shown as a directed factor graph: .. bayesnet:: Directed factor graph of the example model. \node[obs] (y) {$y_n$} ; \node[latent, above left=1.5 and 0.5 of y] (mu) {$\mu$} ; \node[latent, above right=1.5 and 0.5 of y] (tau) {$\tau$} ; \factor[above=of mu] {mu-f} {left:$\mathcal{N}$} {} {mu} ; \factor[above=of tau] {tau-f} {left:$\mathcal{G}$} {} {tau} ; \factor[above=of y] {y-f} {left:$\mathcal{N}$} {mu,tau} {y}; \plate {} {(y)(y-f)(y-f-caption)} {$n=0,\ldots,9$} ; This model can be constructed in BayesPy as follows: .. code:: python from bayespy.nodes import GaussianARD, Gamma mu = GaussianARD(0, 1e-6) tau = Gamma(1e-6, 1e-6) y = GaussianARD(mu, tau, plates=(10,)) .. currentmodule:: bayespy.nodes This is quite self-explanatory given the model definitions above. We have used two types of nodes :class:`GaussianARD` and :class:`Gamma` to represent Gaussian and gamma distributions, respectively. There are much more distributions in :mod:`bayespy.nodes` so you can construct quite complex conjugate exponential family models. The node :code:`y` uses keyword argument :code:`plates` to define the plates :math:`n=0,\ldots,9`. Now that we have created the model, we can provide our data by setting :code:`y` as observed: .. code:: python y.observe(data) Next we want to estimate the posterior distribution. In principle, we could use different inference engines (e.g., MCMC or EP) but currently only variational Bayesian (VB) engine is implemented. The engine is initialized by giving all the nodes of the model: .. code:: python from bayespy.inference import VB Q = VB(mu, tau, y) The inference algorithm can be run as long as wanted (max. 20 iterations in this case): .. code:: python Q.update(repeat=20) .. parsed-literal:: Iteration 1: loglike=-5.789562e+01 (0.010 seconds) Iteration 2: loglike=-5.612083e+01 (0.000 seconds) Iteration 3: loglike=-5.611848e+01 (0.010 seconds) Iteration 4: loglike=-5.611846e+01 (0.000 seconds) Converged. Now the algorithm converged after four iterations, before the requested 20 iterations. VB approximates the true posterior :math:`p(\mu,\tau|\mathbf{y})` with a distribution which factorizes with respect to the nodes: :math:`q(\mu)q(\tau)`\ . The resulting approximate posterior distributions :math:`q(\mu)` and :math:`q(\tau)` can be examined, for instance, by plotting the marginal probability density functions: .. code:: python import bayespy.plot as bpplt # The following two two lines are just for enabling matplotlib plotting in notebooks %matplotlib inline bpplt.pyplot.plot([]) bpplt.pyplot.subplot(2, 1, 1) bpplt.pdf(mu, np.linspace(-10, 20, num=100), color='k', name=r'\mu') bpplt.pyplot.subplot(2, 1, 2) bpplt.pdf(tau, np.linspace(1e-6, 0.08, num=100), color='k', name=r'\tau'); .. image:: quickstartbackup_files/quickstartbackup_14_0.png This example was a very simple introduction to using BayesPy. The model can be much more complex and each phase contains more options to give the user more control over the inference. The following sections give more details about the phases. bayespy-0.4.0/doc/source/user_guide/quickstart.rst0000644000175000017500000001341112533543154024036 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. .. testsetup:: from matplotlib import pyplot pyplot.ion() import numpy numpy.random.seed(1) Quick start guide ================= This short guide shows the key steps in using BayesPy for variational Bayesian inference by applying BayesPy to a simple problem. The key steps in using BayesPy are the following: - Construct the model - Observe some of the variables by providing the data in a proper format - Run variational Bayesian inference - Examine the resulting posterior approximation To demonstrate BayesPy, we'll consider a very simple problem: we have a set of observations from a Gaussian distribution with unknown mean and variance, and we want to learn these parameters. In this case, we do not use any real-world data but generate some artificial data. The dataset consists of ten samples from a Gaussian distribution with mean 5 and standard deviation 10. This dataset can be generated with NumPy as follows: >>> import numpy as np >>> data = np.random.normal(5, 10, size=(10,)) Constructing the model ---------------------- Now, given this data we would like to estimate the mean and the standard deviation as if we didn't know their values. The model can be defined as follows: .. math:: \begin{split} p(\mathbf{y}|\mu,\tau) &= \prod^{9}_{n=0} \mathcal{N}(y_n|\mu,\tau) \\ p(\mu) &= \mathcal{N}(\mu|0,10^{-6}) \\ p(\tau) &= \mathcal{G}(\tau|10^{-6},10^{-6}) \end{split} where :math:`\mathcal{N}` is the Gaussian distribution parameterized by its mean and precision (i.e., inverse variance), and :math:`\mathcal{G}` is the gamma distribution parameterized by its shape and rate parameters. Note that we have given quite uninformative priors for the variables :math:`\mu` and :math:`\tau`\ . This simple model can also be shown as a directed factor graph: .. bayesnet:: Directed factor graph of the example model. \node[obs] (y) {$y_n$} ; \node[latent, above left=1.5 and 0.5 of y] (mu) {$\mu$} ; \node[latent, above right=1.5 and 0.5 of y] (tau) {$\tau$} ; \factor[above=of mu] {mu-f} {left:$\mathcal{N}$} {} {mu} ; \factor[above=of tau] {tau-f} {left:$\mathcal{G}$} {} {tau} ; \factor[above=of y] {y-f} {left:$\mathcal{N}$} {mu,tau} {y}; \plate {} {(y)(y-f)(y-f-caption)} {$n=0,\ldots,9$} ; This model can be constructed in BayesPy as follows: >>> from bayespy.nodes import GaussianARD, Gamma >>> mu = GaussianARD(0, 1e-6) >>> tau = Gamma(1e-6, 1e-6) >>> y = GaussianARD(mu, tau, plates=(10,)) .. currentmodule:: bayespy.nodes This is quite self-explanatory given the model definitions above. We have used two types of nodes :class:`GaussianARD` and :class:`Gamma` to represent Gaussian and gamma distributions, respectively. There are much more distributions in :mod:`bayespy.nodes` so you can construct quite complex conjugate exponential family models. The node :code:`y` uses keyword argument :code:`plates` to define the plates :math:`n=0,\ldots,9`. Performing inference -------------------- Now that we have created the model, we can provide our data by setting ``y`` as observed: >>> y.observe(data) Next we want to estimate the posterior distribution. In principle, we could use different inference engines (e.g., MCMC or EP) but currently only variational Bayesian (VB) engine is implemented. The engine is initialized by giving all the nodes of the model: >>> from bayespy.inference import VB >>> Q = VB(mu, tau, y) The inference algorithm can be run as long as wanted (max. 20 iterations in this case): >>> Q.update(repeat=20) Iteration 1: loglike=-6.020956e+01 (... seconds) Iteration 2: loglike=-5.820527e+01 (... seconds) Iteration 3: loglike=-5.820290e+01 (... seconds) Iteration 4: loglike=-5.820288e+01 (... seconds) Converged at iteration 4. Now the algorithm converged after four iterations, before the requested 20 iterations. VB approximates the true posterior :math:`p(\mu,\tau|\mathbf{y})` with a distribution which factorizes with respect to the nodes: :math:`q(\mu)q(\tau)`\ . Examining posterior approximation --------------------------------- The resulting approximate posterior distributions :math:`q(\mu)` and :math:`q(\tau)` can be examined, for instance, by plotting the marginal probability density functions: >>> import bayespy.plot as bpplt >>> bpplt.pyplot.subplot(2, 1, 1) >>> bpplt.pdf(mu, np.linspace(-10, 20, num=100), color='k', name=r'\mu') [] >>> bpplt.pyplot.subplot(2, 1, 2) >>> bpplt.pdf(tau, np.linspace(1e-6, 0.08, num=100), color='k', name=r'\tau') [] >>> bpplt.pyplot.tight_layout() >>> bpplt.pyplot.show() .. plot:: import numpy as np np.random.seed(1) data = np.random.normal(5, 10, size=(10,)) from bayespy.nodes import GaussianARD, Gamma mu = GaussianARD(0, 1e-6) tau = Gamma(1e-6, 1e-6) y = GaussianARD(mu, tau, plates=(10,)) y.observe(data) from bayespy.inference import VB Q = VB(mu, tau, y) Q.update(repeat=20) import bayespy.plot as bpplt bpplt.pyplot.subplot(2, 1, 1) bpplt.pdf(mu, np.linspace(-10, 20, num=100), color='k', name=r'\mu') bpplt.pyplot.subplot(2, 1, 2) bpplt.pdf(tau, np.linspace(1e-6, 0.08, num=100), color='k', name=r'\tau') bpplt.pyplot.tight_layout() bpplt.pyplot.show() This example was a very simple introduction to using BayesPy. The model can be much more complex and each phase contains more options to give the user more control over the inference. The following sections give more details about the phases. bayespy-0.4.0/doc/source/user_guide/quickstartbackup.py0000664000175000017500000000646412346330025025051 0ustar jluttinejluttine00000000000000 # coding: utf-8 ## Quick start guide # This short guide shows the key steps in using BayesPy for variational Bayesian inference by applying BayesPy to a simple problem. The key steps in using BayesPy are the following: # # * Construct the model # # * Observe some of the variables by providing the data in a proper format # # * Run variational Bayesian inference # # * Examine the resulting posterior approximation # # To demonstrate BayesPy, we'll consider a very simple problem: we have a set of observations from a Gaussian distribution with unknown mean and variance, and we want to learn these parameters. In this case, we do not use any real-world data but generate some artificial data. The dataset consists of ten samples from a Gaussian distribution with mean 5 and standard deviation 10. This dataset can be generated with NumPy as follows: # In[1]: import numpy as np data = np.random.normal(5, 10, size=(10,)) # Now, given this data we would like to estimate the mean and the standard deviation as if we didn't know their values. The model can be defined as follows: # # $$ # \begin{split} # p(\mathbf{y}|\mu,\tau) &= \prod^{9}_{n=0} \mathcal{N}(y_n|\mu,\tau) \\ # p(\mu) &= \mathcal{N}(\mu|0,10^{-6}) \\ # p(\tau) &= \mathcal{G}(\tau|10^{-6},10^{-6}) # \end{split} # $$ # # where $\mathcal{N}$ is the Gaussian distribution parameterized by its mean and precision (i.e., inverse variance), and $\mathcal{G}$ is the gamma distribution parameterized by its shape and rate parameters. Note that we have given quite uninformative priors for the variables $\mu$ and $\tau$. This simple model can also be shown as a directed factor graph: # This model can be constructed in BayesPy as follows: # In[2]: from bayespy.nodes import GaussianARD, Gamma mu = GaussianARD(0, 1e-6) tau = Gamma(1e-6, 1e-6) y = GaussianARD(mu, tau, plates=(10,)) # In[3]: y.observe(data) # Next we want to estimate the posterior distribution. In principle, we could use different inference engines (e.g., MCMC or EP) but currently only variational Bayesian (VB) engine is implemented. The engine is initialized by giving all the nodes of the model: # In[4]: from bayespy.inference import VB Q = VB(mu, tau, y) # The inference algorithm can be run as long as wanted (max. 20 iterations in this case): # In[5]: Q.update(repeat=20) # Now the algorithm converged after four iterations, before the requested 20 iterations. # # VB approximates the true posterior $p(\mu,\tau|\mathbf{y})$ with a distribution which factorizes with respect to the nodes: $q(\mu)q(\tau)$. The resulting approximate posterior distributions $q(\mu)$ and $q(\tau)$ can be examined, for instance, by plotting the marginal probability density functions: # In[6]: import bayespy.plot as bpplt # The following two two lines are just for enabling matplotlib plotting in notebooks get_ipython().magic('matplotlib inline') bpplt.pyplot.plot([]) bpplt.pyplot.subplot(2, 1, 1) bpplt.pdf(mu, np.linspace(-10, 20, num=100), color='k', name=r'\mu') bpplt.pyplot.subplot(2, 1, 2) bpplt.pdf(tau, np.linspace(1e-6, 0.08, num=100), color='k', name=r'\tau'); # This example was a very simple introduction to using BayesPy. The model can be much more complex and each phase contains more options to give the user more control over the inference. The following sections give more details about the phases. bayespy-0.4.0/doc/source/user_guide/advanced.rst0000644000175000017500000003423512533543154023420 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014-2015 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. .. testsetup:: import numpy as np np.random.seed(1) # This is the PCA model from the previous sections from bayespy.nodes import GaussianARD, Gamma, Dot D = 3 X = GaussianARD(0, 1, shape=(D,), plates=(1,100), name='X') alpha = Gamma(1e-3, 1e-3, plates=(D,), name='alpha') C = GaussianARD(0, alpha, shape=(D,), plates=(10,1), name='C') F = Dot(C, X) tau = Gamma(1e-3, 1e-3, name='tau') Y = GaussianARD(F, tau, name='Y') c = np.random.randn(10, 2) x = np.random.randn(2, 100) data = np.dot(c, x) + 0.1*np.random.randn(10, 100) Y.observe(data) from bayespy.inference import VB import bayespy.plot as bpplt Q = VB(Y, C, X, alpha, tau) X.initialize_from_parameters(np.random.randn(1, 100, D), 10) from bayespy.inference.vmp import transformations rotX = transformations.RotateGaussianARD(X) rotC = transformations.RotateGaussianARD(C, alpha) R = transformations.RotationOptimizer(rotC, rotX, D) Q = VB(Y, C, X, alpha, tau) Q.callback = R.rotate Q.update(repeat=1000, tol=1e-6, verbose=False) import warnings warnings.simplefilter('error', UserWarning) Advanced topics =============== This section contains brief information on how to implement some advanced methods in BayesPy. These methods include Riemannian conjugate gradient methods, pattern search, simulated annealing, collapsed variational inference and stochastic variational inference. In order to use these methods properly, the user should understand them to some extent. They are also considered experimental, thus you may encounter bugs or unimplemented features. In any case, these methods may provide huge performance improvements easily compared to the standard VB-EM algorithm. Gradient-based optimization --------------------------- Variational Bayesian learning basically means that the parameters of the approximate posterior distributions are optimized to maximize the lower bound of the marginal log likelihood :cite:`Honkela:2010`. This optimization can be done by using gradient-based optimization methods. In order to improve the gradient-based methods, it is recommended to take into account the information geometry by using the Riemannian (a.k.a. natural) gradient. In fact, the standard VB-EM algorithm is equivalent to a gradient ascent method which uses the Riemannian gradient and step length 1. Thus, it is natural to try to improve this method by using non-linear conjugate gradient methods instead of gradient ascent. These optimization methods are especially useful when the VB-EM update equations are not available but one has to use fixed form approximation. But it is possible that the Riemannian conjugate gradient method improve performance even when the VB-EM update equations are available. .. currentmodule:: bayespy.inference The optimization algorithm in :func:`VB.optimize` has a simple interface. Instead of using the default Riemannian geometry, one can use the Euclidean geometry by giving :code:`riemannian=False`. It is also possible to choose the optimization method from gradient ascent (:code:`method='gradient'`) or conjugate gradient methods (only :code:`method='fletcher-reeves'` implemented at the moment). For instance, we could optimize nodes ``C`` and ``X`` jointly using Euclidean gradient ascent as: >>> Q = VB(Y, C, X, alpha, tau) >>> Q.optimize(C, X, riemannian=False, method='gradient', maxiter=5) Iteration ... Note that this is very inefficient way of updating those nodes (bad geometry and not using conjugate gradients). Thus, one should understand the idea of these optimization methods, otherwise one may do something extremely inefficient. Most likely this method can be found useful in combination with the advanced tricks in the following sections. .. note:: The Euclidean gradient has not been implemented for all nodes yet. The Euclidean gradient is required by the Euclidean geometry based optimization but also by the conjugate gradient methods in the Riemannian geometry. Thus, the Riemannian conjugate gradient may not yet work for all models. It is possible to construct custom optimization algorithms with the tools provided by :class:`VB`. For instance, :func:`VB.get_parameters` and :func:`VB.set_parameters` can be used to handle the parameters of nodes. :func:`VB.get_gradients` is used for computing the gradients of nodes. The parameter and gradient objects are not numerical arrays but more complex nested lists not meant to be accessed by the user. Thus, for simple arithmetics with the parameter and gradient objects, use functions :func:`VB.add` and :func:`VB.dot`. Finally, :func:`VB.compute_lowerbound` and :func:`VB.has_converged` can be used to monitor the lower bound. Collapsed inference ------------------- The optimization method can be used efficiently in such a way that some of the variables are collapsed, that is, marginalized out :cite:`Hensman:2012`. The collapsed variables must be conditionally independent given the observations and all other variables. Probably, one also wants that the size of the marginalized variables is large and the size of the optimized variables is small. For instance, in our PCA example, we could optimize as follows: >>> Q.optimize(C, tau, maxiter=10, collapsed=[X, alpha]) Iteration ... The collapsed variables are given as a list. This optimization does basically the following: It first computes the gradients for ``C`` and ``tau`` and takes an update step using the desired optimization method. Then, it updates the collapsed variables by using the standard VB-EM update equations. These two steps are taken in turns. Effectively, this corresponds to collapsing the variables ``X`` and ``alpha`` in a particular way. The point of this method is that the number of parameters in the optimization reduces significantly and the collapsed variables are updated optimally. For more details, see :cite:`Hensman:2012`. It is possible to use this method in such a way, that the collapsed variables are not conditionally independent given the observations and all other variables. However, in that case, the method does not anymore correspond to collapsing the variables but just using VB-EM updates after gradient-based updates. The method does not check for conditional independence, so the user is free to do this. .. note:: Although the Riemannian conjugate gradient method has not yet been implemented for all nodes, it may be possible to collapse those nodes and optimize the other nodes for which the Euclidean gradient is already implemented. Pattern search -------------- The pattern search method estimates the direction in which the approximate posterior distributions are updating and performs a line search in that direction :cite:`Honkela:2003`. The search direction is based on the difference in the VB parameters on successive updates (or several updates). The idea is that the VB-EM algorithm may be slow because it just zigzags and this can be fixed by moving to the direction in which the VB-EM is slowly moving. BayesPy offers a simple built-in pattern search method :func:`VB.pattern_search`. The method updates the nodes twice, measures the difference in the parameters and performs a line search with a small number of function evaluations: >>> Q.pattern_search(C, X) Iteration ... Similarly to the collapsed optimization, it is possible to collapse some of the variables in the pattern search. The same rules of conditional independence apply as above. The collapsed variables are given as list: >>> Q.pattern_search(C, tau, collapsed=[X, alpha]) Iteration ... Also, a maximum number of iterations can be set by using ``maxiter`` keyword argument. It is not always obvious whether a pattern search will improve the rate of convergence or not but if it seems that the convergence is slow because of zigzagging, it may be worth a try. Note that the computational cost of the pattern search is quite high, thus it is not recommended to perform it after every VB-EM update but every now and then, for instance, after every 10 iterations. In addition, it is possible to write a more customized VB learning algorithm which uses pattern searches by using the different methods of :class:`VB` discussed above. Deterministic annealing ----------------------- The standard VB-EM algorithm converges to a local optimum which can often be inferior to the global optimum and many other local optima. Deterministic annealing aims at finding a better local optimum, hopefully even the global optimum :cite:`Katahira:2008`. It does this by increasing the weight on the entropy of the posterior approximation in the VB lower bound. Effectively, the annealed lower bound becomes closer to a uniform function instead of the original multimodal lower bound. The weight on the entropy is recovered slowly and the optimization is much more robust to initialization. In BayesPy, the annealing can be set by using :func:`VB.set_annealing`. The given annealing should be in range :math:`(0,1]` but this is not validated in case the user wants to do something experimental. If annealing is set to 1, the original VB lower bound is recovered. Annealing with 0 would lead to an improper uniform distribution, thus it will lead to errors. The entropy term is weighted by the inverse of this annealing term. An alternative view is that the model probability density functions are raised to the power of the annealing term. Typically, the annealing is used in such a way that the annealing is small at the beginning and increased after every convergence of the VB algorithm until value 1 is reached. After the annealing value is increased, the algorithm continues from where it had just converged. The annealing can be used for instance as: >>> beta = 0.1 >>> while beta < 1.0: ... beta = min(beta*1.5, 1.0) ... Q.set_annealing(beta) ... Q.update(repeat=100, tol=1e-4) Iteration ... Here, the ``tol`` keyword argument is used to adjust the threshold for convergence. In this case, it is a bit larger than by default so the algorithm does not need to converge perfectly but a rougher convergence is sufficient for the next iteration with a new annealing value. Stochastic variational inference -------------------------------- In stochastic variational inference :cite:`Hoffman:2013`, the idea is to use mini-batches of large datasets to compute noisy gradients and learn the VB distributions by using stochastic gradient ascent. In order for it to be useful, the model must be such that it can be divided into "intermediate" and "global" variables. The number of intermediate variables increases with the data but the number of global variables remains fixed. The global variables are learnt in the stochastic optimization. By denoting the data as :math:`Y=[Y_1, \ldots, Y_N]`, the intermediate variables as :math:`Z=[Z_1, \ldots, Z_N]` and the global variables as :math:`\theta`, the model needs to have the following structure: .. math:: p(Y, Z, \theta) &= p(\theta) \prod^N_{n=1} p(Y_n|Z_n,\theta) p(Z_n|\theta) The algorithm consists of three steps which are iterated: 1) a random mini-batch of the data is selected, 2) the corresponding intermediate variables are updated by using normal VB update equations, and 3) the global variables are updated with (stochastic) gradient ascent as if there was as many replications of the mini-batch as needed to recover the original dataset size. The learning rate for the gradient ascent must satisfy: .. math:: \sum^\infty_{i=1} \alpha_i = \infty \qquad \text{and} \qquad \sum^\infty_{i=1} \alpha^2 < \infty, where :math:`i` is the iteration number. An example of a valid learning parameter is :math:`\alpha_i = (\delta + i)^{-\gamma}`, where :math:`\delta \geq 0` is a delay and :math:`\gamma\in (0.5, 1]` is a forgetting rate. Stochastic variational inference is relatively easy to use in BayesPy. The idea is that the user creates a model for the size of a mini-batch and specifies a multiplier for those plate axes that are replicated. For the PCA example, the mini-batch model can be costructed as follows. We decide to use ``X`` as an intermediate variable and the other variables are global. The global variables ``alpha``, ``C`` and ``tau`` are constructed identically as before. The intermediate variable ``X`` is constructed as: >>> X = GaussianARD(0, 1, ... shape=(D,), ... plates=(1,5), ... plates_multiplier=(1,20), ... name='X') Note that the plates are ``(1,5)`` whereas they are ``(1,100)`` in the full model. Thus, we need to provide a plates multiplier ``(1,20)`` to define how the plates are replicated to get the full dataset. These multipliers do not need to be integers, in this case the latter plate axis is multiplied by :math:`100/5=20`. The remaining variables are defined as before: >>> F = Dot(C, X) >>> Y = GaussianARD(F, tau, name='Y') Note that the plates of ``Y`` and ``F`` also correspond to the size of the mini-batch and they also deduce the plate multipliers from their parents, thus we do not need to specify the multiplier here explicitly (although it is ok to do so). Let us construct the inference engine for the new mini-batch model: >>> Q = VB(Y, C, X, alpha, tau) Use random initialization for ``C`` to break the symmetry in ``C`` and ``X``: >>> C.initialize_from_random() Then, stochastic variational inference algorithm could look as follows: >>> Q.ignore_bound_checks = True >>> for n in range(200): ... subset = np.random.choice(100, 5) ... Y.observe(data[:,subset]) ... Q.update(X) ... learning_rate = (n + 2.0) ** (-0.7) ... Q.gradient_step(C, alpha, tau, scale=learning_rate) Iteration ... First, we ignore the bound checks because they are noisy. Then, the loop consists of three parts: 1) Draw a random mini-batch of the data (5 samples from 100). 2) Update the intermediate variable ``X``. 3) Update global variables with gradient ascent using a proper learning rate. Black-box variational inference ------------------------------- NOT YET IMPLEMENTED. bayespy-0.4.0/doc/source/user_guide/plot.rst0000644000175000017500000003625712533543154022637 0ustar jluttinejluttine00000000000000.. Copyright (C) 2014 Jaakko Luttinen This file is licensed under the MIT License. See LICENSE for a text of the license. .. testsetup:: # This is the PCA model from the previous sections import numpy as np np.random.seed(1) from bayespy.nodes import GaussianARD, Gamma, Dot D = 3 X = GaussianARD(0, 1, shape=(D,), plates=(1,100), name='X') alpha = Gamma(1e-3, 1e-3, plates=(D,), name='alpha') C = GaussianARD(0, alpha, shape=(D,), plates=(10,1), name='C') F = Dot(C, X) tau = Gamma(1e-3, 1e-3, name='tau') Y = GaussianARD(F, tau, name='Y') c = np.random.randn(10, 2) x = np.random.randn(2, 100) data = np.dot(c, x) + 0.1*np.random.randn(10, 100) Y.observe(data) Y.observe(data, mask=[[True], [False], [False], [True], [True], [False], [True], [True], [True], [False]]) X.initialize_from_parameters(np.random.randn(1, 100, D), 10) from bayespy.inference import VB Q = VB(Y, C, X, alpha, tau) X.initialize_from_parameters(np.random.randn(1, 100, D), 10) from bayespy.inference.vmp import transformations rotX = transformations.RotateGaussianARD(X) rotC = transformations.RotateGaussianARD(C, alpha) R = transformations.RotationOptimizer(rotC, rotX, D) Q = VB(Y, C, X, alpha, tau) Q.callback = R.rotate Q.update(repeat=1000, tol=1e-6, verbose=False) from bayespy.nodes import Gaussian Examining the results ===================== After the results have been obtained, it is important to be able to examine the results easily. The results can be examined either numerically by inspecting numerical arrays or visually by plotting distributions of the nodes. In addition, the posterior distributions can be visualized during the learning algorithm and the results can saved into a file. Plotting the results -------------------- .. currentmodule:: bayespy The module :mod:`plot` offers some plotting basic functionality: >>> import bayespy.plot as bpplt .. currentmodule:: bayespy.plot The module contains ``matplotlib.pyplot`` module if the user needs that. For instance, interactive plotting can be enabled as: >>> bpplt.pyplot.ion() The :mod:`plot` module contains some functions but it is not a very comprehensive collection, thus the user may need to write some problem- or model-specific plotting functions. The current collection is: * :func:`pdf`: show probability density function of a scalar * :func:`contour`: show probability density function of two-element vector * :func:`hinton`: show the Hinton diagram * :func:`plot`: show value as a function The probability density function of a scalar random variable can be plotted using the function :func:`pdf`: >>> bpplt.pyplot.figure() >>> bpplt.pdf(Q['tau'], np.linspace(60, 140, num=100)) [] .. plot:: # This is the PCA model from the previous sections import numpy as np np.random.seed(1) from bayespy.nodes import GaussianARD, Gamma, Dot D = 3 X = GaussianARD(0, 1, shape=(D,), plates=(1,100), name='X') alpha = Gamma(1e-3, 1e-3, plates=(D,), name='alpha') C = GaussianARD(0, alpha, shape=(D,), plates=(10,1), name='C') F = Dot(C, X) tau = Gamma(1e-3, 1e-3, name='tau') Y = GaussianARD(F, tau) c = np.random.randn(10, 2) x = np.random.randn(2, 100) data = np.dot(c, x) + 0.1*np.random.randn(10, 100) Y.observe(data) Y.observe(data, mask=[[True], [False], [False], [True], [True], [False], [True], [True], [True], [False]]) from bayespy.inference import VB Q = VB(Y, C, X, alpha, tau) X.initialize_from_parameters(np.random.randn(1, 100, D), 10) from bayespy.inference.vmp import transformations rotX = transformations.RotateGaussianARD(X) rotC = transformations.RotateGaussianARD(C, alpha) R = transformations.RotationOptimizer(rotC, rotX, D) Q = VB(Y, C, X, alpha, tau) Q.callback = R.rotate Q.update(repeat=1000, tol=1e-6) Q.update(repeat=50, tol=np.nan) import bayespy.plot as bpplt bpplt.pdf(Q['tau'], np.linspace(60, 140, num=100)) The variable ``tau`` models the inverse variance of the noise, for which the true value is :math:`0.1^{-2}=100`. Thus, the posterior captures the true value quite accurately. Similarly, the function :func:`contour` can be used to plot the probability density function of a 2-dimensional variable, for instance: >>> V = Gaussian([3, 5], [[4, 2], [2, 5]]) >>> bpplt.pyplot.figure() >>> bpplt.contour(V, np.linspace(1, 5, num=100), np.linspace(3, 7, num=100)) .. plot:: # This is the PCA model from the previous sections import numpy as np np.random.seed(1) from bayespy.nodes import GaussianARD, Gamma, Dot D = 3 X = GaussianARD(0, 1, shape=(D,), plates=(1,100), name='X') alpha = Gamma(1e-3, 1e-3, plates=(D,), name='alpha') C = GaussianARD(0, alpha, shape=(D,), plates=(10,1), name='C') F = Dot(C, X) tau = Gamma(1e-3, 1e-3, name='tau') Y = GaussianARD(F, tau) c = np.random.randn(10, 2) x = np.random.randn(2, 100) data = np.dot(c, x) + 0.1*np.random.randn(10, 100) Y.observe(data) Y.observe(data, mask=[[True], [False], [False], [True], [True], [False], [True], [True], [True], [False]]) from bayespy.inference import VB Q = VB(Y, C, X, alpha, tau) X.initialize_from_parameters(np.random.randn(1, 100, D), 10) from bayespy.inference.vmp import transformations rotX = transformations.RotateGaussianARD(X) rotC = transformations.RotateGaussianARD(C, alpha) R = transformations.RotationOptimizer(rotC, rotX, D) Q = VB(Y, C, X, alpha, tau) Q.callback = R.rotate Q.update(repeat=1000, tol=1e-6) Q.update(repeat=50, tol=np.nan) import bayespy.plot as bpplt from bayespy.nodes import Gaussian V = Gaussian([3, 5], [[4, 2], [2, 5]]) bpplt.contour(V, np.linspace(1, 5, num=100), np.linspace(3, 7, num=100)) Both :func:`pdf` and :func:`contour` require that the user provides the grid on which the probability density function is computed. They also support several keyword arguments for modifying the output, similarly as ``plot`` and ``contour`` in ``matplotlib.pyplot``. These functions can be used only for stochastic nodes. A few other plot types are also available as built-in functions. A Hinton diagram can be plotted as: >>> bpplt.pyplot.figure() >>> bpplt.hinton(C) .. plot:: # This is the PCA model from the previous sections import numpy as np np.random.seed(1) from bayespy.nodes import GaussianARD, Gamma, Dot D = 3 X = GaussianARD(0, 1, shape=(D,), plates=(1,100), name='X') alpha = Gamma(1e-3, 1e-3, plates=(D,), name='alpha') C = GaussianARD(0, alpha, shape=(D,), plates=(10,1), name='C') F = Dot(C, X) tau = Gamma(1e-3, 1e-3, name='tau') Y = GaussianARD(F, tau) c = np.random.randn(10, 2) x = np.random.randn(2, 100) data = np.dot(c, x) + 0.1*np.random.randn(10, 100) Y.observe(data) Y.observe(data, mask=[[True], [False], [False], [True], [True], [False], [True], [True], [True], [False]]) from bayespy.inference import VB Q = VB(Y, C, X, alpha, tau) X.initialize_from_parameters(np.random.randn(1, 100, D), 10) from bayespy.inference.vmp import transformations rotX = transformations.RotateGaussianARD(X) rotC = transformations.RotateGaussianARD(C, alpha) R = transformations.RotationOptimizer(rotC, rotX, D) Q = VB(Y, C, X, alpha, tau) Q.callback = R.rotate Q.update(repeat=1000, tol=1e-6) Q.update(repeat=50, tol=np.nan) import bayespy.plot as bpplt from bayespy.nodes import Gaussian bpplt.hinton(C) The diagram shows the elements of the matrix :math:`C`. The size of the filled rectangle corresponds to the absolute value of the element mean, and white and black correspond to positive and negative values, respectively. The non-filled rectangle shows standard deviation. From this diagram it is clear that the third column of :math:`C` has been pruned out and the rows that were missing in the data have zero mean and column-specific variance. The function :func:`hinton` is a simple wrapper for node-specific Hinton diagram plotters, such as :func:`gaussian_hinton` and :func:`dirichlet_hinton`. Thus, the keyword arguments depend on the node which is plotted. Another plotting function is :func:`plot`, which just plots the values of the node over one axis as a function: >>> bpplt.pyplot.figure() >>> bpplt.plot(X, axis=-2) .. plot:: # This is the PCA model from the previous sections import numpy as np np.random.seed(1) from bayespy.nodes import GaussianARD, Gamma, Dot D = 3 X = GaussianARD(0, 1, shape=(D,), plates=(1,100), name='X') alpha = Gamma(1e-3, 1e-3, plates=(D,), name='alpha') C = GaussianARD(0, alpha, shape=(D,), plates=(10,1), name='C') F = Dot(C, X) tau = Gamma(1e-3, 1e-3, name='tau') Y = GaussianARD(F, tau) c = np.random.randn(10, 2) x = np.random.randn(2, 100) data = np.dot(c, x) + 0.1*np.random.randn(10, 100) Y.observe(data) Y.observe(data, mask=[[True], [False], [False], [True], [True], [False], [True], [True], [True], [False]]) from bayespy.inference import VB Q = VB(Y, C, X, alpha, tau) X.initialize_from_parameters(np.random.randn(1, 100, D), 10) from bayespy.inference.vmp import transformations rotX = transformations.RotateGaussianARD(X) rotC = transformations.RotateGaussianARD(C, alpha) R = transformations.RotationOptimizer(rotC, rotX, D) Q = VB(Y, C, X, alpha, tau) Q.callback = R.rotate Q.update(repeat=1000, tol=1e-6) Q.update(repeat=50, tol=np.nan) import bayespy.plot as bpplt from bayespy.nodes import Gaussian bpplt.plot(X, axis=-2) Now, the ``axis`` is the second last axis which corresponds to :math:`n=0,\ldots,N-1`. As :math:`D=3`, there are three subplots. For Gaussian variables, the function shows the mean and two standard deviations. The plot shows that the third component has been pruned out, thus the method has been able to recover the true dimensionality of the latent space. It also has similar keyword arguments to ``plot`` function in ``matplotlib.pyplot``. Again, :func:`plot` is a simple wrapper over node-specific plotting functions, thus it supports only some node classes. Monitoring during the inference algorithm ----------------------------------------- It is possible to plot the distribution of the nodes during the learning algorithm. This is useful when the user is interested to see how the distributions evolve during learning and what is happening to the distributions. In order to utilize monitoring, the user must set plotters for the nodes that he or she wishes to monitor. This can be done either when creating the node or later at any time. The plotters are set by creating a plotter object and providing this object to the node. The plotter is a wrapper of one of the plotting functions mentioned above: :class:`PDFPlotter`, :class:`ContourPlotter`, :class:`HintonPlotter` or :class:`FunctionPlotter`. Thus, our example model could use the following plotters: >>> tau.set_plotter(bpplt.PDFPlotter(np.linspace(60, 140, num=100))) >>> C.set_plotter(bpplt.HintonPlotter()) >>> X.set_plotter(bpplt.FunctionPlotter(axis=-2)) These could have been given at node creation as a keyword argument ``plotter``: >>> V = Gaussian([3, 5], [[4, 2], [2, 5]], ... plotter=bpplt.ContourPlotter(np.linspace(1, 5, num=100), ... np.linspace(3, 7, num=100))) When the plotter is set, one can use the ``plot`` method of the node to perform plotting: >>> V.plot() Nodes can also be plotted using the ``plot`` method of the inference engine: >>> Q.plot('C') This method remembers the figure in which a node has been plotted and uses that every time it plots the same node. In order to monitor the nodes during learning, it is possible to use the keyword argument ``plot``: >>> Q.update(repeat=5, plot=True, tol=np.nan) Iteration 19: loglike=-1.221354e+02 (... seconds) Iteration 20: loglike=-1.221354e+02 (... seconds) Iteration 21: loglike=-1.221354e+02 (... seconds) Iteration 22: loglike=-1.221354e+02 (... seconds) Iteration 23: loglike=-1.221354e+02 (... seconds) Each node which has a plotter set will be plotted after it is updated. Note that this may slow down the inference significantly if the plotting operation is time consuming. Posterior parameters and moments -------------------------------- If the built-in plotting functions are not sufficient, it is possible to use ``matplotlib.pyplot`` for custom plotting. Each node has ``get_moments`` method which returns the moments and they can be used for plotting. Stochastic exponential family nodes have natural parameter vectors which can also be used. In addition to plotting, it is also possible to just print the moments or parameters in the console. Saving and loading results -------------------------- .. currentmodule:: bayespy.inference The results of the inference engine can be easily saved and loaded using :func:`VB.save` and :func:`VB.load` methods: >>> Q.save(filename='tmp.hdf5') >>> Q.load(filename='tmp.hdf5') The results are stored in a HDF5 file. The user may set an autosave file in which the results are automatically saved regularly. Autosave filename can be set at creation time by ``autosave_filename`` keyword argument or later using :func:`VB.set_autosave` method. If autosave file has been set, the :func:`VB.save` and :func:`VB.load` methods use that file by default. In order for the saving to work, all stochastic nodes must have been given (unique) names. However, note that these methods do *not* save nor load the node definitions. It means that the user must create the nodes and the inference engine and then use :func:`VB.load` to set the state of the nodes and the inference engine. If there are any differences in the model that was saved and the one which is tried to update using loading, then loading does not work. Thus, the user should keep the model construction unmodified in a Python file in order to be able to load the results later. Or if the user wishes to share the results, he or she must share the model construction Python file with the HDF5 results file. bayespy-0.4.0/doc/Makefile0000644000175000017500000001612112600471211017105 0ustar jluttinejluttine00000000000000# Makefile for Sphinx documentation # # You can set these variables from the command line. SPHINXOPTS = -v SPHINXBUILD = python -m sphinx # sphinx-build PAPER = BUILDDIR = build # User-friendly check for sphinx-build #ifeq ($(shell which ${SPHINXBUILD%% *} >/dev/null 2>&1; echo $$?), 1) #$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '${SPHINXBUILD%% *}' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) #endif # Internal variables. PAPEROPT_a4 = -D latex_paper_size=a4 PAPEROPT_letter = -D latex_paper_size=letter ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source # the i18n builder cannot share the environment and doctrees with the others I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext help: @echo "Please use \`make ' where is one of" @echo " html to make standalone HTML files" @echo " dirhtml to make HTML files named index.html in directories" @echo " singlehtml to make a single large HTML file" @echo " pickle to make pickle files" @echo " json to make JSON files" @echo " htmlhelp to make HTML files and a HTML help project" @echo " qthelp to make HTML files and a qthelp project" @echo " devhelp to make HTML files and a Devhelp project" @echo " epub to make an epub" @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" @echo " latexpdf to make LaTeX files and run them through pdflatex" @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" @echo " text to make text files" @echo " man to make manual pages" @echo " texinfo to make Texinfo files" @echo " info to make Texinfo files and run them through makeinfo" @echo " gettext to make PO message catalogs" @echo " changes to make an overview of all changed/added/deprecated items" @echo " xml to make Docutils-native XML files" @echo " pseudoxml to make pseudoxml-XML files for display purposes" @echo " linkcheck to check all external links for integrity" @echo " doctest to run all doctests embedded in the documentation (if enabled)" clean: rm -rf $(BUILDDIR)/* rm -rf source/*/generated rm -rf source/*/*_files gh-pages: make clean cd build && git clone https://github.com/bayespy/bayespy.git -b gh-pages html make html cd build/html && git add -f *.png *.html *.txt *.js *.pdf *.py *.css * && git commit -a && git push html: latexpdf $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html cp $(BUILDDIR)/latex/BayesPy.pdf $(BUILDDIR)/html/_static/ cp -r build/latex/user_guide/* build/html/user_guide/ cp -r build/latex/examples/* build/html/examples/ @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." dirhtml: $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." singlehtml: $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml @echo @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." pickle: $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle @echo @echo "Build finished; now you can process the pickle files." json: $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json @echo @echo "Build finished; now you can process the JSON files." htmlhelp: $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp @echo @echo "Build finished; now you can run HTML Help Workshop with the" \ ".hhp project file in $(BUILDDIR)/htmlhelp." qthelp: $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp @echo @echo "Build finished; now you can run "qcollectiongenerator" with the" \ ".qhcp project file in $(BUILDDIR)/qthelp, like this:" @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/BayesPy.qhcp" @echo "To view the help file:" @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/BayesPy.qhc" devhelp: $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp @echo @echo "Build finished." @echo "To view the help file:" @echo "# mkdir -p $$HOME/.local/share/devhelp/BayesPy" @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/BayesPy" @echo "# devhelp" epub: $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub @echo @echo "Build finished. The epub file is in $(BUILDDIR)/epub." latex: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." @echo "Run \`make' in that directory to run these through (pdf)latex" \ "(use \`make latexpdf' here to do that automatically)." latexpdf: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through pdflatex..." $(MAKE) -C $(BUILDDIR)/latex all-pdf @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." latexpdfja: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through platex and dvipdfmx..." $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." text: $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text @echo @echo "Build finished. The text files are in $(BUILDDIR)/text." man: $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man @echo @echo "Build finished. The manual pages are in $(BUILDDIR)/man." texinfo: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." @echo "Run \`make' in that directory to run these through makeinfo" \ "(use \`make info' here to do that automatically)." info: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo "Running Texinfo files through makeinfo..." make -C $(BUILDDIR)/texinfo info @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." gettext: $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale @echo @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." changes: $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes @echo @echo "The overview file is in $(BUILDDIR)/changes." linkcheck: $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck @echo @echo "Link check complete; look for any errors in the above output " \ "or in $(BUILDDIR)/linkcheck/output.txt." doctest: $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest @echo "Testing of doctests in the sources finished, look at the " \ "results in $(BUILDDIR)/doctest/output.txt." xml: $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml @echo @echo "Build finished. The XML files are in $(BUILDDIR)/xml." pseudoxml: $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml @echo @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." bayespy-0.4.0/README.rst0000644000175000017500000000753312535613115016406 0ustar jluttinejluttine00000000000000BayesPy - Bayesian Python ========================= BayesPy provides tools for Bayesian inference with Python. The user constructs a model as a Bayesian network, observes data and runs posterior inference. The goal is to provide a tool which is efficient, flexible and extendable enough for expert use but also accessible for more casual users. Currently, only variational Bayesian inference for conjugate-exponential family (variational message passing) has been implemented. Future work includes variational approximations for other types of distributions and possibly other approximate inference methods such as expectation propagation, Laplace approximations, Markov chain Monte Carlo (MCMC) and other methods. Contributions are welcome. Project information ------------------- Copyright (C) 2011-2015 Jaakko Luttinen and other contributors (see below) BayesPy including the documentation is licensed under the MIT License. See LICENSE file for a text of the license or visit http://opensource.org/licenses/MIT. * Documentation: * http://bayespy.org * `PDF file `_ * RST format in ``doc`` directory * Repository: https://github.com/bayespy/bayespy.git * Bug reports: https://github.com/bayespy/bayespy/issues * Mailing list: bayespy@googlegroups.com * IRC: #bayespy @ `freenode `_ * Author: Jaakko Luttinen jaakko.luttinen@iki.fi * Latest release: .. image:: https://badge.fury.io/py/bayespy.svg :target: https://pypi.python.org/pypi/bayespy * Build status: .. image:: https://travis-ci.org/bayespy/bayespy.png?branch=master :target: https://travis-ci.org/bayespy/bayespy/ * Unit test coverage: .. image:: https://coveralls.io/repos/bayespy/bayespy/badge.png?branch=master :target: https://coveralls.io/r/bayespy/bayespy?branch=master Similar projects ---------------- `VIBES `_ (http://vibes.sourceforge.net/) allows variational inference to be performed automatically on a Bayesian network. It is implemented in Java and released under revised BSD license. `Bayes Blocks `_ (http://research.ics.aalto.fi/bayes/software/) is a C++/Python implementation of the variational building block framework. The framework allows easy learning of a wide variety of models using variational Bayesian learning. It is available as free software under the GNU General Public License. `Infer.NET `_ (http://research.microsoft.com/infernet/) is a .NET framework for machine learning. It provides message-passing algorithms and statistical routines for performing Bayesian inference. It is partly closed source and licensed for non-commercial use only. `PyMC `_ (https://github.com/pymc-devs/pymc) provides MCMC methods in Python. It is released under the Academic Free License. `OpenBUGS `_ (http://www.openbugs.info) is a software package for performing Bayesian inference using Gibbs sampling. It is released under the GNU General Public License. `Dimple `_ (http://dimple.probprog.org/) provides Gibbs sampling, belief propagation and a few other inference algorithms for Matlab and Java. It is released under the Apache License. `Stan `_ (http://mc-stan.org/) provides inference using MCMC with an interface for R and Python. It is released under the New BSD License. `PBNT - Python Bayesian Network Toolbox `_ (http://pbnt.berlios.de/) is Bayesian network library in Python supporting static networks with discrete variables. There was no information about the license. Contributors ------------ The list of contributors: * Jaakko Luttinen * Hannu Hartikainen Each file or the git log can be used for more detailed information. bayespy-0.4.0/LICENSE0000644000175000017500000000210212533543154015712 0ustar jluttinejluttine00000000000000The MIT License (MIT) Copyright (c) 2011-2015 BayesPy developers Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. bayespy-0.4.0/bayespy.egg-info/0000755000175000017500000000000012615647103020060 5ustar jluttinejluttine00000000000000bayespy-0.4.0/bayespy.egg-info/entry_points.txt0000644000175000017500000000007212615647067023366 0ustar jluttinejluttine00000000000000[nose.plugins] warnaserror = bayespy.testing:WarnAsError bayespy-0.4.0/bayespy.egg-info/dependency_links.txt0000644000175000017500000000000112615647067024137 0ustar jluttinejluttine00000000000000 bayespy-0.4.0/bayespy.egg-info/PKG-INFO0000644000175000017500000001316612615647067021175 0ustar jluttinejluttine00000000000000Metadata-Version: 1.1 Name: bayespy Version: 0.4.0 Summary: Variational Bayesian inference tools for Python Home-page: http://bayespy.org Author: Jaakko Luttinen Author-email: jaakko.luttinen@iki.fi License: MIT Description: BayesPy - Bayesian Python ========================= BayesPy provides tools for Bayesian inference with Python. The user constructs a model as a Bayesian network, observes data and runs posterior inference. The goal is to provide a tool which is efficient, flexible and extendable enough for expert use but also accessible for more casual users. Currently, only variational Bayesian inference for conjugate-exponential family (variational message passing) has been implemented. Future work includes variational approximations for other types of distributions and possibly other approximate inference methods such as expectation propagation, Laplace approximations, Markov chain Monte Carlo (MCMC) and other methods. Contributions are welcome. Project information ------------------- Copyright (C) 2011-2015 Jaakko Luttinen and other contributors (see below) BayesPy including the documentation is licensed under the MIT License. See LICENSE file for a text of the license or visit http://opensource.org/licenses/MIT. * Documentation: * http://bayespy.org * `PDF file `_ * RST format in ``doc`` directory * Repository: https://github.com/bayespy/bayespy.git * Bug reports: https://github.com/bayespy/bayespy/issues * Mailing list: bayespy@googlegroups.com * IRC: #bayespy @ `freenode `_ * Author: Jaakko Luttinen jaakko.luttinen@iki.fi * Latest release: .. image:: https://badge.fury.io/py/bayespy.svg :target: https://pypi.python.org/pypi/bayespy * Build status: .. image:: https://travis-ci.org/bayespy/bayespy.png?branch=master :target: https://travis-ci.org/bayespy/bayespy/ * Unit test coverage: .. image:: https://coveralls.io/repos/bayespy/bayespy/badge.png?branch=master :target: https://coveralls.io/r/bayespy/bayespy?branch=master Similar projects ---------------- `VIBES `_ (http://vibes.sourceforge.net/) allows variational inference to be performed automatically on a Bayesian network. It is implemented in Java and released under revised BSD license. `Bayes Blocks `_ (http://research.ics.aalto.fi/bayes/software/) is a C++/Python implementation of the variational building block framework. The framework allows easy learning of a wide variety of models using variational Bayesian learning. It is available as free software under the GNU General Public License. `Infer.NET `_ (http://research.microsoft.com/infernet/) is a .NET framework for machine learning. It provides message-passing algorithms and statistical routines for performing Bayesian inference. It is partly closed source and licensed for non-commercial use only. `PyMC `_ (https://github.com/pymc-devs/pymc) provides MCMC methods in Python. It is released under the Academic Free License. `OpenBUGS `_ (http://www.openbugs.info) is a software package for performing Bayesian inference using Gibbs sampling. It is released under the GNU General Public License. `Dimple `_ (http://dimple.probprog.org/) provides Gibbs sampling, belief propagation and a few other inference algorithms for Matlab and Java. It is released under the Apache License. `Stan `_ (http://mc-stan.org/) provides inference using MCMC with an interface for R and Python. It is released under the New BSD License. `PBNT - Python Bayesian Network Toolbox `_ (http://pbnt.berlios.de/) is Bayesian network library in Python supporting static networks with discrete variables. There was no information about the license. Contributors ------------ The list of contributors: * Jaakko Luttinen * Hannu Hartikainen Each file or the git log can be used for more detailed information. Keywords: variational Bayes,probabilistic programming,Bayesian networks,graphical models,variational message passing Platform: UNKNOWN Classifier: Programming Language :: Python :: 3 :: Only Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Development Status :: 4 - Beta Classifier: Environment :: Console Classifier: Intended Audience :: Developers Classifier: Intended Audience :: Science/Research Classifier: License :: OSI Approved :: MIT License Classifier: Operating System :: OS Independent Classifier: Topic :: Scientific/Engineering Classifier: Topic :: Scientific/Engineering :: Information Analysis bayespy-0.4.0/bayespy.egg-info/top_level.txt0000644000175000017500000000001012615647067022612 0ustar jluttinejluttine00000000000000bayespy bayespy-0.4.0/bayespy.egg-info/requires.txt0000644000175000017500000000006212615647067022467 0ustar jluttinejluttine00000000000000numpy>=1.8.0 scipy>=0.13.0 matplotlib>=1.2.0 h5py bayespy-0.4.0/bayespy.egg-info/SOURCES.txt0000644000175000017500000001303312615647073021752 0ustar jluttinejluttine00000000000000INSTALL.rst LICENSE MANIFEST.in README.rst setup.cfg setup.py bayespy/__init__.py bayespy/discrete_example.py bayespy/plot.py bayespy/testing.py bayespy/version.py bayespy.egg-info/PKG-INFO bayespy.egg-info/SOURCES.txt bayespy.egg-info/dependency_links.txt bayespy.egg-info/entry_points.txt bayespy.egg-info/requires.txt bayespy.egg-info/top_level.txt bayespy/demos/__init__.py bayespy/demos/annealing.py bayespy/demos/black_box.py bayespy/demos/categorical.py bayespy/demos/collapsed_cg.py bayespy/demos/gamma_shape.py bayespy/demos/hmm.py bayespy/demos/lda.py bayespy/demos/lssm.py bayespy/demos/lssm_sd.py bayespy/demos/lssm_tvd.py bayespy/demos/mog.py bayespy/demos/pattern_search.py bayespy/demos/pca.py bayespy/demos/saving.py bayespy/demos/stochastic_inference.py bayespy/inference/__init__.py bayespy/inference/vmp/__init__.py bayespy/inference/vmp/transformations.py bayespy/inference/vmp/vmp.py bayespy/inference/vmp/nodes/CovarianceFunctions.py bayespy/inference/vmp/nodes/GaussianProcesses.py bayespy/inference/vmp/nodes/__init__.py bayespy/inference/vmp/nodes/add.py bayespy/inference/vmp/nodes/bernoulli.py bayespy/inference/vmp/nodes/beta.py bayespy/inference/vmp/nodes/binomial.py bayespy/inference/vmp/nodes/categorical.py bayespy/inference/vmp/nodes/categorical_markov_chain.py bayespy/inference/vmp/nodes/concatenate.py bayespy/inference/vmp/nodes/constant.py bayespy/inference/vmp/nodes/deterministic.py bayespy/inference/vmp/nodes/dirichlet.py bayespy/inference/vmp/nodes/dot.py bayespy/inference/vmp/nodes/expfamily.py bayespy/inference/vmp/nodes/exponential.py bayespy/inference/vmp/nodes/gamma.py bayespy/inference/vmp/nodes/gate.py bayespy/inference/vmp/nodes/gaussian.py bayespy/inference/vmp/nodes/gaussian_markov_chain.py bayespy/inference/vmp/nodes/gp.py bayespy/inference/vmp/nodes/logistic.py bayespy/inference/vmp/nodes/logpdf.py bayespy/inference/vmp/nodes/mixture.py bayespy/inference/vmp/nodes/multinomial.py bayespy/inference/vmp/nodes/node.py bayespy/inference/vmp/nodes/point_estimate.py bayespy/inference/vmp/nodes/poisson.py bayespy/inference/vmp/nodes/stochastic.py bayespy/inference/vmp/nodes/wishart.py bayespy/inference/vmp/nodes/tests/__init__.py bayespy/inference/vmp/nodes/tests/test_bernoulli.py bayespy/inference/vmp/nodes/tests/test_beta.py bayespy/inference/vmp/nodes/tests/test_binomial.py bayespy/inference/vmp/nodes/tests/test_categorical.py bayespy/inference/vmp/nodes/tests/test_categorical_markov_chain.py bayespy/inference/vmp/nodes/tests/test_concatenate.py bayespy/inference/vmp/nodes/tests/test_deterministic.py bayespy/inference/vmp/nodes/tests/test_dirichlet.py bayespy/inference/vmp/nodes/tests/test_dot.py bayespy/inference/vmp/nodes/tests/test_gamma.py bayespy/inference/vmp/nodes/tests/test_gate.py bayespy/inference/vmp/nodes/tests/test_gaussian.py bayespy/inference/vmp/nodes/tests/test_gaussian_markov_chain.py bayespy/inference/vmp/nodes/tests/test_mixture.py bayespy/inference/vmp/nodes/tests/test_multinomial.py bayespy/inference/vmp/nodes/tests/test_node.py bayespy/inference/vmp/nodes/tests/test_poisson.py bayespy/inference/vmp/nodes/tests/test_wishart.py bayespy/inference/vmp/tests/__init__.py bayespy/inference/vmp/tests/test_annealing.py bayespy/inference/vmp/tests/test_transformations.py bayespy/nodes/__init__.py bayespy/tests/__init__.py bayespy/tests/test_plot.py bayespy/tests/baseline_images/test_plot/contour.png bayespy/tests/baseline_images/test_plot/gaussian_mixture.png bayespy/tests/baseline_images/test_plot/hinton_p.png bayespy/tests/baseline_images/test_plot/hinton_r.png bayespy/tests/baseline_images/test_plot/hinton_z.png bayespy/tests/baseline_images/test_plot/pdf.png bayespy/utils/__init__.py bayespy/utils/linalg.py bayespy/utils/misc.py bayespy/utils/optimize.py bayespy/utils/random.py bayespy/utils/covfunc/__init__.py bayespy/utils/covfunc/covariance.py bayespy/utils/tests/__init__.py bayespy/utils/tests/test_linalg.py bayespy/utils/tests/test_misc.py bayespy/utils/tests/test_random.py doc/Makefile doc/source/conf.py doc/source/demos.rst doc/source/index.rst doc/source/intro.rst doc/source/nodes.rst doc/source/references.rst doc/source/_templates/autosummary/class.rst doc/source/_templates/autosummary/module.rst doc/source/_templates/autosummary/short_module.rst doc/source/dev_api/dev_api.rst doc/source/dev_api/distributions.rst doc/source/dev_api/moments.rst doc/source/dev_api/nodes.rst doc/source/dev_api/utils.rst doc/source/dev_guide/advanced.rst doc/source/dev_guide/dev_guide.rst doc/source/dev_guide/engine.rst doc/source/dev_guide/vmp.rst doc/source/dev_guide/workflow.rst doc/source/dev_guide/writingnodes.rst doc/source/dev_guide/vmp/vmp_gamma.rst doc/source/dev_guide/vmp/vmp_gaussian.rst doc/source/dev_guide/vmp/vmp_gaussian_gamma.rst doc/source/dev_guide/vmp/vmp_gaussian_wishart.rst doc/source/dev_guide/vmp/vmp_mixture.rst doc/source/dev_guide/vmp/vmp_normal.rst doc/source/dev_guide/vmp/vmp_normal_gamma.rst doc/source/dev_guide/vmp/vmp_wishart.rst doc/source/examples/additive_fhmm.rst doc/source/examples/bmm.rst doc/source/examples/examples.rst doc/source/examples/gmm.rst doc/source/examples/hmm.rst doc/source/examples/lda.rst doc/source/examples/lssm.rst doc/source/examples/my_test.rst doc/source/examples/pca.rst doc/source/examples/regression.rst doc/source/user_api/user_api.rst doc/source/user_guide/advanced.rst doc/source/user_guide/inference.rst doc/source/user_guide/install.rst doc/source/user_guide/modelconstruct.rst doc/source/user_guide/plot.rst doc/source/user_guide/quickstart.rst doc/source/user_guide/quickstartbackup.py doc/source/user_guide/quickstartbackup.rst doc/source/user_guide/user_guide.rstbayespy-0.4.0/setup.cfg0000664000175000017500000000021612615647103016534 0ustar jluttinejluttine00000000000000[egg_info] tag_build = tag_date = 0 tag_svn_revision = 0 [aliases] release = egg_info -RDb '' release_pypi = release register sdist upload