bayespy-0.1/0000775000175000017500000000000012174043062014545 5ustar jluttinejluttine00000000000000bayespy-0.1/PKG-INFO0000664000175000017500000000165512174043062015651 0ustar jluttinejluttine00000000000000Metadata-Version: 1.1 Name: bayespy Version: 0.1 Summary: Bayesian inference tools for Python Home-page: http://bayespy.org Author: Jaakko Luttinen Author-email: jaakko.luttinen@iki.fi License: GPLv3 Description: Bayesian inference tools for Python. The package provides tools for building models and performing posterior inference using variational Bayesian message passing. Platform: UNKNOWN Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 3 Classifier: Development Status :: 2 - Pre-Alpha Classifier: Environment :: Console Classifier: Intended Audience :: Developers Classifier: Intended Audience :: Science/Research Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3) Classifier: Operating System :: OS Independent Classifier: Topic :: Scientific/Engineering Classifier: Topic :: Scientific/Engineering :: Information Analysis bayespy-0.1/bayespy.egg-info/0000775000175000017500000000000012174043062017713 5ustar jluttinejluttine00000000000000bayespy-0.1/bayespy.egg-info/SOURCES.txt0000664000175000017500000000670412174043023021603 0ustar jluttinejluttine00000000000000INSTALL.rst LICENSE MANIFEST.in README.rst setup.cfg setup.py bayespy/__init__.py bayespy.egg-info/PKG-INFO bayespy.egg-info/SOURCES.txt bayespy.egg-info/dependency_links.txt bayespy.egg-info/requires.txt bayespy.egg-info/top_level.txt bayespy/_nodes_dev/CovarianceFunctions.py bayespy/_nodes_dev/GaussianProcesses.py bayespy/_nodes_dev/__init__.py bayespy/_nodes_dev/categorical.py bayespy/_nodes_dev/constant.py bayespy/_nodes_dev/dirichlet.py bayespy/_nodes_dev/dot.py bayespy/_nodes_dev/gamma.py bayespy/_nodes_dev/gaussian.py bayespy/_nodes_dev/gaussian_markov_chain.py bayespy/_nodes_dev/mixture.py bayespy/_nodes_dev/node.py bayespy/_nodes_dev/normal.py bayespy/_nodes_dev/variable.py bayespy/_nodes_dev/wishart.py bayespy/demos/__init__.py bayespy/demos/demo_categorical.py bayespy/demos/demo_gaussian_markov_chain.py bayespy/demos/demo_gaussian_mixture.py bayespy/demos/demo_gp.py bayespy/demos/demo_gp_sparse.py bayespy/demos/demo_gpfa.py bayespy/demos/demo_kalman_filter.py bayespy/demos/demo_lssm.py bayespy/demos/demo_lssm_drift.py bayespy/demos/demo_pca.py bayespy/demos/demo_rotation.py bayespy/demos/demo_saving.py bayespy/inference/__init__.py bayespy/inference/vmp/__init__.py bayespy/inference/vmp/transformations.py bayespy/inference/vmp/vmp.py bayespy/inference/vmp/nodes/CovarianceFunctions.py bayespy/inference/vmp/nodes/GaussianProcesses.py bayespy/inference/vmp/nodes/__init__.py bayespy/inference/vmp/nodes/categorical.py bayespy/inference/vmp/nodes/constant.py bayespy/inference/vmp/nodes/deterministic.py bayespy/inference/vmp/nodes/dirichlet.py bayespy/inference/vmp/nodes/dot.py bayespy/inference/vmp/nodes/expfamily.py bayespy/inference/vmp/nodes/gamma.py bayespy/inference/vmp/nodes/gaussian.py bayespy/inference/vmp/nodes/gaussian_markov_chain.py bayespy/inference/vmp/nodes/mixture.py bayespy/inference/vmp/nodes/node.py bayespy/inference/vmp/nodes/node2.py bayespy/inference/vmp/nodes/normal.py bayespy/inference/vmp/nodes/stochastic.py bayespy/inference/vmp/nodes/variable.py bayespy/inference/vmp/nodes/wishart.py bayespy/inference/vmp/nodes/tests/__init__.py bayespy/inference/vmp/nodes/tests/test_deterministic.py bayespy/inference/vmp/nodes/tests/test_dot.py bayespy/inference/vmp/nodes/tests/test_gaussian_markov_chain.py bayespy/inference/vmp/nodes/tests/test_node.py bayespy/inference/vmp/tests/__init__.py bayespy/inference/vmp/tests/gp.py bayespy/inference/vmp/tests/vmp.py bayespy/plot/__init__.py bayespy/plot/plotting.py bayespy/utils/__init__.py bayespy/utils/linalg.py bayespy/utils/optimize.py bayespy/utils/random.py bayespy/utils/utils.py bayespy/utils/covfunc/__init__.py bayespy/utils/covfunc/covariance.py bayespy/utils/covfunc/distance.pyx bayespy/utils/covfunc/sparse_distance/sparse_distance.c bayespy/utils/covfunc/sparse_distance/sparse_distance.h bayespy/utils/covfunc/tests/__init__.py bayespy/utils/covfunc/tests/test_distance.py bayespy/utils/tests/__init__.py bayespy/utils/tests/test_linalg.py bayespy/utils/tests/test_utils.py doc/Makefile doc/source/api.rst doc/source/conf.py doc/source/devguide.rst doc/source/examples.rst doc/source/index.rst doc/source/install.rst doc/source/intro.rst doc/source/nodes.rst doc/source/userguide.rst doc/source/vmp.rst doc/source/examples/example_01.py doc/source/vmp/vmp_gamma.rst doc/source/vmp/vmp_gaussian.rst doc/source/vmp/vmp_gaussian_gamma.rst doc/source/vmp/vmp_gaussian_wishart.rst doc/source/vmp/vmp_mixture.rst doc/source/vmp/vmp_normal.rst doc/source/vmp/vmp_normal_gamma.rst doc/source/vmp/vmp_wishart.rstbayespy-0.1/bayespy.egg-info/PKG-INFO0000664000175000017500000000165512174043023021014 0ustar jluttinejluttine00000000000000Metadata-Version: 1.1 Name: bayespy Version: 0.1 Summary: Bayesian inference tools for Python Home-page: http://bayespy.org Author: Jaakko Luttinen Author-email: jaakko.luttinen@iki.fi License: GPLv3 Description: Bayesian inference tools for Python. The package provides tools for building models and performing posterior inference using variational Bayesian message passing. Platform: UNKNOWN Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 3 Classifier: Development Status :: 2 - Pre-Alpha Classifier: Environment :: Console Classifier: Intended Audience :: Developers Classifier: Intended Audience :: Science/Research Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3) Classifier: Operating System :: OS Independent Classifier: Topic :: Scientific/Engineering Classifier: Topic :: Scientific/Engineering :: Information Analysis bayespy-0.1/bayespy.egg-info/requires.txt0000664000175000017500000000007012174043023022305 0ustar jluttinejluttine00000000000000numpy>=1.7.1 scipy>=0.11.0 matplotlib>=1.2.0 cython h5pybayespy-0.1/bayespy.egg-info/dependency_links.txt0000664000175000017500000000000112174043023023756 0ustar jluttinejluttine00000000000000 bayespy-0.1/bayespy.egg-info/top_level.txt0000664000175000017500000000001012174043023022431 0ustar jluttinejluttine00000000000000bayespy bayespy-0.1/doc/0000775000175000017500000000000012174043062015312 5ustar jluttinejluttine00000000000000bayespy-0.1/doc/source/0000775000175000017500000000000012174043062016612 5ustar jluttinejluttine00000000000000bayespy-0.1/doc/source/userguide.rst0000664000175000017500000003450712174023546021356 0ustar jluttinejluttine00000000000000.. Copyright (C) 2011,2012 Jaakko Luttinen This file is licensed under Version 3.0 of the GNU General Public License. See LICENSE for a text of the license. This file is part of BayesPy. BayesPy is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License version 3 as published by the Free Software Foundation. BayesPy is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with BayesPy. If not, see . Quick user guide ================ * Construct the model (Bayesian network) * Put the data in * Run inference * Examine posterior results Simple example -------------- Let's begin with a simple example which shows the basic steps. In this case, we do not use any real data but generate some toy data. The dataset consists of ten samples from the Gaussian distribution with mean 5 and standard deviation 10: .. literalinclude:: examples/example_01.py :language: python3 :start-after: (0) :end-before: (1) Now, given this data we would like to estimate the mean and the standard deviation. We can construct a simple model shown below as a directed factor graph. .. bayesnet:: Directed factor graph of the example model. \node[obs] (y) {$y$} ; \node[latent, above left=1.5 and 0.5 of y] (mu) {$\mu$} ; \node[latent, above right=1.5 and 0.5 of y] (tau) {$\tau$} ; \node[const, above=of mu, xshift=-0.5cm] (mumu) {$0$} ; \node[const, above=of mu, xshift=0.5cm] (taumu) {$10^{-3}$} ; \node[const, above=of tau, xshift=-0.5cm] (atau) {$10^{-3}$} ; \node[const, above=of tau, xshift=0.5cm] (btau) {$10^{-3}$} ; \factor[above=of y] {y-f} {left:$\mathcal{N}$} {mu,tau} {y}; \factor[above=of mu] {} {left:$\mathcal{N}$} {mumu,taumu} {mu}; \factor[above=of tau] {} {left:$\mathcal{G}$} {atau,btau} {tau}; \plate {} {(y)(y-f)(y-f-caption)} {10} ; Alternatively, the model can also be defined using explicit mathematical notation: .. math:: p(\mathbf{y}|\mu,\tau) &= \prod^{10}_{n=1} \mathcal{N}(y_n|\mu,\tau) \\ p(\mu) &= \mathcal{N}(\mu|0,10^{-3}) \\ p(\tau) &= \mathcal{G}(\tau|10^{-3},10^{-3}) Note that we parameterize the normal distribution using the mean and the precision (i.e., the inverse of the variance). The model can be constructed in BayesPy as follows: .. literalinclude:: examples/example_01.py :language: python3 :start-after: (1) :end-before: (2) This is quite self-explanatory given the model definitions above. Now, we use the generated data: .. literalinclude:: examples/example_01.py :language: python3 :start-after: (2) :end-before: (3) Next we want to estimate the posterior distribution. In principle, we could use different inference engines (e.g., MCMC or EP) but currently only variational Bayesian (VB) engine is implemented. The engine is initialized by giving the nodes and the inference algorithm can be run as long as wanted (20 iterations in this case): .. literalinclude:: examples/example_01.py :language: python3 :start-after: (3) :end-before: (4) In VB, the true posterior :math:`p(\mu,\tau|\mathbf{y})` is approximated with a factorized distribution :math:`q(\mu)q(\tau)`. The resulting approximate posterior distributions :math:`q(\mu)` and :math:`q(\tau)` can be examined as: .. literalinclude:: examples/example_01.py :language: python3 :start-after: (4) .. todo:: Add an example of visualizing the results. This example was a very simple introduction to using BayesPy. The model can be much more complex and each phase contains more options to give the user more control over the inference. The following sections give more details. Constructing the model ---------------------- In BayesPy, the model is constructed by creating nodes which form a network. Roughly speaking, a node corresponds to a random variable from a specific probability distribution. In the example, ``mu`` was ``Normal`` node corresponding to :math:`\mu` from the normal distribution. However, a node can also correspond to a set of random variables or nodes can be deterministic not corresponding to any random variable. When you create a node, you give its parents as parameters. The role and the number of the parents depend on the node. For instance, ``Normal`` node takes two parents (mean and precision) and ``Gamma`` node takes two parents (scale and rate). .. warning:: Currently, it is important that the parent has the correct node type, because the model construction and VB inference engine are not yet separated. For instance, the parents mean and precision of ``Normal`` node must be ``Normal`` and ``Gamma`` nodes (or other nodes that have similar output), respectively. Thus, currently one can build only conjugate-exponential family models. Name and plates +++++++++++++++ In general, the nodes take some optional parameters: ``name`` and ``plates``. The parameter ``name`` is used to give a name for the variable. The parameter ``plates`` is used to define plates, that is, a repetitive collection of nodes that are independent given the parents. For instance, the following set of i.i.d. random variables .. math:: y_{mn} \sim \mathcal{N}(\mu, \tau),\qquad m=1,\ldots,10, \quad n=1,\ldots,30 would be created as .. code-block:: python3 y = bp.nodes.Normal(mu, tau, plates=(10,30)) It is also possible that the parents have plates. The validity of the plates between a child and a parent is checked by comparing the plates plate-wise from the trailing plates and working the way forward. A plate of the child is compatible with a plate of the parent if either of the following conditions is met: 1) The two plates have equal size 2) The parent has size 1 (or no plate) Table below shows an example of compatible plates for a child and two parent nodes. +---------+-------------------------+ | node | plates | +=========+===+===+====+===+===+====+ | parent1 | | 9 | 1 | 5 | 1 | 10 | +---------+---+---+----+---+---+----+ | parent2 | | | 15 | 5 | 1 | 1 | +---------+---+---+----+---+---+----+ | child | 5 | 9 | 15 | 5 | 1 | 10 | +---------+---+---+----+---+---+----+ For instance, a model .. math:: \mu_m &\sim \mathcal{N}(0, 10^{-3}), \\ \tau_n &\sim \mathcal{G}(10^{-3}, 10^{-3}), \\ y_{mn} &\sim \mathcal{N}(\mu_m, \tau_n),\qquad m=1,\ldots,10, \quad n=1,\ldots,30 could be created as .. code-block:: python3 mu = bp.nodes.Normal(0, 1e-3, plates=(10,1)) tau = bp.nodes.Gamma(1e-3, 1e-3, plates=(30,)) y = bp.nodes.Normal(mu, tau, plates=(10,30)) Multi-dimensional nodes +++++++++++++++++++++++ Sometimes a random variable is multi-dimensional. For instance, a multivariate normal distribution is a probability distribution for vectors. Quite often, the dimensionality can be deduced implicitly from the parents, thus the user may not need to provide it explicitly. However, it is important to know that the values are stored in a NumPy array where the plates are the leading axes and the dimensions are the trailing axes. This becomes relevant, for instance, when providing the data for an observed multi-dimensional node. To make a clear distinction between scalar and multi-dimensional distributions, there is often a multi-dimensional counterpart of a scalar node. For instance, the normal distribution for scalars is provided by the node ``Normal``, but the node for the multivariate normal distribution is ``Gaussian``. Below is a more complete table of correspondence. ============== ================== Scalar Multi-dimensional ============== ================== ``Normal`` ``Gaussian`` ``Gamma`` ``Wishart`` ``Bernoulli`` ``Categorical`` ``Binomial`` ``Multinomial`` ``Beta`` ``Dirichlet`` ============== ================== Deterministic and constant nodes ++++++++++++++++++++++++++++++++ In addition to the random variable nodes, there are two special types of nodes: constant and deterministic. Neither one has any probability distribution associated with them. A constant node has no parents and the value of the node is fixed. Constant nodes are created implicitly as parent nodes when you give numeric values as parents when creating a node, thus, the user is not required to create any constant nodes explicitly. A deterministic, on the other hand, defines some function. It transforms the parents to produce a new variable which is a non-random function of the parents. For instance, ``Dot`` node computes the dot product of its parents. Providing the data ------------------ The data is provided by simply calling ``observe`` method of the node: .. code-block:: python3 y.observe(data) It is important that the shape of the ``data`` array matches the shape of the node ``y``, which is the combination of the plates and the dimensionality. For instance, if ``y`` is ``Wishart`` node for :math:`3\times 3` matrices with plates ``(5,1,10)``, the actual shape of ``y`` would be ``(5,1,10,3,3)``. The data array must have this shape exactly, that is, no broadcasting rules are applied. Missing values ++++++++++++++ It is possible to mark missing values by providing a mask: .. code-block:: python3 y.observe(data, mask=[True, False, False, True, True, False, True, True, True, False]) ``True`` means that the value is observed and ``False`` means that the value is missing. To be more precise, the mask is applied to the plates, *not* to the data array directly. Unlike for the data itself, standard NumPy broadcasting rules are applied for the mask with respect to the plates. So, if the variable has plates ``(5,1,10)``, the mask could have a shape ``(1,)``, ``(10,)``, ``(5,1,1)`` or ``(5,1,10)``. From implementational point of view, the inference algorithms ignore the missing plates automatically if they are not needed. Thus, the missing values are integrated out giving more accurate approximations and the computations may also be faster. Performing inference -------------------- Approximation of the posterior distribution can be divided into several steps: * Choosing and constructing the inference engine * Initializing the engine * Running the inference algorithm Choosing the inference method +++++++++++++++++++++++++++++ Inference methods can be found in ``bayespy.inference`` package. Currently, only variational Bayesian approximation is implemented (``bayespy.inference.VB``). The inference engine is constructed by giving the nodes of the model. .. code-block:: python3 from bayespy.inference import VB Q = VB(node1, node2, node3, node4) Initializing the inference ++++++++++++++++++++++++++ The inference engines give some initialization to the nodes by default. However, the inference algorithms can be sensitive to the initialization, thus it is sometimes necessary to have full control over the initialization. There may be different initialization methods, but for VB you can, for instance, initialize in one of the following ways: * ``initialize_from_prior``: Use only parent nodes to update the node. * ``initialize_from_parameters``: Use the given parameter values for the distribution. A random initialization for VB has to be performed manually, because it is not obvious what is actually wanted. For instance, one way to achieve it is to first update from the parents, then to draw a random sample from that distribution and to set the values of the parameters based on that. For ``Normal`` node, one could draw the mean parameter randomly and choose the precision parameter arbitrarily: .. code-block:: python3 x = bp.nodes.Normal(mu, tau, plates=(10,)) x.initialize_from_prior() x.initialize_from_parameters(x.random(), 1) In this case, the precision was set to one. The default initialization method is ``initialization_from_prior``, which is performed when the node is created. If the initialization uses the values of the parents, they should be initialized before the children. .. For VB, a random initialization could perhaps be achieved as follows: Draw random samples of the parent parameters but extend this sampling to the plates of the child node. These values can be used to set the parameters of the child's distribution? No, it doesn't work: Consider constant parent nodes. Then the child would have constant values. Running the inference algorithm +++++++++++++++++++++++++++++++ The approximation methods are based on iterative algorithms, which can be run using ``update`` method. By default, it takes one iteration step updating all nodes once. However, you can give as arguments the nodes you want to update and they are updated in the given order. It is possible to give same nodes several times, for instance: .. code-block:: python3 Q.update(node1, node3, node1, node4) This would update ``node3`` and ``node4`` once, and ``node1`` twice. In order to update several times, one can use the optional argument ``repeat``. .. code-block:: python3 Q.update(node3, node4, repeat=5) Q.update(node1, node2, node3, node4, repeat=10) This first updates ``node3`` and ``node4`` five times and then all the nodes ten times. This might be useful, for instance, if updating some nodes is expensive and should be done rarely or if updating some nodes in the beginning would cause the algorithm to converge to a bad solution. .. warning:: Ideally, one constructs the model and then chooses the inference method to be used - possibly trying several different methods. However, the model construction is not yet separated from the model construction, that is, the constructed model network is also the variational message passing network for VB inference. Examining the results --------------------- After the results have been obtained, it is important to be able to examine the results easily. ``show`` method prints the approximate posterior distribution of the node. Also, ``get_moments`` can be used to obtain the sufficient statistics of the node. .. todo:: In order to examine the results more carefully, ``get_parameters`` method should return the parameter values of the approximate posterior distribution. The user may use these values for arbitrarily complex further analysis. bayespy-0.1/doc/source/vmp/0000775000175000017500000000000012174043062017414 5ustar jluttinejluttine00000000000000bayespy-0.1/doc/source/vmp/vmp_wishart.rst0000664000175000017500000000360212174013072022510 0ustar jluttinejluttine00000000000000Wishart distribution -------------------- .. math:: \mathbf{\Lambda} &\sim \mathcal{W}(n, \mathbf{V}), .. math:: n > D-1, \quad \mathbf{\Lambda}, \mathbf{V} \in \mathbb{R}^{D \times D}, \quad \mathbf{\Lambda}, \mathbf{V} \text{ symmetric positive definite} .. math:: \log\mathcal{W}( \mathbf{\Lambda} | n, \mathbf{V} ) &= - \frac{1}{2} \operatorname{tr} (\mathbf{\Lambda V}) + \frac{n}{2} \log |\mathbf{\Lambda}| + \frac{n}{2} \log |\mathbf{V}| - \frac{D+1}{2} \log |\mathbf{\Lambda}| - \frac{nD}{2} \log 2 - \log \Gamma_D \left(\frac{n}{2}\right) .. math:: \mathbf{u} (\mathbf{\Lambda}) &= \left[ \begin{matrix} \mathbf{\Lambda} \\ \log |\mathbf{\Lambda}| \end{matrix} \right] \\ \boldsymbol{\phi} (n, \mathbf{V}) &= \left[ \begin{matrix} - \frac{1}{2} \mathbf{V} \\ \frac{1}{2} n \end{matrix} \right] \\ \boldsymbol{\phi}_{n} (\mathbf{\Lambda}, \mathbf{V}) &= \left[ \begin{matrix} \frac{1}{2}\log|\mathbf{\Lambda}| + \frac{1}{2}\log|\mathbf{V}| + \frac{D}{2} \log 2 \\ -1 \end{matrix} \right] \\ \boldsymbol{\phi}_{\mathbf{V}} (\mathbf{\Lambda}, n) &= \left[ \begin{matrix} - \frac{1}{2} \mathbf{\Lambda} \\ \frac{1}{2} n \end{matrix} \right] \\ g (n, \mathbf{V}) &= \frac{n}{2} \log|\mathbf{V}| - \frac{nD}{2}\log 2 - \log \Gamma_D \left(\frac{n}{2}\right) \\ g_{\boldsymbol{\phi}} (\boldsymbol{\phi}) &= \boldsymbol{\phi}_2 \log|-\boldsymbol{\phi}_1| - \log \Gamma_D (\boldsymbol{\phi}_2) \\ f(\mathbf{\Lambda}) &= - \frac{D+1}{2} \log|\mathbf{\Lambda}| \\ \overline{\mathbf{u}} (\boldsymbol{\phi}) &= \left[ \begin{matrix} - \boldsymbol{\phi}_2 \boldsymbol{\phi}^{-1}_1 \\ - \log|-\boldsymbol{\phi}_1| + \psi_D(\boldsymbol{\phi}_2) \end{matrix} \right] bayespy-0.1/doc/source/vmp/vmp_gaussian_wishart.rst0000664000175000017500000000007512174013072024403 0ustar jluttinejluttine00000000000000Gaussian-Wishart distribution ----------------------------- bayespy-0.1/doc/source/vmp/vmp_gaussian_gamma.rst0000664000175000017500000000007112174013072024000 0ustar jluttinejluttine00000000000000 Gaussian-Gamma distribution --------------------------- bayespy-0.1/doc/source/vmp/vmp_gamma.rst0000664000175000017500000000004612174013072022110 0ustar jluttinejluttine00000000000000Gamma distribution ------------------ bayespy-0.1/doc/source/vmp/vmp_normal.rst0000664000175000017500000000007612174013072022321 0ustar jluttinejluttine00000000000000Univariate normal distribution ------------------------------ bayespy-0.1/doc/source/vmp/vmp_gaussian.rst0000664000175000017500000000464112174013072022645 0ustar jluttinejluttine00000000000000Multivariate normal distribution -------------------------------- .. math:: \mathbf{x} &\sim \mathcal{N}(\boldsymbol{\mu}, \mathbf{\Lambda}), .. math:: \mathbf{x},\boldsymbol{\mu} \in \mathbb{R}^{D}, \quad \mathbf{\Lambda} \in \mathbb{R}^{D \times D}, \quad \mathbf{\Lambda} \text{ symmetric positive definite} .. math:: \log\mathcal{N}( \mathbf{x} | \boldsymbol{\mu}, \mathbf{\Lambda} ) &= - \frac{1}{2} \mathbf{x}^{\mathrm{T}} \mathbf{\Lambda} \mathbf{x} + \mathbf{x}^{\mathrm{T}} \mathbf{\Lambda} \boldsymbol{\mu} - \frac{1}{2} \boldsymbol{\mu}^{\mathrm{T}} \mathbf{\Lambda} \boldsymbol{\mu} + \frac{1}{2} \log |\mathbf{\Lambda}| - \frac{D}{2} \log (2\pi) .. math:: \mathbf{u} (\mathbf{x}) &= \left[ \begin{matrix} \mathbf{x} \\ \mathbf{xx}^{\mathrm{T}} \end{matrix} \right] \\ \boldsymbol{\phi} (\boldsymbol{\mu}, \mathbf{\Lambda}) &= \left[ \begin{matrix} \mathbf{\Lambda} \boldsymbol{\mu} \\ - \frac{1}{2} \mathbf{\Lambda} \end{matrix} \right] \\ \boldsymbol{\phi}_{\boldsymbol{\mu}} (\mathbf{x}, \mathbf{\Lambda}) &= \left[ \begin{matrix} \mathbf{\Lambda} \mathbf{x} \\ - \frac{1}{2} \mathbf{\Lambda} \end{matrix} \right] \\ \boldsymbol{\phi}_{\mathbf{\Lambda}} (\mathbf{x}, \boldsymbol{\mu}) &= \left[ \begin{matrix} - \frac{1}{2} \mathbf{xx}^{\mathrm{T}} + \frac{1}{2} \mathbf{x}\boldsymbol{\mu}^{\mathrm{T}} + \frac{1}{2} \boldsymbol{\mu}\mathbf{x}^{\mathrm{T}} - \frac{1}{2} \boldsymbol{\mu\mu}^{\mathrm{T}} \\ \frac{1}{2} \end{matrix} \right] \\ g (\boldsymbol{\mu}, \mathbf{\Lambda}) &= - \frac{1}{2} \operatorname{tr}(\boldsymbol{\mu\mu}^{\mathrm{T}} \mathbf{\Lambda} ) + \frac{1}{2} \log |\mathbf{\Lambda}| \\ g_{\boldsymbol{\phi}} (\boldsymbol{\phi}) &= \frac{1}{4} \boldsymbol{\phi}^{\mathrm{T}}_1 \boldsymbol{\phi}^{-1}_2 \boldsymbol{\phi}_1 + \frac{1}{2} \log | -2 \boldsymbol{\phi}_2 | \\ f(\mathbf{x}) &= - \frac{D}{2} \log(2\pi) \\ \overline{\mathbf{u}} (\boldsymbol{\phi}) &= \left[ \begin{matrix} - \frac{1}{2} \boldsymbol{\phi}^{-1}_2 \boldsymbol{\phi}_1 \\ \frac{1}{4} \boldsymbol{\phi}^{-1}_2 \boldsymbol{\phi}_1 \boldsymbol{\phi}^{\mathrm{T}}_1 \boldsymbol{\phi}^{-1}_2 - \frac{1}{2} \boldsymbol{\phi}^{-1}_2 \end{matrix} \right] bayespy-0.1/doc/source/vmp/vmp_normal_gamma.rst0000664000175000017500000000006412174013072023460 0ustar jluttinejluttine00000000000000Normal-Gamma distribution ------------------------- bayespy-0.1/doc/source/vmp/vmp_mixture.rst0000664000175000017500000000632612174013072022532 0ustar jluttinejluttine00000000000000Mixture distribution -------------------- .. math:: \mathbf{x} &\sim \mathrm{Mix}_{\mathcal{D}} \left( \lambda, \left\{ \mathbf{\Theta}^{(n)}_1, \ldots, \mathbf{\Theta}^{(n)}_K \right\}^N_{n=1} \right) .. math:: \lambda \in \{1, \ldots, N\}, \quad \mathcal{D} \text{ is an exp.fam. distribution}, \quad \mathbf{\Theta}^{(n)}_k \text{ are parameters of } \mathcal{D} .. math:: \log\mathrm{Mix}_{\mathcal{D}} \left( \mathbf{x} \left| \lambda, \left\{ \mathbf{\Theta}^{(n)}_1, \ldots, \mathbf{\Theta}^{(n)}_K \right\}^N_{n=1} \right. \right) &= \sum^N_{n=1} [\lambda=n] \mathbf{u}_{\mathcal{D}}(\mathbf{x})^{\mathrm{T}} \boldsymbol{\phi}_{\mathcal{D}} \left( \mathbf{\Theta}^{(n)}_1, \ldots, \mathbf{\Theta}^{(n)}_K \right) \\ & \quad + \sum^N_{n=1} [\lambda=n] g_{\mathcal{D}} \left( \mathbf{\Theta}^{(n)}_1, \ldots, \mathbf{\Theta}^{(n)}_K \right) + f_{\mathcal{D}} (\mathbf{x}) .. math:: \mathbf{u} (\mathbf{x}) &= \mathbf{u}_{\mathcal{D}} (\mathbf{x}) \\ \boldsymbol{\phi} \left( \lambda, \left\{ \mathbf{\Theta}^{(n)}_1, \ldots, \mathbf{\Theta}^{(n)}_K \right\}^N_{n=1} \right) &= \sum^N_{n=1} [\lambda=n] \boldsymbol{\phi}_{\mathcal{D}} \left( \mathbf{\Theta}^{(n)}_1, \ldots, \mathbf{\Theta}^{(n)}_K \right) % \\ % \boldsymbol{\phi}_{\lambda} \left( \mathbf{x}, \left\{ \mathbf{\Theta}^{(n)}_1, \ldots, \mathbf{\Theta}^{(n)}_K \right\}^N_{n=1} \right) &= \left[\begin{matrix} \mathbf{u}_{\mathcal{D}} (\mathbf{x})^{\mathrm{T}} \boldsymbol{\phi}_{\mathcal{D}} \left( \mathbf{\Theta}^{(1)}_1, \ldots, \mathbf{\Theta}^{(1)}_K \right) + g_{\mathcal{D}} \left( \mathbf{\Theta}^{(1)}_1, \ldots, \mathbf{\Theta}^{(1)}_K \right) \\ \vdots \\ \mathbf{u}_{\mathcal{D}} (\mathbf{x})^{\mathrm{T}} \boldsymbol{\phi}_{\mathcal{D}} \left( \mathbf{\Theta}^{(N)}_1, \ldots, \mathbf{\Theta}^{(N)}_K \right) + g_{\mathcal{D}} \left( \mathbf{\Theta}^{(N)}_1, \ldots, \mathbf{\Theta}^{(N)}_K \right) \end{matrix}\right] % \\ % \boldsymbol{\phi}_{\mathbf{\Theta}^{(m)}_l} \left( \mathbf{x}, \lambda, \left\{ \mathbf{\Theta}^{(n)}_1, \ldots, \mathbf{\Theta}^{(n)}_K \right\}^N_{n=1} \setminus \left\{ \mathbf{\Theta}^{(m)}_l \right\} \right) &= [\lambda=m] \boldsymbol{\phi}_{\mathcal{D}\rightarrow\mathbf{\Theta}_l} \left( \mathbf{x}, \left\{ \mathbf{\Theta}^{(m)}_k \right\}_{k\neq l} \right) % \\ % g \left( \lambda, \left\{ \mathbf{\Theta}^{(n)}_1, \ldots, \mathbf{\Theta}^{(n)}_K \right\}^N_{n=1} \right) &= \sum^N_{n=1} [\lambda=n] g_{\mathcal{D}} \left( \mathbf{\Theta}^{(n)}_1, \ldots, \mathbf{\Theta}^{(n)}_K \right) \\ g (\boldsymbol{\phi}) &= g_{\mathcal{D}} (\boldsymbol{\phi}) \\ f(\mathbf{x}) &= f_{\mathcal{D}} (\mathbf{x}) \\ \overline{\mathbf{u}} (\boldsymbol{\phi}) &= \overline{\mathbf{u}}_{\mathcal{D}} (\boldsymbol{\phi}) bayespy-0.1/doc/source/conf.py0000664000175000017500000002335312174013072020115 0ustar jluttinejluttine00000000000000# -*- coding: utf-8 -*- # # BayesPy documentation build configuration file, created by # sphinx-quickstart on Mon Aug 27 12:22:11 2012. # # This file is execfile()d with the current directory set to its containing dir. # # Note that not all possible configuration values are present in this # autogenerated file. # # All configuration values have a default; values that are commented out # serve to show the default. import sys, os # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. #sys.path.insert(0, os.path.abspath('.')) # Import some information from the setup.py script. sys.path.insert(0, os.path.abspath('../..')) import setup as setupfile # -- General configuration ----------------------------------------------------- # If your documentation needs a minimal Sphinx version, state it here. #needs_sphinx = '1.0' # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = [ 'sphinx.ext.autodoc', 'sphinx.ext.mathjax', 'sphinx.ext.todo', 'sphinx.ext.autosummary', 'sphinx.ext.viewcode', 'sphinxcontrib.tikz', 'sphinxcontrib.bayesnet', 'numpydoc' ] # Include TODOs in the documentation? todo_include_todos = True # Add a path to binary files that are necessary for readthedocs.org to # build HTML with tikz extension if os.environ.get('READTHEDOCS', None) == 'True': os.environ["PATH"] += os.pathsep + os.path.abspath('bin') os.environ["LD_LIBRARY_PATH"] = os.path.abspath('bin') # Generate autosummary stub pages automatically #autosummary_generate = False autosummary_generate = True # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # The suffix of source filenames. source_suffix = '.rst' # The encoding of source files. #source_encoding = 'utf-8-sig' # The master toctree document. master_doc = 'index' # General information about the project. project = setupfile.NAME copyright = u'2011-2013, Jaakko Luttinen' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The short X.Y version. version = setupfile.VERSION # The full version, including alpha/beta/rc tags. release = setupfile.VERSION # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. #language = None # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: #today = '' # Else, today_fmt is used as the format for a strftime call. #today_fmt = '%B %d, %Y' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. exclude_patterns = [] # The reST default role (used for this markup: `text`) to use for all documents. #default_role = None # If true, '()' will be appended to :func: etc. cross-reference text. #add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). #add_module_names = True # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. #show_authors = False # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' # A list of ignored prefixes for module index sorting. #modindex_common_prefix = [] # -- Options for HTML output --------------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. html_theme = 'default' # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. #html_theme_options = {} # Add any paths that contain custom themes here, relative to this directory. #html_theme_path = [] # The name for this set of Sphinx documents. If None, it defaults to # " v documentation". #html_title = None # A shorter title for the navigation bar. Default is the same as html_title. #html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. #html_logo = None # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. #html_favicon = None # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['_static'] # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, # using the given strftime format. #html_last_updated_fmt = '%b %d, %Y' # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. #html_use_smartypants = True # Custom sidebar templates, maps document names to template names. #html_sidebars = {} # Additional templates that should be rendered to pages, maps page names to # template names. #html_additional_pages = {} # If false, no module index is generated. #html_domain_indices = True # If false, no index is generated. #html_use_index = True # If true, the index is split into individual pages for each letter. #html_split_index = False # If true, links to the reST sources are added to the pages. #html_show_sourcelink = True # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. #html_show_sphinx = True # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. #html_show_copyright = True # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. #html_use_opensearch = '' # This is the file name suffix for HTML files (e.g. ".xhtml"). #html_file_suffix = None # Output file base name for HTML help builder. htmlhelp_basename = 'BayesPydoc' # -- Options for LaTeX output -------------------------------------------------- latex_elements = { # The paper size ('letterpaper' or 'a4paper'). #'papersize': 'letterpaper', # The font size ('10pt', '11pt' or '12pt'). #'pointsize': '10pt', # Additional stuff for the LaTeX preamble. 'preamble': r''' \usepackage{tikz} \usepackage{amssymb} ''' # \usetikzlibrary{shapes, fit, chains, arrows} # \usetikzlibrary{bayesnet} } #latex_additional_files = ['images/bayesnet.sty',] # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, author, documentclass [howto/manual]). latex_documents = [ ('index', 'BayesPy.tex', u'BayesPy Documentation', u'Jaakko Luttinen', 'manual'), ] # The name of an image file (relative to this directory) to place at the top of # the title page. #latex_logo = None # For "manual" documents, if this is true, then toplevel headings are parts, # not chapters. #latex_use_parts = False # If true, show page references after internal links. #latex_show_pagerefs = False # If true, show URL addresses after external links. #latex_show_urls = False # Documents to append as an appendix to all manuals. #latex_appendices = [] # If false, no module index is generated. #latex_domain_indices = True # -- Options for manual page output -------------------------------------------- # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ ('index', 'bayespy', u'BayesPy Documentation', [u'Jaakko Luttinen'], 1) ] # If true, show URL addresses after external links. #man_show_urls = False # -- Options for Texinfo output ------------------------------------------------ # Grouping the document tree into Texinfo files. List of tuples # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ ('index', 'BayesPy', u'BayesPy Documentation', u'Jaakko Luttinen', 'BayesPy', 'One line description of project.', 'Miscellaneous'), ] # Documents to append as an appendix to all manuals. #texinfo_appendices = [] # If false, no module index is generated. #texinfo_domain_indices = True # How to display URL addresses: 'footnote', 'no', or 'inline'. #texinfo_show_urls = 'footnote' # -- Options for Epub output --------------------------------------------------- # Bibliographic Dublin Core info. epub_title = u'BayesPy' epub_author = u'Jaakko Luttinen' epub_publisher = u'Jaakko Luttinen' epub_copyright = u'2012, Jaakko Luttinen' # The language of the text. It defaults to the language option # or en if the language is not set. #epub_language = '' # The scheme of the identifier. Typical schemes are ISBN or URL. #epub_scheme = '' # The unique identifier of the text. This can be a ISBN number # or the project homepage. #epub_identifier = '' # A unique identification for the text. #epub_uid = '' # A tuple containing the cover image and cover page html template filenames. #epub_cover = () # HTML files that should be inserted before the pages created by sphinx. # The format is a list of tuples containing the path and title. #epub_pre_files = [] # HTML files shat should be inserted after the pages created by sphinx. # The format is a list of tuples containing the path and title. #epub_post_files = [] # A list of files that should not be packed into the epub file. #epub_exclude_files = [] # The depth of the table of contents in toc.ncx. #epub_tocdepth = 3 # Allow duplicate toc entries. #epub_tocdup = True bayespy-0.1/doc/source/api.rst0000664000175000017500000000266512174013072020124 0ustar jluttinejluttine00000000000000.. Copyright (C) 2011,2012 Jaakko Luttinen This file is licensed under Version 3.0 of the GNU General Public License. See LICENSE for a text of the license. This file is part of BayesPy. BayesPy is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License version 3 as published by the Free Software Foundation. BayesPy is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with BayesPy. If not, see . API *** Stochastic nodes in VMP ======================= .. currentmodule:: bayespy.inference.vmp.nodes .. autosummary:: :toctree: generated/ normal.Normal gamma.Gamma gaussian.Gaussian wishart.Wishart dirichlet.Dirichlet categorical.Categorical mixture.Mixture gaussian_markov_chain.GaussianMarkovChain Deterministic ============= .. autosummary:: :toctree: generated/ dot.Dot Utility functions ================= .. currentmodule:: bayespy.utils.utils .. autosummary:: :toctree: generated/ kalman_filter rts_smoother .. currentmodule:: bayespy .. autosummary:: :toctree: generated/ demos inference utils.utils bayespy-0.1/doc/source/examples/0000775000175000017500000000000012174043062020430 5ustar jluttinejluttine00000000000000bayespy-0.1/doc/source/examples/example_01.py0000664000175000017500000000067612174013072022744 0ustar jluttinejluttine00000000000000# (0) Generate data import numpy as np data = np.random.normal(5, 10, size=(10,)) # (1) Construct the model import bayespy as bp mu = bp.nodes.Normal(0, 1e-3) tau = bp.nodes.Gamma(1e-3, 1e-3) y = bp.nodes.Normal(mu, tau, plates=(10,)) # (2) Observe data y.observe(data) # (3) Run inference from bayespy.inference import VB Q = VB(mu, tau, y) Q.update(repeat=20) # (4) Show posterior results mu.show() tau.show() import bayespy.plotting as plt bayespy-0.1/doc/source/intro.rst0000664000175000017500000000151112174013072020473 0ustar jluttinejluttine00000000000000.. Copyright (C) 2011,2012 Jaakko Luttinen This file is licensed under Version 3.0 of the GNU General Public License. See LICENSE for a text of the license. This file is part of BayesPy. BayesPy is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License version 3 as published by the Free Software Foundation. BayesPy is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with BayesPy. If not, see . Introduction ============ .. include:: ../../README.rst :start-line: 2 bayespy-0.1/doc/source/devguide.rst0000664000175000017500000000166012174013072021141 0ustar jluttinejluttine00000000000000.. Copyright (C) 2011,2012 Jaakko Luttinen This file is licensed under Version 3.0 of the GNU General Public License. See LICENSE for a text of the license. This file is part of BayesPy. BayesPy is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License version 3 as published by the Free Software Foundation. BayesPy is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with BayesPy. If not, see . Developer guide =============== How to document: https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt Variational message passing --------------------------- bayespy-0.1/doc/source/nodes.rst0000664000175000017500000000220012174013072020444 0ustar jluttinejluttine00000000000000.. Copyright (C) 2011,2012 Jaakko Luttinen This file is licensed under Version 3.0 of the GNU General Public License. See LICENSE for a text of the license. This file is part of BayesPy. BayesPy is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License version 3 as published by the Free Software Foundation. BayesPy is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with BayesPy. If not, see . Nodes ===== Stochastic nodes ---------------- Normal ++++++ Gaussian ++++++++ Gamma +++++ Wishart +++++++ Bernoulli +++++++++ Categorical +++++++++++ Binomial ++++++++ Multinomial +++++++++++ Beta ++++ Dirichlet +++++++++ Mixture +++++++ Constant ++++++++ Deterministic nodes ------------------- Dot product (of Gaussian variables) +++++++++++++++++++++++++++++++++++ bayespy-0.1/doc/source/vmp.rst0000664000175000017500000001262712174013072020154 0ustar jluttinejluttine00000000000000.. Copyright (C) 2012 Jaakko Luttinen This file is licensed under Version 3.0 of the GNU General Public License. See LICENSE for a text of the license. This file is part of BayesPy. BayesPy is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License version 3 as published by the Free Software Foundation. BayesPy is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with BayesPy. If not, see . Variational message passing =========================== The general update equation for factorized approximation: .. math:: :label: vmp_general_update \log q(\boldsymbol{\theta}) &= \langle \log p\left( \boldsymbol{\theta} | \mathrm{pa}(\boldsymbol{\theta}) \right) \rangle + \sum_{\mathbf{x} \in \mathrm{ch}(\boldsymbol{\theta})} \langle \log p(\mathbf{x}|\mathrm{pa}(\mathbf{x})) \rangle + \mathrm{const}, where :math:`\mathrm{pa}(\boldsymbol{\theta})` and :math:`\mathrm{ch}(\boldsymbol{\theta})` are the set of parents and children of :math:`\boldsymbol{\theta}`, respectively. The expectations are over the approximate distribution of all other variables than :math:`\boldsymbol{\theta}`. Actually, not all the variables are needed, because the non-constant part uses only the Markov blanket of :math:`\boldsymbol{\theta}`. Thus, the optimization can be done locally using messages from neighbouring nodes. The messages are simple for conjugate-exponential models. Exponential-family distributions have the form .. math:: :label: likelihood \log p(\mathbf{x}|\mathbf{\Theta}) &= \mathbf{u}_{\mathbf{x}}(\mathbf{x})^{\mathrm{T}} \boldsymbol{\phi}_{\mathbf{x}}(\mathbf{\Theta}) + g_{\mathbf{x}}(\mathbf{\Theta}) + f_{\mathbf{x}}(\mathbf{x}), where :math:`\mathbf{\Theta}=\{\boldsymbol{\theta}_j\}` is the set of parents. If a parent has a conjugate prior, :eq:`likelihood` is linear with respect to the parent's natural statistics. Thus, :eq:`likelihood` can be re-organized with respect to :math:`\boldsymbol{\theta}_j` as .. math:: \log p(\mathbf{x}|\mathbf{\Theta}) &= \mathbf{u}_{\boldsymbol{\theta}_j}(\boldsymbol{\theta}_j)^{\mathrm{T}} \boldsymbol{\phi}_{\mathbf{x}\rightarrow\boldsymbol{\theta}_j} (\mathbf{x}, \{\boldsymbol{\theta}_k\}_{k\neq j}) + \mathrm{const}, where :math:`\mathbf{u}_{\boldsymbol{\theta}_j}` is the natural statistics of :math:`\boldsymbol{\theta}_j`. Thus, the update equation :eq:`vmp_general_update` can be given as .. math:: \log q(\boldsymbol{\theta}_j) &= \mathbf{u}_{\boldsymbol{\theta}_j}(\boldsymbol{\theta}_j)^{\mathrm{T}} \left( \langle \boldsymbol{\phi}_{\boldsymbol{\theta}_j} \rangle + \sum_{\mathbf{x} \in \mathrm{ch}(\boldsymbol{\theta}_j)} \langle \boldsymbol{\phi}_{\mathbf{x}\rightarrow\boldsymbol{\theta}_j} \rangle \right) + f_{\boldsymbol{\theta}_j}(\boldsymbol{\theta}_j) + \mathrm{const}, where the summation is over all the child nodes of :math:`\boldsymbol{\theta}_j`. Because of the conjugacy, :math:`\langle\boldsymbol{\phi}_{\boldsymbol{\theta}_j}\rangle` depends (multi)linearly on the expectations of the parents' natural statistics. Similarly, :math:`\langle \boldsymbol{\phi}_{\mathbf{x}\rightarrow\boldsymbol{\theta}_j} \rangle` depends (multi)linearly on the expectations of the children's and co-parents' natural statistics. The required expectations can be computed locally by using messages from the parents and the children. The message from a parent node :math:`\boldsymbol{\theta}_j` to a child node :math:`\mathbf{x}` is .. math:: \mathbf{m}_{\mathbf{\boldsymbol{\theta}_j}\rightarrow\mathbf{x}} &= \langle \mathbf{u}_{\boldsymbol{\theta}_j} \rangle = \tilde{\mathbf{u}}_{\boldsymbol{\theta}_j} (\tilde{\boldsymbol{\phi}}_{\mathbf{\boldsymbol{\theta}_j}}), and the message from a child node :math:`\mathbf{x}` to a parent node :math:`\boldsymbol{\theta}_j` is .. math:: \mathbf{m}_{\mathbf{x}\rightarrow\boldsymbol{\theta}_j} &= \langle \boldsymbol{\phi}_{\mathbf{x}\rightarrow\boldsymbol{\theta}_j} \rangle = \boldsymbol{\phi}_{\mathbf{x}\rightarrow\boldsymbol{\theta}_j} \left( \langle \mathbf{u}_{\mathbf{x}} \rangle, \{ \mathbf{m}_{\theta_k\rightarrow\mathbf{x}} \}_ %\{ \langle \mathbf{u}_{\theta_k} \rangle \}_ {k \neq j} %\in \mathrm{cp}({\boldsymbol{\theta}_j}, \mathbf{x})} \right). Using the messages, the natural parameters of :math:`q(\boldsymbol{\theta})` can be updated as .. math:: \tilde{\boldsymbol{\phi}}_{\boldsymbol{\theta}} &= \boldsymbol{\phi}_{\boldsymbol{\theta}} \left( \{ \mathbf{m}_{\mathbf{z}\rightarrow\boldsymbol{\theta}} \}_ {\mathbf{z} \in \mathrm{pa}(\boldsymbol{\theta})} \right) + \sum_{\mathbf{x} \in \mathrm{ch}(\boldsymbol{\theta})} \mathbf{m}_{\mathbf{x}\rightarrow\boldsymbol{\theta}}. .. include:: vmp/vmp_normal.rst .. include:: vmp/vmp_gaussian.rst .. include:: vmp/vmp_gamma.rst .. include:: vmp/vmp_wishart.rst .. include:: vmp/vmp_normal_gamma.rst .. include:: vmp/vmp_gaussian_wishart.rst .. include:: vmp/vmp_gaussian_gamma.rst .. include:: vmp/vmp_mixture.rst bayespy-0.1/doc/source/examples.rst0000664000175000017500000001674312174035761021204 0ustar jluttinejluttine00000000000000.. Copyright (C) 2011,2012 Jaakko Luttinen This file is licensed under Version 3.0 of the GNU General Public License. See LICENSE for a text of the license. This file is part of BayesPy. BayesPy is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License version 3 as published by the Free Software Foundation. BayesPy is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with BayesPy. If not, see . .. Examples Principal component analysis TODO Gaussian mixture model TODO Example: Linear state-space model ================================= In linear state-space models a sequence of :math:`M`-dimensional observations :math:`\mathbf{Y}=(\mathbf{y}_1,\ldots,\mathbf{y}_N)` is assumed to be generated from latent :math:`D`-dimensional states :math:`\mathbf{X}=(\mathbf{x}_1,\ldots,\mathbf{x}_N)` which follow a first-order Markov process: .. math:: \begin{align} \mathbf{x}_{n} &= \mathbf{A}\mathbf{x}_{n-1} + \text{noise} \,, \\ \mathbf{y}_{n} &= \mathbf{C}\mathbf{x}_{n} + \text{noise} \,, \end{align} where the noise is Gaussian, :math:`\mathbf{A}` is the :math:`D\times D` state dynamics matrix and :math:`\mathbf{C}` is the :math:`M\times D` loading matrix. Usually, the latent space dimensionality :math:`D` is assumed to be much smaller than the observation space dimensionality :math:`M` in order to model the dependencies of high-dimensional observations efficiently. .. todo:: Add a graphical model and model definition, more detailed comments about the code First, let us generate some toy data: .. code-block:: python import numpy as np # # Simulate some data # M = 30 N = 400 w = 0.3 a = np.array([[np.cos(w), -np.sin(w), 0, 0], [np.sin(w), np.cos(w), 0, 0], [0, 0, 1, 0], [0, 0, 0, 0]]) c = np.random.randn(M,4) x = np.empty((N,4)) f = np.empty((M,N)) y = np.empty((M,N)) x[0] = 10*np.random.randn(4) f[:,0] = np.dot(c,x[0]) y[:,0] = f[:,0] + 3*np.random.randn(M) for n in range(N-1): x[n+1] = np.dot(a,x[n]) + np.random.randn(4) f[:,n+1] = np.dot(c,x[n+1]) y[:,n+1] = f[:,n+1] + 3*np.random.randn(M) The linear state-space model can be constructed as follows: .. code-block:: python from bayespy.inference.vmp.nodes.gaussian_markov_chain import GaussianMarkovChain from bayespy.inference.vmp.nodes.gaussian import Gaussian from bayespy.inference.vmp.nodes.gamma import Gamma from bayespy.inference.vmp.nodes.normal import Normal from bayespy.inference.vmp.nodes.dot import Dot from bayespy.inference.vmp.nodes.gamma import diagonal D = 10 # Dynamics matrix with ARD alpha = Gamma(1e-5, 1e-5, plates=(D,), name='alpha') A = Gaussian(np.zeros(D), diagonal(alpha), plates=(D,), name='A') # Latent states with dynamics X = GaussianMarkovChain(np.zeros(D), # mean of x0 1e-3*np.identity(D), # prec of x0 A, # dynamics np.ones(D), # innovation n=N, # time instances name='X', initialize=False) X.initialize_from_value(np.zeros((N,D))) # just some empty values, X is # updated first anyway # Mixing matrix from latent space to observation space using ARD gamma = Gamma(1e-5, 1e-5, plates=(D,), name='gamma') C = Gaussian(np.zeros(D), diagonal(gamma), plates=(M,1), name='C') # Initialize nodes (must use some randomness for C, and update X before C) C.initialize_from_random() # Observation noise tau = Gamma(1e-5, 1e-5, name='tau') # Observations CX = Dot(C, X.as_gaussian(), name='CX') Y = Normal(CX, tau, name='Y') An inference machine using variational Bayesian inference with variational message passing is then construced as .. code-block:: python from bayespy.inference.vmp.vmp import VB Q = VB(X, C, gamma, A, alpha, tau, Y) Observe the data partially (80% is marked missing): .. code-block:: python from bayespy.utils import random # Add missing values randomly (keep only 20%) mask = random.mask(M, N, p=0.2) Y.observe(y, mask=mask) Then inference (100 iterations) can be run simply as .. code-block:: python Q.update(repeat=100) Speeding up with parameter expansion ------------------------------------ VB inference can converge extremely slowly if the variables are strongly coupled. Because VMP updates one variable at a time, it may lead to slow zigzagging. This can be solved by using parameter expansion which reduces the coupling. In state-space models, the states :math:`\mathbf{x}_n` and the loadings :math:`\mathbf{C}` are coupled through a dot product :math:`\mathbf{Cx}_n`, which is unaltered if the latent space is rotated arbitrarily: .. math:: \mathbf{y}_n &= \mathbf{C}\mathbf{x}_n = \mathbf{C}\mathbf{R}^{-1}\mathbf{R}\mathbf{x}_n \,. Thus, one intuitive transformation would be :math:`\mathbf{C}\rightarrow\mathbf{C}\mathbf{R}^{-1}` and :math:`\mathbf{X}\rightarrow\mathbf{R}\mathbf{X}`. In order to keep the dynamics of the latent states unaffected by the transformation, the state dynamics matrix :math:`\mathbf{A}` must be transformed accordingly: .. math:: \mathbf{R}\mathbf{x}_n &= \mathbf{R}\mathbf{A}\mathbf{R}^{-1} \mathbf{R}\mathbf{x}_{n-1} \,, resulting in a transformation :math:`\mathbf{A}\rightarrow\mathbf{R}\mathbf{A}\mathbf{R}^{-1}`. For more details, refer to *Fast Variational Bayesian Linear State-Space Model (Luttinen, 2013)*. In BayesPy, the transformations can be used as follows: .. code-block:: python # Import the parameter expansion module from bayespy.inference.vmp import transformations # Rotator of the state dynamics matrix rotA = transformations.RotateGaussianARD(Q['A'], Q['alpha']) # Rotator of the states (includes rotation of the state dynamics matrix) rotX = transformations.RotateGaussianMarkovChain(Q['X'], Q['A'], rotA) # Rotator of the loading matrix rotC = transformations.RotateGaussianARD(Q['C'], Q['gamma']) # Rotation optimizer R = transformations.RotationOptimizer(rotX, rotC, D) Note that it is crucial to select the correct rotation class which corresponds to the particular model block exactly. The rotation can be performed after each full VB update: .. code-block:: python for ind in range(100): Q.update() R.rotate() If you want to implement your own rotations or check the existing ones, you may use debugging utilities: .. code-block:: python for ind in range(100): Q.update() R.rotate(check_bound=Q.compute_lowerbound, check_bound_terms=Q.compute_lowerbound_terms, check_gradient=True) bayespy-0.1/doc/source/install.rst0000664000175000017500000000143612174013072021014 0ustar jluttinejluttine00000000000000.. Copyright (C) 2011,2012 Jaakko Luttinen This file is licensed under Version 3.0 of the GNU General Public License. See LICENSE for a text of the license. This file is part of BayesPy. BayesPy is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License version 3 as published by the Free Software Foundation. BayesPy is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with BayesPy. If not, see . .. include:: ../../INSTALL.rst bayespy-0.1/doc/source/index.rst0000664000175000017500000000204712174021062020452 0ustar jluttinejluttine00000000000000.. Copyright (C) 2011,2012 Jaakko Luttinen This file is licensed under Version 3.0 of the GNU General Public License. See LICENSE for a text of the license. This file is part of BayesPy. BayesPy is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License version 3 as published by the Free Software Foundation. BayesPy is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with BayesPy. If not, see . Documentation for BayesPy -- Bayesian Python ============================================ .. toctree:: :maxdepth: 2 intro install userguide examples devguide nodes vmp api Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * :ref:`search` bayespy-0.1/doc/Makefile0000664000175000017500000001354612174013072016761 0ustar jluttinejluttine00000000000000# Makefile for Sphinx documentation # # You can set these variables from the command line. SPHINXOPTS = # Use Python 3 version of Sphinx (or use virtualenv!) #SPHINXBUILD = /usr/share/sphinx/scripts/python3/sphinx-build SPHINXBUILD = sphinx-build PAPER = BUILDDIR = build # Internal variables. PAPEROPT_a4 = -D latex_paper_size=a4 PAPEROPT_letter = -D latex_paper_size=letter ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source # the i18n builder cannot share the environment and doctrees with the others I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext help: @echo "Please use \`make ' where is one of" @echo " html to make standalone HTML files" @echo " dirhtml to make HTML files named index.html in directories" @echo " singlehtml to make a single large HTML file" @echo " pickle to make pickle files" @echo " json to make JSON files" @echo " htmlhelp to make HTML files and a HTML help project" @echo " qthelp to make HTML files and a qthelp project" @echo " devhelp to make HTML files and a Devhelp project" @echo " epub to make an epub" @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" @echo " latexpdf to make LaTeX files and run them through pdflatex" @echo " text to make text files" @echo " man to make manual pages" @echo " texinfo to make Texinfo files" @echo " info to make Texinfo files and run them through makeinfo" @echo " gettext to make PO message catalogs" @echo " changes to make an overview of all changed/added/deprecated items" @echo " linkcheck to check all external links for integrity" @echo " doctest to run all doctests embedded in the documentation (if enabled)" clean: -rm -rf $(BUILDDIR)/* source/generated/* html: $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." dirhtml: $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." singlehtml: $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml @echo @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." pickle: $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle @echo @echo "Build finished; now you can process the pickle files." json: $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json @echo @echo "Build finished; now you can process the JSON files." htmlhelp: $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp @echo @echo "Build finished; now you can run HTML Help Workshop with the" \ ".hhp project file in $(BUILDDIR)/htmlhelp." qthelp: $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp @echo @echo "Build finished; now you can run "qcollectiongenerator" with the" \ ".qhcp project file in $(BUILDDIR)/qthelp, like this:" @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/BayesPy.qhcp" @echo "To view the help file:" @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/BayesPy.qhc" devhelp: $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp @echo @echo "Build finished." @echo "To view the help file:" @echo "# mkdir -p $$HOME/.local/share/devhelp/BayesPy" @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/BayesPy" @echo "# devhelp" epub: $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub @echo @echo "Build finished. The epub file is in $(BUILDDIR)/epub." latex: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." @echo "Run \`make' in that directory to run these through (pdf)latex" \ "(use \`make latexpdf' here to do that automatically)." latexpdf: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through pdflatex..." $(MAKE) -C $(BUILDDIR)/latex all-pdf @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." text: $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text @echo @echo "Build finished. The text files are in $(BUILDDIR)/text." man: $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man @echo @echo "Build finished. The manual pages are in $(BUILDDIR)/man." texinfo: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." @echo "Run \`make' in that directory to run these through makeinfo" \ "(use \`make info' here to do that automatically)." info: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo "Running Texinfo files through makeinfo..." make -C $(BUILDDIR)/texinfo info @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." gettext: $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale @echo @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." changes: $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes @echo @echo "The overview file is in $(BUILDDIR)/changes." linkcheck: $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck @echo @echo "Link check complete; look for any errors in the above output " \ "or in $(BUILDDIR)/linkcheck/output.txt." doctest: $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest @echo "Testing of doctests in the sources finished, look at the " \ "results in $(BUILDDIR)/doctest/output.txt." gh-pages: rm -rf build/html git clone ../ build/html cd build/html && git checkout gh-pages && pwd make html cd build/html && \ git add -A && \ git commit -m "Generated gh-pages for `git log master -1 --pretty=short --abbrev-commit`" && \ git push origin gh-pages bayespy-0.1/bayespy/0000775000175000017500000000000012174043062016221 5ustar jluttinejluttine00000000000000bayespy-0.1/bayespy/inference/0000775000175000017500000000000012174043062020157 5ustar jluttinejluttine00000000000000bayespy-0.1/bayespy/inference/vmp/0000775000175000017500000000000012174043062020761 5ustar jluttinejluttine00000000000000bayespy-0.1/bayespy/inference/vmp/vmp.py0000664000175000017500000002354212174013072022141 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011-2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np import matplotlib.pyplot as plt import warnings import time import h5py import datetime import tempfile from bayespy import utils from bayespy.inference.vmp.nodes.node import Node class VB(): def __init__(self, *nodes, tol=1e-6, autosave_iterations=0, autosave_filename=None, callback=None): # Remove duplicate nodes self.model = utils.utils.unique(nodes) self.iter = 0 self.L = np.array(()) self.l = dict(zip(self.model, len(self.model)*[np.array([])])) self.autosave_iterations = autosave_iterations if not autosave_filename: date = datetime.datetime.today().strftime('%Y%m%d%H%M%S') prefix = 'vb_autosave_%s_' % date tmpfile = tempfile.NamedTemporaryFile(prefix=prefix, suffix='.hdf5') self.autosave_filename = tmpfile.name self.filename = None else: self.autosave_filename = autosave_filename self.filename = autosave_filename # Check uniqueness of the node names names = [node.name for node in self.model] if len(names) != len(self.model): raise Exception("Use unique names for nodes.") self.callback = callback self.callback_output = None def set_autosave(self, filename, iterations=None): self.autosave_filename = filename self.filename = filename if iterations is not None: self.autosave_iterations = iterations def update(self, *nodes, repeat=1): # TODO/FIXME: # # If no nodes are given and thus everything is updated, the update order # should be from down to bottom. Or something similar.. # Append the cost arrays self.L = np.append(self.L, utils.utils.nans(repeat)) for (node, l) in self.l.items(): self.l[node] = np.append(l, utils.utils.nans(repeat)) # By default, update all nodes if len(nodes) == 0: nodes = self.model for i in range(repeat): t = time.clock() # Update nodes for node in nodes: X = self[node] if hasattr(X, 'update') and callable(X.update): X.update() # Call the custom function provided by the user if callable(self.callback): z = self.callback() if z is not None: z = np.array(z)[...,np.newaxis] if self.callback_output is None: self.callback_output = z else: self.callback_output = np.concatenate((self.callback_output,z), axis=-1) # Compute lower bound L = self.loglikelihood_lowerbound() print("Iteration %d: loglike=%e (%.3f seconds)" % (self.iter+1, L, time.clock()-t)) # Check the progress of the iteration if self.iter > 0: # Check for errors if self.L[self.iter-1] - L > 1e-6: L_diff = (self.L[self.iter-1] - L) warnings.warn("Lower bound decreased %e! Bug somewhere or " "numerical inaccuracy?" % L_diff) # Check for convergence if L - self.L[self.iter-1] < 1e-12: print("Converged.") self.L[self.iter] = L self.iter += 1 # Auto-save, if requested if (self.autosave_iterations > 0 and np.mod(self.iter, self.autosave_iterations) == 0): self.save(self.autosave_filename) print('Auto-saved to %s' % self.autosave_filename) def compute_lowerbound(self): L = 0 for node in self.model: L += node.lower_bound_contribution() return L def compute_lowerbound_terms(self, *nodes): if len(nodes) == 0: nodes = self.model return {node: node.lower_bound_contribution() for node in nodes} def loglikelihood_lowerbound(self): L = 0 for node in self.model: lp = node.lower_bound_contribution() L += lp self.l[node][self.iter] = lp return L def plot_iteration_by_nodes(self): """ Plot the cost function per node during the iteration. Handy tool for debugging. """ D = len(self.l) N = self.iter L = np.empty((N,D)) legends = [] for (d, node) in enumerate(self.l): L[:,d] = self.l[node] legends += [node.name] plt.plot(np.arange(N)+1, L) plt.legend(legends, loc='lower right') plt.title('Lower bound contributions by nodes') plt.xlabel('Iteration') def get_iteration_by_nodes(self): return self.l def save(self, filename=None): if self.iter == 0: # Check HDF5 version. if h5py.version.hdf5_version_tuple < (1,8,7): warnings.warn("WARNING! Your HDF5 version is %s. HDF5 versions " "<1.8.7 are not able to save empty arrays, thus " "you may experience problems if you for instance " "try to save before running any iteration steps." % str(h5py.version.hdf5_version_tuple)) # By default, use the same file as for auto-saving if not filename: if self.filename: filename = self.filename else: raise Exception("Filename must be given.") # Open HDF5 file h5f = h5py.File(filename, 'w') try: # Write each node nodegroup = h5f.create_group('nodes') for node in self.model: if node.name == '': raise Exception("In order to save nodes, they must have " "(unique) names.") if hasattr(node, 'save') and callable(node.save): node.save(nodegroup.create_group(node.name)) # Write iteration statistics utils.utils.write_to_hdf5(h5f, self.L, 'L') utils.utils.write_to_hdf5(h5f, self.iter, 'iter') if self.callback_output is not None: utils.utils.write_to_hdf5(h5f, self.callback_output, 'callback_output') boundgroup = h5f.create_group('boundterms') for node in self.model: utils.utils.write_to_hdf5(boundgroup, self.l[node], node.name) finally: # Close file h5f.close() def load(self, *nodes, filename=None): # By default, use the same file as for auto-saving if not filename: if self.filename: filename = self.filename else: raise Exception("Filename must be given.") # Open HDF5 file h5f = h5py.File(filename, 'r') try: # Get nodes to load if len(nodes) == 0: nodes = self.model else: nodes = [self[node] for node in nodes if node is not None] # Read each node for node_id in nodes: node = self[node_id] if node.name == '': h5f.close() raise Exception("In order to load nodes, they must have " "(unique) names.") if hasattr(node, 'load') and callable(node.load): try: node.load(h5f['nodes'][node.name]) except KeyError: h5f.close() raise Exception("File does not contain variable %s" % node.name) # Read iteration statistics self.L = h5f['L'][...] self.iter = h5f['iter'][...] for node in self.model: self.l[node] = h5f['boundterms'][node.name][...] try: self.callback_output = h5f['callback_output'][...] except KeyError: pass finally: # Close file h5f.close() def __getitem__(self, name): if name in self.model: return name else: # Dictionary for mapping node names to nodes dictionary = {node.name: node for node in self.model} return dictionary[name] bayespy-0.1/bayespy/inference/vmp/transformations.py0000664000175000017500000017621512174013072024576 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np import warnings import scipy from bayespy import utils from bayespy.utils.linalg import dot, tracedot class RotationOptimizer(): def __init__(self, block1, block2, D): self.block1 = block1 self.block2 = block2 self.D = D def rotate(self, maxiter=10, check_gradient=False, verbose=False, check_bound=None, check_bound_terms=None): """ Optimize the rotation of two separate model blocks jointly. If some variable is the dot product of two Gaussians, rotating the two Gaussians optimally can make the inference algorithm orders of magnitude faster. First block is rotated with :math:`\mathbf{R}` and the second with :math:`\mathbf{R}^{-T}`. Blocks must have methods: `bound(U,s,V)` and `rotate(R)`. """ I = np.identity(self.D) piv = np.arange(self.D) def cost(r): # Make vector-r into matrix-R R = np.reshape(r, (self.D,self.D)) # Compute SVD invR = np.linalg.inv(R) logdetR = np.linalg.slogdet(R)[1] # Compute lower bound terms (b1,db1) = self.block1.bound(R, logdet=logdetR, inv=invR) (b2,db2) = self.block2.bound(invR.T, logdet=-logdetR, inv=R.T) # Apply chain rule for the second gradient: # d b(invR.T) # = tr(db.T * d(invR.T)) # = tr(db * d(invR)) # = -tr(db * invR * (dR) * invR) # = -tr(invR * db * invR * dR) db2 = -dot(invR.T, db2.T, invR.T) # Compute the cost function c = -(b1+b2) dc = -(db1+db2) return (c, np.ravel(dc)) def get_bound_terms(r): """ Returns a dictionary of bound terms for the nodes. """ # Make vector-r into matrix-R R = np.reshape(r, (self.D,self.D)) # Compute SVD invR = np.linalg.inv(R) logdetR = np.linalg.slogdet(R)[1] # Compute lower bound terms dict1 = self.block1.get_bound_terms(R, logdet=logdetR, inv=invR) dict2 = self.block2.get_bound_terms(invR.T, logdet=-logdetR, inv=R.T) dict1.update(dict2) return dict1 self.block1.setup() self.block2.setup() if check_gradient: R = np.random.randn(self.D, self.D) utils.optimize.check_gradient(cost, np.ravel(R), verbose=True) # Initial rotation is identity matrix r0 = np.ravel(np.identity(self.D)) (cost_begin, _) = cost(r0) if check_bound is not None and check_bound is not False: bound_begin = check_bound() if check_bound_terms: bound_terms_begin = get_bound_terms(r0) true_bound_terms_begin = check_bound_terms() # Run optimization #print("Initial cost:", cost(r0)[0]) r = utils.optimize.minimize(cost, r0, maxiter=maxiter, verbose=verbose) #if check_bound is not None and check_bound is not False: (cost_end, _) = cost(r) if check_bound_terms: bound_terms_end = get_bound_terms(r) # Apply the optimal rotation R = np.reshape(r, (self.D,self.D)) invR = np.linalg.inv(R) logdetR = np.linalg.slogdet(R)[1] self.block1.rotate(R, inv=invR, logdet=logdetR) self.block2.rotate(invR.T, inv=R.T, logdet=-logdetR) # Check that the cost function and the true lower bound changed equally cost_change = cost_end - cost_begin if check_bound is not None and check_bound is not False: bound_end = check_bound() bound_change = bound_end - bound_begin if not np.allclose(bound_change, -cost_change): #print(cost_end, cost_begin, bound_end, bound_begin) warnings.warn("Rotation cost function is not consistent with " "the true lower bound. Bound changed %g but " "optimized function changed %g." % (bound_change, -cost_change)) # Check that we really have improved the bound. if bound_change < 0: warnings.warn("Rotation made the lower bound worse by %g. " "Probably a bug in the rotation functions." % (bound_change,)) # Check that we really have improved the bound. if cost_change > 0: warnings.warn("Rotation optimization made the cost function worse by %g. " "Probably a bug in the gradient of the rotation functions." % (cost_change,)) ## if cost_change > 0: ## raise Exception("WTF! Bug in optimization..") if check_bound_terms: true_bound_terms_end = check_bound_terms() for rot in bound_terms_begin.keys(): opt_bound_change = (bound_terms_end[rot] - bound_terms_begin[rot]) true_bound_change = 0 for node in rot.nodes(): try: true_bound_change += (true_bound_terms_end[node] - true_bound_terms_begin[node]) except KeyError: raise Exception("The node %s is part of the " "transformation but not part of the " "model. Check your VB construction." % node.name) ## print("DEBUG in check %s:" ## % [node.name for node in rot.nodes()], ## true_bound_change, ## opt_bound_change) if not np.allclose(opt_bound_change, true_bound_change): #print(rot.nodes()[0].name, opt_bound_change, true_bound_change) warnings.warn("Rotation cost function is not consistent " "with the true lower bound for node %s. " "Bound changed %g but optimized function " "changed %g." % (rot.nodes()[0].name, true_bound_change, opt_bound_change)) raise Exception() class RotateGaussian(): def __init__(self, X): self.X = X def rotate(self, R, inv=None, logdet=None): self.X.rotate(R, inv=inv, logdet=logdet) def setup(self): """ This method should be called just before optimization. """ mask = self.X.mask[...,np.newaxis,np.newaxis] # Number of plates self.N = self.X.plates[0] #np.sum(mask) # Compute the sum over plates self.XX = utils.utils.sum_multiply(self.X.get_moments()[1], mask, axis=(-1,-2), sumaxis=False, keepdims=False) # Parent's moments self.Lambda = self.X.parents[1].get_moments()[0] def _compute_bound(self, R, logdet=None, inv=None, gradient=False): """ Rotate q(X) as X->RX: q(X)=N(R*mu, R*Cov*R') Assume: :math:`p(\mathbf{X}) = \prod^M_{m=1} N(\mathbf{x}_m|0, \mathbf{\Lambda})` """ # TODO/FIXME: X and alpha should NOT contain observed values!! Check # that. # TODO/FIXME: Allow non-zero prior mean! # Assume constant mean and precision matrix over plates.. # Compute rotated moments XX_R = dot(R, self.XX, R.T) inv_R = inv logdet_R = logdet # Compute entropy H(X) logH_X = utils.random.gaussian_entropy(-2*self.N*logdet_R, 0) # Compute logp_X = utils.random.gaussian_logpdf(np.vdot(XX_R, self.Lambda), 0, 0, 0, 0) # Compute the bound bound = logp_X + logH_X if gradient: # Compute dH(X) dlogH_X = utils.random.gaussian_entropy(-2*self.N*inv_R.T, 0) # Compute d dXX = 2*dot(self.Lambda, R, self.XX) dlogp_X = utils.random.gaussian_logpdf(dXX, 0, 0, 0, 0) d_bound = dlogp_X + dlogH_X return (bound, d_bound) else: return bound def bound(self, R, logdet=None, inv=None): return self._compute_bound(R, logdet=logdet, inv=inv, gradient=True) def get_bound_terms(self, R, logdet=None, inv=None): bound = self._compute_bound(R, logdet=logdet, inv=inv, gradient=False) return {self: bound} def nodes(self): return [self.X] class RotateGaussianARD(): # TODO: Let the mean be non-zero! def __init__(self, X, alpha): self.node_X = X self.node_alpha = alpha def nodes(self): return [self.node_X, self.node_alpha] def rotate(self, R, inv=None, logdet=None, Q=None): self.node_X.rotate(R, inv=inv, logdet=logdet, Q=Q) self.node_alpha.update() def setup(self, rotate_plates=False): """ This method should be called just before optimization. If using Q, set rotate_plates to True. """ # Get the mean parameter. It is not rotated. node_mu = self.node_X.parents[0] (mu, self.mumu) = node_mu.get_moments() mask = self.node_X.mask[...,np.newaxis,np.newaxis] # Number of plates self.N = self.node_X.plates[0] #np.sum(mask) self.X = self.node_X.get_moments()[0] * self.node_X.mask[...,np.newaxis] if not rotate_plates: # Compute the sum over plates self.XX = utils.utils.sum_multiply(self.node_X.get_moments()[1], mask, axis=(-1,-2), sumaxis=False, keepdims=False) self.Xmu = utils.utils.sum_multiply(self.X[...,:,np.newaxis], mu[...,np.newaxis,:], axis=(-1,-2), sumaxis=False, keepdims=False) else: XX = self.node_X.get_moments()[1] * mask self.CovX = XX - utils.linalg.outer(self.X, self.X) self.mu = mu #self.Xmu = self.X[...,:,np.newaxis] * mu[...,np.newaxis,:] # Parent's moments self.a = np.ravel(self.node_alpha.phi[1]) # TODO/FIXME: Handle vector valued parents a0 and b0 self.a0 = self.node_alpha.parents[0].get_moments()[0] self.b0 = self.node_alpha.parents[1].get_moments()[0] # Compute (summed over plates) #self.mu = np.atleast_2d(self.mu) if len(node_mu.plates) == 0 or node_mu.plates[0] == 1: self.mumu = self.N * np.diag(utils.utils.sum_to_dim(self.mumu, 2)) #print("no plates", node_mu.plates) else: self.mumu = np.diag(utils.utils.sum_to_dim(self.mumu, 2)) #print("yes plates", node_mu.plates, np.shape(self.mumu)) def _compute_bound(self, R, logdet=None, inv=None, Q=None, gradient=False): """ Rotate q(X) and q(alpha). Assume: p(X|alpha) = prod_m N(x_m|0,diag(alpha)) p(alpha) = prod_d G(a_d,b_d) """ # TODO/FIXME: X and alpha should NOT contain observed values!! Check that. # # Transform the distributions and moments # # Compute rotated second moment if Q is not None: # Rotate plates sumQ = np.sum(Q, axis=0) QX = np.einsum('ik,kj->ij', Q, self.X) XX = (np.einsum('ki,kj->ij', QX, QX) + np.einsum('d,dij->ij', sumQ**2, self.CovX)) logdet_Q = np.sum(np.log(np.abs(sumQ))) X = QX X_mu = utils.utils.sum_multiply(X[...,:,np.newaxis], self.mu[...,np.newaxis,:], axis=(-1,-2), sumaxis=False, keepdims=False) else: X = self.X XX = self.XX logdet_Q = 0 X_mu = self.Xmu # TODO/FIXME: X can be summed to the plates of mu!? #X_R = np.einsum('...jk,...k->...j', R, X) XX_R = dot(R, XX, R.T) X_mu = dot(R, X_mu) ## X_mu = utils.utils.sum_multiply(X_R[...,:,np.newaxis], ## self.mu[...,np.newaxis,:], ## sumaxis=False, ## axis=(-1,-2)) XmuXmu = (XX_R - X_mu - X_mu.T + self.mumu) # Compute q(alpha) a_alpha = self.a b_alpha = self.b0 + 0.5*np.diag(XmuXmu) #b_alpha = self.b0 + 0.5*np.diag(XX_R) alpha_R = a_alpha / b_alpha logalpha_R = - np.log(b_alpha) # + const logdet_R = logdet inv_R = inv N = self.N D = np.shape(R)[0] # # Compute the cost # # Compute entropy H(X) logH_X = utils.random.gaussian_entropy(-2*N*logdet_R - 2*D*logdet_Q, 0) # Compute entropy H(alpha) logH_alpha = utils.random.gamma_entropy(0, np.sum(np.log(b_alpha)), 0, 0, 0) # Compute #print(np.shape(self.mu), np.shape(X_R), np.shape(alpha_R)) logp_X = utils.random.gaussian_logpdf(np.einsum('ii,i', XmuXmu, alpha_R), 0, 0, N*np.sum(logalpha_R), 0) # Compute logp_alpha = utils.random.gamma_logpdf(self.b0*np.sum(alpha_R), np.sum(logalpha_R), self.a0*np.sum(logalpha_R), 0, 0) # Compute the bound bound = ( logp_X + logp_alpha + logH_X + logH_alpha ) #print(np.shape(bound), np.shape(logp_X), np.shape(logp_alpha), np.shape(logH_X), np.shape(logH_alpha)) if not gradient: return bound # # Compute the gradient with respect R # # Compute dH(X) dlogH_X = utils.random.gaussian_entropy(-2*N*inv_R.T, 0) # Compute dH(alpha) d_log_b = np.einsum('i,ik,kj->ij', 1/b_alpha, R, XX) dlogH_alpha = utils.random.gamma_entropy(0, d_log_b, 0, 0, 0) # Compute d # TODO/FIXME: Fix these gradients! d_log_alpha = -d_log_b dXX_alpha = 2*np.einsum('i,ik,kj->ij', alpha_R, R, XX) XX_dalpha = -np.einsum('i,i,ii,ik,kj->ij', alpha_R, 1/b_alpha, XX_R, R, XX) dXmu_alpha = 0 dlogp_X = utils.random.gaussian_logpdf(dXX_alpha + XX_dalpha, dXmu_alpha, 0, N*d_log_alpha, 0) # Compute d d_alpha = -np.einsum('i,i,ik,kj->ij', alpha_R, 1/b_alpha, R, XX) dlogp_alpha = utils.random.gamma_logpdf(self.b0*d_alpha, d_log_alpha, self.a0*d_log_alpha, 0, 0) dR_bound = ( dlogp_X + dlogp_alpha + dlogH_X + dlogH_alpha ) if Q is None: return (bound, dR_bound) # # Compute the gradient with respect to Q (if Q given) # def d_helper(v): R_v_R = np.einsum('ki,k,kj->ij', R, v, R) tr_R_v_R_Cov = np.einsum('ij,dji->d', R_v_R, self.CovX) return (dot(QX, R_v_R, self.X.T) + sumQ * tr_R_v_R_Cov) # Compute dH(X) dQ_logHX = utils.random.gaussian_entropy(-2*D/sumQ, 0) # Compute dH(alpha) ## R_b_R = np.einsum('ki,k,kj->ij', R, 1/b_alpha, R) ## tr_R_b_R_Cov = np.einsum('ij,dji->d', R_b_R, self.CovX) ## d_log_b = (dot(QX, R_b_R, self.X.T) ## + sumQ * tr_R_b_R_Cov) d_log_b = d_helper(1/b_alpha) dQ_logHalpha = utils.random.gamma_entropy(0, d_log_b, 0, 0, 0) # Compute d # TODO/FIXME: Fix these gradients to take into account the mean parameter! dXX_alpha = 2*d_helper(alpha_R) XX_dalpha = -d_helper(np.diag(XX_R)*alpha_R/b_alpha) d_log_alpha = -d_log_b dQ_logpX = utils.random.gaussian_logpdf(dXX_alpha + XX_dalpha, 0, 0, N*d_log_alpha, 0) # Compute d ## R_alpha_b_R = np.einsum('ki,k,kj->ij', ## R, alpha_R/b_alpha, R) ## tr_R_alpha_b_R_Cov = np.einsum('ij,dji->d', ## R_alpha_b_R, self.CovX) ## d_alpha = (dot(QX, R_alpha_b_R, self.X.T) ## + sumQ * tr_R_alpha_b_R_Cov) d_alpha = -d_helper(alpha_R/b_alpha) dQ_logpalpha = utils.random.gamma_logpdf(self.b0*d_alpha, d_log_alpha, self.a0*d_log_alpha, 0, 0) dQ_bound = ( dQ_logpX + dQ_logpalpha + dQ_logHX + dQ_logHalpha ) return (bound, dR_bound, dQ_bound) def bound(self, R, logdet=None, inv=None, Q=None): return self._compute_bound(R, logdet=logdet, inv=inv, Q=Q, gradient=True) def get_bound_terms(self, R, logdet=None, inv=None, Q=None): bound = self._compute_bound(R, logdet=logdet, inv=inv, Q=Q, gradient=False) return {self: bound} class RotateGaussianMarkovChain(): """ Assume the following model. Constant, unit isotropic innovation noise. :math:`A` may vary in time. Shape of A: (N,D,D) Shape of AA: (N,D,D,D) No plates for X. """ def __init__(self, X, A, A_rotator): self.X_node = X self.A_node = A self.A_rotator = A_rotator self.N = X.dims[0][0] def nodes(self): # A node is in the A rotator. return [self.X_node] def rotate(self, R, inv=None, logdet=None): self.X_node.rotate(R, inv=inv, logdet=logdet) self.A_rotator.rotate(inv.T, inv=R.T, logdet=-logdet, Q=R) def setup(self): """ This method should be called just before optimization. """ # Get moments of X (X, XnXn, XpXn) = self.X_node.get_moments() XpXp = XnXn[:-1,:,:] # Get moments of A (and make sure they include time axis) (A, AA) = self.X_node.parents[2].get_moments() A = utils.utils.atleast_nd(A, 3) AA = utils.utils.atleast_nd(AA, 4) CovA = AA - A[...,:,np.newaxis]*A[...,np.newaxis,:] # # Expectations with respect to X # self.X0 = X[0,:] self.X0X0 = XnXn[0,:,:] #self.XpXp = np.sum(XpXp, axis=0) self.XnXn = np.sum(XnXn[1:,:,:], axis=0) #self.XpXn = np.sum(XpXn, axis=0) # # Expectations with respect to A and X # # Compute: \sum_n self.A_XpXn = np.sum(dot(A, XpXn), axis=0) # Compute: \sum_n ^T self.A_XpXp_A = np.sum(dot(A, XpXp, utils.utils.T(A)), axis=0) # Compute: \sum_n tr(CovA_n ) self.CovA_XpXp = np.einsum('ndij,nij->d', CovA, XpXp) # Get moments of the fixed parameter nodes mu = self.X_node.parents[0].get_moments()[0] self.Lambda = self.X_node.parents[1].get_moments()[0] self.Lambda_mu_X0 = np.outer(np.dot(self.Lambda,mu), self.X0) self.A_rotator.setup(rotate_plates=True) # Innovation noise is assumed to be I #self.v = self.X_node.parents[3].get_moments()[0] def _compute_bound(self, R, logdet=None, inv=None, gradient=False): """ Rotate q(X) as X->RX: q(X)=N(R*mu, R*Cov*R') Assume: :math:`p(\mathbf{X}) = \prod^M_{m=1} N(\mathbf{x}_m|0, \mathbf{\Lambda})` Assume unit innovation noise covariance. """ # TODO/FIXME: X and alpha should NOT contain observed values!! Check # that. # TODO/FIXME: Allow non-zero prior mean! # Assume constant mean and precision matrix over plates.. invR = inv logdetR = logdet # Transform moments of X: # Transform moments of A: Lambda_R_X0X0 = dot(self.Lambda, R, self.X0X0) R_XnXn = dot(R, self.XnXn) RA_XpXp_A = dot(R, self.A_XpXp_A) sumr = np.sum(R, axis=0) R_CovA_XpXp = sumr * self.CovA_XpXp ## if not gradient: ## print("DEBUG TOO", dot(R_XnXn,R.T)) # Compute entropy H(X) logH_X = utils.random.gaussian_entropy(-2*self.N*logdetR, 0) # Compute yy = tracedot(R_XnXn, R.T) + tracedot(Lambda_R_X0X0, R.T) yz = tracedot(dot(R,self.A_XpXn),R.T) + tracedot(self.Lambda_mu_X0, R.T) zz = tracedot(RA_XpXp_A, R.T) + np.dot(R_CovA_XpXp, sumr) #RR_CovA_XpXp logp_X = utils.random.gaussian_logpdf(yy, yz, zz, 0, 0) # Compute dH(X) dlogH_X = utils.random.gaussian_entropy(-2*self.N*invR.T, 0) # Compute the bound bound = logp_X + logH_X #if not gradient: # print("Debug in transformations", bound) if gradient: # Compute d dyy = 2 * (R_XnXn + Lambda_R_X0X0) dyz = dot(R, self.A_XpXn + self.A_XpXn.T) + self.Lambda_mu_X0 dzz = 2 * (RA_XpXp_A + R_CovA_XpXp) dlogp_X = utils.random.gaussian_logpdf(dyy, dyz, dzz, 0, 0) d_bound = dlogp_X + dlogH_X return (bound, d_bound) else: return bound def bound(self, R, logdet=None, inv=None): (bound_X, d_bound_X) = self._compute_bound(R, logdet=logdet, inv=inv, gradient=True) # Compute cost and gradient from A (bound_A, dR_bound_A, dQ_bound_A) = self.A_rotator.bound(inv.T, inv=R.T, logdet=-logdet, Q=R) # TODO/FIXME: Also apply the gradient of invR.T to the result dR_bound_A = -dot(inv.T, dR_bound_A.T, inv.T) # Compute the bound bound = bound_X + bound_A d_bound = d_bound_X + dR_bound_A + dQ_bound_A return (bound, d_bound) def get_bound_terms(self, R, logdet=None, inv=None): bound_dict = self.A_rotator.get_bound_terms(inv.T, inv=R.T, logdet=-logdet, Q=R) bound_X = self._compute_bound(R, logdet=logdet, inv=inv, gradient=False) bound_dict.update({self: bound_X}) return bound_dict class RotateGaussianMatrixARD(): """ Assume the following model: :math:`p(X|\alpha) = \prod_{dk} N(x_{dk} | 0, 1 / \alpha_{dk})` :math:`p(\alpha) = \prod_k G(\alpha_{dk} | a, b)` Reshape vector x to matrix X and multiply either on the left or on the right: R*X or X*R'. """ def __init__(self, X, alpha, axis='rows', D1=None, D2=None): self.X_node = X self.alpha_node = alpha self.D1 = D1 self.D2 = D2 ## D = X.dims[0][0] ## self.D2 = alpha.plates[0] ## self.D1 = D / self.D2 # Whether to multiply on the left or on the right if axis == 'rows': self.axis = 'rows' elif axis == 'cols': self.axis = 'cols' else: raise Exception("Unknown axis") def nodes(self): return [self.X_node, self.alpha_node] def rotate(self, R, inv=None, logdet=None, Q=None): if self.axis == 'rows': # Multiply on the left by R R1 = R inv1 = inv logdet1 = logdet R2 = np.identity(self.D2) inv2 = R2 logdet2 = 0 else: # Multiply on the right by R' R1 = np.identity(self.D1) inv1 = R1 logdet1 = 0 R2 = R inv2 = inv logdet2 = logdet self.X_node.rotate_matrix(R1, R2, inv1=inv1, logdet1=logdet1, inv2=inv2, logdet2=logdet2, Q=Q) self.alpha_node.update() #print("debug in rotazione", self.alpha_node.get_moments()[0]) def setup(self, rotate_plates=False): """ This method should be called just before optimization. """ #mask = self.X_node.mask[...,np.newaxis,np.newaxis] # Number of plates self.N = self.X_node.plates[0] #np.sum(mask) if not rotate_plates: # Compute the sum over plates self.XX = utils.utils.sum_multiply(self.X_node.get_moments()[1], self.X_node.mask[...,np.newaxis,np.newaxis], axis=(-1,-2), sumaxis=False, keepdims=False) else: self.X = (self.X_node.get_moments()[0] * self.X_node.mask[...,np.newaxis]) XX = (self.X_node.get_moments()[1] * self.X_node.mask[...,np.newaxis,np.newaxis]) self.CovX = XX - utils.linalg.outer(self.X, self.X) # Parent's moments self.a = np.ravel(self.alpha_node.phi[1]) #self.b = -np.ravel(self.alpha_node.phi[0]) # TODO/FIXME: Handle vector valued parents a0 and b0 self.a0 = self.alpha_node.parents[0].get_moments()[0] self.b0 = self.alpha_node.parents[1].get_moments()[0] def _compute_bound(self, R, logdet=None, inv=None, Q=None, gradient=False): """ Rotate q(X) and q(alpha). """ # TODO/FIXME: X and alpha should NOT contain observed values!! Check that. # # Transform the distributions and moments # D1 = self.D1 D2 = self.D2 N = self.N D = D1 * D2 # Compute rotated second moment if Q is not None: X = np.reshape(self.X, (N,D1,D2)) CovX = np.reshape(self.CovX, (N,D1,D2,D1,D2)) # Rotate plates sumQ = np.sum(Q, axis=0) QX = np.einsum('ik,kab->iab', Q, X) logdet_Q = np.sum(np.log(np.abs(sumQ))) if self.axis == 'cols': # Sum "rows" #X = np.einsum('nkj->nj', X) # Rotate "columns" X_R = np.einsum('jk,nik->nij', R, X) r_CovX_r = np.einsum('bk,bl,nakal->nab', R, R, CovX) XX = (np.einsum('kai,kaj->aij', QX, QX) + np.einsum('d,daiaj->aij', sumQ**2, CovX)) else: # Rotate "rows" #print("OR THE BUG IS HERE...") X_R = np.einsum('ik,nkj->nji', R, X) r_CovX_r = np.einsum('ak,al,nkblb->nba', R, R, CovX) XX = (np.einsum('kib,kjb->bij', QX, QX) + np.einsum('d,dibjb->bij', sumQ**2, CovX)) Q_X_R = np.einsum('nk,kij->nij', Q, X_R) else: # Reshape into matrix form sh = (D1,D2,D1,D2) XX = np.reshape(self.XX, sh) if self.axis == 'cols': XX = np.einsum('aiaj->aij', XX) else: XX = np.einsum('ibjb->bij', XX) logdet_Q = 0 # Reshape vector to matrix if self.axis == 'cols': # Apply rotation on the right R_XX = np.einsum('ik,akj->aij', R, XX) r_XX_r = np.einsum('ik,il,akl->ai', R, R, XX) if Q is not None: # Debug stuff: #print("debug for Q", np.shape(XX_R)) r_XQQX_r = np.einsum('ab->ab', r_XX_r) #r_XX_r #np.einsum('abab->b', XX_R) # Compute q(alpha) a_alpha = self.a b_alpha = self.b0 + 0.5 * r_XX_r #np.einsum('ab->b', r_XX_r) alpha_R = a_alpha / b_alpha logalpha_R = -np.log(b_alpha) # + const logdet_R = logdet inv_R = inv # Bug in here: XX_dalpha = -np.einsum('ab,ab,abj->bj', alpha_R/b_alpha, r_XX_r, R_XX) #XX = np.einsum('aiaj->ij', XX) #XX_R = np.einsum('aiaj->ij', XX_R) #print("THERE") alpha_R_XX = np.einsum('ai,aij->ij', alpha_R, R_XX) # BUG IN HERE? In gradient dalpha_R_XX = np.einsum('ai,aij->ij', alpha_R/b_alpha, R_XX) invb_R_XX = np.einsum('ai,aij->ij', 1/b_alpha, R_XX) #dalpha_RXXR = np.einsum('i,ii->i', alpha_R/b_alpha, XX_R) ND = self.N * D1 else: # Apply rotation on the left R_XX = np.einsum('ik,bkj->bij', R, XX) r_XX_r = np.einsum('ik,il,bkl->bi', R, R, XX) if Q is not None: # Debug stuff: #print("debug for Q", np.shape(XX_R)) r_XQQX_r = np.einsum('bi->bi', r_XX_r) #np.einsum('abab->b', XX_R) # Compute q(alpha) a_alpha = self.a b_alpha = self.b0 + 0.5 * r_XX_r #np.einsum('bi->b', r_XX_r) #b_alpha = self.b0 + 0.5 * np.einsum('abab->b', XX_R) alpha_R = a_alpha / b_alpha logalpha_R = -np.log(b_alpha) # + const logdet_R = logdet inv_R = inv #print("HERE IS THE BUG SOMEWHERE") # Compute: _* R < #print(np.shape(alpha_R), np.shape(R), np.shape(XX), np.shape(R_XX), QX.shape, D1, D2) alpha_R_XX = np.einsum('bi,bij->ij', alpha_R, R_XX) dalpha_R_XX = np.einsum('bi,bij->ij', alpha_R/b_alpha, R_XX) invb_R_XX = np.einsum('bi,bij->ij', 1/b_alpha, R_XX) #dalpha_RXXR = np.einsum('b,ibib->i', alpha_R/b_alpha, XX_R) XX_dalpha = -np.einsum('ba,ba,baj->aj', alpha_R/b_alpha, r_XX_r, R_XX) #XX = np.einsum('ibjb->ij', XX) #XX_R = np.nan * np.einsum('ibjb->ij', XX_R) ND = self.N * D2 # # Compute the cost # # Compute entropy H(X) logH_X = utils.random.gaussian_entropy(-2*ND*logdet_R - 2*D*logdet_Q, 0) # Compute entropy H(alpha) logH_alpha = utils.random.gamma_entropy(0, np.sum(np.log(b_alpha)), 0, 0, 0) # Compute #logp_X = utils.random.gaussian_logpdf(np.einsum('ii,i', XX_R, alpha_R), logp_X = utils.random.gaussian_logpdf(tracedot(alpha_R_XX, R.T), 0, 0, N*np.sum(logalpha_R), 0) # Compute logp_alpha = utils.random.gamma_logpdf(self.b0*np.sum(alpha_R), np.sum(logalpha_R), self.a0*np.sum(logalpha_R), 0, 0) # Compute the bound bound = (0 + logp_X + logp_alpha + logH_X + logH_alpha ) ## if not gradient: ## print("debug xx", ## dot(R_XX, R.T), ## np.shape(XX), ## self.X_node.name, ## logp_X, ## logp_alpha, ## logH_X, ## logH_alpha, ## alpha_R) if not gradient: return bound # # Compute the gradient with respect to R # # Compute dH(X) dlogH_X = utils.random.gaussian_entropy(-2*ND*inv_R.T, 0) # Compute dH(alpha) d_log_b = invb_R_XX #np.einsum('i,ik,kj->ij', 1/b_alpha, R, XX) dlogH_alpha = utils.random.gamma_entropy(0, d_log_b, 0, 0, 0) # Compute d d_log_alpha = -d_log_b dXX_alpha = 2*alpha_R_XX #np.einsum('i,ik,kj->ij', alpha_R, R, XX) #dalpha_xx = np.einsum('id,di', dalpha_R_XX, R.T) #XX_dalpha = -np.einsum('i,ik,kj', dalpha_RXXR, R, XX) # BUG IS IN THIS TERM!!!! #np.einsum('i,i,ii,ik,kj->ij', alpha_R, 1/b_alpha, XX_R, R, XX) # TODO/FIXME: This gradient term seems to have a bug. # # DEBUG: If you set these gradient terms to zero, the gradient is more # accurate..?! #dXX_alpha = 0 #XX_dalpha = 0 dlogp_X = utils.random.gaussian_logpdf(dXX_alpha + XX_dalpha, 0, 0, N*d_log_alpha, 0) # Compute d d_alpha = -dalpha_R_XX #np.einsum('i,i,ik,kj->ij', alpha_R, 1/b_alpha, R, XX) dlogp_alpha = utils.random.gamma_logpdf(self.b0*d_alpha, d_log_alpha, self.a0*d_log_alpha, 0, 0) dR_bound = (0*dlogp_X + dlogp_X + dlogp_alpha + dlogH_X + dlogH_alpha ) if Q is None: return (bound, dR_bound) # # Compute the gradient with respect to Q (if Q given) # def d_helper(v): return (np.einsum('iab,ab,jab->ij', Q_X_R, v, X_R) + np.einsum('n,ab,nab->n', sumQ, v, r_CovX_r)) # Compute dH(X) dQ_logHX = utils.random.gaussian_entropy(-2*D/sumQ, 0) # Compute dH(alpha) d_log_b = d_helper(1/b_alpha) dQ_logHalpha = utils.random.gamma_entropy(0, d_log_b, 0, 0, 0) # Compute d dXX_alpha = 2*d_helper(alpha_R) XX_dalpha = -d_helper(r_XQQX_r*alpha_R/b_alpha) d_log_alpha = -d_log_b dQ_logpX = utils.random.gaussian_logpdf(dXX_alpha + XX_dalpha, 0, 0, N*d_log_alpha, 0) # Compute d d_alpha = -d_helper(alpha_R/b_alpha) dQ_logpalpha = utils.random.gamma_logpdf(self.b0*d_alpha, d_log_alpha, self.a0*d_log_alpha, 0, 0) dQ_bound = (0*dQ_logpX + dQ_logpX + dQ_logpalpha + dQ_logHX + dQ_logHalpha ) return (bound, dR_bound, dQ_bound) def bound(self, R, logdet=None, inv=None, Q=None): return self._compute_bound(R, logdet=logdet, inv=inv, Q=Q, gradient=True) def get_bound_terms(self, R, logdet=None, inv=None, Q=None): bound = self._compute_bound(R, logdet=logdet, inv=inv, Q=Q, gradient=False) return {self: bound} ## class RotateGaussianMatrixARD(): ## """ ## Assume the following model: ## :math:`p(X|\alpha) = \prod_{dk} N(x_{dk} | 0, 1 / \alpha_k)` ## :math:`p(\alpha) = \prod_k G(\alpha_k | a, b)` ## Reshape vector x to matrix X and multiply either on the left or on the ## right: R*X or X*R'. ## """ ## def __init__(self, X, alpha, axis='rows'): ## self.X_node = X ## self.alpha_node = alpha ## D = X.dims[0][0] ## self.D2 = alpha.plates[0] ## self.D1 = D / self.D2 ## # Whether to multiply on the left or on the right ## if axis == 'rows': ## self.axis = 'rows' ## elif axis == 'cols': ## self.axis = 'cols' ## else: ## raise Exception("Unknown axis") ## def nodes(self): ## return [self.X_node, self.alpha_node] ## def rotate(self, R, inv=None, logdet=None, Q=None): ## if self.axis == 'rows': ## # Multiply on the left by R ## R1 = R ## inv1 = inv ## logdet1 = logdet ## R2 = np.identity(self.D2) ## inv2 = R2 ## logdet2 = 0 ## else: ## # Multiply on the right by R' ## R1 = np.identity(self.D1) ## inv1 = R1 ## logdet1 = 0 ## R2 = R ## inv2 = inv ## logdet2 = logdet ## self.X_node.rotate_matrix(R1, R2, ## inv1=inv1, ## logdet1=logdet1, ## inv2=inv2, ## logdet2=logdet2, ## Q=Q) ## self.alpha_node.update() ## #print("debug in rotazione", self.alpha_node.get_moments()[0]) ## def setup(self, rotate_plates=False): ## """ ## This method should be called just before optimization. ## """ ## #mask = self.X_node.mask[...,np.newaxis,np.newaxis] ## # Number of plates ## self.N = self.X_node.plates[0] #np.sum(mask) ## if not rotate_plates: ## # Compute the sum over plates ## self.XX = utils.utils.sum_multiply(self.X_node.get_moments()[1], ## self.X_node.mask[...,np.newaxis,np.newaxis], ## axis=(-1,-2), ## sumaxis=False, ## keepdims=False) ## else: ## self.X = (self.X_node.get_moments()[0] ## * self.X_node.mask[...,np.newaxis]) ## XX = (self.X_node.get_moments()[1] ## * self.X_node.mask[...,np.newaxis,np.newaxis]) ## self.CovX = XX - utils.linalg.outer(self.X, self.X) ## # Parent's moments ## self.a = np.ravel(self.alpha_node.phi[1]) ## #self.b = -np.ravel(self.alpha_node.phi[0]) ## # TODO/FIXME: Handle vector valued parents a0 and b0 ## self.a0 = self.alpha_node.parents[0].get_moments()[0] ## self.b0 = self.alpha_node.parents[1].get_moments()[0] ## def _compute_bound(self, R, logdet=None, inv=None, Q=None, gradient=False): ## """ ## Rotate q(X) and q(alpha). ## """ ## # TODO/FIXME: X and alpha should NOT contain observed values!! Check that. ## # ## # Transform the distributions and moments ## # ## D1 = self.D1 ## D2 = self.D2 ## N = self.N ## D = D1 * D2 ## # Compute rotated second moment ## if Q is not None: ## X = np.reshape(self.X, (N,D1,D2)) ## CovX = np.reshape(self.CovX, (N,D1,D2,D1,D2)) ## # Rotate plates ## sumQ = np.sum(Q, axis=0) ## QX = np.einsum('ik,kab->iab', Q, X) ## logdet_Q = np.sum(np.log(np.abs(sumQ))) ## if self.axis == 'cols': ## # Sum "rows" ## X = np.einsum('nkj->nj', X) ## # Rotate "columns" ## X_R = np.einsum('jk,nk->nj', R, X) ## r_CovX_r = np.einsum('ik,il,nakal->ni', R, R, CovX) ## XX = (np.einsum('kai,kaj->aij', QX, QX) ## + np.einsum('d,daiaj->aij', sumQ**2, CovX)) ## else: ## # Rotate "rows" ## #print("OR THE BUG IS HERE...") ## X_R = np.einsum('ik,nkj->nj', R, X) ## r_CovX_r = np.einsum('ak,al,nkili->ni', R, R, CovX) ## XX = (np.einsum('kib,kjb->bij', QX, QX) ## + np.einsum('d,dibjb->bij', sumQ**2, CovX)) ## Q_X_R = np.einsum('nk,ki->ni', Q, X_R) ## else: ## # Reshape into matrix form ## sh = (D1,D2,D1,D2) ## XX = np.reshape(self.XX, sh) ## if self.axis == 'cols': ## XX = np.einsum('aiaj->aij', XX) ## else: ## XX = np.einsum('ibjb->bij', XX) ## logdet_Q = 0 ## # Reshape vector to matrix ## if self.axis == 'cols': ## # Apply rotation on the right ## R_XX = np.einsum('ik,akj->aij', R, XX) ## r_XX_r = np.einsum('ik,il,akl->ai', R, R, XX) ## if Q is not None: ## # Debug stuff: ## #print("debug for Q", np.shape(XX_R)) ## r_XQQX_r = np.einsum('ab->b', r_XX_r) #r_XX_r #np.einsum('abab->b', XX_R) ## # Compute q(alpha) ## a_alpha = self.a ## b_alpha = self.b0 + 0.5 * np.einsum('ab->b', r_XX_r) ## alpha_R = a_alpha / b_alpha ## logalpha_R = -np.log(b_alpha) # + const ## logdet_R = logdet ## inv_R = inv ## # Bug in here: ## XX_dalpha = -np.einsum('b,ab,abj->bj', alpha_R/b_alpha, r_XX_r, R_XX) ## #XX = np.einsum('aiaj->ij', XX) ## #XX_R = np.einsum('aiaj->ij', XX_R) ## #print("THERE") ## alpha_R_XX = np.einsum('i,aij->ij', alpha_R, R_XX) # BUG IN HERE? In gradient ## dalpha_R_XX = np.einsum('i,aij->ij', alpha_R/b_alpha, R_XX) ## invb_R_XX = np.einsum('i,aij->ij', 1/b_alpha, R_XX) ## #dalpha_RXXR = np.einsum('i,ii->i', alpha_R/b_alpha, XX_R) ## ND = self.N * D1 ## else: ## # Apply rotation on the left ## R_XX = np.einsum('ik,bkj->bij', R, XX) ## r_XX_r = np.einsum('ik,il,bkl->bi', R, R, XX) ## if Q is not None: ## # Debug stuff: ## #print("debug for Q", np.shape(XX_R)) ## r_XQQX_r = np.einsum('bi->b', r_XX_r) #np.einsum('abab->b', XX_R) ## # Compute q(alpha) ## a_alpha = self.a ## b_alpha = self.b0 + 0.5 * np.einsum('bi->b', r_XX_r) ## #b_alpha = self.b0 + 0.5 * np.einsum('abab->b', XX_R) ## alpha_R = a_alpha / b_alpha ## logalpha_R = -np.log(b_alpha) # + const ## logdet_R = logdet ## inv_R = inv ## #print("HERE IS THE BUG SOMEWHERE") ## # Compute: _* R < ## alpha_R_XX = dot(R, np.einsum('b,bij->ij', alpha_R, XX)) ## dalpha_R_XX = dot(R, np.einsum('b,bij->ij', alpha_R/b_alpha, XX)) ## invb_R_XX = dot(R, np.einsum('b,bij->ij', 1/b_alpha, XX)) ## #dalpha_RXXR = np.einsum('b,ibib->i', alpha_R/b_alpha, XX_R) ## XX_dalpha = -np.einsum('b,ba,baj->aj', alpha_R/b_alpha, r_XX_r, R_XX) ## #XX = np.einsum('ibjb->ij', XX) ## #XX_R = np.nan * np.einsum('ibjb->ij', XX_R) ## ND = self.N * D2 ## # ## # Compute the cost ## # ## # Compute entropy H(X) ## logH_X = utils.random.gaussian_entropy(-2*ND*logdet_R - 2*D*logdet_Q, ## 0) ## # Compute entropy H(alpha) ## logH_alpha = utils.random.gamma_entropy(0, ## np.sum(np.log(b_alpha)), ## 0, ## 0, ## 0) ## # Compute ## #logp_X = utils.random.gaussian_logpdf(np.einsum('ii,i', XX_R, alpha_R), ## logp_X = utils.random.gaussian_logpdf(tracedot(alpha_R_XX, R.T), ## 0, ## 0, ## (N*D1)*np.sum(logalpha_R), ## 0) ## # Compute ## logp_alpha = utils.random.gamma_logpdf(self.b0*np.sum(alpha_R), ## np.sum(logalpha_R), ## self.a0*np.sum(logalpha_R), ## 0, ## 0) ## # Compute the bound ## bound = (0 ## + logp_X ## + logp_alpha ## + logH_X ## + logH_alpha ## ) ## ## if not gradient: ## ## print("debug xx", ## ## dot(R_XX, R.T), ## ## np.shape(XX), ## ## self.X_node.name, ## ## logp_X, ## ## logp_alpha, ## ## logH_X, ## ## logH_alpha, ## ## alpha_R) ## if not gradient: ## return bound ## # ## # Compute the gradient with respect to R ## # ## # Compute dH(X) ## dlogH_X = utils.random.gaussian_entropy(-2*ND*inv_R.T, ## 0) ## # Compute dH(alpha) ## d_log_b = invb_R_XX #np.einsum('i,ik,kj->ij', 1/b_alpha, R, XX) ## dlogH_alpha = utils.random.gamma_entropy(0, ## d_log_b, ## 0, ## 0, ## 0) ## # Compute d ## d_log_alpha = -d_log_b ## dXX_alpha = 2*alpha_R_XX #np.einsum('i,ik,kj->ij', alpha_R, R, XX) ## #dalpha_xx = np.einsum('id,di', dalpha_R_XX, R.T) ## #XX_dalpha = -np.einsum('i,ik,kj', dalpha_RXXR, R, XX) # BUG IS IN THIS TERM!!!! ## #np.einsum('i,i,ii,ik,kj->ij', alpha_R, 1/b_alpha, XX_R, R, XX) ## # TODO/FIXME: This gradient term seems to have a bug. ## # ## # DEBUG: If you set these gradient terms to zero, the gradient is more ## # accurate..?! ## #dXX_alpha = 0 ## #XX_dalpha = 0 ## dlogp_X = utils.random.gaussian_logpdf(dXX_alpha + XX_dalpha, ## 0, ## 0, ## (N*D1)*d_log_alpha, ## 0) ## # Compute d ## d_alpha = -dalpha_R_XX #np.einsum('i,i,ik,kj->ij', alpha_R, 1/b_alpha, R, XX) ## dlogp_alpha = utils.random.gamma_logpdf(self.b0*d_alpha, ## d_log_alpha, ## self.a0*d_log_alpha, ## 0, ## 0) ## dR_bound = (0*dlogp_X ## + dlogp_X ## + dlogp_alpha ## + dlogH_X ## + dlogH_alpha ## ) ## if Q is None: ## return (bound, dR_bound) ## # ## # Compute the gradient with respect to Q (if Q given) ## # ## def d_helper(v): ## return (np.einsum('id,d,jd->ij', Q_X_R, v, X_R) ## + np.einsum('n,d,nd->n', sumQ, v, r_CovX_r)) ## # Compute dH(X) ## dQ_logHX = utils.random.gaussian_entropy(-2*D/sumQ, ## 0) ## # Compute dH(alpha) ## d_log_b = d_helper(1/b_alpha) ## dQ_logHalpha = utils.random.gamma_entropy(0, ## d_log_b, ## 0, ## 0, ## 0) ## # Compute d ## dXX_alpha = 2*d_helper(alpha_R) ## XX_dalpha = -d_helper(r_XQQX_r*alpha_R/b_alpha) ## d_log_alpha = -d_log_b ## dQ_logpX = utils.random.gaussian_logpdf(dXX_alpha + XX_dalpha, ## 0, ## 0, ## (N*D1)*d_log_alpha, ## 0) ## # Compute d ## d_alpha = -d_helper(alpha_R/b_alpha) ## dQ_logpalpha = utils.random.gamma_logpdf(self.b0*d_alpha, ## d_log_alpha, ## self.a0*d_log_alpha, ## 0, ## 0) ## dQ_bound = (0*dQ_logpX ## + dQ_logpX ## + dQ_logpalpha ## + dQ_logHX ## + dQ_logHalpha ## ) ## return (bound, dR_bound, dQ_bound) ## def bound(self, R, logdet=None, inv=None, Q=None): ## return self._compute_bound(R, ## logdet=logdet, ## inv=inv, ## Q=Q, ## gradient=True) ## def get_bound_terms(self, R, logdet=None, inv=None, Q=None): ## bound = self._compute_bound(R, ## logdet=logdet, ## inv=inv, ## Q=Q, ## gradient=False) ## return {self: bound} class RotateDriftingMarkovChain(): """ Assume the following model. Constant, unit isotropic innovation noise. :math:`A_n = \sum_k B_k s_{kn}` Shape of B: (D,D*K) Shape of BB: (D,D*K,D*K) Shape of S: (N,K) Shape of SS: (N,K,K) No plates for X. """ def __init__(self, X, B, S, B_rotator): self.X_node = X self.B_node = B self.S_node = S self.B_rotator = B_rotator self.N = X.dims[0][0] self.D = X.dims[0][-1] self.K = S.dims[0][-1] def nodes(self): # B node is in the B rotator. # S is not rotated. return [self.X_node] def rotate(self, R, inv=None, logdet=None): self.X_node.rotate(R, inv=inv, logdet=logdet) self.B_rotator.rotate(inv.T, inv=R.T, logdet=-logdet, Q=R) def setup(self): """ This method should be called just before optimization. """ # Get moments of X (X, XnXn, XpXn) = self.X_node.get_moments() XpXp = XnXn[:-1,:,:] # Get moments of B and S (B, BB) = self.B_node.get_moments() u_S = self.S_node.get_moments() S = u_S[0] SS = u_S[1] CovB = BB - B[...,:,np.newaxis]*B[...,np.newaxis,:] B = np.reshape(B, (self.D,self.D,self.K)) CovB = np.reshape(CovB, (self.D,self.D,self.K,self.D,self.K)) # # Expectations with respect to X # self.X0 = X[0,:] self.X0X0 = XnXn[0,:,:] #self.XpXp = np.sum(XpXp, axis=0) self.XnXn = np.sum(XnXn[1:,:,:], axis=0) #self.XpXn = np.sum(XpXn, axis=0) # # Expectations with respect to A and X # # Compute: \sum_n S_XpXn = np.einsum('nk,nij->kij', S, XpXn) self.A_XpXn = np.einsum('dik,kij->dj', B, S_XpXn) # Compute: \sum_n ^T SS_XpXp = np.einsum('nkl,nij->ikjl', SS, XpXp) B_SS_XpXp = np.einsum('dik,ikjl->djl', B, SS_XpXp) self.A_XpXp_A = np.einsum('djl,ejl->de', B_SS_XpXp, B) #np.sum(dot(A, XpXp, utils.utils.T(A)), # axis=0) # Compute: \sum_n tr(CovA_n ) self.CovA_XpXp = np.einsum('dikjl,ikjl->d', CovB, SS_XpXp) #self.CovA_XpXp = np.einsum('ndij,nij->d', CovA, XpXp) # Get moments of the fixed parameter nodes mu = self.X_node.parents[0].get_moments()[0] self.Lambda = self.X_node.parents[1].get_moments()[0] self.Lambda_mu_X0 = np.outer(np.dot(self.Lambda,mu), self.X0) self.B_rotator.setup(rotate_plates=True) # Innovation noise is assumed to be I #self.v = self.X_node.parents[3].get_moments()[0] def _compute_bound(self, R, logdet=None, inv=None, gradient=False): """ Rotate q(X) as X->RX: q(X)=N(R*mu, R*Cov*R') Assume: :math:`p(\mathbf{X}) = \prod^M_{m=1} N(\mathbf{x}_m|0, \mathbf{\Lambda})` Assume unit innovation noise covariance. """ # TODO/FIXME: X and alpha should NOT contain observed values!! Check # that. # TODO/FIXME: Allow non-zero prior mean! # Assume constant mean and precision matrix over plates.. invR = inv logdetR = logdet # Transform moments of X: # Transform moments of A: Lambda_R_X0X0 = dot(self.Lambda, R, self.X0X0) R_XnXn = dot(R, self.XnXn) RA_XpXp_A = dot(R, self.A_XpXp_A) sumr = np.sum(R, axis=0) R_CovA_XpXp = sumr * self.CovA_XpXp ## if not gradient: ## print("DEBUG TOO", dot(R_XnXn,R.T)) # Compute entropy H(X) logH_X = utils.random.gaussian_entropy(-2*self.N*logdetR, 0) # Compute yy = tracedot(R_XnXn, R.T) + tracedot(Lambda_R_X0X0, R.T) yz = tracedot(dot(R,self.A_XpXn),R.T) + tracedot(self.Lambda_mu_X0, R.T) zz = tracedot(RA_XpXp_A, R.T) + np.dot(R_CovA_XpXp, sumr) #RR_CovA_XpXp logp_X = utils.random.gaussian_logpdf(yy, yz, zz, 0, 0) # Compute dH(X) dlogH_X = utils.random.gaussian_entropy(-2*self.N*invR.T, 0) # Compute the bound bound = (0 + logp_X + logH_X ) #if not gradient: # print("Debug in transformations", bound) if gradient: # Compute d dyy = 2 * (R_XnXn + Lambda_R_X0X0) dyz = dot(R, self.A_XpXn + self.A_XpXn.T) + self.Lambda_mu_X0 dzz = 2 * (RA_XpXp_A + R_CovA_XpXp) dlogp_X = utils.random.gaussian_logpdf(dyy, dyz, dzz, 0, 0) d_bound = (0*dlogp_X + dlogp_X + dlogH_X ) return (bound, d_bound) else: return bound def bound(self, R, logdet=None, inv=None): (bound_X, d_bound_X) = self._compute_bound(R, logdet=logdet, inv=inv, gradient=True) # Compute cost and gradient from A (bound_B, dR_bound_B, dQ_bound_B) = self.B_rotator.bound(inv.T, inv=R.T, logdet=-logdet, Q=R) # TODO/FIXME: Also apply the gradient of invR.T to the result dR_bound_B = -dot(inv.T, dR_bound_B.T, inv.T) # Compute the bound bound = bound_X + bound_B d_bound = d_bound_X + dR_bound_B + dQ_bound_B return (bound, d_bound) def get_bound_terms(self, R, logdet=None, inv=None): bound_dict = self.B_rotator.get_bound_terms(inv.T, inv=R.T, logdet=-logdet, Q=R) bound_X = self._compute_bound(R, logdet=logdet, inv=inv, gradient=False) bound_dict.update({self: bound_X}) return bound_dict ## def get_gradient_terms(self, R, logdet=None, inv=None): ## grad_dict = self.B_rotator.get_gradient_terms(inv.T, ## inv=R.T, ## logdet=-logdet, ## Q=R) ## (_, grad_X) = self._compute_bound(R, ## logdet=logdet, ## inv=inv, ## gradient=True) ## grad_dict.update({self: grad_X}) ## return grad_dict bayespy-0.1/bayespy/inference/vmp/tests/0000775000175000017500000000000012174043062022123 5ustar jluttinejluttine00000000000000bayespy-0.1/bayespy/inference/vmp/tests/vmp.py0000664000175000017500000003102712174013072023300 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np #import scipy as sp import matplotlib.pyplot as plt import time # Profiling stuff #import profile import cProfile import pstats import imp import utils import Nodes.ExponentialFamily as EF import Nodes.CovarianceFunctions as CF import Nodes.GaussianProcesses as GP imp.reload(utils) imp.reload(EF) imp.reload(CF) imp.reload(GP) # * Model # * * Nodes # * * * GaussianScalar # * * * GaussianVector # * * * Gamma # * Inference # * * VB # * * * Nodes # * * * * GaussianScalar # * * * * GaussianVector # * * * * Gamma # * * * * Constant # * * * * Delta # * * EP # * * MCMC # * * * Nodes # * * * * Gibbs # * * * * * GaussianScalar # * * * * * GaussianVector # * Plotting # VB.update(W,X,tau, repeat=20) # Protocols: # Value # VB.GaussianScalar # VB.GaussianVector # VB.Gamma # EP.GaussianScalar # ... # input_protocols = [VB.GaussianScalar, VB.Gamma] # output_protocol = VB.GassianVector # # or do i need to check input protocols.. child checks parents' output # protocols and that's it. def m_plot(x, Y, style): Y = np.atleast_2d(Y) M = Y.shape[-2] for i in range(M): plt.subplot(M,1,i+1) plt.plot(x, Y[i], style) def m_errorplot(x, Y, L, U): Y = np.atleast_2d(Y) L = np.atleast_2d(L) U = np.atleast_2d(U) M = Y.shape[-2] ## print(np.shape(Y)) ## print(np.shape(L)) ## print(np.shape(U)) ## print(np.shape(M)) for i in range(M): plt.subplot(M,1,i+1) lower = Y[i] - L[i] upper = Y[i] + U[i] #print(upper-lower) #if np.any(lower>=upper): #print('WTF?!') plt.fill_between(x, upper, lower, #where=(upper>=lower), facecolor=(0.6,0.6,0.6,1), edgecolor=(0,0,0,0), #edgecolor=(0.6,0.6,0.6,1), linewidth=0, interpolate=True) plt.plot(x, Y[i], color=(0,0,0,1)) plt.ylabel(str(i)) # MULTIVARIATE GP!! def test_sparse_gp(): ## Generate data # Noisy observations from a sinusoid N = 10000 func = lambda x: np.sin(x*2*np.pi/20) x = np.random.uniform(low=0, high=N, size=(N,)) f = func(x) y = f + np.random.normal(0, 0.2, np.shape(f)) # Plot data plt.clf() plt.plot(x,y,'r+') ## Construct model # Covariance function stuff ls = EF.NodeConstantScalar(3, name='lengthscale') amp = EF.NodeConstantScalar(2.0, name='amplitude') noise = EF.NodeConstantScalar(0.6, name='noise') # Latent process covariance #K_f = CF.SquaredExponential(amp, ls) K_f = CF.PiecewisePolynomial2(amp, ls) # Noise process covariance K_noise = CF.Delta(noise) # Observation process covariance K_y = CF.Sum(K_f, K_noise) # Joint covariance #K_joint = CF.Multiple([[K_f, K_f],[K_f,K_y]], sparse=True) # Mean function stuff M = GP.Constant(lambda x: (x/10-2)*(x/10+1)) # Means for latent and observation processes #M_multi = GP.Multiple([M, M]) # Gaussian process F = GP.GaussianProcess(M, [[K_f, K_f], [K_f, K_y]]) #F = GP.GaussianProcess(M, [[K_f, K_f], [K_f, K_y]]) #F = GP.GaussianProcess(M_multi, K_joint) ## Inference F.observe([[],x], y) utils.vb_optimize_nodes(ls, amp, noise) F.update() u = F.get_parameters() ## Show results # Print hyperparameters print('parameters') print(ls.name, ls.u[0]) print(amp.name, amp.u[0]) print(noise.name, noise.u[0]) # Posterior predictions xh = np.arange(np.min(x)-5, np.max(x)+10, 0.1) (fh, varfh) = u([[],xh], covariance=1) #(fh, varfh) = u([xh,[]], covariance=1) # Plot predictive distribution varfh[varfh<0] = 0 errfh = np.sqrt(varfh) m_errorplot(xh, fh, errfh, errfh) return def test_gp(): # Generate data func = lambda x: np.sin(x*2*np.pi/5) x = np.random.uniform(low=0, high=10, size=(100,)) f = func(x) f = f + np.random.normal(0, 0.2, np.shape(f)) plt.clf() plt.plot(x,f,'r+') #plt.plot(x,y,'r+') # Construct model ls = EF.NodeConstantScalar(1.5, name='lengthscale') amp = EF.NodeConstantScalar(2.0, name='amplitude') noise = EF.NodeConstantScalar(0.6, name='noise') K = CF.SquaredExponential(amp, ls) K_noise = CF.Delta(noise) K_sum = CF.Sum(K, K_noise) M = GP.Constant(lambda x: (x/10-2)*(x/10+1)) method = 'multi' if method == 'sum': # Sum GP F = GP.GaussianProcess(M, K_sum) elif method == 'multi': # Joint for latent function and observation process M_multi = GP.Multiple([M, M]) K_zeros = CF.Zeros() #K_multi = CF.Multiple([[K, K],[K,K_sum]]) K_multi1 = CF.Multiple([[K, K],[K,K]]) #K_multi2 = CF.Multiple([[None, None],[None, K_noise]]) K_multi2 = CF.Multiple([[K_zeros, K_zeros],[K_zeros,K_noise]]) xp = np.arange(0,5,1) F = GP.GaussianProcess(M_multi, K_multi1, k_sparse=K_multi2, #pseudoinputs=[[],xp]) pseudoinputs=None) #F = GP.GaussianProcess(M_multi, K_multi, pseudoinputs=[[],xp]) # Observations are from the latter process: #xf = np.array([]) #x_pseudo = [[], x] #x_full = [np.array([15, 20]), []] #xy = x #x = [xf, xy] #f = np.concatenate([func(xf), f]) #f_pseudo = f #f_full = func(x_full) x = [[], x] # Inference #F.observe(x_pseudo, f_pseudo, pseudo=True) F.observe(x, f) utils.vb_optimize_nodes(ls, amp, noise) F.update() u = F.get_parameters() print('parameters') print(ls.name) print(ls.u[0]) print(amp.name) print(amp.u[0]) print(noise.name) print(noise.u[0]) #print(F.lower_bound_contribution()) # Posterior predictions xh = np.arange(-5, 20, 0.1) if method == 'multi': # Choose which process you want to examine: (fh, varfh) = u([[],xh], covariance=1) #(fh, varfh) = u([xh,[]], covariance=1) else: (fh, varfh) = u(xh, covariance=1) #print(fh) #print(np.shape(fh)) #print(np.shape(varfh)) varfh[varfh<0] = 0 errfh = np.sqrt(varfh) #print(varfh) #print(errfh) m_errorplot(xh, fh, errfh, errfh) return # Construct a GP k = gp_cov_se(magnitude=theta1, lengthscale=theta2) f = NodeGP(0, k) f.observe(x, y) f.update() (mp, kp) = f.get_parameters() def test_pca(): # Dimensionalities dataset = 1 if dataset == 1: M = 10 N = 100 D_y = 3 D = 3+2 # Generate data w = np.random.normal(0, 1, size=(M,1,D_y)) x = np.random.normal(0, 1, size=(1,N,D_y)) f = sum_product(w, x, axes_to_sum=[-1])#np.einsum('...i,...i', w, x) y = f + np.random.normal(0, 0.5, size=(M,N)) elif dataset == 2: # Data from matlab comparison f = np.genfromtxt('/home/jluttine/matlab/fa/data_pca_01_f.txt') y = np.genfromtxt('/home/jluttine/matlab/fa/data_pca_01_y.txt') D = np.genfromtxt('/home/jluttine/matlab/fa/data_pca_01_d.txt') (M,N) = np.shape(y) # Construct the PCA model with ARD alpha = NodeGamma(1e-10, 1e-10, plates=(D,), name='alpha') alpha.update() diag_alpha = NodeWishartFromGamma(alpha) #Lambda = NodeWishart(D, D * np.identity(D), plates=(), name='Lambda') #Lambda.update() X = NodeGaussian(np.zeros(D), np.identity(D), name="X", plates=(1,N)) X.update() X.u[0] = X.random() W = NodeGaussian(np.zeros(D), diag_alpha, name="W", plates=(M,1)) #W = NodeGaussian(np.zeros(D), Lambda, name="W", plates=(M,1)) #W = NodeGaussian(np.zeros(D), np.identity(D), name="W", plates=(M,1)) W.update() W.u[0] = W.random() WX = NodeDot(W,X,S,R) tau = NodeGamma(1e-5, 1e-5, name="tau", plates=(M,N)) tau.update() Y = NodeNormal(WX, tau, name="Y", plates=(M,N)) Y.update() # Initialize (from prior) # Y.update() # mask = True # mask = np.ones((M,N), dtype=np.bool) mask = np.random.rand(M,N) < 0.4 mask[:,20:40] = False Y.observe(y, mask) # Inference L_last = -np.inf for i in range(100): t = time.clock() X.update() W.update() tau.update() #Lambda.update() alpha.update() L_X = X.lower_bound_contribution() L_W = W.lower_bound_contribution() L_tau = tau.lower_bound_contribution() L_Y = Y.lower_bound_contribution() L_alpha = alpha.lower_bound_contribution() #print("X: %f, W: %f, tau: %f, Y: %f" % (L_X, L_W, L_tau, L_Y)) L = L_X + L_W + L_tau + L_Y print("Iteration %d: loglike=%e (%.3f seconds)" % (i+1, L, time.clock()-t)) if L_last > L: L_diff = (L_last - L) #raise Exception("Lower bound decreased %e! Bug somewhere or numerical inaccuracy?" % L_diff) if L - L_last < 1e-12: print("Converged.") #break L_last = L #return #print(shape(yh)) plt.figure(1) plt.clf() WX_params = WX.get_parameters() fh = WX_params[0] * np.ones(y.shape) err_fh = 2*np.sqrt(WX_params[1]) * np.ones(y.shape) m_errorplot(np.arange(N), fh, err_fh, err_fh) m_plot(np.arange(N), f, 'g') m_plot(np.arange(N), y, 'r+') #plt.figure(2) #plt.clf() #alpha.show() print(alpha.u[0]) tau.show() def test_normal(): M = 10 N = 5 # mu mu = NodeNormal(0.0, 10**-5, name="mu", plates=()) print("Prior for mu:") mu.update() mu.show() # tau tau = NodeGamma(10**-5, 10**-5, plates=(N,), name="tau") print("Prior for tau:") tau.update() tau.show() # x x = NodeNormal(mu, tau, plates=(M,N), name="x") print("Prior for x:") x.update() x.show() # y (generate data) y = NodeNormal(x, 1, plates=(M,N), name="y") y.observe(random.normal(loc=10, scale=10, size=(M,N))) # Inference for i in range(50): x.update() mu.update() tau.update() print("Posterior for mu:") mu.show() print("Posterior for tau:") tau.show() print("Posterior for x:") x.show() return def test_multivariate(): D = 3 N = 100 M = 200 # mu mu = NodeGaussian(np.zeros(D), 10**(-10)*np.identity(D), plates=(M,1), name='mu') print("Prior for mu:") mu.update() mu.show() # Lambda Lambda = NodeWishart(D, (10**-10) * np.identity(D), plates=(1,N), name='Lambda') print("Prior for Lambda:") Lambda.update() Lambda.show() #Y = NodeGaussian(mu, 10**(-2)*identity(D), plates=(M,N), name='Y') Y = NodeGaussian(mu, Lambda, plates=(M,N), name='Y') Y.observe(random.normal(loc=10, scale=10, size=(M,N,D))) ## # y (generate data) ## for i in range(100): ## y = NodeGaussian(mu, Lambda) ## v = random.normal(0,1, D) ## y.fix(v) # Inference try: for i in range(50): mu.update() Lambda.update() print("Posterior for mu:") mu.show() print("Posterior for Lambda:") Lambda.show() except Exception: pass if __name__ == '__main__': # FOR INTERACTIVE SESSIONS, NON-BLOCKING PLOTTING: plt.ion() test_sparse_gp() #test_gp() #test_pca() #cProfile.run('test_sparse_gp()', 'profile.tmp') #S = pstats.Stats('profile.tmp') #test_normal() #test_multivariate() bayespy-0.1/bayespy/inference/vmp/tests/__init__.py0000664000175000017500000000000012174013072024220 0ustar jluttinejluttine00000000000000bayespy-0.1/bayespy/inference/vmp/tests/gp.py0000664000175000017500000005711112174013072023106 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import itertools import numpy as np import scipy as sp import scipy.linalg.decomp_cholesky as decomp import scipy.linalg as linalg import scipy.special as special import matplotlib.pyplot as plt import time import profile import scipy.spatial.distance as distance from nodes import * from utils import * # Computes log probability density function of the Gaussian # distribution def gaussian_logpdf(y_invcov_y, y_invcov_mu, mu_invcov_mu, logdetcov, D): return (-0.5*D*np.log(2*np.pi) -0.5*logdetcov -0.5*y_invcov_y +y_invcov_mu -0.5*mu_invcov_mu) # m prior mean function # k prior covariance function # x data inputs # z processed data outputs (z = inv(Cov) * (y-m(x))) # U data covariance Cholesky factor def gp_posterior_moment_function(m, k, x, y, noise=None): # Prior mu = m(x)[0] K = k(x,x)[0] if noise != None: K += noise #print('hereiamagain') #print(K) # Compute posterior GP N = len(y) if N == 0: U = None z = None else: U = chol(K) z = chol_solve(U, y-mu) def get_moments(xh, covariance=1, mean=True): (kh,) = k(x, xh) # Function for computing posterior moments if mean: # Mean vector mh = m(xh) if z != None: mh += np.dot(kh.T, z) else: mh = None if covariance: if covariance == 1: # Variance vector khh = k(xh) if U != None: khh -= np.einsum('i...,i...', kh, chol_solve(U, kh)) elif covariance == 2: # Full covariance matrix khh = k(xh,xh) if U != None: khh -= np.dot(kh.T, chol_solve(U,kh)) else: khh = None return [mh, khh] return get_moments # m prior mean function # k prior covariance function # x data inputs # z processed data outputs (z = inv(Cov) * (y-m(x))) # U data covariance Cholesky factor ## def gp_multi_posterior_moment_function(m, k, x, y, noise=None): ## # Prior ## mu = m(x)[0] ## K = k(x,x)[0] ## if noise != None: ## K += noise ## #print('hereiamagain') ## #print(K) ## # Compute posterior GP ## N = len(y) ## if N == 0: ## U = None ## z = None ## else: ## U = chol(K) ## z = chol_solve(U, y-mu) ## def get_moments(xh, covariance=1, mean=True): ## (kh,) = k(x, xh) ## # Function for computing posterior moments ## if mean: ## # Mean vector ## mh = m(xh) ## if z != None: ## mh += np.dot(kh.T, z) ## else: ## mh = None ## if covariance: ## if covariance == 1: ## # Variance vector ## khh = k(xh) ## if U != None: ## khh -= np.einsum('i...,i...', kh, chol_solve(U, kh)) ## elif covariance == 2: ## # Full covariance matrix ## khh = k(xh,xh) ## if U != None: ## khh -= np.dot(kh.T, chol_solve(U,kh)) ## else: ## khh = None ## return [mh, khh] ## return get_moments def gp_cov_se(D2, overwrite=False): if overwrite: K = D2 K *= -0.5 np.exp(K, out=K) else: K = np.exp(-0.5*D2) return K def gp_cov_delta(N): return np.identity(N) def squared_distance(x1, x2): # Reshape arrays to 2-D arrays sh1 = np.shape(x1)[:-1] sh2 = np.shape(x2)[:-1] d = np.shape(x1)[-1] x1 = np.reshape(x1, (-1,d)) x2 = np.reshape(x2, (-1,d)) # Compute squared Euclidean distance D2 = distance.cdist(x1, x2, metric='sqeuclidean') # Reshape the result D2 = np.reshape(D2, sh1 + sh2) return D2 # General rule for the parameters for covariance functions: # # (value, [ [dvalue1, ...], [dvalue2, ...], [dvalue3, ...], ...]) # # For instance, # # k = covfunc_se((1.0, []), (15, [ [1,update_grad] ])) # K = k((x1, [ [dx1,update_grad] ]), (x2, [])) # # Plain values are converted as: # value -> (value, []) def gp_standardize_input(x): if np.ndim(x) == 0: x = add_trailing_axes(x, 2) elif np.ndim(x) == 1: x = add_trailing_axes(x, 1) return x def gp_preprocess_inputs(*args): args = list(args) if len(args) < 1 or len(args) > 2: raise Exception("Number of inputs must be one or two") if len(args) == 2: if args[0] is args[1]: args[0] = gp_standardize_input(args[0]) args[1] = args[0] else: args[1] = gp_standardize_input(args[1]) args[0] = gp_standardize_input(args[0]) else: args[0] = gp_standardize_input(args[0]) return args def covfunc_delta(theta, *inputs, gradient=False): amplitude = theta[0] if gradient: gradient_amplitude = gradient[0] else: gradient_amplitude = [] inputs = gp_preprocess_inputs(*inputs) # Compute distance and covariance matrix if len(inputs) == 1: # Only variance vector asked x = inputs[0] K = np.ones(np.shape(x)[:-1]) * amplitude**2 else: # Full covariance matrix asked x1 = inputs[0] x2 = inputs[1] # Number of inputs x1 N1 = np.shape(x1)[-2] # x1 == x2? if x1 is x2: delta = True # Delta covariance K = gp_cov_delta(N1) * amplitude**2 else: delta = False # Number of inputs x2 N2 = np.shape(x2)[-2] # Zero covariance K = np.zeros((N1,N2)) # Gradient w.r.t. amplitude if gradient: for ind in range(len(gradient_amplitude)): gradient_amplitude[ind] = K * (2 * gradient_amplitude[ind] / amplitude) if gradient: return (K, gradient) else: return K def covfunc_se(theta, *inputs, gradient=False): amplitude = theta[0] lengthscale = theta[1] ## print('in se') ## print(amplitude) ## print(lengthscale) if gradient: gradient_amplitude = gradient[0] gradient_lengthscale = gradient[1] else: gradient_amplitude = [] gradient_lengthscale = [] inputs = gp_preprocess_inputs(*inputs) # Compute covariance matrix if len(inputs) == 1: x = inputs[0] # Compute variance vector K = np.ones(np.shape(x)[:-1]) K *= amplitude**2 # Compute gradient w.r.t. lengthscale for ind in range(len(gradient_lengthscale)): gradient_lengthscale[ind] = np.zeros(np.shape(x)[:-1]) else: x1 = inputs[0] / (lengthscale) x2 = inputs[1] / (lengthscale) # Compute distance matrix K = squared_distance(x1, x2) # Compute gradient partly if gradient: for ind in range(len(gradient_lengthscale)): gradient_lengthscale[ind] = K * ((lengthscale**-1) * gradient_lengthscale[ind]) # Compute covariance matrix gp_cov_se(K, overwrite=True) K *= amplitude**2 # Compute gradient w.r.t. lengthscale if gradient: for ind in range(len(gradient_lengthscale)): gradient_lengthscale[ind] *= K # Gradient w.r.t. amplitude if gradient: for ind in range(len(gradient_amplitude)): gradient_amplitude[ind] = K * (2 * gradient_amplitude[ind] / amplitude) # Return values if gradient: return (K, gradient) else: return K class NodeCovarianceFunction(Node): def __init__(self, covfunc, *args, **kwargs): self.covfunc = covfunc params = list(args) for i in range(len(args)): # Check constant parameters if is_numeric(args[i]): params[i] = NodeConstant([np.asanyarray(args[i])], dims=[np.shape(args[i])]) # TODO: Parameters could be constant functions? :) Node.__init__(self, *params, dims=[(np.inf, np.inf)], **kwargs) def message_to_child(self, gradient=False): params = [parent.message_to_child(gradient=gradient) for parent in self.parents] return self.covariance_function(*params) def covariance_function(self, *params): params = list(params) gradient_params = list() for ind in range(len(params)): if isinstance(params[ind], tuple): gradient_params.append(params[ind][1]) params[ind] = params[ind][0][0] else: gradient_params.append([]) params[ind] = params[ind][0] def cov(*inputs, gradient=False): if gradient: grads = [[grad[0] for grad in gradient_params[ind]] for ind in range(len(gradient_params))] (K, dK) = self.covfunc(params, *inputs, gradient=grads) for ind in range(len(dK)): for (grad, dk) in zip(gradient_params[ind], dK[ind]): grad[0] = dk K = [K] dK = [] for grad in gradient_params: dK += grad return (K, dK) else: K = self.covfunc(params, *inputs, gradient=False) return [K] return cov class NodeCovarianceFunctionSum(NodeCovarianceFunction): def __init__(self, *args, **kwargs): NodeCovarianceFunction.__init__(self, None, *args, **kwargs) def covariance_function(self, *covfuncs): def cov(*inputs, gradient=False): K_sum = 0 if gradient: dK_sum = list() for k in covfuncs: if gradient: (K, dK) = k(*inputs, gradient=gradient) dK_sum += dK else: K = k(*inputs, gradient=gradient) K_sum += K[0] if gradient: return ([K_sum], dK_sum) else: return [K_sum] return cov class NodeCovarianceFunctionDelta(NodeCovarianceFunction): def __init__(self, amplitude, **kwargs): NodeCovarianceFunction.__init__(self, covfunc_delta, amplitude, **kwargs) class NodeCovarianceFunctionSquaredExponential(NodeCovarianceFunction): def __init__(self, amplitude, lengthscale, **kwargs): NodeCovarianceFunction.__init__(self, covfunc_se, amplitude, lengthscale, **kwargs) class NodeMultiCovarianceFunction(NodeCovarianceFunction): def __init__(self, *args, **kwargs): NodeCovarianceFunction.__init__(self, None, *args, **kwargs) def covfunc(self, *covfuncs): def cov(*inputs, gradient=False): K_sum = 0 if gradient: dK_sum = list() for k in covfuncs: if gradient: (K, dK) = k(*inputs, gradient=gradient) dK_sum += dK else: K = k(*inputs, gradient=gradient) K_sum += K[0] if gradient: return ([K_sum], dK_sum) else: return [K_sum] return cov class NodeConstantGaussianProcess(Node): def __init__(self, f, **kwargs): self.f = f Node.__init__(self, dims=[(np.inf,)], **kwargs) def message_to_child(self, gradient=False): # Wrapper def func(x, gradient=False): if gradient: return ([self.f(x)], []) else: return [self.f(x)] return func # At least for now, simplify this GP node such that a GP is either # observed or latent. If it is observed, it doesn't take messages from # children, actually, it should not even have children! class NodeMultiGaussianProcess(NodeVariable): def __init__(self, m, k, **kwargs): self.x = [] self.f = [] # By default, posterior == prior self.m = m self.k = k # Ignore plates NodeVariable.__init__(self, m, k, plates=(), dims=[(np.inf,), (np.inf,np.inf)], **kwargs) def message_to_parent(self, index): if index == 0: k = self.parents[1].message_to_child()[0] K = k(self.x, self.x) return [self.x, self.mu, K] if index == 1: raise Exception("not implemented yet") def message_to_child(self): if self.observed: raise Exception("Observable GP should not have children.") return self.u def get_parameters(self): return self.u def observe(self, x, f): if np.ndim(x) == 1: if np.shape(f) != np.shape(x): print(np.shape(f)) print(np.shape(x)) raise Exception("Number of inputs and function values do not match") elif np.shape(f) != np.shape(x)[:-1]: print(np.shape(f)) print(np.shape(x)) raise Exception("Number of inputs and function values do not match") self.observed = True self.x = x self.f = f ## self.x_obs = x ## self.f_obs = f # You might want: # - mean for x # - covariance (and mean) for x # - variance (and mean) for x # - i.e., mean and/or (co)variance for x # - covariance for x1 and x2 def lower_bound_contribution(self, gradient=False): m = self.parents[0].message_to_child(gradient=gradient) k = self.parents[1].message_to_child(gradient=gradient) ## m = self.parents[0].message_to_child(gradient=gradient)[0] ## k = self.parents[1].message_to_child(gradient=gradient)[0] # Prior if gradient: (mu, dmus) = m(self.x, gradient=True) (K, dKs) = k(self.x, self.x, gradient=True) else: mu = m(self.x) K = k(self.x, self.x) dmus = [] dKs = [] mu = mu[0] K = K[0] # Log pdf if self.observed: # Vector of f-mu f0 = np.vstack([(f-m) for (f,m) in zip(self.f,mu)]) # Full covariance matrix K_full = np.bmat(K) try: U = chol(K_full) except linalg.LinAlgError: print('non positive definite, return -inf') return -np.inf z = chol_solve(U, f0) #print(K) L = gaussian_logpdf(np.dot(f0, z), 0, 0, logdet_chol(U), np.size(self.f)) for (dmu, func) in dmus: # Derivative w.r.t. mean vector d = -np.sum(z) # Send the derivative message func += d #func(d) for (dK, func) in dKs: # Compute derivative w.r.t. covariance matrix d = 0.5 * (np.dot(z, np.dot(dK, z)) - np.trace(chol_solve(U, dK))) # Send the derivative message #print('add gradient') #func += d func(d) else: raise Exception('Not implemented yet') return L ## Let f1 be observed and f2 latent function values. # Compute #L = gaussian_logpdf(sum_product(np.outer(self.f,self.f) + self.Cov, # Compute def update(self): # Messages from parents m = self.parents[0].message_to_child() k = self.parents[1].message_to_child() ## m = self.parents[0].message_to_child()[0] ## k = self.parents[1].message_to_child()[0] if self.observed: # Observations of this node self.u = gp_posterior_moment_function(m, k, self.x, self.f) else: x = np.array([]) y = np.array([]) # Messages from children for (child,index) in self.children: (msg, mask) = child.message_to_parent(index) # Ignoring masks and plates.. # m[0] is the inputs x = np.concatenate((x, msg[0]), axis=-2) # m[1] is the observations y = np.concatenate((y, msg[1])) # m[2] is the covariance matrix V = linalg.block_diag(V, msg[2]) self.u = gp_posterior_moment_function(m, k, x, y, covariance=V) self.x = x self.f = y class NodeGaussianProcess(NodeVariable): def __init__(self, m, k, **kwargs): self.x = np.array([]) self.f = np.array([]) ## self.x_obs = np.zeros((0,1)) ## self.f_obs = np.zeros((0,)) # By default, posterior == prior self.m = m self.k = k # Ignore plates NodeVariable.__init__(self, m, k, plates=(), dims=[(np.inf,), (np.inf,np.inf)], **kwargs) def message_to_parent(self, index): if index == 0: k = self.parents[1].message_to_child()[0] K = k(self.x, self.x) return [self.x, self.mu, K] if index == 1: raise Exception("not implemented yet") def message_to_child(self): if self.observed: raise Exception("Observable GP should not have children.") return self.u def get_parameters(self): return self.u def observe(self, x, f): if np.ndim(x) == 1: if np.shape(f) != np.shape(x): print(np.shape(f)) print(np.shape(x)) raise Exception("Number of inputs and function values do not match") elif np.shape(f) != np.shape(x)[:-1]: print(np.shape(f)) print(np.shape(x)) raise Exception("Number of inputs and function values do not match") self.observed = True self.x = x self.f = f ## self.x_obs = x ## self.f_obs = f # You might want: # - mean for x # - covariance (and mean) for x # - variance (and mean) for x # - i.e., mean and/or (co)variance for x # - covariance for x1 and x2 def lower_bound_contribution(self, gradient=False): m = self.parents[0].message_to_child(gradient=gradient) k = self.parents[1].message_to_child(gradient=gradient) ## m = self.parents[0].message_to_child(gradient=gradient)[0] ## k = self.parents[1].message_to_child(gradient=gradient)[0] # Prior if gradient: (mu, dmus) = m(self.x, gradient=True) (K, dKs) = k(self.x, self.x, gradient=True) else: mu = m(self.x) K = k(self.x, self.x) dmus = [] dKs = [] mu = mu[0] K = K[0] # Log pdf if self.observed: f0 = self.f - mu #print('hereiam') #print(K) try: U = chol(K) except linalg.LinAlgError: print('non positive definite, return -inf') return -np.inf z = chol_solve(U, f0) #print(K) L = gaussian_logpdf(np.dot(f0, z), 0, 0, logdet_chol(U), np.size(self.f)) for (dmu, func) in dmus: # Derivative w.r.t. mean vector d = -np.sum(z) # Send the derivative message func += d #func(d) for (dK, func) in dKs: # Compute derivative w.r.t. covariance matrix d = 0.5 * (np.dot(z, np.dot(dK, z)) - np.trace(chol_solve(U, dK))) # Send the derivative message #print('add gradient') #func += d func(d) else: raise Exception('Not implemented yet') return L ## Let f1 be observed and f2 latent function values. # Compute #L = gaussian_logpdf(sum_product(np.outer(self.f,self.f) + self.Cov, # Compute def update(self): # Messages from parents m = self.parents[0].message_to_child() k = self.parents[1].message_to_child() ## m = self.parents[0].message_to_child()[0] ## k = self.parents[1].message_to_child()[0] if self.observed: # Observations of this node self.u = gp_posterior_moment_function(m, k, self.x, self.f) else: x = np.array([]) y = np.array([]) # Messages from children for (child,index) in self.children: (msg, mask) = child.message_to_parent(index) # Ignoring masks and plates.. # m[0] is the inputs x = np.concatenate((x, msg[0]), axis=-2) # m[1] is the observations y = np.concatenate((y, msg[1])) # m[2] is the covariance matrix V = linalg.block_diag(V, msg[2]) self.u = gp_posterior_moment_function(m, k, x, y, covariance=V) self.x = x self.f = y bayespy-0.1/bayespy/inference/vmp/__init__.py0000664000175000017500000000000012174013072023056 0ustar jluttinejluttine00000000000000bayespy-0.1/bayespy/inference/vmp/nodes/0000775000175000017500000000000012174043062022071 5ustar jluttinejluttine00000000000000bayespy-0.1/bayespy/inference/vmp/nodes/GaussianProcesses.py0000664000175000017500000006435312174013072026115 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import itertools import numpy as np #import scipy as sp #import scipy.linalg.decomp_cholesky as decomp import scipy.linalg as linalg #import scipy.special as special #import matplotlib.pyplot as plt #import time #import profile #import scipy.spatial.distance as distance import scipy.sparse as sp import utils import Nodes.ExponentialFamily as EF import Nodes.CovarianceFunctions as CF import imp imp.reload(utils) imp.reload(EF) imp.reload(CF) class CovarianceMatrix: def cholesky(self): pass def multiply(A, B): return np.multiply(A,B) # m prior mean function # k prior covariance function # x data inputs # z processed data outputs (z = inv(Cov) * (y-m(x))) # U data covariance Cholesky factor def gp_posterior_moment_function(m, k, x, y, k_sparse=None, pseudoinputs=None, noise=None): # Prior # FIXME: We are ignoring the covariance of mu now.. mu = m(x)[0] ## if np.ndim(mu) == 1: ## mu = np.asmatrix(mu).T ## else: ## mu = np.asmatrix(mu) K_noise = None if noise != None: if K_noise is None: K_noise = noise else: K_noise += noise if k_sparse != None: if K_noise is None: K_noise = k_sparse(x,x)[0] else: K_noise += k_sparse(x,x)[0] if pseudoinputs != None: p = pseudoinputs #print('in pseudostuff') #print(K_noise) #print(np.shape(K_noise)) K_pp = k(p,p)[0] K_xp = k(x,p)[0] U = utils.chol(K_noise) # Compute Lambda Lambda = K_pp + np.dot(K_xp.T, utils.chol_solve(U, K_xp)) U_lambda = utils.chol(Lambda) # Compute statistics for posterior predictions #print(np.shape(U_lambda)) #print(np.shape(y)) z = utils.chol_solve(U_lambda, np.dot(K_xp.T, utils.chol_solve(U, y - mu))) U = utils.chol(K_pp) # Now we can forget the location of the observations and # consider only the pseudoinputs when predicting. x = p else: K = K_noise if K is None: K = k(x,x)[0] else: try: K += k(x,x)[0] except: K = K + k(x,x)[0] # Compute posterior GP N = len(y) U = None z = None if N > 0: U = utils.chol(K) z = utils.chol_solve(U, y-mu) def get_moments(h, covariance=1, mean=True): K_xh = k(x, h)[0] if k_sparse != None: try: # This may not work, for instance, if either one is a # sparse matrix. K_xh += k_sparse(x, h)[0] except: K_xh = K_xh + k_sparse(x, h)[0] # NumPy has problems when mixing matrices and arrays. # Matrices may appear, for instance, when you sum an array and # a sparse matrix. Make sure the result is either an array or # a sparse matrix (not dense matrix!), because matrix objects # cause lots of problems: # # array.dot(array) = array # matrix.dot(array) = matrix # sparse.dot(array) = array if not sp.issparse(K_xh): K_xh = np.asarray(K_xh) # Function for computing posterior moments if mean: # Mean vector # FIXME: Ignoring the covariance of prior mu m_h = m(h)[0] if z != None: m_h += K_xh.T.dot(z) else: m_h = None # Compute (co)variance matrix/vector if covariance: if covariance == 1: ## Compute variance vector k_h = k(h)[0] if k_sparse != None: k_h += k_sparse(h)[0] if U != None: if isinstance(K_xh, np.ndarray): k_h -= np.einsum('i...,i...', K_xh, utils.chol_solve(U, K_xh)) else: # TODO: This isn't very efficient way, but # einsum doesn't work for sparse matrices.. # This may consume A LOT of memory for sparse # matrices. k_h -= np.asarray(K_xh.multiply(utils.chol_solve(U, K_xh))).sum(axis=0) if pseudoinputs != None: if isinstance(K_xh, np.ndarray): k_h += np.einsum('i...,i...', K_xh, utils.chol_solve(U_lambda, K_xh)) else: # TODO: This isn't very efficient way, but # einsum doesn't work for sparse matrices.. # This may consume A LOT of memory for sparse # matrices. k_h += np.asarray(K_xh.multiply(utils.chol_solve(U_lambda, K_xh))).sum(axis=0) # Ensure non-negative variances k_h[k_h<0] = 0 return (m_h, k_h) elif covariance == 2: ## Compute full covariance matrix K_hh = k(h,h)[0] if k_sparse != None: K_hh += k_sparse(h)[0] if U != None: K_hh -= K_xh.T.dot(utils.chol_solve(U,K_xh)) #K_hh -= np.dot(K_xh.T, utils.chol_solve(U,K_xh)) if pseudoinputs != None: K_hh += K_xh.T.dot(utils.chol_solve(U_lambda, K_xh)) #K_hh += np.dot(K_xh.T, utils.chol_solve(U_lambda, K_xh)) return (m_h, K_hh) else: return (m_h, None) return get_moments # Constant function using GP mean protocol class Constant(EF.Node): def __init__(self, f, **kwargs): self.f = f EF.Node.__init__(self, dims=[(np.inf,)], **kwargs) def message_to_child(self, gradient=False): # Wrapper def func(x, gradient=False): if gradient: return ([self.f(x), None], []) else: return [self.f(x), None] return func #class MultiDimensional(EF.NodeVariable): # """ A multi-dimensional Gaussian process f(x). """ ## class ToGaussian(EF.NodeVariable): ## """ Deterministic node which transform a Gaussian process into ## finite-dimensional Gaussian variable. """ ## def __init__(self, f, x, **kwargs): ## EF.NodeVariable.__init__(self, ## f, ## x, ## plates= ## dims= # Deterministic node for creating a set of GPs which can be used as a # mean function to a general GP node. class Multiple(EF.NodeVariable): def __init__(self, GPs, **kwargs): # Ignore plates EF.NodeVariable.__init__(self, *GPs, plates=(), dims=[(np.inf,), (np.inf,np.inf)], **kwargs) def message_to_parent(self, index): raise Exception("not implemented yet") def message_to_child(self, gradient=False): u = [parent.message_to_child() for parent in self.parents] def get_moments(xh, **kwargs): mh_all = [] khh_all = [] for i in range(len(self.parents)): xi = np.array(xh[i]) #print(xi) #print(np.shape(xi)) #print(xi) # FIXME: We are ignoring the covariance of mu now.. if gradient: ((mh, khh), dm) = u[i](xi, **kwargs) else: (mh, khh) = u[i](xi, **kwargs) #mh = u[i](xi, **kwargs)[0] #print(mh) #print(mh_all) ## print(mh) ## print(khh) ## print(np.shape(mh)) mh_all = np.concatenate([mh_all, mh]) #print(np.shape(mh_all)) if khh != None: print(khh) raise Exception('Not implemented yet for covariances') #khh_all = np.concatenate([khh_all, khh]) # FIXME: Compute gradients! if gradient: return ([mh_all, khh_all], []) else: return [mh_all, khh_all] #return [mh_all, khh_all] return get_moments # Gaussian process distribution class GaussianProcess(EF.NodeVariable): def __init__(self, m, k, k_sparse=None, pseudoinputs=None, **kwargs): self.x = np.array([]) self.f = np.array([]) ## self.x_obs = np.zeros((0,1)) ## self.f_obs = np.zeros((0,)) if pseudoinputs != None: pseudoinputs = EF.NodeConstant([pseudoinputs], dims=[np.shape(pseudoinputs)]) # By default, posterior == prior self.m = None #m self.k = None #k if isinstance(k, list) and isinstance(m, list): if len(k) != len(m): raise Exception('The number of mean and covariance functions must be equal.') k = CF.Multiple(k) m = Multiple(m) elif isinstance(k, list): D = len(k) k = CF.Multiple(k) m = Multiple(D*[m]) elif isinstance(m, list): D = len(m) k = CF.Multiple(D*[k]) m = Multiple(m) # Ignore plates EF.NodeVariable.__init__(self, m, k, k_sparse, pseudoinputs, plates=(), dims=[(np.inf,), (np.inf,np.inf)], **kwargs) def __call__(self, x, covariance=None): if not covariance: return self.u(x, covariance=False)[0] elif covariance.lower() == 'vector': return self.u(x, covariance=1) elif covariance.lower() == 'matrix': return self.u(x, covariance=2) else: raise Exception("Unknown covariance type requested") def message_to_parent(self, index): if index == 0: k = self.parents[1].message_to_child()[0] K = k(self.x, self.x) return [self.x, self.mu, K] if index == 1: raise Exception("not implemented yet") def message_to_child(self): if self.observed: raise Exception("Observable GP should not have children.") return self.u def get_parameters(self): return self.u def observe(self, x, f): self.observed = True self.x = x self.f = f ## if np.ndim(f) == 1: ## self.f = np.asmatrix(f).T ## else: ## self.f = np.asmatrix(f) # You might want: # - mean for x # - covariance (and mean) for x # - variance (and mean) for x # - i.e., mean and/or (co)variance for x # - covariance for x1 and x2 def lower_bound_contribution(self, gradient=False): # Get moment functions from parents m = self.parents[0].message_to_child(gradient=gradient) k = self.parents[1].message_to_child(gradient=gradient) if self.parents[2]: k_sparse = self.parents[2].message_to_child(gradient=gradient) else: k_sparse = None if self.parents[3]: pseudoinputs = self.parents[3].message_to_child(gradient=gradient) #pseudoinputs = self.parents[3].message_to_child(gradient=gradient)[0] else: pseudoinputs = None ## m = self.parents[0].message_to_child(gradient=gradient)[0] ## k = self.parents[1].message_to_child(gradient=gradient)[0] # Compute the parameters (covariance matrices etc) using # parents' moment functions DKs_xx = [] DKd_xx = [] DKd_xp = [] DKd_pp = [] Dxp = [] Dmu = [] if gradient: # FIXME: We are ignoring the covariance of mu now.. ((mu, _), Dmu) = m(self.x, gradient=True) ## if k_sparse: ## ((Ks_xx,), DKs_xx) = k_sparse(self.x, self.x, gradient=True) if pseudoinputs: ((Ks_xx,), DKs_xx) = k_sparse(self.x, self.x, gradient=True) ((xp,), Dxp) = pseudoinputs ((Kd_pp,), DKd_pp) = k(xp,xp, gradient=True) ((Kd_xp,), DKd_xp) = k(self.x, xp, gradient=True) else: ((K_xx,), DKd_xx) = k(self.x, self.x, gradient=True) if k_sparse: ((Ks_xx,), DKs_xx) = k_sparse(self.x, self.x, gradient=True) try: K_xx += Ks_xx except: K_xx = K_xx + Ks_xx else: # FIXME: We are ignoring the covariance of mu now.. (mu, _) = m(self.x) ## if k_sparse: ## (Ks_xx,) = k_sparse(self.x, self.x) if pseudoinputs: (Ks_xx,) = k_sparse(self.x, self.x) (xp,) = pseudoinputs (Kd_pp,) = k(xp, xp) (Kd_xp,) = k(self.x, xp) else: (K_xx,) = k(self.x, self.x) if k_sparse: (Ks_xx,) = k_sparse(self.x, self.x) try: K_xx += Ks_xx except: K_xx = K_xx + Ks_xx mu = mu[0] #K = K[0] # Log pdf if self.observed: ## Log pdf for directly observed GP f0 = self.f - mu #print('hereiam') #print(K) if pseudoinputs: ## Pseudo-input approximation # Decompose the full-rank sparse/noise covariance matrix try: Us_xx = utils.cholesky(Ks_xx) except linalg.LinAlgError: print('Noise/sparse covariance not positive definite') return -np.inf # Use Woodbury-Sherman-Morrison formula with the # following notation: # # y2 = f0' * inv(Kd_xp*inv(Kd_pp)*Kd_xp' + Ks_xx) * f0 # # z = Ks_xx \ f0 # Lambda = Kd_pp + Kd_xp'*inv(Ks_xx)*Kd_xp # nu = inv(Lambda) * (Kd_xp' * (Ks_xx \ f0)) # rho = Kd_xp * inv(Lambda) * (Kd_xp' * (Ks_xx \ f0)) # # y2 = f0' * z - z' * rho z = Us_xx.solve(f0) Lambda = Kd_pp + np.dot(Kd_xp.T, Us_xx.solve(Kd_xp)) ## z = utils.chol_solve(Us_xx, f0) ## Lambda = Kd_pp + np.dot(Kd_xp.T, ## utils.chol_solve(Us_xx, Kd_xp)) try: U_Lambda = utils.cholesky(Lambda) #U_Lambda = utils.chol(Lambda) except linalg.LinAlgError: print('Lambda not positive definite') return -np.inf nu = U_Lambda.solve(np.dot(Kd_xp.T, z)) #nu = utils.chol_solve(U_Lambda, np.dot(Kd_xp.T, z)) rho = np.dot(Kd_xp, nu) y2 = np.dot(f0, z) - np.dot(z, rho) # Use matrix determinant lemma # # det(Kd_xp*inv(Kd_pp)*Kd_xp' + Ks_xx) # = det(Kd_pp + Kd_xp'*inv(Ks_xx)*Kd_xp) # * det(inv(Kd_pp)) * det(Ks_xx) # = det(Lambda) * det(Ks_xx) / det(Kd_pp) try: Ud_pp = utils.cholesky(Kd_pp) #Ud_pp = utils.chol(Kd_pp) except linalg.LinAlgError: print('Covariance of pseudo inputs not positive definite') return -np.inf logdet = (U_Lambda.logdet() + Us_xx.logdet() - Ud_pp.logdet()) ## logdet = (utils.logdet_chol(U_Lambda) ## + utils.logdet_chol(Us_xx) ## - utils.logdet_chol(Ud_pp)) # Compute the log pdf L = gaussian_logpdf(y2, 0, 0, logdet, np.size(self.f)) # Add the variational cost of the pseudo-input # approximation # Compute gradients for (dmu, func) in Dmu: # Derivative w.r.t. mean vector d = np.nan # Send the derivative message func(d) for (dKs_xx, func) in DKs_xx: # Compute derivative w.r.t. covariance matrix d = np.nan # Send the derivative message func(d) for (dKd_xp, func) in DKd_xp: # Compute derivative w.r.t. covariance matrix d = np.nan # Send the derivative message func(d) V = Ud_pp.solve(Kd_xp.T) Z = Us_xx.solve(V.T) ## V = utils.chol_solve(Ud_pp, Kd_xp.T) ## Z = utils.chol_solve(Us_xx, V.T) for (dKd_pp, func) in DKd_pp: # Compute derivative w.r.t. covariance matrix d = (0.5 * np.trace(Ud_pp.solve(dKd_pp)) - 0.5 * np.trace(U_Lambda.solve(dKd_pp)) + np.dot(nu, np.dot(dKd_pp, nu)) + np.trace(np.dot(dKd_pp, np.dot(V,Z)))) ## d = (0.5 * np.trace(utils.chol_solve(Ud_pp, dKd_pp)) ## - 0.5 * np.trace(utils.chol_solve(U_Lambda, dKd_pp)) ## + np.dot(nu, np.dot(dKd_pp, nu)) ## + np.trace(np.dot(dKd_pp, ## np.dot(V,Z)))) # Send the derivative message func(d) for (dxp, func) in Dxp: # Compute derivative w.r.t. covariance matrix d = np.nan # Send the derivative message func(d) else: ## Full exact (no pseudo approximations) try: U = utils.cholesky(K_xx) #U = utils.chol(K_xx) except linalg.LinAlgError: print('non positive definite, return -inf') return -np.inf z = U.solve(f0) #z = utils.chol_solve(U, f0) #print(K) L = utils.gaussian_logpdf(np.dot(f0, z), 0, 0, U.logdet(), ## utils.logdet_chol(U), np.size(self.f)) for (dmu, func) in Dmu: # Derivative w.r.t. mean vector d = -np.sum(z) # Send the derivative message func(d) for (dK, func) in DKd_xx: # Compute derivative w.r.t. covariance matrix # # TODO: trace+chol_solve should be handled better # for sparse matrices. Use sparse-inverse! d = 0.5 * (dK.dot(z).dot(z) - U.trace_solve_gradient(dK)) ## - np.trace(U.solve(dK))) ## d = 0.5 * (dK.dot(z).dot(z) ## - np.trace(utils.chol_solve(U, dK))) #print('derivate', d, dK) ## d = 0.5 * (np.dot(z, np.dot(dK, z)) ## - np.trace(utils.chol_solve(U, dK))) # # Send the derivative message func(d) for (dK, func) in DKs_xx: # Compute derivative w.r.t. covariance matrix d = 0.5 * (dK.dot(z).dot(z) - U.trace_solve_gradient(dK)) ## - np.trace(U.solve(dK))) ## d = 0.5 * (dK.dot(z).dot(z) ## - np.trace(utils.chol_solve(U, dK))) ## d = 0.5 * (np.dot(z, np.dot(dK, z)) ## - np.trace(utils.chol_solve(U, dK))) # Send the derivative message func(d) else: ## Log pdf for latent GP raise Exception('Not implemented yet') return L ## Let f1 be observed and f2 latent function values. # Compute #L = gaussian_logpdf(sum_product(np.outer(self.f,self.f) + self.Cov, # Compute def update(self): # Messages from parents m = self.parents[0].message_to_child() k = self.parents[1].message_to_child() if self.parents[2]: k_sparse = self.parents[2].message_to_child() else: k_sparse = None if self.parents[3]: pseudoinputs = self.parents[3].message_to_child()[0] else: pseudoinputs = None ## m = self.parents[0].message_to_child()[0] ## k = self.parents[1].message_to_child()[0] if self.observed: # Observations of this node self.u = gp_posterior_moment_function(m, k, self.x, self.f, k_sparse=k_sparse, pseudoinputs=pseudoinputs) else: x = np.array([]) y = np.array([]) # Messages from children for (child,index) in self.children: (msg, mask) = child.message_to_parent(index) # Ignoring masks and plates.. # m[0] is the inputs x = np.concatenate((x, msg[0]), axis=-2) # m[1] is the observations y = np.concatenate((y, msg[1])) # m[2] is the covariance matrix V = linalg.block_diag(V, msg[2]) self.u = gp_posterior_moment_function(m, k, x, y, covariance=V) self.x = x self.f = y # At least for now, simplify this GP node such that a GP is either # observed or latent. If it is observed, it doesn't take messages from # children, actually, it should not even have children! ## # Pseudo for GPFA: ## k1 = gp_cov_se(magnitude=theta1, lengthscale=theta2) ## k2 = gp_cov_periodic(magnitude=.., lengthscale=.., period=..) ## k3 = gp_cov_rq(magnitude=.., lengthscale=.., alpha=..) ## f = NodeGPSet(0, [k1,k2,k3]) # assumes block diagonality ## # f = NodeGPSet(0, [[k11,k12,k13],[k21,k22,k23],[k31,k32,k33]]) ## X = GaussianFromGP(f, [ [[t0,0],[t0,1],[t0,2]], [t1,0],[t1,1],[t1,2], ..]) ## ... ## # Construct a sum of GPs if interested only in the sum term ## k1 = gp_cov_se(magnitude=theta1, lengthscale=theta2) ## k2 = gp_cov_periodic(magnitude=.., lengthscale=.., period=..) ## k = gp_cov_sum(k1, k2) ## f = NodeGP(0, k) ## f.observe(x, y) ## f.update() ## (mp, kp) = f.get_parameters() ## # Construct a sum of GPs when interested also in the individual ## # GPs: ## k1 = gp_cov_se(magnitude=theta1, lengthscale=theta2) ## k2 = gp_cov_periodic(magnitude=.., lengthscale=.., period=..) ## k3 = gp_cov_delta(magnitude=theta3) ## f = NodeGPSum(0, [k1,k2,k3]) ## x = np.array([1,2,3,4,5,6,7,8,9,10]) ## y = np.sin(x[0]) + np.random.normal(0, 0.1, (10,)) ## # Observe the sum (index 0) ## f.observe((0,x), y) ## # Inference ## f.update() ## (mp, kp) = f.get_parameters() ## # Mean of the sum ## mp[0](...) ## # Mean of the individual terms ## mp[1](...) ## mp[2](...) ## mp[3](...) ## # Covariance of the sum ## kp[0][0](..., ...) ## # Other covariances ## kp[1][1](..., ...) ## kp[2][2](..., ...) ## kp[3][3](..., ...) ## kp[1][2](..., ...) ## kp[1][3](..., ...) ## kp[2][3](..., ...) bayespy-0.1/bayespy/inference/vmp/nodes/wishart.py0000664000175000017500000001355512174013072024133 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np import scipy.special as special from bayespy.utils import utils #from .variable import Variable from .expfamily import ExponentialFamily from .constant import Constant def WishartPrior(k): class _WishartPrior: """ Conjugate prior node for the degrees of freedom of the Wishart distribution. This isn't a distribution but can be used to compute the correct fixed moments, e.g., for Constant node.""" # Number of trailing axes for variable dimensions in # observations. The other axes correspond to plates. ndim_observations = 0 @staticmethod def compute_fixed_moments(n): """ Compute moments for fixed x. """ u0 = np.asanyarray(n) u1 = special.multigammaln(0.5*u0, k) return [u0, u1] @staticmethod def compute_dims_from_values(n): """ Compute the dimensions of phi or u. """ return ( (), () ) return _WishartPrior class Wishart(ExponentialFamily): ndims = (2, 0) ndims_parents = [None, (2, 0)] # Observations/values are 2-D matrices ndim_observations = 2 #parameter_distributions = (WishartPrior, Wishart) ## @staticmethod ## def compute_fixed_parameter_moments(*args): ## """ Compute the moments of the distribution parameters for ## fixed values.""" ## n = args[0] ## V = args[1] ## k = np.shape(V)[-1] ## u_n = WishartPrior(k).compute_fixed_moments(n) ## u_V = Wishart.compute_fixed_moments(V) ## return (u_n, u_V) @staticmethod def compute_fixed_moments(Lambda): """ Compute moments for fixed x. """ ldet = utils.m_chol_logdet(utils.m_chol(Lambda)) u = [Lambda, ldet] return u @staticmethod def _compute_cgf_from_parents(*u_parents): n = u_parents[0][0] gammaln_n = u_parents[0][1] V = u_parents[1][0] logdet_V = u_parents[1][1] k = np.shape(V)[-1] #k = self.dims[0][0] # TODO: Check whether this is correct: #g = 0.5*n*logdet_V - special.multigammaln(n/2, k) g = 0.5*n*logdet_V - 0.5*k*n*np.log(2) - gammaln_n #special.multigammaln(n/2, k) return g @staticmethod def _compute_phi_from_parents(*u_parents): return [-0.5 * u_parents[1][0], 0.5 * u_parents[0][0]] @staticmethod def _compute_moments_and_cgf(phi, mask=True): U = utils.m_chol(-phi[0]) k = np.shape(phi[0])[-1] #k = self.dims[0][0] logdet_phi0 = utils.m_chol_logdet(U) u0 = phi[1][...,np.newaxis,np.newaxis] * utils.m_chol_inv(U) u1 = -logdet_phi0 + utils.m_digamma(phi[1], k) u = [u0, u1] g = phi[1] * logdet_phi0 - special.multigammaln(phi[1], k) return (u, g) @staticmethod def compute_fixed_u_and_f(Lambda): """ Compute u(x) and f(x) for given x. """ k = np.shape(Lambda)[-1] ldet = utils.m_chol_logdet(utils.m_chol(Lambda)) u = [Lambda, ldet] f = -(k+1)/2 * ldet return (u, f) @staticmethod def message(index, u, u_parents): if index == 0: raise Exception("No analytic solution exists") elif index == 1: return (-0.5 * u[0], 0.5 * u_parents[0][0]) @staticmethod def compute_dims(*parents): """ Compute the dimensions of phi/u. """ # Has the same dimensionality as the second parent. return parents[1].dims @staticmethod def compute_dims_from_values(x): """ Compute the dimensions of phi and u. """ if np.ndim(x) < 2: raise ValueError("Values for Wishart distribution must be at least " "2-D arrays.") if np.shape(x)[-1] != np.shape(x)[-2]: raise ValueError("Values for Wishart distribution must be square " "matrices, thus the two last axes must have equal " "length.") d = np.shape(x)[-1] return ( (d,d), () ) # Wishart(n, inv(V)) def __init__(self, n, V, plates=(), **kwargs): # Check for constant V if np.isscalar(V) or isinstance(V, np.ndarray): V = Constant(Wishart)(V) k = V.dims[0][-1] # Check for constant n if np.isscalar(n) or isinstance(n, np.ndarray): n = Constant(WishartPrior(k))(n) self.parameter_distributions = (WishartPrior(k), Wishart) super().__init__(n, V, plates=plates, **kwargs) def show(self): print("%s ~ Wishart(n, A)" % self.name) print(" n =") print(2*self.phi[1]) print(" A =") print(0.5 * self.u[0] / self.phi[1][...,np.newaxis,np.newaxis]) bayespy-0.1/bayespy/inference/vmp/nodes/mixture.py0000664000175000017500000003741612174013072024151 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import itertools import numpy as np import scipy as sp import scipy.linalg.decomp_cholesky as decomp import scipy.linalg as linalg import scipy.special as special import scipy.spatial.distance as distance from bayespy.utils import utils from .expfamily import ExponentialFamily from .constant import Constant from .categorical import Categorical def Mixture(distribution, cluster_plate=-1): if cluster_plate >= 0: raise Exception("Give negative value for axis index cluster_plates") class _Mixture(ExponentialFamily): """ Mixtured distributions must implement: _compute_cgf_from_parents _compute_fixed_moments_and_f(x, mask) _compute_dims Sub-classes of ExponentialFamily should implement anyway: _compute_message_to_parent _compute_phi_from_parents _compute_moments_and_cgf(phi, mask) ExponentialFamily implements already: _compute_logpdf """ ndims = distribution.ndims @staticmethod def _compute_phi_from_parents(*u_parents): # Compute weighted average of the parameters # Cluster parameters Phi = distribution._compute_phi_from_parents(*(u_parents[1:])) # Contributions/weights/probabilities P = u_parents[0][0] phi = list() for ind in range(len(Phi)): # Compute element-wise product and then sum over K clusters. # Note that the dimensions aren't perfectly aligned because # the cluster dimension (K) may be arbitrary for phi, and phi # also has dimensions (Dd,..,D0) of the parameters. # Shape(phi) = [Nn,..,K,..,N0,Dd,..,D0] # Shape(p) = [Nn,..,N0,K] # Shape(result) = [Nn,..,N0,Dd,..,D0] # General broadcasting rules apply for Nn,..,N0, that is, # preceding dimensions may be missing or dimension may be # equal to one. Probably, shape(phi) has lots of missing # dimensions and/or dimensions that are one. if cluster_plate < 0: cluster_axis = cluster_plate - distribution.ndims[ind] #else: # cluster_axis = cluster_plate # Move cluster axis to the last: # Shape(phi) = [Nn,..,N0,Dd,..,D0,K] phi.append(utils.moveaxis(Phi[ind], cluster_axis, -1)) # Add axes to p: # Shape(p) = [Nn,..,N0,K,1,..,1] p = utils.add_trailing_axes(P, distribution.ndims[ind]) # Move cluster axis to the last: # Shape(p) = [Nn,..,N0,1,..,1,K] p = utils.moveaxis(p, -(distribution.ndims[ind]+1), -1) #print('Mixture.compute_phi, p:', np.sum(p, axis=-1)) #print('mixture.compute_phi shapes:') #print(np.shape(p)) #print(np.shape(phi[ind])) # Now the shapes broadcast perfectly and we can sum # p*phi over the last axis: # Shape(result) = [Nn,..,N0,Dd,..,D0] phi[ind] = utils.sum_product(p, phi[ind], axes_to_sum=-1) return phi @staticmethod def _compute_cgf_from_parents(*u_parents): # Compute weighted average of g over the clusters. # Shape(g) = [Nn,..,K,..,N0] # Shape(p) = [Nn,..,N0,K] # Shape(result) = [Nn,..,N0] # Compute g for clusters: # Shape(g) = [Nn,..,K,..,N0] g = distribution._compute_cgf_from_parents(*(u_parents[1:])) # Move cluster axis to last: # Shape(g) = [Nn,..,N0,K] g = utils.moveaxis(g, cluster_plate, -1) # Cluster assignments/contributions/probabilities/weights: # Shape(p) = [Nn,..,N0,K] p = u_parents[0][0] # Weighted average of g over the clusters. As p and g are # properly aligned, you can just sum p*g over the last # axis and utilize broadcasting: # Shape(result) = [Nn,..,N0] #print('mixture.compute_g_from_parents p and g:', np.shape(p), np.shape(g)) g = utils.sum_product(p, g, axes_to_sum=-1) #print('mixture.compute_g_from_parents g:', np.sum(g), np.shape(g)) return g @staticmethod def _compute_moments_and_cgf(phi, mask=True): return distribution._compute_moments_and_cgf(phi, mask=mask) @staticmethod def _compute_fixed_moments_and_f(x, mask=True): """ Compute u(x) and f(x) for given x. """ return distribution._compute_fixed_moments_and_f(x, mask=True) @staticmethod def _compute_message_to_parent(parent, index, u, *u_parents): """ . """ #print('Mixture.compute_message:') if index == 0: # Shape(phi) = [Nn,..,K,..,N0,Dd,..,D0] # Shape(L) = [Nn,..,K,..,N0] # Shape(u) = [Nn,..,N0,Dd,..,D0] # Shape(result) = [Nn,..,N0,K] # Compute g: # Shape(g) = [Nn,..,K,..,N0] g = distribution._compute_cgf_from_parents(*(u_parents[1:])) # Reshape(g): # Shape(g) = [Nn,..,N0,K] g = utils.moveaxis(g, cluster_plate, -1) # Compute phi: # Shape(phi) = [Nn,..,K,..,N0,Dd,..,D0] phi = distribution._compute_phi_from_parents(*(u_parents[1:])) # Reshape phi: # Shape(phi) = [Nn,..,N0,K,Dd,..,D0] for ind in range(len(phi)): phi[ind] = utils.moveaxis(phi[ind], cluster_plate-distribution.ndims[ind], -1-distribution.ndims[ind]) # Reshape u: # Shape(u) = [Nn,..,N0,1,Dd,..,D0] u_self = list() for ind in range(len(u)): u_self.append(np.expand_dims(u[ind], axis=(-1-distribution.ndims[ind]))) # Compute logpdf: # Shape(L) = [Nn,..,N0,K] L = distribution._compute_logpdf(u_self, phi, g, 0) # Sum over other than the cluster dimensions? No! # Hmm.. I think the message passing method will do # that automatically ## print(np.shape(phi[0])) ## print(np.shape(u_self[0])) ## print(np.shape(g)) ## print(np.shape(L)) return [L] elif index >= 1: # Parent index for the distribution used for the # mixture. index = index - 1 # Reshape u: # Shape(u) = [Nn,..1,..,N0,Dd,..,D0] u_self = list() for ind in range(len(u)): if cluster_plate < 0: cluster_axis = cluster_plate - distribution.ndims[ind] else: cluster_axis = cluster_plate u_self.append(np.expand_dims(u[ind], axis=cluster_axis)) # Message from the mixed distribution m = distribution._compute_message_to_parent(index, u_self, *(u_parents[1:])) # Weigh the messages with the responsibilities for i in range(len(m)): # Shape(m) = [Nn,..,K,..,N0,Dd,..,D0] # Shape(p) = [Nn,..,N0,K] # Shape(result) = [Nn,..,K,..,N0,Dd,..,D0] # Number of axes for the variable dimensions for # the parent message. D = distribution.ndims_parents[index][i] # Responsibilities for clusters are the first # parent's first moment: # Shape(p) = [Nn,..,N0,K] p = u_parents[0][0] # Move the cluster axis to the proper place: # Shape(p) = [Nn,..,K,..,N0] p = utils.moveaxis(p, -1, cluster_plate) # Add axes for variable dimensions to the contributions # Shape(p) = [Nn,..,K,..,N0,1,..,1] p = utils.add_trailing_axes(p, D) if cluster_plate < 0: # Add the variable dimensions cluster_axis = cluster_plate - D # Add axis for clusters: # Shape(m) = [Nn,..,1,..,N0,Dd,..,D0] #m[i] = np.expand_dims(m[i], axis=cluster_axis) # # TODO: You could do summing here already so that # you wouldn't compute huge matrices as # intermediate result. Use einsum. ## print(np.shape(m[i])) ## print(np.shape(p)) # Compute the message contributions for each # cluster: # Shape(result) = [Nn,..,K,..,N0,Dd,..,D0] m[i] = m[i] * p #print(np.shape(m[i])) return m @staticmethod def compute_dims(*parents): """ Compute the dimensions of phi and u. """ return distribution.compute_dims(*parents[1:]) def __init__(self, z, *args, **kwargs): # Check for constant mu if np.isscalar(z) or isinstance(z, np.ndarray): z = ConstantCategorical(z) # Construct super().__init__(z, *args, **kwargs) #_compute_mask_to_parent(index, mask) #_plates_to_parent(self, index) #_plates_from_parent(self, index) def _plates_to_parent(self, index): if index == 0: return self.plates else: if cluster_plate < 0: plates = list(self.plates) if cluster_plate < 0: k = len(self.plates) + cluster_plate + 1 else: k = cluster_plate plates.insert(k, self.parents[0].dims[0][0]) plates = tuple(plates) #print('plates_to_parent', cluster_plate, plates) ## plates = (self.plates[:cluster_plate] + ## self.parents[0].dims[0] + ## self.plates[cluster_plate:]) return plates def _plates_from_parent(self, index): if index == 0: return self.parents[index].plates else: plates = list(self.parents[index].plates) plates.pop(cluster_plate) return tuple(plates) def integrated_logpdf_from_parents(self, x, index): """ Approximates the posterior predictive pdf \int p(x|parents) q(parents) dparents in log-scale as \int q(parents_i) exp( \int q(parents_\i) \log p(x|parents) dparents_\i ) dparents_i.""" if index == 0: # Integrate out the cluster assignments # First, integrate the cluster parameters in log-scale # compute_logpdf(cls, u, phi, g, f): # Shape(x) = [M1,..,Mm,N1,..,Nn,D1,..,Dd] # Add the cluster axis to x: # Shape(x) = [M1,..,Mm,N1,..,1,..,Nn,D1,..,Dd] cluster_axis = cluster_plate - distribution.ndim_observations x = np.expand_dims(x, axis=cluster_axis) u_parents = self._message_from_parents() #u_parents = self.moments_from_parents() # Shape(u) = [M1,..,Mm,N1,..,1,..,Nn,D1,..,Dd] # Shape(f) = [M1,..,Mm,N1,..,1,..,Nn] (u, f) = distribution._compute_fixed_moments_and_f(x) # Shape(phi) = [N1,..,K,..,Nn,D1,..,Dd] phi = distribution._compute_phi_from_parents(*(u_parents[1:])) # Shape(g) = [N1,..,K,..,Nn] g = distribution._compute_cgf_from_parents(*(u_parents[1:])) # Shape(lpdf) = [M1,..,Mm,N1,..,K,..,Nn] lpdf = distribution._compute_logpdf(u, phi, g, f) # From logpdf to pdf, but avoid over/underflow lpdf_max = np.max(lpdf, axis=cluster_plate, keepdims=True) pdf = np.exp(lpdf-lpdf_max) # Move cluster axis to be the last: # Shape(pdf) = [M1,..,Mm,N1,..,Nn,K] pdf = utils.moveaxis(pdf, cluster_plate, -1) #print('integrated_logpdf', pdf) # Cluster assignments/probabilities/weights # Shape(p) = [N1,..,Nn,K] p = u_parents[0][0] #self.parents[0].show() #print('integrated_logpdf, p:', p) # Weighted average. TODO/FIXME: Use einsum! # Shape(pdf) = [M1,..,Mm,N1,..,Nn] pdf = np.sum(pdf * p, axis=cluster_plate) #print('integrated_logpdf', pdf) # Back to log-scale (add the overflow fix!) lpdf_max = np.squeeze(lpdf_max, axis=cluster_plate) lpdf = np.log(pdf) + lpdf_max return lpdf raise NotImplementedError() return _Mixture ## def show(self): ## p = self.u[0] #np.exp(self.phi[0]) ## #p /= np.sum(p, axis=-1, keepdims=True) ## print("Categorical(p)") ## print(" p = ") ## print(p) ## def observe(self, x): ## # TODO: You could check that x has proper dimensions ## x = np.array(x, dtype=np.int) ## # Initial array of zeros ## d = self.dims[0][0] ## self.u[0] = np.zeros(np.shape(x)+(d,)) ## # Compute indices ## x += d*np.arange(np.size(x),dtype=int).reshape(np.shape(x)) ## x = x[...,np.newaxis] ## # Set 1 to elements corresponding to the observations ## np.put(self.u[0], x, 1) ## self.show() ## self.fix_u_and_f(self.u, 0) bayespy-0.1/bayespy/inference/vmp/nodes/constant.py0000664000175000017500000000514312174013072024275 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np from .node import Node class ConstantNumeric(Node): def __init__(self, x, ndim, **kwargs): # Compute moments self.u = [np.asarray(x)] # Dimensions and plates of the moments ind_dim = np.ndim(x) - ndim dims = np.shape(x)[ind_dim:] plates = np.shape(x)[:ind_dim] # Parent constructor super().__init__(dims=dims, plates=plates, **kwargs) def get_moments(self): return self.u def Constant(distribution): class _Constant(Node): def __init__(self, x, **kwargs): x = np.asanyarray(x) # Compute moments self.u = distribution.compute_fixed_moments(x) # Dimensions of the moments dims = distribution.compute_dims_from_values(x) # Number of plate axes plates_ndim = np.ndim(x) - distribution.ndim_observations plates = np.shape(x)[:plates_ndim] # Parent constructor super().__init__(dims=dims, plates=plates, **kwargs) @staticmethod def compute_fixed_moments(x): """ Compute u(x) for given x. """ return distribution.compute_fixed_moments(x) @staticmethod def compute_fixed_u_and_f(x): """ Compute u(x) and f(x) for given x. """ raise Exception("I think constants should NOT need this function?") return distribution.compute_fixed_u_and_f(x) def get_moments(self): return self.u return _Constant bayespy-0.1/bayespy/inference/vmp/nodes/dot.py0000664000175000017500000002704012174013072023232 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np from bayespy.utils import utils from .node import Node from .deterministic import Deterministic from .constant import Constant from .gaussian import Gaussian def _message_sum_multiply(plates_parent, dims_parent, *arrays): """ Compute message to parent and sum over plates. """ # The shape of the full message shapes = [np.shape(array) for array in arrays] shape_full = utils.broadcasted_shape(*shapes) # Find axes that should be summed shape_parent = plates_parent + dims_parent sum_axes = utils.axes_to_collapse(shape_full, shape_parent) # Compute the multiplier for cancelling the # plate-multiplier. Because we are summing over the # dimensions already in this function (for efficiency), we # need to cancel the effect of the plate-multiplier # applied in the message_to_parent function. r = 1 for j in sum_axes: if j >= 0 and j < len(plates_parent): r *= shape_full[j] elif j < 0 and j < -len(dims_parent): r *= shape_full[j] # Compute the sum-product m = utils.sum_multiply(*arrays, axis=sum_axes, sumaxis=True, keepdims=True) / r # Remove extra axes m = utils.squeeze_to_dim(m, len(shape_parent)) return m def _mask_sum(plates_parent, mask): # Compute the mask axes = utils.axes_to_collapse(np.shape(mask), plates_parent) mask = np.any(mask, axis=axes, keepdims=True) mask = utils.squeeze_to_dim(mask, len(plates_parent)) return mask class Dot(Deterministic): """ A deterministic node for computing vector product of two Gaussians. See also: --------- MatrixDot """ # This node satisfies Normal-protocol to children and # Gaussian-protocol to parents # This node is deterministic, just handles message processing. # y(i0,i1,...,in) = sum_d prod_k xk(i0,i1,...,in,d) def __init__(self, *parents, **kwargs): # For now, do not use plates other than from the parents, # although it would be possible (it would mean that you'd # create several "datasets" with identical "PCA # distribution"). Maybe it is better to create such plates in # children nodes? #plates = [] parents = list(parents) # Convert constant arrays to constant nodes for n in range(len(parents)): if utils.is_numeric(parents[n]): parents[n] = Constant(Gaussian)(parents[n]) parent_dims = [parent.dims for parent in parents] parent_plates = [parent.plates for parent in parents] # Check that the parents have equal dimensions for n in range(len(parent_dims)-1): if parent_dims[n] != parent_dims[n+1]: raise ValueError("Dimensions of the Gaussians do not " "match: %s" % (parent_dims,)) ## try: ## plates = utils.broadcasted_shape(*parent_plates) ## except ValueError: ## raise ValueError("The plates of the parents are " ## "incompatible: %s" % (parent_plates,)) super().__init__(*parents, #plates=tuple(plates), dims=((),()), **kwargs) def _compute_moments(self, *u_parents): if len(self.parents) == 0: return [0, 0] str1 = '...i' + ',...i' * (len(self.parents)-1) str2 = '...ij' + ',...ij' * (len(self.parents)-1) u1 = list() u2 = list() for u in u_parents: u1.append(u[0]) u2.append(u[1]) x = [np.einsum(str1, *u1), np.einsum(str2, *u2)] return x def get_parameters(self): # Compute mean and variance u = self.get_moments() u[1] -= u[0]**2 return u def _compute_message_and_mask_to_parent(self, index, m, *u_parents): # Normally we don't need to care about masks when computing the # message. However, in this node we want to avoid computing huge message # arrays so we sum some axis already here. Thus, we need to apply the # mask mask = self.mask parent = self.parents[index] # Compute both messages for i in range(2): # Add extra axes to the message from children m[i] = utils.add_trailing_axes(m[i], i+1) # List of elements to multiply together A = [m[i]] for k in range(len(u_parents)): if k != index: A.append(u_parents[k][i]) # Compute the sum over some axes already here in order to avoid huge # message matrices. m[i] = _message_sum_multiply(parent.plates, parent.dims[i], *A) # Compute the mask s = utils.axes_to_collapse(np.shape(mask), parent.plates) mask = np.any(mask, axis=s, keepdims=True) mask = utils.squeeze_to_dim(mask, len(parent.plates)) return (m, mask) class MatrixDot(Deterministic): """ A deterministic node for computing matrix-vector product of Gaussians. The matrix is given in vector form. Therefore, the first parent should have dimensionality that is a multiple of the dimensionality of the second parent. The dimensionality of the result is that multiplier: A : (M*N)-dimensional X : N-dimensional result : M-dimensional `Dot` computes vector-vector product of Gaussians, thus the result is scalar Normal. This node assumes that you have a full joint Gaussian distribution for all elements of a matrix, thus the result is vector Gaussian. See also -------- Dot """ def __init__(self, A, X, **kwargs): # Convert arrays to constant nodes if utils.is_numeric(A): A = Constant(Gaussian)(A) if utils.is_numeric(X): X = Constant(Gaussian)(X) MN = A.dims[0][0] N = X.dims[0][0] # Check parents if A.dims != ( (MN,), (MN, MN) ): raise ValueError("Invalid dimensionality of the first parent.") if X.dims != ( (N,), (N, N) ): raise ValueError("Invalid dimensionality of the second parent.") if (MN % N) != 0: raise ValueError("The dimensionality of the first parent should be " "a multiple of the second parent.") # Dimensionality of the output M = int(MN / N) dims = ( (M,), (M,M) ) Node.__init__(self, A, X, dims=dims, **kwargs) def _compute_moments(self, u_A, u_X): """ Get the moments of the Gaussian output. """ # Helpful variables to clarify the code A = u_A[0] AA = u_A[1] X = u_X[0] XX = u_X[1] (A, AA) = self._reshape_to_matrix(A, AA) #(A, AA) = self._reshape_to_matrix(A, AA, X.shape[-1]) # Compute matrix-vector products Y = np.einsum('...ij,...j->...i', A, X) YY = np.einsum('...jlik,...lk->...ij', AA, XX) return [Y, YY] def _compute_message_and_mask_to_parent(self, index, m, u_A, u_X): """ Compute the message to a parent node. """ mask = self.mask parent = self.parents[index] # Message from children VY = m[0] # (..., D) V = m[1] # (..., D, D) D = np.shape(V)[-1] if index == 0: # Message to A X = u_X[0] # (..., K) XX = u_X[1] # (..., K, K) K = np.shape(X)[-1] # m0 : (..., D,K) -> (...,D*K) # m1 : (..., D,K,D,K) -> (...,D*K,D*K) # In order to avoid huge memory consumption, compute the sum-product # already in this method (instead of Node._message_to_parent). That # is, some axes are summed over already here. m0 = _message_sum_multiply(parent.plates, (D,K), VY[...,:,np.newaxis], X[...,np.newaxis,:]) m1 = _message_sum_multiply(parent.plates, (D,K,D,K), V[...,:,np.newaxis,:,np.newaxis], XX[...,np.newaxis,:,np.newaxis,:]) # Do the same for the mask mask = _mask_sum(parent.plates, mask) (m0, m1) = self._reshape_to_vector(m0, m1) elif index == 1: # Message to X (A, AA) = self._reshape_to_matrix(u_A[0], u_A[1]) #K = parent.dims[0][0] K = np.shape(A)[-1] # A : (...,D,K) # AA : (...,D,K,D,K) # In order to avoid huge memory consumption, compute the sum-product # already in this method (instead of Node._message_to_parent). That # is, some axes are summed over already here. m0 = _message_sum_multiply(parent.plates, (1,K), A, VY[...,:,np.newaxis]) m0 = m0[...,0,:] m1 = _message_sum_multiply(parent.plates, (1,K,1,K), AA, V[...,:,np.newaxis,:,np.newaxis]) m1 = m1[...,0,:,0,:] # Do the same for the mask mask = _mask_sum(parent.plates, mask) #m0 = np.einsum('...ij,...i->...j', A, VY) #m1 = np.einsum('...kilj,...kl->...ij', AA, V) return ([m0,m1], mask) def _reshape_to_matrix(self, A, AA): """ Reshape vector form moments to matrix form. A : (...,M*N) AA : (...,M*N,M*N) Reshape to: A : (...,M,N) AA : (...,M,N,M,N) """ # The dimensionalities MN = self.parents[0].dims[0][0] N = self.parents[1].dims[0][0] M = int(MN / N) # Reshape vector A to a matrix sh_A = np.shape(A)[:-1] + (M,N) sh_AA = np.shape(A)[:-1] + (M,N,M,N) A = A.reshape(sh_A) AA = AA.reshape(sh_AA) return (A, AA) def _reshape_to_vector(self, A, AA): """ Reshape matrix form moments to vector form. A : (...,M,N) AA : (...,M,N,M,N) Reshape to: A : (...,M*N) AA : (...,M*N,M*N) """ MN = A.shape[-2] * A.shape[-1] sh_A = A.shape[:-2] + (MN,) sh_AA = AA.shape[:-4] + (MN,MN) A = A.reshape(sh_A) AA = AA.reshape(sh_AA) return (A, AA) bayespy-0.1/bayespy/inference/vmp/nodes/normal.py0000664000175000017500000001065612174023362023744 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np from .expfamily import ExponentialFamily from .constant import Constant from .gamma import Gamma class Normal(ExponentialFamily): ndims = (0, 0) ndims_parents = [(0, 0), (0, 0)] # Observations are scalars (0-D): ndim_observations = 0 @staticmethod def compute_fixed_moments(x): """ Compute moments u(x) for given x. """ return [x, x**2] @staticmethod def _compute_phi_from_parents(*u_parents): phi = [u_parents[1][0] * u_parents[0][0], -u_parents[1][0] / 2] return phi @staticmethod def _compute_cgf_from_parents(*u_parents): mu = u_parents[0][0] mumu = u_parents[0][1] tau = u_parents[1][0] log_tau = u_parents[1][1] g = -0.5 * mumu*tau + 0.5 * log_tau return g @staticmethod def _compute_moments_and_cgf(phi, mask=True): u0 = -phi[0] / (2*phi[1]) u1 = u0**2 - 1 / (2*phi[1]) u = [u0, u1] g = (-0.5 * u[0] * phi[0] + 0.5 * np.log(-2*phi[1])) return (u, g) @staticmethod def _compute_fixed_moments_and_f(x, mask=True): """ Compute u(x) and f(x) for given x. """ u = [x, x**2] f = -np.log(2*np.pi)/2 return (u, f) @staticmethod def _compute_message_to_parent(parent, index, u, *u_parents): """ . """ if index == 0: return [u_parents[1][0] * u[0], -0.5 * u_parents[1][0]] elif index == 1: return [-0.5 * (u[1] - 2*u[0]*u_parents[0][0] + u_parents[0][1]), 0.5] @staticmethod def compute_dims(*parents): """ Compute the dimensions of phi/u. """ # Both moments are scalars, thus, shapes are () return ( (), () ) @staticmethod def compute_dims_from_values(x): """ Compute the dimensions of phi and u. """ return ( (), () ) # Normal(mu, 1/tau) def __init__(self, mu, tau, **kwargs): # Check for constant mu if np.isscalar(mu) or isinstance(mu, np.ndarray): mu = Constant(Normal)(mu) # Check for constant tau if np.isscalar(tau) or isinstance(tau, np.ndarray): tau = Constant(Gamma)(tau) # Construct super().__init__(mu, tau, **kwargs) def show(self, parameters=True, mean=True, mode=True, median=True): mu = self.u[0] tau = -2 * self.phi[1] #s2 = self.u[1] - mu**2 print("%s ~ Normal(mu, tau)" % self.name) print(" mu =", mu) #print(mu) print(" tau =", tau) #print(tau) #print("Normal(" + str(mu) + ", " + str(s2) + ")") def plot(self): pass def predict(self): """ Compute posterior predictive distribution. Integrate out the mean parameter analytically by forming Q(X|mu) and then integrating out mu. This gives more accurate posterior predictive distribution. Observations are ignored. The predictive distribution is returned as a tuple containing the predictive mean and variance. """ m = self._message_from_children() tau = self.parents[1].get_moments()[0] (mu, mu2) = self.parents[0].get_moments() varmu = mu2 - mu**2 z1 = 1 / (tau + m[1]) z2 = tau*mu + m[0] return (z1 * z2, z1 + z1**2 * tau**2 * varmu) bayespy-0.1/bayespy/inference/vmp/nodes/stochastic.py0000664000175000017500000001646212174013072024616 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np from bayespy.utils import utils from .node import Node class Stochastic(Node): """ Base class for nodes that are stochastic. u observed Sub-classes must implement: _compute_message_to_parent(parent, index, u_self, *u_parents) _update_distribution_and_lowerbound(self, m, *u) lowerbound(self) _compute_dims If you want to be able to observe the variable: _compute_fixed_moments_and_f Sub-classes may need to re-implement: 1. If they manipulate plates: _compute_mask_to_parent(index, mask) _plates_to_parent(self, index) _plates_from_parent(self, index) """ def __init__(self, *args, initialize=True, **kwargs): super().__init__(*args, dims=self.compute_dims(*args), **kwargs) # Initialize moment array axes = len(self.plates)*(1,) self.u = [utils.nans(axes+dim) for dim in self.dims] # Not observed self.observed = False if initialize: self.initialize_from_prior() # TODO: Write the initialization method. def get_moments(self): return self.u ## @staticmethod ## def _compute_message_to_parent(index, u_self, *u_parents): ## # Sub-classes should implement this ## raise NotImplementedError() def _get_message_and_mask_to_parent(self, index): u_parents = self._message_from_parents(exclude=index) m = self._compute_message_to_parent(self.parents[index], index, self.u, *u_parents) mask = self._compute_mask_to_parent(index, self.mask) return (m, mask) ## def get_mask(self): ## return self.mask ## def _get_message_mask(self): ## return self.mask def _set_mask(self, mask): self.mask = np.logical_or(mask, self.observed) def _set_moments(self, u, mask=True): # Store the computed moments u but do not change moments for # observations, i.e., utilize the mask. for ind in range(len(u)): # Add axes to the mask for the variable dimensions (mask # contains only axes for the plates). u_mask = utils.add_trailing_axes(mask, self.ndims[ind]) # Enlarge self.u[ind] as necessary so that it can store the # broadcasted result. sh = utils.broadcasted_shape_from_arrays(self.u[ind], u[ind], u_mask) self.u[ind] = utils.repeat_to_shape(self.u[ind], sh) # TODO/FIXME/BUG: The mask of observations is not used, observations # may be overwritten!!! ??? # Hah, this function is used to set the observations! The caller # should be careful what mask he uses! If you want to set only # latent variables, then use such a mask. # Use mask to update only unobserved plates and keep the # observed as before np.copyto(self.u[ind], u[ind], where=u_mask) # Make sure u has the correct number of dimensions: # TODO/FIXME: Maybe it would be good to also check that u has a # shape that is a sub-shape of get_shape. shape = self.get_shape(ind) ndim = len(shape) ndim_u = np.ndim(self.u[ind]) if ndim > ndim_u: self.u[ind] = utils.add_leading_axes(u[ind], ndim - ndim_u) elif ndim < ndim_u: raise Exception("The size of the variable's moments array " "is larger than it should be based on the " "plates and dimension information. Check " "that you have provided plates properly.") def update(self): if not np.all(self.observed): u_parents = self._message_from_parents() m_children = self._message_from_children() self._update_distribution_and_lowerbound(m_children, *u_parents) def observe(self, x, mask=True): """ Fix moments, compute f and propagate mask. """ # Compute fixed moments (u, f) = self._compute_fixed_moments_and_f(x, mask=mask) # Check the dimensionality of the observations for (i,v) in enumerate(u): # This is what the dimensionality "should" be s = self.plates + self.dims[i] t = np.shape(v) if s != t: msg = "Dimensionality of the observations incorrect." msg += "\nShape of input: " + str(t) msg += "\nExpected shape: " + str(s) msg += "\nCheck plates." raise Exception(msg) # Set the moments self._set_moments(u, mask=mask) # TODO/FIXME: Use the mask? self.f = f # Observed nodes should not be ignored self.observed = mask self._update_mask() def unobserve(self): # Update mask self.observed = False self._update_mask() def lowerbound(self): # Sub-class should implement this raise NotImplementedError() ## @staticmethod ## def _compute_fixed_moments_and_f(x, mask=True): ## # Sub-classes should implement this ## raise NotImplementedError() def _update_distribution_and_lowerbound(self, m_children, *u_parents): # Sub-classes should implement this raise NotImplementedError() def save(self, group): """ Save the state of the node into a HDF5 file. group can be the root """ ## if name is None: ## name = self.name ## subgroup = group.create_group(name) for i in range(len(self.u)): utils.write_to_hdf5(group, self.u[i], 'u%d' % i) utils.write_to_hdf5(group, self.observed, 'observed') def load(self, group): """ Load the state of the node from a HDF5 file. """ # TODO/FIXME: Check that the shapes are correct! for i in range(len(self.u)): ui = group['u%d' % i][...] self.u[i] = ui old_observed = self.observed self.observed = group['observed'][...] # Update masks if necessary if np.any(old_observed != self.observed): self._update_mask() bayespy-0.1/bayespy/inference/vmp/nodes/gamma.py0000664000175000017500000001611412174013072023526 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np import scipy.special as special from .node import Node from .deterministic import Deterministic from .expfamily import ExponentialFamily from .constant import Constant from bayespy.utils import utils def diagonal(alpha): """ Create a diagonal Wishart node from a Gamma node. """ return _GammaToDiagonalWishart(alpha, name=alpha.name + " as Wishart") class GammaPrior(Node): """ Conjugate prior node for the shape of the gamma distribution. This isn't a distribution but can be used to compute the correct fixed moments, e.g., for Constant node.""" # Number of trailing axes for variable dimensions in # observations. The other axes correspond to plates. ndim_observations = 0 @staticmethod def compute_fixed_moments(a): """ Compute moments for fixed x. """ u0 = a u1 = special.gammaln(a) return [u0, u1] @staticmethod def compute_dims_from_values(a): """ Compute the dimensions of phi or u. """ return [(), ()] class Gamma(ExponentialFamily): ndims = (0, 0) # Observations/values are scalars (0-dimensional) ndim_observations = 0 def __init__(self, a, b, **kwargs): self.parameter_distributions = (GammaPrior, Gamma) # TODO: USE asarray(a) # Check for constant a if utils.is_numeric(a): a = Constant(GammaPrior)(a) # Check for constant b if utils.is_numeric(b): b = Constant(Gamma)(b) #b = NodeConstant([b, np.log(b)], plates=np.shape(b), dims=[(),()]) # Construct super().__init__(a, b, **kwargs) @staticmethod def compute_fixed_moments(x): """ Compute moments for fixed x. """ u0 = x # TODO/FIXME: Shouldn't this be just log? u1 = special.gammaln(x) return [u0, u1] @staticmethod def _compute_phi_from_parents(*u_parents): return [-u_parents[1][0], 1*u_parents[0][0]] @staticmethod def _compute_cgf_from_parents(*u_parents): a = u_parents[0][0] gammaln_a = u_parents[0][1] #special.gammaln(a) b = u_parents[1][0] log_b = u_parents[1][1] g = a * log_b - gammaln_a return g @staticmethod def _compute_moments_and_cgf(phi, mask=True): log_b = np.log(-phi[0]) u0 = phi[1] / (-phi[0]) u1 = special.digamma(phi[1]) - log_b u = [u0, u1] g = phi[1] * log_b - special.gammaln(phi[1]) return (u, g) @staticmethod def _compute_fixed_moments_and_f(x, mask=True): """ Compute u(x) and f(x) for given x. """ u = [x, np.log(x)] f = 0 return (u, f) @staticmethod def _compute_message_to_parent(parent, index, u, *u_parents): """ . """ if index == 0: raise Exception("No analytic solution exists") elif index == 1: return [-u[0], u_parents[0][0]] @staticmethod def compute_dims(*parents): """ Compute the dimensions of phi/u. """ return ( (), () ) @staticmethod def compute_dims_from_values(x): """ Compute the dimensions of phi and u. """ return ( (), () ) def get_shape_of_value(self): # Dimensionality of a realization return () def show(self): a = self.phi[1] b = -self.phi[0] print("%s ~ Gamma(a, b)" % self.name) print(" a =", a) print(" b =", b) def as_diagonal_wishart(self): return _GammaToDiagonalWishart(self, name=self.name + " as Wishart") class _GammaToDiagonalWishart(Deterministic): """ Transform a set of gamma scalars into a diagonal Wishart matrix. The last plate is used as the diagonal dimension. """ ndims = (2, 0) ndims_parents = [None, (2, 0)] # Observations/values are 2-D matrices ndim_observations = 2 def __init__(self, alpha, **kwargs): # Check for constant if utils.is_numeric(alpha): alpha = Constant(Gamma)(alpha) # Remove the last plate... #plates = alpha.plates[:-1] # ... and use it as the dimensionality of the Wishart # distribution if len(alpha.plates) == 0: raise Exception("Gamma variable needs to have plates in " "order to be used as a diagonal Wishart.") D = alpha.plates[-1] dims = ( (D,D), () ) # Construct the node super().__init__(alpha, #plates=plates, dims=dims, **kwargs) def _plates_to_parent(self, index): D = self.dims[0][0] return self.plates + (D,) def _plates_from_parent(self, index): return self.parents[index].plates[:-1] @staticmethod def _compute_mask_to_parent(index, mask): return mask[..., np.newaxis] def get_moments(self): u = self.parents[0].get_moments() # Form a diagonal matrix from the gamma variables return [np.identity(self.dims[0][0]) * u[0][...,np.newaxis], np.sum(u[1], axis=(-1))] @staticmethod def _compute_message_to_parent(index, m_children, *u_parents): # Take the diagonal m0 = np.einsum('...ii->...i', m_children[0]) m1 = np.reshape(m_children[1], np.shape(m_children[1]) + (1,)) return [m0, m1] ## def _compute_message_and_mask_to_parent(self, index, m_children, *u_parents): ## m = self._message_from_children() ## #(m, mask) = self.message_from_children() ## # Take the diagonal ## m[0] = np.einsum('...ii->...i', m[0]) ## # TODO/FIXME: I think m[1] is wrong.. ## m[1] = np.reshape(m[1], np.shape(m[1]) + (1,)) ## # m[1] is ok ## mask = self._compute_mask_to_parent(index, self.mask) ## #mask = mask[...,np.newaxis] ## return (m, mask) ## #return m bayespy-0.1/bayespy/inference/vmp/nodes/dirichlet.py0000664000175000017500000001010012174013072024400 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np import scipy.special as special #from .variable import Variable from .expfamily import ExponentialFamily from .constant import Constant class Dirichlet(ExponentialFamily): class ConjugatePrior: """ Conjugate prior for Dirichlet distribution. This isn't a distribution but can be used to compute the correct fixed moments, e.g., for Constant node.""" # Number of trailing axes for variable dimensions in # observations. The other axes correspond to plates. ndim_observations = 1 @staticmethod def compute_fixed_moments(alpha): """ Compute moments for fixed x. """ gammaln_sum = special.gammaln(np.sum(alpha, axis=-1)) sum_gammaln = np.sum(special.gammaln(alpha), axis=-1) z = gammaln_sum - sum_gammaln return [alpha, z] @staticmethod def compute_dims_from_values(alpha): """ Compute the dimensions of phi and u. """ d = np.shape(alpha)[-1] return [(d,), ()] ndims = (1,) @staticmethod def _compute_phi_from_parents(*u_parents): return [u_parents[0][0]] #return [u_parents[0][0].copy()] @staticmethod def _compute_cgf_from_parents(*u_parents): return u_parents[0][1] @staticmethod def _compute_moments_and_cgf(phi, mask=True): sum_gammaln = np.sum(special.gammaln(phi[0]), axis=-1) gammaln_sum = special.gammaln(np.sum(phi[0], axis=-1)) psi_sum = special.psi(np.sum(phi[0], axis=-1, keepdims=True)) # Moments u0 = special.psi(phi[0]) - psi_sum u = [u0] # G g = gammaln_sum - sum_gammaln return (u, g) @staticmethod def compute_message(index, u, u_parents): """ . """ if index == 0: return [u[0], 1] @staticmethod def compute_dims(*parents): """ Compute the dimensions of phi/u. """ # Has the same dimensionality as the parent for its first # moment. return parents[0].dims[:1] def __init__(self, alpha, plates=(), **kwargs): # Check for constant alpha if np.isscalar(alpha) or isinstance(alpha, np.ndarray): alpha = Constant(self.ConjugatePrior)(alpha) ## gammaln_sum = special.gammaln(np.sum(alpha, axis=-1)) ## sum_gammaln = np.sum(special.gammaln(alpha), axis=-1) ## z = gammaln_sum - sum_gammaln ## d = np.shape(alpha)[-1] ## alpha = NodeConstant([alpha, z], ## plates=np.shape(alpha)[:-1], ## dims=((d,), ())) # Construct super().__init__(alpha, plates=plates, **kwargs) def random(self): raise NotImplementedError() def show(self): alpha = self.phi[0] print("%s ~ Dirichlet(alpha)" % self.name) print(" alpha = ") print(alpha) bayespy-0.1/bayespy/inference/vmp/nodes/gaussian.py0000664000175000017500000003205412174013072024257 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # Bayespy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np from bayespy import utils from bayespy.utils.linalg import dot, mvdot from .expfamily import ExponentialFamily from .constant import Constant from .wishart import Wishart class Gaussian(ExponentialFamily): r""" VMP node for Gaussian variable. The node represents a :math:`D`-dimensional vector from the Gaussian distribution: .. math:: \mathbf{x} &\sim \mathcal{N}(\boldsymbol{\mu}, \mathbf{\Lambda}), where :math:`\boldsymbol{\mu}` is the mean vector and :math:`\mathbf{\Lambda}` is the precision matrix (i.e., inverse of the covariance matrix). .. math:: \mathbf{x},\boldsymbol{\mu} \in \mathbb{R}^{D}, \quad \mathbf{\Lambda} \in \mathbb{R}^{D \times D}, \quad \mathbf{\Lambda} \text{ symmetric positive definite} Plates! Parent nodes? Child nodes? See also -------- bayespy.inference.vmp.nodes.wishart.Wishart inference.vmp.nodes.wishart.Wishart vmp.nodes.wishart.Wishart nodes.wishart.Wishart wishart.Wishart Wishart Notes ----- Message passing equations: .. math:: \mathbf{x} &\sim \mathcal{N}(\boldsymbol{\mu}, \mathbf{\Lambda}), .. math:: \mathbf{x},\boldsymbol{\mu} \in \mathbb{R}^{D}, \quad \mathbf{\Lambda} \in \mathbb{R}^{D \times D}, \quad \mathbf{\Lambda} \text{ symmetric positive definite} .. math:: \log\mathcal{N}( \mathbf{x} | \boldsymbol{\mu}, \mathbf{\Lambda} ) &= - \frac{1}{2} \mathbf{x}^{\mathrm{T}} \mathbf{\Lambda} \mathbf{x} + \mathbf{x}^{\mathrm{T}} \mathbf{\Lambda} \boldsymbol{\mu} - \frac{1}{2} \boldsymbol{\mu}^{\mathrm{T}} \mathbf{\Lambda} \boldsymbol{\mu} + \frac{1}{2} \log |\mathbf{\Lambda}| - \frac{D}{2} \log (2\pi) .. math:: \mathbf{u} (\mathbf{x}) &= \left[ \begin{matrix} \mathbf{x} \\ \mathbf{xx}^{\mathrm{T}} \end{matrix} \right] \\ \boldsymbol{\phi} (\boldsymbol{\mu}, \mathbf{\Lambda}) &= \left[ \begin{matrix} \mathbf{\Lambda} \boldsymbol{\mu} \\ - \frac{1}{2} \mathbf{\Lambda} \end{matrix} \right] \\ \boldsymbol{\phi}_{\boldsymbol{\mu}} (\mathbf{x}, \mathbf{\Lambda}) &= \left[ \begin{matrix} \mathbf{\Lambda} \mathbf{x} \\ - \frac{1}{2} \mathbf{\Lambda} \end{matrix} \right] \\ \boldsymbol{\phi}_{\mathbf{\Lambda}} (\mathbf{x}, \boldsymbol{\mu}) &= \left[ \begin{matrix} - \frac{1}{2} \mathbf{xx}^{\mathrm{T}} + \frac{1}{2} \mathbf{x}\boldsymbol{\mu}^{\mathrm{T}} + \frac{1}{2} \boldsymbol{\mu}\mathbf{x}^{\mathrm{T}} - \frac{1}{2} \boldsymbol{\mu\mu}^{\mathrm{T}} \\ \frac{1}{2} \end{matrix} \right] \\ g (\boldsymbol{\mu}, \mathbf{\Lambda}) &= - \frac{1}{2} \operatorname{tr}(\boldsymbol{\mu\mu}^{\mathrm{T}} \mathbf{\Lambda} ) + \frac{1}{2} \log |\mathbf{\Lambda}| \\ g_{\boldsymbol{\phi}} (\boldsymbol{\phi}) &= \frac{1}{4} \boldsymbol{\phi}^{\mathrm{T}}_1 \boldsymbol{\phi}^{-1}_2 \boldsymbol{\phi}_1 + \frac{1}{2} \log | -2 \boldsymbol{\phi}_2 | \\ f(\mathbf{x}) &= - \frac{D}{2} \log(2\pi) \\ \overline{\mathbf{u}} (\boldsymbol{\phi}) &= \left[ \begin{matrix} - \frac{1}{2} \boldsymbol{\phi}^{-1}_2 \boldsymbol{\phi}_1 \\ \frac{1}{4} \boldsymbol{\phi}^{-1}_2 \boldsymbol{\phi}_1 \boldsymbol{\phi}^{\mathrm{T}}_1 \boldsymbol{\phi}^{-1}_2 - \frac{1}{2} \boldsymbol{\phi}^{-1}_2 \end{matrix} \right] """ ndims = (1, 2) ndims_parents = [(1, 2), (2, 0)] # Observations are vectors (1-D): ndim_observations = 1 def __init__(self, mu, Lambda, **kwargs): self.parameter_distributions = (Gaussian, Wishart) # Check for constant mu if utils.utils.is_numeric(mu): mu = Constant(Gaussian)(np.atleast_1d(mu)) # Check for constant Lambda if utils.utils.is_numeric(Lambda): Lambda = Constant(Wishart)(np.atleast_2d(Lambda)) ## # You could check whether the dimensions of mu and Lambda ## # match (and Lambda is square) ## if Lambda.dims[0][-1] != mu.dims[0][-1]: ## raise Exception("Dimensionalities of mu and Lambda do not match.") # Construct super().__init__(mu, Lambda, **kwargs) @staticmethod def compute_fixed_moments(x): """ Compute moments for fixed x. """ return [x, utils.utils.m_outer(x,x)] @staticmethod def _compute_phi_from_parents(*u_parents): return [utils.utils.m_dot(u_parents[1][0], u_parents[0][0]), -0.5 * u_parents[1][0]] @staticmethod def _compute_cgf_from_parents(u_mu, u_Lambda): mu = u_mu[0] mumu = u_mu[1] Lambda = u_Lambda[0] logdet_Lambda = u_Lambda[1] g = (-0.5 * np.einsum('...ij,...ij',mumu,Lambda) + 0.5 * logdet_Lambda) return g @staticmethod def _compute_moments_and_cgf(phi, mask=True): # TODO: Compute -2*phi[1] and simplify the formulas L = utils.utils.m_chol(-2*phi[1]) k = np.shape(phi[0])[-1] # Moments u0 = utils.utils.m_chol_solve(L, phi[0]) u1 = utils.utils.m_outer(u0, u0) + utils.utils.m_chol_inv(L) u = [u0, u1] # G g = (-0.5 * np.einsum('...i,...i', u[0], phi[0]) + 0.5 * utils.utils.m_chol_logdet(L)) #+ 0.5 * np.log(2) * self.dims[0][0]) return (u, g) @staticmethod def _compute_fixed_moments_and_f(x, mask=True): """ Compute u(x) and f(x) for given x. """ k = np.shape(x)[-1] u = [x, utils.utils.m_outer(x,x)] f = -k/2*np.log(2*np.pi) return (u, f) @staticmethod def _compute_message_to_parent(parent, index, u, *u_parents): """ . """ if index == 0: return [utils.utils.m_dot(u_parents[1][0], u[0]), -0.5 * u_parents[1][0]] elif index == 1: xmu = utils.utils.m_outer(u[0], u_parents[0][0]) return [-0.5 * (u[1] - xmu - xmu.swapaxes(-1,-2) + u_parents[0][1]), 0.5] @staticmethod def compute_dims(mu, Lambda): """ Compute the dimensions of phi and u using the parent nodes. Also, check that the dimensionalities of the parents are consistent with each other. Parameters: ----------- mu : Node A VB node with ( (D,), (D,D) ) dimensional Gaussian output. Lambda: Node A VB node with ( (D,D), () ) dimensional Wishart output. """ D = mu.dims[0][0] if mu.dims != ( (D,), (D,D) ): raise Exception("First parent has wrong dimensionality") if Lambda.dims != ( (D,D), () ): raise Exception("Second parent has wrong dimensionality") return ( (D,), (D,D) ) @staticmethod def compute_dims_from_values(x): """ Compute the dimensions of phi and u from a value. The last axis tells the dimensionality, the other axes are plates. Parameters: ----------- x : ndarray """ if np.ndim(x) == 0: raise ValueError("The value must be at least 1-D array.") D = np.shape(x)[-1] return ( (D,), (D,D) ) def get_shape_of_value(self): # Dimensionality of a realization return self.dims[0] def random(self): # TODO/FIXME: You shouldn't draw random values for # observed/fixed elements! # Note that phi[1] is -0.5*inv(Cov) U = utils.utils.m_chol(-2*self.phi[1]) mu = self.u[0] z = np.random.normal(0, 1, self.get_shape(0)) # Compute mu + U'*z z = utils.utils.m_solve_triangular(U, z, trans='T', lower=False) return mu + z def show(self): mu = self.u[0] Cov = self.u[1] - utils.utils.m_outer(mu, mu) print("%s ~ Gaussian(mu, Cov)" % self.name) print(" mu = ") print(mu) print(" Cov = ") print(str(Cov)) def rotate(self, R, inv=None, logdet=None, Q=None): if inv is not None: invR = inv else: invR = np.linalg.inv(R) if logdet is not None: logdetR = logdet else: logdetR = np.linalg.slogdet(R)[1] # It would be more efficient and simpler, if you just rotated the # moments and didn't touch phi. However, then you would need to call # update() before lower_bound_contribution. This is more error-safe. # Rotate plates, if plate rotation matrix is given. Assume that there's # only one plate-axis #logdet_old = np.sum(utils.linalg.logdet_cov(-2*self.phi[1])) if Q is not None: # Rotate moments using Q self.u[0] = np.einsum('ik,kj->ij', Q, self.u[0]) sumQ = np.sum(Q, axis=0) # Rotate natural parameters using Q self.phi[1] = np.einsum('d,dij->dij', sumQ**(-2), self.phi[1]) self.phi[0] = np.einsum('dij,dj->di', -2*self.phi[1], self.u[0]) # Transform parameters using R self.phi[0] = mvdot(invR.T, self.phi[0]) self.phi[1] = dot(invR.T, self.phi[1], invR) if Q is not None: self._update_moments_and_cgf() else: # Transform moments and g using R self.u[0] = mvdot(R, self.u[0]) self.u[1] = dot(R, self.u[1], R.T) self.g -= logdetR def rotate_matrix(self, R1, R2, inv1=None, logdet1=None, inv2=None, logdet2=None, Q=None): """ The vector is reshaped into a matrix by stacking the row vectors. Computes R1*X*R2', which is identical to kron(R1,R2)*x (??) Note that this is slightly different from the standard Kronecker product definition because Numpy stacks row vectors instead of column vectors. """ if Q is not None: # Rotate moments using Q #print("Debug in rotate matrix", np.shape(self.u[0]), self.get_shape(0)) self.u[0] = np.einsum('ik,kj->ij', Q, self.u[0]) sumQ = np.sum(Q, axis=0) # Rotate natural parameters using Q self.phi[1] = np.einsum('d,dij->dij', sumQ**(-2), self.phi[1]) self.phi[0] = np.einsum('dij,dj->di', -2*self.phi[1], self.u[0]) if inv1 is None: inv1 = np.linalg.inv(R1) if logdet1 is None: logdet1 = np.linalg.slogdet(R1)[1] if inv2 is None: inv2 = np.linalg.inv(R2) if logdet2 is None: logdet2 = np.linalg.slogdet(R2)[1] D1 = np.shape(R1)[0] D2 = np.shape(R2)[0] # Reshape into matrices sh0 = np.shape(self.phi[0])[:-1] + (D1,D2) sh1 = np.shape(self.phi[1])[:-2] + (D1,D2,D1,D2) phi0 = np.reshape(self.phi[0], sh0) phi1 = np.reshape(self.phi[1], sh1) # Apply rotations to phi #phi0 = dot(inv1, phi0, inv2.T) phi0 = dot(inv1.T, phi0, inv2) phi1 = np.einsum('...ia,...abcd->...ibcd', inv1.T, phi1) phi1 = np.einsum('...ic,...abcd->...abid', inv1.T, phi1) phi1 = np.einsum('...ib,...abcd->...aicd', inv2.T, phi1) phi1 = np.einsum('...id,...abcd->...abci', inv2.T, phi1) # Reshape back into vectors self.phi[0] = np.reshape(phi0, self.phi[0].shape) self.phi[1] = np.reshape(phi1, self.phi[1].shape) # It'd be better to rotate the moments too.. #g0 = np.sum(np.ones(self.plates)*self.g) self._update_moments_and_cgf() #g1 = np.sum(np.ones(self.plates)*self.g) #dg = g1 - g0 #print("debug rotate", np.sum(self.u[1],axis=0), self.name) ## XX = np.sum(np.reshape(self.u[1], (-1,D1,D2,D1,D2)), ## axis=(0,1,3)) ## print("DEBUG", XX) bayespy-0.1/bayespy/inference/vmp/nodes/tests/0000775000175000017500000000000012174043062023233 5ustar jluttinejluttine00000000000000bayespy-0.1/bayespy/inference/vmp/nodes/tests/test_gaussian_markov_chain.py0000664000175000017500000001662012174013072031202 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### """ Unit tests for gaussian_markov_chain module. """ import unittest import numpy as np from numpy import testing from ..gaussian_markov_chain import GaussianMarkovChain from ..gaussian import Gaussian from ..wishart import Wishart from ..gamma import Gamma from bayespy import utils class TestGaussianMarkovChain(unittest.TestCase): def create_model(self, N, D): # Construct the model Mu = Gaussian(np.random.randn(D), np.identity(D)) Lambda = Wishart(D, utils.random.covariance(D)) A = Gaussian(np.random.randn(D,D), np.identity(D)) V = Gamma(D, np.random.rand(D)) X = GaussianMarkovChain(Mu, Lambda, A, V, n=N) Y = Gaussian(X.as_gaussian(), np.identity(D)) return (Y, X, Mu, Lambda, A, V) def test_plates(self): """ Test that plates are handled correctly. """ def test_message_to_mu0(self): pass def test_message_to_Lambda0(self): pass def test_message_to_A(self): pass def test_message_to_v(self): pass def test_message_to_child(self): pass def test_moments(self): """ Test the updating of GaussianMarkovChain. Check that the moments and the lower bound contribution are computed correctly. """ # TODO: Add plates and missing values! # Dimensionalities D = 3 N = 5 (Y, X, Mu, Lambda, A, V) = self.create_model(N, D) # Inference with arbitrary observations y = np.random.randn(N,D) Y.observe(y) X.update() (x_vb, xnxn_vb, xpxn_vb) = X.get_moments() # Get parameter moments (mu0, mumu0) = Mu.get_moments() (icov0, logdet0) = Lambda.get_moments() (a, aa) = A.get_moments() (icov_x, logdetx) = V.get_moments() icov_x = np.diag(icov_x) # Prior precision Z = np.einsum('...kij,...kk->...ij', aa, icov_x) U_diag = [icov0+Z] + (N-2)*[icov_x+Z] + [icov_x] U_super = (N-1) * [-np.dot(a.T, icov_x)] U = utils.utils.block_banded(U_diag, U_super) # Prior mean mu_prior = np.zeros(D*N) mu_prior[:D] = np.dot(icov0,mu0) # Data Cov = np.linalg.inv(U + np.identity(D*N)) mu = np.dot(Cov, mu_prior + y.flatten()) # Moments xx = mu[:,np.newaxis]*mu[np.newaxis,:] + Cov mu = np.reshape(mu, (N,D)) xx = np.reshape(xx, (N,D,N,D)) # Check results testing.assert_allclose(x_vb, mu, err_msg="Incorrect mean") for n in range(N): testing.assert_allclose(xnxn_vb[n,:,:], xx[n,:,n,:], err_msg="Incorrect second moment") for n in range(N-1): testing.assert_allclose(xpxn_vb[n,:,:], xx[n,:,n+1,:], err_msg="Incorrect lagged second moment") # Compute the entropy H(X) ldet = utils.linalg.logdet_cov(Cov) H = utils.random.gaussian_entropy(-ldet, N*D) # Compute xx = np.reshape(xx, (N*D, N*D)) mu = np.reshape(mu, (N*D,)) ldet = -logdet0 - np.sum(np.ones((N-1,D))*logdetx) P = utils.random.gaussian_logpdf(np.einsum('...ij,...ij', xx, U), np.einsum('...i,...i', mu, mu_prior), np.einsum('...ij,...ij', mumu0, icov0), -ldet, N*D) # The VB bound from the net l = X.lower_bound_contribution() testing.assert_allclose(l, H+P) # Compute the true bound + H(X) def test_smoothing(self): """ Test the posterior estimation of GaussianMarkovChain. Create time-variant dynamics and compare the results of BayesPy VB inference and standard Kalman filtering & smoothing. This is not that useful anymore, because the moments are checked much better in another test method. """ # # Set up an artificial system # # Dimensions N = 500 D = 2 # Dynamics (time varying) A0 = np.array([[.9, -.4], [.4, .9]]) A1 = np.array([[.98, -.1], [.1, .98]]) l = np.linspace(0, 1, N-1).reshape((-1,1,1)) A = (1-l)*A0 + l*A1 # Innovation covariance matrix (time varying) v = np.random.rand(D) V = np.diag(v) # Observation noise covariance matrix C = np.identity(D) # # Simulate data # X = np.empty((N,D)) Y = np.empty((N,D)) x = np.array([0.5, -0.5]) X[0,:] = x Y[0,:] = x + np.random.multivariate_normal(np.zeros(D), C) for n in range(N-1): x = np.dot(A[n,:,:],x) + np.random.multivariate_normal(np.zeros(D), V) X[n+1,:] = x Y[n+1,:] = x + np.random.multivariate_normal(np.zeros(D), C) # # BayesPy inference # # Construct VB model Xh = GaussianMarkovChain(np.zeros(D), np.identity(D), A, 1/v, n=N) Yh = Gaussian(Xh.as_gaussian(), np.identity(D), plates=(N,)) # Put data Yh.observe(Y) # Run inference Xh.update() # Store results Xh_vb = Xh.u[0] CovXh_vb = Xh.u[1] - Xh_vb[...,np.newaxis,:] * Xh_vb[...,:,np.newaxis] # # "The ground truth" using standard Kalman filter and RTS smoother # V = N*(V,) UY = Y U = N*(C,) (Xh, CovXh) = utils.utils.kalman_filter(UY, U, A, V, np.zeros(D), np.identity(D)) (Xh, CovXh) = utils.utils.rts_smoother(Xh, CovXh, A, V) # # Check results # self.assertTrue(np.allclose(Xh_vb, Xh)) self.assertTrue(np.allclose(CovXh_vb, CovXh)) ## if __name__ == '__main__': ## unittest.main() bayespy-0.1/bayespy/inference/vmp/nodes/tests/test_dot.py0000664000175000017500000002152312174013072025433 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### """ Unit tests for `dot` module. """ import unittest import numpy as np import scipy from numpy import testing from ..dot import Dot, MatrixDot from ..gaussian import Gaussian from ...vmp import VB from bayespy.utils import utils from bayespy.utils import linalg from bayespy.utils import random class TestMatrixDot(unittest.TestCase): # Dimensionalities M = 3 N = 4 # Plates # Y: D4,D3,D2,D1 # A: 1,D3,D2, 1 # X: 1,D3, 1,D1 D4 = 5 D3 = 7 D2 = 6 D1 = 2 def setUp(self): """ Generate artificial data for the tests. """ # Generate data self.a = np.random.randn(self.D3,self.D2,1,self.M,self.N) self.x = np.random.randn(self.D3,1,self.D1,self.N) f = np.einsum('...ij,...j->...i', self.a, self.x) v = np.random.randn(self.M,self.M) self.vv = np.dot(v.T, v) self.cov = np.linalg.inv(self.vv) l_cov = scipy.linalg.cholesky(self.cov, lower=True) self.y = f + np.einsum('...ij,...j->...i', l_cov, np.random.randn(self.D4, self.D3, self.D2, self.D1, self.M)) self.Lambda = self.vv super().setUp() def check_A(self, mask): """ Check update equations of parent A. Use a simple model that has parent nodes and a child node around MatrixDot. """ # Parent nodes A = Gaussian(np.zeros(self.M*self.N), np.identity(self.M*self.N), plates=(self.D3,self.D2,1), name='A') X = Gaussian(self.x, np.identity(self.N), name='X') # Node itself AX = MatrixDot(A, X, name='AX') # Child node Y = Gaussian(AX, self.Lambda, plates=(self.D4,self.D3,self.D2,self.D1), name='Y') # Put in data Y.observe(self.y, mask=mask) # VB model Q = VB(Y, A) Q.update(A, repeat=1) u = A.get_moments() # Compute true solution mask = np.ones((self.D4,self.D3,self.D2,self.D1)) * mask vv = mask[...,np.newaxis,np.newaxis] * self.vv xx = (self.x[...,:,np.newaxis] * self.x[...,np.newaxis,:] + np.identity(self.N)) Cov_A = (vv[...,:,np.newaxis,:,np.newaxis] * xx[...,np.newaxis,:,np.newaxis,:]) # from data Cov_A = np.sum(Cov_A, axis=(0,3), keepdims=True) Cov_A = np.reshape(Cov_A, (self.D3,self.D2,1,self.M*self.N,self.M*self.N)) Cov_A = Cov_A + np.identity(self.M*self.N) # add prior Cov_A = linalg.chol_inv(linalg.chol(Cov_A)) mu_A = np.einsum('...ij,...j,...k->...ik', vv, self.y, self.x) mu_A = np.sum(mu_A, axis=(0,3)) mu_A = np.reshape(mu_A, (self.D3,self.D2,1,self.M*self.N)) mu_A = np.einsum('...ij,...j->...i', Cov_A, mu_A) # Compare VB results to the analytic solution: Cov_vb = u[1] - u[0][...,np.newaxis,:]*u[0][...,:,np.newaxis] testing.assert_allclose(Cov_vb, Cov_A, err_msg="Incorrect second moment.") testing.assert_allclose(u[0], mu_A, err_msg="Incorrect first moment.") def test_parent_A(self): """ Test messages from MatrixDot to A without missing values. Compare the results of A to the analytic solution, which can be obtained because only one node is unknown. """ self.check_A(np.array([True])) def test_parent_A_mv(self): """ Test messages from MatrixDot to A with missing values. """ self.check_A(random.mask(self.D4,self.D3,self.D2,self.D1,p=0.5)) def check_X(self, mask): """ Check update equations of parent X. Use a simple model that has parent nodes and a child node around MatrixDot. """ # Parent nodes A = Gaussian(self.a.reshape((self.D3,self.D2,1,self.M*self.N)), np.identity(self.M*self.N), name='A') X = Gaussian(np.zeros(self.N), np.identity(self.N), plates=(self.D3,1,self.D1), name='X') # Node itself AX = MatrixDot(A, X, name='AX') # Child node Y = Gaussian(AX, self.Lambda, plates=(self.D4,self.D3,self.D2,self.D1), name='Y') # Put in data Y.observe(self.y, mask=mask) # VB model Q = VB(Y, X) Q.update(X, repeat=1) u = X.get_moments() # Compute true solution mask = np.ones((self.D4,self.D3,self.D2,self.D1)) * mask vv = mask[...,np.newaxis,np.newaxis] * self.vv aa = np.einsum('...ki,...kl,...lj->...ij', self.a, vv, self.a) aa = aa + np.einsum('...ij,...kl,...kl->...ij', np.identity(self.N), np.identity(self.M), vv) aa = np.sum(aa, axis=2, keepdims=True) aa = np.sum(aa, axis=0) Cov_X = linalg.chol_inv(linalg.chol(aa + np.identity(self.N))) mu_X = np.einsum('...kj,...kl,...l->...j', self.a, vv, self.y) mu_X = np.sum(mu_X, axis=2, keepdims=True) mu_X = np.sum(mu_X, axis=0) mu_X = np.einsum('...ij,...j->...i', Cov_X, mu_X) # Compare VB results to the analytic solution: Cov_vb = u[1] - u[0][...,np.newaxis,:]*u[0][...,:,np.newaxis] testing.assert_allclose(Cov_vb, Cov_X, err_msg="Incorrect second moment.") testing.assert_allclose(u[0], mu_X, err_msg="Incorrect first moment.") def test_parent_X(self): """ Test messages from MatrixDot to X without missing values. Compare the results of X to the analytic VB solution. """ self.check_X(np.array([True])) def test_parent_X_mv(self): """ Test messages from MatrixDot to X with missing values. Compare the results of X to the analytic VB solution. """ self.check_X(random.mask(self.D4,self.D3,self.D2,self.D1,p=0.5)) def test_child(self): """ Test messages from MatrixDot to children. """ # Parent nodes A = Gaussian(self.a.reshape((self.D3,self.D2,1,self.M*self.N)), np.identity(self.M*self.N), name='A') X = Gaussian(self.x, np.identity(self.N), plates=(self.D3,1,self.D1), name='X') # Node itself AX = MatrixDot(A, X, name='AX') # Moments from the model (u0, u1) = AX.get_moments() # Compute the true moments ax = np.einsum('...ij,...j->...i', self.a, self.x) aa = (np.einsum('...ij,...kl->...ijkl', self.a, self.a) + np.identity(self.N*self.M).reshape((self.M, self.N, self.M, self.N))) xx = (np.einsum('...i,...j->...ij', self.x, self.x) + np.identity(self.N)) axxa = np.einsum('...ikjl,...kl->...ij', aa, xx) # Check the moments testing.assert_allclose(u0, ax, err_msg="Incorrect first moment") testing.assert_allclose(u1, axxa, err_msg="Incorrect second moment") bayespy-0.1/bayespy/inference/vmp/nodes/tests/test_deterministic.py0000664000175000017500000002723412174013072027515 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### """ Unit tests for `deterministic` module. """ import unittest import numpy as np import scipy from numpy import testing from ..node import Node from ..deterministic import tile from bayespy import utils class TestTile(unittest.TestCase): def check_message_to_children(self, tiles, u_parent, u_tiled, dims=None, plates=None): # Set up the dummy model class Dummy(Node): def get_moments(self): return u_parent X = Dummy(dims=dims, plates=plates) Y = tile(X, tiles) u_Y = Y._compute_moments(u_parent) for (x,y) in zip(u_Y, u_tiled): self.assertEqual(np.shape(x), np.shape(y), msg="Incorrect shape.") testing.assert_allclose(x, y, err_msg="Incorrect moments.") def test_message_to_children(self): """ Test the moments of Tile node. """ # Define th check function check_message_to_children = self.check_message_to_children # Check scalar (and broadcasting) check_message_to_children(2, (5,), (5,), dims=[()], plates=()), # Check 1-D check_message_to_children(2, ([1,2],), ([1,2,1,2],), dims=[()], plates=(2,)) # Check N-D check_message_to_children(2, ([[1,2], [3,4], [5,6]],), ([[1,2,1,2], [3,4,3,4], [5,6,5,6]],), dims=[()], plates=(3,2)) # Check not-last plate check_message_to_children([2,1], ([[1,2], [3,4]],), ([[1,2], [3,4], [1,2], [3,4]],), dims=[()], plates=(2,2)) # Check several plates check_message_to_children([2,3], ([[1,2], [3,4]],), ([[1,2,1,2,1,2], [3,4,3,4,3,4], [1,2,1,2,1,2], [3,4,3,4,3,4]],), dims=[()], plates=(2,2)) # Check non-zero dimensional variables check_message_to_children(2, ([[1,2], [3,4]],), ([[1,2], [3,4], [1,2], [3,4]],), dims=[(2,)], plates=(2,)) # Check several moments check_message_to_children(2, ([[1,2], [3,4]], [1,2]), ([[1,2], [3,4], [1,2], [3,4]], [1,2,1,2]), dims=[(2,),()], plates=(2,)) # Check broadcasting of tiled plate check_message_to_children(2, ([[1,], [2,]],), ([[1,], [2,]],), dims=[()], plates=(2,2)) # Check broadcasting of non-tiled plate check_message_to_children(2, ([[1,2]],), ([[1,2,1,2]],), dims=[()], plates=(2,2)) # Check broadcasting of leading plates that are not in parent check_message_to_children([2,1], ([1,2],), ([1,2],), dims=[()], plates=(2,)) def check_message_to_parent(self, tiles, m_children, m_true, dims=None, plates_parent=None, plates_children=None): # Set up the dummy model class Dummy(Node): pass X = Dummy(dims=dims, plates=plates_parent) Y = tile(X, tiles) m = Y._compute_message_to_parent(0, m_children, None) for (x,y) in zip(m, m_true): self.assertEqual(np.shape(x), np.shape(y), msg="Incorrect shape.") testing.assert_allclose(x, y, err_msg="Incorrect message.") def test_message_to_parent(self): """ Test the parent message of Tile node. """ # Define th check function check = self.check_message_to_parent # Check scalar check(2, ([5,5],), (10,), dims=[()], plates_parent=(), plates_children=(2,)), # Check 1-D check(2, ([1,2,3,4],), ([4,6],), dims=[()], plates_parent=(2,), plates_children=(4,)) # Check N-D check(2, ([[1,2,7,8], [3,4,9,0], [5,6,1,2]],), ([[8,10], [12,4], [6,8]],), dims=[()], plates_parent=(3,2), plates_children=(3,4)) # Check not-last plate check([2,1], ([[1,2], [3,4], [5,6], [7,8]],), ([[6,8], [10,12]],), dims=[()], plates_parent=(2,2), plates_children=(4,2)) # Check several plates check([2,3], ([[1,2,1,2,1,2], [3,4,3,4,3,4], [1,2,1,2,1,2], [3,4,3,4,3,4]],), ([[6,12], [18,24]],), dims=[()], plates_parent=(2,2), plates_children=(4,6)) # Check broadcasting if message has unit axis for tiled plate check(2, ([[1,], [2,], [3,]],), ([[2,], [4,], [6,]],), dims=[()], plates_parent=(3,2), plates_children=(3,4)) # Check broadcasting if message has unit axis for non-tiled plate check(2, ([[1,2,3,4]],), ([[4,6]],), dims=[()], plates_parent=(3,2), plates_children=(3,4)) # Check non-zero dimensional variables check(2, ([[1,2], [3,4], [5,6], [7,8]],), ([[6,8], [10,12]],), dims=[(2,)], plates_parent=(2,), plates_children=(4,)) # Check several moments check(2, ([[1,2], [3,4], [5,6], [7,8]], [1,2,3,4]), ([[6,8], [10,12]], [4,6]), dims=[(2,),()], plates_parent=(2,), plates_children=(4,)) def check_mask_to_parent(self, tiles, mask_child, mask_true, plates_parent=None, plates_children=None): # Set up the dummy model class Dummy(Node): pass X = Dummy(dims=[()], plates=plates_parent) Y = tile(X, tiles) mask = Y._compute_mask_to_parent(0, mask_child) self.assertEqual(np.shape(mask), np.shape(mask_true), msg="Incorrect shape.") testing.assert_equal(mask, mask_true, err_msg="Incorrect mask.") def test_mask_to_parent(self): """ Test the mask message to parent of Tile node. """ # Define th check function check = self.check_mask_to_parent # Check scalar parent check(2, [True,False], True, plates_parent=(), plates_children=(2,)) check(2, [False,False], False, plates_parent=(), plates_children=(2,)) # Check 1-D check(2, [True,False,False,False], [True,False], plates_parent=(2,), plates_children=(4,)) # Check N-D check(2, [[True,False,True,False], [False,True,False,False]], [[True,False], [False,True]], plates_parent=(2,2), plates_children=(2,4)) # Check not-last plate check([2,1], [[True,False], [False,True], [True,False], [False,False]], [[True,False], [False,True]], plates_parent=(2,2), plates_children=(4,2)) # Check several plates check([2,3], [[False,False,True,False,False,False], [False,False,False,False,False,False], [False,False,False,False,False,False], [False,True,False,False,False,False]], [[True,False], [False,True]], plates_parent=(2,2), plates_children=(4,6)) # Check broadcasting if message has unit axis for tiled plate check(2, [[True,], [False,], [True,]], [[True,], [False,], [True,]], plates_parent=(3,2), plates_children=(3,4)) # Check broadcasting if message has unit axis for non-tiled plate check(2, [[False,False,False,True]], [[False,True]], plates_parent=(3,2), plates_children=(3,4)) bayespy-0.1/bayespy/inference/vmp/nodes/tests/__init__.py0000664000175000017500000000000012174013072025330 0ustar jluttinejluttine00000000000000bayespy-0.1/bayespy/inference/vmp/nodes/tests/test_node.py0000664000175000017500000002121112174013072025564 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### """ Unit tests for `dot` module. """ import unittest import numpy as np import scipy from numpy import testing from ..node import Node from ...vmp import VB from bayespy import utils class TestNode(unittest.TestCase): def check_message_to_parent(self, plates_child, plates_message, plates_mask, plates_parent, dims=(2,)): # Dummy message msg = np.random.randn(*(plates_message+dims)) # Mask with every other True and every other False mask = np.mod(np.arange(np.prod(plates_mask)).reshape(plates_mask), 2) == 0 # Set up the dummy model class Dummy(Node): def _get_message_and_mask_to_parent(self, index): return ([msg], mask) parent = Dummy(dims=[dims], plates=plates_parent) child = Dummy(parent, dims=[dims], plates=plates_child) m = child._message_to_parent(0)[0] * np.ones(plates_parent+dims) # Brute-force computation of the message without too much checking m_true = msg * mask[...,np.newaxis] * np.ones(plates_child+dims) for ind in range(len(plates_child)): axis = -ind - 2 if ind >= len(plates_parent): m_true = np.sum(m_true, axis=axis, keepdims=False) elif plates_parent[-ind-1] == 1: m_true = np.sum(m_true, axis=axis, keepdims=True) testing.assert_allclose(m, m_true, err_msg="Incorrect message.") def test_message_to_parent(self): """ Test plate handling in _message_to_parent. """ # Test empty plates with scalar messages self.check_message_to_parent((), (), (), (), dims=()) # Test singular plates self.check_message_to_parent((2,3,4), (2,3,4), (2,3,4), (2,3,4)) self.check_message_to_parent((2,3,4), (2,1,4), (2,3,4), (2,3,4)) self.check_message_to_parent((2,3,4), (2,3,4), (2,1,4), (2,3,4)) self.check_message_to_parent((2,3,4), (2,3,4), (2,3,4), (2,1,4)) self.check_message_to_parent((2,3,4), (2,1,4), (2,1,4), (2,3,4)) self.check_message_to_parent((2,3,4), (2,3,4), (2,1,4), (2,1,4)) self.check_message_to_parent((2,3,4), (2,1,4), (2,3,4), (2,1,4)) self.check_message_to_parent((2,3,4), (2,1,4), (2,1,4), (2,1,4)) # Test missing plates self.check_message_to_parent((4,3), (4,3), (4,3), (4,3)) self.check_message_to_parent((4,3), ( 3,), (4,3), (4,3)) self.check_message_to_parent((4,3), (4,3), ( 3,), (4,3)) self.check_message_to_parent((4,3), (4,3), (4,3), ( 3,)) self.check_message_to_parent((4,3), ( 3,), ( 3,), (4,3)) self.check_message_to_parent((4,3), ( 3,), (4,3), ( 3,)) self.check_message_to_parent((4,3), (4,3), ( 3,), ( 3,)) self.check_message_to_parent((4,3), ( 3,), ( 3,), ( 3,)) # A complex test self.check_message_to_parent((7,6,5,4,3), ( 6,1,4,3), (1,1,5,4,1), ( 6,5,1,3)) # Test errors for inconsistent shapes self.assertRaises(ValueError, self.check_message_to_parent, (3,), (1,3,), (3,), (3,)) self.assertRaises(ValueError, self.check_message_to_parent, (3,), (3,), (1,3,), (3,)) self.assertRaises(ValueError, self.check_message_to_parent, (3,), (1,3,), (1,3,), (3,)) self.assertRaises(ValueError, self.check_message_to_parent, (3,), (4,), (3,), (3,)) self.assertRaises(ValueError, self.check_message_to_parent, (3,), (3,), (4,), (3,)) self.assertRaises(ValueError, self.check_message_to_parent, (3,), (4,), (4,), (3,)) self.assertRaises(ValueError, self.check_message_to_parent, (3,), (4,), (3,), (1,)) self.assertRaises(ValueError, self.check_message_to_parent, (3,), (3,), (4,), (1,)) self.assertRaises(ValueError, self.check_message_to_parent, (3,), (4,), (4,), (1,)) self.assertRaises(ValueError, self.check_message_to_parent, (1,), (4,), (3,), (1,)) self.assertRaises(ValueError, self.check_message_to_parent, (1,), (3,), (4,), (1,)) self.assertRaises(ValueError, self.check_message_to_parent, (1,), (4,), (4,), (1,)) bayespy-0.1/bayespy/inference/vmp/nodes/expfamily.py0000664000175000017500000002201612174013072024440 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np from bayespy.utils import utils from .stochastic import Stochastic class ExponentialFamily(Stochastic): """ A base class for nodes using natural parameterization `phi`. phi Sub-classes must implement the following static methods: _compute_message_to_parent(index, u_self, *u_parents) _compute_phi_from_parents(*u_parents, mask) _compute_moments_and_cgf(phi, mask) Sub-classes may need to re-implement: 1. If they manipulate plates: _compute_mask_to_parent(index, mask) _plates_to_parent(self, index) _plates_from_parent(self, index) """ def __init__(self, *args, initialize=True, **kwargs): super().__init__(*args, initialize=initialize, **kwargs) if not initialize: axes = len(self.plates)*(1,) self.phi = [utils.nans(axes+dim) for dim in self.dims] # Terms for the lower bound (G for latent and F for observed) self.g = np.array(0) self.f = np.array(0) def initialize_from_prior(self): if not np.all(self.observed): # Messages from parents #u_parents = [parent.message_to_child() for parent in self.parents] u_parents = self._message_from_parents() # Update natural parameters using parents self._update_phi_from_parents(*u_parents) # Update moments mask = np.logical_not(self.observed) (u, g) = self._compute_moments_and_cgf(self.phi, mask=mask) # TODO/FIXME/BUG: You should use observation mask in order to not # overwrite them! self._set_moments_and_cgf(u, g, mask=mask) def initialize_from_parameters(self, *args): # Get the moments of the parameters if they were fixed to the # given values. #u_parents = self.compute_fixed_parameter_moments(*args) u_parents = list() for (ind, x) in enumerate(args): u = self.parameter_distributions[ind].compute_fixed_moments(x) u_parents.append(u) # Update natural parameters self._update_phi_from_parents(*u_parents) # Update moments # TODO/FIXME: Use the mask of observations! mask = np.logical_not(self.observed) (u, g) = self._compute_moments_and_cgf(self.phi, mask=mask) self._set_moments_and_cgf(u, g, mask=mask) def initialize_from_value(self, x): # Update moments from value if np.shape(x) != self.plates + self.get_shape_of_value(): raise ValueError("Invalid shape of the value for initialization.") mask = np.logical_not(self.observed) (u, f) = self._compute_fixed_moments_and_f(x, mask=mask) self._set_moments_and_cgf(u, np.inf, mask=mask) def initialize_from_random(self): self.initialize_from_prior() X = self.random() self.initialize_from_value(X) def _update_phi_from_parents(self, *u_parents): # TODO/FIXME: Could this be combined to the function # _update_distribution_and_lowerbound ? # No, because some initialization methods may want to use this. # This makes correct broadcasting self.phi = self._compute_phi_from_parents(*u_parents) self.phi = list(self.phi) # Make sure phi has the correct number of axes. It makes life # a bit easier elsewhere. for i in range(len(self.phi)): axes = len(self.plates) + self.ndims[i] - np.ndim(self.phi[i]) if axes > 0: # Add axes self.phi[i] = utils.add_leading_axes(self.phi[i], axes) elif axes < 0: # Remove extra leading axes first = -(len(self.plates)+self.ndims[i]) sh = np.shape(self.phi[i])[first:] self.phi[i] = np.reshape(self.phi[i], sh) def _set_moments_and_cgf(self, u, g, mask=True): self._set_moments(u, mask=mask) # TODO/FIXME: Apply mask to g too!! self.g = g def _update_distribution_and_lowerbound(self, m_children, *u_parents): # Update phi first from parents.. self._update_phi_from_parents(*u_parents) # .. then just add children's message for i in range(len(self.phi)): self.phi[i] = self.phi[i] + m_children[i] # Update u and g self._update_moments_and_cgf() def _update_moments_and_cgf(self): """ Update moments and cgf based on current phi. """ # Mask for plates to update (i.e., unobserved plates) update_mask = np.logical_not(self.observed) # Compute the moments (u) and CGF (g)... (u, g) = self._compute_moments_and_cgf(self.phi, mask=update_mask) # ... and store them self._set_moments_and_cgf(u, g, mask=update_mask) def lower_bound_contribution(self, gradient=False): # Compute E[ log p(X|parents) - log q(X) ] over q(X)q(parents) # Messages from parents #u_parents = [parent.message_to_child() for parent in self.parents] u_parents = self._message_from_parents() phi = self._compute_phi_from_parents(*u_parents) # G from parents L = self._compute_cgf_from_parents(*u_parents) # L = g # G for unobserved variables (ignored variables are handled # properly automatically) latent_mask = np.logical_not(self.observed) #latent_mask = np.logical_and(self.mask, np.logical_not(self.observed)) # F for observed, G for latent L = L + np.where(self.observed, self.f, -self.g) for (phi_p, phi_q, u_q, dims) in zip(phi, self.phi, self.u, self.dims): # Form a mask which puts observed variables to zero and # broadcasts properly latent_mask_i = utils.add_axes(latent_mask, len(self.plates) - np.ndim(latent_mask), len(dims)) axis_sum = tuple(range(-len(dims),0)) # Compute the term phi_q = np.where(latent_mask_i, phi_q, 0) # TODO/FIXME: Use einsum here? Z = np.sum((phi_p-phi_q) * u_q, axis=axis_sum) L = L + Z return (np.sum(np.where(self.mask, L, 0)) * self._plate_multiplier(self.plates, np.shape(L), np.shape(self.mask))) #return L @classmethod def _compute_logpdf(cls, u, phi, g, f): """ Compute E[log p(X)] given E[u], E[phi], E[g] and E[f]. Does not sum over plates.""" # TODO/FIXME: Should I take into account what is latent or # observed, or what is even totally ignored (by the mask). L = g + f for (phi_i, u_i, ndims_i) in zip(phi, u, cls.ndims): #for (phi_i, u_i, len_dims_i) in zip(phi, u, len_dims): # Axes to sum (dimensions of the variable, not the plates) axis_sum = tuple(range(-ndims_i,0)) #axis_sum = tuple(range(-len_dims_i,0)) # Compute the term # TODO/FIXME: Use einsum! L = L + np.sum(phi_i * u_i, axis=axis_sum) return L def save(self, group): """ Save the state of the node into a HDF5 file. group can be the root """ ## if name is None: ## name = self.name ## subgroup = group.create_group(name) for i in range(len(self.phi)): utils.write_to_hdf5(group, self.phi[i], 'phi%d' % i) utils.write_to_hdf5(group, self.f, 'f') utils.write_to_hdf5(group, self.g, 'g') super().save(group) def load(self, group): """ Load the state of the node from a HDF5 file. """ # TODO/FIXME: Check that the shapes are correct! for i in range(len(self.phi)): phii = group['phi%d' % i][...] self.phi[i] = phii self.f = group['f'][...] self.g = group['g'][...] super().load(group) bayespy-0.1/bayespy/inference/vmp/nodes/__init__.py0000664000175000017500000000612612174013072024205 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### # Import some most commonly used nodes from . import * #from .node import Node #from .variable import Variable #from .constant import Constant from .gaussian import Gaussian from .wishart import Wishart from .normal import Normal from .gamma import Gamma from .dirichlet import Dirichlet from .categorical import Categorical from .dot import Dot, MatrixDot from .mixture import Mixture from .gaussian_markov_chain import GaussianMarkovChain ## from . import variable ## import imp ## if 'Variable' in locals(): ## print('VARIABLE!') ## #del Variable #imp.reload(Variable) ## #del Dirichlet ## #imp.reload(variable) ## import variable ## from .variable import Variable ## from .constant import Constant ## from .wishart import Wishart ## from .gaussian import Gaussian ## from .gamma import Gamma ## from .normal import Normal ## from .dirichlet import Dirichlet ## from .categorical import Categorical ## from .mixture import Mixture ## from .dot import Dot ## from . import variable ## from . import constant ## # BLAAH, they must in the correct order in order them to work.. :/ ## from . import wishart ## from . import gaussian ## from . import gamma ## from . import normal ## from . import dirichlet ## from . import categorical ## from . import mixture ## from . import dot ## import constant ## # BLAAH, they must in the correct order in order them to work.. :/ ## import wishart ## import gaussian ## import gamma ## import normal ## import dirichlet ## import categorical ## import mixture ## import dot ## import imp ## imp.reload(variable) ## imp.reload(constant) ## imp.reload(wishart) ## imp.reload(gaussian) ## imp.reload(gamma) ## imp.reload(normal) ## imp.reload(dirichlet) ## imp.reload(categorical) ## imp.reload(mixture) ## imp.reload(dot) ## from .Gaussian import Gaussian ## from .Wishart import Wishart ## from .Normal import Normal ## from .Gamma import Gamma ## from .Dirichlet import Dirichlet ## from .Categorical import Categorical ## from .Mixture import Mixture ## from .Dot import Dot bayespy-0.1/bayespy/inference/vmp/nodes/CovarianceFunctions.py0000664000175000017500000007154312174013072026416 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import itertools import numpy as np #import scipy as sp import scipy.sparse as sp # prefer CSC format #import scipy.linalg.decomp_cholesky as decomp #import scipy.linalg as linalg #import scipy.special as special #import matplotlib.pyplot as plt #import time #import profile import scipy.spatial.distance as dist import scikits.sparse.distance as spdist import Nodes.ExponentialFamily as ef import utils import imp imp.reload(spdist) imp.reload(ef) imp.reload(utils) # Covariance matrices can be either arrays or matrices so be careful # with products and powers! Use explicit multiply or dot instead of # *-operator. def gp_cov_se(D2, overwrite=False): if overwrite: K = D2 K *= -0.5 np.exp(K, out=K) else: K = np.exp(-0.5*D2) return K def gp_cov_pp2_new(r, d, derivative=False): # Dimension dependent parameter q = 2 j = np.floor(d/2) + q + 1 # Polynomial coefficients a2 = j**2 + 4*j + 3 a1 = 3*j + 6 a0 = 3 # Two parts of the covariance function k1 = (1-r) ** (j+2) k2 = (a2*r**2 + a1*r + 3) # TODO: Check that derivative is 0, 1 or 2! if derivative == 0: # Return covariance return k1 * k2 / 3 dk1 = - (j+2) * (1-r)**(j+1) dk2 = 2*a2*r + a1 if derivative == 1: # Return first derivative of the covariance return (k1 * dk2 + dk1 * k2) / 3 ddk1 = (j+2) * (j+1) * (1-r)**j ddk2 = 2*a2 if derivative == 2: # Return second derivative of the covariance return (ddk1*k2 + 2*dk1*dk2 + k1*ddk2) / 3 def gp_cov_pp2(r, d, gradient=False): # Dimension dependent parameter j = np.floor(d/2) + 2 + 1 # Polynomial coefficients a2 = j**2 + 4*j + 3 a1 = 3*j + 6 a0 = 3 # Two parts of the covariance function k1 = (1-r) ** (j+2) k2 = (a2*r**2 + a1*r + 3) # The covariance function k = k1 * k2 / 3 if gradient: # The gradient w.r.t. r dk = k * (j+2) / (r-1) + k1 * (2*a2*r + a1) / 3 return (k, dk) else: return k def gp_cov_delta(N): # TODO: Use sparse matrices here! if N > 0: #print('in gpcovdelta', N, sp.identity(N).shape) return sp.identity(N) else: # Sparse matrices do not allow zero-length dimensions return np.identity(N) #return np.identity(N) #return np.asmatrix(np.identity(N)) def squared_distance(x1, x2): ## # Reshape arrays to 2-D arrays ## sh1 = np.shape(x1)[:-1] ## sh2 = np.shape(x2)[:-1] ## d = np.shape(x1)[-1] ## x1 = np.reshape(x1, (-1,d)) ## x2 = np.reshape(x2, (-1,d)) (m1,n1) = x1.shape (m2,n2) = x2.shape if m1 == 0 or m2 == 0: D2 = np.empty((m1,m2)) else: # Compute squared Euclidean distance D2 = dist.cdist(x1, x2, metric='sqeuclidean') #D2 = np.asmatrix(D2) # Reshape the result #D2 = np.reshape(D2, sh1 + sh2) return D2 # General rule for the parameters for covariance functions: # # (value, [ [dvalue1, ...], [dvalue2, ...], [dvalue3, ...], ...]) # # For instance, # # k = covfunc_se((1.0, []), (15, [ [1,update_grad] ])) # K = k((x1, [ [dx1,update_grad] ]), (x2, [])) # # Plain values are converted as: # value -> (value, []) def gp_standardize_input(x): if np.size(x) == 0: x = np.reshape(x, (0,0)) elif np.ndim(x) == 0: x = np.reshape(x, (1,1)) elif np.ndim(x) == 1: x = np.reshape(x, (-1,1)) elif np.ndim(x) == 2: x = np.atleast_2d(x) else: raise Exception("Standard GP inputs must be 2-dimensional") return x def gp_preprocess_inputs(x1,x2=None): #args = list(args) #if len(args) < 1 or len(args) > 2: #raise Exception("Number of inputs must be one or two") if x2 is None: x1 = gp_standardize_input(x1) return x1 else: if x1 is x2: x1 = gp_standardize_input(x1) x2 = x1 else: x1 = gp_standardize_input(x1) x2 = gp_standardize_input(x2) return (x1, x2) #return args ## def gp_preprocess_inputs(x1,x2=None): ## #args = list(args) ## #if len(args) < 1 or len(args) > 2: ## #raise Exception("Number of inputs must be one or two") ## if x2 is not None: len(args) == 2: ## if args[0] is args[1]: ## args[0] = gp_standardize_input(args[0]) ## args[1] = args[0] ## else: ## args[1] = gp_standardize_input(args[1]) ## args[0] = gp_standardize_input(args[0]) ## else: ## args[0] = gp_standardize_input(args[0]) ## return args # TODO: # General syntax for these covariance functions: # covfunc(hyper1, # hyper2, # ... # hyperN, # x1, # x2=None, # gradient=list_of_booleans_for_each_hyperparameter) def covfunc_zeros(x1, x2=None, gradient=False): inputs = gp_preprocess_inputs(*inputs) # Compute distance and covariance matrix if x2 is None: x1 = gp_preprocess_inputs(x1) # Only variance vector asked N = np.shape(x1)[0] # TODO: Use sparse matrices! K = np.zeros(N) #K = np.asmatrix(np.zeros((N,1))) else: (x1,x2) = gp_preprocess_inputs(x1,x2) # Full covariance matrix asked #x1 = inputs[0] #x2 = inputs[1] # Number of inputs x1 N1 = np.shape(x1)[0] N2 = np.shape(x2)[0] # TODO: Use sparse matrices! K = np.zeros((N1,N2)) #K = np.asmatrix(np.zeros((N1,N2))) if gradient is not False: return (K, []) else: return K def covfunc_delta(amplitude, x1, x2=None, gradient=False): # Make sure that amplitude is a scalar, not an array object amplitude = utils.array_to_scalar(amplitude) ## if gradient: ## gradient_amplitude = gradient[0] ## else: ## gradient_amplitude = [] ## inputs = gp_preprocess_inputs(*inputs) # Compute distance and covariance matrix if x2 is None: x1 = gp_preprocess_inputs(x1) # Only variance vector asked #x = inputs[0] N = np.shape(x1)[0] K = np.ones(N) * amplitude**2 else: (x1,x2) = gp_preprocess_inputs(x1,x2) # Full covariance matrix asked #x1 = inputs[0] #x2 = inputs[1] # Number of inputs x1 N1 = np.shape(x1)[0] # x1 == x2? if x1 is x2: delta = True # Delta covariance # # FIXME: Broadcasting doesn't work with sparse matrices, # so must use scalar multiplication K = gp_cov_delta(N1) * amplitude**2 #K = gp_cov_delta(N1).multiply(amplitude**2) else: delta = False # Number of inputs x2 N2 = np.shape(x2)[0] # Zero covariance if N1 > 0 and N2 > 0: K = sp.csc_matrix((N1,N2)) else: K = np.zeros((N1,N2)) # Gradient w.r.t. amplitude if gradient: # FIXME: Broadcasting doesn't work with sparse matrices, # so must use scalar multiplication gradient_amplitude = K*(2/amplitude) print("noise grad", gradient_amplitude) return (K, (gradient_amplitude,)) else: return K def covfunc_pp2(amplitude, lengthscale, x1, x2, gradient=False): # Make sure that hyperparameters are scalars, not an array objects amplitude = utils.array_to_scalar(amplitude) lengthscale = utils.array_to_scalar(lengthscale) #amplitude = theta[0] #lengthscale = theta[1] ## if gradient: ## gradient_amplitude = gradient[0] ## gradient_lengthscale = gradient[1] ## else: ## gradient_amplitude = [] ## gradient_lengthscale = [] ## inputs = gp_preprocess_inputs(*inputs) # Compute covariance matrix if x2 is None: x1 = gp_preprocess_inputs(x1) # Compute variance vector K = np.ones(np.shape(x)[:-1]) K *= amplitude**2 # Compute gradient w.r.t. lengthscale if gradient: gradient_lengthscale = np.zeros(np.shape(x1)[:-1]) else: (x1,x2) = gp_preprocess_inputs(x1,x2) # Compute (sparse) distance matrix if x1 is x2: x1 = inputs[0] / (lengthscale) x2 = x1 D2 = spdist.pdist(x1, 1.0, form="full", format="csc") else: x1 = inputs[0] / (lengthscale) x2 = inputs[1] / (lengthscale) D2 = spdist.cdist(x1, x2, 1.0, format="csc") r = np.sqrt(D2.data) N1 = np.shape(x1)[0] N2 = np.shape(x2)[0] # Compute the covariances if gradient: (k, dk) = gp_cov_pp2(r, np.shape(x1)[-1], gradient=True) else: k = gp_cov_pp2(r, np.shape(x1)[-1]) k *= amplitude**2 # Compute gradient w.r.t. lengthscale if gradient: if N1 >= 1 and N2 >= 1: dk *= r * (-amplitude**2 / lengthscale) gradient_lengthscale = sp.csc_matrix((dk, D2.indices, D2.indptr), shape=(N1,N2)) else: gradient_lengthscale = np.empty((N1,N2)) # Form sparse covariance matrix if N1 >= 1 and N2 >= 1: ## K = sp.csc_matrix((k, ij), shape=(N1,N2)) K = sp.csc_matrix((k, D2.indices, D2.indptr), shape=(N1,N2)) else: K = np.empty((N1, N2)) #print(K.__class__) # Gradient w.r.t. amplitude if gradient: gradient_amplitude = K * (2 / amplitude) # Return values if gradient: print("pp2 grad", gradient_lengthscale) return (K, (gradient_amplitude, gradient_lengthscale)) else: return K def covfunc_se(amplitude, lengthscale, x1, x2=None, gradient=False): # Make sure that hyperparameters are scalars, not an array objects amplitude = utils.array_to_scalar(amplitude) lengthscale = utils.array_to_scalar(lengthscale) # Compute covariance matrix if x2 is None: x1 = gp_preprocess_inputs(x1) #x = inputs[0] # Compute variance vector N = np.shape(x1)[0] K = np.ones(N) np.multiply(K, amplitude**2, out=K) # Compute gradient w.r.t. lengthscale if gradient: # TODO: Use sparse matrices? gradient_lengthscale = np.zeros(N) else: (x1,x2) = gp_preprocess_inputs(x1,x2) x1 = x1 / (lengthscale) x2 = x2 / (lengthscale) # Compute distance matrix K = squared_distance(x1, x2) # Compute gradient partly if gradient: gradient_lengthscale = np.divide(K, lengthscale) # Compute covariance matrix gp_cov_se(K, overwrite=True) np.multiply(K, amplitude**2, out=K) # Compute gradient w.r.t. lengthscale if gradient: gradient_lengthscale *= K # Gradient w.r.t. amplitude if gradient: gradient_amplitude = K * (2 / amplitude) # Return values if gradient: print("se grad", gradient_amplitude, gradient_lengthscale) return (K, (gradient_amplitude, gradient_lengthscale)) else: return K class CovarianceFunctionWrapper(): def __init__(self, covfunc, *params): # Parse parameter values and their gradients to separate lists self.covfunc = covfunc self.params = list(params) self.gradient_params = list() ## print(params) for ind in range(len(params)): if isinstance(params[ind], tuple): # Parse the value and the list of gradients from the # form: # ([value, ...], [ [grad1, ...], [grad2, ...], ... ]) self.gradient_params.append(params[ind][1]) self.params[ind] = params[ind][0][0] else: # No gradients, parse from the form: # [value, ...] self.gradient_params.append([]) self.params[ind] = params[ind][0] def fixed_covariance_function(self, *inputs, gradient=False): # What if this is called several times?? if gradient: ## grads = [[grad[0] for grad in self.gradient_params[ind]] ## for ind in range(len(self.gradient_params))] ## (K, dK) = self.covfunc(self.params, ## *inputs, ## gradient=self.gradient_params) arguments = tuple(self.params) + tuple(inputs) (K, dK) = self.covfunc(*arguments, gradient=True) ## (K, dK) = self.covfunc(self.params, ## *inputs, ## gradient=grads) DK = [] for ind in range(len(dK)): # Gradient w.r.t. covariance function's ind-th # hyperparameter dk = dK[ind] # Chain rule: Multiply by the gradient of the # hyperparameter w.r.t. parent node and append the # list DK: # DK = [ (dx1_1, callback), ..., (dx1_n, callback) ] for grad in self.gradient_params[ind]: #print(grad[0]) #print(grad[1:]) #print(dk) if sp.issparse(dk): print(dk.shape) print(grad[0].shape) DK += [ [dk.multiply(grad[0])] + grad[1:] ] else: DK += [ [np.multiply(dk,grad[0])] + grad[1:] ] #DK += [ [np.multiply(grad[0], dk)] + grad[1:] ] ## DK += [ (np.multiply(grad, dk),) + grad[1:] ## for grad in self.gradient_params[ind] ] ## for grad in self.gradient_params[ind]: ## DK += ( (np.multiply(grad, dk),) + grad[1:] ) ## DK = [] ## for ind in range(len(dK)): ## for (grad, dk) in zip(self.gradient_params[ind], dK[ind]): ## DK += [ [dk] + grad[1:] ] K = [K] return (K, DK) else: arguments = tuple(self.params) + tuple(inputs) #print(arguments) K = self.covfunc(*arguments, gradient=False) return [K] class CovarianceFunction(ef.Node): def __init__(self, covfunc, *args, **kwargs): self.covfunc = covfunc params = list(args) for i in range(len(args)): # Check constant parameters if utils.is_numeric(args[i]): params[i] = ef.NodeConstant([np.asanyarray(args[i])], dims=[np.shape(args[i])]) # TODO: Parameters could be constant functions? :) ef.Node.__init__(self, *params, dims=[(np.inf, np.inf)], **kwargs) def __call__(self, x1, x2): """ Compute covariance matrix for inputs x1 and x2. """ covfunc = self.message_to_child() return covfunc(x1, x2)[0] def message_to_child(self, gradient=False): params = [parent.message_to_child(gradient=gradient) for parent in self.parents] covfunc = self.get_fixed_covariance_function(*params) return covfunc def get_fixed_covariance_function(self, *params): get_cov_func = CovarianceFunctionWrapper(self.covfunc, *params) return get_cov_func.fixed_covariance_function ## def covariance_function(self, *params): ## # Parse parameter values and their gradients to separate lists ## params = list(params) ## gradient_params = list() ## print(params) ## for ind in range(len(params)): ## if isinstance(params[ind], tuple): ## # Parse the value and the list of gradients from the ## # form: ## # ([value, ...], [ [grad1, ...], [grad2, ...], ... ]) ## gradient_params.append(params[ind][1]) ## params[ind] = params[ind][0][0] ## else: ## # No gradients, parse from the form: ## # [value, ...] ## gradient_params.append([]) ## params[ind] = params[ind][0] ## # This gradient_params changes mysteriously.. ## print('grad_params before') ## if isinstance(self, SquaredExponential): ## print(gradient_params) ## def cov(*inputs, gradient=False): ## if gradient: ## print('grad_params after') ## print(gradient_params) ## grads = [[grad[0] for grad in gradient_params[ind]] ## for ind in range(len(gradient_params))] ## print('CovarianceFunction.cov') ## #if isinstance(self, SquaredExponential): ## #print(self.__class__) ## #print(grads) ## (K, dK) = self.covfunc(params, ## *inputs, ## gradient=grads) ## for ind in range(len(dK)): ## for (grad, dk) in zip(gradient_params[ind], dK[ind]): ## grad[0] = dk ## K = [K] ## dK = [] ## for grad in gradient_params: ## dK += grad ## return (K, dK) ## else: ## K = self.covfunc(params, ## *inputs, ## gradient=False) ## return [K] ## return cov class Sum(CovarianceFunction): def __init__(self, *args, **kwargs): CovarianceFunction.__init__(self, None, *args, **kwargs) def get_fixed_covariance_function(self, *covfunc_parents): def covfunc(*inputs, gradient=False): K_sum = None if gradient: dK_sum = list() for k in covfunc_parents: if gradient: (K, dK) = k(*inputs, gradient=gradient) print("dK in sum", dK) dK_sum += dK #print("dK_sum in sum", dK_sum) else: K = k(*inputs, gradient=gradient) if K_sum is None: K_sum = K[0] else: try: K_sum += K[0] except: # You have to do this way, for instance, if # K_sum is sparse and K[0] is dense. K_sum = K_sum + K[0] if gradient: #print("dK_sum on: ", dK_sum) #print('covsum', dK_sum) return ([K_sum], dK_sum) else: return [K_sum] return covfunc class Delta(CovarianceFunction): def __init__(self, amplitude, **kwargs): CovarianceFunction.__init__(self, covfunc_delta, amplitude, **kwargs) class Zeros(CovarianceFunction): def __init__(self, **kwargs): CovarianceFunction.__init__(self, covfunc_zeros, **kwargs) class SquaredExponential(CovarianceFunction): def __init__(self, amplitude, lengthscale, **kwargs): CovarianceFunction.__init__(self, covfunc_se, amplitude, lengthscale, **kwargs) class PiecewisePolynomial2(CovarianceFunction): def __init__(self, amplitude, lengthscale, **kwargs): CovarianceFunction.__init__(self, covfunc_pp2, amplitude, lengthscale, **kwargs) # TODO: Rename to Blocks or Joint ? class Multiple(CovarianceFunction): def __init__(self, covfuncs, **kwargs): self.d = len(covfuncs) #self.sparse = sparse parents = [covfunc for row in covfuncs for covfunc in row] CovarianceFunction.__init__(self, None, *parents, **kwargs) def get_fixed_covariance_function(self, *covfuncs): def cov(*inputs, gradient=False): # Computes the covariance matrix from blocks which all # have their corresponding covariance functions if len(inputs) < 2: # For one input, return the variance vector instead of # the covariance matrix x1 = inputs[0] # Collect variance vectors from the covariance # functions corresponding to the diagonal blocks K = [covfuncs[i*self.d+i](x1[i], gradient=gradient)[0] for i in range(self.d)] # Form the variance vector from the collected vectors if gradient: raise Exception('Gradient not yet implemented.') else: ## print("in cov multiple") ## for (k,kf) in zip(K,covfuncs): ## print(np.shape(k), k.__class__, kf) #K = np.vstack(K) K = np.concatenate(K) else: x1 = inputs[0] x2 = inputs[1] # Collect the covariance matrix (and possibly # gradients) from each block. #print('cov mat collection begins') K = [[covfuncs[i*self.d+j](x1[i], x2[j], gradient=gradient) for j in range(self.d)] for i in range(self.d)] #print('cov mat collection ends') # Remove matrices that have zero length dimensions? if gradient: K = [[K[i][j] for j in range(self.d) if np.shape(K[i][j][0][0])[1] != 0] for i in range(self.d) if np.shape(K[i][0][0][0])[0] != 0] else: K = [[K[i][j] for j in range(self.d) if np.shape(K[i][j][0])[1] != 0] for i in range(self.d) if np.shape(K[i][0][0])[0] != 0] n_blocks = len(K) #print("nblocks", n_blocks) #print("K", K) # Check whether all blocks are sparse is_sparse = True for i in range(n_blocks): for j in range(n_blocks): if gradient: A = K[i][j][0][0] else: A = K[i][j][0] if not sp.issparse(A): is_sparse = False if gradient: ## Compute the covariance matrix and the gradients # Create block matrices of zeros. This helps in # computing the gradient. if is_sparse: # Empty sparse matrices. Some weird stuff here # because sparse matrices can't have zero # length dimensions. Z = [[sp.csc_matrix(np.shape(K[i][j][0][0])) for j in range(n_blocks)] for i in range(n_blocks)] else: # Empty dense matrices Z = [[np.zeros(np.shape(K[i][j][0][0])) for j in range(n_blocks)] for i in range(n_blocks)] ## for j in range(self.d)] ## for i in range(self.d)] # Compute gradients block by block dK = list() for i in range(n_blocks): for j in range(n_blocks): # Store the zero block z_old = Z[i][j] # Go through the gradients for the (i,j) # block for dk in K[i][j][1]: # Keep other blocks at zero and set # the gradient to (i,j) block. Form # the matrix from blocks if is_sparse: Z[i][j] = dk[0] dk[0] = sp.bmat(Z).tocsc() else: if sp.issparse(dk[0]): Z[i][j] = dk[0].toarray() else: Z[i][j] = dk[0] #print("Z on:", Z) dk[0] = np.asarray(np.bmat(Z)) # Append the computed gradient matrix # to the list of gradients dK.append(dk) # Restore the zero block Z[i][j] = z_old ## Compute the covariance matrix but not the ## gradients if is_sparse: # Form the full sparse covariance matrix from # blocks. Ignore blocks having a zero-length # axis because sparse matrices consider zero # length as an invalid shape (BUG IN SCIPY?). K = [[K[i][j][0][0] for j in range(n_blocks)] for i in range(n_blocks)] K = sp.bmat(K).tocsc() else: # Form the full dense covariance matrix from # blocks. Transform sparse blocks to dense # blocks. K = [[K[i][j][0][0] if not sp.issparse(K[i][j][0][0]) else K[i][j][0][0].toarray() for j in range(n_blocks)] for i in range(n_blocks)] K = np.asarray(np.bmat(K)) else: ## Compute the covariance matrix but not the ## gradients if is_sparse: # Form the full sparse covariance matrix from # blocks. Ignore blocks having a zero-length # axis because sparse matrices consider zero # length as an invalid shape (BUG IN SCIPY?). K = [[K[i][j][0] for j in range(n_blocks)] for i in range(n_blocks)] K = sp.bmat(K).tocsc() else: # Form the full dense covariance matrix from # blocks. Transform sparse blocks to dense # blocks. K = [[K[i][j][0] if not sp.issparse(K[i][j][0]) else K[i][j][0].toarray() for j in range(n_blocks)] for i in range(n_blocks)] K = np.asarray(np.bmat(K)) if gradient: return ([K], dK) else: return [K] return cov bayespy-0.1/bayespy/inference/vmp/nodes/variable.py0000664000175000017500000004333512174013072024236 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011-2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np from bayespy.utils import utils from .node import Node raise DeprecationWarning("Don't use this module. Use stochastic or expfamily " "instead.") # nodes/ # gp/ # # nodes.random.Gaussian # nodes.random.Wishart # nodes.random.Mixture # nodes.deterministic.Dot # nodes.deterministic.Constant # or # nodes.variable.Gaussian # nodes.variable.Wishart # nodes.variable.Dot # nodes.variable.Constant # nodes.gp.GaussianProcess # nodes.gp.Constant # MAP/ML: # X = MAP(prior=Gaussian(mu,Cov)) # X = ML() # or: # X = Delta(prior=Gaussian(mu,Cov)) class Variable(Node): # Overwrite this ndims = None ndims_parents = None parameter_distributions = None @classmethod def compute_logpdf(cls, u, phi, g, f): """ Compute E[log p(X)] given E[u], E[phi], E[g] and E[f]. Does not sum over plates.""" # TODO/FIXME: Should I take into account what is latent or # observed, or what is even totally ignored (by the mask). L = g + f for (phi_i, u_i, ndims_i) in zip(phi, u, cls.ndims): #for (phi_i, u_i, len_dims_i) in zip(phi, u, len_dims): # Axes to sum (dimensions of the variable, not the plates) axis_sum = tuple(range(-ndims_i,0)) #axis_sum = tuple(range(-len_dims_i,0)) # Compute the term # TODO/FIXME: Use einsum! L = L + np.sum(phi_i * u_i, axis=axis_sum) return L @staticmethod def compute_fixed_parameter_moments(*args): """ Compute the moments of the distribution parameters for fixed values.""" raise NotImplementedError() @staticmethod def compute_phi_from_parents(u_parents): """ Compute the natural parameters using parents' moments. Compute E[phi] over q(parents) """ raise NotImplementedError() @staticmethod def compute_g_from_parents(u_parents): """ Compute E[g(phi)] over q(parents) """ raise NotImplementedError() @staticmethod def compute_u_and_g(phi, mask=True): """ Compute E[u] and g(phi) for given phi """ raise NotImplementedError() @staticmethod def compute_fixed_u_and_f(x): """ Compute u(x) and f(x) for given x. """ raise NotImplementedError() @staticmethod def compute_message(index, u, u_parents): """ . """ raise NotImplementedError() @staticmethod def compute_dims(*parents): """ Compute the dimensions of phi and u. """ raise NotImplementedError() @staticmethod def compute_dims_from_values(x): """ Compute the dimensions of phi and u. """ raise NotImplementedError() def __init__(self, *args, initialize=True, **kwargs): super().__init__(*args, dims=self.compute_dims(*args), **kwargs) # Natural parameters and moments. Note that the size of the # array self.u or self.phi doesn't actually need to equal # self.dims+self.plates, it will be broadcasted automatically # (although this might be extremely rare: it would mean that # there are identical posterior distributions over # plates). Anyway, for this reason (and some others), we need # to explicityly have the dimensionalities dims and plates? Do # we need dims? Yes, because you need to be able to check what # plates are missing. #self.phi = [np.array([]) for i in range(len(self.dims))] #self.u = [np.array([]) for i in range(len(self.dims))] #self.phi = [np.array(0.0) for i in range(len(self.dims))] #self.u = [np.array(0.0) for i in range(len(self.dims))] axes = len(self.plates)*(1,) self.phi = [utils.nans(axes+dim) for dim in self.dims] self.u = [utils.nans(axes+dim) for dim in self.dims] # self.u = [np.array(0.0) for i in range(len(self.dims))] # Terms for the lower bound (G for latent and F for observed) self.g = 0 self.f = 0 # Not observed self.observed = False # By default, ignore all plates self.mask = False if initialize: self.initialize_from_prior() def get_vb_term(self): return {self.lower_bound_contribution} def get_message(self, index, u_parents): """ Get the message and the mask for a parent. Parameters: ----------- index : int Index of the parent requesting the message. u_parents : list of list of ndarrays List of parents' moments. Returns: -------- m : list of ndarrays Message as a list of arrays. mask : boolean ndarray Mask telling which plates should be taken into account. """ return (list(self.compute_message(index, self.u, u_parents)), self.mask) def initialize_from_prior(self): if not np.all(self.observed): # Messages from parents #u_parents = [parent.message_to_child() for parent in self.parents] u_parents = self.moments_from_parents() # Update natural parameters using parents self.update_phi_from_parents(u_parents) # Update moments (u, g) = self.compute_u_and_g(self.phi, mask=True) self.update_u_and_g(u, g, mask=True) def initialize_from_parameters(self, *args): # Get the moments of the parameters if they were fixed to the # given values. #u_parents = self.compute_fixed_parameter_moments(*args) u_parents = list() for (ind, x) in enumerate(args): u = self.parameter_distributions[ind].compute_fixed_moments(x) #u = self.parents[ind].compute_fixed_moments(x) #(u, _) = self.parents[ind].compute_fixed_u_and_f(x) u_parents.append(u) # Update natural parameters self.update_phi_from_parents(u_parents) # Update moments # TODO/FIXME: Use the mask of observations! (u, g) = self.compute_u_and_g(self.phi, mask=True) self.update_u_and_g(u, g, mask=True) # TODO: Initialization where you could give the distribution # values in standard parameterization! def initialize_from_value(self, x): # Update moments from value if np.shape(x) != self.plates + self.get_shape_of_value(): raise ValueError("Invalid shape of the value for initialization.") (u, f) = self.compute_fixed_u_and_f(x) self.update_u_and_g(u, np.nan, mask=True) ## def initialize_from_random(self): ## self.initialize_from_prior() ## self.initialize_from_value(self.random()) def update(self): if not np.all(self.observed): # Messages from parents u_parents = [parent.message_to_child() for parent in self.parents] # Update natural parameters using parents self.update_phi_from_parents(u_parents) # Update natural parameters using children (just add the # messages to phi) for (child,index) in self.children: # TODO/FIXME: These m are nans.. (m, mask) = child.message_to_parent(index) # Combine masks # # TODO: Maybe you would like to compute a new mask # at every update? self.mask = np.logical_or(self.mask,mask) for i in range(len(self.phi)): if m[i] is not None: ## try: ## # Try exploiting broadcasting rules ## # ## # TODO/FIXME: This has the problem that if phi ## # is updated from parents such that phi is a ## # view into parent's moments thus modifying ## # phi would modify parents moments.. Maybe one ## # should check that nobody else is viewing ## # phi? ## self.phi[i] += m[i] ## except ValueError: ## self.phi[i] = self.phi[i] + m[i] # TODO/FIXME: You should take into account the # mask when adding the message!!!!??? self.phi[i] = self.phi[i] + m[i] self.update_from_phi() def _update_from_phi(self): """ Update moments and cgf based on current phi. """ # Mask for plates to update (i.e., unobserved plates) update_mask = np.logical_not(self.observed) # Compute u and g (u, g) = self.compute_u_and_g(self.phi, mask=update_mask) # Update moments self.update_u_and_g(u, g, mask=update_mask) def phi_from_parents(self, gradient=False): # Messages from parents u_parents = [parent.message_to_child(gradient=gradient) for parent in self.parents] # Compute and return phi return self.compute_phi_from_parents(u_parents, gradient=gradient) def update_phi_from_parents(self, u_parents): # This makes correct broadcasting self.phi = self.compute_phi_from_parents(u_parents) self.phi = list(self.phi) # Make sure phi has the correct number of axes. It makes life # a bit easier elsewhere. for i in range(len(self.phi)): axes = len(self.plates) + self.ndims[i] - np.ndim(self.phi[i]) if axes > 0: # Add axes self.phi[i] = utils.add_leading_axes(self.phi[i], axes) elif axes < 0: # Remove extra leading axes first = -(len(self.plates)+self.ndims[i]) sh = np.shape(self.phi[i])[first:] self.phi[i] = np.reshape(self.phi[i], sh) ## for i in range(len(self.phi)): ## self.phi[i].fill(0) ## try: ## # Try exploiting broadcasting rules ## self.phi[i] += phi[i] ## except ValueError: ## self.phi[i] = self.phi[i] + phi[i] def get_moments(self): return self.u def get_children_vb_bound(self): raise NotImplementedError("Not implemented for " + str(self.__class__)) def start_optimization(self): raise NotImplementedError("Not implemented for " + str(self.__class__)) def stop_optimization(self): raise NotImplementedError("Not implemented for " + str(self.__class__)) def message(self, index, u_parents): raise NotImplementedError("Not implemented for " + str(self.__class__)) def update_u(self, u, mask=True): # Store the computed moments u but do not change moments for # observations, i.e., utilize the mask. for ind in range(len(u)): # Add axes to the mask for the variable dimensions (mask # contains only axes for the plates). u_mask = utils.add_trailing_axes(mask, self.ndims[ind]) # Enlarge self.u[ind] as necessary so that it can store the # broadcasted result. sh = utils.broadcasted_shape_from_arrays(self.u[ind], u[ind], u_mask) self.u[ind] = utils.repeat_to_shape(self.u[ind], sh) # Use mask to update only unobserved plates and keep the # observed as before np.copyto(self.u[ind], u[ind], where=u_mask) # Make sure u has the correct number of dimensions: shape = self.get_shape(ind) ndim = len(shape) ndim_u = np.ndim(self.u[ind]) if ndim > ndim_u: self.u[ind] = utils.add_leading_axes(u[ind], ndim - ndim_u) elif ndim < ndim_u: raise Exception("The size of the variable's moments array " "is larger than it should be based on the " "plates and dimension information. Check " "that you have provided plates properly.") def update_u_and_g(self, u, g, mask=True): self.update_u(u, mask=mask) # TODO/FIXME: Apply mask to g too!! self.g = g ## # Store the computed moments u but do not change moments for ## # observations, i.e., utilize the mask. ## for ind in range(len(u)): ## # Add axes to the mask for the variable dimensions (mask ## # contains only axes for the plates). ## u_mask = utils.add_trailing_axes(mask, self.ndims[ind]) ## # Enlarge self.u[ind] as necessary so that it can store the ## # broadcasted result. ## sh = utils.broadcasted_shape_from_arrays(self.u[ind], u[ind], u_mask) ## self.u[ind] = utils.repeat_to_shape(self.u[ind], sh) ## # Use mask to update only unobserved plates and keep the ## # observed as before ## np.copyto(self.u[ind], ## u[ind], ## where=u_mask) ## # TODO/FIXME: Apply mask to g too!! ## self.g = g ## # Make sure u has the correct number of dimensions: ## shape = self.get_shape(ind) ## ndim = len(shape) ## ndim_u = np.ndim(self.u[ind]) ## if ndim > ndim_u: ## self.u[ind] = utils.add_leading_axes(u[ind], ndim - ndim_u) ## elif ndim < np.ndim(self.u[ind]): ## raise Exception("Weird, this shouldn't happen.. :)") def lower_bound_contribution(self, gradient=False): # Compute E[ log p(X|parents) - log q(X) ] over q(X)q(parents) # Messages from parents u_parents = [parent.message_to_child() for parent in self.parents] phi = self.compute_phi_from_parents(u_parents) # G from parents L = self.compute_g_from_parents(u_parents) # L = g # G for unobserved variables (ignored variables are handled # properly automatically) latent_mask = np.logical_not(self.observed) #latent_mask = np.logical_and(self.mask, np.logical_not(self.observed)) # F for observed, G for latent L = L + np.where(self.observed, self.f, -self.g) for (phi_p, phi_q, u_q, dims) in zip(phi, self.phi, self.u, self.dims): # Form a mask which puts observed variables to zero and # broadcasts properly latent_mask_i = utils.add_axes(latent_mask, len(self.plates) - np.ndim(latent_mask), len(dims)) axis_sum = tuple(range(-len(dims),0)) # Compute the term phi_q = np.where(latent_mask_i, phi_q, 0) # TODO/FIXME: Use einsum here? Z = np.sum((phi_p-phi_q) * u_q, axis=axis_sum) L = L + Z return (np.sum(np.where(self.mask, L, 0)) * self.plate_multiplier(self.plates, np.shape(L), np.shape(self.mask))) #return L def fix_u_and_f(self, u, f, mask=True): for (i,v) in enumerate(u): # This is what the dimensionality "should" be s = self.plates + self.dims[i] t = np.shape(v) if s != t: msg = "Dimensionality of the observations incorrect." msg += "\nShape of input: " + str(t) msg += "\nExpected shape: " + str(s) msg += "\nCheck plates." raise Exception(msg) #self.phi[i] = 0 self.update_u(u, mask=mask) ## for i in range(len(self.u)): ## obs_mask = utils.add_axes(mask, ## len(self.plates) - np.ndim(mask), ## len(self.dims[i])) ## # TODO/FIXME: Use copyto! ## self.u[i] = (obs_mask * u[i] ## + np.logical_not(obs_mask) * self.u[i]) # TODO/FIXME: Use the mask? self.f = f def observe(self, x, mask=True): (u, f) = self.compute_fixed_u_and_f(x) self.fix_u_and_f(u, f, mask=mask) # Observed nodes should not be ignored self.observed = mask self.mask = np.logical_or(self.mask, self.observed) def integrated_logpdf_from_parents(self, index): r""" Approximates the posterior predictive pdf. :math:`\int p(x|parents) q(parents) dparents in log-scale as \int q(parents_i) exp( \int q(parents_\i) \log p(x|parents) dparents_\i ) dparents_i`. """ raise NotImplementedError() bayespy-0.1/bayespy/inference/vmp/nodes/gaussian_markov_chain.py0000664000175000017500000005372712174013072027012 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2012-2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### """ This module contains VMP nodes for Gaussian Markov chains. """ import numpy as np import scipy import matplotlib.pyplot as plt import bayespy.plot.plotting as bpplt from bayespy import utils from bayespy.utils.linalg import dot, mvdot from .node import Node, message_sum_multiply from .deterministic import Deterministic from .expfamily import ExponentialFamily from .constant import Constant, ConstantNumeric from .gaussian import Gaussian from .wishart import Wishart from .gamma import Gamma # TODO/FIXME: The plates of masks are not handled properly! Try having # a plate of GMCs and then the message mask to A or v.. class GaussianMarkovChain(ExponentialFamily): r""" VMP node for Gaussian Markov chain. Parents are: `mu` is the mean of x0 (Gaussian) `Lambda` is the precision of x0 (Wishart) `A` is the dynamic matrix (Gaussian) `v` is the diagonal precision of the innovation (Gamma) An additional dummy parent is created: 'N' is the number of time instances Output is Gaussian variables. Time dimension is over the last plate. Hmm.. The number of time instances is one more than the plates in A and V. Input N -> Output N+1. .. bayesnet:: \node[latent] (x1) {$\mathbf{x}_1$}; \node[latent, right=of x1] (x2) {$\mathbf{x}_2$}; \node[right=of x2] (dots) {$\cdots$}; \node[latent, right=of dots] (xn) {$\mathbf{x}_n$}; \edge {x1}{x2}; \edge {x2}{dots}; \edge {dots}{xn}; See also -------- bayespy.inference.vmp.nodes.gaussian.Gaussian bayespy.inference.vmp.nodes.wishart.Wishart """ # phi[0] is (N,D), phi[1] is (N,D,D), phi[2] is (N-1,D,D) ndims = (2, 3, 3) ndims_parents = [(1, 2), (2, 0), (1, 2), (0, 0)] # Observations are a set of vectors (thus 2-D matrix): ndim_observations = 2 def __init__(self, mu, Lambda, A, v, n=None, **kwargs): """ `mu` is the mean of x_0 `Lambda` is the precision of x_0 `A` is the dynamic matrix `v` is the diagonal precision of the innovation """ self.parameter_distributions = (Gaussian, Wishart, Gaussian, Gamma) # Check for constant mu if utils.utils.is_numeric(mu): mu = Constant(Gaussian)(np.atleast_1d(mu)) # Check for constant Lambda if utils.utils.is_numeric(Lambda): Lambda = Constant(Wishart)(np.atleast_2d(Lambda)) # Check for constant A if utils.utils.is_numeric(A): A = Constant(Gaussian)(np.atleast_2d(A)) # Check for constant V if utils.utils.is_numeric(v): v = Constant(Gamma)(np.atleast_1d(v)) # A dummy wrapper for the number of time instances. n_A = 1 if len(A.plates) >= 2: n_A = A.plates[-2] n_v = 1 if len(v.plates) >= 2: n_v = v.plates[-2] if n_v != n_A and n_v != 1 and n_A != 1: raise Exception("Plates of A and v are giving different number of time instances") n_A = max(n_v, n_A) if n is None: if n_A == 1: raise Exception("""The number of time instances could not be determined automatically. Give the number of time instances.""") n = n_A + 1 elif n_A != 1 and n_A+1 != n: raise Exception("The number of time instances must match " "the number of last plates of parents: " "%d != %d+1" % (n, n_A)) N = ConstantNumeric(n, 0) # Check that the dimensions and plates of the parents are # consistent with the dimensionality of the process. The # plates are checked in the super constructor. #if mu.dims[0][0] != # Construct super().__init__(mu, Lambda, A, v, N, **kwargs) @staticmethod def compute_fixed_moments(x): raise NotImplementedError() @staticmethod def _compute_phi_from_parents(u_mu, u_Lambda, u_A, u_v, u_N): """ Compute the natural parameters using parents' moments. Parameters ---------- u_parents : list of list of arrays List of parents' lists of moments. Returns ------- phi : list of arrays Natural parameters. dims : tuple Shape of the variable part of phi. """ # Dimensionality of the Gaussian states D = np.shape(u_mu[0])[-1] # Number of time instances in the process N = u_N[0] # TODO/FIXME: Take into account plates! phi0 = np.zeros((N,D)) phi1 = np.zeros((N,D,D)) phi2 = np.zeros((N-1,D,D)) # Parameters for x0 mu = u_mu[0] # (..., D) Lambda = u_Lambda[0] # (..., D, D) phi0[...,0,:] = np.dot(Lambda, mu) phi1[...,0,:,:] = Lambda # Helpful variables (show shapes in comments) A = u_A[0] # (..., N-1, D, D) AA = u_A[1] # (..., N-1, D, D, D) v = u_v[0] # (..., N-1, D) # DEBUGGING: #AA = A[...,np.newaxis] * A[...,np.newaxis,:] #CovA = AA - A[...,np.newaxis] * A[...,np.newaxis,:] #print('gmc.compphi', CovA) # Diagonal blocks: -0.5 * (V_i + A_{i+1}' * V_{i+1} * A_{i+1}) phi1[..., 1:, :, :] = v[...,np.newaxis]*np.identity(D) phi1[..., :-1, :, :] += np.einsum('...kij,...k->...ij', AA, v) phi1 *= -0.5 # Super-diagonal blocks: 0.5 * A.T * V # However, don't multiply by 0.5 because there are both super- and # sub-diagonal blocks (sum them together) phi2[..., :, :, :] = np.einsum('...ji,...j->...ij', A, v) return (phi0, phi1, phi2) @staticmethod def _compute_cgf_from_parents(*u_parents): """ Compute CGF using the moments of the parents. """ u_mu = u_parents[0] u_Lambda = u_parents[1] u_A = u_parents[2] u_v = u_parents[3] u_N = u_parents[4] N = u_N[0] mumu = u_mu[1] Lambda = u_Lambda[0] logdet_Lambda = u_Lambda[1] logdet_v = u_v[1] g0 = -0.5 * np.einsum('...ij,...ij->...', mumu, Lambda) g1 = 0.5 * logdet_Lambda if np.ndim(logdet_v) == 1: g1 = g1 + 0.5 * (N-1) * np.sum(logdet_v, axis=-1) elif np.shape(logdet_v)[-2] == 1: g1 = g1 + 0.5 * (N-1) * np.sum(logdet_v, axis=(-1,-2)) else: g1 = g1 + 0.5 * np.sum(logdet_v, axis=(-1,-2)) return g0 + g1 @staticmethod def _compute_moments_and_cgf(phi, mask=True): """ Compute the moments and the cumulant-generating function. Parameters ---------- phi Returns ------- u g """ # Solve the Kalman filtering and smoothing problem y = phi[0] A = -2*phi[1] # Don't multiply phi[2] by two because it is a sum of the super- and # sub-diagonal blocks so we would need to divide by two anyway. B = -phi[2] (CovXnXn, CovXpXn, Xn, ldet) = utils.utils.block_banded_solve(A, B, y) ## # DEBUGGING ## if np.shape(phi[0])[-1] == 1: ## C = utils.utils.block_banded(A, B) ## L = np.linalg.cholesky(C) ## N = np.shape(C)[-1] ## print('C', C) ## print('L', L)#utils.linalg.chol(C)) ## print('gmc.compmom', np.linalg.cond(C)) ## print('Use inv') ## #invC = np.linalg.inv(C) ## invC = scipy.linalg.cho_solve((L,True), np.identity(N)) ## CovXnXn = np.diag(invC)[...,np.newaxis,np.newaxis] ## CovXpXn = np.diag(invC, k=1)[...,np.newaxis,np.newaxis] # Compute moments u0 = Xn u1 = CovXnXn + Xn[...,:,np.newaxis] * Xn[...,np.newaxis,:] u2 = CovXpXn + Xn[...,:-1,:,np.newaxis] * Xn[...,1:,np.newaxis,:] u = [u0, u1, u2] # Compute cumulant-generating function g = -0.5 * np.einsum('...ij,...ij', u[0], phi[0]) + 0.5*ldet return (u, g) @staticmethod def _compute_fixed_moments_and_f(x, mask=True): """ Compute u(x) and f(x) for given x. """ u0 = x u1 = x[...,:,np.newaxis] * x[...,np.newaxis,:] u2 = x[...,:-1,:,np.newaxis] * x[...,1:,np.newaxis,:] u = [u0, u1, u2] f = -0.5 * np.shape(x)[-2] * np.shape(x)[-1] * np.log(2*np.pi) return (u, f) @staticmethod def _compute_mask_to_parent(index, mask): if index == 0: # mu return mask elif index == 1: # Lambda return mask elif index == 2: # A return mask[...,np.newaxis,np.newaxis] elif index == 3: # v return mask[...,np.newaxis,np.newaxis] elif index == 4: # N return mask @staticmethod def _compute_message_to_parent(parent, index, u, u_mu, u_Lambda, u_A, u_v, u_N): """ Compute a message to a parent. Parameters: ----------- index : int Index of the parent requesting the message. u : list of ndarrays Moments of this node. u_mu : list of ndarrays Moments of parent `mu`. u_Lambda : list of ndarrays Moments of parent `Lambda`. u_A : list of ndarrays Moments of parent `A`. u_v : list of ndarrays Moments of parent `v`. u_N : list of ndarrays Moments of parent `N`. """ if index == 0: # mu raise NotImplementedError() elif index == 1: # Lambda raise NotImplementedError() elif index == 2: # A XnXn = u[1] XpXn = u[2] v = u_v[0] m0 = v[...,np.newaxis] * XpXn.swapaxes(-1,-2) # The following message matrix could be huge, so let's use a help # function which computes sum(v*XnXn) without computing the huge # v*XnXn explicitly. m1 = -0.5 * message_sum_multiply(parent.plates, parent.dims[1], v[...,np.newaxis,np.newaxis], XnXn[..., :-1, np.newaxis, :, :]) #m1 = -0.5 * v[...,np.newaxis,np.newaxis] * XnXn[..., :-1, np.newaxis, :, :] elif index == 3: # v XnXn = u[1] # (...,N,D,D) XpXn = u[2] # (...,N-1,D,D) A = u_A[0] # (...,N-1,D,D) AA = u_A[1] # (...,N-1,D,D,D) m0 = (- 0.5*np.einsum('...ii->...i', XnXn[...,1:,:,:]) + np.einsum('...ik,...ki->...i', A, XpXn) - 0.5*np.einsum('...ikl,...kl->...i', AA, XnXn[...,:-1,:,:])) m1 = 0.5 elif index == 4: # N raise NotImplementedError() return [m0, m1] @staticmethod def compute_dims(mu, Lambda, A, v, N): """ Compute the dimensions of phi and u. The plates and dimensions of the parents should be: mu: (...) and D-dimensional Lambda: (...) and D-dimensional A: (...,1,D) or (...,N-1,D) and D-dimensional v: (...,1,D) or (...,N-1,D) and 0-dimensional N: () and 0-dimensional (dummy parent) Check that the dimensionalities of the parents are proper. For instance, A should be a collection of DxD matrices, thus the dimensionality and the last plate should both equal D. Similarly, `v` should be a collection of diagonal innovation matrix elements, thus the last plate should equal D. """ D = mu.dims[0][0] M = N.get_moments()[0] # Check mu if mu.dims != ( (D,), (D,D) ): raise Exception("First parent has wrong dimensionality") # Check Lambda if Lambda.dims != ( (D,D), () ): raise Exception("Second parent has wrong dimensionality") # Check A if A.dims != ( (D,), (D,D) ): raise Exception("Third parent has wrong dimensionality") if len(A.plates) == 0 or A.plates[-1] != D: raise Exception("Third parent should have a last plate " "equal to the dimensionality of the " "system.") if (len(A.plates) >= 2 and A.plates[-2] != 1 and A.plates[-2] != M-1): raise ValueError("The second last plate of the third " "parent should have length equal to one or " "N-1, where N is the number of time " "instances.") # Check v if v.dims != ( (), () ): raise Exception("Fourth parent has wrong dimensionality") if len(v.plates) == 0 or v.plates[-1] != D: raise Exception("Fourth parent should have a last plate " "equal to the dimensionality of the " "system.") if (len(v.plates) >= 2 and v.plates[-2] != 1 and v.plates[-2] != M-1): raise ValueError("The second last plate of the fourth " "parent should have length equal to one or " "N-1 where N is the number of time " "instances.") return ( (M,D), (M,D,D), (M-1,D,D) ) @staticmethod def compute_dims_from_values(x): """ Compute the dimensions of phi and u. """ raise NotImplementedError() def get_shape_of_value(self): # Dimensionality of a realization return self.dims[0] def _plates_to_parent(self, index): """ Computes the plates of this node with respect to a parent. If this node has plates (...), the latent dimensionality is D and the number of time instances is N, the plates with respect to the parents are: mu: (...) Lambda: (...) A: (...,N-1,D) v: (...,N-1,D) N: () Parameters: ----------- index : int The index of the parent node to use. """ N = self.dims[0][0] D = self.dims[0][1] if index == 0: # mu return self.plates elif index == 1: # Lambda return self.plates elif index == 2: # A return self.plates + (N-1,D) #raise NotImplementedError() elif index == 3: # v return self.plates + (N-1,D) elif index == 4: # N return () raise ValueError("Invalid parent index.") #raise NotImplementedError() def _plates_from_parent(self, index): """ Compute the plates using information of a parent node. If the plates of the parents are: mu: (...) Lambda: (...) A: (...,N-1,D) v: (...,N-1,D) N: () the resulting plates of this node are (...) Parameters ---------- index : int Index of the parent to use. """ if index == 0: # mu return self.parents[0].plates elif index == 1: # Lambda return self.parents[1].plates elif index == 2: # A return self.parents[2].plates[:-2] elif index == 3: # v return self.parents[3].plates[:-2] elif index == 4: # N return () raise ValueError("Invalid parent index.") def random(self): raise NotImplementedError() def show(self): raise NotImplementedError() def as_gaussian(self): return _MarkovChainToGaussian(self, name=self.name+" as Gaussian") def plot(self): x = self.u[0] xx = self.u[1] var = np.einsum('...ii->...i', xx) - x**2 plt.figure() D = np.shape(x)[-1] for d in range(D): plt.subplot(D,1,d+1) bpplt.errorplot(y=x[...,d], error=2*np.sqrt(var[...,d])) def rotate(self, R, inv=None, logdet=None): if inv is not None: invR = inv else: invR = np.linalg.inv(R) if logdet is not None: logdetR = logdet else: logdetR = np.linalg.slogdet(R)[1] # It would be more efficient and simpler, if you just rotated the # moments and didn't touch phi. However, then you would need to call # update() before lower_bound_contribution. This is more error-safe. #print('rotate debug in gmc', self.phi[0]) #print(R, invR, np.shape(self.phi[0])) # Transform parameters self.phi[0] = mvdot(invR.T, self.phi[0]) self.phi[1] = dot(invR.T, self.phi[1], invR) self.phi[2] = dot(invR.T, self.phi[2], invR) N = self.dims[0][0] if False: #print(self.phi[0]) self._update_moments_and_cgf() else: # Transform moments and g u0 = mvdot(R, self.u[0]) u1 = dot(R, self.u[1], R.T) u2 = dot(R, self.u[2], R.T) self.u = [u0, u1, u2] self.g -= N*logdetR #print("DEBUG", np.sum(u1[1:], axis=0)) class _MarkovChainToGaussian(Deterministic): """ Transform a Gaussian Markov chain node into a Gaussian node. This node is deterministic. """ def __init__(self, X, **kwargs): # Check for constant n if utils.utils.is_numeric(X): X = Constant(GaussianMarkovChain)(X) # Make the time dimension a plate dimension... #plates = X.plates + (X.dims[0][0],) # ... and remove it from the variable dimensions dims = ( X.dims[0][-1:], X.dims[1][-2:] ) super().__init__(X, #plates=plates, dims=dims, **kwargs) def _plates_to_parent(self, index): """ Return the number of plates to the parent node. Normally, the parent sees the same number of plates as the node itself. However, now that one of the variable dimensions of the parents corresponds to a plate in this node, it is necessary to fix it here: the last plate is ignored when calculating plates with respect to the parent. Parent: Plates = (...) Dims = (N, ...) This node: Plates = (..., N) Dims = (...) """ return self.plates[:-1] def _plates_from_parent(self, index): # Sub-classes may want to overwrite this if they manipulate plates if index != 0: raise ValueError("Invalid parent index.") parent = self.parents[0] plates = parent.plates + (parent.dims[0][0],) return plates def _compute_moments(self, u): """ Transform the moments of a GMC to moments of a Gaussian. There is no need to worry about the plates and variable dimensions because the child node is free to interpret the axes as it pleases. However, the Gaussian moments contain only and but not , thus the last moment is discarded. """ # Get the moments from the parent Gaussian Markov Chain #u = self.parents[0].get_moments() #message_to_child() # Send only moments and but not return u[:2] @staticmethod def _compute_mask_to_parent(index, mask): # Remove the last axis of the mask if np.ndim(mask) >= 1: mask = np.any(mask, axis=-1) return mask @staticmethod def _compute_message_to_parent(index, m_children, *u_parents): """ Transform a message to a Gaussian into a message to a GMC. The messages to a Gaussian are almost correct, there are only two minor things to be done: 1) The last plate is changed into a variable/time dimension. Because a message mask is applied for plates only, the last axis of the mask must be applied to the message because the last plate is changed to a variable/time dimension. 2) Because the message does not contain part, we'll put the last/third message to None meaning that it is empty. Parameters: ----------- index : int Index of the parent requesting the message. u_parents : list of list of ndarrays List of parents' moments. Returns: -------- m : list of ndarrays Message as a list of arrays. mask : boolean ndarray Mask telling which plates should be taken into account. """ #(m, mask) = self.message_from_children() # Add the third empty message return [m_children[0], m_children[1], None] #mask = self._compute_mask #return (m, mask) bayespy-0.1/bayespy/inference/vmp/nodes/node2.py0000664000175000017500000003471512174013072023462 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### """ This module contains a sketch of a new implementation of the framework. """ class Node(): """ Base class for all nodes. dims plates parents children name """ def __init__(self, *parents, dims=None, plates=None, name=""): if dims is None: raise Exception("You need to specify the dimensionality of the " "distribution for class %s" % str(self.__class__)) self.dims = dims self.name = name # Parents self.parents = parents # Inform parent nodes for (index,parent) in enumerate(self.parents): if parent: parent._add_child(self, index) # Check plates parent_plates = [self.plates_from_parent(index) for index in range(len(self.parents))] if plates is None: # By default, use the minimum number of plates determined # from the parent nodes try: self.plates = utils.broadcasted_shape(*parent_plates) except ValueError: raise ValueError("The plates of the parents do not broadcast.") else: # Use custom plates self.plates = plates # TODO/FIXME: Check that these plates are consistent with parents. # This is not a good test yet.. You need to check that the # parent_plates are a subset of plates. try: plates_broadcasted = utils.broadcasted_shape(plates, *parent_plates) except ValueError: raise ValueError("The given plates and the plates of the " "parents do not broadcast.") # Children self.children = list() @staticmethod def _compute_dims_from_parents(*parents): """ Compute the dimensions of phi and u. """ raise NotImplementedError() @staticmethod def _compute_dims_from_values(x): """ Compute the dimensions of phi and u. """ raise NotImplementedError() def _add_child(self, child, index): """ Add a child node. Parameters ---------- child : node index : int The parent index of this node for the child node. The child node recognizes its parents by their index number. """ self.children.append((child, index)) def _set_mask(self, mask): # Sub-classes may want to overwrite this pass def _update_mask(self): # Combine masks from children mask = False for (child, index) in self.children: mask = np.logical_or(mask, child._mask_to_parent(index)) # Set the mask of this node self._set_mask(mask) # Tell parents to update their masks for parent in self.parents: parent._update_mask() def _mask_to_parent(self, index): # Sub-classes should implement this raise NotImplementedError() def _message_to_child(self): return self.get_moments() def _message_to_parent(self, index): # Compute the message, check plates, apply mask and sum over some plates if index >= len(self.parents): raise ValueError("Parent index larger than the number of parents") # Compute the message m = self._get_message_to_parent(index) my_mask = self._get_mask() # The parent we're sending the message to parent = self.parents[index] # Compact the message to a proper shape for i in range(len(m)): # Empty messages are given as None. We can ignore those. if m[i] is not None: # Ignorations (add extra axes to broadcast properly). # This sends zero messages to parent from such # variables we are ignoring in this node. This is # useful for handling missing data. # Apply the mask to the message # Sum the dimensions of the message matrix to match # the dimensionality of the parents natural # parameterization (as there may be less plates for # parents) # Extend the plate mask to include unit axes for variable # dimensions. shape_mask = np.shape(my_mask) + (1,) * len(parent.dims[i]) my_mask2 = np.reshape(my_mask, shape_mask) # Apply the mask to the message m[i] = np.where(my_mask2, m[i], 0) shape_m = np.shape(m[i]) # If some dimensions of the message matrix were # singleton although the corresponding dimension # of the parents natural parameterization is not, # multiply the message (i.e., broadcasting) # Plates in the message dim_parent = len(self.parents[index].dims[i]) if dim_parent > 0: plates_m = shape_m[:-dim_parent] else: plates_m = shape_m # Compute the multiplier (multiply by the number of # plates for which both the message and parent have # single plates) plates_self = self.plates_to_parent(index) try: ## print('node.msg', self.name, plates_self, ## plates_m, ## parent.plates) r = self.plate_multiplier(plates_self, plates_m, parent.plates) except ValueError: raise ValueError("The plates of the message and " "parent[%d] node (%s) are not a " "broadcastable subset of the plates " "of this node (%s). The message has " "shape %s, meaning plates %s. This " "node has plates %s with respect to " "the parent[%d], which has plates %s." % (index, parent.name, self.name, np.shape(m[i]), plates_m, plates_self, index, parent.plates)) shape_parent = parent.get_shape(i) s = utils.axes_to_collapse(shape_m, shape_parent) m[i] = np.sum(m[i], axis=s, keepdims=True) m[i] = utils.squeeze_to_dim(m[i], len(shape_parent)) m[i] *= r return m def _message_from_children(self): msg = [np.array(0.0) for i in range(len(self.dims))] for (child,index) in self.children: m = child._message_to_parent(index) for i in range(len(self.dims)): if m[i] is not None: # Check broadcasting shapes sh = utils.broadcasted_shape(self.get_shape(i), np.shape(m[i])) try: # Try exploiting broadcasting rules msg[i] += m[i] except ValueError: msg[i] = msg[i] + m[i] return msg def _message_from_parents(self, ignore=None): return [parent._message_to_child() if ind != ignore else None for (ind,parent) in enumerate(self.parents)] def get_moments(self): raise NotImplementedError() class Stochastic(Node): """ Base class for nodes that are stochastic. u mask observed """ def __init__(self, *args, initialize=True, **kwargs): super().__init__(*args, dims=self.compute_dims(*args), **kwargs) axes = len(self.plates)*(1,) self.u = [utils.nans(axes+dim) for dim in self.dims] # Terms for the lower bound (G for latent and F for observed) #self.g = 0 #self.f = 0 # Not observed self.observed = False # By default, ignore all plates self.mask = False if initialize: self.initialize_from_prior() def get_moments(self): return self.u def _get_message_to_parent(self, index): u_parents = self._message_from_parents(ignore=index) return self._compute_message_to_parent(index, self.u, *u_parents) @staticmethod def _compute_mask_to_parent(index, mask): # Sub-classes may want to overwrite this if they do something to plates. return mask def _mask_to_parent(self, index): return self._compute_mask_to_parent(index, self.mask) def _set_mask(self, mask): self.mask = np.logical_or(mask, self.observed) def _set_moments(self, u, mask=True): # Store the computed moments u but do not change moments for # observations, i.e., utilize the mask. for ind in range(len(u)): # Add axes to the mask for the variable dimensions (mask # contains only axes for the plates). u_mask = utils.add_trailing_axes(mask, self.ndims[ind]) # Enlarge self.u[ind] as necessary so that it can store the # broadcasted result. sh = utils.broadcasted_shape_from_arrays(self.u[ind], u[ind], u_mask) self.u[ind] = utils.repeat_to_shape(self.u[ind], sh) # TODO/FIXME/BUG: The mask of observations is not used, observations # may be overwritten!!! ??? # Use mask to update only unobserved plates and keep the # observed as before np.copyto(self.u[ind], u[ind], where=u_mask) # Make sure u has the correct number of dimensions: shape = self.get_shape(ind) ndim = len(shape) ndim_u = np.ndim(self.u[ind]) if ndim > ndim_u: self.u[ind] = utils.add_leading_axes(u[ind], ndim - ndim_u) elif ndim < ndim_u: raise Exception("The size of the variable's moments array " "is larger than it should be based on the " "plates and dimension information. Check " "that you have provided plates properly.") def update(self): if not np.all(self.observed): u_parents = self._message_from_parents() m_children = self._message_from_children() self._update_distribution_and_lowerbound(m_children, *u_parents) def observe(y, mask=True): """ Fix moments, compute f and propagate mask. """ # Compute fixed moments (u, f) = self._compute_fixed_moments_and_f(x, mask=mask) # Check the dimensionality of the observations for (i,v) in enumerate(u): # This is what the dimensionality "should" be s = self.plates + self.dims[i] t = np.shape(v) if s != t: msg = "Dimensionality of the observations incorrect." msg += "\nShape of input: " + str(t) msg += "\nExpected shape: " + str(s) msg += "\nCheck plates." raise Exception(msg) # Set the moments self._set_moments(u, mask=mask) # TODO/FIXME: Use the mask? self.f = f # Observed nodes should not be ignored self.observed = mask self._update_mask() def lowerbound(): # Sub-class should implement this raise NotImplementedError() @staticmethod def _compute_fixed_moments_and_f(x, mask=True): # Sub-classes should implement this raise NotImplementedError() @staticmethod def _compute_message_to_parent(index, u_self, *u_parents): # Sub-classes should implement this raise NotImplementedError() def _update_distribution_and_lowerbound(self, m_children, *u_parents): # Sub-classes should implement this raise NotImplementedError() class Deterministic(Node): """ Base class for nodes that are deterministic. """ def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) def _mask_to_parent(self, index): # Sub-classes should implement this mask = False for (child, index) in self.children: mask = np.logical_or(mask, child._mask_to_parent(index)) return self._compute_mask_to_parent(index, mask) def get_moments(self): u_parents = [parent._message_to_child() for parent in self.parents] return self._compute_moments(*u_parents) def _get_message_to_parent(self, index): u_parents = self._message_from_parents(ignore=index) m_children = self._message_from_children() return self._compute_message_to_parent(index, m_children, *u_parents) def _compute_moments(*u_parents): # Sub-classes should implement this raise NotImplementedError() @staticmethod def _compute_message_to_parent(index, m_children, *u_parents): # Sub-classes should implement this raise NotImplementedError() bayespy-0.1/bayespy/inference/vmp/nodes/deterministic.py0000664000175000017500000002206412174013072025310 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import functools import numpy as np from bayespy.utils import utils from .node import Node class Deterministic(Node): """ Base class for nodes that are deterministic. Sub-classes must implement: 1. For implementing the deterministic function: _compute_moments(self, *u) 2. One of the following options: a) Simple methods: _compute_message_to_parent(self, index, m, *u) _compute_mask_to_parent(self, index, mask) b) More control with: _compute_message_and_mask_to_parent(self, index, m, *u) Sub-classes may need to re-implement: 1. If they manipulate plates: _compute_mask_to_parent(index, mask) _plates_to_parent(self, index) _plates_from_parent(self, index) """ def __init__(self, *args, **kwargs): super().__init__(*args, plates=None, **kwargs) def get_moments(self): u_parents = [parent._message_to_child() for parent in self.parents] return self._compute_moments(*u_parents) def _compute_message_and_mask_to_parent(self, index, m_children, *u_parents): # The following methods should be implemented by sub-classes. m = self._compute_message_to_parent(index, m_children, *u_parents) mask = self._compute_mask_to_parent(index, self.mask) return (m, mask) def _get_message_and_mask_to_parent(self, index): u_parents = self._message_from_parents(exclude=index) m_children = self._message_from_children() return self._compute_message_and_mask_to_parent(index, m_children, *u_parents) def _compute_moments(self, *u_parents): # Sub-classes should implement this raise NotImplementedError() def tile(X, tiles): """ Tile the plates of the input node. x = [a,b,c] y = tile(x, 2) = [a,b,c,a,b,c] There should be no need to tile plates that have unit length because they are handled properly by the broadcasting rules already. Parameters: ----------- X : Node Input node to be tiled. tiles : int, tuple Tiling of the plates (broadcasting rules for plates apply). See also: --------- numpy.tile """ # Make sure `tiles` is tuple (even if an integer is given) tiles = tuple(np.ravel(tiles)) class _Tile(Deterministic): def __init__(self, X, **kwargs): super().__init__(X, dims=X.dims, **kwargs) def _plates_to_parent(self, index): plates = list(self.plates) for i in range(-len(tiles), 0): plates[i] = plates[i] // tiles[i] return tuple(plates) def _plates_from_parent(self, index): return tuple(utils.multiply_shapes(self.parents[index].plates, tiles)) def _compute_mask_to_parent(self, index, mask): # Idea: Reshape the message array such that every other axis # will be summed and every other kept. # Make plates equal length plates = self._plates_to_parent(index) shape_m = np.shape(mask) (plates, tiles_m, shape_m) = utils.make_equal_length(plates, tiles, shape_m) # Handle broadcasting rules for axes that have unit length in # the message (although the plate may be non-unit length). Also, # compute the corresponding plate_multiplier. plates = list(plates) tiles_m = list(tiles_m) for j in range(len(plates)): if shape_m[j] == 1: plates[j] = 1 tiles_m[j] = 1 # Combine the tuples by picking every other from tiles_ind and # every other from shape shape = functools.reduce(lambda x,y: x+y, zip(tiles_m, plates)) # ..and reshape the array, that is, every other axis corresponds # to tiles and every other to plates/dimensions in parents mask = np.reshape(mask, shape) # Sum over every other axis axes = tuple(range(0,len(shape),2)) mask = np.any(mask, axis=axes) # Remove extra leading axes ndim_parent = len(self.parents[index].plates) mask = utils.squeeze_to_dim(mask, ndim_parent) return mask def _compute_message_to_parent(self, index, m, u_X): m = list(m) for ind in range(len(m)): # Idea: Reshape the message array such that every other axis # will be summed and every other kept. shape_ind = self._plates_to_parent(index) + self.dims[ind] # Add variable dimensions to tiles tiles_ind = tiles + (1,)*len(self.dims[ind]) # Make shape tuples equal length shape_m = np.shape(m[ind]) (tiles_ind, shape, shape_m) = utils.make_equal_length(tiles_ind, shape_ind, shape_m) # Handle broadcasting rules for axes that have unit length in # the message (although the plate may be non-unit length). Also, # compute the corresponding plate_multiplier. r = 1 shape = list(shape) tiles_ind = list(tiles_ind) for j in range(len(shape)): if shape_m[j] == 1: r *= tiles_ind[j] shape[j] = 1 tiles_ind[j] = 1 # Combine the tuples by picking every other from tiles_ind and # every other from shape shape = functools.reduce(lambda x,y: x+y, zip(tiles_ind, shape)) # ..and reshape the array, that is, every other axis corresponds # to tiles and every other to plates/dimensions in parents m[ind] = np.reshape(m[ind], shape) # Sum over every other axis axes = tuple(range(0,len(shape),2)) m[ind] = r * np.sum(m[ind], axis=axes) # Remove extra leading axes ndim_parent = len(self.parents[index].get_shape(ind)) m[ind] = utils.squeeze_to_dim(m[ind], ndim_parent) return m def _compute_moments(self, u_X): """ Tile the plates of the parent's moments. """ # Utilize broadcasting: If a tiled axis is unit length in u_X, there # is no need to tile it. u = list() for ind in range(len(u_X)): ui = u_X[ind] shape_u = np.shape(ui) if np.ndim(ui) > 0: # Add variable dimensions tiles_ind = tiles + (1,)*len(self.dims[ind]) # Utilize broadcasting: Do not tile leading empty axes nd = min(len(tiles_ind), np.ndim(ui)) tiles_ind = tiles_ind[(-nd):] # For simplicity, make tiles and shape equal length (tiles_ind, shape_u) = utils.make_equal_length(tiles_ind, shape_u) # Utilize broadcasting: Use tiling only if the parent's # moment has non-unit axis length. tiles_ind = [tile if sh > 1 else 1 for (tile, sh) in zip(tiles_ind, shape_u)] # Tile ui = np.tile(ui, tiles_ind) u.append(ui) return u return _Tile(X, name="tile(%s, %s)" % (X.name, tiles)) bayespy-0.1/bayespy/inference/vmp/nodes/categorical.py0000664000175000017500000000744112174013072024724 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np from .expfamily import ExponentialFamily from .constant import Constant from .dirichlet import Dirichlet def Categorical(p, **kwargs): # Get the number of categories (static methods may need this) if np.isscalar(p) or isinstance(p, np.ndarray): n_categories = np.shape(p)[-1] else: n_categories = p.dims[0][0] # The actual categorical distribution node class _Categorical(ExponentialFamily): ndims = (1,) @staticmethod def _compute_phi_from_parents(*u_parents): return [u_parents[0][0]] @staticmethod def _compute_cgf_from_parents(*u_parents): return 0 @staticmethod def _compute_moments_and_cgf(phi, mask=True): # For numerical reasons, scale contributions closer to # one, i.e., subtract the maximum of the log-contributions. max_phi = np.max(phi[0], axis=-1, keepdims=True) p = np.exp(phi[0]-max_phi) sum_p = np.sum(p, axis=-1, keepdims=True) # Moments u0 = p / sum_p u = [u0] # G g = -np.log(sum_p) - max_phi g = np.squeeze(g, axis=-1) #print('Categorical.compute_u_and_g, g:', np.sum(g), np.shape(g), np.sum(max_phi)) return (u, g) @staticmethod def _compute_fixed_moments_and_f(x, mask=True): """ Compute u(x) and f(x) for given x. """ # TODO: You could check that x has proper dimensions x = np.array(x, dtype=np.int) u0 = np.zeros((np.size(x), n_categories)) u0[[np.arange(np.size(x)), x]] = 1 f = 0 return ([u0], f) @staticmethod def _compute_message_to_parent(parent, index, u, *u_parents): """ . """ #print('message in categorical:', u[0]) if index == 0: return [ u[0].copy() ] @staticmethod def compute_dims(*parents): """ Compute the dimensions of phi/u. """ # Has the same dimensionality as the parent. return parents[0].dims def __init__(self, p, **kwargs): # Check for constant mu if np.isscalar(p) or isinstance(p, np.ndarray): p = ConstantDirichlet(p) # Construct super().__init__(p, **kwargs) def random(self): raise NotImplementedError() def show(self): p = self.u[0] #np.exp(self.phi[0]) #p /= np.sum(p, axis=-1, keepdims=True) print("%s ~ Categorical(p)" % self.name) print(" p = ") print(p) return _Categorical(p, **kwargs) bayespy-0.1/bayespy/inference/vmp/nodes/node.py0000664000175000017500000005534512174013072023402 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np from bayespy.utils import utils """ This module contains a sketch of a new implementation of the framework. """ def message_sum_multiply(plates_parent, dims_parent, *arrays): """ Compute message to parent and sum over plates. Divide by the plate multiplier. """ # The shape of the full message shapes = [np.shape(array) for array in arrays] shape_full = utils.broadcasted_shape(*shapes) # Find axes that should be summed shape_parent = plates_parent + dims_parent sum_axes = utils.axes_to_collapse(shape_full, shape_parent) # Compute the multiplier for cancelling the # plate-multiplier. Because we are summing over the # dimensions already in this function (for efficiency), we # need to cancel the effect of the plate-multiplier # applied in the message_to_parent function. r = 1 for j in sum_axes: if j >= 0 and j < len(plates_parent): r *= shape_full[j] elif j < 0 and j < -len(dims_parent): r *= shape_full[j] # Compute the sum-product m = utils.sum_multiply(*arrays, axis=sum_axes, sumaxis=True, keepdims=True) / r # Remove extra axes m = utils.squeeze_to_dim(m, len(shape_parent)) return m class Node(): """ Base class for all nodes. mask dims plates parents children name Sub-classes must implement: 1. For computing the message to children: get_moments(self): 2. For computing the message to parents: _get_message_and_mask_to_parent(self, index) Sub-classes may need to re-implement: 1. If they manipulate plates: _compute_mask_to_parent(index, mask) _plates_to_parent(self, index) _plates_from_parent(self, index) """ def __init__(self, *parents, dims=None, plates=None, name=""): if dims is None: raise Exception("You need to specify the dimensionality of the " "distribution for class %s" % str(self.__class__)) self.dims = dims self.name = name # Parents self.parents = parents # Inform parent nodes for (index,parent) in enumerate(self.parents): if parent: parent._add_child(self, index) # Check plates parent_plates = [self._plates_from_parent(index) for index in range(len(self.parents))] if plates is None: # By default, use the minimum number of plates determined # from the parent nodes try: self.plates = utils.broadcasted_shape(*parent_plates) except ValueError: raise ValueError("The plates of the parents do not broadcast.") else: # Use custom plates self.plates = plates # Check that the parent_plates are a subset of plates. for p in parent_plates: if not utils.is_shape_subset(p, plates): raise ValueError("The plates of the parents are not " "subsets of the given plates.") # By default, ignore all plates self.mask = np.array(False) # Children self.children = list() ## @staticmethod ## def _compute_dims_from_parents(*parents): ## """ Compute the dimensions of phi and u. """ ## raise NotImplementedError("Not implemented for %s" % self.__class__) ## @staticmethod ## def _compute_dims_from_values(x): ## """ Compute the dimensions of phi and u. """ ## raise NotImplementedError() def _plates_to_parent(self, index): # Sub-classes may want to overwrite this if they manipulate plates return self.plates def _plates_from_parent(self, index): # Sub-classes may want to overwrite this if they manipulate plates return self.parents[index].plates def get_shape(self, ind): return self.plates + self.dims[ind] def _add_child(self, child, index): """ Add a child node. Parameters ---------- child : node index : int The parent index of this node for the child node. The child node recognizes its parents by their index number. """ self.children.append((child, index)) def get_mask(self): return self.mask ## def _get_message_mask(self): ## return self.mask def _set_mask(self, mask): # Sub-classes may overwrite this method if they have some other masks to # be combined (for instance, observation mask) self.mask = mask def _update_mask(self): # Combine masks from children mask = np.array(False) for (child, index) in self.children: mask = np.logical_or(mask, child._mask_to_parent(index)) # Set the mask of this node self._set_mask(mask) if not utils.is_shape_subset(np.shape(self.mask), self.plates): raise ValueError("The mask of the node %s has updated " "incorrectly. The plates in the mask %s are not a " "subset of the plates of the node %s." % (self.name, np.shape(self.mask), self.plates)) # Tell parents to update their masks for parent in self.parents: parent._update_mask() ## @staticmethod ## def _compute_mask_to_parent(index, mask): ## # Sub-classes may want to overwrite this if they do something to plates. ## return mask # TODO: Rename to _compute_message_mask_to_parent(index, mask) @staticmethod def _compute_mask_to_parent(index, mask): """ Compute the mask used for messages sent to parent[index]. The mask tells which plates in the messages are active. This method is used for obtaining the mask which is used to set plates in the messages to parent to zero. Sub-classes may want to overwrite this method if they do something to plates so that the mask is somehow altered. """ return mask def _mask_to_parent(self, index): """ Get the mask with respect to parent[index]. The mask tells which plate connections are active. The mask is "summed" (logical or) and reshaped into the plate shape of the parent. Thus, it can't be used for masking messages, because some plates have been summed already. This method is used for propagating the mask to parents. """ mask = self._compute_mask_to_parent(index, self.mask) # Check the shape of the mask plates_to_parent = self._plates_to_parent(index) if not utils.is_shape_subset(np.shape(mask), plates_to_parent): print(self._plates_to_parent(index)) raise ValueError("In node %s, the mask being sent to " "parent[%d] (%s) has invalid shape: The shape of " "the mask %s is not a sub-shape of the plates of " "the node with respect to the parent %s. It could " "be that this node (%s) is manipulating plates " "but has not overwritten the method " "_compute_mask_to_parent." % (self.name, index, self.parents[index].name, np.shape(mask), plates_to_parent, self.__class__.__name__)) # "Sum" (i.e., logical or) over the plates that have unit length in # the parent node. parent_plates = self.parents[index].plates s = utils.axes_to_collapse(np.shape(mask), parent_plates) mask = np.any(mask, axis=s, keepdims=True) mask = utils.squeeze_to_dim(mask, len(parent_plates)) return mask #return self._compute_mask_to_parent(index, self.get_mask()) def _message_to_child(self): return self.get_moments() #def _get_message_and_mask_to_parent(self, index): # raise NotImplementedError() def _message_to_parent(self, index): # Compute the message, check plates, apply mask and sum over some plates if index >= len(self.parents): raise ValueError("Parent index larger than the number of parents") # Compute the message and mask (m, mask) = self._get_message_and_mask_to_parent(index) # Plates in the mask plates_mask = np.shape(mask) # The parent we're sending the message to parent = self.parents[index] # Compact the message to a proper shape for i in range(len(m)): # Empty messages are given as None. We can ignore those. if m[i] is not None: # Plates in the message shape_m = np.shape(m[i]) dim_parent = len(parent.dims[i]) if dim_parent > 0: plates_m = shape_m[:-dim_parent] else: plates_m = shape_m # Compute the multiplier (multiply by the number of plates for # which the message, the mask and the parent have single # plates). Such a plate is meant to be broadcasted but because # the parent has singular plate axis, it won't broadcast (and # sum over it), so we need to multiply it. plates_self = self._plates_to_parent(index) try: r = self._plate_multiplier(plates_self, plates_m, plates_mask, parent.plates) except ValueError: raise ValueError("The plates of the message, the mask and " "parent[%d] node (%s) are not a " "broadcastable subset of the plates of " "this node (%s). The message has shape " "%s, meaning plates %s. The mask has " "plates %s. This node has plates %s with " "respect to the parent[%d], which has " "plates %s." % (index, parent.name, self.name, np.shape(m[i]), plates_m, plates_mask, plates_self, index, parent.plates)) # Add variable axes to the mask shape_mask = np.shape(mask) + (1,) * len(parent.dims[i]) mask_i = np.reshape(mask, shape_mask) # Sum over plates that are not in the message nor in the parent shape_parent = parent.get_shape(i) shape_msg = utils.broadcasted_shape(shape_m, shape_parent) axes_mask = utils.axes_to_collapse(shape_mask, shape_msg) mask_i = np.sum(mask_i, axis=axes_mask, keepdims=True) # Compute the masked message and sum over the plates that the # parent does not have. axes_msg = utils.axes_to_collapse(shape_msg, shape_parent) m[i] = utils.sum_multiply(mask_i, m[i], r, axis=axes_msg, keepdims=True) # Remove leading singular plates if the parent does not have # those plate axes. m[i] = utils.squeeze_to_dim(m[i], len(shape_parent)) return m def _message_from_children(self): msg = [np.array(0.0) for i in range(len(self.dims))] for (child,index) in self.children: m = child._message_to_parent(index) for i in range(len(self.dims)): if m[i] is not None: # Check broadcasting shapes sh = utils.broadcasted_shape(self.get_shape(i), np.shape(m[i])) try: # Try exploiting broadcasting rules msg[i] += m[i] except ValueError: msg[i] = msg[i] + m[i] return msg def _message_from_parents(self, exclude=None): return [parent._message_to_child() if ind != exclude else None for (ind,parent) in enumerate(self.parents)] def get_moments(self): raise NotImplementedError() @staticmethod def _plate_multiplier(plates, *args): # Check broadcasting of the shapes for arg in args: utils.broadcasted_shape(plates, arg) # Check that each arg-plates are a subset of plates? for arg in args: if not utils.is_shape_subset(arg, plates): raise ValueError("The shapes in args are not a sub-shape of " "plates.") r = 1 for j in range(-len(plates),0): mult = True for arg in args: if not (-j > len(arg) or arg[j] == 1): mult = False if mult: r *= plates[j] return r def move_plates(self, from_plate, to_plate): return _MovePlate(self, from_plate, to_plate, name=self.name + ".move_plates") def add_plate_axis(self, to_plate): return AddPlateAxis(to_plate)(self, name=self.name+".add_plate_axis") from .deterministic import Deterministic def AddPlateAxis(to_plate): if to_plate >= 0: raise Exception("Give negative value for axis index to_plate.") class _AddPlateAxis(Deterministic): def __init__(self, X, **kwargs): nonlocal to_plate N = len(X.plates) + 1 # Check the parameters if to_plate >= 0 or to_plate < -N: raise ValueError("Invalid plate position to add.") # Use positive indexing only ## if to_plate < 0: ## to_plate += N # Use negative indexing only if to_plate >= 0: to_plate -= N #self.to_plate = to_plate super().__init__(X, dims=X.dims, **kwargs) def _plates_to_parent(self, index): plates = list(self.plates) plates.pop(to_plate) return tuple(plates) #return self.plates[:to_plate] + self.plates[(to_plate+1):] def _plates_from_parent(self, index): plates = list(self.parents[index].plates) plates.insert(len(plates)-to_plate+1, 1) return tuple(plates) #raise Exception("IMPLEMENT THIS") @staticmethod def _compute_mask_to_parent(index, mask): # Ouch, how can you compute this in a static function? # Maybe this does not have to be a static funtion? # Maybe Mixture node requires this and _plates_to/from_parent to be # static as well.. # Remove the added mask plate #diff = len(self.plates) - np.ndim(mask) #mask = utils.add_leading_axes(mask, diff) if abs(to_plate) <= np.ndim(mask): sh_mask = list(np.shape(mask)) sh_mask.pop(to_plate) mask = np.reshape(mask, sh_mask) return mask #raise Exception("IMPLEMENT THIS") def _compute_message_and_mask_to_parent(self, index, m, *u_parents): """ Compute the message to a parent node. """ # Get the message from children #(m, mask) = self.message_from_children() # Remove the added message plate for i in range(len(m)): # Make sure the message has all the axes #diff = len(self.plates) + len(self.dims[i]) - np.ndim(m[i]) #m[i] = utils.add_leading_axes(m[i], diff) # Remove the axis if np.ndim(m[i]) >= abs(to_plate) + len(self.dims[i]): axis = to_plate - len(self.dims[i]) #ndims = np.ndim(m[i]) - len(self.dims[i]) + 1 sh_m = list(np.shape(m[i])) sh_m.pop(axis) m[i] = np.reshape(m[i], sh_m) mask = self._compute_mask_to_parent(index, self.mask) return (m, mask) def _compute_moments(self, u): """ Get the moments with an added plate axis. """ # Get parents' moments #u = self.parents[0].message_to_child() # Move a plate axis u = list(u) for i in range(len(u)): # Make sure the moments have all the axes #diff = len(self.plates) + len(self.dims[i]) - np.ndim(u[i]) - 1 #u[i] = utils.add_leading_axes(u[i], diff) # The location of the new axis/plate: axis = np.ndim(u[i]) - abs(to_plate) - len(self.dims[i]) + 1 if axis > 0: # Add one axes to the correct position sh_u = list(np.shape(u[i])) sh_u.insert(axis, 1) u[i] = np.reshape(u[i], sh_u) return u return _AddPlateAxis ## class _MovePlate(Node): ## """ ## Move a plate to a given position. ## NOTE: This has NOT been tested yet.. ## """ ## def __init__(self, X, from_plate, to_plate, plates=None, **kwargs): ## if plates is not None: ## raise ValueError("Do not specify plates.") ## plates = X.plates ## # Check the parameters ## if from_plate >= len(plates) or from_plate < -len(plates): ## raise ValueError("Invalid plate to move from.") ## if to_plate >= len(plates) or to_plate < -len(plates): ## raise ValueError("Invalid plate to move to.") ## # Use positive indexing only ## if from_plate < 0: ## from_plate = len(plates) + from_plate ## if to_plate < 0: ## to_plate = len(plates) + to_plate + 1 ## # Move the plate ## plates = list(plates) ## plates.insert(to_plate, plates.pop(from_plate)) ## super().__init__(X, ## plates=tuple(plates), ## dims=X.dims, ## **kwargs) ## def get_moments(self): ## """ ## Get the moments with moved plates. ## """ ## # Get parents' moments ## u = self.parents[0].message_to_child() ## # Move a plate axis ## u = list(u) ## for i in range(len(u)): ## u[i] = utils.moveaxis(u[i], self.from_plate, self.to_plate) ## return tuple(u) ## def get_message(self, index, u_parents): ## """ ## Compute the message to a parent node. ## """ ## # Get the message from children ## (m, mask) = self.message_from_children() ## # Move message plates ## for i in range(len(m)): ## diff = len(self.plates) + len(self.dims[i]) - self.np.ndim(m[i]) ## m[i] = utils.add_leading_axes(m[i], diff) ## m[i] = utils.moveaxis(m[i], self.to_plate, self.from_plate) ## # Move mask plates ## mask = utils.add_leading_axes(mask, len(self.plates) - np.ndim(mask)) ## mask = utils.moveaxis(mask, self.to_plate, self.from_plate) ## return (m, mask) class NodeConstant(Node): def __init__(self, u, **kwargs): self.u = u Node.__init__(self, **kwargs) def message_to_child(self, gradient=False): if gradient: return (self.u, []) else: return self.u class NodeConstantScalar(NodeConstant): @staticmethod def compute_fixed_u_and_f(x): """ Compute u(x) and f(x) for given x. """ return ([x], 0) def __init__(self, a, **kwargs): NodeConstant.__init__(self, [a], plates=np.shape(a), dims=[()], **kwargs) def start_optimization(self): # FIXME: Set the plate sizes appropriately!! x0 = self.u[0] #self.gradient = np.zeros(np.shape(x0)) def transform(x): # E.g., for positive scalars you could have exp here. self.gradient = np.zeros(np.shape(x0)) self.u[0] = x def gradient(): # This would need to apply the gradient of the # transformation to the computed gradient return self.gradient return (x0, transform, gradient) def add_to_gradient(self, d): self.gradient += d def message_to_child(self, gradient=False): if gradient: return (self.u, [ [np.ones(np.shape(self.u[0])), #self.gradient] ]) self.add_to_gradient] ]) else: return self.u def stop_optimization(self): #raise Exception("Not implemented for " + str(self.__class__)) pass bayespy-0.1/bayespy/inference/__init__.py0000664000175000017500000000214012174023032022260 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### from .vmp.vmp import VB bayespy-0.1/bayespy/_nodes_dev/0000775000175000017500000000000012174043062020326 5ustar jluttinejluttine00000000000000bayespy-0.1/bayespy/_nodes_dev/GaussianProcesses.py0000664000175000017500000006425312174013072024351 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import itertools import numpy as np #import scipy as sp #import scipy.linalg.decomp_cholesky as decomp import scipy.linalg as linalg #import scipy.special as special #import matplotlib.pyplot as plt #import time #import profile #import scipy.spatial.distance as distance import scipy.sparse as sp import utils import Nodes.ExponentialFamily as EF import Nodes.CovarianceFunctions as CF import imp imp.reload(utils) imp.reload(EF) imp.reload(CF) class CovarianceMatrix: def cholesky(self): pass def multiply(A, B): return np.multiply(A,B) # m prior mean function # k prior covariance function # x data inputs # z processed data outputs (z = inv(Cov) * (y-m(x))) # U data covariance Cholesky factor def gp_posterior_moment_function(m, k, x, y, k_sparse=None, pseudoinputs=None, noise=None): # Prior # FIXME: We are ignoring the covariance of mu now.. mu = m(x)[0] ## if np.ndim(mu) == 1: ## mu = np.asmatrix(mu).T ## else: ## mu = np.asmatrix(mu) K_noise = None if noise != None: if K_noise is None: K_noise = noise else: K_noise += noise if k_sparse != None: if K_noise is None: K_noise = k_sparse(x,x)[0] else: K_noise += k_sparse(x,x)[0] if pseudoinputs != None: p = pseudoinputs #print('in pseudostuff') #print(K_noise) #print(np.shape(K_noise)) K_pp = k(p,p)[0] K_xp = k(x,p)[0] U = utils.chol(K_noise) # Compute Lambda Lambda = K_pp + np.dot(K_xp.T, utils.chol_solve(U, K_xp)) U_lambda = utils.chol(Lambda) # Compute statistics for posterior predictions #print(np.shape(U_lambda)) #print(np.shape(y)) z = utils.chol_solve(U_lambda, np.dot(K_xp.T, utils.chol_solve(U, y - mu))) U = utils.chol(K_pp) # Now we can forget the location of the observations and # consider only the pseudoinputs when predicting. x = p else: K = K_noise if K is None: K = k(x,x)[0] else: try: K += k(x,x)[0] except: K = K + k(x,x)[0] # Compute posterior GP N = len(y) U = None z = None if N > 0: U = utils.chol(K) z = utils.chol_solve(U, y-mu) def get_moments(h, covariance=1, mean=True): K_xh = k(x, h)[0] if k_sparse != None: try: # This may not work, for instance, if either one is a # sparse matrix. K_xh += k_sparse(x, h)[0] except: K_xh = K_xh + k_sparse(x, h)[0] # NumPy has problems when mixing matrices and arrays. # Matrices may appear, for instance, when you sum an array and # a sparse matrix. Make sure the result is either an array or # a sparse matrix (not dense matrix!), because matrix objects # cause lots of problems: # # array.dot(array) = array # matrix.dot(array) = matrix # sparse.dot(array) = array if not sp.issparse(K_xh): K_xh = np.asarray(K_xh) # Function for computing posterior moments if mean: # Mean vector # FIXME: Ignoring the covariance of prior mu m_h = m(h)[0] if z != None: m_h += K_xh.T.dot(z) else: m_h = None # Compute (co)variance matrix/vector if covariance: if covariance == 1: ## Compute variance vector k_h = k(h)[0] if k_sparse != None: k_h += k_sparse(h)[0] if U != None: if isinstance(K_xh, np.ndarray): k_h -= np.einsum('i...,i...', K_xh, utils.chol_solve(U, K_xh)) else: # TODO: This isn't very efficient way, but # einsum doesn't work for sparse matrices.. # This may consume A LOT of memory for sparse # matrices. k_h -= np.asarray(K_xh.multiply(utils.chol_solve(U, K_xh))).sum(axis=0) if pseudoinputs != None: if isinstance(K_xh, np.ndarray): k_h += np.einsum('i...,i...', K_xh, utils.chol_solve(U_lambda, K_xh)) else: # TODO: This isn't very efficient way, but # einsum doesn't work for sparse matrices.. # This may consume A LOT of memory for sparse # matrices. k_h += np.asarray(K_xh.multiply(utils.chol_solve(U_lambda, K_xh))).sum(axis=0) # Ensure non-negative variances k_h[k_h<0] = 0 return (m_h, k_h) elif covariance == 2: ## Compute full covariance matrix K_hh = k(h,h)[0] if k_sparse != None: K_hh += k_sparse(h)[0] if U != None: K_hh -= K_xh.T.dot(utils.chol_solve(U,K_xh)) #K_hh -= np.dot(K_xh.T, utils.chol_solve(U,K_xh)) if pseudoinputs != None: K_hh += K_xh.T.dot(utils.chol_solve(U_lambda, K_xh)) #K_hh += np.dot(K_xh.T, utils.chol_solve(U_lambda, K_xh)) return (m_h, K_hh) else: return (m_h, None) return get_moments # Constant function using GP mean protocol class Constant(EF.Node): def __init__(self, f, **kwargs): self.f = f EF.Node.__init__(self, dims=[(np.inf,)], **kwargs) def message_to_child(self, gradient=False): # Wrapper def func(x, gradient=False): if gradient: return ([self.f(x), None], []) else: return [self.f(x), None] return func #class MultiDimensional(EF.NodeVariable): # """ A multi-dimensional Gaussian process f(x). """ ## class ToGaussian(EF.NodeVariable): ## """ Deterministic node which transform a Gaussian process into ## finite-dimensional Gaussian variable. """ ## def __init__(self, f, x, **kwargs): ## EF.NodeVariable.__init__(self, ## f, ## x, ## plates= ## dims= # Deterministic node for creating a set of GPs which can be used as a # mean function to a general GP node. class Multiple(EF.NodeVariable): def __init__(self, GPs, **kwargs): # Ignore plates EF.NodeVariable.__init__(self, *GPs, plates=(), dims=[(np.inf,), (np.inf,np.inf)], **kwargs) def message_to_parent(self, index): raise Exception("not implemented yet") def message_to_child(self, gradient=False): u = [parent.message_to_child() for parent in self.parents] def get_moments(xh, **kwargs): mh_all = [] khh_all = [] for i in range(len(self.parents)): xi = np.array(xh[i]) #print(xi) #print(np.shape(xi)) #print(xi) # FIXME: We are ignoring the covariance of mu now.. if gradient: ((mh, khh), dm) = u[i](xi, **kwargs) else: (mh, khh) = u[i](xi, **kwargs) #mh = u[i](xi, **kwargs)[0] #print(mh) #print(mh_all) ## print(mh) ## print(khh) ## print(np.shape(mh)) mh_all = np.concatenate([mh_all, mh]) #print(np.shape(mh_all)) if khh != None: print(khh) raise Exception('Not implemented yet for covariances') #khh_all = np.concatenate([khh_all, khh]) # FIXME: Compute gradients! if gradient: return ([mh_all, khh_all], []) else: return [mh_all, khh_all] #return [mh_all, khh_all] return get_moments # Gaussian process distribution class GaussianProcess(EF.NodeVariable): def __init__(self, m, k, k_sparse=None, pseudoinputs=None, **kwargs): self.x = np.array([]) self.f = np.array([]) ## self.x_obs = np.zeros((0,1)) ## self.f_obs = np.zeros((0,)) if pseudoinputs != None: pseudoinputs = EF.NodeConstant([pseudoinputs], dims=[np.shape(pseudoinputs)]) # By default, posterior == prior self.m = None #m self.k = None #k if isinstance(k, list) and isinstance(m, list): if len(k) != len(m): raise Exception('The number of mean and covariance functions must be equal.') k = CF.Multiple(k) m = Multiple(m) elif isinstance(k, list): D = len(k) k = CF.Multiple(k) m = Multiple(D*[m]) elif isinstance(m, list): D = len(m) k = CF.Multiple(D*[k]) m = Multiple(m) # Ignore plates EF.NodeVariable.__init__(self, m, k, k_sparse, pseudoinputs, plates=(), dims=[(np.inf,), (np.inf,np.inf)], **kwargs) def __call__(self, x, covariance=None): if not covariance: return self.u(x, covariance=False)[0] elif covariance.lower() == 'vector': return self.u(x, covariance=1) elif covariance.lower() == 'matrix': return self.u(x, covariance=2) else: raise Exception("Unknown covariance type requested") def message_to_parent(self, index): if index == 0: k = self.parents[1].message_to_child()[0] K = k(self.x, self.x) return [self.x, self.mu, K] if index == 1: raise Exception("not implemented yet") def message_to_child(self): if self.observed: raise Exception("Observable GP should not have children.") return self.u def get_parameters(self): return self.u def observe(self, x, f): self.observed = True self.x = x self.f = f ## if np.ndim(f) == 1: ## self.f = np.asmatrix(f).T ## else: ## self.f = np.asmatrix(f) # You might want: # - mean for x # - covariance (and mean) for x # - variance (and mean) for x # - i.e., mean and/or (co)variance for x # - covariance for x1 and x2 def lower_bound_contribution(self, gradient=False): # Get moment functions from parents m = self.parents[0].message_to_child(gradient=gradient) k = self.parents[1].message_to_child(gradient=gradient) if self.parents[2]: k_sparse = self.parents[2].message_to_child(gradient=gradient) else: k_sparse = None if self.parents[3]: pseudoinputs = self.parents[3].message_to_child(gradient=gradient) #pseudoinputs = self.parents[3].message_to_child(gradient=gradient)[0] else: pseudoinputs = None ## m = self.parents[0].message_to_child(gradient=gradient)[0] ## k = self.parents[1].message_to_child(gradient=gradient)[0] # Compute the parameters (covariance matrices etc) using # parents' moment functions DKs_xx = [] DKd_xx = [] DKd_xp = [] DKd_pp = [] Dxp = [] Dmu = [] if gradient: # FIXME: We are ignoring the covariance of mu now.. ((mu, _), Dmu) = m(self.x, gradient=True) ## if k_sparse: ## ((Ks_xx,), DKs_xx) = k_sparse(self.x, self.x, gradient=True) if pseudoinputs: ((Ks_xx,), DKs_xx) = k_sparse(self.x, self.x, gradient=True) ((xp,), Dxp) = pseudoinputs ((Kd_pp,), DKd_pp) = k(xp,xp, gradient=True) ((Kd_xp,), DKd_xp) = k(self.x, xp, gradient=True) else: ((K_xx,), DKd_xx) = k(self.x, self.x, gradient=True) if k_sparse: ((Ks_xx,), DKs_xx) = k_sparse(self.x, self.x, gradient=True) try: K_xx += Ks_xx except: K_xx = K_xx + Ks_xx else: # FIXME: We are ignoring the covariance of mu now.. (mu, _) = m(self.x) ## if k_sparse: ## (Ks_xx,) = k_sparse(self.x, self.x) if pseudoinputs: (Ks_xx,) = k_sparse(self.x, self.x) (xp,) = pseudoinputs (Kd_pp,) = k(xp, xp) (Kd_xp,) = k(self.x, xp) else: (K_xx,) = k(self.x, self.x) if k_sparse: (Ks_xx,) = k_sparse(self.x, self.x) try: K_xx += Ks_xx except: K_xx = K_xx + Ks_xx mu = mu[0] #K = K[0] # Log pdf if self.observed: ## Log pdf for directly observed GP f0 = self.f - mu #print('hereiam') #print(K) if pseudoinputs: ## Pseudo-input approximation # Decompose the full-rank sparse/noise covariance matrix try: Us_xx = utils.cholesky(Ks_xx) except linalg.LinAlgError: print('Noise/sparse covariance not positive definite') return -np.inf # Use Woodbury-Sherman-Morrison formula with the # following notation: # # y2 = f0' * inv(Kd_xp*inv(Kd_pp)*Kd_xp' + Ks_xx) * f0 # # z = Ks_xx \ f0 # Lambda = Kd_pp + Kd_xp'*inv(Ks_xx)*Kd_xp # nu = inv(Lambda) * (Kd_xp' * (Ks_xx \ f0)) # rho = Kd_xp * inv(Lambda) * (Kd_xp' * (Ks_xx \ f0)) # # y2 = f0' * z - z' * rho z = Us_xx.solve(f0) Lambda = Kd_pp + np.dot(Kd_xp.T, Us_xx.solve(Kd_xp)) ## z = utils.chol_solve(Us_xx, f0) ## Lambda = Kd_pp + np.dot(Kd_xp.T, ## utils.chol_solve(Us_xx, Kd_xp)) try: U_Lambda = utils.cholesky(Lambda) #U_Lambda = utils.chol(Lambda) except linalg.LinAlgError: print('Lambda not positive definite') return -np.inf nu = U_Lambda.solve(np.dot(Kd_xp.T, z)) #nu = utils.chol_solve(U_Lambda, np.dot(Kd_xp.T, z)) rho = np.dot(Kd_xp, nu) y2 = np.dot(f0, z) - np.dot(z, rho) # Use matrix determinant lemma # # det(Kd_xp*inv(Kd_pp)*Kd_xp' + Ks_xx) # = det(Kd_pp + Kd_xp'*inv(Ks_xx)*Kd_xp) # * det(inv(Kd_pp)) * det(Ks_xx) # = det(Lambda) * det(Ks_xx) / det(Kd_pp) try: Ud_pp = utils.cholesky(Kd_pp) #Ud_pp = utils.chol(Kd_pp) except linalg.LinAlgError: print('Covariance of pseudo inputs not positive definite') return -np.inf logdet = (U_Lambda.logdet() + Us_xx.logdet() - Ud_pp.logdet()) ## logdet = (utils.logdet_chol(U_Lambda) ## + utils.logdet_chol(Us_xx) ## - utils.logdet_chol(Ud_pp)) # Compute the log pdf L = gaussian_logpdf(y2, 0, 0, logdet, np.size(self.f)) # Add the variational cost of the pseudo-input # approximation # Compute gradients for (dmu, func) in Dmu: # Derivative w.r.t. mean vector d = np.nan # Send the derivative message func(d) for (dKs_xx, func) in DKs_xx: # Compute derivative w.r.t. covariance matrix d = np.nan # Send the derivative message func(d) for (dKd_xp, func) in DKd_xp: # Compute derivative w.r.t. covariance matrix d = np.nan # Send the derivative message func(d) V = Ud_pp.solve(Kd_xp.T) Z = Us_xx.solve(V.T) ## V = utils.chol_solve(Ud_pp, Kd_xp.T) ## Z = utils.chol_solve(Us_xx, V.T) for (dKd_pp, func) in DKd_pp: # Compute derivative w.r.t. covariance matrix d = (0.5 * np.trace(Ud_pp.solve(dKd_pp)) - 0.5 * np.trace(U_Lambda.solve(dKd_pp)) + np.dot(nu, np.dot(dKd_pp, nu)) + np.trace(np.dot(dKd_pp, np.dot(V,Z)))) ## d = (0.5 * np.trace(utils.chol_solve(Ud_pp, dKd_pp)) ## - 0.5 * np.trace(utils.chol_solve(U_Lambda, dKd_pp)) ## + np.dot(nu, np.dot(dKd_pp, nu)) ## + np.trace(np.dot(dKd_pp, ## np.dot(V,Z)))) # Send the derivative message func(d) for (dxp, func) in Dxp: # Compute derivative w.r.t. covariance matrix d = np.nan # Send the derivative message func(d) else: ## Full exact (no pseudo approximations) try: U = utils.cholesky(K_xx) #U = utils.chol(K_xx) except linalg.LinAlgError: print('non positive definite, return -inf') return -np.inf z = U.solve(f0) #z = utils.chol_solve(U, f0) #print(K) L = utils.gaussian_logpdf(np.dot(f0, z), 0, 0, U.logdet(), ## utils.logdet_chol(U), np.size(self.f)) for (dmu, func) in Dmu: # Derivative w.r.t. mean vector d = -np.sum(z) # Send the derivative message func(d) for (dK, func) in DKd_xx: # Compute derivative w.r.t. covariance matrix # # TODO: trace+chol_solve should be handled better # for sparse matrices. Use sparse-inverse! d = 0.5 * (dK.dot(z).dot(z) - U.trace_solve_gradient(dK)) ## - np.trace(U.solve(dK))) ## d = 0.5 * (dK.dot(z).dot(z) ## - np.trace(utils.chol_solve(U, dK))) #print('derivate', d, dK) ## d = 0.5 * (np.dot(z, np.dot(dK, z)) ## - np.trace(utils.chol_solve(U, dK))) # # Send the derivative message func(d) for (dK, func) in DKs_xx: # Compute derivative w.r.t. covariance matrix d = 0.5 * (dK.dot(z).dot(z) - U.trace_solve_gradient(dK)) ## - np.trace(U.solve(dK))) ## d = 0.5 * (dK.dot(z).dot(z) ## - np.trace(utils.chol_solve(U, dK))) ## d = 0.5 * (np.dot(z, np.dot(dK, z)) ## - np.trace(utils.chol_solve(U, dK))) # Send the derivative message func(d) else: ## Log pdf for latent GP raise Exception('Not implemented yet') return L ## Let f1 be observed and f2 latent function values. # Compute #L = gaussian_logpdf(sum_product(np.outer(self.f,self.f) + self.Cov, # Compute def update(self): # Messages from parents m = self.parents[0].message_to_child() k = self.parents[1].message_to_child() if self.parents[2]: k_sparse = self.parents[2].message_to_child() else: k_sparse = None if self.parents[3]: pseudoinputs = self.parents[3].message_to_child()[0] else: pseudoinputs = None ## m = self.parents[0].message_to_child()[0] ## k = self.parents[1].message_to_child()[0] if self.observed: # Observations of this node self.u = gp_posterior_moment_function(m, k, self.x, self.f, k_sparse=k_sparse, pseudoinputs=pseudoinputs) else: x = np.array([]) y = np.array([]) # Messages from children for (child,index) in self.children: (msg, mask) = child.message_to_parent(index) # Ignoring masks and plates.. # m[0] is the inputs x = np.concatenate((x, msg[0]), axis=-2) # m[1] is the observations y = np.concatenate((y, msg[1])) # m[2] is the covariance matrix V = linalg.block_diag(V, msg[2]) self.u = gp_posterior_moment_function(m, k, x, y, covariance=V) self.x = x self.f = y # At least for now, simplify this GP node such that a GP is either # observed or latent. If it is observed, it doesn't take messages from # children, actually, it should not even have children! ## # Pseudo for GPFA: ## k1 = gp_cov_se(magnitude=theta1, lengthscale=theta2) ## k2 = gp_cov_periodic(magnitude=.., lengthscale=.., period=..) ## k3 = gp_cov_rq(magnitude=.., lengthscale=.., alpha=..) ## f = NodeGPSet(0, [k1,k2,k3]) # assumes block diagonality ## # f = NodeGPSet(0, [[k11,k12,k13],[k21,k22,k23],[k31,k32,k33]]) ## X = GaussianFromGP(f, [ [[t0,0],[t0,1],[t0,2]], [t1,0],[t1,1],[t1,2], ..]) ## ... ## # Construct a sum of GPs if interested only in the sum term ## k1 = gp_cov_se(magnitude=theta1, lengthscale=theta2) ## k2 = gp_cov_periodic(magnitude=.., lengthscale=.., period=..) ## k = gp_cov_sum(k1, k2) ## f = NodeGP(0, k) ## f.observe(x, y) ## f.update() ## (mp, kp) = f.get_parameters() ## # Construct a sum of GPs when interested also in the individual ## # GPs: ## k1 = gp_cov_se(magnitude=theta1, lengthscale=theta2) ## k2 = gp_cov_periodic(magnitude=.., lengthscale=.., period=..) ## k3 = gp_cov_delta(magnitude=theta3) ## f = NodeGPSum(0, [k1,k2,k3]) ## x = np.array([1,2,3,4,5,6,7,8,9,10]) ## y = np.sin(x[0]) + np.random.normal(0, 0.1, (10,)) ## # Observe the sum (index 0) ## f.observe((0,x), y) ## # Inference ## f.update() ## (mp, kp) = f.get_parameters() ## # Mean of the sum ## mp[0](...) ## # Mean of the individual terms ## mp[1](...) ## mp[2](...) ## mp[3](...) ## # Covariance of the sum ## kp[0][0](..., ...) ## # Other covariances ## kp[1][1](..., ...) ## kp[2][2](..., ...) ## kp[3][3](..., ...) ## kp[1][2](..., ...) ## kp[1][3](..., ...) ## kp[2][3](..., ...) bayespy-0.1/bayespy/_nodes_dev/wishart.py0000664000175000017500000001256712174013072022372 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np import scipy.special as special from bayespy.utils import utils from .variable import Variable from .constant import Constant def WishartPrior(k): class _WishartPrior: """ Conjugate prior node for the degrees of freedom of the Wishart distribution. This isn't a distribution but can be used to compute the correct fixed moments, e.g., for Constant node.""" # Number of trailing axes for variable dimensions in # observations. The other axes correspond to plates. ndim_observations = 0 @staticmethod def compute_fixed_moments(n): """ Compute moments for fixed x. """ u0 = n u1 = special.multigammaln(0.5*n, k) return [u0, u1] @staticmethod def compute_dims_from_values(n): """ Compute the dimensions of phi or u. """ return [(), ()] return _WishartPrior class Wishart(Variable): ndims = (2, 0) ndims_parents = [None, (2, 0)] # Observations/values are 2-D matrices ndim_observations = 2 #parameter_distributions = (WishartPrior, Wishart) ## @staticmethod ## def compute_fixed_parameter_moments(*args): ## """ Compute the moments of the distribution parameters for ## fixed values.""" ## n = args[0] ## V = args[1] ## k = np.shape(V)[-1] ## u_n = WishartPrior(k).compute_fixed_moments(n) ## u_V = Wishart.compute_fixed_moments(V) ## return (u_n, u_V) @staticmethod def compute_fixed_moments(Lambda): """ Compute moments for fixed x. """ ldet = utils.m_chol_logdet(utils.m_chol(Lambda)) u = [Lambda, ldet] return u @staticmethod def compute_g_from_parents(u_parents): n = u_parents[0][0] gammaln_n = u_parents[0][1] V = u_parents[1][0] logdet_V = u_parents[1][1] k = np.shape(V)[-1] #k = self.dims[0][0] # TODO: Check whether this is correct: #g = 0.5*n*logdet_V - special.multigammaln(n/2, k) g = 0.5*n*logdet_V - 0.5*k*n*np.log(2) - gammaln_n #special.multigammaln(n/2, k) return g @staticmethod def compute_phi_from_parents(u_parents): return [-0.5 * u_parents[1][0], 0.5 * u_parents[0][0]] @staticmethod def compute_u_and_g(phi, mask=True): U = utils.m_chol(-phi[0]) k = np.shape(phi[0])[-1] #k = self.dims[0][0] logdet_phi0 = utils.m_chol_logdet(U) u0 = phi[1][...,np.newaxis,np.newaxis] * utils.m_chol_inv(U) u1 = -logdet_phi0 + utils.m_digamma(phi[1], k) u = [u0, u1] g = phi[1] * logdet_phi0 - special.multigammaln(phi[1], k) return (u, g) @staticmethod def compute_fixed_u_and_f(Lambda): """ Compute u(x) and f(x) for given x. """ k = np.shape(Lambda)[-1] ldet = utils.m_chol_logdet(utils.m_chol(Lambda)) u = [Lambda, ldet] f = -(k+1)/2 * ldet return (u, f) @staticmethod def message(index, u, u_parents): if index == 0: raise Exception("No analytic solution exists") elif index == 1: return (-0.5 * u[0], 0.5 * u_parents[0][0]) @staticmethod def compute_dims(*parents): """ Compute the dimensions of phi/u. """ # Has the same dimensionality as the second parent. return parents[1].dims @staticmethod def compute_dims_from_values(x): """ Compute the dimensions of phi and u. """ d = np.shape(x)[-1] return [(d,d), ()] # Wishart(n, inv(V)) def __init__(self, n, V, plates=(), **kwargs): # Check for constant V if np.isscalar(V) or isinstance(V, np.ndarray): V = Constant(Wishart)(V) k = V.dims[0][-1] # Check for constant n if np.isscalar(n) or isinstance(n, np.ndarray): n = Constant(WishartPrior(k))(n) #n = NodeConstantScalar(n) self.parameter_distributions = (WishartPrior(k), Wishart) super().__init__(n, V, plates=plates, **kwargs) def show(self): print("%s ~ Wishart(n, A)" % self.name) print(" n =") print(2*self.phi[1]) print(" A =") print(0.5 * self.u[0] / self.phi[1][...,np.newaxis,np.newaxis]) bayespy-0.1/bayespy/_nodes_dev/mixture.py0000664000175000017500000003503612174013072022402 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import itertools import numpy as np import scipy as sp import scipy.linalg.decomp_cholesky as decomp import scipy.linalg as linalg import scipy.special as special import scipy.spatial.distance as distance import bayespy.utils as utils from .variable import Variable from .constant import Constant from .categorical import Categorical def Mixture(distribution, cluster_plate=-1): if cluster_plate >= 0: raise Exception("Give negative value for axis index cluster_plates") class _Mixture(Variable): ndims = distribution.ndims @staticmethod def compute_phi_from_parents(u_parents): # Compute weighted average of the parameters # Cluster parameters Phi = distribution.compute_phi_from_parents(u_parents[1:]) # Contributions/weights/probabilities P = u_parents[0][0] phi = list() for ind in range(len(Phi)): # Compute element-wise product and then sum over K clusters. # Note that the dimensions aren't perfectly aligned because # the cluster dimension (K) may be arbitrary for phi, and phi # also has dimensions (Dd,..,D0) of the parameters. # Shape(phi) = [Nn,..,K,..,N0,Dd,..,D0] # Shape(p) = [Nn,..,N0,K] # Shape(result) = [Nn,..,N0,Dd,..,D0] # General broadcasting rules apply for Nn,..,N0, that is, # preceding dimensions may be missing or dimension may be # equal to one. Probably, shape(phi) has lots of missing # dimensions and/or dimensions that are one. if cluster_plate < 0: cluster_axis = cluster_plate - distribution.ndims[ind] #else: # cluster_axis = cluster_plate # Move cluster axis to the last: # Shape(phi) = [Nn,..,N0,Dd,..,D0,K] phi.append(utils.moveaxis(Phi[ind], cluster_axis, -1)) # Add axes to p: # Shape(p) = [Nn,..,N0,K,1,..,1] p = utils.add_trailing_axes(P, distribution.ndims[ind]) # Move cluster axis to the last: # Shape(p) = [Nn,..,N0,1,..,1,K] p = utils.moveaxis(p, -(distribution.ndims[ind]+1), -1) #print('Mixture.compute_phi, p:', np.sum(p, axis=-1)) #print('mixture.compute_phi shapes:') #print(np.shape(p)) #print(np.shape(phi[ind])) # Now the shapes broadcast perfectly and we can sum # p*phi over the last axis: # Shape(result) = [Nn,..,N0,Dd,..,D0] phi[ind] = utils.sum_product(p, phi[ind], axes_to_sum=-1) return phi @staticmethod def compute_g_from_parents(u_parents): # Compute weighted average of g over the clusters. # Shape(g) = [Nn,..,K,..,N0] # Shape(p) = [Nn,..,N0,K] # Shape(result) = [Nn,..,N0] # Compute g for clusters: # Shape(g) = [Nn,..,K,..,N0] g = distribution.compute_g_from_parents(u_parents[1:]) # Move cluster axis to last: # Shape(g) = [Nn,..,N0,K] g = utils.moveaxis(g, cluster_plate, -1) # Cluster assignments/contributions/probabilities/weights: # Shape(p) = [Nn,..,N0,K] p = u_parents[0][0] # Weighted average of g over the clusters. As p and g are # properly aligned, you can just sum p*g over the last # axis and utilize broadcasting: # Shape(result) = [Nn,..,N0] #print('mixture.compute_g_from_parents p and g:', np.shape(p), np.shape(g)) g = utils.sum_product(p, g, axes_to_sum=-1) #print('mixture.compute_g_from_parents g:', np.sum(g), np.shape(g)) return g @staticmethod def compute_u_and_g(phi, mask=True): return distribution.compute_u_and_g(phi, mask=mask) @staticmethod def compute_fixed_u_and_f(x): """ Compute u(x) and f(x) for given x. """ return distribution.compute_fixed_u_and_f(x) @staticmethod def compute_message(index, u, u_parents): """ . """ #print('Mixture.compute_message:') if index == 0: # Shape(phi) = [Nn,..,K,..,N0,Dd,..,D0] # Shape(L) = [Nn,..,K,..,N0] # Shape(u) = [Nn,..,N0,Dd,..,D0] # Shape(result) = [Nn,..,N0,K] # Compute g: # Shape(g) = [Nn,..,K,..,N0] g = distribution.compute_g_from_parents(u_parents[1:]) # Reshape(g): # Shape(g) = [Nn,..,N0,K] g = utils.moveaxis(g, cluster_plate, -1) # Compute phi: # Shape(phi) = [Nn,..,K,..,N0,Dd,..,D0] phi = distribution.compute_phi_from_parents(u_parents[1:]) # Reshape phi: # Shape(phi) = [Nn,..,N0,K,Dd,..,D0] for ind in range(len(phi)): phi[ind] = utils.moveaxis(phi[ind], cluster_plate-distribution.ndims[ind], -1-distribution.ndims[ind]) # Reshape u: # Shape(u) = [Nn,..,N0,1,Dd,..,D0] u_self = list() for ind in range(len(u)): u_self.append(np.expand_dims(u[ind], axis=(-1-distribution.ndims[ind]))) # Compute logpdf: # Shape(L) = [Nn,..,N0,K] L = distribution.compute_logpdf(u_self, phi, g, 0) # Sum over other than the cluster dimensions? No! # Hmm.. I think the message passing method will do # that automatically ## print(np.shape(phi[0])) ## print(np.shape(u_self[0])) ## print(np.shape(g)) ## print(np.shape(L)) return [L] elif index >= 1: # Parent index for the distribution used for the # mixture. index = index - 1 # Reshape u: # Shape(u) = [Nn,..1,..,N0,Dd,..,D0] u_self = list() for ind in range(len(u)): if cluster_plate < 0: cluster_axis = cluster_plate - distribution.ndims[ind] else: cluster_axis = cluster_plate u_self.append(np.expand_dims(u[ind], axis=cluster_axis)) # Message from the mixed distribution m = distribution.compute_message(index, u_self, u_parents[1:]) # Weigh the messages with the responsibilities for i in range(len(m)): # Shape(m) = [Nn,..,K,..,N0,Dd,..,D0] # Shape(p) = [Nn,..,N0,K] # Shape(result) = [Nn,..,K,..,N0,Dd,..,D0] # Number of axes for the variable dimensions for # the parent message. D = distribution.ndims_parents[index][i] # Responsibilities for clusters are the first # parent's first moment: # Shape(p) = [Nn,..,N0,K] p = u_parents[0][0] # Move the cluster axis to the proper place: # Shape(p) = [Nn,..,K,..,N0] p = utils.moveaxis(p, -1, cluster_plate) # Add axes for variable dimensions to the contributions # Shape(p) = [Nn,..,K,..,N0,1,..,1] p = utils.add_trailing_axes(p, D) if cluster_plate < 0: # Add the variable dimensions cluster_axis = cluster_plate - D # Add axis for clusters: # Shape(m) = [Nn,..,1,..,N0,Dd,..,D0] #m[i] = np.expand_dims(m[i], axis=cluster_axis) # # TODO: You could do summing here already so that # you wouldn't compute huge matrices as # intermediate result. Use einsum. ## print(np.shape(m[i])) ## print(np.shape(p)) # Compute the message contributions for each # cluster: # Shape(result) = [Nn,..,K,..,N0,Dd,..,D0] m[i] = m[i] * p #print(np.shape(m[i])) return m @staticmethod def compute_dims(*parents): """ Compute the dimensions of phi and u. """ return distribution.compute_dims(*parents[1:]) def __init__(self, z, *args, **kwargs): # Check for constant mu if np.isscalar(z) or isinstance(z, np.ndarray): z = ConstantCategorical(z) # Construct super().__init__(z, *args, **kwargs) def plates_to_parent(self, index): if index == 0: return self.plates else: if cluster_plate < 0: plates = list(self.plates) if cluster_plate < 0: k = len(self.plates) + cluster_plate + 1 else: k = cluster_plate plates.insert(k, self.parents[0].dims[0][0]) plates = tuple(plates) #print('plates_to_parent', cluster_plate, plates) ## plates = (self.plates[:cluster_plate] + ## self.parents[0].dims[0] + ## self.plates[cluster_plate:]) return plates def integrated_logpdf_from_parents(self, x, index): """ Approximates the posterior predictive pdf \int p(x|parents) q(parents) dparents in log-scale as \int q(parents_i) exp( \int q(parents_\i) \log p(x|parents) dparents_\i ) dparents_i.""" if index == 0: # Integrate out the cluster assignments # First, integrate the cluster parameters in log-scale # compute_logpdf(cls, u, phi, g, f): # Shape(x) = [M1,..,Mm,N1,..,Nn,D1,..,Dd] # Add the cluster axis to x: # Shape(x) = [M1,..,Mm,N1,..,1,..,Nn,D1,..,Dd] cluster_axis = cluster_plate - distribution.ndim_observations x = np.expand_dims(x, axis=cluster_axis) u_parents = self.moments_from_parents() # Shape(u) = [M1,..,Mm,N1,..,1,..,Nn,D1,..,Dd] # Shape(f) = [M1,..,Mm,N1,..,1,..,Nn] (u, f) = distribution.compute_fixed_u_and_f(x) # Shape(phi) = [N1,..,K,..,Nn,D1,..,Dd] phi = distribution.compute_phi_from_parents(u_parents[1:]) # Shape(g) = [N1,..,K,..,Nn] g = distribution.compute_g_from_parents(u_parents[1:]) # Shape(lpdf) = [M1,..,Mm,N1,..,K,..,Nn] lpdf = distribution.compute_logpdf(u, phi, g, f) # From logpdf to pdf, but avoid over/underflow lpdf_max = np.max(lpdf, axis=cluster_plate, keepdims=True) pdf = np.exp(lpdf-lpdf_max) # Move cluster axis to be the last: # Shape(pdf) = [M1,..,Mm,N1,..,Nn,K] pdf = utils.moveaxis(pdf, cluster_plate, -1) #print('integrated_logpdf', pdf) # Cluster assignments/probabilities/weights # Shape(p) = [N1,..,Nn,K] p = u_parents[0][0] #self.parents[0].show() #print('integrated_logpdf, p:', p) # Weighted average. TODO/FIXME: Use einsum! # Shape(pdf) = [M1,..,Mm,N1,..,Nn] pdf = np.sum(pdf * p, axis=cluster_plate) #print('integrated_logpdf', pdf) # Back to log-scale (add the overflow fix!) lpdf_max = np.squeeze(lpdf_max, axis=cluster_plate) lpdf = np.log(pdf) + lpdf_max return lpdf raise NotImplementedError() return _Mixture ## def show(self): ## p = self.u[0] #np.exp(self.phi[0]) ## #p /= np.sum(p, axis=-1, keepdims=True) ## print("Categorical(p)") ## print(" p = ") ## print(p) ## def observe(self, x): ## # TODO: You could check that x has proper dimensions ## x = np.array(x, dtype=np.int) ## # Initial array of zeros ## d = self.dims[0][0] ## self.u[0] = np.zeros(np.shape(x)+(d,)) ## # Compute indices ## x += d*np.arange(np.size(x),dtype=int).reshape(np.shape(x)) ## x = x[...,np.newaxis] ## # Set 1 to elements corresponding to the observations ## np.put(self.u[0], x, 1) ## self.show() ## self.fix_u_and_f(self.u, 0) bayespy-0.1/bayespy/_nodes_dev/constant.py0000664000175000017500000000403712174013072022533 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np from .node import Node def Constant(distribution): class _Constant(Node): @staticmethod def compute_fixed_moments(x): """ Compute u(x) for given x. """ return distribution.compute_fixed_moments(x) @staticmethod def compute_fixed_u_and_f(x): """ Compute u(x) and f(x) for given x. """ return distribution.compute_fixed_u_and_f(x) def __init__(self, x, **kwargs): # Compute moments self.u = distribution.compute_fixed_moments(x) # Dimensions of the moments dims = distribution.compute_dims_from_values(x) # Number of plate axes plates_ndim = np.ndim(x) - distribution.ndim_observations plates = np.shape(x)[:plates_ndim] # Parent constructor super().__init__(dims=dims, plates=plates, **kwargs) def get_moments(self): return self.u return _Constant bayespy-0.1/bayespy/_nodes_dev/dot.py0000664000175000017500000001556312174013072021476 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np from bayespy.utils import utils from .node import Node class Dot(Node): # This node satisfies Normal-protocol to children and # Gaussian-protocol to parents # This node is deterministic, just handles message processing. # y(i0,i1,...,in) = sum_d prod_k xk(i0,i1,...,in,d) def __init__(self, *args, **kwargs): # For now, do not use plates other than from the parents, # although it would be possible (it would mean that you'd # create several "datasets" with identical "PCA # distribution"). Maybe it is better to create such plates in # children nodes? plates = [] for x in args: # Convert constant matrices to nodes if np.isscalar(x) or isinstance(x, np.ndarray): x = NodeConstantGaussian(x) # Dimensionality of the Gaussian(s). You should check that # all the parents have the same dimensionality! self.d = x.dims[0] # Check consistency of plates (broadcasting rules!) for ind in range(min(len(plates),len(x.plates))): if plates[-ind-1] == 1: plates[-ind-1] = x.plates[-ind-1] elif x.plates[-ind-1] != 1 and plates[-ind-1] != x.plates[-ind-1]: raise Exception('Plates do not match') # Add new extra plates plates = list(x.plates[:(len(x.plates)-len(plates))]) + plates Node.__init__(self, *args, plates=tuple(plates), dims=[(),()], **kwargs) def get_moments(self): if len(self.parents) == 0: return [0, 0] str1 = '...i' + ',...i' * (len(self.parents)-1) str2 = '...ij' + ',...ij' * (len(self.parents)-1) u1 = list() u2 = list() for parent in self.parents: u = parent.message_to_child() u1.append(u[0]) u2.append(u[1]) x = [np.einsum(str1, *u1), np.einsum(str2, *u2)] return x def get_parameters(self): # Compute mean and variance u = self.get_moments() u[1] -= u[0]**2 return u def get_message(self, index, u_parents): (m, mask) = self.message_from_children() parent = self.parents[index] # Compute both messages for i in range(2): # Add extra axes to the message from children #m_shape = np.shape(m[i]) + (1,) * (i+1) #m[i] = np.reshape(m[i], m_shape) # Put masked elements to zero np.copyto(m[i], 0, where=np.logical_not(mask)) # Add extra axes to the mask from children #mask_shape = np.shape(mask) + (1,) * (i+1) #mask_i = np.reshape(mask, mask_shape) #mask_i = mask m[i] = utils.add_trailing_axes(m[i], i+1) #for k in range(i+1): #m[i] = np.expand_dims(m[i], axis=-1) #mask_i = np.expand_dims(mask_i, axis=-1) # List of elements to multiply together A = [m[i]] for k in range(len(u_parents)): if k != index: A.append(u_parents[k][i]) # Find out which axes are summed over. Also, full_shape = utils.broadcasted_shape_from_arrays(*A) axes = utils.axes_to_collapse(full_shape, parent.get_shape(i)) # Compute the multiplier for cancelling the # plate-multiplier. Because we are summing over the # dimensions already in this function (for efficiency), we # need to cancel the effect of the plate-multiplier # applied in the message_to_parent function. r = 1 for j in axes: r *= full_shape[j] # Compute dot product (and cancel plate-multiplier) m[i] = utils.sum_product(*A, axes_to_sum=axes, keepdims=True) / r # Compute the mask s = utils.axes_to_collapse(np.shape(mask), parent.plates) mask = np.any(mask, axis=s, keepdims=True) mask = utils.squeeze_to_dim(mask, len(parent.plates)) return (m, mask) def OLD_get_message(self, index, u_parents): (m, mask) = self.message_from_children() parent = self.parents[index] # Compute both messages for i in range(2): # Add extra axes to the message from children #m_shape = np.shape(m[i]) + (1,) * (i+1) #m[i] = np.reshape(m[i], m_shape) # Add extra axes to the mask from children mask_shape = np.shape(mask) + (1,) * (i+1) mask_i = np.reshape(mask, mask_shape) mask_i = mask for k in range(i+1): m[i] = np.expand_dims(m[i], axis=-1) mask_i = np.expand_dims(mask_i, axis=-1) # List of elements to multiply together A = [m[i], mask_i] for k in range(len(u_parents)): if k != index: A.append(u_parents[k][i]) # Find out which axes are summed over. Also, because # we are summing over the dimensions already in this # function (for efficiency), we need to cancel the # effect of the plate-multiplier applied in the # message_to_parent function. full_shape = utils.broadcasted_shape_from_arrays(*A) axes = utils.axes_to_collapse(full_shape, parent.get_shape(i)) r = 1 for j in axes: r *= full_shape[j] # Compute dot product m[i] = utils.sum_product(*A, axes_to_sum=axes, keepdims=True) / r # Compute the mask s = utils.axes_to_collapse(np.shape(mask), parent.plates) mask = np.any(mask, axis=s, keepdims=True) mask = utils.squeeze_to_dim(mask, len(parent.plates)) return (m, mask) bayespy-0.1/bayespy/_nodes_dev/normal.py0000664000175000017500000000723012174013072022170 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np from .variable import Variable from .constant import Constant from .gamma import Gamma class Normal(Variable): ndims = (0, 0) ndims_parents = [(0, 0), (0, 0)] # Observations are scalars (0-D): ndim_observations = 0 @staticmethod def compute_fixed_moments(x): """ Compute moments u(x) for given x. """ return [x, x**2] @staticmethod def compute_phi_from_parents(u_parents): phi = [u_parents[1][0] * u_parents[0][0], -u_parents[1][0] / 2] return phi @staticmethod def compute_g_from_parents(u_parents): mu = u_parents[0][0] mumu = u_parents[0][1] tau = u_parents[1][0] log_tau = u_parents[1][1] g = -0.5 * mumu*tau + 0.5 * log_tau return g @staticmethod def compute_u_and_g(phi, mask=True): u0 = -phi[0] / (2*phi[1]) u1 = u0**2 - 1 / (2*phi[1]) u = [u0, u1] g = (-0.5 * u[0] * phi[0] + 0.5 * np.log(-2*phi[1])) return (u, g) @staticmethod def compute_fixed_u_and_f(x): """ Compute u(x) and f(x) for given x. """ u = [x, x**2] f = -np.log(2*np.pi)/2 return (u, f) @staticmethod def compute_message(index, u, u_parents): """ . """ if index == 0: return [u_parents[1][0] * u[0], -0.5 * u_parents[1][0]] elif index == 1: return [-0.5 * (u[1] - 2*u[0]*u_parents[0][0] + u_parents[0][1]), 0.5] raise NotImplementedError() @staticmethod def compute_dims(*parents): """ Compute the dimensions of phi/u. """ # Both moments are scalars, thus, shapes are () return [(), ()] @staticmethod def compute_dims_from_values(x): """ Compute the dimensions of phi and u. """ return [(), ()] # Normal(mu, 1/tau) def __init__(self, mu, tau, plates=(), **kwargs): # Check for constant mu if np.isscalar(mu) or isinstance(mu, np.ndarray): mu = Constant(Normal)(mu) # Check for constant tau if np.isscalar(tau) or isinstance(tau, np.ndarray): tau = Constant(Gamma)(tau) # Construct super().__init__(mu, tau, plates=plates, **kwargs) def show(self, parameters=True, mean=True, mode=True, median=True): mu = self.u[0] tau = self.phi[1] #s2 = self.u[1] - mu**2 print("%s ~ Normal(mu, tau)" % self.name) print(" mu =", mu) #print(mu) print(" tau =", tau) #print(tau) #print("Normal(" + str(mu) + ", " + str(s2) + ")") def plot(self): pass bayespy-0.1/bayespy/_nodes_dev/gamma.py0000664000175000017500000001662212174013072021767 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np import scipy.special as special from .node import Node from .variable import Variable from .constant import Constant class GammaPrior: """ Conjugate prior node for the shape of the gamma distribution. This isn't a distribution but can be used to compute the correct fixed moments, e.g., for Constant node.""" # Number of trailing axes for variable dimensions in # observations. The other axes correspond to plates. ndim_observations = 0 @staticmethod def compute_fixed_moments(a): """ Compute moments for fixed x. """ u0 = a u1 = special.gammaln(a) return [u0, u1] @staticmethod def compute_dims_from_values(a): """ Compute the dimensions of phi or u. """ return [(), ()] class Gamma(Variable): ndims = (0, 0) # Observations/values are scalars (0-dimensional) ndim_observations = 0 @staticmethod def compute_fixed_moments(x): """ Compute moments for fixed x. """ u0 = x u1 = special.gammaln(x) return [u0, u1] @staticmethod def compute_phi_from_parents(u_parents): return [-u_parents[1][0], 1*u_parents[0][0]] @staticmethod def compute_g_from_parents(u_parents): a = u_parents[0][0] gammaln_a = u_parents[0][1] #special.gammaln(a) b = u_parents[1][0] log_b = u_parents[1][1] g = a * log_b - gammaln_a return g @staticmethod def compute_u_and_g(phi, mask=True): log_b = np.log(-phi[0]) u0 = phi[1] / (-phi[0]) u1 = special.digamma(phi[1]) - log_b u = [u0, u1] g = phi[1] * log_b - special.gammaln(phi[1]) return (u, g) @staticmethod def compute_message(index, u, u_parents): """ . """ if index == 0: raise Exception("No analytic solution exists") elif index == 1: return [-u[0], u_parents[0][0]] @staticmethod def compute_dims(*parents): """ Compute the dimensions of phi/u. """ return [(), ()] @staticmethod def compute_dims_from_values(x): """ Compute the dimensions of phi and u. """ return [(), ()] def __init__(self, a, b, plates=(), **kwargs): self.parameter_distributions = (GammaPrior, Gamma) # TODO: USE asarray(a) # Check for constant a if np.isscalar(a) or isinstance(a, np.ndarray): a = Constant(GammaPrior)(a) # Check for constant b if np.isscalar(b) or isinstance(b, np.ndarray): b = Constant(Gamma)(b) #b = NodeConstant([b, np.log(b)], plates=np.shape(b), dims=[(),()]) # Construct super().__init__(a, b, plates=plates, **kwargs) def show(self): a = self.phi[1] b = -self.phi[0] print("%s ~ Gamma(a, b)" % self.name) print(" a =", a) #print(a) print(" b =", b) #print(b) class GammaToDiagonalWishart(Node): ndims = (2, 0) ndims_parents = [None, (2, 0)] # Observations/values are 2-D matrices ndim_observations = 2 ## @staticmethod ## def compute_fixed_moments(Lambda): ## """ Compute moments for fixed x. """ ## return Wishart.compute_fixed_moments(Lambda) ## @staticmethod ## def compute_g_from_parents(u_parents): ## n = u_parents[0][0] ## gammaln_n = u_parents[0][1] ## V = u_parents[1][0] ## logdet_V = u_parents[1][1] ## k = np.shape(V)[-1] ## #k = self.dims[0][0] ## # TODO: Check whether this is correct: ## #g = 0.5*n*logdet_V - special.multigammaln(n/2, k) ## g = 0.5*n*logdet_V - 0.5*k*n*np.log(2) - gammaln_n #special.multigammaln(n/2, k) ## return g ## @staticmethod ## def compute_phi_from_parents(u_parents): ## return [-0.5 * u_parents[1][0], ## 0.5 * u_parents[0][0]] ## @staticmethod ## def compute_u_and_g(phi, mask=True): ## U = utils.m_chol(-phi[0]) ## k = np.shape(phi[0])[-1] ## #k = self.dims[0][0] ## logdet_phi0 = utils.m_chol_logdet(U) ## u0 = phi[1][...,np.newaxis,np.newaxis] * utils.m_chol_inv(U) ## u1 = -logdet_phi0 + utils.m_digamma(phi[1], k) ## u = [u0, u1] ## g = phi[1] * logdet_phi0 - special.multigammaln(phi[1], k) ## return (u, g) ## @staticmethod ## def compute_fixed_u_and_f(Lambda): ## """ Compute u(x) and f(x) for given x. """ ## k = np.shape(Lambda)[-1] ## ldet = utils.m_chol_logdet(utils.m_chol(Lambda)) ## u = [Lambda, ## ldet] ## f = -(k+1)/2 * ldet ## return (u, f) ## @staticmethod ## def message(index, u, u_parents): ## if index == 0: ## raise Exception("No analytic solution exists") ## elif index == 1: ## return (-0.5 * u[0], ## 0.5 * u_parents[0][0]) ## @staticmethod ## def compute_dims(*parents): ## """ Compute the dimensions of phi/u. """ ## # Has the same dimensionality as the second parent. ## return parents[1].dims ## @staticmethod ## def compute_dims_from_values(x): ## """ Compute the dimensions of phi and u. """ ## d = np.shape(x)[-1] ## return [(d,d), ()] def __init__(self, alpha, **kwargs): # Check for constant n if np.isscalar(alpha) or isinstance(alpha, np.ndarray): alpha = Constant(Gamma)(alpha) #ExponentialFamily.__init__(self, n, V, plates=plates, dims=V.dims, **kwargs) k = alpha.plates[-1] super().__init__(alpha, plates=alpha.plates[:-1], dims=[(k,k),()], **kwargs) def get_moments(self): u = self.parents[0].message_to_child() # Form a diagonal matrix from the gamma variables return [np.identity(self.dims[0][0]) * u[0][...,np.newaxis], np.sum(u[1], axis=(-1))] def get_message(self, index, u_parents): (m, mask) = self.message_from_children() # Take the diagonal m[0] = np.einsum('...ii->...i', m[0]) # TODO/FIXME: I think m[1] is wrong.. m[1] = np.reshape(m[1], np.shape(m[1]) + (1,)) # m[1] is ok mask = mask[...,np.newaxis] return (m, mask) bayespy-0.1/bayespy/_nodes_dev/dirichlet.py0000664000175000017500000000767612174013072022665 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np import scipy.special as special from .variable import Variable from .constant import Constant class Dirichlet(Variable): class ConjugatePrior: """ Conjugate prior for Dirichlet distribution. This isn't a distribution but can be used to compute the correct fixed moments, e.g., for Constant node.""" # Number of trailing axes for variable dimensions in # observations. The other axes correspond to plates. ndim_observations = 1 @staticmethod def compute_fixed_moments(alpha): """ Compute moments for fixed x. """ gammaln_sum = special.gammaln(np.sum(alpha, axis=-1)) sum_gammaln = np.sum(special.gammaln(alpha), axis=-1) z = gammaln_sum - sum_gammaln return [alpha, z] @staticmethod def compute_dims_from_values(alpha): """ Compute the dimensions of phi and u. """ d = np.shape(alpha)[-1] return [(d,), ()] ndims = (1,) @staticmethod def compute_phi_from_parents(u_parents): return [u_parents[0][0]] #return [u_parents[0][0].copy()] @staticmethod def compute_g_from_parents(u_parents): return u_parents[0][1] @staticmethod def compute_u_and_g(phi, mask=True): sum_gammaln = np.sum(special.gammaln(phi[0]), axis=-1) gammaln_sum = special.gammaln(np.sum(phi[0], axis=-1)) psi_sum = special.psi(np.sum(phi[0], axis=-1, keepdims=True)) # Moments u0 = special.psi(phi[0]) - psi_sum u = [u0] # G g = gammaln_sum - sum_gammaln return (u, g) @staticmethod def compute_message(index, u, u_parents): """ . """ if index == 0: return [u[0], 1] @staticmethod def compute_dims(*parents): """ Compute the dimensions of phi/u. """ # Has the same dimensionality as the parent for its first # moment. return parents[0].dims[:1] def __init__(self, alpha, plates=(), **kwargs): # Check for constant alpha if np.isscalar(alpha) or isinstance(alpha, np.ndarray): alpha = Constant(self.ConjugatePrior)(alpha) ## gammaln_sum = special.gammaln(np.sum(alpha, axis=-1)) ## sum_gammaln = np.sum(special.gammaln(alpha), axis=-1) ## z = gammaln_sum - sum_gammaln ## d = np.shape(alpha)[-1] ## alpha = NodeConstant([alpha, z], ## plates=np.shape(alpha)[:-1], ## dims=((d,), ())) # Construct super().__init__(alpha, plates=plates, **kwargs) def random(self): raise NotImplementedError() def show(self): alpha = self.phi[0] print("%s ~ Dirichlet(alpha)" % self.name) print(" alpha = ") print(alpha) bayespy-0.1/bayespy/_nodes_dev/gaussian.py0000664000175000017500000000406612174013072022516 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np from bayespy.utils import utils from .variable import Variable #from .constant import Constant #from .wishart import Wishart class Gaussian(Variable): #ndims = (1, 2) #ndims_parents = [(1, 2), (2, 0)] # Observations are vectors (1-D): ndim_observations = 1 # Gaussian(mu, inv(Lambda)) def __init__(self, mu, Lambda, plates=(), **kwargs): ## # Check for constant mu ## if np.isscalar(mu) or isinstance(mu, np.ndarray): ## mu = Constant(Gaussian)(mu) ## # Check for constant Lambda ## if np.isscalar(Lambda) or isinstance(Lambda, np.ndarray): ## Lambda = Constant(Wishart)(Lambda) ## # You could check whether the dimensions of mu and Lambda ## # match (and Lambda is square) ## if Lambda.dims[0][-1] != mu.dims[0][-1]: ## raise Exception("Dimensionalities of mu and Lambda do not match.") # Construct super().__init__(mu, Lambda, plates=plates, **kwargs) bayespy-0.1/bayespy/_nodes_dev/__init__.py0000664000175000017500000000264112174013072022440 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### # Import some most commonly used nodes #from .node import Node #from .variables.variable import Variable #from .variables.constant import Constant from .gaussian import Gaussian from .wishart import Wishart from .normal import Normal from .gamma import Gamma, GammaToDiagonalWishart from .dirichlet import Dirichlet from .categorical import Categorical from .dot import Dot from .mixture import Mixture bayespy-0.1/bayespy/_nodes_dev/CovarianceFunctions.py0000664000175000017500000007144312174013072024652 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import itertools import numpy as np #import scipy as sp import scipy.sparse as sp # prefer CSC format #import scipy.linalg.decomp_cholesky as decomp #import scipy.linalg as linalg #import scipy.special as special #import matplotlib.pyplot as plt #import time #import profile import scipy.spatial.distance as dist import scikits.sparse.distance as spdist import Nodes.ExponentialFamily as ef import utils import imp imp.reload(spdist) imp.reload(ef) imp.reload(utils) # Covariance matrices can be either arrays or matrices so be careful # with products and powers! Use explicit multiply or dot instead of # *-operator. def gp_cov_se(D2, overwrite=False): if overwrite: K = D2 K *= -0.5 np.exp(K, out=K) else: K = np.exp(-0.5*D2) return K def gp_cov_pp2_new(r, d, derivative=False): # Dimension dependent parameter q = 2 j = np.floor(d/2) + q + 1 # Polynomial coefficients a2 = j**2 + 4*j + 3 a1 = 3*j + 6 a0 = 3 # Two parts of the covariance function k1 = (1-r) ** (j+2) k2 = (a2*r**2 + a1*r + 3) # TODO: Check that derivative is 0, 1 or 2! if derivative == 0: # Return covariance return k1 * k2 / 3 dk1 = - (j+2) * (1-r)**(j+1) dk2 = 2*a2*r + a1 if derivative == 1: # Return first derivative of the covariance return (k1 * dk2 + dk1 * k2) / 3 ddk1 = (j+2) * (j+1) * (1-r)**j ddk2 = 2*a2 if derivative == 2: # Return second derivative of the covariance return (ddk1*k2 + 2*dk1*dk2 + k1*ddk2) / 3 def gp_cov_pp2(r, d, gradient=False): # Dimension dependent parameter j = np.floor(d/2) + 2 + 1 # Polynomial coefficients a2 = j**2 + 4*j + 3 a1 = 3*j + 6 a0 = 3 # Two parts of the covariance function k1 = (1-r) ** (j+2) k2 = (a2*r**2 + a1*r + 3) # The covariance function k = k1 * k2 / 3 if gradient: # The gradient w.r.t. r dk = k * (j+2) / (r-1) + k1 * (2*a2*r + a1) / 3 return (k, dk) else: return k def gp_cov_delta(N): # TODO: Use sparse matrices here! if N > 0: #print('in gpcovdelta', N, sp.identity(N).shape) return sp.identity(N) else: # Sparse matrices do not allow zero-length dimensions return np.identity(N) #return np.identity(N) #return np.asmatrix(np.identity(N)) def squared_distance(x1, x2): ## # Reshape arrays to 2-D arrays ## sh1 = np.shape(x1)[:-1] ## sh2 = np.shape(x2)[:-1] ## d = np.shape(x1)[-1] ## x1 = np.reshape(x1, (-1,d)) ## x2 = np.reshape(x2, (-1,d)) (m1,n1) = x1.shape (m2,n2) = x2.shape if m1 == 0 or m2 == 0: D2 = np.empty((m1,m2)) else: # Compute squared Euclidean distance D2 = dist.cdist(x1, x2, metric='sqeuclidean') #D2 = np.asmatrix(D2) # Reshape the result #D2 = np.reshape(D2, sh1 + sh2) return D2 # General rule for the parameters for covariance functions: # # (value, [ [dvalue1, ...], [dvalue2, ...], [dvalue3, ...], ...]) # # For instance, # # k = covfunc_se((1.0, []), (15, [ [1,update_grad] ])) # K = k((x1, [ [dx1,update_grad] ]), (x2, [])) # # Plain values are converted as: # value -> (value, []) def gp_standardize_input(x): if np.size(x) == 0: x = np.reshape(x, (0,0)) elif np.ndim(x) == 0: x = np.reshape(x, (1,1)) elif np.ndim(x) == 1: x = np.reshape(x, (-1,1)) elif np.ndim(x) == 2: x = np.atleast_2d(x) else: raise Exception("Standard GP inputs must be 2-dimensional") return x def gp_preprocess_inputs(x1,x2=None): #args = list(args) #if len(args) < 1 or len(args) > 2: #raise Exception("Number of inputs must be one or two") if x2 is None: x1 = gp_standardize_input(x1) return x1 else: if x1 is x2: x1 = gp_standardize_input(x1) x2 = x1 else: x1 = gp_standardize_input(x1) x2 = gp_standardize_input(x2) return (x1, x2) #return args ## def gp_preprocess_inputs(x1,x2=None): ## #args = list(args) ## #if len(args) < 1 or len(args) > 2: ## #raise Exception("Number of inputs must be one or two") ## if x2 is not None: len(args) == 2: ## if args[0] is args[1]: ## args[0] = gp_standardize_input(args[0]) ## args[1] = args[0] ## else: ## args[1] = gp_standardize_input(args[1]) ## args[0] = gp_standardize_input(args[0]) ## else: ## args[0] = gp_standardize_input(args[0]) ## return args # TODO: # General syntax for these covariance functions: # covfunc(hyper1, # hyper2, # ... # hyperN, # x1, # x2=None, # gradient=list_of_booleans_for_each_hyperparameter) def covfunc_zeros(x1, x2=None, gradient=False): inputs = gp_preprocess_inputs(*inputs) # Compute distance and covariance matrix if x2 is None: x1 = gp_preprocess_inputs(x1) # Only variance vector asked N = np.shape(x1)[0] # TODO: Use sparse matrices! K = np.zeros(N) #K = np.asmatrix(np.zeros((N,1))) else: (x1,x2) = gp_preprocess_inputs(x1,x2) # Full covariance matrix asked #x1 = inputs[0] #x2 = inputs[1] # Number of inputs x1 N1 = np.shape(x1)[0] N2 = np.shape(x2)[0] # TODO: Use sparse matrices! K = np.zeros((N1,N2)) #K = np.asmatrix(np.zeros((N1,N2))) if gradient is not False: return (K, []) else: return K def covfunc_delta(amplitude, x1, x2=None, gradient=False): # Make sure that amplitude is a scalar, not an array object amplitude = utils.array_to_scalar(amplitude) ## if gradient: ## gradient_amplitude = gradient[0] ## else: ## gradient_amplitude = [] ## inputs = gp_preprocess_inputs(*inputs) # Compute distance and covariance matrix if x2 is None: x1 = gp_preprocess_inputs(x1) # Only variance vector asked #x = inputs[0] N = np.shape(x1)[0] K = np.ones(N) * amplitude**2 else: (x1,x2) = gp_preprocess_inputs(x1,x2) # Full covariance matrix asked #x1 = inputs[0] #x2 = inputs[1] # Number of inputs x1 N1 = np.shape(x1)[0] # x1 == x2? if x1 is x2: delta = True # Delta covariance # # FIXME: Broadcasting doesn't work with sparse matrices, # so must use scalar multiplication K = gp_cov_delta(N1) * amplitude**2 #K = gp_cov_delta(N1).multiply(amplitude**2) else: delta = False # Number of inputs x2 N2 = np.shape(x2)[0] # Zero covariance if N1 > 0 and N2 > 0: K = sp.csc_matrix((N1,N2)) else: K = np.zeros((N1,N2)) # Gradient w.r.t. amplitude if gradient: # FIXME: Broadcasting doesn't work with sparse matrices, # so must use scalar multiplication gradient_amplitude = K*(2/amplitude) print("noise grad", gradient_amplitude) return (K, (gradient_amplitude,)) else: return K def covfunc_pp2(amplitude, lengthscale, x1, x2, gradient=False): # Make sure that hyperparameters are scalars, not an array objects amplitude = utils.array_to_scalar(amplitude) lengthscale = utils.array_to_scalar(lengthscale) #amplitude = theta[0] #lengthscale = theta[1] ## if gradient: ## gradient_amplitude = gradient[0] ## gradient_lengthscale = gradient[1] ## else: ## gradient_amplitude = [] ## gradient_lengthscale = [] ## inputs = gp_preprocess_inputs(*inputs) # Compute covariance matrix if x2 is None: x1 = gp_preprocess_inputs(x1) # Compute variance vector K = np.ones(np.shape(x)[:-1]) K *= amplitude**2 # Compute gradient w.r.t. lengthscale if gradient: gradient_lengthscale = np.zeros(np.shape(x1)[:-1]) else: (x1,x2) = gp_preprocess_inputs(x1,x2) # Compute (sparse) distance matrix if x1 is x2: x1 = inputs[0] / (lengthscale) x2 = x1 D2 = spdist.pdist(x1, 1.0, form="full", format="csc") else: x1 = inputs[0] / (lengthscale) x2 = inputs[1] / (lengthscale) D2 = spdist.cdist(x1, x2, 1.0, format="csc") r = np.sqrt(D2.data) N1 = np.shape(x1)[0] N2 = np.shape(x2)[0] # Compute the covariances if gradient: (k, dk) = gp_cov_pp2(r, np.shape(x1)[-1], gradient=True) else: k = gp_cov_pp2(r, np.shape(x1)[-1]) k *= amplitude**2 # Compute gradient w.r.t. lengthscale if gradient: if N1 >= 1 and N2 >= 1: dk *= r * (-amplitude**2 / lengthscale) gradient_lengthscale = sp.csc_matrix((dk, D2.indices, D2.indptr), shape=(N1,N2)) else: gradient_lengthscale = np.empty((N1,N2)) # Form sparse covariance matrix if N1 >= 1 and N2 >= 1: ## K = sp.csc_matrix((k, ij), shape=(N1,N2)) K = sp.csc_matrix((k, D2.indices, D2.indptr), shape=(N1,N2)) else: K = np.empty((N1, N2)) #print(K.__class__) # Gradient w.r.t. amplitude if gradient: gradient_amplitude = K * (2 / amplitude) # Return values if gradient: print("pp2 grad", gradient_lengthscale) return (K, (gradient_amplitude, gradient_lengthscale)) else: return K def covfunc_se(amplitude, lengthscale, x1, x2=None, gradient=False): # Make sure that hyperparameters are scalars, not an array objects amplitude = utils.array_to_scalar(amplitude) lengthscale = utils.array_to_scalar(lengthscale) # Compute covariance matrix if x2 is None: x1 = gp_preprocess_inputs(x1) #x = inputs[0] # Compute variance vector N = np.shape(x1)[0] K = np.ones(N) np.multiply(K, amplitude**2, out=K) # Compute gradient w.r.t. lengthscale if gradient: # TODO: Use sparse matrices? gradient_lengthscale = np.zeros(N) else: (x1,x2) = gp_preprocess_inputs(x1,x2) x1 = x1 / (lengthscale) x2 = x2 / (lengthscale) # Compute distance matrix K = squared_distance(x1, x2) # Compute gradient partly if gradient: gradient_lengthscale = np.divide(K, lengthscale) # Compute covariance matrix gp_cov_se(K, overwrite=True) np.multiply(K, amplitude**2, out=K) # Compute gradient w.r.t. lengthscale if gradient: gradient_lengthscale *= K # Gradient w.r.t. amplitude if gradient: gradient_amplitude = K * (2 / amplitude) # Return values if gradient: print("se grad", gradient_amplitude, gradient_lengthscale) return (K, (gradient_amplitude, gradient_lengthscale)) else: return K class CovarianceFunctionWrapper(): def __init__(self, covfunc, *params): # Parse parameter values and their gradients to separate lists self.covfunc = covfunc self.params = list(params) self.gradient_params = list() ## print(params) for ind in range(len(params)): if isinstance(params[ind], tuple): # Parse the value and the list of gradients from the # form: # ([value, ...], [ [grad1, ...], [grad2, ...], ... ]) self.gradient_params.append(params[ind][1]) self.params[ind] = params[ind][0][0] else: # No gradients, parse from the form: # [value, ...] self.gradient_params.append([]) self.params[ind] = params[ind][0] def fixed_covariance_function(self, *inputs, gradient=False): # What if this is called several times?? if gradient: ## grads = [[grad[0] for grad in self.gradient_params[ind]] ## for ind in range(len(self.gradient_params))] ## (K, dK) = self.covfunc(self.params, ## *inputs, ## gradient=self.gradient_params) arguments = tuple(self.params) + tuple(inputs) (K, dK) = self.covfunc(*arguments, gradient=True) ## (K, dK) = self.covfunc(self.params, ## *inputs, ## gradient=grads) DK = [] for ind in range(len(dK)): # Gradient w.r.t. covariance function's ind-th # hyperparameter dk = dK[ind] # Chain rule: Multiply by the gradient of the # hyperparameter w.r.t. parent node and append the # list DK: # DK = [ (dx1_1, callback), ..., (dx1_n, callback) ] for grad in self.gradient_params[ind]: #print(grad[0]) #print(grad[1:]) #print(dk) if sp.issparse(dk): print(dk.shape) print(grad[0].shape) DK += [ [dk.multiply(grad[0])] + grad[1:] ] else: DK += [ [np.multiply(dk,grad[0])] + grad[1:] ] #DK += [ [np.multiply(grad[0], dk)] + grad[1:] ] ## DK += [ (np.multiply(grad, dk),) + grad[1:] ## for grad in self.gradient_params[ind] ] ## for grad in self.gradient_params[ind]: ## DK += ( (np.multiply(grad, dk),) + grad[1:] ) ## DK = [] ## for ind in range(len(dK)): ## for (grad, dk) in zip(self.gradient_params[ind], dK[ind]): ## DK += [ [dk] + grad[1:] ] K = [K] return (K, DK) else: arguments = tuple(self.params) + tuple(inputs) #print(arguments) K = self.covfunc(*arguments, gradient=False) return [K] class CovarianceFunction(ef.Node): def __init__(self, covfunc, *args, **kwargs): self.covfunc = covfunc params = list(args) for i in range(len(args)): # Check constant parameters if utils.is_numeric(args[i]): params[i] = ef.NodeConstant([np.asanyarray(args[i])], dims=[np.shape(args[i])]) # TODO: Parameters could be constant functions? :) ef.Node.__init__(self, *params, dims=[(np.inf, np.inf)], **kwargs) def __call__(self, x1, x2): """ Compute covariance matrix for inputs x1 and x2. """ covfunc = self.message_to_child() return covfunc(x1, x2)[0] def message_to_child(self, gradient=False): params = [parent.message_to_child(gradient=gradient) for parent in self.parents] covfunc = self.get_fixed_covariance_function(*params) return covfunc def get_fixed_covariance_function(self, *params): get_cov_func = CovarianceFunctionWrapper(self.covfunc, *params) return get_cov_func.fixed_covariance_function ## def covariance_function(self, *params): ## # Parse parameter values and their gradients to separate lists ## params = list(params) ## gradient_params = list() ## print(params) ## for ind in range(len(params)): ## if isinstance(params[ind], tuple): ## # Parse the value and the list of gradients from the ## # form: ## # ([value, ...], [ [grad1, ...], [grad2, ...], ... ]) ## gradient_params.append(params[ind][1]) ## params[ind] = params[ind][0][0] ## else: ## # No gradients, parse from the form: ## # [value, ...] ## gradient_params.append([]) ## params[ind] = params[ind][0] ## # This gradient_params changes mysteriously.. ## print('grad_params before') ## if isinstance(self, SquaredExponential): ## print(gradient_params) ## def cov(*inputs, gradient=False): ## if gradient: ## print('grad_params after') ## print(gradient_params) ## grads = [[grad[0] for grad in gradient_params[ind]] ## for ind in range(len(gradient_params))] ## print('CovarianceFunction.cov') ## #if isinstance(self, SquaredExponential): ## #print(self.__class__) ## #print(grads) ## (K, dK) = self.covfunc(params, ## *inputs, ## gradient=grads) ## for ind in range(len(dK)): ## for (grad, dk) in zip(gradient_params[ind], dK[ind]): ## grad[0] = dk ## K = [K] ## dK = [] ## for grad in gradient_params: ## dK += grad ## return (K, dK) ## else: ## K = self.covfunc(params, ## *inputs, ## gradient=False) ## return [K] ## return cov class Sum(CovarianceFunction): def __init__(self, *args, **kwargs): CovarianceFunction.__init__(self, None, *args, **kwargs) def get_fixed_covariance_function(self, *covfunc_parents): def covfunc(*inputs, gradient=False): K_sum = None if gradient: dK_sum = list() for k in covfunc_parents: if gradient: (K, dK) = k(*inputs, gradient=gradient) print("dK in sum", dK) dK_sum += dK #print("dK_sum in sum", dK_sum) else: K = k(*inputs, gradient=gradient) if K_sum is None: K_sum = K[0] else: try: K_sum += K[0] except: # You have to do this way, for instance, if # K_sum is sparse and K[0] is dense. K_sum = K_sum + K[0] if gradient: #print("dK_sum on: ", dK_sum) #print('covsum', dK_sum) return ([K_sum], dK_sum) else: return [K_sum] return covfunc class Delta(CovarianceFunction): def __init__(self, amplitude, **kwargs): CovarianceFunction.__init__(self, covfunc_delta, amplitude, **kwargs) class Zeros(CovarianceFunction): def __init__(self, **kwargs): CovarianceFunction.__init__(self, covfunc_zeros, **kwargs) class SquaredExponential(CovarianceFunction): def __init__(self, amplitude, lengthscale, **kwargs): CovarianceFunction.__init__(self, covfunc_se, amplitude, lengthscale, **kwargs) class PiecewisePolynomial2(CovarianceFunction): def __init__(self, amplitude, lengthscale, **kwargs): CovarianceFunction.__init__(self, covfunc_pp2, amplitude, lengthscale, **kwargs) # TODO: Rename to Blocks or Joint ? class Multiple(CovarianceFunction): def __init__(self, covfuncs, **kwargs): self.d = len(covfuncs) #self.sparse = sparse parents = [covfunc for row in covfuncs for covfunc in row] CovarianceFunction.__init__(self, None, *parents, **kwargs) def get_fixed_covariance_function(self, *covfuncs): def cov(*inputs, gradient=False): # Computes the covariance matrix from blocks which all # have their corresponding covariance functions if len(inputs) < 2: # For one input, return the variance vector instead of # the covariance matrix x1 = inputs[0] # Collect variance vectors from the covariance # functions corresponding to the diagonal blocks K = [covfuncs[i*self.d+i](x1[i], gradient=gradient)[0] for i in range(self.d)] # Form the variance vector from the collected vectors if gradient: raise Exception('Gradient not yet implemented.') else: ## print("in cov multiple") ## for (k,kf) in zip(K,covfuncs): ## print(np.shape(k), k.__class__, kf) #K = np.vstack(K) K = np.concatenate(K) else: x1 = inputs[0] x2 = inputs[1] # Collect the covariance matrix (and possibly # gradients) from each block. #print('cov mat collection begins') K = [[covfuncs[i*self.d+j](x1[i], x2[j], gradient=gradient) for j in range(self.d)] for i in range(self.d)] #print('cov mat collection ends') # Remove matrices that have zero length dimensions? if gradient: K = [[K[i][j] for j in range(self.d) if np.shape(K[i][j][0][0])[1] != 0] for i in range(self.d) if np.shape(K[i][0][0][0])[0] != 0] else: K = [[K[i][j] for j in range(self.d) if np.shape(K[i][j][0])[1] != 0] for i in range(self.d) if np.shape(K[i][0][0])[0] != 0] n_blocks = len(K) #print("nblocks", n_blocks) #print("K", K) # Check whether all blocks are sparse is_sparse = True for i in range(n_blocks): for j in range(n_blocks): if gradient: A = K[i][j][0][0] else: A = K[i][j][0] if not sp.issparse(A): is_sparse = False if gradient: ## Compute the covariance matrix and the gradients # Create block matrices of zeros. This helps in # computing the gradient. if is_sparse: # Empty sparse matrices. Some weird stuff here # because sparse matrices can't have zero # length dimensions. Z = [[sp.csc_matrix(np.shape(K[i][j][0][0])) for j in range(n_blocks)] for i in range(n_blocks)] else: # Empty dense matrices Z = [[np.zeros(np.shape(K[i][j][0][0])) for j in range(n_blocks)] for i in range(n_blocks)] ## for j in range(self.d)] ## for i in range(self.d)] # Compute gradients block by block dK = list() for i in range(n_blocks): for j in range(n_blocks): # Store the zero block z_old = Z[i][j] # Go through the gradients for the (i,j) # block for dk in K[i][j][1]: # Keep other blocks at zero and set # the gradient to (i,j) block. Form # the matrix from blocks if is_sparse: Z[i][j] = dk[0] dk[0] = sp.bmat(Z).tocsc() else: if sp.issparse(dk[0]): Z[i][j] = dk[0].toarray() else: Z[i][j] = dk[0] #print("Z on:", Z) dk[0] = np.asarray(np.bmat(Z)) # Append the computed gradient matrix # to the list of gradients dK.append(dk) # Restore the zero block Z[i][j] = z_old ## Compute the covariance matrix but not the ## gradients if is_sparse: # Form the full sparse covariance matrix from # blocks. Ignore blocks having a zero-length # axis because sparse matrices consider zero # length as an invalid shape (BUG IN SCIPY?). K = [[K[i][j][0][0] for j in range(n_blocks)] for i in range(n_blocks)] K = sp.bmat(K).tocsc() else: # Form the full dense covariance matrix from # blocks. Transform sparse blocks to dense # blocks. K = [[K[i][j][0][0] if not sp.issparse(K[i][j][0][0]) else K[i][j][0][0].toarray() for j in range(n_blocks)] for i in range(n_blocks)] K = np.asarray(np.bmat(K)) else: ## Compute the covariance matrix but not the ## gradients if is_sparse: # Form the full sparse covariance matrix from # blocks. Ignore blocks having a zero-length # axis because sparse matrices consider zero # length as an invalid shape (BUG IN SCIPY?). K = [[K[i][j][0] for j in range(n_blocks)] for i in range(n_blocks)] K = sp.bmat(K).tocsc() else: # Form the full dense covariance matrix from # blocks. Transform sparse blocks to dense # blocks. K = [[K[i][j][0] if not sp.issparse(K[i][j][0]) else K[i][j][0].toarray() for j in range(n_blocks)] for i in range(n_blocks)] K = np.asarray(np.bmat(K)) if gradient: return ([K], dK) else: return [K] return cov bayespy-0.1/bayespy/_nodes_dev/variable.py0000664000175000017500000004222412174013072022467 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np from bayespy.utils import utils from .node import Node # nodes/ # gp/ # # nodes.random.Gaussian # nodes.random.Wishart # nodes.random.Mixture # nodes.deterministic.Dot # nodes.deterministic.Constant # or # nodes.variable.Gaussian # nodes.variable.Wishart # nodes.variable.Dot # nodes.variable.Constant # nodes.gp.GaussianProcess # nodes.gp.Constant #print(Node) #print(Node.Node) # MAP/ML: # X = MAP(prior=Gaussian(mu,Cov)) # X = ML() # or: # X = Delta(prior=Gaussian(mu,Cov)) class Variable(Node): # Overwrite this ndims = None ndims_parents = None parameter_distributions = None @classmethod def compute_logpdf(cls, u, phi, g, f): """ Compute E[log p(X)] given E[u], E[phi], E[g] and E[f]. Does not sum over plates.""" # TODO/FIXME: Should I take into account what is latent or # observed, or what is even totally ignored (by the mask). L = g + f for (phi_i, u_i, ndims_i) in zip(phi, u, cls.ndims): #for (phi_i, u_i, len_dims_i) in zip(phi, u, len_dims): # Axes to sum (dimensions of the variable, not the plates) axis_sum = tuple(range(-ndims_i,0)) #axis_sum = tuple(range(-len_dims_i,0)) # Compute the term # TODO/FIXME: Use einsum! L = L + np.sum(phi_i * u_i, axis=axis_sum) return L @staticmethod def compute_fixed_parameter_moments(*args): """ Compute the moments of the distribution parameters for fixed values.""" raise NotImplementedError() @staticmethod def compute_phi_from_parents(u_parents): """ Compute E[phi] over q(parents) """ raise NotImplementedError() @staticmethod def compute_g_from_parents(u_parents): """ Compute E[g(phi)] over q(parents) """ raise NotImplementedError() @staticmethod def compute_u_and_g(phi, mask=True): """ Compute E[u] and g(phi) for given phi """ raise NotImplementedError() @staticmethod def compute_fixed_u_and_f(x): """ Compute u(x) and f(x) for given x. """ raise NotImplementedError() @staticmethod def compute_message(index, u, u_parents): """ . """ raise NotImplementedError() @staticmethod def compute_dims(*parents): """ Compute the dimensions of phi and u. """ raise NotImplementedError() @staticmethod def compute_dims_from_values(x): """ Compute the dimensions of phi and u. """ raise NotImplementedError() def __init__(self, *args, initialize=True, **kwargs): super().__init__(*args, dims=self.compute_dims(*args), **kwargs) # Natural parameters and moments. Note that the size of the # array self.u or self.phi doesn't actually need to equal # self.dims+self.plates, it will be broadcasted automatically # (although this might be extremely rare: it would mean that # there are identical posterior distributions over # plates). Anyway, for this reason (and some others), we need # to explicityly have the dimensionalities dims and plates? Do # we need dims? Yes, because you need to be able to check what # plates are missing. #self.phi = [np.array([]) for i in range(len(self.dims))] #self.u = [np.array([]) for i in range(len(self.dims))] #self.phi = [np.array(0.0) for i in range(len(self.dims))] #self.u = [np.array(0.0) for i in range(len(self.dims))] axes = len(self.plates)*(1,) self.phi = [utils.nans(axes+dim) for dim in self.dims] self.u = [utils.nans(axes+dim) for dim in self.dims] # self.u = [np.array(0.0) for i in range(len(self.dims))] # Terms for the lower bound (G for latent and F for observed) self.g = 0 self.f = 0 # Not observed self.observed = False # By default, ignore all plates self.mask = False if initialize: self.initialize_from_prior() def get_vb_term(self): return {self.lower_bound_contribution} def get_message(self, index, u_parents): return (self.compute_message(index, self.u, u_parents), self.mask) ## return (self.message(index, u_parents), ## self.mask) def initialize_from_prior(self): if not np.all(self.observed): # Messages from parents #u_parents = [parent.message_to_child() for parent in self.parents] u_parents = self.moments_from_parents() # Update natural parameters using parents self.update_phi_from_parents(u_parents) # Update moments (u, g) = self.compute_u_and_g(self.phi, mask=True) self.update_u_and_g(u, g, mask=True) def initialize_from_parameters(self, *args): # Get the moments of the parameters if they were fixed to the # given values. #u_parents = self.compute_fixed_parameter_moments(*args) u_parents = list() for (ind, x) in enumerate(args): #print(self.parents[ind].__class__) u = self.parameter_distributions[ind].compute_fixed_moments(x) #u = self.parents[ind].compute_fixed_moments(x) #(u, _) = self.parents[ind].compute_fixed_u_and_f(x) u_parents.append(u) # Update natural parameters self.update_phi_from_parents(u_parents) # Update moments # TODO/FIXME: Use the mask of observations! (u, g) = self.compute_u_and_g(self.phi, mask=True) self.update_u_and_g(u, g, mask=True) # TODO: Initialization where you could give the distribution # values in standard parameterization! ## def initialize_from_value(self, x): ## # Update moments from value ## (u, f) = self.compute_fixed_u_and_f(x) ## self.update_u_and_g(u, np.nan, mask=True) ## def initialize_from_random(self): ## self.initialize_from_prior() ## self.initialize_from_value(self.random()) def update(self): if not np.all(self.observed): # Messages from parents u_parents = [parent.message_to_child() for parent in self.parents] # Update natural parameters using parents self.update_phi_from_parents(u_parents) #print('update, phi', self.name, self.phi) # Update natural parameters using children (just add the # messages to phi) for (child,index) in self.children: #print('ExpFam.update:', self.name) # TODO/FIXME: These m are nans.. (m, mask) = child.message_to_parent(index) # Combine masks # # TODO: Maybe you would like to compute a new mask # at every update? self.mask = np.logical_or(self.mask,mask) for i in range(len(self.phi)): ## try: ## # Try exploiting broadcasting rules ## # ## # TODO/FIXME: This has the problem that if phi ## # is updated from parents such that phi is a ## # view into parent's moments thus modifying ## # phi would modify parents moments.. Maybe one ## # should check that nobody else is viewing ## # phi? ## self.phi[i] += m[i] ## except ValueError: ## self.phi[i] = self.phi[i] + m[i] #print('update2, phi', i, self.name, self.phi, m[i]) # TODO/FIXME: You should take into account the # mask when adding the message!!!!??? self.phi[i] = self.phi[i] + m[i] # Mask for plates to update (i.e., unobserved plates) update_mask = np.logical_not(self.observed) # Compute u and g (u, g) = self.compute_u_and_g(self.phi, mask=update_mask) # Update moments self.update_u_and_g(u, g, mask=update_mask) def phi_from_parents(self, gradient=False): # Messages from parents u_parents = [parent.message_to_child(gradient=gradient) for parent in self.parents] # Compute and return phi return self.compute_phi_from_parents(u_parents, gradient=gradient) def update_phi_from_parents(self, u_parents): # This makes correct broadcasting self.phi = self.compute_phi_from_parents(u_parents) #print('update_phi, phi', self.phi) # Make sure phi has the correct number of axes. It makes life # a bit easier elsewhere. for i in range(len(self.phi)): axes = len(self.plates) + self.ndims[i] - np.ndim(self.phi[i]) ## print('update_phi_from_parents:') ## print(np.shape(self.phi[i])) ## print(self.plates) ## print(self.dims[i]) if axes > 0: # Add axes self.phi[i] = utils.add_leading_axes(self.phi[i], axes) elif axes < 0: # Remove extra leading axes first = -(len(self.plates)+self.ndims[i]) sh = np.shape(self.phi[i])[first:] self.phi[i] = np.reshape(self.phi[i], sh) #print(np.shape(self.phi[i])) ## for i in range(len(self.phi)): ## self.phi[i].fill(0) ## try: ## # Try exploiting broadcasting rules ## self.phi[i] += phi[i] ## except ValueError: ## self.phi[i] = self.phi[i] + phi[i] def get_moments(self): return self.u def get_children_vb_bound(self): raise NotImplementedError("Not implemented for " + str(self.__class__)) def start_optimization(self): raise NotImplementedError("Not implemented for " + str(self.__class__)) def stop_optimization(self): raise NotImplementedError("Not implemented for " + str(self.__class__)) def message(self, index, u_parents): raise NotImplementedError("Not implemented for " + str(self.__class__)) def update_u(self, u, mask=True): # Store the computed moments u but do not change moments for # observations, i.e., utilize the mask. for ind in range(len(u)): # Add axes to the mask for the variable dimensions (mask # contains only axes for the plates). u_mask = utils.add_trailing_axes(mask, self.ndims[ind]) # Enlarge self.u[ind] as necessary so that it can store the # broadcasted result. sh = utils.broadcasted_shape_from_arrays(self.u[ind], u[ind], u_mask) self.u[ind] = utils.repeat_to_shape(self.u[ind], sh) # Use mask to update only unobserved plates and keep the # observed as before np.copyto(self.u[ind], u[ind], where=u_mask) # Make sure u has the correct number of dimensions: shape = self.get_shape(ind) ndim = len(shape) ndim_u = np.ndim(self.u[ind]) if ndim > ndim_u: self.u[ind] = utils.add_leading_axes(u[ind], ndim - ndim_u) elif ndim < np.ndim(self.u[ind]): raise Exception("Weird, this shouldn't happen.. :)") def update_u_and_g(self, u, g, mask=True): self.update_u(u, mask=mask) # TODO/FIXME: Apply mask to g too!! self.g = g ## # Store the computed moments u but do not change moments for ## # observations, i.e., utilize the mask. ## for ind in range(len(u)): ## # Add axes to the mask for the variable dimensions (mask ## # contains only axes for the plates). ## u_mask = utils.add_trailing_axes(mask, self.ndims[ind]) ## # Enlarge self.u[ind] as necessary so that it can store the ## # broadcasted result. ## sh = utils.broadcasted_shape_from_arrays(self.u[ind], u[ind], u_mask) ## self.u[ind] = utils.repeat_to_shape(self.u[ind], sh) ## # Use mask to update only unobserved plates and keep the ## # observed as before ## np.copyto(self.u[ind], ## u[ind], ## where=u_mask) ## # TODO/FIXME: Apply mask to g too!! ## self.g = g ## # Make sure u has the correct number of dimensions: ## shape = self.get_shape(ind) ## ndim = len(shape) ## ndim_u = np.ndim(self.u[ind]) ## if ndim > ndim_u: ## self.u[ind] = utils.add_leading_axes(u[ind], ndim - ndim_u) ## elif ndim < np.ndim(self.u[ind]): ## raise Exception("Weird, this shouldn't happen.. :)") def lower_bound_contribution(self, gradient=False): # Compute E[ log p(X|parents) - log q(X) ] over q(X)q(parents) # Messages from parents u_parents = [parent.message_to_child() for parent in self.parents] phi = self.compute_phi_from_parents(u_parents) # G from parents L = self.compute_g_from_parents(u_parents) # L = g # G for unobserved variables (ignored variables are handled # properly automatically) latent_mask = np.logical_not(self.observed) #latent_mask = np.logical_and(self.mask, np.logical_not(self.observed)) # F for observed, G for latent L = L + np.where(self.observed, self.f, -self.g) for (phi_p, phi_q, u_q, dims) in zip(phi, self.phi, self.u, self.dims): # Form a mask which puts observed variables to zero and # broadcasts properly latent_mask_i = utils.add_axes(latent_mask, len(self.plates) - np.ndim(latent_mask), len(dims)) axis_sum = tuple(range(-len(dims),0)) # Compute the term phi_q = np.where(latent_mask_i, phi_q, 0) # TODO/FIXME: Use einsum here? Z = np.sum((phi_p-phi_q) * u_q, axis=axis_sum) L = L + Z return (np.sum(np.where(self.mask, L, 0)) * self.plate_multiplier(self.plates, np.shape(L), np.shape(self.mask))) #return L def fix_u_and_f(self, u, f, mask=True): #print(u) for (i,v) in enumerate(u): # This is what the dimensionality "should" be s = self.plates + self.dims[i] t = np.shape(v) if s != t: msg = "Dimensionality of the observations incorrect." msg += "\nShape of input: " + str(t) msg += "\nExpected shape: " + str(s) msg += "\nCheck plates." raise Exception(msg) #self.phi[i] = 0 self.update_u(u, mask=mask) ## for i in range(len(self.u)): ## obs_mask = utils.add_axes(mask, ## len(self.plates) - np.ndim(mask), ## len(self.dims[i])) ## # TODO/FIXME: Use copyto! ## self.u[i] = (obs_mask * u[i] ## + np.logical_not(obs_mask) * self.u[i]) # TODO/FIXME: Use the mask? self.f = f def observe(self, x, mask=True): (u, f) = self.compute_fixed_u_and_f(x) #print(u) self.fix_u_and_f(u, f, mask=mask) # Observed nodes should not be ignored self.observed = mask self.mask = np.logical_or(self.mask, self.observed) #print('observe', self.name, self.u) ## print(x) ## print(mask) ## print(u) def integrated_logpdf_from_parents(self, index): """ Approximates the posterior predictive pdf \int p(x|parents) q(parents) dparents in log-scale as \int q(parents_i) exp( \int q(parents_\i) \log p(x|parents) dparents_\i ) dparents_i.""" raise NotImplementedError() bayespy-0.1/bayespy/_nodes_dev/gaussian_markov_chain.py0000664000175000017500000001470412174013072025237 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np import utils from .variable import Variable from .constant import Constant from .wishart import Wishart class GaussianMarkovChain(Variable): ndims = (1, 2) ndims_parents = [(1, 2), (2, 0)] # Observations are vectors (1-D): ndim_observations = 1 ## @staticmethod ## def compute_fixed_parameter_moments(*args): ## """ Compute the moments of the distribution parameters for ## fixed values.""" ## mu = args[0] ## Lambda = args[1] ## u_mu = Gaussian.compute_fixed_moments(mu) ## u_Lambda = Wishart.compute_fixed_moments(Lambda) ## return (u_mu, u_Lambda) @staticmethod def compute_fixed_moments(x): """ Compute moments for fixed x. """ return [x, utils.m_outer(x,x)] @staticmethod def compute_phi_from_parents(u_parents): ## print('in Gaussian.compute_phi_from_parents') ## print(u_parents) ## print(np.shape(u_parents[1][0])) ## print(np.shape(u_parents[0][0])) return [utils.m_dot(u_parents[1][0], u_parents[0][0]), -0.5 * u_parents[1][0]] @staticmethod def compute_g_from_parents(u_parents): mu = u_parents[0][0] mumu = u_parents[0][1] Lambda = u_parents[1][0] logdet_Lambda = u_parents[1][1] g = (-0.5 * np.einsum('...ij,...ij',mumu,Lambda) + 0.5 * logdet_Lambda) ## g = (-0.5 * np.einsum('...ij,...ij',mumu,Lambda) ## + 0.5 * np.sum(logdet_Lambda)) return g @staticmethod def compute_u_and_g(phi, mask=True): #print(-phi[1]) L = utils.m_chol(-phi[1]) k = np.shape(phi[0])[-1] # Moments u0 = utils.m_chol_solve(L, 0.5*phi[0]) u1 = utils.m_outer(u0, u0) + 0.5 * utils.m_chol_inv(L) u = [u0, u1] # G g = (-0.5 * np.einsum('...i,...i', u[0], phi[0]) + 0.5 * utils.m_chol_logdet(L) + 0.5 * np.log(2) * k) #+ 0.5 * np.log(2) * self.dims[0][0]) return (u, g) @staticmethod def compute_fixed_u_and_f(x): """ Compute u(x) and f(x) for given x. """ k = np.shape(x)[-1] u = [x, utils.m_outer(x,x)] f = -k/2*np.log(2*np.pi) return (u, f) @staticmethod def compute_message(index, u, u_parents): """ . """ if index == 0: return [utils.m_dot(u_parents[1][0], u[0]), -0.5 * u_parents[1][0]] elif index == 1: xmu = utils.m_outer(u[0], u_parents[0][0]) return [-0.5 * (u[1] - xmu - xmu.swapaxes(-1,-2) + u_parents[0][1]), 0.5] @staticmethod def compute_dims(*parents): """ Compute the dimensions of phi and u. """ # Has the same dimensionality as the first parent. ## print('in gaussian compute dims: parent.dims:', parents[0].dims) ## print('in gaussian compute dims: parent.u:', parents[0].u) return parents[0].dims @staticmethod def compute_dims_from_values(x): """ Compute the dimensions of phi and u. """ d = np.shape(x)[-1] return [(d,), (d,d)] # Gaussian(mu, inv(Lambda)) def __init__(self, mu, Lambda, plates=(), **kwargs): self.parameter_distributions = (Gaussian, Wishart) # Check for constant mu if np.isscalar(mu) or isinstance(mu, np.ndarray): mu = Constant(Gaussian)(mu) # Check for constant Lambda if np.isscalar(Lambda) or isinstance(Lambda, np.ndarray): Lambda = Constant(Wishart)(Lambda) # You could check whether the dimensions of mu and Lambda # match (and Lambda is square) if Lambda.dims[0][-1] != mu.dims[0][-1]: raise Exception("Dimensionalities of mu and Lambda do not match.") # Construct super().__init__(mu, Lambda, plates=plates, **kwargs) def random(self): # TODO/FIXME: You shouldn't draw random values for # observed/fixed elements! # Note that phi[1] is -0.5*inv(Cov) U = utils.m_chol(-2*self.phi[1]) mu = self.u[0] z = np.random.normal(0, 1, self.get_shape(0)) # Compute mu + U'*z #return mu + np.einsum('...ij,...i->...j', U, z) #scipy.linalg.solve_triangular(a, b, trans=0, lower=False, unit_diagonal=False, overwrite_b=False, debug=False) #print('gaussian.random', np.shape(mu), np.shape(z)) z = utils.m_solve_triangular(U, z, trans='T', lower=False) return mu + z #return self.u[0] + utils.m_chol_solve(U, z) ## def initialize_random_mean(self): ## # First, initialize the distribution from prior? ## self.initialize_from_prior() ## if not np.all(self.observed): ## # Draw a random sample ## x = self.random() ## # Update parameter for the mean using the sample ## self.phi[0] = -2*utils.m_dot(self.phi[1], x) ## # Update moments ## (u, g) = self.compute_u_and_g(self.phi, mask=True) ## self.update_u_and_g(u, g, mask=True) def show(self): mu = self.u[0] Cov = self.u[1] - utils.m_outer(mu, mu) print("%s ~ Gaussian(mu, Cov)" % self.name) print(" mu = ") print(mu) print(" Cov = ") print(str(Cov)) ## def observe(self, x): ## self.fix_moments([x, utils.m_outer(x,x)]) bayespy-0.1/bayespy/_nodes_dev/categorical.py0000664000175000017500000000723112174013072023156 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np from .variable import Variable from .constant import Constant from .dirichlet import Dirichlet def Categorical(p, **kwargs): # Get the number of categories (static methods may need this) if np.isscalar(p) or isinstance(p, np.ndarray): n_categories = np.shape(p)[-1] else: n_categories = p.dims[0][0] # The actual categorical distribution node class _Categorical(Variable): ndims = (1,) @staticmethod def compute_phi_from_parents(u_parents): return [u_parents[0][0]] @staticmethod def compute_g_from_parents(u_parents): return 0 @staticmethod def compute_u_and_g(phi, mask=True): # For numerical reasons, scale contributions closer to # one, i.e., subtract the maximum of the log-contributions. max_phi = np.max(phi[0], axis=-1, keepdims=True) p = np.exp(phi[0]-max_phi) sum_p = np.sum(p, axis=-1, keepdims=True) # Moments u0 = p / sum_p u = [u0] # G g = -np.log(sum_p) - max_phi g = np.squeeze(g, axis=-1) #print('Categorical.compute_u_and_g, g:', np.sum(g), np.shape(g), np.sum(max_phi)) return (u, g) @staticmethod def compute_fixed_u_and_f(x): """ Compute u(x) and f(x) for given x. """ # TODO: You could check that x has proper dimensions x = np.array(x, dtype=np.int) u0 = np.zeros((np.size(x), n_categories)) u0[[np.arange(np.size(x)), x]] = 1 f = 0 return ([u0], f) @staticmethod def compute_message(index, u, u_parents): """ . """ #print('message in categorical:', u[0]) if index == 0: return [ u[0].copy() ] @staticmethod def compute_dims(*parents): """ Compute the dimensions of phi/u. """ # Has the same dimensionality as the parent. return parents[0].dims def __init__(self, p, **kwargs): # Check for constant mu if np.isscalar(p) or isinstance(p, np.ndarray): p = ConstantDirichlet(p) # Construct super().__init__(p, **kwargs) def random(self): raise NotImplementedError() def show(self): p = self.u[0] #np.exp(self.phi[0]) #p /= np.sum(p, axis=-1, keepdims=True) print("%s ~ Categorical(p)" % self.name) print(" p = ") print(p) return _Categorical(p, **kwargs) bayespy-0.1/bayespy/_nodes_dev/node.py0000664000175000017500000000625312174013072021631 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### ''' General class for a node in a Bayesian network. The node can be stochastic or deterministic. ''' import numpy as np class Node: def __init__(self, *args, dims=None, plates=(), name=""): if dims is None: raise Exception("You need to specify the dimensionality" \ + " of the distribution for class" \ + str(self.__class__)) self.dims = dims self.plates = plates self.name = name # Parents self.parents = args # Inform parent nodes for (index,parent) in enumerate(self.parents): if parent: parent.add_child(self, index) # Children self.children = list() def add_child(self, child, index): self.children.append((child, index)) def plates_to_parent(self, index): return self.plates #def get_shape(self, ind): # return self.plates + self.dims[ind] ## @staticmethod ## def plate_multiplier(plates, *args): ## # Check broadcasting of the shapes ## for arg in args: ## utils.broadcasted_shape(plates, arg) ## r = 1 ## for j in range(-len(plates),0): ## mult = True ## for arg in args: ## if not (-j > len(arg) or arg[j] == 1): ## mult = False ## if mult: ## r *= plates[j] ## return r ## class NodeConstant(Node): ## def __init__(self, u, **kwargs): ## self.u = u ## Node.__init__(self, **kwargs) ## def message_to_child(self, gradient=False): ## if gradient: ## return (self.u, []) ## else: ## return self.u ## class NodeConstantScalar(NodeConstant): ## @staticmethod ## def compute_fixed_u_and_f(x): ## """ Compute u(x) and f(x) for given x. """ ## return ([x], 0) ## def __init__(self, a, **kwargs): ## NodeConstant.__init__(self, ## [a], ## plates=np.shape(a), ## dims=[()], ## **kwargs) bayespy-0.1/bayespy/utils/0000775000175000017500000000000012174043062017361 5ustar jluttinejluttine00000000000000bayespy-0.1/bayespy/utils/utils.py0000664000175000017500000007746512174013072021114 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011-2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### """ General numerical functions and methods. """ import functools import itertools import numpy as np import scipy as sp #import scipy.linalg.decomp_cholesky as decomp import scipy.linalg as linalg import scipy.special as special import scipy.optimize as optimize import scipy.sparse as sparse #import scikits.sparse.cholmod as cholmod import tempfile as tmp import unittest from numpy import testing def is_callable(f): return hasattr(f, '__call__') def atleast_nd(X, d): if np.ndim(X) < d: sh = (d-np.ndim(X))*(1,) + np.shape(X) X = np.reshape(X, sh) return X def T(X): """ Transpose the matrix. """ return np.swapaxes(X, -1, -2) class TestCase(unittest.TestCase): """ Simple base class for unit testing. Adds NumPy's features to Python's unittest. """ def assertAllClose(self, A, B, msg="Arrays not almost equal"): self.assertEqual(np.shape(A), np.shape(B), msg=msg) testing.assert_allclose(A, B, err_msg=msg) def symm(X): """ Make X symmetric. """ return 0.5 * (X + np.swapaxes(X, -1, -2)) def unique(l): """ Remove duplicate items from a list while preserving order. """ seen = set() seen_add = seen.add return [ x for x in l if x not in seen and not seen_add(x)] def tempfile(prefix='', suffix=''): return tmp.NamedTemporaryFile(prefix=prefix, suffix=suffix).name def write_to_hdf5(group, data, name): """ Writes the given array into the HDF5 file. """ try: # Try using compression. It doesn't work for scalars. group.create_dataset(name, data=data, compression='gzip') except TypeError: group.create_dataset(name, data=data) except ValueError: raise ValueError('Could not write %s' % data) def nans(size=()): return np.tile(np.nan, size) def trues(shape): return np.ones(shape, dtype=np.bool) def array_to_scalar(x): # This transforms an N-dimensional array to a scalar. It's most # useful when you know that the array has only one element and you # want it out as a scalar. return np.ravel(x)[0] #def diag(x): def grid(x1, x2): """ Returns meshgrid as a (M*N,2)-shape array. """ (X1, X2) = np.meshgrid(x1, x2) return np.hstack((X1.reshape((-1,1)),X2.reshape((-1,1)))) class CholeskyDense(): def __init__(self, K): self.U = linalg.cho_factor(K) def solve(self, b): if sparse.issparse(b): b = b.toarray() return linalg.cho_solve(self.U, b) def logdet(self): return 2*np.sum(np.log(np.diag(self.U[0]))) def trace_solve_gradient(self, dK): return np.trace(self.solve(dK)) class CholeskySparse(): def __init__(self, K): self.LD = cholmod.cholesky(K) def solve(self, b): if sparse.issparse(b): b = b.toarray() return self.LD.solve_A(b) def logdet(self): return self.LD.logdet() #np.sum(np.log(LD.D())) def trace_solve_gradient(self, dK): # WTF?! numpy.multiply doesn't work for two sparse # matrices.. It returns a result but it is incorrect! # Use the identity trace(K\dK)=sum(inv(K).*dK) by computing # the sparse inverse (lower triangular part) iK = self.LD.spinv(form='lower') return (2*iK.multiply(dK).sum() - iK.diagonal().dot(dK.diagonal())) # Multiply by two because of symmetry (remove diagonal once # because it was taken into account twice) #return np.multiply(self.LD.inv().todense(),dK.todense()).sum() #return self.LD.inv().multiply(dK).sum() # THIS WORKS #return np.multiply(self.LD.inv(),dK).sum() # THIS NOT WORK!! WTF?? iK = self.LD.spinv() return iK.multiply(dK).sum() #return (2*iK.multiply(dK).sum() # - iK.diagonal().dot(dK.diagonal())) #return (2*np.multiply(iK, dK).sum() # - iK.diagonal().dot(dK.diagonal())) # THIS NOT WORK!! #return np.trace(self.solve(dK)) def cholesky(K): if isinstance(K, np.ndarray): return CholeskyDense(K) elif sparse.issparse(K): return CholeskySparse(K) else: raise Exception("Unsupported covariance matrix type") def vb_optimize(x0, set_values, lowerbound, gradient=None): # Function for computing the lower bound def func(x): # Set the value of the nodes set_values(x) # Compute lower bound (and gradient terms) return -lowerbound() #return f # Function for computing the gradient of the lower bound def funcprime(x): # Collect the gradients from the nodes set_values(x) # Compute lower bound (and gradient terms) #lowerbound() return -gradient() #return df # Optimize if gradient != None: check_gradient(x0, func, funcprime, 1e-6) xopt = optimize.fmin_bfgs(func, x0, fprime=funcprime, maxiter=100) #xopt = optimize.fmin_ncg(func, x0, fprime=funcprime, maxiter=50) else: xopt = optimize.fmin_bfgs(func, x0, maxiter=100) #xopt = optimize.fmin_ncg(func, x0, maxiter=50) # Set optimal values to the nodes set_values(xopt) # Optimizes the parameters of the given nodes. def vb_optimize_nodes(*nodes): # Get cost functions lbs = set() for node in nodes: # Add node's cost function lbs |= node.get_all_vb_terms() #.lower_bound_contribution) # Add child nodes' cost functions #for lb in node.get_children_vb_bound(): #lbs.add(lb) # Uniqify nodes? nodes = set(nodes) # Get initial value and transformation/update function ind = 0 ind_all = list() transform_all = list() gradient_all = list() x0_all = np.array([]) for node in nodes: (x0, transform, gradient) = node.start_optimization() # Vector of initial values x0 = np.atleast_1d(x0) x0_all = np.concatenate((x0_all, x0)) # Indices of the vector elements that correspond to this node sz = np.size(x0) ind_all.append((ind, ind+sz)) ind += sz # Function for setting the value of this node transform_all.append(transform) # Gradients gradient_all.append(gradient) # Function for changing the values of the nodes def set_value(x): for (ind, transform) in zip(ind_all, transform_all): # Transform/update variable transform(x[ind[0]:ind[1]]) # Compute the lower bound (and the gradient) def lowerbound(): l = 0 # TODO: Put gradients to zero! for lb in lbs: l += lb(gradient=False) return l # Compute (or get) the gradient def gradient(): for lb in lbs: lb(gradient=True) dl = np.zeros(np.shape(x0_all)) for (ind, gradient) in zip(ind_all, gradient_all): dl[ind[0]:ind[1]] = gradient() return dl #vb_optimize(x0_all, set_value, lowerbound) vb_optimize(x0_all, set_value, lowerbound, gradient=gradient) for node in nodes: node.stop_optimization() # Computes log probability density function of the Gaussian # distribution def gaussian_logpdf(y_invcov_y, y_invcov_mu, mu_invcov_mu, logdetcov, D): return (-0.5*D*np.log(2*np.pi) -0.5*logdetcov -0.5*y_invcov_y +y_invcov_mu -0.5*mu_invcov_mu) def check_gradient(x0, f, df, eps): #f0 = f(x0) grad = np.zeros(np.shape(x0)) for ind in range(len(x0)): xmin = x0.copy() xmax = x0.copy() xmin[ind] -= eps xmax[ind] += eps fmin = f(xmin) fmax = f(xmax) grad[ind] = (fmax-fmin) / (2*eps) print('x: ' + str(x0)) print('Numerical gradient: ' + str(grad)) print('Exact gradient: ' + str(df(x0))) def is_numeric(a): return (np.isscalar(a) or isinstance(a, list) or isinstance(a, np.ndarray)) def multiply_shapes(*shapes): """ Compute element-wise product of lists/tuples. Shorter lists are concatenated with leading 1s in order to get lists with the same length. """ # Make the shapes equal length shapes = make_equal_length(*shapes) # Compute element-wise product f = lambda X,Y: (x*y for (x,y) in zip(X,Y)) shape = functools.reduce(f, shapes) return tuple(shape) def make_equal_length(*shapes): """ Make tuples equal length. Add leading 1s to shorter tuples. """ # Get maximum length max_len = max((len(shape) for shape in shapes)) # Make the shapes equal length shapes = ((1,)*(max_len-len(shape)) + tuple(shape) for shape in shapes) return shapes def sum_to_dim(A, dim): """ Sum leading axes of A such that A has dim dimensions. """ dimdiff = np.ndim(A) - dim if dimdiff > 0: axes = np.arange(dimdiff) A = np.sum(A, axis=axes) return A def sum_multiply(*args, axis=None, sumaxis=True, keepdims=False): # Computes sum(arg[0]*arg[1]*arg[2]*..., axis=axes_to_sum) without # explicitly computing the intermediate product if len(args) == 0: raise ValueError("You must give at least one input array") # Dimensionality of the result max_dim = 0 for k in range(len(args)): max_dim = max(max_dim, np.ndim(args[k])) if sumaxis: if axis is None: # Sum all axes axes = [] else: if np.isscalar(axis): axis = [axis] axes = [i for i in range(max_dim) if i not in axis and (-max_dim+i) not in axis] else: if axis is None: # Keep all axes axes = range(max_dim) else: # Find axes that are kept if np.isscalar(axis): axes = [axis] axes = [i if i >= 0 else i+max_dim for i in axis] axes = sorted(axes) if len(axes) > 0 and (min(axes) < 0 or max(axes) >= max_dim): raise ValueError("Axis index out of bounds") # Form a list of pairs: the array in the product and its axes pairs = list() for i in range(len(args)): a = args[i] a_dim = np.ndim(a) pairs.append(a) pairs.append(range(max_dim-a_dim, max_dim)) # Output axes are those which are not summed pairs.append(axes) # Compute the sum-product try: y = np.einsum(*pairs) except ValueError as err: if str(err) == ("If 'op_axes' or 'itershape' is not NULL in " "theiterator constructor, 'oa_ndim' must be greater " "than zero"): # TODO/FIXME: Handle a bug in NumPy. If all arguments to einsum are # scalars, it raises an error. For scalars we can just use multiply # and forget about summing. Hopefully, in the future, einsum handles # scalars properly and this try-except becomes unnecessary. y = functools.reduce(np.multiply, args) else: raise err # Restore summed axes as singleton axes if keepdims: d = 0 s = () for k in range(max_dim): if k in axes: # Axis not summed s = s + (np.shape(y)[d],) d += 1 else: # Axis was summed s = s + (1,) y = np.reshape(y, s) return y def sum_product(*args, axes_to_keep=None, axes_to_sum=None, keepdims=False): if axes_to_keep is not None: return sum_multiply(*args, axis=axes_to_keep, sumaxis=False, keepdims=keepdims) else: return sum_multiply(*args, axis=axes_to_sum, sumaxis=True, keepdims=keepdims) def moveaxis(A, axis_from, axis_to): """ Move the axis number axis_from to be the axis number axis_to. """ axes = np.arange(np.ndim(A)) axes[axis_from:axis_to] += 1 axes[axis_from:axis_to:-1] -= 1 axes[axis_to] = axis_from return np.transpose(A, axes=axes) def broadcasted_shape_from_arrays(*args): """ Computes the resulting shape if shapes a and b are broadcasted together. """ # The dimensionality (i.e., number of axes) of the result dim = 0 for a in args: dim = max(dim, np.ndim(a)) S = () for i in range(-dim,0): s = 1 for a in args: if -i <= np.ndim(a): if s == 1: s = np.shape(a)[i] elif np.shape(a)[i] != 1 and np.shape(a)[i] != s: raise Exception("Shapes do not broadcast") S = S + (s,) return S def is_shape_subset(sub_shape, full_shape): """ """ if len(sub_shape) > len(full_shape): return False for i in range(len(sub_shape)): ind = -1 - i if sub_shape[ind] != 1 and sub_shape[ind] != full_shape[ind]: return False return True def broadcasted_shape(*shapes): """ Get the resulting shape if the given shapes were broadcasted. Broadcasting rules of NumPy. """ dim = 0 for a in shapes: dim = max(dim, len(a)) S = () for i in range(-dim,0): s = 1 for a in shapes: if -i <= len(a): if s == 1: s = a[i] elif a[i] != 1 and a[i] != s: raise ValueError("Shapes %s do not broadcast" % (shapes,)) S = S + (s,) return S ## def broadcasted_shape(a,b): ## # Computes the resulting shape if shapes a and b are broadcasted ## # together ## l_max = max(len(a), len(b)) ## s = () ## for i in range(-l_max,0): ## if -i > len(b): ## s += (a[i],) ## elif -i > len(a) or a[i] == 1 or a[i] == b[i]: ## s += (b[i],) ## elif b[i] == 1: ## s += (a[i],) ## else: ## raise Exception("Shapes %s and %s do not broadcast" % (a,b)) ## return s def add_leading_axes(x, n): shape = (1,)*n + np.shape(x) return np.reshape(x, shape) def add_trailing_axes(x, n): shape = np.shape(x) + (1,)*n return np.reshape(x, shape) def add_axes(x, lead, trail): shape = (1,)*lead + np.shape(x) + (1,)*trail return np.reshape(x, shape) def nested_iterator(max_inds): s = (range(i) for i in max_inds) return itertools.product(*s) def squeeze_to_dim(X, dim): s = tuple(range(np.ndim(X)-dim)) return np.squeeze(X, axis=s) def axes_to_collapse(shape_x, shape_to): # Solves which axes of shape shape_x need to be collapsed in order # to get the shape shape_to s = () for j in range(-len(shape_x), 0): if shape_x[j] != 1: if -j > len(shape_to) or shape_to[j] == 1: s += (j,) elif shape_to[j] != shape_x[j]: print('Shape from: ' + str(shape_x)) print('Shape to: ' + str(shape_to)) raise Exception('Incompatible shape to squeeze') return tuple(s) def repeat_to_shape(A, s): # Current shape t = np.shape(A) if len(t) > len(s): raise Exception("Can't repeat to a smaller shape") # Add extra axis t = tuple([1]*(len(s)-len(t))) + t A = np.reshape(A,t) # Repeat for i in reversed(range(len(s))): if s[i] != t[i]: if t[i] != 1: raise Exception("Can't repeat non-singular dimensions") else: A = np.repeat(A, s[i], axis=i) return A #def spinv_chol(L): def chol(C): if sparse.issparse(C): # Sparse Cholesky decomposition (returns a Factor object) return cholmod.cholesky(C) else: # Dense Cholesky decomposition return linalg.cho_factor(C)[0] def chol_solve(U, b): if isinstance(U, np.ndarray): if sparse.issparse(b): b = b.toarray() return linalg.cho_solve((U, False), b) elif isinstance(U, cholmod.Factor): if sparse.issparse(b): b = b.toarray() return U.solve_A(b) else: raise ValueError("Unknown type of Cholesky factor") def chol_inv(U): if isinstance(U, np.ndarray): I = np.identity(np.shape(U)[-1]) return linalg.cho_solve((U, False), I) elif isinstance(U, cholmod.Factor): raise NotImplementedError ## if sparse.issparse(b): ## b = b.toarray() ## return U.solve_A(b) else: raise ValueError("Unknown type of Cholesky factor") def chol_logdet(U): if isinstance(U, np.ndarray): return 2*np.sum(np.log(np.diag(U))) elif isinstance(U, cholmod.Factor): return np.sum(np.log(U.D())) else: raise ValueError("Unknown type of Cholesky factor") def logdet_chol(U): if isinstance(U, np.ndarray): return 2*np.sum(np.log(np.diag(U))) elif isinstance(U, cholmod.Factor): return np.sum(np.log(U.D())) def m_solve_triangular(U, B, **kwargs): # Allocate memory U = np.atleast_2d(U) B = np.atleast_1d(B) sh_u = U.shape[:-2] sh_b = B.shape[:-1] l_u = len(sh_u) l_b = len(sh_b) # Check which axis are iterated over with B along with U ind_b = [Ellipsis] * l_b l_min = min(l_u, l_b) jnd_b = tuple(i for i in range(-l_min,0) if sh_b[i]==sh_u[i]) # Shape of the result (broadcasting rules) sh = broadcasted_shape(sh_u, sh_b) #out = np.zeros(np.shape(B)) out = np.zeros(sh + B.shape[-1:]) ## if out == None: ## # Shape of the result (broadcasting rules) ## sh = broadcasted_shape(sh_u, sh_b) ## #out = np.zeros(np.shape(B)) ## out = np.zeros(sh + B.shape[-1:]) for i in nested_iterator(np.shape(U)[:-2]): # The goal is to run triangular solver once for all vectors of # B for which the matrices of U are the same (according to the # broadcasting rules). Thus, we collect all the axes of B for # which U is singleton and form them as a 2-D matrix and then # run the solver once. # Select those axes of B for which U and B are not singleton for j in jnd_b: ind_b[j] = i[j] # Collect all the axes for which U is singleton b = B[tuple(ind_b) + (Ellipsis,)] # Reshape it to a 2-D (or 1-D) array orig_shape = b.shape if b.ndim > 1: b = b.reshape((-1, b.shape[-1])) # Ellipsis to all preceeding axes and ellipsis for the last # axis: if len(ind_b) < len(sh): ind_out = (Ellipsis,) + tuple(ind_b) + (Ellipsis,) else: ind_out = tuple(ind_b) + (Ellipsis,) #print('utils.m_solve_triangular', np.shape(U[i]), np.shape(b)) out[ind_out] = linalg.solve_triangular(U[i], b.T, **kwargs).T.reshape(orig_shape) #out[ind_out] = out[ind_out].T.reshape(orig_shape) return out def m_chol(C): # Computes Cholesky decomposition for a collection of matrices. # The last two axes of C are considered as the matrix. C = np.atleast_2d(C) U = np.empty(np.shape(C)) #print('m_chol', C) for i in nested_iterator(np.shape(U)[:-2]): try: U[i] = linalg.cho_factor(C[i])[0] except np.linalg.linalg.LinAlgError: print(C[i]) raise Exception("Matrix not positive definite") return U def m_chol_solve(U, B, out=None): # Allocate memory U = np.atleast_2d(U) B = np.atleast_1d(B) sh_u = U.shape[:-2] sh_b = B.shape[:-1] l_u = len(sh_u) l_b = len(sh_b) # Check which axis are iterated over with B along with U ind_b = [Ellipsis] * l_b l_min = min(l_u, l_b) jnd_b = tuple(i for i in range(-l_min,0) if sh_b[i]==sh_u[i]) if out == None: # Shape of the result (broadcasting rules) sh = broadcasted_shape(sh_u, sh_b) #out = np.zeros(np.shape(B)) out = np.zeros(sh + B.shape[-1:]) for i in nested_iterator(np.shape(U)[:-2]): # The goal is to run Cholesky solver once for all vectors of B # for which the matrices of U are the same (according to the # broadcasting rules). Thus, we collect all the axes of B for # which U is singleton and form them as a 2-D matrix and then # run the solver once. # Select those axes of B for which U and B are not singleton for j in jnd_b: ind_b[j] = i[j] # Collect all the axes for which U is singleton b = B[tuple(ind_b) + (Ellipsis,)] # Reshape it to a 2-D (or 1-D) array orig_shape = b.shape if b.ndim > 1: b = b.reshape((-1, b.shape[-1])) # Ellipsis to all preceeding axes and ellipsis for the last # axis: if len(ind_b) < len(sh): ind_out = (Ellipsis,) + tuple(ind_b) + (Ellipsis,) else: ind_out = tuple(ind_b) + (Ellipsis,) out[ind_out] = linalg.cho_solve((U[i], False), b.T).T.reshape(orig_shape) return out def m_chol_inv(U): # Allocate memory V = np.tile(np.identity(np.shape(U)[-1]), np.shape(U)[:-2]+(1,1)) for i in nested_iterator(np.shape(U)[:-2]): V[i] = linalg.cho_solve((U[i], False), V[i], overwrite_b=True) # This would need Fortran order return V def m_chol_logdet(U): # Computes Cholesky decomposition for a collection of matrices. return 2*np.sum(np.log(np.einsum('...ii->...i', U)), axis=(-1,)) def m_digamma(a, d): y = 0 for i in range(d): y += special.digamma(a + 0.5*(1-i)) return y def m_outer(A,B): # Computes outer product over the last axes of A and B. The other # axes are broadcasted. Thus, if A has shape (..., N) and B has # shape (..., M), then the result has shape (..., N, M) return A[...,np.newaxis]*B[...,np.newaxis,:] def diagonal(A): return np.diagonal(A, axis1=-2, axis2=-1) def m_dot(A,b): # Compute matrix-vector product over the last two axes of A and # the last axes of b. Other axes are broadcasted. If A has shape # (..., M, N) and b has shape (..., N), then the result has shape # (..., M) #b = reshape(b, shape(b)[:-1] + (1,) + shape(b)[-1:]) #return np.dot(A, b) return np.einsum('...ik,...k->...i', A, b) # TODO: Use einsum!! #return np.sum(A*b[...,np.newaxis,:], axis=(-1,)) def block_banded(D, B): """ Construct a symmetric block-banded matrix. `D` contains square diagonal blocks. `B` contains super-diagonal blocks. The resulting matrix is: D[0], B[0], 0, 0, ..., 0, 0, 0 B[0].T, D[1], B[1], 0, ..., 0, 0, 0 0, B[1].T, D[2], B[2], ..., ..., ..., ... ... ... ... ... ..., B[N-2].T, D[N-1], B[N-1] 0, 0, 0, 0, ..., 0, B[N-1].T, D[N] """ D = [np.atleast_2d(d) for d in D] B = [np.atleast_2d(b) for b in B] # Number of diagonal blocks N = len(D) if len(B) != N-1: raise ValueError("The number of super-diagonal blocks must contain " "exactly one block less than the number of diagonal " "blocks") # Compute the size of the full matrix M = 0 for i in range(N): if np.ndim(D[i]) != 2: raise ValueError("Blocks must be 2 dimensional arrays") d = np.shape(D[i]) if d[0] != d[1]: raise ValueError("Diagonal blocks must be square") M += d[0] for i in range(N-1): if np.ndim(B[i]) != 2: raise ValueError("Blocks must be 2 dimensional arrays") b = np.shape(B[i]) if b[0] != np.shape(D[i])[1] or b[1] != np.shape(D[i+1])[0]: raise ValueError("Shapes of the super-diagonal blocks do not match " "the shapes of the diagonal blocks") A = np.zeros((M,M)) k = 0 for i in range(N-1): (d0, d1) = np.shape(B[i]) # Diagonal block A[k:k+d0, k:k+d0] = D[i] # Super-diagonal block A[k:k+d0, k+d0:k+d0+d1] = B[i] # Sub-diagonal block A[k+d0:k+d0+d1, k:k+d0] = B[i].T k += d0 A[k:,k:] = D[-1] return A def block_banded_solve(A, B, y): """ Invert symmetric, banded, positive-definite matrix. A contains the diagonal blocks. B contains the superdiagonal blocks (their transposes are the subdiagonal blocks). A and B are lists. The length of B is one smaller. The algorithm is basically LU decomposition. Computes only the diagonal and super-diagonal blocks of the inverse. The true inverse is dense, in general. Assume each block has the same size. Return: * inverse blocks * solution to the system * log-determinant """ # Number of diagonal blocks N = len(A) if len(B) != N-1: raise ValueError("The number of super-diagonal blocks must be exactly " "one less than the number of diagonal blocks") # Compute the size of the full matrix D = np.shape(A[0])[0] V = np.empty((N,D,D)) C = np.empty((N-1,D,D)) x = np.empty(np.shape(y)) # # Forward recursion # # In the forward recursion, store the Cholesky factor in V. So you # don't need to recompute them in the backward recursion. # TODO/FIXME: You could store chol_solve(V[n], B[n]) in forward recursion to # C, because it is used in backward recursion too! x[0] = y[0] V[0] = chol(A[0]) ldet = chol_logdet(V[0]) for n in range(N-1): # Compute the solution of the system x[n+1] = y[n+1] - np.dot(B[n].T, chol_solve(V[n], x[n])) # Compute the diagonal block and store the Cholesky factor V[n+1] = A[n+1] - np.dot(B[n].T, chol_solve(V[n], B[n])) V[n+1] = 0.5*V[n+1] + 0.5*V[n+1].T #print('blk bnd solve', n, V[n+1]) V[n+1] = chol(V[n+1]) #V[n+1] = chol(A[n+1] - np.dot(B[n].T, chol_solve(V[n], B[n]))) # Compute the log-det term here, too ldet += chol_logdet(V[n+1]) # # Backward recursion # x[-1] = chol_solve(V[-1], x[-1]) V[-1] = chol_inv(V[-1]) for n in reversed(range(N-1)): # Compute the solution of the system x[n] = chol_solve(V[n], x[n] - np.dot(B[n], x[n+1])) # Compute the superdiagonal block of the inverse Z = chol_solve(V[n], B[n]) C[n] = -np.dot(Z, V[n+1]) # Compute the diagonal block of the inverse V[n] = chol_inv(V[n]) + np.dot(np.dot(Z, V[n+1]), Z.T) V[n] = 0.5*V[n] + 0.5*V[n].T return (V, C, x, ldet) def kalman_filter(y, U, A, V, mu0, Cov0, out=None): """ Perform Kalman filtering to obtain filtered mean and covariance. The parameters of the process may vary in time, thus they are given as iterators instead of fixed values. Parameters ---------- y : (N,D) array "Normalized" noisy observations of the states, that is, the observations multiplied by the precision matrix U (and possibly other transformation matrices). U : (N,D,D) array or N-list of (D,D) arrays Precision matrix (i.e., inverse covariance matrix) of the observation noise for each time instance. A : (N-1,D,D) array or (N-1)-list of (D,D) arrays Dynamic matrix for each time instance. V : (N-1,D,D) array or (N-1)-list of (D,D) arrays Covariance matrix of the innovation noise for each time instance. Returns ------- mu : array Filtered mean of the states. Cov : array Filtered covariance of the states. See also -------- rts_smoother """ mu = mu0 Cov = Cov0 # Allocate memory for the results (N,D) = np.shape(y) X = np.empty((N,D)) CovX = np.empty((N,D,D)) # Update step for t=0 M = np.dot(np.dot(Cov, U[0]), Cov) + Cov L = chol(M) mu = np.dot(Cov, chol_solve(L, np.dot(Cov,y[0]) + mu)) Cov = np.dot(Cov, chol_solve(L, Cov)) X[0,:] = mu CovX[0,:,:] = Cov #for (yn, Un, An, Vn) in zip(y, U, A, V): for n in range(len(y)-1): #(yn, Un, An, Vn) in zip(y, U, A, V): # Prediction step mu = np.dot(A[n], mu) Cov = np.dot(np.dot(A[n], Cov), A[n].T) + V[n] # Update step M = np.dot(np.dot(Cov, U[n+1]), Cov) + Cov L = chol(M) mu = np.dot(Cov, chol_solve(L, np.dot(Cov,y[n+1]) + mu)) Cov = np.dot(Cov, chol_solve(L, Cov)) # Force symmetric covariance (for numeric inaccuracy) Cov = 0.5*Cov + 0.5*Cov.T # Store results X[n+1,:] = mu CovX[n+1,:,:] = Cov return (X, CovX) def rts_smoother(mu, Cov, A, V, removethis=None): """ Perform Rauch-Tung-Striebel smoothing to obtain the posterior. The function returns the posterior mean and covariance of each state. The parameters of the process may vary in time, thus they are given as iterators instead of fixed values. Parameters ---------- mu : (N,D) array Mean of the states from Kalman filter. Cov : (N,D,D) array Covariance of the states from Kalman filter. A : (N-1,D,D) array or (N-1)-list of (D,D) arrays Dynamic matrix for each time instance. V : (N-1,D,D) array or (N-1)-list of (D,D) arrays Covariance matrix of the innovation noise for each time instance. Returns ------- mu : array Posterior mean of the states. Cov : array Posterior covariance of the states. See also -------- kalman_filter """ N = len(mu) #n = N-1 # Start from the last time instance and smoothen backwards x = mu[-1,:] Covx = Cov[-1,:,:] for n in reversed(range(N-1)):#(An, Vn) in zip(reversed(A), reversed(V)): #n = n - 1 #if n <= 0: # break # The predicted value of n x_p = np.dot(A[n], mu[n,:]) Cov_p = np.dot(np.dot(A[n], Cov[n,:,:]), A[n].T) + V[n] # Temporary variable S = np.linalg.solve(Cov_p, np.dot(A[n], Cov[n,:,:])) # Smoothed value of n x = mu[n,:] + np.dot(S.T, x-x_p) Covx = Cov[n,:,:] + np.dot(np.dot(S.T, Covx-Cov_p), S) # Force symmetric covariance (for numeric inaccuracy) Covx = 0.5*Covx + 0.5*Covx.T # Store results mu[n,:] = x Cov[n,:] = Covx return (mu, Cov) ## x_p = A*x; ## Covx_p = A*Covx*A' + Q; ## S = (Covx*A') / Covx_p; ## x = x + S*(x_s-x_p); ## if nargout >= 2 ## Covx = Covx + S*(Covx_s-Covx_p)*S'; ## end pass bayespy-0.1/bayespy/utils/random.py0000664000175000017500000001241412174013072021213 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### """ General functions random sampling and distributions. """ import numpy as np from . import linalg def mask(*shape, p=0.5): """ Return a boolean array of the given shape. Parameters: ----------- d0, d1, ..., dn : int Shape of the output. p : value in range [0,1] A probability that the elements are `True`. """ return np.random.rand(*shape) < p def wishart_rand(nu, V): """ Draw a random sample from the Wishart distribution. Parameters: ----------- nu : int """ # TODO/FIXME: Are these correct.. D = np.shape(V)[0] if nu < D: raise ValueError("Degrees of freedom must be equal or greater than the " "dimensionality of the matrix.") X = np.random.multivariate_normal(np.zeros(D), V, size=nu) return np.dot(X, X.T) def invwishart_rand(nu, V): # TODO/FIXME: Are these correct.. return np.linalg.inv(wishart_rand(nu, V)) def covariance(D, size=()): """ Draw a random covariance matrix. Draws from inverse-Wishart distribution. The distribution of each element is independent of the dimensionality of the matrix. C ~ Inv-W(I, D) Parameters: ----------- D : int Dimensionality of the covariance matrix. Returns: -------- C : (D,D) ndarray Positive-definite symmetric :math:`D\times D` matrix. """ if isinstance(size, int): size = (size,) shape = tuple(size) + (D,D) C = np.random.randn(*shape) C = linalg.dot(C, np.swapaxes(C, -1, -2)) return linalg.inv(C) #return np.linalg.inv(np.dot(C, C.T)) def correlation(D): """ Draw a random correlation matrix. """ X = np.random.randn(D,D); s = np.sqrt(np.sum(X**2, axis=-1, keepdims=True)) X = X / s return np.dot(X, X.T) def gaussian_logpdf(yVy, yVmu, muVmu, logdet_V, D): """ Log-density of a Gaussian distribution. :math:`\mathcal{G}(\mathbf{y}|\boldsymbol{\mu},\mathbf{V}^{-1})` Parameters: ----------- yVy : ndarray or double :math:`\mathbf{y}^T\mathbf{Vy}` yVmu : ndarray or double :math:`\mathbf{y}^T\mathbf{V}\boldsymbol{\mu}` muVmu : ndarray or double :math:`\boldsymbol{\mu}^T\mathbf{V}\boldsymbol{\mu}` logdet_V : ndarray or double Log-determinant of the precision matrix, :math:`\log|\mathbf{V}|`. D : int Dimensionality of the distribution. """ return -0.5*yVy + yVmu - 0.5*muVmu + 0.5*logdet_V - 0.5*D*np.log(2*np.pi) def gaussian_entropy(logdet_V, D): """ Compute the entropy of a Gaussian distribution. If you want to get the gradient, just let each parameter be a gradient of that term. Parameters: ----------- logdet_V : ndarray or double The log-determinant of the precision matrix. D : int The dimensionality of the distribution. """ return -0.5*logdet_V + 0.5*D + 0.5*D*np.log(2*np.pi) def gamma_logpdf(bx, logx, a_logx, a_logb, gammaln_a): """ Log-density of :math:`\mathcal{G}(x|a,b)`. If you want to get the gradient, just let each parameter be a gradient of that term. Parameters: ----------- bx : ndarray :math:`bx` logx : ndarray :math:`\log(x)` a_logx : ndarray :math:`a \log(x)` a_logb : ndarray :math:`a \log(b)` gammaln_a : ndarray :math:`\log\Gamma(a)` """ return a_logb - gammaln_a + a_logx - logx - bx #def gamma_logpdf(a, log_b, gammaln_a, def gamma_entropy(a, log_b, gammaln_a, psi_a, a_psi_a): """ Entropy of :math:`\mathcal{G}(a,b)`. If you want to get the gradient, just let each parameter be a gradient of that term. Parameters: ----------- a : ndarray :math:`a` log_b : ndarray :math:`\log(b)` gammaln_a : ndarray :math:`\log\Gamma(a)` psi_a : ndarray :math:`\psi(a)` a_psi_a : ndarray :math:`a\psi(a)` """ return a - log_b + gammaln_a + psi_a - a_psi_a def orth(D): """ Draw random orthogonal matrix. """ Q = np.random.randn(D,D) (Q, _) = np.linalg.qr(Q) return Q def svd(s): """ Draw a random matrix given its singular values. """ D = len(s) U = orth(D) * s V = orth(D) return np.dot(U, V.T) bayespy-0.1/bayespy/utils/covfunc/0000775000175000017500000000000012174043062021024 5ustar jluttinejluttine00000000000000bayespy-0.1/bayespy/utils/covfunc/distance.pyx0000664000175000017500000002137712174013072023370 0ustar jluttinejluttine00000000000000# Copyright (C) 2009 Nathaniel Smith # Copyright (C) 2012 Jaakko Luttinen # Released under the terms of the GNU GPL v3, or, at your option, any # later version. import warnings cimport stdlib cimport python as py import numpy as np cimport numpy as np from scipy import sparse # Put all relevant distance functions under this namespace from scipy.spatial.distance import pdist, cdist, squareform np.import_array() cdef extern from "numpy/arrayobject.h": # Cython 0.12 complains about PyTypeObject being an "incomplete type" on # this line: #py.PyTypeObject PyArray_Type # So use a hack: struct MyHackReallyPyTypeObject: pass MyHackReallyPyTypeObject PyArray_Type object PyArray_NewFromDescr(MyHackReallyPyTypeObject * subtype, np.dtype descr, int nd, np.npy_intp * dims, np.npy_intp * strides, void * data, int flags, object obj) # This is ridiculous: the description of PyArrayObject in numpy.pxd does # not mention the 'base' member, so we need a separate wrapper just to # expose it: ctypedef struct ndarray_with_base "PyArrayObject": void * base # In Cython 0.14.1, np.NPY_F_CONTIGUOUS is broken, because numpy.pxd # claims that it is of a non-existent type called 'enum requirements', and # new versions of Cython attempt to stash it in a temporary variable of # this type, which then annoys the C compiler. enum: NPY_F_CONTIGUOUS cdef inline np.ndarray set_base(np.ndarray arr, object base): cdef ndarray_with_base * hack = arr py.Py_INCREF(base) hack.base = base return arr cdef extern from "sparse_distance.h": cdef enum: FULL, LOWER, UPPER, STRICTLY_LOWER, STRICTLY_UPPER void sppdist_sqeuclidean(double *X, int m, int n, double threshold, int form, double **out_Dx, int **out_Dij, int *out_nzmax) void spcdist_sqeuclidean(double *X1, int m1, double *X2, int m2, int n, double threshold, double **out_Dx, int **out_Dij, int *out_nzmax) cdef object _integer_py_dtype = np.dtype(np.int32) assert sizeof(int) == _integer_py_dtype.itemsize == 4 cdef object _real_py_dtype = np.dtype(np.float64) assert sizeof(double) == _real_py_dtype.itemsize == 8 cdef object _complex_py_dtype = np.dtype(np.complex128) assert _complex_py_dtype.itemsize == 2 * sizeof(double) == 16 cdef class _CleanupCOO(object): cdef int * _ij cdef double * _x def __dealloc__(self): stdlib.free(self._ij) stdlib.free(self._x) cdef class _CleanupCSC(object): cdef int * _p cdef int * _i cdef double * _x def __dealloc__(self): stdlib.free(self._p) stdlib.free(self._i) stdlib.free(self._x) cdef _py_sparse_coo(double *x, int *ij, int m, int n, int nzmax): cdef _CleanupCOO cleaner = _CleanupCOO() cleaner._ij = ij cleaner._x = x # PyObject *PyArray_SimpleNewFromData(int nd, npy_intp* dims, int typenum, void* data) shape = (m, n) cdef np.npy_intp nz = nzmax cdef np.npy_intp nz2[2] nz2[0] = 2 nz2[1] = nzmax py.Py_INCREF(_integer_py_dtype) ijnd = set_base(PyArray_NewFromDescr(&PyArray_Type, _integer_py_dtype, 2, nz2, NULL, ij, NPY_F_CONTIGUOUS, None), cleaner) py.Py_INCREF(_real_py_dtype) data = set_base(PyArray_NewFromDescr(&PyArray_Type, _real_py_dtype, 1, &nz, NULL, x, NPY_F_CONTIGUOUS, None), cleaner) return sparse.coo_matrix((data, ijnd), shape=shape) cdef _py_sparse_csc(int *p, int *i, double *x, int m, int n, int nzmax): cdef _CleanupCSC cleaner = _CleanupCSC() cleaner._p = p cleaner._i = i cleaner._x = x # PyObject *PyArray_SimpleNewFromData(int nd, npy_intp* dims, int typenum, void* data) shape = (m, n) py.Py_INCREF(_integer_py_dtype) cdef np.npy_intp ncol_plus_1 = n + 1 indptr = set_base(PyArray_NewFromDescr(&PyArray_Type, _integer_py_dtype, 1, &ncol_plus_1, NULL, p, NPY_F_CONTIGUOUS, None), cleaner) py.Py_INCREF(_integer_py_dtype) cdef np.npy_intp nz = nzmax indices = set_base(PyArray_NewFromDescr(&PyArray_Type, _integer_py_dtype, 1, &nz, NULL, i, NPY_F_CONTIGUOUS, None), cleaner) py.Py_INCREF(_real_py_dtype) data = set_base(PyArray_NewFromDescr(&PyArray_Type, _real_py_dtype, 1, &nz, NULL, x, NPY_F_CONTIGUOUS, None), cleaner) return sparse.csc_matrix((data, indices, indptr), shape=shape) def sparse_pdist(X, threshold, form="strictly_lower", format="csc"): # Outputs cdef double *Dx = NULL cdef int *Dij = NULL cdef int nzmax cdef int uplo if np.ndim(X) == 0: X = np.atleast_2d(X) elif np.ndim(X) == 1: X = X[:,np.newaxis] (m,n) = X.shape # Check empty cases if m == 0: return np.empty((m,m)) #if n == 0: # return sp.csc_matrix((m,m)).asformat(format) cdef np.ndarray[np.double_t, ndim=2, mode="c"] X_c X_c = np.ascontiguousarray(X, dtype=np.double) if form.lower() == "strictly_lower": uplo = STRICTLY_LOWER elif form.lower() == "lower": uplo = LOWER elif form.lower() == "strictly_upper": uplo = STRICTLY_UPPER elif form.lower() == "upper": uplo = UPPER elif form.lower() == "full": uplo = FULL else: raise Exception, "Unknown form requested" sppdist_sqeuclidean( X_c.data, m, n, threshold, uplo, &Dx, &Dij, &nzmax) return _py_sparse_coo(Dx, Dij, m, m, nzmax).asformat(format) def sparse_cdist(X1, X2, threshold, format="csc"): # Outputs cdef double *Dx = NULL cdef int *Dij = NULL cdef int nzmax if np.ndim(X1) == 0: X1 = np.atleast_2d(X1) elif np.ndim(X1) == 1: X1 = X1[:,np.newaxis] elif np.ndim(X1) > 2: raise Exception, "Input matrices must be 0-2 -dimensional" if np.ndim(X2) == 0: X2 = np.atleast_2d(X2) elif np.ndim(X2) == 1: X2 = X2[:,np.newaxis] elif np.ndim(X2) > 2: raise Exception, "Input matrices must be 0-2 -dimensional" (m1,n1) = X1.shape (m2,n2) = X2.shape # Check empty cases if m1 == 0 or m2 == 0: return np.empty((m1,m2)) # Check that inputs have the same dimensionality if n1 != n2: raise Exception, "Matrices must have the same number of columns" n = n1 cdef np.ndarray[np.double_t, ndim=2, mode="c"] X1_c cdef np.ndarray[np.double_t, ndim=2, mode="c"] X2_c X1_c = np.ascontiguousarray(X1, dtype=np.double) X2_c = np.ascontiguousarray(X2, dtype=np.double) spcdist_sqeuclidean( X1_c.data, m1, X2_c.data, m2, n, threshold, &Dx, &Dij, &nzmax) return _py_sparse_coo(Dx, Dij, m1, m2, nzmax).asformat(format) __all__ = ["pdist", "cdist"] bayespy-0.1/bayespy/utils/covfunc/covariance.py0000664000175000017500000002767412174013072023526 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import itertools import numpy as np #import scipy as sp import scipy.sparse as sp # prefer CSC format #import scipy.spatial.distance as dist from bayespy.utils import utils from bayespy.utils.covfunc import distance import imp imp.reload(distance) imp.reload(utils) # Covariance matrices can be either arrays or matrices so be careful # with products and powers! Use explicit multiply or dot instead of # *-operator. def gp_cov_se(D2, overwrite=False): if overwrite: K = D2 K *= -0.5 np.exp(K, out=K) else: K = np.exp(-0.5*D2) return K def gp_cov_pp2_new(r, d, derivative=False): # Dimension dependent parameter q = 2 j = np.floor(d/2) + q + 1 # Polynomial coefficients a2 = j**2 + 4*j + 3 a1 = 3*j + 6 a0 = 3 # Two parts of the covariance function k1 = (1-r) ** (j+2) k2 = (a2*r**2 + a1*r + 3) # TODO: Check that derivative is 0, 1 or 2! if derivative == 0: # Return covariance return k1 * k2 / 3 dk1 = - (j+2) * (1-r)**(j+1) dk2 = 2*a2*r + a1 if derivative == 1: # Return first derivative of the covariance return (k1 * dk2 + dk1 * k2) / 3 ddk1 = (j+2) * (j+1) * (1-r)**j ddk2 = 2*a2 if derivative == 2: # Return second derivative of the covariance return (ddk1*k2 + 2*dk1*dk2 + k1*ddk2) / 3 def gp_cov_pp2(r, d, gradient=False): # Dimension dependent parameter j = np.floor(d/2) + 2 + 1 # Polynomial coefficients a2 = j**2 + 4*j + 3 a1 = 3*j + 6 a0 = 3 # Two parts of the covariance function k1 = (1-r) ** (j+2) k2 = (a2*r**2 + a1*r + 3) # The covariance function k = k1 * k2 / 3 if gradient: # The gradient w.r.t. r dk = k * (j+2) / (r-1) + k1 * (2*a2*r + a1) / 3 return (k, dk) else: return k def gp_cov_delta(N): # TODO: Use sparse matrices here! if N > 0: #print('in gpcovdelta', N, sp.identity(N).shape) return sp.identity(N) else: # Sparse matrices do not allow zero-length dimensions return np.identity(N) #return np.identity(N) #return np.asmatrix(np.identity(N)) def squared_distance(x1, x2): ## # Reshape arrays to 2-D arrays ## sh1 = np.shape(x1)[:-1] ## sh2 = np.shape(x2)[:-1] ## d = np.shape(x1)[-1] ## x1 = np.reshape(x1, (-1,d)) ## x2 = np.reshape(x2, (-1,d)) (m1,n1) = x1.shape (m2,n2) = x2.shape if m1 == 0 or m2 == 0: D2 = np.empty((m1,m2)) else: D2 = distance.cdist(x1, x2, metric='sqeuclidean') #D2 = np.asmatrix(D2) # Reshape the result #D2 = np.reshape(D2, sh1 + sh2) return D2 # General rule for the parameters for covariance functions: # # (value, [ [dvalue1, ...], [dvalue2, ...], [dvalue3, ...], ...]) # # For instance, # # k = covfunc_se((1.0, []), (15, [ [1,update_grad] ])) # K = k((x1, [ [dx1,update_grad] ]), (x2, [])) # # Plain values are converted as: # value -> (value, []) def gp_standardize_input(x): if np.size(x) == 0: x = np.reshape(x, (0,0)) elif np.ndim(x) == 0: x = np.reshape(x, (1,1)) elif np.ndim(x) == 1: x = np.reshape(x, (-1,1)) elif np.ndim(x) == 2: x = np.atleast_2d(x) else: raise Exception("Standard GP inputs must be 2-dimensional") return x def gp_preprocess_inputs(x1,x2=None): #args = list(args) #if len(args) < 1 or len(args) > 2: #raise Exception("Number of inputs must be one or two") if x2 is None: x1 = gp_standardize_input(x1) return x1 else: if x1 is x2: x1 = gp_standardize_input(x1) x2 = x1 else: x1 = gp_standardize_input(x1) x2 = gp_standardize_input(x2) return (x1, x2) #return args ## def gp_preprocess_inputs(x1,x2=None): ## #args = list(args) ## #if len(args) < 1 or len(args) > 2: ## #raise Exception("Number of inputs must be one or two") ## if x2 is not None: len(args) == 2: ## if args[0] is args[1]: ## args[0] = gp_standardize_input(args[0]) ## args[1] = args[0] ## else: ## args[1] = gp_standardize_input(args[1]) ## args[0] = gp_standardize_input(args[0]) ## else: ## args[0] = gp_standardize_input(args[0]) ## return args # TODO: # General syntax for these covariance functions: # covfunc(hyper1, # hyper2, # ... # hyperN, # x1, # x2=None, # gradient=list_of_booleans_for_each_hyperparameter) def covfunc_zeros(x1, x2=None, gradient=False): inputs = gp_preprocess_inputs(*inputs) # Compute distance and covariance matrix if x2 is None: x1 = gp_preprocess_inputs(x1) # Only variance vector asked N = np.shape(x1)[0] # TODO: Use sparse matrices! K = np.zeros(N) #K = np.asmatrix(np.zeros((N,1))) else: (x1,x2) = gp_preprocess_inputs(x1,x2) # Full covariance matrix asked #x1 = inputs[0] #x2 = inputs[1] # Number of inputs x1 N1 = np.shape(x1)[0] N2 = np.shape(x2)[0] # TODO: Use sparse matrices! K = np.zeros((N1,N2)) #K = np.asmatrix(np.zeros((N1,N2))) if gradient is not False: return (K, []) else: return K def covfunc_delta(amplitude, x1, x2=None, gradient=False): # Make sure that amplitude is a scalar, not an array object amplitude = utils.array_to_scalar(amplitude) ## if gradient: ## gradient_amplitude = gradient[0] ## else: ## gradient_amplitude = [] ## inputs = gp_preprocess_inputs(*inputs) # Compute distance and covariance matrix if x2 is None: x1 = gp_preprocess_inputs(x1) # Only variance vector asked #x = inputs[0] N = np.shape(x1)[0] K = np.ones(N) * amplitude**2 else: (x1,x2) = gp_preprocess_inputs(x1,x2) # Full covariance matrix asked #x1 = inputs[0] #x2 = inputs[1] # Number of inputs x1 N1 = np.shape(x1)[0] # x1 == x2? if x1 is x2: delta = True # Delta covariance # # FIXME: Broadcasting doesn't work with sparse matrices, # so must use scalar multiplication K = gp_cov_delta(N1) * amplitude**2 #K = gp_cov_delta(N1).multiply(amplitude**2) else: delta = False # Number of inputs x2 N2 = np.shape(x2)[0] # Zero covariance if N1 > 0 and N2 > 0: K = sp.csc_matrix((N1,N2)) else: K = np.zeros((N1,N2)) # Gradient w.r.t. amplitude if gradient: # FIXME: Broadcasting doesn't work with sparse matrices, # so must use scalar multiplication gradient_amplitude = K*(2/amplitude) print("noise grad", gradient_amplitude) return (K, (gradient_amplitude,)) else: return K def covfunc_pp2(amplitude, lengthscale, x1, x2=None, gradient=False): # Make sure that hyperparameters are scalars, not an array objects amplitude = utils.array_to_scalar(amplitude) lengthscale = utils.array_to_scalar(lengthscale) #amplitude = theta[0] #lengthscale = theta[1] ## if gradient: ## gradient_amplitude = gradient[0] ## gradient_lengthscale = gradient[1] ## else: ## gradient_amplitude = [] ## gradient_lengthscale = [] ## inputs = gp_preprocess_inputs(*inputs) # Compute covariance matrix if x2 is None: x1 = gp_preprocess_inputs(x1) # Compute variance vector K = np.ones(np.shape(x)[:-1]) K *= amplitude**2 # Compute gradient w.r.t. lengthscale if gradient: gradient_lengthscale = np.zeros(np.shape(x1)[:-1]) else: (x1,x2) = gp_preprocess_inputs(x1,x2) # Compute (sparse) distance matrix if x1 is x2: x1 = x1 / (lengthscale) x2 = x1 D2 = distance.sparse_pdist(x1, 1.0, form="full", format="csc") else: x1 = x1 / (lengthscale) x2 = x2 / (lengthscale) D2 = distance.sparse_cdist(x1, x2, 1.0, format="csc") r = np.sqrt(D2.data) N1 = np.shape(x1)[0] N2 = np.shape(x2)[0] # Compute the covariances if gradient: (k, dk) = gp_cov_pp2(r, np.shape(x1)[-1], gradient=True) else: k = gp_cov_pp2(r, np.shape(x1)[-1]) k *= amplitude**2 # Compute gradient w.r.t. lengthscale if gradient: if N1 >= 1 and N2 >= 1: dk *= r * (-amplitude**2 / lengthscale) gradient_lengthscale = sp.csc_matrix((dk, D2.indices, D2.indptr), shape=(N1,N2)) else: gradient_lengthscale = np.empty((N1,N2)) # Form sparse covariance matrix if N1 >= 1 and N2 >= 1: ## K = sp.csc_matrix((k, ij), shape=(N1,N2)) K = sp.csc_matrix((k, D2.indices, D2.indptr), shape=(N1,N2)) else: K = np.empty((N1, N2)) #print(K.__class__) # Gradient w.r.t. amplitude if gradient: gradient_amplitude = K * (2 / amplitude) # Return values if gradient: print("pp2 grad", gradient_lengthscale) return (K, (gradient_amplitude, gradient_lengthscale)) else: return K def covfunc_se(amplitude, lengthscale, x1, x2=None, gradient=False): # Make sure that hyperparameters are scalars, not an array objects amplitude = utils.array_to_scalar(amplitude) lengthscale = utils.array_to_scalar(lengthscale) # Compute covariance matrix if x2 is None: x1 = gp_preprocess_inputs(x1) #x = inputs[0] # Compute variance vector N = np.shape(x1)[0] K = np.ones(N) np.multiply(K, amplitude**2, out=K) # Compute gradient w.r.t. lengthscale if gradient: # TODO: Use sparse matrices? gradient_lengthscale = np.zeros(N) else: (x1,x2) = gp_preprocess_inputs(x1,x2) x1 = x1 / (lengthscale) x2 = x2 / (lengthscale) # Compute distance matrix K = squared_distance(x1, x2) # Compute gradient partly if gradient: gradient_lengthscale = np.divide(K, lengthscale) # Compute covariance matrix gp_cov_se(K, overwrite=True) np.multiply(K, amplitude**2, out=K) # Compute gradient w.r.t. lengthscale if gradient: gradient_lengthscale *= K # Gradient w.r.t. amplitude if gradient: gradient_amplitude = K * (2 / amplitude) # Return values if gradient: print("se grad", gradient_amplitude, gradient_lengthscale) return (K, (gradient_amplitude, gradient_lengthscale)) else: return K bayespy-0.1/bayespy/utils/covfunc/tests/0000775000175000017500000000000012174043062022166 5ustar jluttinejluttine00000000000000bayespy-0.1/bayespy/utils/covfunc/tests/test_distance.py0000664000175000017500000000547312174013072025400 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2012-2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### """ Unit tests for distance module. """ import unittest import numpy as np import scipy.sparse as sp import scipy.spatial.distance as dist from .. import distance as spdist #import scikits.sparse.distance as spdist import numpy.random as rand class TestDistance(unittest.TestCase): def test_pdist(self): N = 50 D = 3 x = rand.uniform(size=(N,D)) threshold = 0.1 i = np.arange(N)[:,np.newaxis] j = np.arange(N)[np.newaxis,:] # Compute full&dense distance matrix Dd = dist.squareform(dist.pdist(x, metric="sqeuclidean")) for form in ["lower", "strictly_lower", "upper", "strictly_upper", "full"]: Ds = spdist.sparse_pdist(x, threshold, form=form) self.assertTrue(sp.issparse(Ds)) Ds = Ds.tocsr() Ds.sort_indices() if form == "lower": D0 = Dd[i>=j] elif form == "strictly_lower": D0 = Dd[i>j] elif form == "upper": D0 = Dd[i<=j] elif form == "strictly_upper": D0 = Dd[i. * -------------------------------------------------------------------------- */ /* ----------------------------------------------------------------------------- * * Computes a sparse squared distance matrix. Squared distances that * are larger than a given threshold are ignored in the sparsity * structure. Zero distances are marked with explicit zeros. If X2 is * NULL, then a symmetric distance matrix (in lower triangular form) * is computed for X1. * * -------------------------------------------------------------------------- */ #include #include #include "sparse_distance.h" #define EXPAND_TRIPLET_IF_NEEDED(Dx,Dij,nzmax,nz) \ { \ if (nz == nzmax) \ { \ nzmax *= 2 ; \ Dij = realloc(Dij, 2*nzmax*sizeof(int)) ; \ Dx = realloc(Dx, nzmax*sizeof(double)) ; \ } \ } #define FORM_HAS_DIAGONAL(f) (((f) == FULL) || ((f) == LOWER) || ((f) == UPPER)) #define FORM_HAS_LOWER(f) (((f) == LOWER) || ((f) == STRICTLY_LOWER) || ((f) == FULL)) #define FORM_HAS_UPPER(f) (((f) == UPPER) || ((f) == STRICTLY_UPPER) || ((f) == FULL)) /* Compute squared Euclidean distance of two vectors */ inline double sqeuclidean_distance ( const double *x1, const double *x2, int n ) { int i ; double d, d2 = 0 ; for (i = 0; i < n; i++) { d = x1[i]-x2[i] ; d2 += d*d ; } return d2 ; } /* ========================================================================== */ /* === sppdist_sqeuclidean ================================================== */ /* ========================================================================== */ void sppdist_sqeuclidean ( /* ---- input ---- */ const double *X, /* input matrix, m-by-n in row-major form */ int m, /* m vectors */ int n, /* n-dimensional vectors */ double threshold, /* distance threshold */ int form, /* full / lower / strictly lower */ /* ---- output ---- */ double **out_Dx, /* distances */ int **out_Dij, /* row and column indices */ int *out_nzmax /* number of non-zeros, i.e., length of Di and Dx */ ) { int i, j ; int nz, nzmax; double d ; const double *x1, *x2 ; int *Dij ; double *Dx ; // Initial size for the number of non-zero elements nzmax = 2*m ; // Allocate initial memory Dij = realloc(*out_Dij, 2*nzmax*sizeof(int)) ; Dx = realloc(*out_Dx, nzmax*sizeof(double)) ; // Number of non-zeros thus far nz = 0 ; for (j = 0; j < m; j++) { if (FORM_HAS_DIAGONAL(form)) { Dij[2*nz] = j ; Dij[2*nz+1] = j ; Dx[nz] = 0.0 ; nz++ ; EXPAND_TRIPLET_IF_NEEDED(Dx, Dij, nzmax, nz) ; } for (i = j+1; i < m; i++) { // Compute distance between X1[:,i] and X2[:,j] x1 = X + i*n ; x2 = X + j*n ; d = sqeuclidean_distance(x1, x2, n) ; // Store distances that are small enough if (d <= threshold) { // Store strictly lower triangular part if (FORM_HAS_LOWER(form)) { Dij[2*nz] = i ; Dij[2*nz+1] = j ; Dx[nz] = d ; nz++ ; EXPAND_TRIPLET_IF_NEEDED(Dx, Dij, nzmax, nz) ; } if (FORM_HAS_UPPER(form)) { // Store strictly upper triangular part Dij[2*nz] = j ; Dij[2*nz+1] = i ; Dx[nz] = d ; nz++ ; EXPAND_TRIPLET_IF_NEEDED(Dx, Dij, nzmax, nz) ; } } } } // Store the matrix compactly nzmax = nz ; Dij = realloc(Dij, 2*nzmax*sizeof(int)) ; Dx = realloc(Dx, nzmax*sizeof(double)) ; // Set output variables *out_Dij = Dij ; *out_Dx = Dx ; *out_nzmax = nzmax ; return ; } void spcdist_sqeuclidean ( /* ---- input ---- */ const double *X1, /* input matrix, m-by-n in row-major form */ int m1, /* m1 vectors */ const double *X2, /* input matrix, m-by-n in row-major form */ int m2, /* m2 vectors */ int n, /* n-dimensional vectors */ double threshold, /* distance threshold */ /* ---- output ---- */ double **out_Dx, /* distances */ int **out_Dij, /* row and column indices */ int *out_nzmax /* number of non-zeros */ ) { int i, j ; int nz, nzmax; double d ; const double *x1, *x2 ; int *Dij ; double *Dx ; // Initial size for the number of non-zero elements nzmax = 2 * (m1 > m2 ? m1 : m2) ; // Allocate initial memory Dij = realloc(*out_Dij, 2*nzmax*sizeof(int)) ; Dx = realloc(*out_Dx, nzmax*sizeof(double)) ; // Number of non-zeros thus far nz = 0 ; for (i = 0; i < m1; i++) { for (j = 0; j < m2; j++) { // Compute distance between X1[:,i] and X2[:,j] x1 = X1 + i*n ; x2 = X2 + j*n ; d = sqeuclidean_distance(x1, x2, n) ; // Store distances that are small enough if (d <= threshold) { Dij[2*nz] = i ; Dij[2*nz+1] = j ; Dx[nz] = d ; nz++ ; EXPAND_TRIPLET_IF_NEEDED(Dx, Dij, nzmax, nz) ; } } } // Store the matrix compactly nzmax = nz ; Dij = realloc(Dij, 2*nzmax*sizeof(int)) ; Dx = realloc(Dx, nzmax*sizeof(double)) ; // Set output variables *out_Dij = Dij ; *out_Dx = Dx ; *out_nzmax = nzmax ; return ; } bayespy-0.1/bayespy/utils/covfunc/sparse_distance/sparse_distance.h0000664000175000017500000000465512174013072027523 0ustar jluttinejluttine00000000000000/* ========================================================================== */ /* === sparse_distance.h ==================================================== */ /* ========================================================================== */ /* ----------------------------------------------------------------------------- * Copyright (C) 2012 Jaakko Luttinen * * cholmod_spdist.h is licensed under Version 3.0 of the GNU General * Public License. See LICENSE for a text of the license. * -------------------------------------------------------------------------- */ /* ----------------------------------------------------------------------------- * This file is free software: you can redistribute it and/or modify * it under the terms of the GNU General Public License as published * by the Free Software Foundation, either version 3 of the License, * or (at your option) any later version. * * This file is distributed in the hope that it will be useful, but * WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this file. If not, see . * -------------------------------------------------------------------------- */ #ifndef SPARSE_DISTANCE_H #define SPARSE_DISTANCE_H #define FULL 0 #define LOWER 1 #define UPPER 2 #define STRICTLY_LOWER 3 #define STRICTLY_UPPER 4 void sppdist_sqeuclidean ( /* ---- input ---- */ const double *X, /* input matrix, m-by-n in row-major form */ int m, /* m vectors */ int n, /* n-dimensional vectors */ double threshold, /* distance threshold */ int form, /* full / lower / strictly lower */ /* ---- output ---- */ double **out_Dx, /* distances */ int **out_Dij, /* row and column indices */ int *out_nzmax /* number of non-zeros */ ) ; void spcdist_sqeuclidean ( /* ---- input ---- */ const double *X1, /* input matrix, m-by-n in row-major form */ int m1, /* m1 vectors */ const double *X2, /* input matrix, m-by-n in row-major form */ int m2, /* m2 vectors */ int n, /* n-dimensional vectors */ double threshold, /* distance threshold */ /* ---- output ---- */ double **out_Dx, /* distances */ int **out_Dij, /* row and column indices */ int *out_nzmax /* number of non-zeros */ ) ; #endif bayespy-0.1/bayespy/utils/tests/0000775000175000017500000000000012174043062020523 5ustar jluttinejluttine00000000000000bayespy-0.1/bayespy/utils/tests/test_linalg.py0000664000175000017500000001151012174013072023376 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### """ Unit tests for bayespy.utils.utils module. """ #import unittest import numpy as np #from numpy import testing from ...utils.utils import TestCase from .. import linalg #from bayespy.utils import utils class TestDot(TestCase): def test_dot(self): """ Test dot product multiple multi-dimensional arrays. """ # If no arrays, return 0 self.assertAllClose(linalg.dot(), 0) # If only one array, return itself self.assertAllClose(linalg.dot([[1,2,3], [4,5,6]]), [[1,2,3], [4,5,6]]) # Basic test of two arrays: (2,3) * (3,2) self.assertAllClose(linalg.dot([[1,2,3], [4,5,6]], [[7,8], [9,1], [2,3]]), [[31,19], [85,55]]) # Basic test of four arrays: (2,3) * (3,2) * (2,1) * (1,2) self.assertAllClose(linalg.dot([[1,2,3], [4,5,6]], [[7,8], [9,1], [2,3]], [[4], [5]], [[6,7]]), [[1314,1533], [3690,4305]]) # Test broadcasting: (2,2,2) * (2,2,2,2) self.assertAllClose(linalg.dot([[[1,2], [3,4]], [[5,6], [7,8]]], [[[[1,2], [3,4]], [[5,6], [7,8]]], [[[9,1], [2,3]], [[4,5], [6,7]]]]), [[[[ 7, 10], [ 15, 22]], [[ 67, 78], [ 91, 106]]], [[[ 13, 7], [ 35, 15]], [[ 56, 67], [ 76, 91]]]]) # Inconsistent shapes: (2,3) * (2,3) self.assertRaises(ValueError, linalg.dot, [[1,2,3], [4,5,6]], [[1,2,3], [4,5,6]]) # Other axes do not broadcast: (2,2,2) * (3,2,2) self.assertRaises(ValueError, linalg.dot, [[[1,2], [3,4]], [[5,6], [7,8]]], [[[1,2], [3,4]], [[5,6], [7,8]], [[9,1], [2,3]]]) # Do not broadcast matrix axes: (2,1) * (3,2) self.assertRaises(ValueError, linalg.dot, [[1], [2]], [[1,2,3], [4,5,6]]) # Do not accept less than 2-D arrays: (2) * (2,2) self.assertRaises(ValueError, linalg.dot, [1,2], [[1,2,3], [4,5,6]]) bayespy-0.1/bayespy/utils/tests/__init__.py0000664000175000017500000000000012174013072022620 0ustar jluttinejluttine00000000000000bayespy-0.1/bayespy/utils/tests/test_utils.py0000664000175000017500000002740412174013072023301 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### """ Unit tests for bayespy.utils.utils module. """ import unittest import numpy as np from numpy import testing from .. import utils #from bayespy.utils import utils class TestBroadcasting(unittest.TestCase): def test_is_shape_subset(self): f = utils.is_shape_subset self.assertTrue(f( (), () )) self.assertTrue(f( (), (3,) )) self.assertTrue(f( (1,), (1,) )) self.assertTrue(f( (1,), (3,) )) self.assertTrue(f( (1,), (4,1) )) self.assertTrue(f( (1,), (4,3) )) self.assertTrue(f( (1,), (1,3) )) self.assertTrue(f( (3,), (1,3) )) self.assertTrue(f( (3,), (4,3) )) self.assertTrue(f( (5,1,3), (6,5,4,3) )) self.assertTrue(f( (5,4,3), (6,5,4,3) )) self.assertFalse(f( (1,), () )) self.assertFalse(f( (3,), (1,) )) self.assertFalse(f( (4,3,), (3,) )) self.assertFalse(f( (4,3,), (1,3,) )) self.assertFalse(f( (6,1,4,3,), (6,1,1,3,) )) class TestMultiplyShapes(unittest.TestCase): def test_multiply_shapes(self): f = lambda *shapes: tuple(utils.multiply_shapes(*shapes)) # Basic test self.assertEqual(f((2,), (3,)), (6,)) # Test multiple arguments self.assertEqual(f((2,), (3,), (4,)), (24,)) # Test different lengths and multiple arguments self.assertEqual(f(( 2,3,), (4,5,6,), ( 7,)), (4,10,126,)) # Test empty shapes self.assertEqual(f((), ()), ()) self.assertEqual(f((), (5,)), (5,)) class TestSumMultiply(unittest.TestCase): def check_sum_multiply(self, *shapes, **kwargs): # The set of arrays x = list() for (ind, shape) in enumerate(shapes): x += [np.random.randn(*shape)] # Result from the function yh = utils.sum_multiply(*x, **kwargs) axis = kwargs.get('axis', None) sumaxis = kwargs.get('sumaxis', True) keepdims = kwargs.get('keepdims', False) # Compute the product y = 1 for xi in x: y = y * xi # Compute the sum if sumaxis: y = np.sum(y, axis=axis, keepdims=keepdims) else: axes = np.arange(np.ndim(y)) # TODO/FIXME: np.delete has a bug that it doesn't accept negative # indices. Thus, transform negative axes to positive axes. if len(axis) > 0: axis = [i if i >= 0 else i+np.ndim(y) for i in axis] elif axis < 0: axis += np.ndim(y) axes = np.delete(axes, axis) axes = tuple(axes) if len(axes) > 0: y = np.sum(y, axis=axes, keepdims=keepdims) # Check the result testing.assert_allclose(yh, y, err_msg="Incorrect value.") def test_sum_multiply(self): """ Test utils.sum_multiply. """ # Check empty list returns error self.assertRaises(ValueError, self.check_sum_multiply) # Check scalars self.check_sum_multiply(()) self.check_sum_multiply((), (), ()) # Check doing no summation self.check_sum_multiply((3,), axis=()) self.check_sum_multiply((3,1,5), ( 4,1), ( 5,), ( ), axis=(), keepdims=True) # Check AXES_TO_SUM self.check_sum_multiply((3,1), (1,4), (3,4), axis=(1,)) self.check_sum_multiply((3,1), (1,4), (3,4), axis=(-2,)) self.check_sum_multiply((3,1), (1,4), (3,4), axis=(1,-2)) # Check AXES_TO_SUM and KEEPDIMS self.check_sum_multiply((3,1), (1,4), (3,4), axis=(1,), keepdims=True) self.check_sum_multiply((3,1), (1,4), (3,4), axis=(-2,), keepdims=True) self.check_sum_multiply((3,1), (1,4), (3,4), axis=(1,-2,), keepdims=True) self.check_sum_multiply((3,1,5,6), ( 4,1,6), ( 4,1,1), ( ), axis=(1,-2), keepdims=True) # Check AXES_TO_KEEP self.check_sum_multiply((3,1), (1,4), (3,4), sumaxis=False, axis=(1,)) self.check_sum_multiply((3,1), (1,4), (3,4), sumaxis=False, axis=(-2,)) self.check_sum_multiply((3,1), (1,4), (3,4), sumaxis=False, axis=(1,-2)) # Check AXES_TO_KEEP and KEEPDIMS self.check_sum_multiply((3,1), (1,4), (3,4), sumaxis=False, axis=(1,), keepdims=True) self.check_sum_multiply((3,1), (1,4), (3,4), sumaxis=False, axis=(-2,), keepdims=True) self.check_sum_multiply((3,1), (1,4), (3,4), sumaxis=False, axis=(1,-2,), keepdims=True) self.check_sum_multiply((3,1,5,6), ( 4,1,6), ( 4,1,1), ( ), sumaxis=False, axis=(1,-2,), keepdims=True) # Check errors # Inconsistent shapes self.assertRaises(ValueError, self.check_sum_multiply, (3,4), (3,5)) # Axis index out of bounds self.assertRaises(ValueError, self.check_sum_multiply, (3,4), (3,4), axis=(-3,)) self.assertRaises(ValueError, self.check_sum_multiply, (3,4), (3,4), axis=(2,)) self.assertRaises(ValueError, self.check_sum_multiply, (3,4), (3,4), sumaxis=False, axis=(-3,)) self.assertRaises(ValueError, self.check_sum_multiply, (3,4), (3,4), sumaxis=False, axis=(2,)) # Same axis several times self.assertRaises(ValueError, self.check_sum_multiply, (3,4), (3,4), axis=(1,-1)) self.assertRaises(ValueError, self.check_sum_multiply, (3,4), (3,4), sumaxis=False, axis=(1,-1)) class TestBandedSolve(unittest.TestCase): def test_block_banded_solve(self): """ Test the Gaussian elimination algorithm for block-banded matrices. """ # # Create a block-banded matrix # # Number of blocks N = 40 # Random sizes of the blocks #D = np.random.randint(5, 10, size=N) # Fixed sizes of the blocks D = 5*np.ones(N) # Some helpful variables to create the covariances W = [np.random.randn(D[i], 2*D[i]) for i in range(N)] # The diagonal blocks (covariances) A = [np.dot(W[i], W[i].T) for i in range(N)] # The superdiagonal blocks (cross-covariances) B = [np.dot(W[i][:,-1:], W[i+1][:,:1].T) for i in range(N-1)] C = utils.block_banded(A, B) # Create the system to be solved: y=C*x x_true = np.random.randn(np.sum(D), 3) y = np.dot(C, x_true) x_true = np.reshape(x_true, (N, -1, 3)) y = np.reshape(y, (N, -1, 3)) # # Run tests # # The correct inverse invC = np.linalg.inv(C) # Inverse from the function that is tested (invA, invB, x, ldet) = utils.block_banded_solve(A,B,y) #print("inv(C)=", invC) #print("blocks=", invA, invB) # Check that you get the correct number of blocks self.assertEqual(len(invA), N) self.assertEqual(len(invB), N-1) # Check each block i0 = 0 for i in range(N-1): i1 = i0 + D[i] i2 = i1 + D[i+1] # Check diagonal block self.assertTrue(np.allclose(invA[i], invC[i0:i1, i0:i1])) # Check super-diagonal block self.assertTrue(np.allclose(invB[i], invC[i0:i1, i1:i2])) i0 = i1 # Check last block self.assertTrue(np.allclose(invA[-1], invC[i0:, i0:])) # Check the solution of the system #print(x_true, x) self.assertTrue(np.allclose(x_true, x)) # Check the log determinant self.assertAlmostEqual(ldet/np.linalg.slogdet(C)[1], 1) ## if __name__ == '__main__': ## unittest.main() bayespy-0.1/bayespy/utils/__init__.py0000664000175000017500000000012512174013072021466 0ustar jluttinejluttine00000000000000from . import utils from . import linalg from . import random from . import optimize bayespy-0.1/bayespy/utils/optimize.py0000664000175000017500000000365512174013072021602 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011-2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np from scipy import optimize def minimize(f, x0, maxiter=None, verbose=False): """ Simple wrapper for SciPy's optimize. The given function must return a tuple: (value, gradient). """ options = {'disp': verbose} if maxiter is not None: options['maxiter'] = maxiter opt = optimize.minimize(f, x0, jac=True, method='CG', options=options) return opt.x def check_gradient(f, x0, verbose=True): """ Simple wrapper for SciPy's gradient checker. The given function must return a tuple: (value, gradient). Returns relative """ df = f(x0)[1] df_num = optimize.approx_fprime(x0, lambda x: f(x)[0], optimize.optimize._epsilon) err = np.linalg.norm(df-df_num) / np.linalg.norm(df_num) if verbose: print("Gradient relative error = %g" % err) return err bayespy-0.1/bayespy/utils/linalg.py0000664000175000017500000002333112174013072021201 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011-2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### """ General numerical functions and methods. """ import itertools import numpy as np import scipy as sp #import scipy.linalg.decomp_cholesky as decomp import scipy.linalg as linalg import scipy.special as special import scipy.optimize as optimize import scipy.sparse as sparse #import scikits.sparse.cholmod as cholmod # THIS IS SOME NEW GENERALIZED UFUNC FOR LINALG FEATURE, NOT IN OFFICIAL NUMPY # REPO YET #import numpy.linalg._gufuncs_linalg as gula #import numpy.core.gufuncs_linalg as gula from .utils import nested_iterator def chol(C): if sparse.issparse(C): # Sparse Cholesky decomposition (returns a Factor object) return cholmod.cholesky(C) else: # Computes Cholesky decomposition for a collection of matrices. # The last two axes of C are considered as the matrix. C = np.atleast_2d(C) U = np.empty(np.shape(C)) for i in nested_iterator(np.shape(U)[:-2]): try: U[i] = linalg.cho_factor(C[i])[0] except np.linalg.linalg.LinAlgError: print(C[i]) raise Exception("Matrix not positive definite") return U def chol_solve(U, b): if isinstance(U, np.ndarray): if sparse.issparse(b): b = b.toarray() # Allocate memory U = np.atleast_2d(U) B = np.atleast_1d(B) sh_u = U.shape[:-2] sh_b = B.shape[:-1] l_u = len(sh_u) l_b = len(sh_b) # Check which axis are iterated over with B along with U ind_b = [Ellipsis] * l_b l_min = min(l_u, l_b) jnd_b = tuple(i for i in range(-l_min,0) if sh_b[i]==sh_u[i]) if out == None: # Shape of the result (broadcasting rules) sh = broadcasted_shape(sh_u, sh_b) #out = np.zeros(np.shape(B)) out = np.zeros(sh + B.shape[-1:]) for i in nested_iterator(np.shape(U)[:-2]): # The goal is to run Cholesky solver once for all vectors of B # for which the matrices of U are the same (according to the # broadcasting rules). Thus, we collect all the axes of B for # which U is singleton and form them as a 2-D matrix and then # run the solver once. # Select those axes of B for which U and B are not singleton for j in jnd_b: ind_b[j] = i[j] # Collect all the axes for which U is singleton b = B[tuple(ind_b) + (Ellipsis,)] # Reshape it to a 2-D (or 1-D) array orig_shape = b.shape if b.ndim > 1: b = b.reshape((-1, b.shape[-1])) # Ellipsis to all preceeding axes and ellipsis for the last # axis: if len(ind_b) < len(sh): ind_out = (Ellipsis,) + tuple(ind_b) + (Ellipsis,) else: ind_out = tuple(ind_b) + (Ellipsis,) out[ind_out] = linalg.cho_solve((U[i], False), b.T).T.reshape(orig_shape) return out elif isinstance(U, cholmod.Factor): if sparse.issparse(b): b = b.toarray() return U.solve_A(b) else: raise ValueError("Unknown type of Cholesky factor") def chol_inv(U): if isinstance(U, np.ndarray): # Allocate memory V = np.tile(np.identity(np.shape(U)[-1]), np.shape(U)[:-2]+(1,1)) for i in nested_iterator(np.shape(U)[:-2]): V[i] = linalg.cho_solve((U[i], False), V[i], overwrite_b=True) # This would need Fortran order return V elif isinstance(U, cholmod.Factor): raise NotImplementedError ## if sparse.issparse(b): ## b = b.toarray() ## return U.solve_A(b) else: raise ValueError("Unknown type of Cholesky factor") def chol_logdet(U): if isinstance(U, np.ndarray): return 2*np.sum(np.log(np.diag(U))) elif isinstance(U, cholmod.Factor): return np.sum(np.log(U.D())) else: raise ValueError("Unknown type of Cholesky factor") def logdet_chol(U): if isinstance(U, np.ndarray): # Computes Cholesky decomposition for a collection of matrices. return 2*np.sum(np.log(np.einsum('...ii->...i', U)), axis=(-1,)) elif isinstance(U, cholmod.Factor): return np.sum(np.log(U.D())) def logdet_tri(R): """ Logarithm of the absolute value of the determinant of a triangular matrix. """ return np.sum(np.log(np.abs(np.einsum('...ii->...i', R)))) def logdet_cov(C): return logdet_chol(chol(C)) def m_solve_triangular(U, B, **kwargs): # Allocate memory U = np.atleast_2d(U) B = np.atleast_1d(B) sh_u = U.shape[:-2] sh_b = B.shape[:-1] l_u = len(sh_u) l_b = len(sh_b) # Check which axis are iterated over with B along with U ind_b = [Ellipsis] * l_b l_min = min(l_u, l_b) jnd_b = tuple(i for i in range(-l_min,0) if sh_b[i]==sh_u[i]) # Shape of the result (broadcasting rules) sh = broadcasted_shape(sh_u, sh_b) #out = np.zeros(np.shape(B)) out = np.zeros(sh + B.shape[-1:]) ## if out == None: ## # Shape of the result (broadcasting rules) ## sh = broadcasted_shape(sh_u, sh_b) ## #out = np.zeros(np.shape(B)) ## out = np.zeros(sh + B.shape[-1:]) for i in nested_iterator(np.shape(U)[:-2]): # The goal is to run triangular solver once for all vectors of # B for which the matrices of U are the same (according to the # broadcasting rules). Thus, we collect all the axes of B for # which U is singleton and form them as a 2-D matrix and then # run the solver once. # Select those axes of B for which U and B are not singleton for j in jnd_b: ind_b[j] = i[j] # Collect all the axes for which U is singleton b = B[tuple(ind_b) + (Ellipsis,)] # Reshape it to a 2-D (or 1-D) array orig_shape = b.shape if b.ndim > 1: b = b.reshape((-1, b.shape[-1])) # Ellipsis to all preceeding axes and ellipsis for the last # axis: if len(ind_b) < len(sh): ind_out = (Ellipsis,) + tuple(ind_b) + (Ellipsis,) else: ind_out = tuple(ind_b) + (Ellipsis,) #print('utils.m_solve_triangular', np.shape(U[i]), np.shape(b)) out[ind_out] = linalg.solve_triangular(U[i], b.T, **kwargs).T.reshape(orig_shape) #out[ind_out] = out[ind_out].T.reshape(orig_shape) return out def outer(A,B): # Computes outer product over the last axes of A and B. The other # axes are broadcasted. Thus, if A has shape (..., N) and B has # shape (..., M), then the result has shape (..., N, M) return A[...,np.newaxis]*B[...,np.newaxis,:] def dot(*arrays): """ Compute matrix-matrix product. You can give multiple arrays, the dot product is computed from left to right: A1*A2*A3*...*AN. The dot product is computed over the last two axes of each arrays. All other axes must be broadcastable. """ if len(arrays) == 0: return 0 else: Y = np.asanyarray(arrays[0]) for X in arrays[1:]: X = np.asanyarray(X) if np.ndim(Y) < 2 or np.ndim(X) < 2: raise ValueError("Must be at least 2-D arrays") if np.shape(Y)[-1] != np.shape(X)[-2]: raise ValueError("Dimensions do not match") Y = np.einsum('...ik,...kj->...ij', Y, X) #Y = gula.matrix_multiply(Y, X) return Y def tracedot(A, B): """ Computes trace(A*B). """ return np.einsum('...ij,...ji->...', A, B) def inv(A): if np.ndim(A) == 2: return np.linalg.inv(A) else: raise NotImplementedError() # return gula.inv(A) def mvdot(A, b): """ Compute matrix-vector product. Applies broadcasting. """ # TODO/FIXME: A bug in inner1d: # https://github.com/numpy/numpy/issues/3338 # # b = np.asanyarray(b) # return gula.inner1d(A, b[...,np.newaxis,:]) # # Use einsum instead: return np.einsum('...ik,...k->...i', A, b) def m_dot(A,b): raise DeprecationWarning() # Compute matrix-vector product over the last two axes of A and # the last axes of b. Other axes are broadcasted. If A has shape # (..., M, N) and b has shape (..., N), then the result has shape # (..., M) #b = reshape(b, shape(b)[:-1] + (1,) + shape(b)[-1:]) #return np.dot(A, b) return np.einsum('...ik,...k->...i', A, b) # TODO: Use einsum!! #return np.sum(A*b[...,np.newaxis,:], axis=(-1,)) bayespy-0.1/bayespy/plot/0000775000175000017500000000000012174043062017177 5ustar jluttinejluttine00000000000000bayespy-0.1/bayespy/plot/plotting.py0000664000175000017500000002033712174013072021414 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np import scipy.sparse as sp import matplotlib.pyplot as plt from matplotlib import animation #from matplotlib.pyplot import * from bayespy import utils def timeseries_gaussian(X, axis=-1): """ Parameters: X : node Node with Gaussian moments. axis : int The index of the time axis. """ u_X = X.get_moments() x = u_X[0] xx = u_X[1] std = np.sqrt(np.einsum('...ii->...i', xx) - x**2) _timeseries_mean_and_std(x, std, axis=axis) def timeseries_normal(X, axis=-1): """ Parameters: X : node Node with Gaussian moments. axis : int The index of the time axis. """ u_X = X.get_moments() x = u_X[0] xx = u_X[1] std = np.sqrt(xx - x**2) _timeseries_mean_and_std(x, std, axis=axis) def timeseries(x, axis=-1): return _timeseries_mean_and_std(x, 0*x, axis=axis) def _timeseries_mean_and_std(y, std, axis=-1, **kwargs): # TODO/FIXME: You must multiply by ones(plates) in order to plot # broadcasted plates properly shape = list(np.shape(y)) # Get and remove the length of the time axis T = shape.pop(axis) # Move time axis to first y = np.rollaxis(y, axis) std = np.rollaxis(std, axis) y = np.reshape(y, (T, -1)) std = np.reshape(std, (T, -1)) # Remove 1s shape = [s for s in shape if s > 1] # Calculate number of rows and columns shape = utils.utils.multiply_shapes(shape, (1,1)) if len(shape) > 2: raise Exception("Can plot only in 2 dimensions (rows and columns)") (M, N) = shape # Prefer plotting to rows if M == 1: M = N N = 1 # Plot each timeseries #M = 2 for i in range(M*N): plt.subplot(M,N,i+1) errorplot(y=y[:,i], error=2*std[:,i], **kwargs) def matrix(A): A = np.atleast_2d(A) vmax = np.max(np.abs(A)) return plt.imshow(A, interpolation='nearest', cmap='RdBu_r', vmin=-vmax, vmax=vmax) def matrix_animation(A, **kwargs): A = np.atleast_3d(A) vmax = np.max(np.abs(A)) x = plt.imshow(A[0], interpolation='nearest', cmap='RdBu_r', vmin=-vmax, vmax=vmax, **kwargs) s = plt.title('t = %d' % 0) for (t, a) in enumerate(A): x.set_array(a) s.set_text('t = %d' % t) plt.draw() def matrix_movie(A, filename, fps=25, title='Matrix Movie', artist='BayesPy', dpi=100, **kwargs): FFMpegWriter = animation.writers['ffmpeg'] metadata = dict(title=title, artist=artist) writer = FFMpegWriter(fps=fps, metadata=metadata) A = np.atleast_3d(A) vmax = np.max(np.abs(A)) x = plt.imshow(A[0], interpolation='nearest', cmap='RdBu_r', vmin=-vmax, vmax=vmax, **kwargs) s = plt.title('t = %d' % 0) fig = plt.gcf() with writer.saving(fig, filename, dpi): for (t, a) in enumerate(A): x.set_array(a) s.set_text('t = %d' % t) plt.draw() writer.grab_frame() def binary_matrix(A): A = np.atleast_2d(A) G = np.zeros(np.shape(A) + (3,)) G[A] = [0,0,0] G[np.logical_not(A)] = [1,1,1] plt.imshow(G, interpolation='nearest') def gaussian_mixture(w, mu, Sigma): pass def gaussian_mixture_logpdf(x, w, mu, Sigma): # Shape(x) = (N, D) # Shape(w) = (K,) # Shape(mu) = (K, D) # Shape(Sigma) = (K, D, D) # Shape(result) = (N,) # Dimensionality D = np.shape(x)[-1] # Cholesky decomposition of the covariance matrix U = utils.m_chol(Sigma) # Reshape x: # Shape(x) = (N, 1, D) x = np.expand_dims(x, axis=-2) # (x-mu) and (x-mu)'*inv(Sigma)*(x-mu): # Shape(v) = (N, K, D) # Shape(z) = (N, K) v = x - mu z = np.einsum('...i,...i', v, utils.m_chol_solve(U, v)) # Log-determinant of Sigma: # Shape(ldet) = (K,) ldet = utils.m_chol_logdet(U) # Compute log pdf for each cluster: # Shape(lpdf) = (N, K) lpdf = utils.gaussian_logpdf(z, 0, 0, ldet, D) def matrixplot(A, colorbar=False): if sp.issparse(A): A = A.toarray() plt.imshow(A, interpolation='nearest') if colorbar: plt.colorbar() def contourplot(x1, x2, y, colorbar=False, filled=True): """ Plots 2D contour plot. x1 and x2 are 1D vectors, y contains the function values. y.size must be x1.size*x2.size. """ y = np.reshape(y, (len(x2),len(x1))) if filled: plt.contourf(x1, x2, y) else: plt.contour(x1, x2, y) if colorbar: plt.colorbar() def errorplot(y=None, error=None, x=None, lower=None, upper=None): # Default inputs if x is None: x = np.arange(np.size(y)) # Parse errors (lower=lower/error/upper, upper=upper/error/lower) if lower is None: if error is not None: lower = error elif upper is not None: lower = upper if upper is None: if error is not None: upper = error elif lower is not None: upper = lower # Plot errors if (lower is not None) and (upper is not None): #print(np.max(lower)) #print(np.max(upper)) l = y - lower u = y + upper plt.fill_between(x, u, l, facecolor=(0.6,0.6,0.6,1), edgecolor=(0,0,0,0), linewidth=0, interpolate=True) # Plot function plt.plot(x, y, color=(0,0,0,1)) #def multiplot(plot_function, *args, **kwargs): def m_plot(x, Y, style): Y = np.atleast_2d(Y) M = Y.shape[-2] for i in range(M): plt.subplot(M,1,i+1) plt.plot(x, Y[i], style) ## def multi_errorplot(Y, error=None, x=None, lower=None, upper=None): ## for m in range(M): ## for n in range(N): ## plt.subplot(M,N,m*N+n) ## errorplot(Y[m][n], ## error=error[m][n], ## x=x[m][n], ## lower=lower[m][n], ## upper=upper[m][n]) def m_errorplot(x, Y, L, U): Y = np.atleast_2d(Y) L = np.atleast_2d(L) U = np.atleast_2d(U) M = Y.shape[-2] ## print(np.shape(Y)) ## print(np.shape(L)) ## print(np.shape(U)) ## print(np.shape(M)) for i in range(M): plt.subplot(M,1,i+1) lower = Y[i] - L[i] upper = Y[i] + U[i] #print(upper-lower) #if np.any(lower>=upper): #print('WTF?!') plt.fill_between(x, upper, lower, #where=(upper>=lower), facecolor=(0.6,0.6,0.6,1), edgecolor=(0,0,0,0), #edgecolor=(0.6,0.6,0.6,1), linewidth=0, interpolate=True) plt.plot(x, Y[i], color=(0,0,0,1)) plt.ylabel(str(i)) bayespy-0.1/bayespy/plot/__init__.py0000664000175000017500000000000012174013072021274 0ustar jluttinejluttine00000000000000bayespy-0.1/bayespy/demos/0000775000175000017500000000000012174043062017330 5ustar jluttinejluttine00000000000000bayespy-0.1/bayespy/demos/demo_pca.py0000664000175000017500000000647512174013072021463 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011-2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import time import numpy as np import matplotlib.pyplot as plt from bayespy.plot import plotting as myplt from bayespy import utils from bayespy.inference.vmp import nodes from bayespy.inference.vmp.vmp import VB from bayespy.inference.vmp import transformations from bayespy.inference.vmp.nodes.gamma import diagonal def pca_model(M, N, D): # Construct the PCA model with ARD # ARD alpha = nodes.Gamma(1e-2, 1e-2, plates=(D,), name='alpha') # Loadings W = nodes.Gaussian(np.zeros(D), diagonal(alpha), name="W", plates=(M,1)) # States X = nodes.Gaussian(np.zeros(D), np.identity(D), name="X", plates=(1,N)) # PCA WX = nodes.Dot(W, X, name="WX") # Noise tau = nodes.Gamma(1e-2, 1e-2, name="tau", plates=()) # Noisy observations Y = nodes.Normal(WX, tau, name="Y", plates=(M,N)) return (Y, WX, W, X, tau, alpha) def run(M=10, N=100, D_y=3, D=5): seed = 45 print('seed =', seed) np.random.seed(seed) # Generate data w = np.random.normal(0, 1, size=(M,1,D_y)) x = np.random.normal(0, 1, size=(1,N,D_y)) f = utils.utils.sum_product(w, x, axes_to_sum=[-1]) y = f + np.random.normal(0, 0.5, size=(M,N)) # Construct model (Y, WX, W, X, tau, alpha) = pca_model(M, N, D) # Data with missing values mask = utils.random.mask(M, N, p=0.9) # randomly missing mask[:,20:40] = False # gap missing y[~mask] = np.nan Y.observe(y, mask=mask) # Construct inference machine Q = VB(Y, W, X, tau, alpha) # Initialize some nodes randomly X.initialize_from_value(X.random()) W.initialize_from_value(W.random()) # Inference loop. Q.update(repeat=100) # Plot results plt.clf() WX_params = WX.get_parameters() fh = WX_params[0] * np.ones(y.shape) err_fh = 2*np.sqrt(WX_params[1] + 1/tau.get_moments()[0]) * np.ones(y.shape) for m in range(M): plt.subplot(M,1,m+1) myplt.errorplot(fh[m], x=np.arange(N), error=err_fh[m]) plt.plot(np.arange(N), f[m], 'g') plt.plot(np.arange(N), y[m], 'r+') plt.show() if __name__ == '__main__': run() bayespy-0.1/bayespy/demos/demo_rotation.py0000664000175000017500000000755512174013072022557 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011-2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import time import numpy as np import scipy import matplotlib.pyplot as plt from bayespy.plot import plotting as myplt from bayespy import utils from bayespy.inference.vmp import nodes from bayespy.inference.vmp.vmp import VB from bayespy.inference.vmp import transformations from bayespy.inference.vmp.nodes.gamma import diagonal """ Demonstrate the effect of speed-up transformations for PCA model. """ def pca_model(M, N, D): # Construct the PCA model with ARD # ARD alpha = nodes.Gamma(1e-2, 1e-2, plates=(D,), name='alpha') # Loadings W = nodes.Gaussian(np.zeros(D), diagonal(alpha), name="W", plates=(M,1)) # States X = nodes.Gaussian(np.zeros(D), np.identity(D), name="X", plates=(1,N)) # PCA WX = nodes.Dot(W, X, name="WX") # Noise tau = nodes.Gamma(1e-2, 1e-2, name="tau") # Noisy observations Y = nodes.Normal(WX, tau, name="Y") return (Y, WX, W, X, tau, alpha) def run(M=50, N=200, D_y=10, D=20, maxiter=100): seed = 45 print('seed =', seed) np.random.seed(seed) # Generate data (covariance eigenvalues: 1,1,...,1,2^2,3^2,...,(D_y+1)^2 (q,r) = scipy.linalg.qr(np.random.randn(M,M)) C = np.diag(np.arange(2,2+D_y)) C = np.ones(M) C[:D_y] += np.arange(1,1+D_y) y = C[:,np.newaxis] * np.random.randn(M,N) y = np.dot(q, y) # Construct model (Y, WX, W, X, tau, alpha) = pca_model(M, N, D) # Data with missing values mask = utils.random.mask(M, N, p=0.9) # randomly missing mask[:,20:40] = False # gap missing y[~mask] = np.nan Y.observe(y, mask=mask) # Construct inference machine Q = VB(Y, W, X, tau, alpha, autosave_filename=utils.utils.tempfile()) # Initialize nodes (from prior and randomly) X.initialize_from_value(X.random()) W.initialize_from_value(W.random()) Q.update(repeat=1) Q.save() # # Run inference with rotations. # rotX = transformations.RotateGaussian(X) rotW = transformations.RotateGaussianARD(W, alpha) R = transformations.RotationOptimizer(rotX, rotW, D) for ind in range(maxiter): Q.update() R.rotate(check_gradient=False, maxiter=10, verbose=False, check_bound=Q.compute_lowerbound, check_bound_terms=Q.compute_lowerbound_terms) L_rot = Q.L # # Re-run inference without rotations. # Q.load() Q.update(repeat=maxiter) L_norot = Q.L # # Plot comparison # plt.plot(L_rot) plt.plot(L_norot) plt.legend(['With rotations', 'Without rotations'], loc='lower right') plt.show() if __name__ == '__main__': run() bayespy-0.1/bayespy/demos/demo_gp.py0000664000175000017500000000641612174013072021321 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np #import scipy as sp import matplotlib.pyplot as plt import time import bayespy.plot.plotting as myplt from bayespy.utils import utils # Profiling stuff #import profile #import cProfile #import pstats import imp import Nodes.ExponentialFamily as EF import Nodes.CovarianceFunctions as CF import Nodes.GaussianProcesses as GP imp.reload(myplt) imp.reload(utils) imp.reload(EF) imp.reload(CF) imp.reload(GP) # MULTIVARIATE GP!! def run(): ## Generate data # Noisy observations from a sinusoid N = 100 func = lambda x: np.sin(x*2*np.pi/50) x = np.random.uniform(low=0, high=N, size=(N,)) f = func(x) y = f + np.random.normal(0, 0.2, np.shape(f)) # Plot data plt.clf() plt.plot(x,y,'r+') ## Construct model # Covariance function stuff ls = EF.NodeConstantScalar(10, name='lengthscale') amp = EF.NodeConstantScalar(1.0, name='amplitude') noise = EF.NodeConstantScalar(0.2, name='noise') # Latent process covariance #K_f = CF.SquaredExponential(amp, ls) K_f = CF.SquaredExponential(amp, ls) # Noise process covariance K_noise = CF.Delta(noise) # Observation process covariance K_y = CF.Sum(K_f, K_noise) # Joint covariance #K_joint = CF.Multiple([[K_f, K_f],[K_f,K_y]], sparse=True) # Mean function stuff M = GP.Constant(lambda x: np.zeros(np.shape(x)[0])) # Means for latent and observation processes #M_multi = GP.Multiple([M, M]) # Gaussian process F = GP.GaussianProcess(M, [[K_f, K_f], [K_f, K_y]]) #F = GP.GaussianProcess(M, [[K_f, K_f], [K_f, K_y]]) #F = GP.GaussianProcess(M_multi, K_joint) ## Inference F.observe([[],x], y) utils.vb_optimize_nodes(ls, amp, noise) F.update() u = F.get_parameters() ## Show results # Print hyperparameters #print('parameters') #print(ls.name, ls.u[0]) #print(amp.name, amp.u[0]) #print(noise.name, noise.u[0]) # Posterior predictions xh = np.arange(np.min(x)-5, np.max(x)+100, 0.1) (fh, varfh) = u([[],xh], covariance=1) #(fh, varfh) = u([xh,[]], covariance=1) # Plot predictive distribution varfh[varfh<0] = 0 errfh = np.sqrt(varfh) myplt.errorplot(fh, x=xh, error=errfh) return if __name__ == '__main__': run() bayespy-0.1/bayespy/demos/demo_gaussian_markov_chain.py0000664000175000017500000000612712174013072025245 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np import matplotlib.pyplot as plt from bayespy.inference.vmp.nodes.gaussian_markov_chain import GaussianMarkovChain #from bayespy.inference.vmp.nodes.gaussian_markov_chain import MarkovChainToGaussian from bayespy.inference.vmp.nodes.gaussian import Gaussian from bayespy.utils import utils import bayespy.plot.plotting as bpplt import imp imp.reload(utils) imp.reload(bpplt) def run(): # Create some data N = 500 D = 2 # Initial state x0 = np.array([0.5, -0.5]) # Dynamics (time varying) A0 = np.array([[.9, -.4], [.4, .9]]) A1 = np.array([[.98, -.1], [.1, .98]]) l = np.linspace(0, 1, N-1).reshape((-1,1,1)) A = (1-l)*A0 + l*A1 # Innovation covariance matrix V = np.identity(D) # Observation noise covariance matrix C = np.tile(np.identity(D), (N, 1, 1)) ## C0 = 10*np.array([[1, 0], [0, 1]]) ## C1 = 0.01*np.array([[1, 0], [0, 1]]) ## C = (1-l)**2*C0 + l**2*C1 X = np.empty((N,D)) Y = np.empty((N,D)) # Simulate data x = x0 X[0,:] = x Y[0,:] = x + np.random.multivariate_normal(np.zeros(D), C[0,:,:]) for n in range(N-1): x = np.dot(A[n,:,:],x) + np.random.multivariate_normal(np.zeros(D), V) X[n+1,:] = x Y[n+1,:] = x + np.random.multivariate_normal(np.zeros(D), C[n+1,:,:]) # Invert observation noise covariance to observation precision matrices U = np.empty((N,D,D)) UY = np.empty((N,D)) for n in range(N): U[n,:,:] = np.linalg.inv(C[n,:,:]) UY[n,:] = np.linalg.solve(C[n,:,:], Y[n,:]) # Construct VB model Xh = GaussianMarkovChain(np.zeros(D), np.identity(D), A, np.ones(D), n=N) Yh = Gaussian(Xh.as_gaussian(), np.identity(D), plates=(N,)) Yh.observe(Y) Xh.update() xh = Xh.u[0] varxh = utils.diagonal(Xh.u[1]) - xh**2 #err = 2 * np.sqrt(varxh) plt.figure(1) plt.clf() for d in range(D): plt.subplot(D,1,d) bpplt.errorplot(xh[:,d], error=2*np.sqrt(varxh[:,d])) plt.plot(X[:,d], 'r-') plt.plot(Y[:,d], '.') if __name__ == '__main__': run() plt.show() bayespy-0.1/bayespy/demos/demo_kalman_filter.py0000664000175000017500000000530412174013072023516 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np import matplotlib.pyplot as plt from bayespy.utils import utils import bayespy.plot.plotting as bpplt import imp imp.reload(utils) imp.reload(bpplt) def run(): # Create some data N = 500 D = 2 # Initial state x0 = np.array([0.5, -0.5]) # Dynamics (time varying) A0 = np.array([[.9, -.4], [.4, .9]]) A1 = np.array([[.98, -.1], [.1, .98]]) l = np.linspace(0, 1, N).reshape((-1,1,1)) A = (1-l)*A0 + l*A1 # Innovation covariance matrix V = np.array([[1, 0], [0, 1]]) # Observation noise covariance matrix C0 = 10*np.array([[1, 0], [0, 1]]) C1 = 0.01*np.array([[1, 0], [0, 1]]) C = (1-l)**2*C0 + l**2*C1 X = np.empty((N,D)) Y = np.empty((N,D)) # Simulate data x = x0 for n in range(N): x = np.dot(A[n,:,:],x) + np.random.multivariate_normal(np.zeros(D), V) X[n,:] = x Y[n,:] = x + np.random.multivariate_normal(np.zeros(D), C[n,:,:]) U = np.empty((N,D,D)) UY = np.empty((N,D)) for n in range(N): U[n,:,:] = np.linalg.inv(C[n,:,:]) UY[n,:] = np.linalg.solve(C[n,:,:], Y[n,:]) # Create iterators for the static matrices #U = N*(U,) #A = N*(A,) V = N*(V,) (Xh, CovXh) = utils.kalman_filter(UY, U, A, V, np.zeros(2), 10*np.identity(2)) (Xh, CovXh) = utils.rts_smoother(Xh, CovXh, A, V) plt.clf() for d in range(D): plt.subplot(D,1,d) #plt.plot(Xh[:,d], 'r--') bpplt.errorplot(Xh[:,d], error=2*np.sqrt(CovXh[:,d,d])) plt.plot(X[:,d], 'r-') plt.plot(Y[:,d], '.') plt.show() if __name__ == '__main__': # FOR INTERACTIVE SESSIONS, NON-BLOCKING PLOTTING: #plt.ion() run() bayespy-0.1/bayespy/demos/demo_gp_sparse.py0000664000175000017500000000635112174013072022674 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np #import scipy as sp import matplotlib.pyplot as plt import plotting as myplt import time # Profiling stuff #import profile #import cProfile #import pstats import imp import utils import Nodes.ExponentialFamily as EF import Nodes.CovarianceFunctions as CF import Nodes.GaussianProcesses as GP imp.reload(myplt) imp.reload(utils) imp.reload(EF) imp.reload(CF) imp.reload(GP) # MULTIVARIATE GP!! def run(): ## Generate data # Noisy observations from a sinusoid N = 10000 func = lambda x: np.sin(x*2*np.pi/50) x = np.random.uniform(low=0, high=N, size=(N,)) f = func(x) y = f + np.random.normal(0, 0.2, np.shape(f)) # Plot data plt.clf() plt.plot(x,y,'r+') ## Construct model # Covariance function stuff ls = EF.NodeConstantScalar(3, name='lengthscale') amp = EF.NodeConstantScalar(2.0, name='amplitude') noise = EF.NodeConstantScalar(0.6, name='noise') # Latent process covariance #K_f = CF.SquaredExponential(amp, ls) K_f = CF.PiecewisePolynomial2(amp, ls) # Noise process covariance K_noise = CF.Delta(noise) # Observation process covariance K_y = CF.Sum(K_f, K_noise) # Joint covariance #K_joint = CF.Multiple([[K_f, K_f],[K_f,K_y]], sparse=True) # Mean function stuff M = GP.Constant(lambda x: (x/10-2)*(x/10+1)) # Means for latent and observation processes #M_multi = GP.Multiple([M, M]) # Gaussian process F = GP.GaussianProcess(M, [[K_f, K_f], [K_f, K_y]]) #F = GP.GaussianProcess(M, [[K_f, K_f], [K_f, K_y]]) #F = GP.GaussianProcess(M_multi, K_joint) ## Inference F.observe([[],x], y) utils.vb_optimize_nodes(ls, amp, noise) F.update() u = F.get_parameters() ## Show results # Print hyperparameters print('parameters') print(ls.name, ls.u[0]) print(amp.name, amp.u[0]) print(noise.name, noise.u[0]) # Posterior predictions xh = np.arange(np.min(x)-5, np.max(x)+10, 0.1) (fh, varfh) = u([[],xh], covariance=1) #(fh, varfh) = u([xh,[]], covariance=1) # Plot predictive distribution varfh[varfh<0] = 0 errfh = np.sqrt(varfh) myplt.errorplot(xh, fh, errfh, errfh) return if __name__ == '__main__': run() bayespy-0.1/bayespy/demos/demo_categorical.py0000664000175000017500000000447612174013072023174 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np import matplotlib.pyplot as plt import time from bayespy.utils import utils from bayespy.plot import plotting as myplt import bayespy.inference.vmp.nodes as EF import imp imp.reload(utils) imp.reload(myplt) imp.reload(EF) def categorical_model(M, D): p = EF.Dirichlet(1*np.ones(D), name='p') z = EF.Categorical(p, plates=(M,), name='z') return (z, p) def run(M=30, D=5): # Generate data y = np.random.randint(D, size=(M,)) # Construct model (z, p) = categorical_model(M, D) # Initialize nodes p.update() z.update() # Observe the data with randomly missing values mask = np.random.rand(M) < 0.5 # randomly missing z.observe(y, mask) # Inference loop. L_last = -np.inf for i in range(10): t = time.clock() # Update nodes p.update() z.update() # Compute lower bound L_p = p.lower_bound_contribution() L_z = z.lower_bound_contribution() L = L_p + L_z # Check convergence print("Iteration %d: loglike=%e (%.3f seconds)" % (i+1, L, time.clock()-t)) if L_last > L: L_diff = (L_last - L) if L - L_last < 1e-12: print("Converged.") L_last = L z.show() p.show() if __name__ == '__main__': run() bayespy-0.1/bayespy/demos/demo_gpfa.py0000664000175000017500000000767512174013072021640 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np #import scipy as sp import matplotlib.pyplot as plt import plotting as myplt import time # Profiling stuff #import profile #import cProfile #import pstats import imp import utils import Nodes.ExponentialFamily as EF import Nodes.CovarianceFunctions as CF import Nodes.GaussianProcesses as GP imp.reload(myplt) imp.reload(utils) imp.reload(EF) imp.reload(CF) imp.reload(GP) # MULTIVARIATE GP!! def gpfa_model(x_a, x_s, n_a, n_s, D): # This? for i in range(D): amp[i] = EF.Delta(name='amplitude-'+str(i)) ls[i] = EF.Delta(name='lengthscale-'+str(i)) cf[i] = CF.PiecewisePolynomial2(amp[i], ls[i]) cf_a = CF.Multiple(cf) a = GP.GaussianProcess(0, cf_a, name='a') indices = np.arange(D*n_a).reshape((n_a,D)) # maybe some better way to express this? x_A = [x_a] * D A = GP.ProcessToVector(a, x_A, indices) # This? amp = EF.Delta(name='amplitude', plates=(D,)) ls = EF.Delta(name='lengthscale', plates=(D,)) cf = CF.PiecewisePolynomial2(amp, ls) cf_a = CF.Multiple(cf) a = GP.GaussianProcess(0, cf_a, name='a') A = GP.ProcessToVector(a, x_a) def run(): ## Generate data # Noisy observations from a sinusoid N = 100 #func = lambda x: np.sin(x*2*np.pi/50) x = np.random.uniform(low=0, high=N, size=(N,)) f = func(x) y = f + np.random.normal(0, 0.2, np.shape(f)) # Plot data plt.clf() plt.plot(x,y,'r+') ## Construct model # Covariance function stuff ls = EF.NodeConstantScalar(3, name='lengthscale') amp = EF.NodeConstantScalar(2.0, name='amplitude') noise = EF.NodeConstantScalar(0.6, name='noise') # Latent process covariance #K_f = CF.SquaredExponential(amp, ls) K_f = CF.PiecewisePolynomial2(amp, ls) # Noise process covariance K_noise = CF.Delta(noise) # Observation process covariance K_y = CF.Sum(K_f, K_noise) # Joint covariance #K_joint = CF.Multiple([[K_f, K_f],[K_f,K_y]], sparse=True) # Mean function stuff M = GP.Constant(lambda x: (x/10-2)*(x/10+1)) # Means for latent and observation processes #M_multi = GP.Multiple([M, M]) # Gaussian process F = GP.GaussianProcess(M, [[K_f, K_f], [K_f, K_y]]) #F = GP.GaussianProcess(M, [[K_f, K_f], [K_f, K_y]]) #F = GP.GaussianProcess(M_multi, K_joint) ## Inference F.observe([[],x], y) utils.vb_optimize_nodes(ls, amp, noise) F.update() u = F.get_parameters() ## Show results # Print hyperparameters print('parameters') print(ls.name, ls.u[0]) print(amp.name, amp.u[0]) print(noise.name, noise.u[0]) # Posterior predictions xh = np.arange(np.min(x)-5, np.max(x)+10, 0.1) (fh, varfh) = u([[],xh], covariance=1) #(fh, varfh) = u([xh,[]], covariance=1) # Plot predictive distribution varfh[varfh<0] = 0 errfh = np.sqrt(varfh) myplt.errorplot(xh, fh, errfh, errfh) return if __name__ == '__main__': run() bayespy-0.1/bayespy/demos/demo_lssm_drift.py0000664000175000017500000002455712174013072023067 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### """ Demonstrate linear Gaussian state-space model with drifting dynamics. """ import numpy as np import scipy import matplotlib.pyplot as plt from bayespy.inference.vmp.nodes.gaussian_markov_chain import GaussianMarkovChain from bayespy.inference.vmp.nodes.gaussian import Gaussian from bayespy.inference.vmp.nodes.gamma import Gamma, diagonal from bayespy.inference.vmp.nodes.normal import Normal from bayespy.inference.vmp.nodes.dot import Dot, MatrixDot from bayespy.inference.vmp.nodes.deterministic import tile from bayespy.utils import utils from bayespy.utils import random from bayespy.inference.vmp.vmp import VB from bayespy.inference.vmp import transformations import bayespy.plot.plotting as bpplt def simulate_static_lssm(M, N): # Simulate some data D = 3 c = np.random.randn(M,D) w = 0.3 a = np.array([[np.cos(w), -np.sin(w), 0], [np.sin(w), np.cos(w), 0], [0, 0, 1]]) x = np.empty((N,D)) f = np.empty((M,N)) y = np.empty((M,N)) x[0] = 10*np.random.randn(D) f[:,0] = np.dot(c,x[0]) y[:,0] = f[:,0] + 3*np.random.randn(M) for n in range(N-1): x[n+1] = np.dot(a,x[n]) + np.random.randn(D) f[:,n+1] = np.dot(c,x[n+1]) y[:,n+1] = f[:,n+1] + 3*np.random.randn(M) return (y, f) def simulate_drifting_lssm(M, N): # Simulate some data D = 3 c = np.random.randn(M,D) a = np.empty((N-1,D,D)) n = 0 for l in np.linspace(5, 1, num=N-1): w = 1/l a[n] = np.array([[np.cos(w), -np.sin(w), 0], [np.sin(w), np.cos(w), 0], [0, 0, 1]]) n = n + 1 x = np.empty((N,D)) f = np.empty((M,N)) y = np.empty((M,N)) x[0] = 10*np.random.randn(D) f[:,0] = np.dot(c,x[0]) y[:,0] = f[:,0] + 3*np.random.randn(M) for n in range(N-1): x[n+1] = np.dot(a[n],x[n]) + np.random.randn(D) f[:,n+1] = np.dot(c,x[n+1]) y[:,n+1] = f[:,n+1] + 3*np.random.randn(M) return (y, f) def run_dlssm(y, f, mask, D, K, maxiter): """ Run VB inference for linear state space model with drifting dynamics. """ (M, N) = np.shape(y) # Dynamics matrix with ARD # alpha : (D) x () alpha = Gamma(1e-5, 1e-5, plates=(K,), name='alpha') # A : (K) x (K) A = Gaussian(np.zeros(K), #np.identity(K), diagonal(alpha), plates=(K,), name='A_S') A.initialize_from_value(np.identity(K)) # rho ## rho = Gamma(1e-5, ## 1e-5, ## plates=(K,), ## name="rho") # S : () x (N-1,K) S = GaussianMarkovChain(np.ones(K), 1e-6*np.identity(K), A, np.ones(K), n=N-1, name='S') S.initialize_from_value(1*np.ones((N-1,K))) # Projection matrix of the dynamics matrix # beta : (K) x () beta = Gamma(1e-5, 1e-5, plates=(K,), name='beta') # B : (D) x (D*K) B = Gaussian(np.zeros(D*K), diagonal(tile(beta, D)), plates=(D,), name='B') b = np.zeros((D,D,K)) b[np.arange(D),np.arange(D),np.zeros(D,dtype=int)] = 1 B.initialize_from_value(np.reshape(1*b, (D,D*K))) # A : (N-1,D) x (D) BS = MatrixDot(B, S.as_gaussian().add_plate_axis(-1), name='BS') # Latent states with dynamics # X : () x (N,D) X = GaussianMarkovChain(np.zeros(D), # mean of x0 1e-3*np.identity(D), # prec of x0 BS, # dynamics np.ones(D), # innovation n=N, # time instances name='X', initialize=False) X.initialize_from_value(np.random.randn(N,D)) # Mixing matrix from latent space to observation space using ARD # gamma : (D) x () gamma = Gamma(1e-5, 1e-5, plates=(D,), name='gamma') # C : (M,1) x (D) C = Gaussian(np.zeros(D), diagonal(gamma), plates=(M,1), name='C') C.initialize_from_value(np.random.randn(M,1,D)) # Observation noise # tau : () x () tau = Gamma(1e-5, 1e-5, name='tau') # Observations # Y : (M,N) x () CX = Dot(C, X.as_gaussian()) Y = Normal(CX, tau, name='Y') # # RUN INFERENCE # # Observe data Y.observe(y, mask=mask) # Construct inference machine Q = VB(Y, X, S, A, alpha, B, beta, C, gamma, tau) # # Run inference with rotations. # # Rotate the D-dimensional state space (C, X) rotB = transformations.RotateGaussianMatrixARD(B, beta, axis='rows') rotX = transformations.RotateDriftingMarkovChain(X, B, S, rotB) rotC = transformations.RotateGaussianARD(C, gamma) R_X = transformations.RotationOptimizer(rotX, rotC, D) # Rotate the K-dimensional latent dynamics space (B, S) rotA = transformations.RotateGaussianARD(A, alpha) rotS = transformations.RotateGaussianMarkovChain(S, A, rotA) rotB = transformations.RotateGaussianMatrixARD(B, beta, axis='cols') R_S = transformations.RotationOptimizer(rotS, rotB, K) # Iterate for ind in range(int(maxiter/5)): Q.update(repeat=5) #Q.update(X, S, A, alpha, rho, B, beta, C, gamma, tau, repeat=maxiter) R_X.rotate() R_S.rotate() ## R_X.rotate( ## check_bound=Q.compute_lowerbound, ## check_bound_terms=Q.compute_lowerbound_terms, ## check_gradient=True ## ) ## R_S.rotate( ## check_bound=Q.compute_lowerbound, ## check_bound_terms=Q.compute_lowerbound_terms, ## check_gradient=True ## ) # # SHOW RESULTS # # Mean and standard deviation of the posterior (f_mean, f_squared) = CX.get_moments() f_std = np.sqrt(f_squared - f_mean**2) # Plot observations space for m in range(M): plt.subplot(M,1,m+1) plt.plot(y[m,:], 'r.') plt.plot(f[m,:], 'b-') bpplt.errorplot(y=f_mean[m,:], error=2*f_std[m,:]) def run_lssm(y, f, mask, D, maxiter): """ Run VB inference for linear state space model. """ (M, N) = np.shape(y) # # CONSTRUCT THE MODEL # # Dynamic matrix # alpha: (D) x () alpha = Gamma(1e-5, 1e-5, plates=(D,), name='alpha') # A : (D) x (D) A = Gaussian(np.zeros(D), diagonal(alpha), plates=(D,), name='A') A.initialize_from_value(np.identity(D)) # Latent states with dynamics # X : () x (N,D) X = GaussianMarkovChain(np.zeros(D), # mean of x0 1e-3*np.identity(D), # prec of x0 A, # dynamics np.ones(D), # innovation n=N, # time instances name='X', initialize=False) X.initialize_from_value(np.random.randn(N,D)) # Mixing matrix from latent space to observation space using ARD # gamma : (D) x () gamma = Gamma(1e-5, 1e-5, plates=(D,), name='gamma') # C : (M,1) x (D) C = Gaussian(np.zeros(D), diagonal(gamma), plates=(M,1), name='C') C.initialize_from_value(np.random.randn(M,1,D)) # Observation noise # tau : () x () tau = Gamma(1e-5, 1e-5, name='tau') # Observations # Y : (M,N) x () CX = Dot(C, X.as_gaussian()) Y = Normal(CX, tau, name='Y') # # RUN INFERENCE # # Observe data Y.observe(y, mask=mask) # Construct inference machine Q = VB(Y, X, A, alpha, C, gamma, tau) # Iterate Q.update(X, A, alpha, C, gamma, tau, repeat=maxiter) # # SHOW RESULTS # # Mean and standard deviation of the posterior (f_mean, f_squared) = CX.get_moments() f_std = np.sqrt(f_squared - f_mean**2) # Plot observations space #plt.figure() for m in range(M): plt.subplot(M,1,m+1) plt.plot(y[m,:], 'r.') plt.plot(f[m,:], 'b-') bpplt.errorplot(y=f_mean[m,:], error=2*f_std[m,:]) def run(method): # Seed for random number generator seed = 495 np.random.seed(seed) print("seed = ", seed) # Create data M = 10 N = 200 (y, f) = simulate_drifting_lssm(M, N) # Add missing values randomly mask = random.mask(M, N, p=0.3) # Add missing values to a period of time mask[:,70:120] = False #mask[:] = True y[~mask] = np.nan # BayesPy doesn't require NaNs, they're just for plotting. # Run the method if method == 'lssm': run_lssm(y, f, mask, 3, 200) elif method == 'dlssm': run_dlssm(y, f, mask, 3, 2, 200) else: raise Exception("Unknown method requested") if __name__ == '__main__': run('dlssm') plt.show() bayespy-0.1/bayespy/demos/demo_saving.py0000664000175000017500000001123712174013072022177 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011-2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np import matplotlib.pyplot as plt import h5py import tempfile from bayespy.plot import plotting as myplt from bayespy import utils from bayespy.inference.vmp import nodes from bayespy.inference.vmp.vmp import VB def pca_model(M, N, D): # Construct the PCA model with ARD # ARD alpha = nodes.Gamma(1e-2, 1e-2, plates=(D,), name='alpha') # Loadings W = nodes.Gaussian(np.zeros(D), alpha.as_diagonal_wishart(), name="W", plates=(M,1)) # States X = nodes.Gaussian(np.zeros(D), np.identity(D), name="X", plates=(1,N)) # PCA WX = nodes.Dot(W, X, name="WX") # Noise tau = nodes.Gamma(1e-2, 1e-2, name="tau", plates=()) # Noisy observations Y = nodes.Normal(WX, tau, name="Y", plates=(M,N)) return (Y, WX, W, X, tau, alpha) def run(M=10, N=100, D_y=3, D=5): seed = 45 print('seed =', seed) np.random.seed(seed) # Check HDF5 version. if h5py.version.hdf5_version_tuple < (1,8,7): print("WARNING! Your HDF5 version is %s. HDF5 versions <1.8.7 are not " "able to save empty arrays, thus you may experience problems if " "you for instance try to save before running any iteration steps." % str(h5py.version.hdf5_version_tuple)) # Generate data w = np.random.normal(0, 1, size=(M,1,D_y)) x = np.random.normal(0, 1, size=(1,N,D_y)) f = utils.utils.sum_product(w, x, axes_to_sum=[-1]) y = f + np.random.normal(0, 0.5, size=(M,N)) # Construct model (Y, WX, W, X, tau, alpha) = pca_model(M, N, D) # Data with missing values mask = utils.random.mask(M, N, p=0.9) # randomly missing mask[:,20:40] = False # gap missing y[~mask] = np.nan Y.observe(y, mask=mask) # Construct inference machine Q = VB(Y, W, X, tau, alpha, autosave_iterations=5) # Initialize some nodes randomly X.initialize_from_value(X.random()) W.initialize_from_value(W.random()) # Save the state into a HDF5 file filename = tempfile.NamedTemporaryFile(suffix='hdf5').name Q.update(X, W, alpha, tau, repeat=1) Q.save(filename) # Inference loop. Q.update(X, W, alpha, tau, repeat=10) # Reload the state from the HDF5 file Q.load(filename=filename) # Inference loop again. Q.update(X, W, alpha, tau, repeat=10) # NOTE: Saving and loading requires that you have the model # constructed. "Save" does not store the model structure nor does "load" # read it. They are just used for reading and writing the contents of the # nodes. Thus, if you want to load, you first need to construct the same # model that was used for saving and then use load to set the states of the # nodes. plt.clf() WX_params = WX.get_parameters() fh = WX_params[0] * np.ones(y.shape) err_fh = 2*np.sqrt(WX_params[1] + 1/tau.get_moments()[0]) * np.ones(y.shape) for m in range(M): plt.subplot(M,1,m+1) #errorplot(y, error=None, x=None, lower=None, upper=None): myplt.errorplot(fh[m], x=np.arange(N), error=err_fh[m]) plt.plot(np.arange(N), f[m], 'g') plt.plot(np.arange(N), y[m], 'r+') plt.figure() Q.plot_iteration_by_nodes() plt.figure() plt.subplot(2,2,1) myplt.binary_matrix(W.mask) plt.subplot(2,2,2) myplt.binary_matrix(X.mask) plt.subplot(2,2,3) #myplt.binary_matrix(WX.get_mask()) plt.subplot(2,2,4) myplt.binary_matrix(Y.mask) plt.show() if __name__ == '__main__': run() bayespy-0.1/bayespy/demos/__init__.py0000664000175000017500000000000012174013072021425 0ustar jluttinejluttine00000000000000bayespy-0.1/bayespy/demos/demo_lssm.py0000664000175000017500000001245412174013072021670 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### """ Demonstrate linear Gaussian state-space model. """ import numpy as np import scipy import matplotlib.pyplot as plt from bayespy.inference.vmp.nodes.gaussian_markov_chain import GaussianMarkovChain from bayespy.inference.vmp.nodes.gaussian import Gaussian from bayespy.inference.vmp.nodes.gamma import Gamma from bayespy.inference.vmp.nodes.normal import Normal from bayespy.inference.vmp.nodes.dot import Dot from bayespy.inference.vmp.nodes.gamma import diagonal from bayespy.utils import utils from bayespy.utils import random from bayespy.inference.vmp.vmp import VB from bayespy.inference.vmp import transformations import bayespy.plot.plotting as bpplt def linear_state_space_model(D=3, N=100, M=10): # Dynamics matrix with ARD alpha = Gamma(1e-5, 1e-5, plates=(D,), name='alpha') A = Gaussian(np.zeros(D), diagonal(alpha), plates=(D,), name='A') # Latent states with dynamics X = GaussianMarkovChain(np.zeros(D), # mean of x0 1e-3*np.identity(D), # prec of x0 A, # dynamics np.ones(D), # innovation n=N, # time instances name='X') # Mixing matrix from latent space to observation space using ARD gamma = Gamma(1e-5, 1e-5, plates=(D,), name='gamma') C = Gaussian(np.zeros(D), diagonal(gamma), plates=(M,1), name='C') # Observation noise tau = Gamma(1e-5, 1e-5, name='tau') # Observations CX = Dot(C, X.as_gaussian()) Y = Normal(CX, tau, name='Y') return (Y, CX, X, tau, C, gamma, A, alpha) def run(maxiter=100): seed = 496#np.random.randint(1000) print("seed = ", seed) np.random.seed(seed) # Simulate some data D = 3 M = 6 N = 200 c = np.random.randn(M,D) w = 0.3 a = np.array([[np.cos(w), -np.sin(w), 0], [np.sin(w), np.cos(w), 0], [0, 0, 1]]) x = np.empty((N,D)) f = np.empty((M,N)) y = np.empty((M,N)) x[0] = 10*np.random.randn(D) f[:,0] = np.dot(c,x[0]) y[:,0] = f[:,0] + 3*np.random.randn(M) for n in range(N-1): x[n+1] = np.dot(a,x[n]) + np.random.randn(D) f[:,n+1] = np.dot(c,x[n+1]) y[:,n+1] = f[:,n+1] + 3*np.random.randn(M) # Create the model (Y, CX, X, tau, C, gamma, A, alpha) = linear_state_space_model(D, N, M) # Add missing values randomly mask = random.mask(M, N, p=0.3) # Add missing values to a period of time mask[:,30:80] = False y[~mask] = np.nan # BayesPy doesn't require this. Just for plotting. # Observe the data Y.observe(y, mask=mask) # Initialize nodes (must use some randomness for C) C.initialize_from_random() # Run inference Q = VB(Y, X, C, gamma, A, alpha, tau) # # Run inference with rotations. # rotA = transformations.RotateGaussianARD(A, alpha) rotX = transformations.RotateGaussianMarkovChain(X, A, rotA) rotC = transformations.RotateGaussianARD(C, gamma) R = transformations.RotationOptimizer(rotX, rotC, D) #maxiter = 84 for ind in range(maxiter): Q.update() #print('C term', C.lower_bound_contribution()) R.rotate(maxiter=10, check_gradient=True, verbose=False, check_bound=Q.compute_lowerbound, #check_bound=None, check_bound_terms=Q.compute_lowerbound_terms) #check_bound_terms=None) X_vb = X.u[0] varX_vb = utils.diagonal(X.u[1] - X_vb[...,np.newaxis,:] * X_vb[...,:,np.newaxis]) u_CX = CX.get_moments() CX_vb = u_CX[0] varCX_vb = u_CX[1] - CX_vb**2 # Show results plt.figure(3) plt.clf() for m in range(M): plt.subplot(M,1,m+1) plt.plot(y[m,:], 'r.') plt.plot(f[m,:], 'b-') bpplt.errorplot(y=CX_vb[m,:], error=2*np.sqrt(varCX_vb[m,:])) plt.figure() Q.plot_iteration_by_nodes() if __name__ == '__main__': run() plt.show() bayespy-0.1/bayespy/demos/demo_gaussian_mixture.py0000664000175000017500000001654612174013072024307 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### import numpy as np import matplotlib.pyplot as plt import time from bayespy.utils import utils import bayespy.plot.plotting as myplt from bayespy.inference.vmp import nodes #import nodes.exponential_family as EF ## import imp ## imp.reload(utils) ## imp.reload(myplt) ## imp.reload(EF) # Reload everything (helpful for interactive sessions) ## import imp ## import bayespy.inference.vmp.nodes ## import bayespy.inference.vmp.nodes.node ## import bayespy.inference.vmp.nodes.variable ## import bayespy.inference.vmp.nodes.wishart ## import bayespy.inference.vmp.nodes.gaussian ## import bayespy.inference.vmp.nodes.mixture ## import bayespy.inference.vmp.nodes.dirichlet ## import bayespy.inference.vmp.nodes.categorical ## imp.reload(utils) ## imp.reload(bayespy.inference.vmp.nodes.node) ## imp.reload(bayespy.inference.vmp.nodes.variable) ## imp.reload(bayespy.inference.vmp.nodes.wishart) ## imp.reload(bayespy.inference.vmp.nodes.gaussian) ## imp.reload(bayespy.inference.vmp.nodes.mixture) ## imp.reload(bayespy.inference.vmp.nodes.dirichlet) ## imp.reload(bayespy.inference.vmp.nodes.categorical) ## imp.reload(EF) def gaussianmix_model(N, K, D): # N = number of data vectors # K = number of clusters # D = dimensionality # Construct the Gaussian mixture model # K prior weights (for components) alpha = nodes.Dirichlet(1*np.ones(K), name='alpha') # N K-dimensional cluster assignments (for data) z = nodes.Categorical(alpha, plates=(N,), name='z') # K D-dimensional component means X = nodes.Gaussian(np.zeros(D), 0.01*np.identity(D), plates=(K,), name='X') # K D-dimensional component covariances Lambda = nodes.Wishart(D, 0.01*np.identity(D), plates=(K,), name='Lambda') # N D-dimensional observation vectors Y = nodes.Mixture(nodes.Gaussian)(z, X, Lambda, plates=(N,), name='Y') # TODO: Plates should be learned automatically if not given (it # would be the smallest shape broadcasted from the shapes of the # parents) return (Y, X, Lambda, z, alpha) def run(N=50, K=5, D=2): #plt.ion() #17,31 #np.random.seed(31) # Generate data N1 = np.floor(0.5*N) N2 = N - N1 y = np.vstack([np.random.normal(0, 0.5, size=(N1,D)), np.random.normal(10, 0.5, size=(N2,D))]) # Construct model (Y, X, Lambda, z, alpha) = gaussianmix_model(N,K,D) # Initialize nodes (from prior and randomly) alpha.initialize_from_prior() z.initialize_from_prior() Lambda.initialize_from_parameters(D, 10*np.identity(D)) X.initialize_from_prior() X.initialize_from_parameters(X.random(), np.identity(D)) ## X.initialize_from_parameters(np.random.permutation(y)[:K], ## 0.01*np.identity(D)) #X.initialize_from_random() # Initialize means by selecting random data points #X.initialize_from_value(np.random.permutation(y)[:K]) #return #X.initialize_random_mean() #Y.initialize() ## X.show() ## return # Data with missing values ## mask = np.random.rand(M,N) < 0.4 # randomly missing ## mask[:,20:40] = False # gap missing # Y.observe(y, mask) ## alpha.show() ## Lambda.show() ## z.show() X.show() #return Y.observe(y) ## z.update() ## X.update() ## alpha.update() ## X.show() ## alpha.show() #Lambda.show() #return ## X.show() ## Lambda.show() ## z.update() ## z.show() ## return # Inference loop. maxiter = 30 L_X = np.zeros(maxiter) L_Lambda = np.zeros(maxiter) L_alpha = np.zeros(maxiter) L_z = np.zeros(maxiter) L_Y = np.zeros(maxiter) L = np.zeros(maxiter) L_last = -np.inf for i in range(maxiter): t = time.clock() # Update nodes z.update() alpha.update() X.update() Lambda.update() #Y.show() #z.show() # Compute lower bound L_X[i] = X.lower_bound_contribution() L_Lambda[i] = Lambda.lower_bound_contribution() L_alpha[i] = alpha.lower_bound_contribution() L_z[i] = z.lower_bound_contribution() L_Y[i] = Y.lower_bound_contribution() L[i] = L_X[i] + L_Lambda[i] + L_alpha[i] + L_z[i] + L_Y[i] #print('terms:', L_X[i], L_Lambda[i], L_alpha[i], L_z[i], L_Y[i]) # Check convergence print("Iteration %d: loglike=%e (%.3f seconds)" % (i+1, L[i], time.clock()-t)) if L_last - L[i] > 1e-6: L_diff = (L_last - L[i]) print("Lower bound decreased %e! Bug somewhere or numerical inaccuracy?" % L_diff) #raise Exception("Lower bound decreased %e! Bug somewhere or numerical inaccuracy?" % L_diff) if L[i] - L_last < 1e-12: print("Converged.") #break L_last = L[i] # Predictive stuff zh = nodes.Categorical(alpha, name='zh') Yh = nodes.Mixture(nodes.Gaussian)(zh, X, Lambda, name='Yh') # TODO/FIXME: Messages to parents should use the masks such that # children don't need to be initialized! zh.initialize_from_prior() Yh.initialize_from_prior() zh.update() #zh.show() N1 = 400 N2 = 400 x1 = np.linspace(-3, 15, N1) x2 = np.linspace(-3, 15, N2) xh = utils.grid(x1, x2) lpdf = Yh.integrated_logpdf_from_parents(xh, 0) pdf = np.reshape(np.exp(lpdf), (N2,N1)) #print(pdf) #plt.clf() plt.clf() #plt.imshow(x1, x2, pdf) plt.contourf(x1, x2, pdf, 100) plt.scatter(y[:,0], y[:,1]) print('integrated pdf:', np.sum(pdf)*(18*18)/(N1*N2)) #return X.show() alpha.show() plt.show() #print(y) #print(Y.u[0]) ## plt.clf() ## f = np.vstack([L_X, L_Lambda, L_z, L_alpha, L_Y, L]).T ## #f = np.diff(f, axis=0) ## ax = plt.plot(f) ## plt.legend(ax) ## plt.figure() ## plt.clf() ## WX_params = WX.get_parameters() ## fh = WX_params[0] * np.ones(y.shape) ## err_fh = 2*np.sqrt(WX_params[1]) * np.ones(y.shape) ## for d in range(D): ## myplt.errorplot(np.arange(N), fh[d], err_fh[d], err_fh[d]) ## plt.plot(np.arange(N), f[d], 'g') ## plt.plot(np.arange(N), y[d], 'r+') if __name__ == '__main__': # FOR INTERACTIVE SESSIONS, NON-BLOCKING PLOTTING: #plt.ion() run() bayespy-0.1/bayespy/__init__.py0000664000175000017500000000245612174023005020334 0ustar jluttinejluttine00000000000000###################################################################### # Copyright (C) 2011-2013 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### from . import utils from . import inference # Currently, model construction and the inference network are not separated so # the model is constructed using variational message passing nodes. from .inference.vmp import nodes bayespy-0.1/setup.py0000664000175000017500000001140612174042774016272 0ustar jluttinejluttine00000000000000#!/usr/bin/env python ###################################################################### # Copyright (C) 2011,2012 Jaakko Luttinen # # This file is licensed under Version 3.0 of the GNU General Public # License. See LICENSE for a text of the license. ###################################################################### ###################################################################### # This file is part of BayesPy. # # BayesPy is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License version 3 as # published by the Free Software Foundation. # # BayesPy is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with BayesPy. If not, see . ###################################################################### LONG_DESCRIPTION = \ """ Bayesian inference tools for Python. The package provides tools for building models and performing posterior inference using variational Bayesian message passing. """ NAME = 'bayespy' DESCRIPTION = 'Bayesian inference tools for Python' AUTHOR = 'Jaakko Luttinen' AUTHOR_EMAIL = 'jaakko.luttinen@iki.fi' URL = 'http://bayespy.org' LICENSE = 'GPLv3' VERSION = '0.1' if __name__ == "__main__": from setuptools import setup, Extension, find_packages #from distutils.core import setup, Extension from Cython.Distutils import build_ext import numpy as np # Sparse distance extension. # Use numpy.get_include() in order to use the correct NumPy for building. # Remove Cython dependency for now, in order to keep the installation # simpler. # sparse_distance = Extension('bayespy.utils.covfunc.distance', sources=['bayespy/utils/covfunc/distance.pyx', 'bayespy/utils/covfunc/sparse_distance/sparse_distance.c'], include_dirs=['bayespy/utils/covfunc/sparse_distance', np.get_include()]) # Setup for BayesPy setup( install_requires = ['numpy>=1.7.1', # 1.7.0 contains a memory leak bug fixed in 1.7.1 'scipy>=0.11.0', #'scikits.sparse>=0.1', # required for sparse GPs only 'matplotlib>=1.2.0', 'cython', 'h5py'], ## requires = ['numpy (>=1.7.1)', # 1.7.0 contains a memory leak bug ## 'scipy (>=0.11.0)', ## 'scikits.sparse (>=0.1)', ## 'matplotlib (>=1.2.0)', ## 'cython', ## 'h5py'], ## dependency_links = [ ## 'https://github.com/numpy/numpy/archive/master.zip#egg=numpy-1.8.0', ## ], # These are for sparse_distance Cython extension cmdclass = {'build_ext': build_ext}, ext_modules = [sparse_distance], packages = find_packages(), ## ['bayespy', ## 'bayespy.demos', ## 'bayespy.inference', ## 'bayespy.inference.vmp', ## 'bayespy.inference.vmp.nodes', ## 'bayespy.inference.vmp.nodes.tests', ## 'bayespy.nodes', ## 'bayespy.plot', ## 'bayespy.utils', ## 'bayespy.utils.tests', ## 'bayespy.utils.covfunc', ## 'bayespy.utils.covfunc.tests'], name = NAME, version = VERSION, author = AUTHOR, author_email = AUTHOR_EMAIL, description = DESCRIPTION, license = LICENSE, url = URL, long_description = LONG_DESCRIPTION, classifiers = [ 'Programming Language :: Python', 'Programming Language :: Python :: 3', 'Development Status :: 2 - Pre-Alpha', 'Environment :: Console', 'Intended Audience :: Developers', 'Intended Audience :: Science/Research', 'License :: OSI Approved :: GNU General Public License v3 (GPLv3)', 'Operating System :: OS Independent', 'Topic :: Scientific/Engineering', 'Topic :: Scientific/Engineering :: Information Analysis' ] ) bayespy-0.1/MANIFEST.in0000664000175000017500000000041012174013072016274 0ustar jluttinejluttine00000000000000include README* include INSTALL* include LICENSE # Include header files recursive-include bayespy *.h # Add documentation to sdist-generated tarballs recursive-include doc/source *.rst recursive-include doc/source *.py include doc/Makefile #include doc/make.bat bayespy-0.1/README.rst0000664000175000017500000000513112174013072016232 0ustar jluttinejluttine00000000000000BayesPy - Bayesian Python ========================= BayesPy provides tools for Bayesian inference with Python. The user constructs a model as a Bayesian network, observes data and runs posterior inference. The goal is to provide a tool which is efficient, flexible and extendable enough for expert use but also accessible for more casual users. Currently, only variational Bayesian inference for conjugate-exponential family (variational message passing) has been implemented. Future work includes variational approximations for other types of distributions and possibly other approximate inference methods such as expectation propagation, Laplace approximations, Markov chain Monte Carlo (MCMC) and other methods. Contributions are welcome. Project information ------------------- Copyright (C) 2011-2013 Jaakko Luttinen, Aalto University BayesPy including the documentation is licensed under Version 3.0 of the GNU General Public License. See LICENSE file for a text of the license or visit http://www.gnu.org/copyleft/gpl.html. * Documentation: http://bayespy.org or at ``docs`` directory * Repository: https://github.com/bayespy/bayespy.git * Bug reports: https://github.com/bayespy/bayespy/issues * Mailing list: bayespy@googlegroups.com * IRC: #bayespy @ `freenode `_ * Author: Jaakko Luttinen jaakko.luttinen@iki.fi Similar projects ---------------- `VIBES `_ (http://vibes.sourceforge.net/) allows variational inference to be performed automatically on a Bayesian network. It is implemented in Java and released under revised BSD license. `Bayes Blocks `_ (http://research.ics.aalto.fi/bayes/software/) is a C++/Python implementation of the variational building block framework. The framework allows easy learning of a wide variety of models using variational Bayesian learning. It is available as free software under the GNU General Public License. `Infer.NET `_ (http://research.microsoft.com/infernet/) is a .NET framework for machine learning. It provides message-passing algorithms and statistical routines for performing Bayesian inference. It is partly closed source and licensed for non-commercial use only. `PyMC `_ (https://github.com/pymc-devs/pymc) provides MCMC methods in Python. It is released under the Academic Free License. `OpenBUGS `_ (http://www.openbugs.info) is a software package for performing Bayesian inference using Gibbs sampling. It is released under the GNU General Public License. bayespy-0.1/LICENSE0000664000175000017500000010451312174013072015554 0ustar jluttinejluttine00000000000000 GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007 Copyright (C) 2007 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The GNU General Public License is a free, copyleft license for software and other kinds of works. The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. We, the Free Software Foundation, use the GNU General Public License for most of our software; it applies also to any other work released this way by its authors. You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others. For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. Developers that use the GNU GPL protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License giving you legal permission to copy, distribute and/or modify it. For the developers' and authors' protection, the GPL clearly explains that there is no warranty for this free software. For both users' and authors' sake, the GPL requires that modified versions be marked as changed, so that their problems will not be attributed erroneously to authors of previous versions. Some devices are designed to deny users access to install or run modified versions of the software inside them, although the manufacturer can do so. This is fundamentally incompatible with the aim of protecting users' freedom to change the software. The systematic pattern of such abuse occurs in the area of products for individuals to use, which is precisely where it is most unacceptable. Therefore, we have designed this version of the GPL to prohibit the practice for those products. If such problems arise substantially in other domains, we stand ready to extend this provision to those domains in future versions of the GPL, as needed to protect the freedom of users. Finally, every program is threatened constantly by software patents. States should not allow patents to restrict development and use of software on general-purpose computers, but in those that do, we wish to avoid the special danger that patents applied to a free program could make it effectively proprietary. To prevent this, the GPL assures that patents cannot be used to render the program non-free. The precise terms and conditions for copying, distribution and modification follow. TERMS AND CONDITIONS 0. Definitions. "This License" refers to version 3 of the GNU General Public License. "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. A "covered work" means either the unmodified Program or a work based on the Program. To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 1. Source Code. The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. The Corresponding Source for a work in source code form is that same work. 2. Basic Permissions. All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 3. Protecting Users' Legal Rights From Anti-Circumvention Law. No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 4. Conveying Verbatim Copies. You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 5. Conveying Modified Source Versions. You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 6. Conveying Non-Source Forms. You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. "Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. 7. Additional Terms. "Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or d) Limiting the use for publicity purposes of names of licensors or authors of the material; or e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. 8. Termination. You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. 9. Acceptance Not Required for Having Copies. You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. 10. Automatic Licensing of Downstream Recipients. Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. 11. Patents. A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. 12. No Surrender of Others' Freedom. If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. 13. Use with the GNU Affero General Public License. Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU Affero General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such. 14. Revised Versions of this License. The Free Software Foundation may publish revised and/or new versions of the GNU General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU General Public License, you may choose any version ever published by the Free Software Foundation. If the Program specifies that a proxy can decide which future versions of the GNU General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. 15. Disclaimer of Warranty. THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. Limitation of Liability. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 17. Interpretation of Sections 15 and 16. If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . Also add information on how to contact you by electronic and paper mail. If the program does terminal interaction, make it output a short notice like this when it starts in an interactive mode: Copyright (C) This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, your program's commands might be different; for a GUI interface, you would use an "about box". You should also get your employer (if you work as a programmer) or school, if any, to sign a "copyright disclaimer" for the program, if necessary. For more information on this, and how to apply and follow the GNU GPL, see . The GNU General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. But first, please read . bayespy-0.1/INSTALL.rst0000664000175000017500000001106412174015434016411 0ustar jluttinejluttine00000000000000.. Copyright (C) 2011,2012 Jaakko Luttinen This file is licensed under Version 3.0 of the GNU General Public License. See LICENSE for a text of the license. This file is part of BayesPy. BayesPy is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License version 3 as published by the Free Software Foundation. BayesPy is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with BayesPy. If not, see . Installation ============ .. using `NumPy/SciPy `_ and `Matplotlib `_. BayesPy is a Python 3 package and it can be installed from PyPI or the latest development version from GitHub. The instructions below explain how to set up the system by installing required packages, how to install BayesPy and how to compile this documentation yourself. However, if these instructions contain errors or some relevant details are missing, please file a bug report at https://github.com/bayespy/bayespy/issues. Installing requirements ----------------------- BayesPy requires Python 3.2 and the following packages: * NumPy (>=1.7.1), * SciPy (>=0.11) * matplotlib (>=1.2) * Cython * h5py A proper installation of these packages can be a bit tricky and you may refer to http://www.scipy.org/install.html for more detailed instructions about the SciPy stack. If your system has an older version of any of the packages (NumPy, SciPy or matplotlib) or it does not provide the packages for Python 3.2, you may set up a virtual environment and install the latest versions there. To create and activate a new virtual environment, run .. code-block:: console virtualenv -p python3.2 --system-site-packages ENV source ENV/bin/activate If your system is properly set up, you may be able to install them from PyPI (a C compiler, Python development files, BLAS/LAPACK and other system files may be required). For instance, on Ubuntu (>= 12.10), you may install the dependencies for each package as: .. sudo aptitude install build-essential python3.2-dev libatlas-base-dev gfortran .. code-block:: console sudo apt-get build-dep python3-numpy sudo apt-get build-dep python3-scipy sudo apt-get build-dep python3-matplotlib sudo apt-get build-dep cython sudo apt-get build-dep python-h5py # sudo aptitude install python3-tk tk-dev (for matplotlib?) This guarantees that the required system libraries are installed. Then installation/upgrade from PyPI should work: .. code-block:: console pip install numpy --upgrade pip install scipy matplotlib --upgrade Note that this may take several minutes. You also need to install Cython and h5py, for instance, from PyPI: .. code-block:: console pip install cython h5py If you have problems installing any of these packages, refer to the manual of that package. Installing BayesPy ------------------ Before proceeding, make sure you have installed Cython, h5py and the latest versions of NumPy, Scipy and matplotlib for Python 3.2. After the system has been properly set up and the virtual environment is activated (if wanted), BayesPy can be installed from PyPI simply as .. code-block:: console pip install bayespy If you want to install the latest development version of BayesPy, use GitHub instead: .. code-block:: console pip install https://github.com/bayespy/bayespy/archive/master.zip If you have nose installed in the virtual environment, you can check that BayesPy is working: .. code-block:: console nosetests bayespy Compiling documentation ----------------------- This documentation can be found at http://bayespy.org/. The documentation source files are readable as such in reStructuredText format in ``doc/source/`` directory. It is possible to compile the documentation into HTML or PDF yourself. However, this is not currently recommended as it may turn out to be quite tricky because you need to have a Python 3.2 compatible modification of numpydoc and a few other packages (e.g., sphinxcontrib-tikz and sphinxcontrib-bayesnet). But if you have `Sphinx `_ and all the requirements installed, the documentation can be compiled to HTML and PDF by running the following commands in the ``doc`` folder: .. code-block:: console make html make latexpdf bayespy-0.1/setup.cfg0000664000175000017500000000021612174043062016365 0ustar jluttinejluttine00000000000000[egg_info] tag_build = tag_date = 0 tag_svn_revision = 0 [aliases] release = egg_info -RDb '' release_pypi = release register sdist upload