milk-0.5/ 0000775 0001750 0001750 00000000000 12045730202 014345 5 ustar luispedro luispedro 0000000 0000000 milk-0.5/MANIFEST.in 0000664 0001750 0001750 00000000126 12045730063 016107 0 ustar luispedro luispedro 0000000 0000000 include README.rst
include INSTALL.rst
include COPYING.MIT
include milk/tests/data/*
milk-0.5/README.rst 0000664 0001750 0001750 00000011763 12045730063 016051 0 ustar luispedro luispedro 0000000 0000000 ==============================
MILK: MACHINE LEARNING TOOLKIT
==============================
Machine Learning in Python
--------------------------
Milk is a machine learning toolkit in Python.
Its focus is on supervised classification with several classifiers available:
SVMs (based on libsvm), k-NN, random forests, decision trees. It also performs
feature selection. These classifiers can be combined in many ways to form
different classification systems.
For unsupervised learning, milk supports k-means clustering and affinity
propagation.
Milk is flexible about its inputs. It optimised for numpy arrays, but can often
handle anything (for example, for SVMs, you can use any dataype and any kernel
and it does the right thing).
There is a strong emphasis on speed and low memory usage. Therefore, most of
the performance sensitive code is in C++. This is behind Python-based
interfaces for convenience.
To learn more, check the docs at `http://packages.python.org/milk/
`_ or the code demos included with the source
at ``milk/demos/``.
Examples
--------
Here is how to test how well you can classify some ``features,labels`` data,
measured by cross-validation::
import numpy as np
import milk
features = np.random.rand(100,10) # 2d array of features: 100 examples of 10 features each
labels = np.zeros(100)
features[50:] += .5
labels[50:] = 1
confusion_matrix, names = milk.nfoldcrossvalidation(features, labels)
print 'Accuracy:', confusion_matrix.trace()/float(confusion_matrix.sum())
If want to use a classifier, you instanciate a *learner object* and call its
``train()`` method::
import numpy as np
import milk
features = np.random.rand(100,10)
labels = np.zeros(100)
features[50:] += .5
labels[50:] = 1
learner = milk.defaultclassifier()
model = learner.train(features, labels)
# Now you can use the model on new examples:
example = np.random.rand(10)
print model.apply(example)
example2 = np.random.rand(10)
example2 += .5
print model.apply(example2)
There are several classification methods in the package, but they all use the
same interface: ``train()`` returns a *model* object, which has an ``apply()``
method to execute on new instances.
Details
-------
License: MIT
Author: Luis Pedro Coelho (with code from LibSVM and scikits.learn)
API Documentation: `http://packages.python.org/milk/ `_
Mailing List: `http://groups.google.com/group/milk-users
`__
Features
--------
- SVMs. Using the libsvm solver with a pythonesque wrapper around it.
- LASSO
- K-means using as little memory as possible. It can cluster millions of
instances efficiently.
- Random forests
- Self organising maps
- Stepwise Discriminant Analysis for feature selection.
- Non-negative matrix factorisation
- Affinity propagation
Recent History
--------------
The ChangeLog file contains a more complete history.
New in 0.5 (05 Nov 2012)
~~~~~~~~~~~~~~~~~~~~~~~~
- Add coordinate-descent based LASSO
- Add unsupervised.center function
- Make zscore work with NaNs (by ignoring them)
- Propagate apply_many calls through transformers
- Much faster SVM classification with means a much faster defaultlearner()
[measured 2.5x speedup on yeast dataset!]
New in 0.4.3 (17 Sept 2012)
~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Add select_n_best & rank_corr to featureselection
- Add Euclidean MDS
- Add tree multi-class strategy
- Fix adaboost with boolean weak learners (issue #6, reported by audy
(Austin Richardson))
- Add ``axis`` arguments to zscore()
New in 0.4.2 (16 Jan 2012)
~~~~~~~~~~~~~~~~~~~~~~~~~~
- Make defaultlearner able to take extra arguments
- Make ctransforms_model a supervised_model (adds apply_many)
- Add expanded argument to defaultlearner
- Fix corner case in SDA
- Fix repeated_kmeans
- Fix parallel gridminimise on Windows
- Add multi_label argument to normaliselabels
- Add multi_label argument to nfoldcrossvalidation.foldgenerator
- Do not fork a process in gridminimise if nprocs == 1 (makes for easier
debugging, at the cost of slightly more complex code).
- Add milk.supervised.multi_label
- Fix ext.jugparallel when features is a Task
- Add milk.measures.bayesian_significance
New in 0.4.1
~~~~~~~~~~~~
- Fix important bug in multi-process gridsearch
New in 0.4.0
~~~~~~~~~~~~
- Use multiprocessing to take advantage of multi core machines (off by
default).
- Add perceptron learner
- Set random seed in random forest learner
- Add warning to milk/__init__.py if import fails
- Add return value to ``gridminimise``
- Set random seed in ``precluster_learner``
- Implemented Error-Correcting Output Codes for reduction of multi-class
to binary (including probability estimation)
- Add ``multi_strategy`` argument to ``defaultlearner()``
- Make the dot kernel in svm much, much, faster
- Make sigmoidal fitting for SVM probability estimates faster
- Fix bug in randomforest (patch by Wei on milk-users mailing list)
For older versions, see ``ChangeLog`` file
milk-0.5/setup.py 0000664 0001750 0001750 00000006161 12045727755 016106 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2012, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
from __future__ import division
import os
try:
import setuptools
except:
print '''
setuptools not found.
On linux, the package is often called python-setuptools'''
from sys import exit
exit(1)
from numpy.distutils.core import setup, Extension
execfile('milk/milk_version.py')
long_description = file('README.rst').read()
undef_macros = []
define_macros = []
if os.environ.get('DEBUG'):
undef_macros = ['NDEBUG']
if os.environ.get('DEBUG') == '2':
define_macros = [
('_GLIBCXX_DEBUG','1'),
('EIGEN_INTERNAL_DEBUGGING', '1'),
]
_extensions = {
'milk.unsupervised._kmeans' : ['milk/unsupervised/_kmeans.cpp'],
'milk.unsupervised._som' : ['milk/unsupervised/_som.cpp'],
'milk.supervised._svm' : ['milk/supervised/_svm.cpp'],
'milk.supervised._tree' : ['milk/supervised/_tree.cpp'],
'milk.supervised._perceptron' : ['milk/supervised/_perceptron.cpp'],
'milk.supervised._lasso' : ['milk/supervised/_lasso.cpp'],
}
ext_modules = [
Extension(key,
sources=sources,
undef_macros=undef_macros,
define_macros=define_macros,
extra_compile_args=['--std=c++0x'],
)
for key,sources in _extensions.items()
]
packages = filter(lambda p: p.startswith('milk'), setuptools.find_packages())
package_dir = {
'milk.tests': 'milk/tests',
}
package_data = {
'milk.tests': ['data/*'],
}
setup(name = 'milk',
version = __version__,
description = 'Machine Learning Toolkit',
long_description = long_description,
author = u'Luis Pedro Coelho',
author_email = 'luis@luispedro.org',
url = 'http://luispedro.org/software/milk',
license = 'MIT',
packages = packages,
package_dir = package_dir,
package_data = package_data,
ext_modules = ext_modules,
test_suite = 'nose.collector',
)
milk-0.5/milk/ 0000775 0001750 0001750 00000000000 12045730202 015301 5 ustar luispedro luispedro 0000000 0000000 milk-0.5/milk/measures/ 0000775 0001750 0001750 00000000000 12045730202 017125 5 ustar luispedro luispedro 0000000 0000000 milk-0.5/milk/measures/measures.py 0000664 0001750 0001750 00000010735 11723217323 021337 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2010, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# License: MIT
from __future__ import division
import numpy as np
__all__ = [
'accuracy',
'confusion_matrix',
'waccuracy',
'zero_one_loss',
]
def accuracy(real, other=None, normalisedlabels=False, names=None):
'''
acc = accuracy(real, predicted, normalisedlabels=False, names=None)
Compute accuracy (fraction of correct predictions).
Parameters
----------
real : sequence
The real labels
predicted : sequence
The predicted sequence (must be same type as `real`)
normalisedlabels : boolean, optional
Whether labels have been normalised
names : sequence
The underlying names (unused)
Returns
-------
acc : float
'''
if other is None:
import warnings
warnings.warn('milk.measures.accuracy: calling this with one argument is a deprecated interface.', DeprecationWarning)
cmatrix = np.asanyarray(real)
return cmatrix.trace()/cmatrix.sum()
else:
return np.mean(np.asanyarray(real) == other)
def zero_one_loss(real, predicted, normalisedlabels=False, names=None):
'''
loss = zero_one_loss(real, predicted, normalisedlabels={unused}, names={unused})
Parameters
----------
real : sequence
the underlying labels
predicted : sequence
the predicted labels
normalisedlabels : unused
names: unused
Returns
-------
loss : integer
the number of instances where `real` differs from `predicted`
'''
return np.sum(np.asanyarray(real) != np.asanyarray(predicted))
def waccuracy(real, predicted=None, normalisedlabels=False, names=None):
'''
wacc = waccuracy(real, predicted, normalisedlabels={unused}, names={unused})
Weighted accuracy: average of accuracy for each (real) class. Can be very
different from accuracy if the classes are unbalanced (in particular, if
they are very unbalanced, you can get a high accuracy with a bad
classifier).
Parameters
----------
real : sequence
the underlying labels
predicted : sequence
the predicted labels
normalisedlabels : unused
names: unused
Returns
-------
wacc : float
the weighted accuracy
'''
if predicted is None:
import warnings
warnings.warn('milk.measures.accuracy: calling this with one argument is a deprecated interface.', DeprecationWarning)
cmatrix = np.asanyarray(real)
else:
cmatrix = confusion_matrix(real, predicted, normalisedlabels, names)
return (cmatrix.diagonal() / cmatrix.sum(1)).mean()
def confusion_matrix(real, predicted, normalisedlabels=False, names=None):
'''
cmatrix = confusion_matrix(real, predicted, normalisedlabels=False, names=None)
Computes the confusion matrix
Parameters
----------
real : sequence
The real labels
predicted : sequence
The predicted sequence (must be same type as `real`)
normalisedlabels : boolean, optional
Whether labels have been normalised
names : sequence
The underlying names (unused)
Returns
-------
cmatrix : 2 ndarray
'''
if not normalisedlabels:
from ..supervised.normalise import normaliselabels
real, names = normaliselabels(real)
predicted = map(names.index, predicted)
n = np.max(real)+1
cmat = np.zeros((n,n), int)
for r,p in zip(real, predicted):
cmat[r,p] += 1
return cmat
def bayesian_significance(n, c0, c1):
'''
sig = bayesian_significance(n, c0, c1)
Computes the Bayesian significance of the difference between a classifier
that gets ``c0`` correct versus one that gets ``c1`` right on ``n``
examples.
Parameters
----------
n : int
Total number of examples
c0 : int
Examples that first classifier got correct
c1 : int
Examples that second classifier got correct
Returns
-------
sig : float
Significance value
'''
def _logp(r, n, c):
return c*np.log(r)+(n-c)*np.log(1-r)
r = np.linspace(.0001,.9999,100)
lp0 = _logp(r,n,c0)
lp1 = _logp(r,n,c1)
mat = lp0 + lp1[:,np.newaxis]
mat -= mat.max()
mat = np.exp(mat)
mat /= mat.sum()
sig = np.triu(mat).sum()-mat.trace()/2.
return min(sig, 1. - sig)
## TODO: Implement http://en.wikipedia.org/wiki/Matthews_Correlation_Coefficient
milk-0.5/milk/measures/nfoldcrossvalidation.py 0000664 0001750 0001750 00000017707 11723217323 023750 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2011, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
# LICENSE: MIT
from __future__ import division
from ..supervised.classifier import normaliselabels
import numpy as np
__all__ = ['foldgenerator', 'getfold', 'nfoldcrossvalidation']
def foldgenerator(labels, nfolds=None, origins=None, folds=None, multi_label=False):
'''
for train,test in foldgenerator(labels, nfolds=None, origins=None)
...
This generator breaks up the data into `n` folds (default 10).
If `origins` is given, then all elements that share the same origin will
either be in testing or in training (never in both). This is useful when
you have several replicates that shouldn't be mixed together between
training&testing but that can be otherwise be treated as independent for
learning.
Parameters
----------
labels : a sequence
the labels
nfolds : integer
nr of folds (default 10 or minimum label size)
origins : sequence, optional
if present, must be an array of indices of the same size as labels.
folds : sequence of int, optional
which folds to generate
Returns
-------
iterator over `train, test`, two boolean arrays
'''
labels,names = normaliselabels(labels,multi_label=multi_label)
if origins is None:
origins = np.arange(len(labels))
else:
if len(origins) != len(labels):
raise ValueError(
'milk.nfoldcrossvalidation.foldgenerator: origins must be of same size as labels')
origins = np.asanyarray(origins)
fmin = len(labels)
for ell in xrange(len(names)):
if multi_label:
matching = (orig for i,orig in enumerate(origins) if labels[i,ell])
else:
matching = origins[labels == ell]
curmin = len(set(matching))
fmin = min(fmin, curmin)
if fmin == 1:
raise ValueError('''
milk.nfoldcrossvalidation.foldgenerator: nfolds was reduced to 1 because minimum class size was 1.
If you passed in an origins parameter, it might be caused by having a class come from a single origin.
''')
fold = np.zeros(len(labels))
fold -= 1
if nfolds is None:
nfolds = min(fmin, 10)
elif nfolds > fmin:
from warnings import warn
warn('milk.measures.nfoldcrossvalidation: Reducing the nr. of folds from %s to %s (minimum class size).' % (nfolds, fmin))
nfolds = fmin
if multi_label:
foldweight = np.zeros( (nfolds, len(names)), int)
for orig in np.unique(origins):
(locations,) = np.where(orig == origins)
weight = len(locations)
ell = labels[locations[0]]
f = np.argmin(foldweight[:,ell].sum(1))
fold[locations] = f
foldweight[f,ell] += weight
else:
for lab in set(labels):
locations = (labels == lab)
usedorigins = np.unique(origins[locations])
weights = np.array([np.sum(origins == orig) for orig in usedorigins])
foldweight = np.zeros(nfolds, int)
for w,orig in sorted(zip(weights, usedorigins)):
f = np.argmin(foldweight)
if np.any(fold[origins == orig] > -1):
raise ValueError(
'milk.nfoldcrossvalidation.foldgenerator: something is wrong. Maybe origin %s is present in two labels.' % orig)
fold[origins == orig] = f
foldweight[f] += w
for f in xrange(nfolds):
if folds is not None and f not in folds: continue
yield (fold != f), (fold == f)
def getfold(labels, fold, nfolds=None, origins=None):
'''
trainingset,testingset = getfold(labels, fold, nfolds=None, origins=None)
Get the training and testing set for fold `fold` in `nfolds`
Arguments are the same as for `foldgenerator`
Parameters
----------
labels : ndarray of labels
fold : integer
nfolds : integer
number of folds (default 10 or size of smallest class)
origins : sequence, optional
if given, then objects with same origin are *not* scattered across folds
'''
if nfolds < fold:
raise ValueError('milk.getfold: Attempted to get fold %s out of %s' % (fold, nfolds))
for i,(t,s) in enumerate(foldgenerator(labels, nfolds, origins)):
if i == fold:
return t,s
raise ValueError('milk.getfold: Attempted to get fold %s but the number of actual folds was too small (%s)' % (fold,i))
def nfoldcrossvalidation(features, labels, nfolds=None, learner=None, origins=None, return_predictions=False, folds=None, initial_measure=0, classifier=None,):
'''
Perform n-fold cross validation
cmatrix,names = nfoldcrossvalidation(features, labels, nfolds=10, learner={defaultclassifier()}, origins=None, return_predictions=False)
cmatrix,names,predictions = nfoldcrossvalidation(features, labels, nfolds=10, learner={defaultclassifier()}, origins=None, return_predictions=True)
cmatrix will be a N x N matrix, where N is the number of classes
cmatrix[i,j] will be the number of times that an element of class i was
classified as class j
names[i] will correspond to the label name of class i
Parameters
----------
features : a sequence
labels : an array of labels, where label[i] is the label corresponding to features[i]
nfolds : integer, optional
Nr of folds. Default: 10
learner : learner object, optional
learner should implement the train() method to return a model
(something with an apply() method). defaultclassifier() by default
This parameter used to be called `classifier` and that name is still supported
origins : sequence, optional
Origin ID (see foldgenerator)
return_predictions : bool, optional
whether to return predictions (default: False)
folds : sequence of int, optional
which folds to generate
initial_measure : any, optional
what initial value to use for the results reduction (default: 0)
Returns
-------
cmatrix : ndarray
confusion matrix
names : sequence
sequence of labels so that cmatrix[i,j] corresponds to names[i], names[j]
predictions : sequence
predicted output for each element
'''
import operator
from .measures import confusion_matrix
if len(features) != len(labels):
raise ValueError('milk.measures.nfoldcrossvalidation: len(features) should match len(labels)')
if classifier is not None:
if learner is not None:
raise ValueError('milk.nfoldcrossvalidation: Using both `learner` and `classifier` arguments. They are the same, but `learner` is preferred')
learner = classifier
if learner is None:
from ..supervised.defaultclassifier import defaultclassifier
learner = defaultclassifier()
labels,names = normaliselabels(labels)
if return_predictions:
predictions = np.empty_like(labels)
predictions.fill(-1) # This makes it clearer if there are bugs in the programme
try:
features = np.asanyarray(features)
except:
features = np.asanyarray(features, dtype=object)
if origins is not None:
origins = np.asanyarray(origins)
nclasses = labels.max() + 1
results = []
measure = confusion_matrix
train_kwargs = {}
for trainingset,testingset in foldgenerator(labels, nfolds, origins=origins, folds=folds):
if origins is not None:
train_kwargs = { 'corigins' : origins[trainingset] }
model = learner.train(features[trainingset], labels[trainingset], **train_kwargs)
cur_preds = np.array([model.apply(f) for f in features[testingset]])
if return_predictions:
predictions[testingset] = cur_preds
results.append(measure(labels[testingset], cur_preds))
result = reduce(operator.add, results, initial_measure)
if return_predictions:
return result, names, predictions
return result, names
milk-0.5/milk/measures/cluster_agreement.py 0000664 0001750 0001750 00000003231 11723217323 023214 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2011, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division
import numpy as np
def rand_arand_jaccard(recovered, labels):
'''
rand, a_rand, jaccard = rand_arand_jaccard(recovered, labels)
Compute Rand, Adjusted Rand, and Jaccard indices
These share most of the computation. Therefore, it is best to compute them
together even if you are only going to use some.
Parameters
----------
recovered : sequence of int
The recovered clusters
labels : sequence of int
Underlying labels
Returns
-------
rand : float
Rand index
a_rand : float
Adjusted Rand index
jaccard : float
Jaccard index
References
----------
http://en.wikipedia.org/wiki/Rand_index
http://en.wikipedia.org/wiki/Jaccard_index
'''
from scipy.misc import comb
recovered = np.asanyarray(recovered)
labels = np.asanyarray(labels)
contig,_,_ = np.histogram2d(recovered, labels,np.arange(max(recovered.max()+2,labels.max()+2)))
A_0 = contig.sum(0)
A_1 = contig.sum(1)
Ai2 = np.sum(A_0*(A_0-1)/2.)
Bi2 = np.sum(A_1*(A_1-1)/2.)
n = A_0.sum()
a = comb(contig.ravel(), 2).sum()
b = comb(A_0, 2).sum()-a
c = comb(A_1, 2).sum()-a
d = comb(n, 2)-a-b-c
rand = (a+d)/(a+b+c+d)
jaccard = (a+d)/(b+c+d)
index = np.sum(contig*(contig-1)/2)
expected = Ai2*Bi2/n/(n-1)*2.
maxindex = (Ai2+Bi2)/2.
a_rand = (index-expected)/(maxindex-expected)
return rand, a_rand, jaccard
milk-0.5/milk/measures/curves.py 0000664 0001750 0001750 00000003744 11723217323 021024 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2011, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division
import numpy as np
def precision_recall(values, labels, mode='all', nr_steps=100):
'''
precision, recall = precision_recall(values, labels, mode='all', nr_steps=100)
Compute a precision-recall curve.
For a given threshold ``T``, consider that the positions where ``values >=
T`` are classified as True. Precision is defined as ``TP/(TP+FP)``, while
recall is defined as ``TP/(TP+FN)``.
Parameters
----------
values : sequence of numbers
labels : boolean sequence
mode : str, optional
Which thresholds to consider. Either 'all' (i.e., use all values of
`values` as possible thresholds), or 'step' (using `nr_steps`
equidistant points from ``min(values)`` to ``max(values)``)
nr_steps : integer, optional
How many steps to use. Only meaningfule if ``mode == 'steps'``
Returns
-------
precision : a sequence of floats
recall : a sequence of floats
Actually, ``2 x P`` array is returned.
'''
values = np.asanyarray(values)
labels = np.asanyarray(labels)
if len(values) != len(labels):
raise ValueError('milk.measures.precision_recall: `values` must be of same length as `labels`')
if mode == 'all':
points = list(set(values))
points.sort()
elif mode == 'steps':
points = np.linspace(values.min(), values.max(), nr_steps)
else:
raise ValueError('milk.measures.precision_recall: cannot handle mode: `%s`' % mode)
true_pos = float(np.sum(labels))
precision_recall = np.empty((len(points),2), np.float)
for i,p in enumerate(points):
selected = (values >= p)
selected = labels[selected]
precision_recall[i] = (np.mean(selected), np.sum(selected)/true_pos)
return precision_recall.T
milk-0.5/milk/measures/__init__.py 0000664 0001750 0001750 00000000141 11723217323 021240 0 ustar luispedro luispedro 0000000 0000000 from measures import accuracy, waccuracy, zero_one_loss, confusion_matrix, bayesian_significance
milk-0.5/milk/demos/ 0000775 0001750 0001750 00000000000 12045730202 016410 5 ustar luispedro luispedro 0000000 0000000 milk-0.5/milk/demos/adaboost.py 0000664 0001750 0001750 00000001150 11723217323 020561 0 ustar luispedro luispedro 0000000 0000000 import pylab as plt
import milk.supervised.tree
import milk.supervised.adaboost
from milksets import wine
import milk.supervised.multi
weak = milk.supervised.tree.stump_learner()
learner = milk.supervised.adaboost.boost_learner(weak)
learner = milk.supervised.multi.one_against_one(learner)
features, labels = wine.load()
cmat,names,predictions = milk.nfoldcrossvalidation(features,labels, classifier=learner, return_predictions=True)
colors = "rgb"
codes = "xo"
for y,x,r,p in zip(features.T[0], features.T[1], labels, predictions):
code = codes[int(r == p)]
plt.plot([y],[x], colors[p]+code)
plt.show()
milk-0.5/milk/demos/rf_wine_2d.py 0000664 0001750 0001750 00000001677 11723217323 021021 0 ustar luispedro luispedro 0000000 0000000 from milk.supervised import randomforest
from milk.supervised.multi import one_against_one
import milk.nfoldcrossvalidation
import milk.unsupervised
import pylab
from milksets import wine
# Load 'wine' dataset
features, labels = wine.load()
# random forest learner
rf_learner = randomforest.rf_learner()
# rf is a binary learner, so we transform it into a multi-class classifier
learner = one_against_one(rf_learner)
# cross validate with this learner and return predictions on left-out elements
cmat,names, preds = milk.nfoldcrossvalidation(features, labels, classifier=learner, return_predictions=1)
print 'cross-validation accuracy:', cmat.trace()/float(cmat.sum())
# dimensionality reduction for display
x,v = milk.unsupervised.pca(features)
colors = "rgb" # predicted colour
marks = "xo" # whether the prediction was correct
for (y,x),p,r in zip(x[:,:2], preds, labels):
c = colors[p]
m = marks[p == r]
pylab.plot(y,x,c+m)
pylab.show()
milk-0.5/milk/demos/svm-decision-boundary.py 0000664 0001750 0001750 00000002263 11723217323 023214 0 ustar luispedro luispedro 0000000 0000000 from pylab import *
import numpy as np
from milksets.wine import load
import milk.supervised
import milk.unsupervised.pca
import milk.supervised.svm
features, labels = load()
features = features[labels < 2]
labels = labels[labels < 2]
features,_ = milk.unsupervised.pca(features)
features = features[:,:2]
learner = milk.supervised.svm.svm_raw(kernel=np.dot, C=12)
model = learner.train(features, labels)
w = np.dot(model.svs.T, model.Yw)
b = model.b
x = np.linspace(-.5, .1, 100)
y = -w[0]/w[1]*x + b/w[1]
plot(features[labels == 1][:,0], features[labels == 1][:,1], 'bx')
plot(features[labels == 0][:,0], features[labels == 0][:,1], 'ro')
plot(x,y)
savefig('svm-demo-points.pdf')
clf()
learner = milk.supervised.svm.svm_raw(kernel=milk.supervised.svm.rbf_kernel(1.), C=12)
model = learner.train(features, labels)
Y, X = (np.mgrid[:101,:101]-50)/12.5
values = [model.apply((y,x)) for y,x in zip(Y.ravel(),X.ravel())]
values = np.array(values).reshape(Y.shape)
sfeatures = features*12.5
sfeatures += 50
plot(sfeatures[labels == 0][:,0], sfeatures[labels == 0][:,1], 'bo')
plot(sfeatures[labels == 1][:,0], sfeatures[labels == 1][:,1], 'ro')
imshow(values.T)
savefig('svm-demo-boundary.pdf')
milk-0.5/milk/demos/__init__.py 0000664 0001750 0001750 00000000000 11723217323 020515 0 ustar luispedro luispedro 0000000 0000000 milk-0.5/milk/wrapper/ 0000775 0001750 0001750 00000000000 12045730202 016761 5 ustar luispedro luispedro 0000000 0000000 milk-0.5/milk/wrapper/wraplibsvm.py 0000664 0001750 0001750 00000004573 11723217323 021540 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2010, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division
from milk.supervised.classifier import normaliselabels
try:
from libsvm import svm as libsvm
except ImportError:
try:
import svm as libsvm
except ImportError:
libsvm = None
from tempfile import NamedTemporaryFile
class libsvmModel(object):
def __init__(self, model, names, output_probability):
self.model = model
self.names = names
self.output_probability = output_probability
def apply(self,feats):
if self.output_probability:
return self.model.predict_probability(feats)
res = self.model.predict(feats)
return self.names[int(res)]
def __getstate__(self):
# This is really really really hacky, but it works
N = NamedTemporaryFile()
self.model.save(N.name)
S = N.read()
return S,self.output_probability,self.names
def __setstate__(self,state):
if libsvm is None:
raise RuntimeError('LibSVM Library not found. Cannot use this classifier.')
S,self.output_probability,self.names = state
N = NamedTemporaryFile()
N.write(S)
N.flush()
self.model = libsvm.svm_model(N.name)
class libsvmClassifier(object):
def __init__(self,probability = False, auto_weighting = True):
if libsvm is None:
raise RuntimeError('LibSVM Library not found. Cannot use this classifier.')
self.param = libsvm.svm_parameter(kernel_type = libsvm.RBF, probability = probability)
self.output_probability = probability
self.auto_weighting = auto_weighting
def set_option(self,optname,value):
setattr(self.param, optname, value)
def train(self, features, labels):
labels,names = normaliselabels(labels)
if self.auto_weighting:
nlabels = labels.max() + 1
self.param.nr_weight = int(nlabels)
self.param.weight_label = range(nlabels)
self.param.weight = [(labels != i).mean() for i in xrange(nlabels)]
problem = libsvm.svm_problem(labels.astype(float), features)
model = libsvm.svm_model(problem, self.param)
return libsvmModel(model, names, self.output_probability)
milk-0.5/milk/wrapper/__init__.py 0000664 0001750 0001750 00000000000 11723217323 021066 0 ustar luispedro luispedro 0000000 0000000 milk-0.5/milk/utils/ 0000775 0001750 0001750 00000000000 12045730202 016441 5 ustar luispedro luispedro 0000000 0000000 milk-0.5/milk/utils/parallel.py 0000664 0001750 0001750 00000004655 12017466466 020642 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2011-2012, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division, with_statement
import multiprocessing
max_procs = 1
_used_procs = multiprocessing.Value('i', 1)
_plock = multiprocessing.Lock()
def set_max_processors(value=None):
'''
set_max_processors(value=None)
Set the maximum number of processors to ``value`` (or to the number of
physical CPUs if ``None``).
Note that this is valid for the current process and its children, but not
the parent.
Parameters
----------
value : int, optional
Number of processors to use. Defaults to number of CPUs (as returned by
``multiprocessing.cpu_count()``).
'''
global max_procs
if value is None:
value = multiprocessing.cpu_count()
max_procs = value
def get_proc():
'''
available = get_proc()
Reserve a processor
Returns
-------
available : bool
True if a processor is available
'''
with _plock:
if _used_procs.value >= max_procs:
return False
_used_procs.value += 1
return True
def release_proc():
'''
release_proc()
Returns a processor to the pool
'''
with _plock:
_used_procs.value -= 1
def release_procs(n, count_current=True):
'''
release_procs(n, count_current=True)
Returns ``n`` processors to the pool
Parameters
----------
n : int
Number of processors to release
count_current : bool, optional
Whether the current processor is to be included in ``n`` (default: True)
'''
if count_current:
n -= 1
if n > 0:
with _plock:
_used_procs.value -= n
def get_procs(desired=None, use_current=True):
'''
n = get_procs(desired=None, use_current=True)
Get the up to ``desired`` processors (use None for no maximum).
Parameters
----------
desired : int, optional
Number of processors you wish. By default, there is no maximum
use_current: bool, optional
Whether to count the current processor, True by default.
'''
if desired is None:
desired = 1024 # This should last a few years
n = (1 if use_current else 0)
while n < desired:
if get_proc():
n += 1
else:
return n
return n
milk-0.5/milk/utils/__init__.py 0000664 0001750 0001750 00000000025 12043765776 020575 0 ustar luispedro luispedro 0000000 0000000 from .utils import *
milk-0.5/milk/utils/utils.py 0000664 0001750 0001750 00000005335 12043765776 020207 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2012, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
import numpy as np
import random
__all__ = [
'get_nprandom',
'get_pyrandom',
]
def get_nprandom(R):
'''
R' = get_nprandom(R)
Returns a numpy.RandomState from R
Parameters
----------
R : can be one of:
None : Returns the default numpy global state
integer : Uses it as a seed for constructing a new random generator
RandomState : returns R
Returns
-------
R' : np.RandomState
'''
if R is None:
return np.random.mtrand._rand
if type(R) == int:
return np.random.RandomState(R)
if type(R) is random.Random:
return np.random.RandomState(R.randint(0, 2**30))
if type(R) is np.random.RandomState:
return R
raise TypeError("get_nprandom() does not know how to handle type {0}.".format(type(R)))
def get_pyrandom(R):
'''
R = get_pyrandom(R)
Returns a random.Random object based on R
Parameters
----------
R : can be one of:
None : Returns the default numpy global state
integer : Uses it as a seed for constructing a new random generator
RandomState : returns R
Returns
-------
R' : random.Random
'''
if R is None:
return random.seed.im_self
if type(R) is int:
return random.Random(R)
if type(R) is np.random.RandomState:
return random.Random(R.randint(2**30))
if type(R) is random.Random:
return R
raise TypeError("get_pyrandom() does not know how to handle type {0}.".format(type(R)))
milk-0.5/milk/active/ 0000775 0001750 0001750 00000000000 12045730202 016554 5 ustar luispedro luispedro 0000000 0000000 milk-0.5/milk/active/__init__.py 0000664 0001750 0001750 00000000000 11723217323 020661 0 ustar luispedro luispedro 0000000 0000000 milk-0.5/milk/active/uncertainty.py 0000664 0001750 0001750 00000004216 11723217323 021504 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2010, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
'''
Uncertainty
============
Implements uncertainty-based active learning strategies.
These are strategies that are based on querying those elements in the pool
which we are most uncertain about
Functions
----------
* entropy
* one_minus_max
'''
from __future__ import division
import numpy
def entropy(model, pool):
'''
entropies = entropy(model, pool)
Returns the entropy of each classification output for
members in the pool.
'''
def _entropy(labels, ps):
H = 0.
for p in ps:
if p > 1e-9:
H += p * np.log(p)
return H
return [_entropy(model.apply(u)) for u in pool]
def one_minus_max(model,pool):
'''
oneminus = one_minus_max(model,pool)
oneminus[i] = 1 - max_L { P(pool_i == L) }
Returns one minus the probability for the best label guess.
'''
def _minus1(labels, ps):
return 1. - np.max(ps)
return [_minus1(model.apply(u)) for u in pool]
milk-0.5/milk/active/eimpact.py 0000664 0001750 0001750 00000004604 11723217323 020562 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2010, Luis Pedro Coelho
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
from __future__ import division
import numpy
from ..supervised import svm
from ..supervised.classifier import ctransforms
def expected_impacts(D,labels,U):
'''
EIs = expected_impacts(D,labels,U)
Compute Expected impact for each element of U
Eis[i]: P(label[i] == 1) * IMPACT(label[i] == 1) + P(label[i] == 0) * IMPACT(label[i] == 0)
'''
assert len(D) == len(labels), 'Nr of labeled examples should match lenght of labels vector'
K = svm.rbf_kernel(20000)
prob_classifier = ctransforms(svm.svm_raw(kernel=K,C=4),svm.svm_sigmoidal_correction())
label_classifier = ctransforms(svm.svm_raw(kernel=K,C=4),svm.svm_binary())
prob_classifier.train(D,labels)
u_probs = prob_classifier(U)
u_labels = (u_probs > .5)
impacts = []
for u,p in zip(U,u_probs):
print len(impacts)
label_classifier.train(numpy.vstack((D,u)),numpy.hstack((labels,[0])))
u_labels_0 = label_classifier(U)
label_classifier.train(numpy.vstack((D,u)),numpy.hstack((labels,[1])))
u_labels_1 = label_classifier(U)
e_impact = (1.-p)*(u_labels != u_labels_0).sum() + p*(u_labels != u_labels_1).sum()
impacts.append(e_impact)
return impacts
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
milk-0.5/milk/tests/ 0000775 0001750 0001750 00000000000 12045730202 016443 5 ustar luispedro luispedro 0000000 0000000 milk-0.5/milk/tests/test_measures.py 0000664 0001750 0001750 00000002215 11723217323 021706 0 ustar luispedro luispedro 0000000 0000000 import milk.measures.measures
import numpy as np
import numpy
from milk.measures import accuracy, waccuracy, bayesian_significance
def test_100():
C=numpy.zeros((2,2))
C[0,0]=100
C[1,1]=50
assert accuracy(C) == 1.
assert waccuracy(C) == 1.
def test_0():
C = numpy.array([
[0, 10],
[10, 0]
])
assert waccuracy(C) == 0.
assert accuracy(C) == 0.
def test_50():
C = numpy.array([
[10, 10],
[10, 10]
])
assert accuracy(C) == .5
assert waccuracy(C) == .5
def test_unbalanced():
C = numpy.array([
[20, 10],
[10, 0]
])
assert accuracy(C) == .5
assert waccuracy(C) == 1./3
def test_confusion_matrix():
np.random.seed(323)
labels0 = np.arange(101)%3
labels1 = (labels0 + np.random.rand(101)*2).astype(np.int) % 3
cmat = milk.measures.measures.confusion_matrix(labels0, labels1)
for i in xrange(3):
for j in xrange(3):
assert cmat[i,j] == np.sum( (labels0 == i) & (labels1 == j) )
def test_significance():
assert np.allclose(.5, [bayesian_significance(1024,i,i) for i in xrange(0, 1025, 3)])
milk-0.5/milk/tests/test_nnmf.py 0000664 0001750 0001750 00000003023 11723217323 021016 0 ustar luispedro luispedro 0000000 0000000 import milk.unsupervised
import numpy as np
def test_nnmf():
def test3(method):
np.random.seed(8)
X3 = np.random.rand(20,3)
X = np.c_[ X3,
X3*2+np.random.rand(20,3)/20.,
-X3*2+np.random.rand(20,3)/10.]
W,V = method(X, 3, R=7)
assert np.sum((np.dot(W,V)-X)**2)/np.sum(X**2) < .5
yield test3, milk.unsupervised.lee_seung
yield test3, milk.unsupervised.sparse_nnmf
def test_sparse_nnmf():
# This is really just a smoke test because the test case is not sparse!!
from milk.unsupervised import sparse_nnmf
np.random.seed(8)
X3 = np.random.rand(20,3)
X = np.c_[ X3,
X3*2+np.random.rand(20,3)/20.,
-X3*2+np.random.rand(20,3)/10.]
W,V = sparse_nnmf(X, 3, sparsenessW=.7, sparsenessH=.7, R=7)
assert not np.any(np.isnan(W))
assert not np.any(np.isnan(V))
error = np.dot(W,V)-X
assert error.var() < X.var()
def test_hoyer_project():
from milk.unsupervised.nnmf.hoyer import _L1for, _project
def sp(n, L1, L2):
return (np.sqrt(n) - L1/L2)/(np.sqrt(n) - 1)
sparseness = .6
n = 9.
row = np.arange(int(n))/n
L2 = np.sqrt(np.dot(row, row))
L1 = _L1for(sparseness, row, L2)
assert np.abs(sp(n, L1, L2) - sparseness) < 1.e-4
row_ = _project(row, L1, L2)
assert not np.any(np.isnan(row_))
assert np.all(row_ >= 0)
L2 = np.sqrt(np.dot(row, row))
L1 = np.sum(np.abs(row_))
res = sp(n, L1, L2)
assert np.abs(res - sparseness) < 1.e-4
milk-0.5/milk/tests/test_tree.py 0000664 0001750 0001750 00000005416 11723217323 021027 0 ustar luispedro luispedro 0000000 0000000 import milk.supervised.tree
import milk.supervised._tree
from milk.supervised._tree import set_entropy
from milk.supervised.tree import information_gain, stump_learner
import numpy as np
def test_tree():
from milksets import wine
features, labels = wine.load()
selected = (labels < 2)
features = features[selected]
labels = labels[selected]
C = milk.supervised.tree.tree_classifier()
model = C.train(features,labels)
assert (np.array([model.apply(f) for f in features]) == labels).mean() > .5
def test_split_subsample():
import random
from milksets import wine
features, labels = wine.load()
labels = labels.astype(np.int)
seen = set()
for i in xrange(20):
random.seed(2)
i,s = milk.supervised.tree._split(features[::10], labels[::10], None, milk.supervised.tree.information_gain, 2, random)
seen.add(i)
assert len(seen) <= 2
def test_set_entropy():
labels = np.arange(101)%3
counts = np.zeros(3)
entropy = milk.supervised._tree.set_entropy(labels, counts)
slow_counts = np.array([(labels == i).sum() for i in xrange(3)])
assert np.all(counts == slow_counts)
px = slow_counts.astype(float)/ slow_counts.sum()
slow_entropy = - np.sum(px * np.log(px))
assert np.abs(slow_entropy - entropy) < 1.e-8
def slow_information_gain(labels0, labels1):
H = 0.
N = len(labels0) + len(labels1)
nlabels = 1+max(labels0.max(), labels1.max())
counts = np.empty(nlabels, np.double)
for arg in (labels0, labels1):
H -= len(arg)/float(N) * set_entropy(arg, counts)
return H
def test_information_gain():
np.random.seed(22)
for i in xrange(8):
labels0 = (np.random.randn(20) > .2).astype(int)
labels1 = (np.random.randn(33) > .8).astype(int)
fast = information_gain(labels0, labels1)
slow = slow_information_gain(labels0, labels1)
assert np.abs(fast - slow) < 1.e-8
def test_information_gain_small():
labels1 = np.array([0])
labels0 = np.array([0, 1])
assert information_gain(labels0, labels1) < 0.
def test_z1_loss():
from milk.supervised.tree import z1_loss
L0 = np.zeros(10)
L1 = np.ones(10)
L1[3] = 0
W0 = np.ones(10)
W1 = np.ones(10)
assert z1_loss(L0, L1) == z1_loss(L0, L1, W0, W1)
assert z1_loss(L0, L1) != z1_loss(L0, L1, W0, .8*W1)
assert z1_loss(L0, L1) > 0
def test_stump_learner():
learner = stump_learner()
np.random.seed(111)
for i in xrange(8):
features = np.random.random_sample((40,2))
features[:20,0] += .5
labels = np.repeat((0,1),20)
model = learner.train(features, labels, normalisedlabels=True)
assert not model.apply([0.01,.5])
assert model.apply(np.random.random_sample(2)+.8)
assert model.idx == 0
milk-0.5/milk/tests/test_set2binary_array.py 0000664 0001750 0001750 00000000741 11723217323 023344 0 ustar luispedro luispedro 0000000 0000000 import numpy as np
from milk.supervised import set2binary_array
def test_set2binary_array_len():
s2f = set2binary_array.set2binary_array()
inputs = [ np.arange(1,3)*2, np.arange(4)**2, np.arange(6)+2 ]
labels = [0,0,1]
model = s2f.train(inputs,labels)
assert len(model.apply(inputs[0])) == len(model.apply(inputs[1]))
assert len(model.apply(inputs[0])) == len(model.apply(inputs[2]))
assert len(model.apply(inputs[0])) == len(model.apply(range(128)))
milk-0.5/milk/tests/data/ 0000775 0001750 0001750 00000000000 12045730202 017354 5 ustar luispedro luispedro 0000000 0000000 milk-0.5/milk/tests/data/regression-2-Dec-2009.pp.gz 0000664 0001750 0001750 00000126245 11723217323 023754 0 ustar luispedro luispedro 0000000 0000000 ‹ÀþK regression-2-Dec-2009.pp íýWW[Ô5
ßï_!Š˜¥rÍRŒ ŠD e¨ˆýíßê}ŒxZ{ßó]žÓÚsáÞºÖªªYsŽØGºØmÿú¾st½û±;\ÿþëÛþV³»Û ý÷qwè~lïíïþêöÿÛ‰þ»(¿ûo»—ïwâÿ..Eÿíï$ÿ]h/ü·“Nþžý·º“O>Çwqlþ[J‹É§Åzm¿´3ü·SþwqíÂh&×L~ý7ùõþN5¹2žŸüer]?àË?ï'ÿÜ-ÆÏ'?Íäÿ™\†Ïø'ÆSÖ'_L¾ ù²kÎO Mþ1¹_TÌL¾Á‚Æ™s“ O.hË3r~m’É÷ÝîdiæâßÉÇ©B3ù?N–QM® 'WŽ ¾¼9ùËd•MñFîRMÖ?Ø…U³“LžÝ
3“¯#ÝQþ?Lî>LnÔOvÍr | aëL·ö~Ï>dòg¸3ùÇäÉfx;¹ßäA}o6òÏ'ÿOë ²¨vüôäÊdñÓ}ÿ@ Ž¦‹ŸÕ‡å¸2ùW®KèõÑÙÂä ÉG{„ÏN–eåXcÓùíŒÊKx¯ÃÕö&?'OKäÈM&‡Ð~ëz@"708—ò¥§“*} §Pa•åkÿ¾nƒý™ü»ÏjœHï¯k'çuØ ¾ûöäºÎ?²þÊIv‰®¥ò46àðë›’ Øüô=™¼À°i·OÈÿ’ò¶íÚ^^œ\›_ü¯oìoïâT&Ÿ¶Ù.`„bQH< ~ Bಋ6'd7òË1·´úô™jUÈåïOîºãC»’ɧM&wALˆ|ß®fr¼^sæìíOX=¨^áñä º
o©?ÅÍ°_É9ü\6R6ð¯¼Ð„Fë~Þß¼k7>á>¼
Íàx•±#ÉS+ƒð,È ¦ÿ";½×÷÷;ˆ|Ïs öÂqLnʆãzwE’˜è¡ÈP7C{rNEg㙯3æ_…áÆhÁ²à5a¿¶‡¤‘Wíz/ ûᚬ×We°eØØ(“Ý ’?'vḠÀF©0è@Zx0yµ¾œ<<Î ©
²Fr5Q]{¢šÈmamH’¨U‘ÒŽs“‡µdÁÄo$„¸¥MîO.„6hý—1(»úH(«ëý.WªrðËÞœ…( ¤m29Ij›¯³%Üߎ·å[¼)úÊ2bUc“Öt•¿[—ãU A @Zvº¶ ò0y¨k¨ì†Î}G@†ŽJã$³XUE·¦ô›ÜžÖoˆ¾ËÂ%8^•SÇãƒ6ºª_ ŒÃÆAAmvñÔ_¿oƒj ì—1ìY%(üÖ—©Ü ò4bï9äÁ$Ú¸Í%O ëQÉbÈ'[ ¹Ëe %É„êÛΊXlÌ{Õ9VxG~kcpOe÷E¶[¢oÛÇ“†_ßD:É[U²¦ûVL·ÏnË*ªè#d¯½÷ç
åçöú®ýìH©ÄLäp§Œ"ŒùôHDH?Ö£·kl“¢¯Çø•H»Ì¸XÍŽoû o@RÄØbX})äêuÕU¥ÎvÏCû¨É[uåRÆíÛ
3]Q3ÆïB=:‘5Ί쒃øU_èW<¯ÛCmûkÏEƒ¶©ÈŒ^ O®Æ‡‡ËY°Í}¹pý2Þ®ù-Ä>lØRäB+9'‹ib¥¢||â{Ón²Šm+…‰¶×ÞLy;¬Mh2TüýAs0@¡&Ý¡¤áŠ>ÍoÉ ™üJðs`$\…?ŒÎجdù½¢U#K·û·«/a-‰dh«vŽ…Á›hGùI·?m3Q=CzO³¬põ—}ÛüªPì¾Ve™ª: ó«K¼odz^䶀FuÓ™êœxŒõØmè>N†ÞÎk±Ë¸‹dÔ‹É#³'bü © -@Ùã§](ÅQÕz\Ü=N·²Æ‚ÊCGî ÷k3‰ŠUû"?_`×¾ ùËYáãeY(Ý‘D¼oܧa²|¬‚šwWåkôÈS Ô7N]ÎtE©^—îH+6@=ÔûÐô?Àù¼éY¿ˆ®¿.>Œ[ÈÈ>ÐsФ±üã?¥t»ßÎüˆ‹*=°ì(¦ã˜ ÉMiFs…g°¤Oÿgô<”@*µ4bš4±‚V
šþb½; sÈiõ[â~êñlj켧UxZ²npùµ×÷ 6‹[}k±szø0Ë„Ô/€òžò
—öÑ=Ñж½„1¡90j h{²W÷Z^«›`Ãü²Ï‹f‚
Ç{Çi6l« +½beíÐÝšv4R‹©Qe¢LôÇo'íMœ‹™ƒØ»X5°X³½aþQ e*ÝMsÚfÇ %¨¸RT
í×Qݨ°ñ±úœç߉mqA¼ÝûÐL·.ZºË«âò<?u‹"a¹ŠÄü ÅtçšjÿÎÞ
Îhró§âƒw=
•Þºp‚[éÌŽVm “ìC[zèÓ3[·EØûpqãÙXϪ]ã #Ò ìÜ Ïý ˆ ´Ëw®¬øw¥ÜíE ˜ÐDz¯K°Ð““"T$Ë‘¬ÔòÝšÈOK.$Òþ+..nÚû«wÕ¦·@.3öÎ_äxZõ«†ÂíØYßyÙÇxø$8ëwÁªs ûÔåž^ >€êw®tÊ(^8 žHÂJ±o®óðb-{ÀB,žªIgäöD:J@8Â}ùÈÛ[thšy8IÐVq`Öà®Ãq\Þõt
º‡D¯ò Õ»™—{Üøfê
vÊI"°O³¯7>šóO™œÈ®˜Câb«Ï]¼†-?î„x©åvÕùcþN~“<©Ú9Äkòfîqé,ýâ ç”+¡ÍûñÃEë2«¬¨„ȼi<Ú½_/â.¬Õÿ¡¡¹c˜€0šÄ²Ö/é Æ?-!¯ª 6òn8«(øº«Ú/eÞ[9 Ó°A÷U~Ü;Wû ~r$‚ªé~XWSôöflù£Œ{ÙŸ¨}÷ óš·‹÷ t7!;T²˜6{´~ã¾=Û³õîºÅád×D0á¹pª/;ÖEÓgÂL†óÙ4Ó½ÿ¸Oá™v“ÿÄуàc Åxû÷þ®çŒNá:‘•hª.ªg†M¯"¸fé§^IÄ0#âö΋$ ÈHˆ6˜… a9h#Ò à „¨‰„ …§æÕ… U50$ìAœlÔQ’¤V½
1êµÏ±—ãX]—Q’®+DÉk.ÊE],ºzLç y|5Ž>Ûݹ- –³"àìŸJ~3Dß°„ëö./” 1ÇšD¬«Ýº´ÿbÁ"-M·Y)›Wâ8Ù]…Š‚Ñ>ªÓáÀÃe9ÊÎΊ:³¢{vhŸðZnˆ]ÀË,Ī{üÉQ&}m;mêåÄFÇé]À6VV͸ÀíÑw» €”p‚@<&ºjݳcñ„p¢—?í´‡)@öá÷]‡»ìd —ìE_„hšô«pƸ§Uþxñ—.1dn"zŠb0®ë ÕDœ3£ð ˜¬q;>Ñô ezAÉzÀ]•i¯¼”í1©ÜÚÿ¨Gâmî“=‘÷¦¸¤ –U0íCmóGM,ëƒá’JÌ øUoÙdnì½í™WžòöƒÂ ª!'Ká8€Ô”ÿ¥’™,nÑ ‹‚,½lÈ}À> í¾» ˜Ž™H.0:\ÜVq&§ŒÃ“æŒ#À W¿Ã
ݜ
ŒÉFÄ~£6*dBÕz±mâò>TÓ·ûðg8îKª?#e;«©ŠÍí«BB9¬ÎTtt„mIÙ °Ù€„ĉé«à3Dä‰ a]·íß1æÝO¥4øbÈd«|tÔ¯ì–nîkdJ«Oý~µŠåƒÅ¢ôŽR¯‰EkRÞ=ú@]Vû—篜±K½yÿÚŸ1‘åÚé}!Í(.±Ds <<ÕónIûC¨Ñ³Q¸Õ1‘ã³`Sdë¡ÒÖý7U»ˆ¡hIµ.¨@‰“ŠªÍþ¦Þ±äÝŽÙÍkØß{²k=‹=Ùžhdq´÷ù"ô Mðö5KØçÿ
]0(jæý{˜hcŸoV )Z*4fý³0QGN³B4ÆÕ öŠþpGwEX¸sâxj§Þþ„-¹aŽ¿àËïJI¡.ûö® E<¤ mß¹`eö4?ûØ"3´aÝF…¶E)ðÿî²z¡>+7dûÕ7HüT¸Ýé&!™íºŽ¾úOqw:þ²¥À€{‹Ù# çâoNˆ”"ÓñÜʼªƒX‹pVÏ*QwžÑ•MžÝ®J>½Ð=ÆÝéŽ/ÚÍ~ >¹øgëŽ_hWΪ7‰»ÓV+¨ž ý6
U=Vœ,~’çðµùýA¨Õ¡º‘ül„S*8Cƒ@ÜŒü€&pTŸBnÛnAmátô—µý{€úPò¿(7*»X¤{_
šcyjPg ò¹Õ |P*J,œ¸#:,ªšÌý{B>µj!Ós
Dè`r1²£Œ_ªÿaýÁ.t-·&/t1£žGóÒn‚ÆK&Û›ÛåŠùBÚ0 ²^ùe!†Ñ,) è4ÐOQ
Œ’ÀÍÖkÅdq=y!Px¸ ‚`tÝ3÷*p`8›"»¨.G~Qµ°¥šF®†7Íg§Cï*ÊÖý9 Äþ!•E_Tòº´ï©V¶ñº‰º Æ¥.¼ù<¾âe’Ü깆²rÅ{¹,Kƒ: „îêJ/F¡ ÓW4¿¢µà¨@¢æs½‹—èý%
Bž={ÎKß^¬]fÉØŸ÷j®Ï¶‡Ž/&§Û¿ôöþ¹7^ 5˜€ì²h^Gƒš}u/kouã+]ÅýVíiÝøºæÉìÎ¥Àþ7BÃt®*$±5 T£jb3{|äqᨣ>zà« 1*ïH±¬X‡\<#Ç0Ùw(¶ôÆ/ûØ=‰LvÅÚ òbÓŠ@ŽLr±ð‡òicօˬÏLZeXñ™Ø¼x›Ö-o]9p0}ïÑ~sQV3¦S¸Á½€%!ó:&1¤Vñ½¼óý…È7P‹“2‘¼}“@ýioÿ»Öô+¦†!·N¸è±xg #]˜ ‰" QCvþ¥ PŸÄê†G¢EG—´WŽ›(ã ±2éEZ_wD Žå¯u¾»tßµz÷©¼H£äoufO «1ƒŠÑ8kÐMoé—ÔòjŸÅ¤ÁXT&ui™Ouy|ŽHtÝ¿³ÄA” "‹Zç¯1(õÖ”KJýxëò¢F®œ}½¢1Vè¤,V}¡f4s4ã³´À,XìŒ Y\§V¬±85yÿC´w•-{»ˆØp&_`mU³ê¯aT>’]h!ƒé#«¾ª¤×G¶EƒÍkz–MT½‹obÞ?—x”¢ÍpN+¤R méZ‚DÍ(ÓC%þ¥!K²Q»©¨ÖÖ>wVÔ…õx‘Ëçù' mk^ª¹å"%g(’&Âòä’â!ë9Rã•Þm€T¬ ÒMÍ}òã¯4Ègšù
{Ösª
1ízkz®Ðj1X¥Ó€Ìà2é ?¢ÇûR¤‡™µÈvzšmq‘½‚„éh•WÚ ··ù¾¬È%¤¿d·I6Š[xUwšÀÄ%‹´®-/>¶àW0äšÄŒwF ~#ÛÆ”7”‹U í¶ì¶˜ü8x’ ò¢ï²Åԛǒ÷…jî
ei¬ ÑÓ ™…åbà”ÄØ^“Ù£+öãi!)¦€¤oëÓ٢爪źÍü~1nÖ};+ÛJݯÖ
qzˆ¿ÉiŽîtr!}Z×FˆÜÝ3½u Oë4©U?‘¿ÒŸQ×u´ÆàÂnú–œãXS
uB3Ï ø:ßL}ÜWÇI ‰S¦iÚÉß[õ„å|Õµ£c˜æ;õ)ƒ,ÄTN!•ó7>i¨÷/
=W"-™™Ò±è a‡³—Á dU]N³.~AtlyÒ´ÿ®ïÉ—>A\sœHÛ<Åu·Îɧ"Â4 ŠWK4ÌY ÌËÙ~1/n(Å["‹ˆ©Å¿Ä¦³:Jè¾2€Ù6žOALýð]¬šŸ%ê N÷ÌV¶ a’ý½®¹XÖ¯aF(É|s"rùŽ7˜Ð5u6 ¥^ãŒÔ(íp©.&‚8ž[ ™í`‘ ]“ãåºA7Ýí™|Á>¦¡@+ºSðÍÅÀãWÅ €ŠŽ£—þ›¶Bn?²JjþßSz²%–/zã³Ót
X,<ÎH‘›9Þ[„ÌÞ ÜKœLÕŸÝ_RLǾ ölîÜôY¯mâ\|óß.Ûûl‰ú¢qiïøS´pAœìe*‰z㟥ªVZ ãS+ºŽ?‚,š-•™VÈ*ò¡n’‹zµµâÁO%WÕÑi`d\¥o.‰Ü€ í„=´'$lWD3JããO8JÑõ3«Á;$ÀELüö›¤÷O£ÆS_”ëß‹¹ÕTg6*ý(9³_B‚â•Îªk·ÔùÀó`ïÚ}gð¨Ùܽ 1Cÿhùš*Wgj¨«ÐÂj)Þ=ª]Ò`«ò5š9rÎó8Hµ§*µ¸¬ìŒ’×?~‘!«_×ËxÀšl!$?ª¹ÐË·Œ+¸]~'² ËÀ7ì¦:#n±Tý²,ø]H,î×Þþµ2$»%rB¤ÏŽh ãÈÌ)nùsïŠxfö˜\‚’ª/JõžÍ<:"Y™yŸdºyÖ6bZí ]°QÑ„P{‘¦oDAR–É¿.~г ü
Y^΂*
'K®° øžxÝ—ÚÚw·Ôz(sEö¥Ü bºy~宜ñÐh˜uPY8âHXtUj$ŸG²õzèz¢¸+Õ×½í3¢õãÖ•±^ìŸFG
>9
ÃÔqUêwÁHX¨\€ÄŒ ÎÅœ=ò™Ã~¼PäÇÊ{:–8)äœ4$üÄÊM†Úû1ÌÆ/äþ±fŠ]É"˜¡· g‹_ùñ¾˜bÖÚª î:Ö å`+JŠL¥¤
¤¢ö±+-.=yZwA!þÈ+EÍÔçR|oÇ‘¥l0oó ÐUl¯Qï‘"UÿÎdÅA&ûÂH~.³ß÷!¹ö[U;jhOTÜ^ç¨ÏE€¸˜Ùñùž°2SÑ#5Vl¦H®Î5Óš><¥Çþ*sË{†Ý¹lFŠxÜ®µ:¬¼°fW¾9yq$ü#̽¾~Y,8j®ÈKѼZÚf>úhr÷Ú»
OE¢±ÿ}ááëµa<.ÍÈ¿ \X ªü+Oïˆä^÷+ 4T]í=+˜¨˜‘õ‘Þ®]RP;¢¶§Ë¶° oý³Ge X ئEÑ>£RtzèƒæM ¹c¥÷X$kðÖeŽD €˜+NëÝš¯dÕ£hUºc9wÜ*<Ò …tº ΩPïÏ¢šÊk'¬‘àj,È]æDð¨4 ’ øÕºcÍ|Y‡¦?ãʔڽɦl„³`K±uª|ÒØ3*xNbVr•3kÅid@%[Ö9R‹Þ’·dµÙðËîÛ¥v{OˆÜ²[ˆ‹
ˆ•kö›
ˆ÷)]
òäÕç~Š E†ÈËåû è~=V ÞÝ>U
]}£Ôey<¯waWO¯®Ô Kï’½Mëíá›Wx‹bVÁ«dÅ`ïü»ëûmS3‘*tý\
Æ{UÝ6:•ýð bòªveë}ï’À²÷â Bü˜6Ì«W|»g)ÂÁbp°Í«ß²
Úó‚½ËJ©:+â[-bÉDp¯ Ðd[š@b½t*»05¥P…`fžñ_5æh’AæöA”vrž½¸ñ&ç4/Zý¬Ñ“ ´ Açê›. NbtC–N«œ
4n|$œ5˜™=G¿ˆ”¥—o&ºQ#;4Çö0P™ˆ:R1«möj 'P+
5šÇêåQ’/+ÂS€R¾Ö§«%‡Å?
ѪI+®ùe‘mm|ôuvP E
Ô ìQ›'êª"P_ݪT5Oì~~Æmn°—ƒ™D’ú 0Ù:j˜~YMüH…A&JƒÀ‘üÝÖ††1ªOvq(¿Í¿)!6¢€\ÒzŠÄÉz·ßz–¤K^’oƒ4“Rí¾ÎÒ~Ð¥Ò?j U"‚¥^Ó¯¼¥ ÇR¤ ŠÌ#©Èýìe×PªcXêMF ‹I˜—÷/ÖÕ=‰ÕBÚòýS‰M¸ •Æ@[å|bu¥rC£a¤{.ϼ~p €j€ä7íÛ³P&g.ÖWƒ*eÍ»
qæÊŠ¼æÎ+ ËûGŠÅÑË ‚E³¿~ iBð tm4)ËÑ\;›?þ ,Ø°¯Eõ鶼¹tU%êÆﬡR8‹]ÿ¢ÆWº<«kÞ|±O½=ù]Áî~zëZjº¢] ¾ÄëÇù‘Áøæ¾]¤b¢ñ >ÓúÕÄÅѽ¿SÓvï¶ÄX€XÄÉwAÈ€œ¬A¿U¶¶>@_Þ8£mÙ_< µŠ 4UXk»ädr(Mu.P¹š*0ÑàJ
¨¢…ÛïP´«gÙe¾µ ¡8' Nv ‚ñäHG7ÈØA ‡V4“êq†¸°n¡šÛ‘H¸±Øô"“ÑúuföF²b—»šÉgc£ 3t‚õº 0òØÞPÇΚ÷x ¿°4(VEh :É=šÕÙ&%",ÀüZ
Ù1ÌPØs-(+^¨B2ö y$š”Fv‘íl·ójVDÞP lg6ïeYuÚäà®èG"b6ÁÜÌ€$4N?kÅÞwüøx6P]Qü²'¡êK+o×}S2\êŒÔlë’8±4ˆ‚ìP¢u*î†HsP¬y
źkUË9ÓV¢yîý´<ªîÛ´þðY,þ)¿xÜ-Οsc5ßiøòÞÜMáYØÁÎ;LÁ&ƒ‚RC°ÝpY‘Ñ{¥m.(3ûÅ¥d:„Ͷ¿¿Ø]BiÈ5¼»&t3OD×›îÒ Ð>vþ’hw TmÕš.*Æl„bŒ¯¿-ßØë‘9Êú½î¬“õw]´.æ^¬QÎ8 ó!#á3uy®ªÀ‰PÚN×z
dµO¾/rio^ÝH…â”*F³+‚ aÛ}—_Ž4ÁæÃÌ`?Õˆ^üÄSbŸüó®W¥‡Ï¤ÏÎâ g¾[ÁŒšuö»BóšÊE ˜&ñc¥&¢Ç-´©qÀLÙàIBb2ZŸõî݃»›ýTs³V’X[Ò…R¼Øý€P«›ß8€râT™p2áp¶/Õ=û«[Š§ŽÙ¦I ðÔrsb)iDV1.«]Ó¾ãøù%ŒºoòúN` Ì¢ˆª «I$¤o·,u”³ …Ësç†wÂL œ£˜ÃB:Iø©…¢™eAjX:>©ÔîO³´QG.»¢íZED|™]GŠ¸ .€ð"ˆ”‰º8•œ©«‡?§¶ zõ]5Þüa¯ÁöæøÙm£Õ1‹S
03Ì[b»õF##=დwBÇ´uSÕõ<‹¿Ï–ÔÅŽÔy ¼Zi“\žeSµBA0€ÆÛçëýiÖ] \c¸¡Ñ¹ÅCX„ìôPM&Ü ¶N'¶V »òú±²Á(uä[—ˆ5óæ>Ìx:4κ»³ùB)9ö´:D€s« a& Ø *ù&Ž=3 D®µî®j!úá¿”<Êk"žbõnÈRˆžQ D ÛñŽ±€ÝLCp<ò|h’íÚõˬÔt¾GלüŠéƒ˜r/˽ip†Õü½X8if¸¸Ä¤¹ŸOAÉÕ|`q`Ë’ý/{â3:GÙáSB7k_ò5B0fª“ißN!þGJU¹–j…41BÐhôéÊæøj»õdZ”ÖF: À¸€Ú2B
|µÍoJûqë‰%†?Þ%{½oö@‚©˜Ë{þÊÛàÄ»/TÚº°{ Ý[«wßyÂg£¸Tt4Ž¾\ÕYò§¬Ãk}à¶ó=ñÛ³?qÿG²ù9ô„ Þ ˆ½Y+ûEYíJÏ~£ ®Ô\,ýö%Êæúó9û¦W`–!$ S… w‚1×Wå4\ Wé‘‘<’ˆ1—Ø}>ÚÓ¦oaChV*Dçë ž\x†êï Nßü ˜âáû’énkvMŽ¾4Nt÷{tÅÈk×FÍ¢ìM5€Å #5…Ú§L±p y¤‘¨Ndl ˆ¾^‰†YèøM±éß“÷N6]uAä þU‹¿p|†NÈ”K…G’œ‘ €÷dw$4w `e|æG éÅQ6]í-õ·••ðX 2ìý®$m±RK´ü_!ChØ8Û9f]& v*šÚu1sczá¦ø†¬+vÙŸD%á ñ ’´7Œ5ÅÒërÞk¥>/﩯ÈGÖFF»
p
HX)dûc´J :[X¢ØvéÛåsuÐÑbâ aBâ¦Y`<áJ”ÿ²{¦†à\nBuCãÍú
ûñQ/¯S±ç§kì8¾yC&Ùÿ ‚Öf¶Q³ç C3¦²Â*±ê¹>U|Fô±é +$v
cÄ›e×~Å‚¬¯\ŠäU¬xZ€S÷V´ã»øÇû:€Î—a‘(
ÛyÎëŠ/;gŨ¦ ½Ù2¹Í~âµ8F'ê[z[ŸJ°bcwæTha6=¥
Ï4N}Qr°ZŠzoSŽÆ”Aîˆ0zŒ2›Œçº6ªZAå Br.›8e¿ñK÷ ø¡ym¾Aý&UàUôLUR)V Ë⸙ﾣéNšPhßÁÊ/Z9L"6x•þú<÷¬ú€;Ц}P( N‰Ž 4*+°HÐI§Ñ”ì95¨œÉ<¹2§]“R 8Ÿ¢×ra/VA¤üÓYÛ‹ËŸW2„ÑÑÌzeægiD £‹ÂܗסB¿º°rTiB¸ nt©ËòH…8«ñøˆ9% †Ç‹ ÁÎÀ†7@, :3¶ZÃ*¾º føgc‹ª§‰êÆÅ'…,I¸74’¨(,D‰}*,{GjN&£Ûª
LnÓ]—eÅØqTå4V¿ ˆÉj¿iÙg ÁvÁM³¹ c7} ɆeÑ‘'qƒ¶¬…£ÏpÔiUŒB N‘ÁëìTöj_É|]óFÑÞ½’`ãÊ'/D*¸nx:"É£ …oýß¿óWvƒ 1?[®] ðD"îkB î–7ì†,•zª»nq{[ü7FB¸æ‹ èo-°hŒE3‚=ˆFâqp_ߪɢ3ÏUq-<Ççn{æì&œºûxzÃZÅ¿îÔÚèÇ–=¨uèÁK UQ7 wbRYä´Qn 9«ì5ž6*ÊÉÃÍD:³ø3?p«G¦‚Ñ&˜¹ì”è²ÄÔB‰~‚Š–ú3Q(ýÐ#¼Ï ;@sÍœ^X¤±›Jµ}×Æ tߘ Wå'a.¸ C¯ ðfؽ®"5
Dðôµ¶B"*•"ÞRúDŽ—Wj¯9AA
NbC}Ó’xÌîAtöãCÐ9`ñüÒWotÛ²÷îPêçzˆ¥Ò#±v`KdÓ xVáò{B9Î=V¡ÂZ©âó¡ßã!ãH—+ßoÙµÿГ¨›ÓÌ.›ŸU*%WEœÉ²B P_‡QÛL™6µ{ŽÆñë –ee\$ÆNÄVTNc–»²Ñ”À¦þ…blüðЛä9äMú¢ò´ÝVZ.ÉúÇSAAsu¡yMi%h|}ú×
WQ–|AýIù”ùÙ},á×Á0·tÁÒå a©S‘UsUšîÁ”Ìæú;/‘` µš Õh1U&m3Þuëh?«ÁãÎæ%Qíšè„ÍJ'&Íú7µ˜Y‘DÆÎŽ(Ï
9H‡áLз¡©ÞȯÝDç‘CT!WY‹Ý1¥'bœä¿Cï,Gæ `@‚¥A¶LJ±æx#p‰zѤ„=JÍ”Çn‰ˆé4ÂÂ&ú‘ßæƒ0–¦É„cH Þ`¼!ÚàOiw.&OÌvcm!Œ- yH Í› ’ Š¦þVrLDH(O‘Á>›‰JíD¹y,Ïk°À>¾÷LM,#/ËD~LRÌ?8ÑÁ)mÔ*ê4}ÏIzE€À>Ì—5Î ý ÎOŒ`I•³}´ k”v«B¶aŽ83Ý
?q·xÌéO5l”y7'ÖD«V¡CžÀ´òy’ÀÈ—¡véÃÜg‚¬}:1qÉ ÃrhQV8›?"òwמ;–l+hUغ#š¹âůcÝÌJÿíÀ`=¥¶ûôI7PŒ‡ ·4`o¹¼°I¢¤†e [ˆ»k³=`íh¤”@¼œA‘ kFåO5Àm˜ ß~ çA~-U?/ЈɖިˆÐ7“ÑD·vaiÞð‹V©ÓnˆÐòÅ=סvÎÁx ¤šî¯ž&ôE~h9eAt |;ÚÖƒ‚属a×>QÍWý®]ä‚y~&©Š ܇ŽiþâýQ[—ÁZ‰ßˆ¼4ñb}ªY-!„æš¼Q)ëÐ µÀ`ª314ƒ®)ºÃ&›“å A:/EZ$Î t
VX1Bì¨q…RÎ4½/» —‹‡n$jkŠÁwµ?ž|¿ ˜Icõ=S[=q{A:4(õ3&®ï‚… œš¼d“=Ô 2²Nei½¨Fš‘Y횺èèÝ*äÊE‘'J6>$ª¶.&le?FñÎáŒÐ|o]x)ÅlVÖ$ƒ@w ¼Q§Xö6 žÕ®ðÄŠ'šUšöÌœo‡}„c9ûÒÁ¥~YÔ$MóÜ3/øËo³÷u3xï²Üq8ÚÑcJí#Ù.Õ *Ú®³*1чA£ .Ûk.EÜní¸î?Û?¶íèô)¹$8yÖúXE¯Þ_ÿÛUûbƒÑe‚~mûç±þž±Nî.s2)€¡†
8¿½ÿÒ$ìÑc^C«ÆÚ»™z÷½]Îû¥hQh
ÿÐóþ8e›yARò÷Qœ»JËͤaØ èvR¾õ;ÜðNDhò¾ûäËKÉ+cgDÞvaüÄo)ÙÛl‚ê5ÅÌ¥ÖA´"Æh¼+0g¶a¼ùF f½¢ö5Þ¦ÍÍàÇ*MHlé÷œ*\Rv‘58 /xµW*y÷ÓU±àdû5×vÐ-eI —*ÐÙk®„Ë?<'?3ã£LíîêƒïÈJ‹Ëç–E3·é›w»÷ÐÊã¡sb4Ô1—>]§ÁÀhXªƒ1%Ê–fKƒ|™®ÍºÏýÁùz÷“W¥Bºa¹ðX»Î'íˆ2áö·#'ÖX`M“–{ÉtÞ¾èE9½W6`b•Ëlú©.=‘䃀';Œö4’ì÷"ûrZLûŽåÝé Z¿:z9YÆjo›Ò×Ú‡|FõÌàP«¸
# ¨ô5Ϫ‚dmðö…§©B½š `M3—[ÍìîjmG Èßp`+ÛV´<‹³,ÅÇsõîÜßÄïZËÈçŠÐ‰õÉ5Æ)Fšäø]ôIÐ%|DŒ¨1Ui¦ŸÐöz‘£BžF5seD€ BµÜQSÆžErcSÓ>ÔUría‡‡õî‹gz&ä–yX'·ä\Šµíß3 6€£ "Ш
¿,¶#ÔlDÖf °XaÀ±–3®w—ƒàv\0õ£•«çt!©ÆD`6~$Ë>|¤Èh ?´¯ú
é›f¥víÎå=n©Ý–hpQsI¥Þ_y¢²™°•“˲éQeõåÉ×"©*Eó|Û %÷r3öÁ•Éºµq³ÈßD›p7 šgË}®hìÓnÍ)„e¿vêc5ÉhXòQ· äi]áÜÍB‰
€n«@±Þ™ÂQN† ’ eXC¹7´ áªm V4Î ^úŽ=™
ô 1EÄå;ëÕÕ°.¶Æè2eCw‚Œþ°|˜‘´v-Ì:
k{.ƒÆÇoê Ñž³«ØJ[üvVò;°çõÏz[a¢íUy—¸Q0ØAc ZŒŸ¿ÎÛÏÎj¢LÃ12¨‡ü‰,Í”ç–¼Ã¬ß ùWØž&ð¸Lwçžh7È™*ØΊédð×À@¤s«Fdm»
àÅïýÛRÄ3â]žwmâU#j!—¨íã‹OÑ„š±d¶»(T"¼ÊyÚ”kAti¤âß» ÁðÔù‚Ø/Ò,rº>•ón[©™L |¢¹Ìò1H¶1É`Y^v¥Ö„Rý2€˜én©…(û³
‘r^\ ë'0þ×ÜÓëغÁ½Á¹#*Ôà/Mzq:¡‘‰Ñçf`¢ ~C“*¹‡õæ/ý+Uùpûòö¸>……ð5[±ú%ßOQ,¼AF½,ù0Rœ ~
q°‹{ju
7‚ò¶õÏ ‰íʼnt ßÖ<–¶ÜùàVþŸéq˜ùQyL§>#mZ-zrIEè–í25E¨óŒ†K¤ZT»Djq¤n£Eç}¤^hü…g'Ÿì"“,_, ¾±–Ú$eLö©wªÕÿ¶T¨ƒrw3K9i;o°
§ š*2äç ·¶üqÆíùŠ–w|Çß0Š~¯
!€¢¢1 ¨^‚vÒaMô]{„x‚ rh`ejd ]˜Oæ"‘Gh–o‹F¶,ÉF•ºÀÐËq9 ‚JX¹Œ"£†•Âÿ¾!ðgÙ–^ÍÍ0ˆXé»ôæßkUëŠ*Œ‘gEÃs™åÂ.Hw!$ªñý?a!:ëì d9 6J5;ÕwíçËÂáL”Á90q"Ss£tW³úÂó:ð(l?{ p (ªú)ÇR…Ú…õ¦ /“žÈi:î,£NP[ Ö=“¾ÆwËG·>F‰^üâ.T†#©AˆZFO•µ¸Tla&~ Ç[tµËŽ$miÞK ߦg PˆF¨™^Q .Ú
É*¬!ŽC¨ù%#²'#u*„Ë'†6*:òÀ½Å9G 2¯«êš75£=„è
inœÕèZÆ_&Ô¶«W4F0*òõ1ƒ3ºÓ›™Î¥ Ѳ¢çŒy n¸ntû®F
â:ú î ¶t©êyüº¼6–±?LÈó } ª¯PCï*Dʯð´D‰·ª‚,+wÆ‚ËÔnú¿iÿP#?ÑÈ
wæ» b«iÊMx¾q¶våÛÖÕ —JÌ›²»lãá ¼
ÍÌŒ jqˆÇ±ûЄÉV•ÎÈ|ûb›µ £ü¼G9ÎÔ¼ÃÂþÖÛ~ëcŒT’¢‘Ã:˜4,»Ç?ÖØiF P &†o5\Ëæ1‚NÂ45bÞëþn½æeÁ.¼î¢¼ùãÏB\…Ì°|«35Ôb§8–‘ñf¢ ºìÚw_¶% 4ñ
|jz'¢D€°ýzu"W4ÿ#ve{FÙ¥./^á%{›Ë”ØÏ_Ë_úè·ÂÜÎôdÄéÁåƒÇ–Y¶–•\šÝ, îQ7ƒhêÖ‚ÈJŽCáÁxºÊ˜’g-rÃœ Méb秊ßk°˜ý
u[xÛ¹³gTıÕ!O½U„{aÓÞmAÞ‘_Í%ù™¼ý³Z'Á`]ŸIîœ×0Æ ïÕ}…ÊæèÓ• ’Lÿn5è a渲R¥ã!½öµÆ°5æëòp4£ 7ŽÙ(3
ORN‘&8;y7Ât’äóU”" U~õÌÓbs†A.äÍRËð¦‰"´ËvDjÈK?=¦ Ý;ßÖ»èò“ÿS \ò˜e_8G
®`ç쿧‡2a¡à¹·Ë"¼;t%ô 'O/füO¬ä} ôŠã] ¬k×=±Þʻݹ"RKØvñðFåâ\ƒ{W‹nwWÊUÏg I ,ú.Ÿz¥>yÎöwYÔ ÷h-è*6 ÿè X{!©bEø½E™“ãv´H¢™C`ù E”¤\³;œœ`?¬xR钯ׅ È îŒ®Š—eƒ¡m÷Ô_Äáæè¼5– g£YTk…“äjc ÂP¶B¬Þ†«‰@^ñ/Lx hW|ÛU™dQíBœ†Fé+Ìl°6âÐ,êCRE#á%H¹ï
/msÓË6)-xSåþ¦„ô‰×ßävãÑÞ:F?¶Ró®º#ºÓ «í5Y“
‘#%¾\ö2d».E®P7οˆU3 DzÛ|owv ½<øÁ“hƒÖº¦¹yqÕç]ñµ"ÕŒO‰c°Z[K,ýh<G îÿ‹ A;ZŠ±}]ò¶u°õ˜+GÚ¹¯zÍÌ¡¬
ÜvšúeÊ‘dZ&Fé±Ù#aë[vÑLµNóúìÔv|ñü‰µ* [öÝûÚu’hÍ}/Cé·2Q€#Ä…•"Ðg=R,Â)¸„ÔÝ«î8 <!f7÷íl ûõ£Î§k«•pAÃød.×0œØoþÕ:-Œ’Õ¶&ÉÙÒ: UÅåÀ
käïãðŒoIÌÒƒuéEájÙ
;Dq=w·ãGÄqâöÊï ©jÖ^ƒQ~ï[uó¢h.]
L ˜…qükãnW4ÆÃz3E6°Þßb¤šè¶½‰fвAcþ£š€%‡w„Q›ÆM…¸rë²"=…´Á‡tš¢¿² D¯ä5.ˆT½.waBHïÍxú©ØSöÍY^ Ø£–x1ö UlÏÖndcTN+ŸóëB‡63Ï!Óqk±¶ÇB¥Î5‚=ŠßãN ×kÞ‚‹$Ã1 >=õ2Ïþ „I‚€N`ªRG\¾ \ ø‘ybgU¢xw¯äh£ê¥8AœSÏꃇWƒáÐö®IwTÛœùÁio òè‹£½Z'Ÿ¡è°Æ) ê6‚®o=Ê€ï÷„Î*–V.“»õäqg¢{Y €„×~ÖvØñ/Vz¤ä™±X®OU)2<œÊôïÛ/u8X¨x€
ÕÒ@Ƈƒ¼9GõŸ½!Ò“†µkÔ8u7-êѼÉrtÃEž’\6å´¿áO±;À€Êƒ`<Í0œhñW[D»}¤ÿnÄŠnPpè$Ë(HsÉ–BÑ¥÷¥p?óuú¿þ¦£ÙXQ¤ VZZ ›jzSe¶®®Ò74Éxßy‚k©ÀB1aç
Oý ѧk¢ö |ÞBíN3 Ùüòæ–I£‡"ëÞ:V—¸ßåZ&±†ä ‡‹˜ˆN¡|Iç®{òbSŠBTŒ‰‹ë¡ð‚¢åè¼H¸Ë¡Â,QD“:rKj‰ñ•àÖØ`¶ýi°3z&š2F™ì5)ÕeÄn yízŠhÆDè‚utÉ¢Ç1UR©ýÓE,LÒVLJŽ::NØ…sh ^´`$ìi`œàÙœÌHoÌêù)åçŸÿaòc3Ûob‡3ƒÅ99P „ÞèÎÿ«-=R•ÇÜ¥¯K{GâS÷çž×@0Ê°Qª~Q v¡Ã*IŸ_$ 4$…9ðy“Ö¾Ã[¢"gQäÒ$Ðõµ¿/ß7EâñxÃÒhÊD‘iK¢è.5’Y14@CºèýžÊŒ^ü-‘L Å¿ÄO) â½`a nÔ(ïrÆÔðf¥Y]Š0$—Æd²´S"â´`ZZ`™"
ÎuÉù†Ñ;ôôtÕK¥Æñj½T¡JEÐ0N‰w‚EfaÉ¢wOD\ó#T×ôЬÞ
$1Ž¯üõúPOcY% îjñô@ƒCÁX_A®Ù¤ÿZ•Øƒ×e]MúëP¾ÆX»Ú¨fÃׄå$°HÅDCrÐýËZºE¨ÎŠ03“ƒ®` [ö5Q;Ž0GHi´ØYl¸œâ‚”6¨~SŽ%]-rÀ ’|[µ£ÂR*$&õ¢¦ÂÌub…:‘ÝÔlíךÞLaɱÓá»5''7ZÙñoõyTwÙZ[Gc i¦–I"} dÒc=”ækÆ…7¶9Ö%î>¯¨f33 TÓ´¥‡O%àÖ¨Y`xë
¹–ÒûHñE?ðç¦è%öw Þâ¬?]ÕµEñ ÚxQ=.£ù£®RnКˆìvd9‡Yæ/t#;×oñÞQ$¶B»
²ÇÈí?³¹c?û&
êôV¹Zæ@¨%OeÝÚf }·$ÏaOrû®—-Sˆõ"hY$Ÿ ãÝwnÏû,)˜Ì!èé¹?,ë M`d«ðž(‡k BèŽðz|I±Ê1}%Ÿj¯[Òʶ¼B…^wmw盼
K¦¹öB‘vØ>(Á¤4d[^¡BÖ¶ƒÆaÔ@,ò7oÞä/äg]7]»ñE4é*M¤á¬çʹlóÏÁà hõ9\.?÷öΥ¼<<Þ½¥âÓ±MÁJoƒŸrp“ *E±àùXªj#«;¡è}Õ%(†jÆ+,Î>d« Ìó7ÜþxEõ0jtÎ
(¾zø°v=ä„‘éSÉŠÜšlµäÀ3(`8RËò)ëÆh7 Mþy&ª‘ ÊS$¡¸ \8mkPv«•SÚÖ6¹þʉ!›½JE£yE¾§Š¦a?»+&Ï©I=‰q÷g ¼
úUAÄD8¾c‡y÷ÉYí"R<íÉ]7ìÈö}*O©ú-•y¦v¸qtîï‡LEØàóku!ÐI˜9€\í²Òý\pf5ZfRO½}u#ºˆ÷Kßi2îú'QC,—}²ï\-ž™[^™÷²´Ë†©5Q §2!ÁˆÙJгÜåˆ2ƒæ»+CZö=‹G/E˲h9p<#˜¼®ïsbJ œ\¹4µéŠ÷^‰œ jµÑ:ô5¥ýÈÕÿlì&GbÑÓð–èö’z®É÷Ä[ Z,m‹¾¤Nw=˜27 6({ó05 ¯Ý@UkB6É·²¢ †Ér± ¥uû}mù¶hT<˜Y*ñä¦,
Ëå4i‚¸”´UDã1ƒxÌBJù¨ˆP訅KË‹ö¯Šž©†)ûñï-Ù6VÄørp°Î‡éÛ÷}Ïzp´Ë˜_ÙmËÃ&´xÐÔÛ‹ßsSeg›·:¬Ò¨¼_WÀYe_‚Xí¶S倶RvJ‰?YU#'ƒJÁV#>ˆö®8oiúX¤,#Kéýóö wv˜C¥^äuJëuY_Ÿ÷Êþ,6Äaæ
¿ ˆ‹g¢ ð|b,%£È.ùÎêËóšÉƒy§>A_™#qúé6.œ-¼ãëÉžl/“Q4•IV¥‰™ô£1™¥r¾à.YË/ê ÞN}×Uµ¬N¬¤×X|«EGa[)I¹–-lº5Q²„ ª3ÍÐà*ƒ¼6›[3`Cv¾öÉ5‰¼4s󘨑ù«Ì¹«rÃ*wi5«êa0Ɖ£–µÑh³Tœ¤Ò(ÅŒÃS…½‚¸®´{Û"Ë"ši½9ËÊWfâh ܹcFãì!!J%ê’Î*çªa0x¯OîË» n?Ú3À(rVÖgè&ãxµ“=¯`€‚&šÖîqúð£
䯈4{/tÀÖh± §'j uXœÝÖ>Û¤ºêWÆQ© hlÍ €$®]^[6: ,9 Äé² ©ÓŒÌôÉ¢÷½&ìñ©T®WºÖ
d Œ Ž‰oÙ-ýASc_CG½ n
r®Êˆþd¥>)EÕ•Aùvöʲ¥‘sv wÛ0½5ãÍ÷ªa]
‚¦„ZS‘÷ îO«#md,Ã
Šzà¢01 b‚ è!$Y]TϨ¿gYï¥È6¾ÌˆØÔžÉ ˜™½˜nàg»Á²Mr20ÝÀ†*#
„„ÅσJ¨¸Xßll,ϼ%Ã&€å²CÃjíft Lc0ÐÉ%¶«¹Ã>²Ä½BP˜sâê‘Z±
:—ˆ2æáÜijƒzoTÅñ_GðH‘òÖ©&kK¿ÃP Lz‰‘„ËX «ÚõÅN}.€ü}EíÜ^žT…9…šzÔ Ve-¾üÞO÷ª
pbÆÜ[}¨¤hjŸtÖþÒÀ—æ3ñV<È÷÷ོ×}j¬ê»¢—^ï#¦ö^0*PÕÉ@„*}¯ëËo!QÜy}ZÄ/5Å‚ƒ‡¬År´~êªâ×dÊÝß²Ëò˜£–T!çª žÜôV¬Š£×(ToGôñK†!7“ŸBØnØTª¡HŠ‰¿¢&åÜ?hÜNNĉõ©Á³Fz|ÏÃ`t+â%R!ïGQþeTÛ÷ÜÓ–A ªE
e!K ‰á;²vÒJ×ÿK¸qr¿áB±:ö>ë„0óçߢ÷r‡#l×I}[!ƒÄ{£„Xf Äý]ÏŠ8´Jý0‰yÌÙg}ßT}Ÿß®ð°ÛE˜™<õÔR†}á^ÂçÔ=ç4j6(lmøUßC˜ Á ÞE „eàp·TÈ9L\5sxÀ¼Ùlí²&¢òDN”Õ{‰j#ýR`¶‹ ‡ÙÖZº™dâeÿãrº7
\j£ãmämgQ¶ßV'Qí›6Àå/_}•s±ùž¸Ä&v ´Q,=õ¾ÇòÐæhI'u*TÙ¸(ÄãÅWÊtÞ‰F4¾{ÔƒØ1iqI¶3Îýêµù©Rò´3²¦«¯Ï®‹Rê‚Ø-
NÆâ’FßP¾ÜVw5·§ ×ó(’6B•a_
%Y !s¯Q”Ñ%Í Z £ÊÃÉ4q9lŠeÇØ.©0úByïžÂ²{'¯2øvfïfÑ&¹·u¦;!ôû{áƒ59Tú<É£œSœ½ñEc>Á²ôLãVt”»Þ¿}Ÿi„‹>J¬1X£D!EªG"(¢@.Jù.hï÷•:¨¶¹©|F}vPýàîÝ\„áw R3ÐÜm³|Sı ¨hÄà }¨ñ3PÐ$ƒè1Js©Œ¾é€f¬N™Ð¼9Êíëš\ôvŠ£Øèïö›yŒ9ÖìÍ6ÞµÛxÿ¹¼"ŒË*[ögÞäÅù·ëõ©
²¡{ ÙbæEè'ÉløcÑ8#ún:\ðň¯ñõK»˜Ù–`Á ˆm¬þiwj-©°Qç>{§}ȼ$©ß?kÄ·(Œ)œ±°ô˜ Pi4ªWH×sâ*ÕÞ€¯ªÛjKuÂg¾UÚ=´—‰Í—+žçzˆ]qU(7\ÞÅ”@Û}ÒŠàgúˆZ¦•ù#ò&l«v‘%ÿLcKTEøQƒò„iä‹ú¿ª ÆÍ#‘ª¦Å0lƒ39©•W«è)<°“!Ðc ÄÂ. ýLvÍüÖ¸Œâ\ÕF©Ù;p>]ñm›lª#›ÚuÜ8QÇØŸ©ƒ¦3ê ¦ªQ‡µÏEuÓa±÷oêSeycƒPmÎlìEÿV$¾òÙ=a¸Ñy/?Okƒ‘S”íÖ¶0F®í¿†úûV|8熚*tšd· »AFgÍÊõ:(9ºr ‡±ÐLÛÝAh|Ð7vyª”0X½I¬º\£H Ýw¬-^ƒê`ŒUöB´~è3.¯©cØï¨~âÞ<éHû¦8Tc‡gðHCOLì/Ôá®
ÄT1Ç'Ü«o˜¦óâYŸ#ë¿PNÀ?h¸çئPÕ D
½OÅšà 5à:o•ÒñÔ@$‰p¨ƒ–;¢1{‡'êS¥k4¬1íÒO™éhãE¹šljo#¬ÜÒŸ²‡]ùLˆ›-ª &‘AÝeö20':ïí¡óÇXAÿ#izÒ¾>]ÇÕ´³ Ô¿A‡0¬ÈƒhNÛHu®¢m, _žQ)àÍ1¸¤j£>5Ú“t[¼ÿ&¢…®°±;ö]úDlËǾ› ^}æçÕ‹öþ—Ñ
›¨…í’ÆÛ8\ÔÔGVÉÅÊ`¦„x —ã¬;›íÙ_;nd·Ž K¢cDdЖ/=A´ïvÈ bp¼8}h’
¤Õ¨§î¶I d·ÕZ.£ Í…„hÊ)45VK–’×H´Ý³éÚ¥a Å•iEnƒÀs“ü
K|0e·›AŒùZ
„@Qà_à ŽxÊqßñ{¡ÚÝ‹«þEßqþáIíGªôb.w\\ûµõÝᬘI²£K:Á@Mü}ï–õlØC³_óóû Š>TŒ:Þ^Qnß³öá bO2¼›Y². $R‹E‰¹è¸® pý¸cªÇ7Ï\]ôºñ^8ÿéxÐ ×5UýS<½|Ýu3|ZPà; æõÝ n_`*µìÉoümC’5á{× >Yúâ õâ¶'J¡%äûÀÂr«êŽ_h´²öO¸WIÓdÏ…qº øÔ¢ MÅ)sÁ¥m%!öøb‘HV »»Ð3„ #݃+ŒHM‘Ϫ±ÃTY‘Âo0¾íéÝ;%Ÿø™txÿ_8¢×©UßÁè €p-‹ƒÀšHk? íM€]±‹ f–ún¶ï¬;ù—™ýª4¶æiqî—bÌX &:SÒeÛ8,MÕÄéÉ¢.xÊü¿ò ãÁ¸_öâõ¾=©G+rV4ÔÆ ñCƒömQ ékA9ÀÿŽc}:+÷®ËkCÙÄAè.ʯ©^)õæÂqÖÚîša_ÏôT.T§?j Ngÿf`£áó¦*;+qhƒï`§ñýù>zDoâúlË -0U¡ÎÈhþÝhs¡ð~ñg¼ÉWåO+ì%¬W‹÷cÒe¥0xÉ<Ïöeí†RZ#$BYpÌ ¨óÒzQ$lPVšó– ,éì]ö`âðF ¦‹Ê÷ÝÀ{
3¯D²sÈ™U$,ífKÞR…lþ³ˆÅÜßA§A Ö>š?_±bd§Ä×J7†æÏ]ÙB&»ôÚE÷¶¬ ²˜éåŒêôvùØ=íÆJv&—~QðBM¸‰ÿ÷99y°çw§bÒ?ÛÌÕ¢à‰Â€èŽ@ÿ75cÛZ™Fâ°-ú’
Þœ¸«’ÁÂçèhæ *"h éEgsl¾MÛ¼‡ZCæÉ 2”6}íFº°¢f¬}ÇN”ŠŒ4±ùÝ6‡§#¤éØñ, ²ééë&*Ó1Ç>èÃzA¹kø!¾o¤Ò]‰eSéh¸Ù‡c~yöýÓ_v
G"ð«*™cE8ø
xäÞbê¢Gnê0\—#ÃMCú‘H¬ÂÞ"4_jä¡’-²/Ò=÷ìgCí:Ÿ°?BuhŸŒ›š¹çhŽã;¹à3€Š/^‰‰UKWÐK«V’ÓÔ%¶q«p’n~) UK»‘µ'Þ_ßWp² =âw$¤þDß&OðlJÝäR¾óµAcÇ‹Øšnp1$)Âär“°¨’7Ìpg-|ë×Zr 5ÅRP‘¹„N["ÔÎ ë„L34 N«£Á¶ªš¸Ï•'w„Žm¶,Šÿù§•K•:*U§N¸®‹)ÇhAXÇiÐ>‚“l@¢´K˜;Á3ŠË1›ÖBŽ‘,±·Þtåâ;ø1nQˆJî}(ÿx¼†uqÕèô'¤AꊌͳnŠ¦ ¥G°z È}º°Q»pÆù+‡³jfꯞ Ó +ùµE
ÌhŽÀUòš7º^S³ÉŸy%›lU¶K„ÕAˆªwRamKÑEg ?"
:€ º§Ò‰ûúá-eßöžOW¸¿ñFïý¾‘‰ïáì{ž»£FG‘þ›üËöç_t8ˆlžÐ×Á”H±¯›xûת´ŒŠêô¹:ãšç$Œy¬1ÀNm$´:è€jq Kf ‡4ï6!¨9;5o òXô2ûl—T~¼•ëq‘õq ñYû6Oßìj€k1ï* Ú!æõœ"«~W?
Ø
Û1ð}›rûê8g¯v8Y¯É ÂSŠL}|ùŠ2A²dî‡Úõ:u B{ÿ*”ÏåƒÕ:H¨©42Âô–\@ 1¡¾
Q¾$wv,‰Ì@ÀÝn„᩶ÚÿÙG<ÇVl½¹ç˜ñ#blÙŸÚÁ]"/wÅŒcïMÍý$M“r/ `Ã%ýÑS+"Vv®š»^ ³d´±Úð®ô&ÌzV¢ïåžUduþ¬ÈËXß”Ýþ]_]/MüŠ’9Y•›ÛÐ X"NǧUÙ.ʃ„ëozÕÛtoô@ͺ‹ßÕnÆ (†SJÌ
Bw•Î
è▀Ͻ´ÎÅíÊe&ÇÄÍEõ;8dï nÅuAÒdtQU•Êñ !âzóþ£u?íËóèÊ uÐRdîHdO¤xKÀÌ7‚=‡ª[i±9™s¸¤t£ ‰œiÑýa(žØõBøVç¹Æ ¿½bèÛ«_Rµº\Æ»i;$DÛg²[È+Ä 6Âß.#DÉ„#bKu j܃©ªŒåË'¨¼È×Íz~%èP‰pª0ž+ X
h½Ð!äèòà¸Ø|†æ‰'x†àÒÃ#5gÍÞ
çOT³«éïÀWt&¤ùÙèÓm²Z§v9Í+ìîà¼ÉÕëŒ=(Î.‚æ ƒ36Ì
£Ëg…Çfef7àËvE“¥4Ð×i’«'Õ¾»+o«÷Î~Š¼RÊ û…JÕì Ha¨È˜²jâo¹¯ˆmHëU•6¥öÆp ú«ñ
{é·;8»²ˆC¤cW ¾Ó .Àšd±G8f)j¢J\Ó§Ûe‹'tw•Õ{9U›êÝv@환[¥Yêcö Ê\ج†æò‰4´9dlóËšŽ‡óï2{Íž*âH>ì˜ ª§Piò3—,@£Wòœ)Ñ|ûUq>$¤)&Ý·ÞŒâŽ!Åçn
GËÝÔ"(Õ/âX`WÔôXŽ½GŠø-q¬øNCpƒL²×I` ¶~õKvz
€53ͧ›¡nÕ»
©Ñ>¯Ê܈ 9Þª¸‹Ûµ šÆQÀáøiöMsëC%áÚä T8;qì€6ý{g“¾Üƒy‡¶
UÁ²Í©8ý$-GÛz°L"Ëi8º;{˜ •°§ø&Âa(‰ yy3ñ'ZF¨lêÁ
´h«@9ä«VàÉø°V8mËŽ[áô²$4ëjßÎQ¢¸{í¯dŠ,S[3á–úY†¼ˆnr¸Àk7“QA~¸¦ #ª(6IÅÜ¢4œSIÕœC%GUIö'›n´^d¶š—Õdß>ÖnVžÀC‡B2¨4(dhºhÉj¬ X `1S)v$å:Øš Ó Aþ.ÛÁdµ›,Ù¨¸t-RF1[(0Ô;Š7¯ÁH¦»® Õ‘TxÛµ³sB¬¦ùì÷Ÿ3Y2¹UkÁ€Ö*å°˜™Ö®²ƒ{< ÝØRíä©øK§™*Ón@_»ˆnƒp:£Ê÷¬E õ3t¶„µ¤ßilª¤Q°€ðvÆšiV„†@`ŽjDp
obÃ$AP‡‰Èx®ÐÊÓÏ…ºöº¾p^U»ZêcñjçÚ牎¹6Ñm{—i¤—ߢÌúd ze“îœýsí‹È
׆’ÁïþÚÜ;+VpiäôWý »÷|ýJM‚SßÈ~õ•8ÒtÇ"«9_øàÍå{‰ôh„휺Vk© ÆF'a²ut,–Sðæu;][ø¨Ò$pm'å(µé[èÊÖM÷D°8¢Ê5`#v"ìTù<—ç)ξ›ÿç_U8PR Œ ¶6å}Y—#ˆÒ…¸öý…Z1†øÙ¡5v Sg$5v¡ìæot$2DZSmÁ=c ô¡ioÏ
Fší'¢R§˜‹€ {Õôñ&F¢9pä¼{ ¤ EF3ø©I ŽíÍ‘`-½ "›PK]u Š™ŠÎÕMÎ=•{WíGÇVÌÇ:_¿©O
‚hŠ»ï÷Uþ^”¼u® Où=l! ú– /yÓ‡]Ùó³K‰¡Î¾º=%´6öõiu‚Ä&k×H_ÓÉ7á<›~ÁŸ)1[5ìÁqƒ,º”]ƒ%À$è%,üª¸ š>ÈeÎs$"Ö óXè\;V ¶´£‘èBã1™·Kãü‡SÒ¹s%Hب¶Iv9(É€zc3EzÞ•ú´Öê랉îˆÛµŽkÃ7 ª8a±UÝø!ƒ²_¤/¥o›®ë¤-öß,xc}ì7 cý…3w°Ú§¶Wsµf`À_³Y»áŠïÄ`n’¯Å)cçšê®(
òœ* Pî²ÅG°á¯/ižG·%Tݺ< XæÅ7‘êáü ¶ g€â„멽v©ôˆæÿª…÷ÝÙ‚™õƳË81Ìz°Ö7gÁ‚Ê*ÜîÔHÅJ9–%çy%‘ZÜ¡y°.ÄbÔ½wÑg
¢Ë_UfZ{*c¯SÑ1æc $Õnè|…¸jëð®xJü‘.~ úƒçò'Ñ»t¹ 5Þ{JŒ“N£©±!ØÓ
‹)?[?[ƒÈ¬¼GEW&¯|‹š= ƒNÈÆiëûßöãµÏƒ‡:ëçiÔ©ŠŽj¦¶ˆžŽ$p(ÉUSûALYûW ¯^#ü6‰§„gTûV†Ôk„-—ð-É%¢@mbp•gÐx…—qÖ…‹ëÿ\©æ†f:g ËuÌ áýÁ+œ.Ÿ[„½ýfµöÁŽì`ùz©,eèe¹¸i–_U ÅÒ㿶`)³dµÍ
È.råPŸ ò‚Ì LH¼lò›j •±š“öJ@ȧACYý:ñìê ÕÖ± MàÀyŸ
ºå"ì*LŸŒ D¡¦NÛ„ÎH_Ï% Š_€!º³"þ¹›S".^,É‘ð¯Tß´w{$¿ÂÕU²#»#—å'Žu£È÷ +XÛj˜>= Á¯
b¡c͉&¸ø™¿ŒçÜ>¸ Zþ¬ˆD«±LW a”'`ή²u‰g¼-'Çr@¦¬"èlîÂ/׈ûò/uo1CRza)ÑeuÁöœª/«øÊz»¬ª\ßÓ}%mv jìýx)Êì(ilŠ<°YkËb¤âͱ gð'Ûc±ˆTŠ5*)hVäæq˜@ßÕ4Šü‹ƒ|‘ŒPþúuQC*–—èù£Ù`ÅÎín¬¬GU¤—ƒs€•;¶Çv“ðñk…¢ppˆ1´«ÒãÊÄÁÄ5ý;¿uQ2-'‰²s™TGoˆ€SÙcÖÒ‰¨—5,íÿˆBm©eåRf;Yp£‚Œ"2ƒÚ²^rÛD( jmZ/r]Ñ܇dû΋lùѪûÝݳ/õ]íböM}j¦iäª*/0jåBWÕk+Å\‰»€%]æYX3rŽ)ôå,(ø
ÚŠ—iÕºLSý¦lÞ«”=$µZ¸ÙÈl‹š½ªf1F+äŒÈ+šMåÓBÞÆ5Aq£¢ Þ6À†×篌h$–HU¨¾+-ó×";[ Ê"ž
2À›·/Ÿ=¬ƒ°ò39kΔ´6›$Ø”r"#ZLENHŸÇc‡i±üxRí|5”O5®îúðk…dU(†vΨÀ ¦‘+`[Ш²–d$ŒNMQ]õ{Áu²i&{Ë9{áÏå_ 4?ø_äss3(ÁeêQáãnsÍ~sWÖG4B=EâÎÓ‚z?T/6RPWÞ»¥R9~zðn€7ÇWmy\û\=0“䦙¥ëŒßþþ Ûû<\z«Ñ†^ô¬lîÜ,©g£oÔMÇéu,J…Ø*ç\Ý™bNÌ|긶„\P¿‚~‘rôb;°?ð¦\ió™á TÌObƒ°ÑB,à¹^\é]Çn‹à?SªISûÛCU£ÅÅEñ„mÑCh/’‘wV®ÑKPÀÊåÚ¶Ÿ¾|× D€¿4ц ÉK¥J{µðHŸè2A±\žò-.Ô®P´ígÿxvï‘r×# `Œâ8:mÐî>œÁ<ø/'$X“ØöYʶÏýTØsPÙéNı+ïæŠ]Ïùë?dÝ‘¦=;ïï‘
™r$‚À{^ã´ÂI †¾8®] ý‰ýF/‰Ä˜aâ›ÐçÜø 0™QÛ£×*kùò@×
$@+µYJ樕ëÛ/“€êa,Šc×f
ûüäò3±š&;†.÷¹ñÁÄ
-ÐYhjkóe‡Þ?’£¶ ˜Í K¸/g³ÛÛ9´TËHóîC¼ƒ+ÛåÌ’¿T«/ü‘Bjjúê|9ìÚìS2?„À(N—r2Sˆp^óãªpmÉv”ι\VŽ•|ͺ¥}Š–³²%$}w+‘21Œ%æj7°Æ¼©î…ÑJU’Pæ{
åêÒð†¹W5ìð}ÖE`E ?=?{Ï«t¿@[,:Ázý|’O5FÑÖ×ö0fç/úuQzYG‹§Õç ‚À;WêèÅ"ØérðÊUaó±Ü¸qÕÓ9¸žylê*5è &/{GÕ3ç5M\ õë½móÐhÓ¦h,&R²ÒõÊ÷¿dY6ÝÆ Âjö\D`%8ì%¯™%¯²_/¼ÖÄ!³Ç s'7/Û[Á©Ê»?£w|ZöWf2:Q¹Ö#&Ž3 ¢ýþ¡v@3Å{¦Öq§Kh3zH¸ü†ÞÍÔZìÓ j
ó½?ˆtêUÇñB`ù¦
Q×”÷<1µÕMǾÈ"?aLd…QŠˆ‹OïìÝÐ2©½¢YQ»‡|¥rÕ¦é˜:®
o¤½¡øÕà‰‰5°*¢¢@x*Jƒ&ù—6Æø)
%b)È gÎ{wŸ¹/xå~õ®Ù¢ 8$v‹§\VͳYù[CÃ%ÑÔ>ß:aÆ”¬Æ+¿*ž‡döOû¸tÜ4f×Å—_Õ§RhGš|€ò_Ö±¦>òŠROàøxV˜›óÝ¡ìzÏM’rx9Hóë ‚”AP%V‰]i\Þ'Ó ڸÿðŸ5™¶] Û³œâh+Lò,XØÐÑÁùSŸ*¨o®™dƒÛ
?‰=*v±J^Ö×8¢¨• >¤†ïu`mgÎ ¼ÈšWÑ ™Q‡adOX˜Å®fä¹Ûgüe4"Jë°Ê¯È‰|ô}XŽoßž¹oouï‚ ÚÍòwµª°›³^ 2÷B0K. ÔÞ§+ÌŸô”ÝظbóÒùP¶pßÀ§%€Ú4ÔqœaÒ™g.ŽŽ+ÝÞ¨X¬]±ÝPiÏTè7Ñ%aïÉiÝ_S 5U¶£A‡ÍÛC{ÁkV`2‹õ¥w;5
IJCS»/
¢ôñ»ý þ©1“èk72•eñÊs‘¹üHÓÆ´fE6éVíZI4 S ¼Æä\öEüÂLëWû³´Dt»W å¢h– ó:Sãi‰U•ñm vŠ÷½‚4¨°3k¡cFÒm± ƒ‘C“=Å&—!+Î{²cÑšf/è ß»<Ö[1m°»¯‹>ý³@7!ɤì;l¡E«O½`·xîŠ÷ö ³0Ò¸£zTíh×|—g² + Þg3“i¬Ø{å+ž>%¯ª=4Vþ¾x yQ&3lIØE]Ê ¹ ëÇhµBÅLÐg+v?eºuº×^kÉúœÆMôOK\ƒ @ÇwÚ¢ÏuZD_8S®×.YmÂãÇå››.*¸Q»f@,:ÌëS•Ïc²nlìj0ÙÙÖÛ&¨l{œ×> !À“}…Œ¦] }C»‚¶Ôˆ½µ±Èîð–•9îi[~^^_÷xP«dê'úÓ†pÂÍe©°1šD~ÔGOìy]D³Ù¹¿Sl] \мü,œ9ù4§8 4!JÔØŽzx
Ç|·Ö ÷¡û½!\ ¡QM+IÙØ0 •åSµRV瑶ŒÊ졪Ԥ£«œÍ+¦ 'Ì‚‘{j|È™eß ý…•‰Ì£[ËŠï3ü›ZXöô@8
j íg„,mµôî{ɳ0šâƒÁ—˜">Øzï 4zB^ø~'¿fŠ8—ù ™RK‰ÇFc§[ϬŒó/ýFsAÙ¯¬Bß.7º{@CPZBɾeúsOt¯!›ÔðE°QùeGÄ_UÔ§ÞE¯¢¹ƒ o¥EÊD¸âìØST«þ9 ½¦ÛÑÓâ–d'Jfrõ½^å¼ZTjé¦Æ–èC÷ý¾è/¦Ùé—´»w~)½1ãr#FçNu4ÿ㻀ªçmeÂv[ZŽsïvhÀÖ*%
'äMwÄÎ#6„áDÈ óLa0Íwí{âç¸7lhz,¡7ˆä1-®©uGÝϨÖvÛS7:ûî÷¿)a3 ½ìmöHÁ^¶/ìĪdj±Qyâ5Šé³IÒj¨¸ÏÑ[³{h¿™Y’Ýí²-98_…õWcu6þ–m^œ½¿ óàní£,‘š£ÆL*~ÿeï£ f³‰’óçô’âÔÐRÊnÚÚ›² Ú»[ÝË8Z¾h½e¡µ6yôÑu;{,
„¹ÙHܱÉÀ”yÝx —‡ñ žrÜù]èˆýoÈ)w4o‘½Ífiíœí&ýq 3ÊîfØ'j}i¬¶·¢˜‡ü°v…¿±àƳFÃÁ¼ÉÛÃÎVÛXœ´qÖ&Ú¡œVkŽIqG#‡ºüŸ °FL¹´9U,"éýñÒjÂdC MvdÉ+ÄNoä;ƒ?QHâR®–u ´ØþTŸJÑòÝ
(
j]kh\ÿu;*e¿ ¬£$QäÝ}O¬ Î^6ÕÅ> ¹/z§·®YüÑ‹i–`rÚuySÈÒá£OZÒÌ*ÝóºyTì‡è¥h
Ú²³û<ȉgÌädžg¹ñQšúMèúK›Šä>¨`}
~Þhyl °SƨÃM÷F/”’˜ÍZX#½vnL…Ü™ážÂlaÞ „Y¡1”>sˆÃw!™AQèFQp'5ÀÙÙ
;@˜v´õ±õœ{ZH~eíJe`l´ÙÁ[zu”Ž•Ø¢Œ@8W bç„ñ¯³8ž Ê¥D¬Êy+4µ>pP(ž5¢.X½$¦zÓ}’¥³b‡”½"‡[?Jë"hÚ?²Œr¼~Î^óëígª€`ÐSÜÝYz¡ñ¦ÊjÈŒÝ0^cY
-!*ô²rJ±©eØI™9¡º³´ ò ‡4Rï´±ç]w’ˇÍÙ™ðÎácè1öTi‘>¾j>ÙU3®øþç]Yˆ7CŽÑ#JÔGUÒœÒ@'2瀱‰ƒÕ‰Ö<Õä À†-Iõ›ç30Ãáç3r¤C €8‰Ç ßE™9¦‘É°‘Š°4ð“ºh“wä™>ËR÷!R0«êWíΰï8rä8ÄÙë3øÑY‘ÁÄI…‚ñBŒßNÕ®HÉÅäìŸáÚŽÚúþ#§6ÈH㋧¥Fª= —8áò^„a©Š‡õäE /`ZA© Hˆèƒ '³`·LoU¶°p_~tIVLšªüÊ»´dç‡ÅÚwbÖ3Õ袇’~ñ´Î^ÓÙ%<â‘šð|Û\N–¿/^
rêh‚Tk7¤Ò7‰daUsU08Æöm<$U+¯ºzøTd2iGÓmú@÷ÏÔ§€c67ÈQ}§-²ÿa ºÙÙ´^òz‡Góg_ÚÍÕœ)æu)Ob+ÀŠNfh7ìÓi¼×<¿íMœí¹
?-rï—<€±ñች˾Fž¨ŒÔ©páüü†øÓtñj’¦»ôãŽê¦îW`\©SÙE«‡Só¦™ùF¡wõ—E´ ‰¯fLhÑÅÛÁ4L9},´áG‰fÅ-OØ AÏá™}ÒUˆMWÉܼB.G<ã˜I=]Û‹ÓVâ XÄŠ=ŒW·õé^‡š/VPV²½ª½2ŽœL»-È.t©Ú:Æ[áD3Õ.‹£™úÔR¦Ñx †ÈÛ.f›ê‹®ÎÚ7¬Œá˜øOèÔ ]¼}ò\OY¹ÖöN`bX3öŽŸ¥£_÷jcDWJ´¿Œ“ýçl± ÐÒhI£“†:{ÝKï¯ FâS §±kÙUM!±xÞÙw,}kê ÷Ó±§#ÅvRKÛ…H,£Î / ÓB¢ÛìºÂF ‹,ô+MíF'5ìî4[›êoT¼ÏæýkJD±Žˆ¡sq!½ŠRíoøN" 1‚ ‰Ërv¥ž
kvú¬Sv›fðBMf£„zi¤ô ¹jŸjZ÷ D¼˜À‚%TÐÔÚ„ãF}ªy”Ex8—FÓƒÜ ª^uÏx/
^üÄÆ.˜´ ’`ö%ÊôÆð<ä03Õ^ójìèÂppBc0X)¹úMhÓ
ÿÃÆMvbÿã÷@r„3÷»ÚüÆJ~‘Ê•;8Ì“€ŒâÓ#sÎ;Ö͆Pblhklj2ÿ![×· ¢q&£l$Ôxä3²JwÑD’–3²Ÿ,[ŸÞâq’pUKo!š1ŬW6´w€¾vƒSÿ
"Ê£2öáÉ2B*3Æ£7Èiq6ú@×3ÿ,ñ_uÉ;9"ÎäNB?]úy º:-…Ýüƒ'²MÚ e{Zívês`¦¬"% Ï‹ ²’EeŒªtA\‚"6®Ý Ÿ²ûâ hxQÞ¿
fD>Ð’’ÿñncvåúŠîæ`e+ ÿ¿ˆÁ[þã‘»žyL7woDA?JŠâøôaÿ úmÜW4ŒIlU @
…qœ#¹îä”auW(Ç´ÁEº5¬üíëS£T¢ÁŸ^…¯¶òÄ–jØj“Έ9ìJ× Z J‡ÝëVð,+6¦Ù¸Hˆ@ƒ•Ç%[h¨©-:µ½,kˆ0>˜~ž
ˆ¨#IŒ#+Ö( õ”9Ø0¿Œÿ\
ŒÄ¬výb"•:Uxó,ë…ZJ´vÆI;³þ o9‹‰zyJqöFV⃈í„ÏiÌÍfà[tü‘ïÏj_èÜ)vш‰ax„ëO +D °0œ÷ß0n…ªt†;ÑWÉÉ)äøT »ÑâÑ;?ÓÐÆìñ@žcôL2k ‘¸Y?Ñ ç+¿›ñ· ) ýUtx CÅ;ê–kþ8¶+¬#&ÆŒECÄZ'AàVQþ}A½ZZÔ xʶKƒ§W¢ïÐÄ@ ñKL€*Þ¼]ŸÊkf
–ªÿ¦CS6¡ÎR)‹ '^|~¹µxÓ©Ï o©.€þÿ^f%Ö€mØ7úZŽæJÒÉù²Ÿ½Å]Zñ#Ø‘ ¥O§F£j€jÈ–ÀWŽŒeG9°©›¹ øQ€ ¥Eð}¶ñš#¬¨ø!;ct™lH.íÒí‹ZæLÀV`ôŽîË þùL”Òp<Ã[Ñ#{Ù%CŒÆ>¾ÒbOÇ`vÄ$]2€,8Îe=ÒzÔÏ«q|ßUH¦S~µ¦m¦ŽÕ’ ô±cÿæ¡ú¹ÝX!Ýû‹êG¹¼iüÇ<Ç)²«-o ×®§‡æ¤¿ü®FùÑmvž»·þ?Ø«˜› Ï£ „Æ*NCÉÙÒ¼¬]ÒcöéàHÅ´Åí;Ag„°ú˜¶œ¡ÔOß~i×s¨\fæBu€ê§^
e©6ÆŸ»–Å ~ k²Z˜òr»Õƒ‡¾Œ}µ{ï¾Xì*°Á"àUG7sÙÜ53 =xØÔ<ÔP·wÚó
±&hD
v=Ò×óÜRï.c÷‰UðÔø@ïèçwuÊϺ+¯J^(5r#«Ë‹ šO—C#ÛâîÞG J›ÃÎS‘ZÃP•æ.@z]D‡©žÍß 9û×mµ`;Ÿ‚‚ føb$&jÂ2XÏœzøH¯0*ÿÛÙ2†è=s(²Ï¶pg7z‹ƒX, jýÙ2/•Ý6Ìæoo pliŽ4øÖ‹“ Ж¶@ §Ö4¢ÀVClÉÀÿ>Ÿ$R}JóÝfTwu}U£ö†å@0žß«ë×Ú]ÙV€®`†êA И Ýÿ‘óóÃleÝúª1ñØS’ÑôO“ß¼<ëј^´LÈ×nøÓ£'ØÌ(eÄ.oCýC˜ŠÝSüåe`Ïcl9!6(
Z®,„Šøƒ´
6¦ASh;Ég[ÏušÒÚëê¾{}gÒ ÐM í-Ï’lgÝÈ–²Ä´ºppª¦"p‚Šú[ N…oÿœ½$v»›ÔkÄ»bá ™qZï r† 4c·Ðjª™÷CŸªLÀÄ€¡¯}9·$Jïï+qÅ~9`i£2S‡½EPá[¼Ô@Öå"Î9Íê ÙV“ûMa‰R¬´þ%f`Qq2öE^ŒCg—Ž\œö-b ù'/húô®™Z0!n„Ö†ƒ:˜v˜+2Tà9–´ËíÚͽ`(«£xíô1Á 1E6ùýí=ƒi\_b•Ùe•g ‰jb2}ë›~ƒÀ⌢bhEuŠq™I‰ê)êˆÈjô}‘°ÂMa(Ïé`Š3/áe»Bb¬ymz´µ‡5YŠ ŽT¯ k?[Š
¢¨ƒŠ#PÜ•YáàŽC0m-RS(ªµ Å o°ƒ+[FÉ æ¥5J(zŸ ÁŒcQ6[TÃf=…°q¸Üœééf™V Kä¨íüï°b"šv»fž´P É MyFoãMÍÉ`&Lhx[c [—–€uaÙ={¡±¬Bj›Wµ¨ƒ`llÏÑ ½;›g÷ä Lìxóü]Ušl¦à ßZl¾žÖÙ#zÛu½€Ÿ´¹§WÆC4² %9û[W@ÌŽyM³Bx¾K–`Bµök²Ê/¼kίŠQ7blaXOÝš3 °ò‰¿çx98Ä?\ê̌иÎ?“,`Q"ÏÈâc»ÌXîç/šxËqÈ™ü˜è»Ó¾
\¡ÞÉÞ›ò;ÉTzï ×»µËïb<}ãueÁF! Rü ÉR.ÙóˆÔ 7Øž¢—fΨö²ÃN?9@4¼uÉ›©:ÃQQÅåÄçþ+tÜ¡s#K€3»’¹\ 1Þ³TOĺ™LÃé7Å*`YU¹qº!:—Ÿ-“°¨Cö GúU½ì¼>•ðÆnñø–½†Â"\v?¡…X=Y¢•þÙ¼ù5Ô®µ´æe ‰¿®™~ñ|Q$Z<:¸¿•Í‹Á_Ö®©(«^bså’ ië÷VGþþ ;£,á•qᓨ“Õ ^„†e/½~iÉ3!
”™æí• ôAt9g% QÎ3"8‘ï0½SN‹Êy(ã·¥=mÅÊÆñƒ,?Šüª"N“°òµ»áThu6;À`¸ˆÛ‰QãÌñ×¼Ê>óW¤÷vý¢âÜ„y8ǺwâÏ„1Räì0l“ÉÓì.÷œ´¨:Žsn}ç …HÐRu¯Û÷½"´mmƒ·ZT¤0ž"§7aíþdá,Däê§Z«ˆÇ¹ç »ÂºMù9[þ bÀ×!w_®ÞB#g˼ Ë %Œ>ç
lŽéž{œ¬
ThÛ¢YŽXæ`ÈqDßEiRkþ¢Þ
žvµk^Ÿ·jsÿ;ûîm
ìÙ~êwS×™g3Î刯|!2ÉqW¾Ì ,z€»‹W#²‹¾úî&•²%N á+øé‰o§ç§×-3ά·Oöjœµ¬(|òE–ä»æ<•u»3¿&šûýáYíºS3mIJÅz7çöP„X£ú2*†²ÕˆF{ÕE!dÏŽn™Ü¤ƒJ jJ'›ª÷.göùWÑQÄôõ²OšêÏz»TùÜ~ŠPkXdâoZ§Xq€â°vÝ -Á. ^dËþ°› ³AbUhÆÅ!g]UØ©Q»|EÄ\ÏÛ–ƒß‰z¢ÃÄ-ÛA6q÷ÔÓ'MÓq@/ nZ–PöÕ{+&çHwú¸†zÝ=¡Yo®67Ö
ÑEw ÿô¬C89&Æúb4È*o,•¦ëûCÆá|UÎ dn;¿]ýõÜú[
Ê÷KþlO ˆ 6³¾/Î ~½µó×>y €öÙ¼÷7éÒ÷¯5í yãÅ!cN{‚gÛ–ÁÄBí õ6vIx+¤Èå`’ìž½à.J3pÉðè©ßT y¬)ŒsX«†·!þƒ)‘ë‡
]?íÜÉðâŸ'×ÞܸŒá>GŠËÖF‘M 8]ƒ4¦2f|²OþTjQ¸È$0ôr¿ÞW_
î6…äX¶!43™bsInD²]>•G¯MËn¾ƒÕ¢¨ ¦Ô[¤Õ¥kþØy§©Ç'ÂJÁµÒ¿Ÿ6ÖϪ ELl1VvAÙž–Ÿ}ÃæêSQ:NChk…±ï(hmWŠ`SÆja—zóéV e&\r»Þ?¿åy˜S®³ElÈ+v;@š~c;pÊ6ª^ H¾Ã* -ç\õHsî"5dì©)(·r½IÜ‘CØÅ‹+JOAÑ‹áÚÙçë²±~(î‘h6¯íe Ý„…DÞ
¤ùoöU¿¯ëfâUÿò£æpð
å€Ï9”S_Å& L@Sl: RŽ³ëP W¦!qiÀ8 UÙolJæPœÜÚ/²¤XüŽ¾ ˜Vλ<‚¹fZU˜¾x\î ‹(§Ñ»»]¨îEj¹ìÚ•Ôf6xçr¶÷ä÷v¾†óGÊpL=è‹€¥ù£ ¥úq0–ÔŽDa;†ì̪§Š1úpµöÍZµô2crÉ°Ì *ÊÒ;LøhìŸUACuã‡4™¨s'B#!>Ú ðE1˜Ð!}ÈŸ%r±í[g¨ÙÌ‚3¶E‡ÊãØGkÊéA,G˜%½Dë};#‚jÂZû^ó1uºy+µpïê I>–…BüI@$Á{Û ÕpÐõ]úWô+(‘c1¹ç7ŠþõGQЮgԾ̺Â$ y0ŒÉ,/å \øY\ñ ÉõjB¦Š]”B–¶¡Ó½–%ǵ›;)®3’¤0¯*‹›Î T½)>‡Él‘F(éß^Uוï4oî]ÛR¹km ]†«ê¸9 í’°
+Ï©57ŸËa·Ý” 3CbÏó¼¾5S¦!=~ÙçÀ‡÷TéïaŠ9§¥ý}N¼výÖðF°¤ÓƒEˆ-è®ÜŒ×fux¬Æ/î[Å5Ú{îõpá¾bGÝÌûé@çT¢û*õ:Mi,MÓ™Y…Iôµ">©X>•zÂLwµQ§B„ˆ©V!¬þ8²{Wv¯j7Yƒ™¨öÕ1Y%©J¨d3#e‰0o’¡ÿøëÉBíí
´}²~Ú¾Á¹kšcc–€z6\{{°á%~Ñ%Å3yÉ>ð£hTÿ*®ƒª`=˜‰²ýUA ¾³:^Ï+þe:$HÅæËJØC”dÀÿ'¯ëJuó•o®Ï–¦…Òš@O?!óM!$^¡VÄ”þˆF4³bqθù4x /jÎhh²ÖØçdïÉ“}[QÏx9v1 ¹T?0ùøåAàå)߀Q^Œs‚::xã%äಾçE…U±m„ ¸Ô E œÍÿ8E± 9ôêü;½ÙèíSþù0”±K|ûù™¢…ÿ†®W'’åÂyíBöP(D€‘ÿ,‘CbTÕã·Ö8ƒ¢°1*ß=8 2ŠÝ[óÞO·“¦ú@³.
¶“ujùÁ QÆä)õ7Ž…ê)C < ÝSÇNͧ&9¥ãåÜ*½»ÌV>DíÔFŒoëÜç,ãH”ÜSñ' ‚K×ζ8ezp T"2¹Šî<øâ_ÌšòD¿ËwµC+™šF†„2vˆ´™<•=ï³³ª¬i
‡i„M×¾÷4¾Ûv«²½mÀ ,/.LêScµ;*|@¯·ü0J!$KÀPŽVXÜ4ãÊf €˜ð0Ó?ZGDÜnŠ-¤3ܨ:´j“°áÇ7ÏCåSKG‡‰SƒÖ§\øƒXÁ&˜r0ZuC‹Nm—ƒ˜1´„$Œj“Tü³#¾üӮચ‰Ð¡©6½8bÌ<Ôu«2:
"NÒw~qçtÖo>gû‰Fn£`²sbœ±‚ º,2ŠT1^/‚Ž½D•ªgï×ɨ±)õ
ZEe4J([0-ï@c2¯%ˆFZ’¥UÅ—û:(·ß¨}<<ãŽoÛ¯6ïh>H=qe
¯ ÒNmª›/V…P l•£}u
ÛnࣕM6O¯Ø|.Ò©j®[2¿§aO¨KdT†Q!V|4BÓë†È{jyT·;aFåL(Œ¼’ 1äºYb~ÇH!¾J¦ßWçÆÜP¾& ÒÞ{ç¸}Áî È
ý ZŒÄR.ÕÚGûÒ«A¯K´¯Ež$oÚë&LtÞnMH€ó¦Ûú”1à Ô1þRP ³za¤—ßÞ:úG—‘ä¼FîÆÿhvøGD’ lûÈS?h‡ ª4ŒÂe β4¾2'Ø?÷¾
«ÈHÏœ^¬kW£C#,0Nù.ÕÄhh•‡ýœtÕ
iU|qyKÀs2™ÿ'Ã> hˆ¤ÅlõÖ ¹“%Ò¦x¯†F¦é|~º°üé4º² `´v¯Ñ\¥¸¥PÈ”ld[!¦
düZñ^Xì‚øo5* ¯q ghnæ[jÚ·þÆ~Pâl ¬áÂÕw‹ùÈ ‹Èn
Õ„Úw±›‡^ 5Cÿûšì…é¼M¥çm¤ìŸüÅ B\¦ð‰yD‹–~¤™žã‘p“ -ü„{£ ~bJœÕà.ö¶ ~Û7Cßõd§Éæh¯îÎÝQ£ùÖ ˆ…¤«M ð…§X®«ó ñÇ¢ãûšP@MàZ@M´í Ϊ¹ìS侉ôéÞó=›s‘
y¦wìîQ² S ÄéÃG°šûö¢§ˆ13H.Éï }P5Š%0»µóÅÙˆÇÀ:B¸|ƒ9‘nv>OÀ‡µëp@¤
9=,ݔ
RT¤ºy^Ñ< i ádƒ¶.½}\Ÿe†¨àytñ = Ò å ûbÏÖõË¥±¦gt´0Jk5‰…..'Æ:…Ù×5ΧZ üÍ"4Øb @Õƒ]î©ÆðäZͲ©\ uu¦–þ(8T4¶·“ÜEè?º&GÊæDB”j7PÓZ!1’ï-*½Qa2ªî²ÙÔ÷ðñû;"À&ŽØ¾¬Y$DIuVNÊ& ‰=Ýߎ/FÕPd·#ÑHMúÇbÕ|ç/?!Ð2gÎl×Y±œãÜ)ð {£žMÊ$ÿ³wRL8}Pà“Ñs}.#ØEúdGñ²@¾³;Q*§/-«6<
f‰X·%E‚ÏcùÉ–°¥ßè‚ÿÏfŽ…Q Dí9i#vNä6 ¨Ð=ëÀÙf®‰¥r6x¡Šk©È·ÑM¬0?÷FX\v}KÖ\pì`,Ê®?ÑÆ(~ûà^à€“sÍ’ŠÇ’G‚%ÙTš@c°PîýÊ#E\Z¿ãG—Årˆ ³ösGûÚM[cïð.eM™Š’a 9N µ»k`ž`a'Пk—/m=ç²
’W˜…c ︬Ââ{Kb´„ÏÆþÜ ”—škP©Fa+û œÍåß”j‹ 0h|Q¾vjÿw±³7÷Ų 1icâ-æ^2ý…MÜ^ª³]'`VùXƒæcwýµÇÀy,šj§9ŒN}ýx°'^$Gì iŠðú¶õjõc–WÒéÜTÞ™®'BGLáË¿Ö.ù˜¶ óLŒðké7Í%©1¾»}è
é|…¤jÊ«ra# ǧâËä;šÖ³w˼Ã|"`/Ô§c :†Æ
Óà£@ö©¡ØÑL»JÍ‹4þ ¢Œì‘-]Q ¥šÄæÐH‚øÊh¶9Š\sù{Å`%>—÷Ã9ÔÀ£õ„¯+? )nÞ¶º¡iv]qÚ0©‘èTm_îÕ%;£ÕàA“ˆY§ŠX3áY`0Z9MŸ$›zÒÖ®>±â £"^¾ãåÛ{Çg…]LòЪâÃ
!£8ýwÆ ž)| :f‚N‚œY8À2îQ6,Âæ÷sÕÖbË< ·å?yB«Æ#g Øå«Ÿ2´ÑÇÓþíF`v„à‰ù RŒz=:®ÖýÂÈ×eÕnt$wrüñ™}TĹ e³(—u˜°lKÖH$Hå¡Þ2ú —±2Bý *Øá¾\Ën éBìéi,ƒÃFâ¿kžt@¶~ž5·*‹åi“¼4›ãwÌ—9ÀŠº-p¨2/ü[E5bz· cîË£9=¬+9,Òç1 » ¿ªO—a@Ù?
l9Îx+d•ìšÉõ}T3"°ßþg·Ó!ú-ÜqwÑ¿;£Iˆ”Z÷i‚„6Zê-Fœ4j¤:ؼ©k`|ɇªv]ѯG¸> ÒÎÄS“œY2ùšúazˆ}tFDÝD³ŠÂ ,–à\ufÀYþÎDj“ñ^;X¨Ù¾¦ZËzX× B0¬ø?¢¼ jÍ
!ΣçÅZÿVjLo(€o3KÂ,€¤‡õ&Ô¾µt¶P”„Äal@md›:Ðab|‰ê˶üªyØ öÀMµ\$ÃüP M$8—á“Ž®–å†Èïª,±ñ »mÉ·ÃWiWügì/dóëP]Iv¶¹³ þqžÞ•³ºã#Í÷Ú%¢³ðÍ(ªkæTÏ’üÒãsPð[þx™Â ¹·ŸUÓ™òa^„#!|Èw¶)±¡¬QH¾mžOÑⲂ, ’¤-‡mÖY±“ª‘QsÚJlüžé¥è¨&Ze› ï ý²Ç”uBÑæz÷2çŠ/6!:š o5ÞŸÈ „é,´‰ãô-ÞüÓU¡:v…ÂÇ<ß³ŠAWS!C€,Ú¯â–Äj t6Û˜èÌ6ù‰0Óé8
\`°féb¾ÞÃðïh1o£ZÝtfàƒïÜØ®õ Ɖ'Œ#Fc§Ýmežz-Ff×dû÷ŽvV)NvXVÁ®P™×Îômã³ÎÁb‰Ð¡)W¿*[©Ó1ç#T__¶w§dʾX†ÐleK:Q¯ÛïßÖ§R‘c4ÅNR2´¢~ Ã4"ÎÒ¬R¨¡^}f®© ¢rÍ •ëvPP©7h;)21ÈÝœhçßU½¤¢ýsõ…mµåxÅsP#öÕ`®¸¯Ñ8L!/m
)$ïÔ"a4é·ò…òLŸy k#Ø$öª!hSQ©F7hŒ‰1nã`÷üw:Ôs ƒ|'.°7>æAT¢+7À—KºÕe}ªP³²æa+;ՉŔX\“½h=¥Tñ–XÎUò7À¶ùª`2lb—Ý®\ô)4rDãH¬~q+«|åñíÀ\ÍѬ¸šúT»!!ü8QØò5:ƒ5°† À&
àN4tWj1
AqUT,lèùÊ»PÄ{öº‰ÂhUÇöÕÓÓJ€ñzsnZƒ ƒ—ßqüZTÛvÅ®öàžz”ÍÒ–·P 0H½ûwúÜ»ª ÆbÞ[m±Ê0];TJ*˜iæ1 vÇnÎÓòU\¾xl/0ßþüÂV->H&§Ò!‚8D?SuJQ}Z(Ÿ¾Æ¶ŒAZ×/ë6·ïœ„ó¾zF°°¡PÁ22ép/‚;T'SÛªˆ þ°á§Ý NÛ&¥jãRaÄ›Näí ª"´jDS›'Þ×æÅ@ f¿?/Bvݼ‹Í=8…B—|ƒýäêqíæk¹t35OÁIk…l*Þè”ÚgSWü[Së·"¯ªòÉOÿnc’«OC±{E Lš|MËÅÕh_„Ð ¨%4žm™ i=¯ƒn‘q8=Q9çš"Èš½f|RwàÃË…™2àö¦dúã½ÖvTâP¥fsWôƒXT ;„Åûì¦UÞ Å^Á6‘sšdÈŽ éñ‘¬&bîýžX@N¶bÕ·¨€‘»)æÜÑ7¸Õ¼n¢Cø!ºÑNŸÍ G/Óí¿9VÁ©& ¨L±'JŠb’ÙWª
!äK}¯xø¦ñ =Ó¬Ü^ºe¼¥EþJÛ¢¢s¡ñ†G¶ÄY¥?c–Ÿ Ž»Á·…Ð#cpTÙ®¼Û”))FuбFñw ôf\ vjbqO{Þ)¡e1T©syÒ ¼2š¿ '×·M>d…8 E¦°ÌN#nÒ±§yò±v ßä>ØË/RòºçÉ)îׄ X( )eZ€Z=ĸEÛaä ÛvT4CØÒªºïºüýuƒLuAöΙíù—wO{ñgC÷”•›°qÑL>Ný
L5Z±£h*Nü!×ïê±hòUõ©2 9®ep]^Ÿj 0G²o y•¿½:hRÍÀ¬l@4À´˜;}ˆžIô³ZKXïÑx¼{
üáeÌÐÿù$hìû,v3 tN„þ¾ fÈÿ©ÕÖÿ#D¬Hƒá}ÿÇ fSˆ¾=0IC^i£Œ!ÃÝ’^}ó† ,dÕ>@†[³Pƒ_@ÈÞP4_7Øeî
W®«yM„¤bN~RhOŽ
æv¢¿ ME„¼"ÔK ‹éW6qƒÙD òEcÞ6B'›ÎÁ¢±œRWùЂ ÄÅÊbM wü’ÝÚ=‘èZç ó™Zk…`]5wüŽE6…ÔS?kwÈS ǧZ/ѤC+ Ã4ÀI\U©·ä ÈJìظ¿|¥ôô…-nš¿ö5ÀVêæYUÑdÕx±Ê €,=&4ø™ê ÑŽ¾åïÓ9Yqó{_-Lõ pÖ¯/ °p'T`—™Z÷[~Ù:iþVK°*µ¬B Û jtŒ™Ðl^¤áßnARÞ½éÙ$F«î+µå]>úI‹CõaŠ²Ž<ëÝm1Íò?'Ýÿ‰ !ÝÆŽ|ÜQ½‡ª7œóûo˜ÇXªiD‡¸NŒì¶5/Ô5àÓʳµÎÆ#ÿ¨aø,á Â~êX‰ÍÓ - (|ù…=þßW5—§“ÓŒƒL.¦Á5vĈ€”VN¹ª˜ÊbWï•Ž
×5—}Ž>ZôÐ ³ôÂŽ}ú§4_ýâ5a3:TL©T§Š¬Tœ~í2.öOÔÜWoÐë!B 1Î×ɹ gŠ•Ó•ž ¹ºl_Ûg“Ú;
×8 AŽÌŽübϬ 2)kwýyqQƒ® £UlLmk×M‹²«Ú"¶þ&4Œxw…K‡Jq-«ÿè$œQy¦ “Bú3rßѽÐXÂ+1Ť½¼‡D÷¸ªÉŸZJx)ÔfŠ×'r™½k½ˆ¥FSTÃqâì%Í€ñrëç®Ñr7”´O5'é˜î•¢±©Ýx©ÎµŸ°Ê…{Št ™ºƒ„á’ãÒ·PC¯E°‹/ø76%!Rg¾
‰A`«hEâi§*9›ù/¢ ºØ’úV±ÐkÉ ÊFLa®Þ9Øí€ÃoÿФo”(ïƒtò[{ƒR<ù!|èa-ÛGg4ªi5̼%ôêôa¸ÉÿÍõߊn¤µOãëJç NâBeÁ#d~d¦(\¬ÏÖÞ@}zçÜþÉoW#v¿=“ªf
kàÅÙZç×.‰ μU»>†£¤õ…ÚD#´Ð”1ç•ñÅ7ï )WÞ0FIÌôÄ£ðËZº:Åù·`à*++óB,•¢e(siAžÍÑ‘?VÄLr¸¡Äi-ú Ý᣾R`qÄ ù69·l?^ÚF{Èêa8Ö¤ƒÉËa8©Ú°VÎô/A¸™Ð¤ÂB|ºå§b8ÑêŠ :§y¯= 2UðF}fDZ¥¸¡HÍNo`U¼æÆ´öõ©tvF)kMk
œÈ=‹è*B"s2µÍOdc…¿î[¿fÀ†`El#SiØŽû3xD¼
•=§ U?I]` ìYV_Ø.çsâÚaq°!ì¸x_Èð’é'tUXBžœö?iË'æíwî«k¤ÛÒB°ŠžjxZ³„ªÑ?ÇDíå;BÈ>sc â:}Wk£Ò@.çH Îî\¿î?ëÈY[ÿT;‡Y!±Xï]yåëEK/?Dî2Í9˜¸p6 {B ŠÐ(u‘üsZÐÛ¯5·að_™HMƒÖ*faÜ}5†Jù– bï ™È¼˜LÀ&Vrºrf¿>½¤´ìáH·Ô ú'´-qŽ*¤•¸,LB¯¢Ý¯ƒŒ‰3"äL÷kU™ëâ*“tBî ŒÑù«’ò@7\äœü®UüaL{e`òf}JHÜ›ûxÙ?·j×çÂ
FfâUò6¬9È=ä &ªP.4©Ñ“ÓTH¶ÅÝ9ù–]ËG뾑ô™ìi¶|Êj-âqÄÀÚ®¹[+Êо7ÀÀÖ YB˜Ww^C BcWÄWhò„ØÝ¢žˆîë£õù=û¢ 4¤b‘P)ÂÒU´0_ó;EôÐoM…¨ ƒ.Xò
ÓŸåì…—V^5ñì^ÃW$ ®V?%8›Ú*=8¬ãpnä…io2ªG&d§Á♑qî È`4“…í´¡·YIŠ¾Ë‰/hFÏÍÓ÷P@Sïk7
Fv ßþ¡G—קr Æâ7ì
œ,MR.ƪï
€6µLn
Œ›ë÷ö”B¸_» :ÀéÀ #áì³j#²®ÇúŒ›TÈAØœàê<¨Bô„A”kÐű[Zï2l³ä °Çf.¦‡ÁwÒê{"…h 7:‡ƒ¡4U ØËñ2röYäVø[ jn18²<¿p¾.Íèì÷Úµe •‰åSkJ@n=$ Ußï¨,rß;9#öeÆfÁ[ —: cìª/.(ñZrȺéÞ‚?•*.â¸)_«±ñöZáÈ }B»&»_U¯+Kµˆ#GïÔîM„Ù ¶“k¨ˆ‰Yßåé®ì7j~©¨Ò²ƒp’$¶×nê›ktº;qÊ€æý:H ŽäEc
kæuÛL‰öÌwï´7.¢|ùáW&®†~% ÔI!€S*¯áâ÷Uñ➺M2²ú1c’SÀ#Æ0@dn,‹¯8”°ìñ®Lä´hJ®&8 Ò-u ç C’ rÁàéZg÷ÌjO!›bGÿ7³;µogÉAÏL½l”ØF¿q»ÿ@@a£î¨ë`R
ÕÆ邬‚JƒG¿‰•¦×ðV6cj”Žé
ù5ó;ÿ¼U røsÿôî0g” ýõÛwoG
ùâ'¹Ìt·j˜Ê · N=´{ñçxS%O<ð.*• “mÙ•S°ÌdÛEÑ
ñØ+‡4‹R‰sj—+^çqJc N€µ°'ÁÚ<¬µBhˆ‚¸*mfÝÒµ~ rY‡V6"h ò ùŠìr¯ù.²{÷ν–³‡ Yóº4%æO#ÿDNnz8û/ä@O „JÒîÖÚfAõdöÍ-Y«c sM´#›¿ä§u³ÉŸjbôŠi[€urcµvF×hõi·¿d¸ç ¨£ßX4*õ®Å?Ùj™€ôVM0[™ÑyC›ÐhV™¥|:zDšrÌ E<+2Ú¨¿Ø(º4lXÀéËÚU²²¦“¯ÉB“3ÂLãè
7ß Ÿ*ktcµ:ËÚÐ v„j\È«ïk‹«ö–'ûPØYÓY6º-ÏŒÆÙóÊzAü–!ƒDŽÊšÄa1T,†
ZqÜ„q^ÈŽ6ŒC—[V ß–ÓÄY©\i È‚«:0ÓP;CãséùóG¢……ªï È°s ¥€ýf»võ¯Ì[aY+dPóT57±×׫å_0ÐòÅZ‘ǺêHéq}ªáÌ !†q€eã;·MÁ!Ù‘]ì8„Ìær”²u= 1fôjFŽliµ.b”Á3*ü!ÆøKW»ŽˆV0žFhõ¢çu-GYJ¢ÈÝ ã_Èš¶ý¥tÌ~²–m¦”[õ,ˆÂW¦na³ƒ #†_:Lœèÿ†L¸€»¿ÌâÞn¼wªÞª
MÂ0âAíj6»vçÈR†f4Ä Ú 4©_1}”PwÃ'W9óUö V™0¼þo‘Î,*Ž-l×Òµ~Ôˆ(P€2³Ö øÍ8»PdWýû=µ²†#”³ ïEÓʯÕS'0}7ŒNmÝý+ìË5Îã~ò7
eʋ
c «;©ºÝ…7fâòæ[lÛçÚƒ‹ T¶÷Õ¹OöxP˜dPŸo;'»i8ò:2ÒÂ?5† «KС‰ÊËmkŸd³!®‹\ 4²‚U&)yÇ¢R¢bú›<Ðu΂Lhw!7ÿG‡ü X¯[ðk" !’
ÏÆÛ;ì¬ee<óQ+#1ݤª’W£Ë¶°ûJq°©ŒŸ%˜íB/3µœ÷ó¶b*FV 7è5{¬+žyÝÄZ (J‚LNTÉvi µ¯9'f {ÚÂkï\›»Gkÿä»-e\SoæAÞUù ¶½¨°T»
6ßÑÂÅ-
ÊX¶VŠˆö…ªSÝôÅÁÓ|¬Š?.Ÿ®»éƒßB¹ì 0($gñ¿D“ªõ…UKl8®MD£vÎ édÇvß
׶¨\ôõEÔÓH=#
$h¬!Û|hW
𠿖ܾ;^\0! öB±($Ætbë)ÐkH‚a¿a.%ˆ`" ¦¤WDPÈŒÞ\-3_Í'áàõ_…›+ ƒÞo^«]?O¥ô²õ•Ì,›êŽjf»“ù‚£—b´ú²u9 ½öN–¦ôÙ}ïØî& ÕD“ hÇ鯙ð9¼:jÌ_è u¶$6?“ƒ,Ä}ÔY²á ½[›··Í¯WÁ.¤r„=“2¦€• Þð‚V@½~åµ½)Òþ8y¤«]Aˆ»
sw̘~– £3½õ¦vÙœì1¢Ý¢ü%6nÚ²Ù]G÷õP·ÈzX½ˆw-€íC4—ǦgIU6¯Ñ‘ ¼0Ñæ&°rí’*—× è©æ^åêáº~pâUjuܥϞ*Y½ G™¯7ùëí)Y£1j8OËK·*Ágl³ ÑÖ¦vý‚M;ÔÎWg31éÙ Öé dJd3U "¤™#ÚÆ7÷¯y¦ëÆ5,óÙ+[ŸxáÓ¤˜ { 4ÚC7ŽÏç½±Ù«/ݺ0uHÖñú¼sFSçâɺqš0 Àë0n—«dMð¦É?{ç1ŒDG“Ó
Ñ^å/:xQí6Þ¦3N ¶ÊÅ—wý]CRŠÛ°_ðKŒ“³Io¼î³(„ª#*%{Cµo©,bñFg{¾x.0í0Ìd kÌŽ²ˆÀTg;Í®íåƒ{ˆ„¸L…Hå;@7‰Ð3-Ý ð¶XÎÒjÿ@«Ê Ó„
B˜è ¨EV[ÙäÖi;$–y:Ù‰Ve^¯)sn Q§Ð
?n,øÇ·†1“§ºDÁCæ;J(«ìçãÀ¯¦¥_]»®†§Šv˜,å»B™"Løe*@¯H,XÙsWaÖül±ïú—Š[j0Ú’b$À‚ qxxðHdno‡bQÄJãåÝ DÚ&Ý_&Pµ¶˜ÕHcP9ÃQdÜÊs{²|Ç3Ùc¹Í>²tuV˜ƒµ 0ÒûÄ?ªMׯn
œNŠm³Œ €^˜HšYª5ï–BV| Ί-^Ëy pùÅñúÓïòÖ¬¼þŽ6t<& Œ×Lj7a l“N-Mþ'üô€ÃÛ7ŽCmÖ^/LÀ
p—€Æ6£ÅÓ»Ó\!í”à–æÓ.!®€ Eõ®³Pïž)m**œ+:ŒÎˆ8AR+¶|ëŹe¤§·
‰I˜™¬ˆåºþÎí#$Aò®’ʈEÌÌ ymMÛFKŠèõëõ©áQ±º V–T&åà â x€ùR»V<¼¬ì·KÅá6o D3¥n”Š—·P½n‰‹¤ª'€áôÈxj² ë£áÒ«ª,å`Ü"”Ö"!4ú&Viy¯™¨3* ê‘Z±eÄnaH¦½ïÙUÆ ÖÚ‚Yi•W^øo'NÿÛ߉³ÿÚÍè¿‹›ñ—"|ÿ·É/ŠÿVwârò)¾‰cƒÏÌ›“OÖ.le¸¾úorÅäëý$šü8ÁÓÿÖ.ÌM¾L’ÿ–———®ÅÿñÿµKQ4ùpÀÿþùß?ÿûçÿ˜Lþ÷ßÿûïÿý÷ÿKÿ
ý›AãæÿµÐê‹ñõ¨Šò¨,Ó¤(ò$/K|[ü×mÿú¾st½û±;\ÿþëÛþV³»Û ý·×5ßšÝÉMJ¢±O=-øC§„9‘ÀBHª‰!Nž]¢xòàáZ„ÏÓø¿ëÿ? (¨¹« milk-0.5/milk/tests/data/__init__.py 0000664 0001750 0001750 00000000000 11723217323 021461 0 ustar luispedro luispedro 0000000 0000000 milk-0.5/milk/tests/data/jugparallel_jugfile.py 0000664 0001750 0001750 00000000563 11723217323 023747 0 ustar luispedro luispedro 0000000 0000000 import milk.ext.jugparallel
from milksets.wine import load
from milk.tests.fast_classifier import fast_classifier
features,labels = load()
classified = milk.ext.jugparallel.nfoldcrossvalidation(features, labels, learner=fast_classifier())
classified_wpred = milk.ext.jugparallel.nfoldcrossvalidation(features, labels, learner=fast_classifier(), return_predictions=True)
milk-0.5/milk/tests/data/jugparallel_kmeans_jugfile.py 0000664 0001750 0001750 00000000355 11723217323 025304 0 ustar luispedro luispedro 0000000 0000000 import milk.ext.jugparallel
from milksets.wine import load
from milk.tests.fast_classifier import fast_classifier
features,labels = load()
clustered = milk.ext.jugparallel.kmeans_select_best(features, ks=(2,8), repeats=2, max_iters=6)
milk-0.5/milk/tests/test_perceptron.py 0000664 0001750 0001750 00000001330 11723217323 022240 0 ustar luispedro luispedro 0000000 0000000 import numpy as np
from milk.supervised.perceptron import perceptron_learner
from milk.supervised import _perceptron
from milksets.yeast import load
def test_raw():
np.random.seed(23)
data = np.random.random((100,10))
data[50:] += .5
labels = np.repeat((0,1), 50)
weights = np.zeros((11))
eta = 0.1
for i in xrange(20):
_perceptron.perceptron(data, labels, weights, eta)
errs = _perceptron.perceptron(data, labels, weights, eta)
assert errs < 10
def test_wrapper():
features,labels = load()
labels = (labels >= 5)
learner = perceptron_learner()
model = learner.train(features, labels)
test = map(model.apply, features)
assert np.mean(labels != test) < .35
milk-0.5/milk/tests/test_ecoc_learner.py 0000664 0001750 0001750 00000002574 11723217323 022513 0 ustar luispedro luispedro 0000000 0000000 from milk.supervised.multi import ecoc_learner
from milk.supervised.classifier import ctransforms
from milk.supervised import svm
import milk.tests.fast_classifier
import milk.supervised.multi
from milksets.yeast import load
import numpy as np
def test_ecoc_learner():
base = milk.tests.fast_classifier.fast_classifier()
learner = milk.supervised.multi.ecoc_learner(base)
features, labels = load()
nlabels = len(set(labels))
model = learner.train(features[::2],labels[::2])
testl = np.array(model.apply_many(features[1::2]))
assert np.mean(testl == labels[1::2]) > 1./nlabels
assert testl.min() >= 0
assert testl.max() < nlabels
# This failed at one point:
learner = ecoc_learner(svm.svm_to_binary(svm.svm_raw(kernel=svm.dot_kernel(), C=1.)))
model = learner.train(features[:200], labels[:200])
assert (model is not None)
def test_ecoc_probability():
features,labels = load()
features = features[labels < 5]
labels = labels[labels < 5]
raw = svm.svm_raw(kernel=svm.dot_kernel(), C=1.)
base = ctransforms(raw, svm.svm_sigmoidal_correction())
learner = ecoc_learner(base, probability=True)
model = learner.train(features[::2], labels[::2])
results = map(model.apply, features[1::2])
results = np.array(results)
assert results.shape[1] == len(set(labels))
assert np.mean(results.argmax(1) == labels[1::2]) > .5
milk-0.5/milk/tests/test_defaultlearner.py 0000664 0001750 0001750 00000000362 11723217323 023060 0 ustar luispedro luispedro 0000000 0000000 import milk
def test_extra_arg():
from milksets.wine import load
features,labels = load()
learner = milk.defaultlearner()
model = learner.train(features[::2],labels[::2], extra_arg=5)
assert model.apply(features[1]) < 12.
milk-0.5/milk/tests/test_normaliselabels.py 0000664 0001750 0001750 00000001702 11723217323 023236 0 ustar luispedro luispedro 0000000 0000000 from milk.supervised.normalise import normaliselabels
import numpy as np
def test_normaliselabels():
np.random.seed(22)
labels = np.zeros(120, np.uint8)
labels[40:] += 1
labels[65:] += 1
reorder = np.argsort(np.random.rand(len(labels)))
labels = labels[reorder]
labels2,names = normaliselabels(labels)
for new_n,old_n in enumerate(names):
assert np.all( (labels == old_n) == (labels2 == new_n) )
def test_normaliselabels_multi():
np.random.seed(30)
r = np.random.random
for v in xrange(10):
labels = []
p = np.array([.24,.5,.1,.44])
for i in xrange(100):
cur = [j for j in xrange(4) if r() < p[j]]
if not cur: cur = [0]
labels.append(cur)
nlabels, names = normaliselabels(labels, True)
assert len(labels) == len(nlabels)
assert len(nlabels[0]) == max(map(max,labels))+1
assert nlabels.sum() == sum(map(len,labels))
milk-0.5/milk/tests/test_ext_jugparallel.py 0000664 0001750 0001750 00000001324 11723217323 023244 0 ustar luispedro luispedro 0000000 0000000 try:
import jug
from jug import value
import jug.options
from jug.tests.utils import task_reset, simple_execute
except ImportError:
from nose import SkipTest
def task_reset(f):
def g():
raise SkipTest()
return g
@task_reset
def test_nfoldcrossvalidation():
store, space = jug.jug.init('milk/tests/data/jugparallel_jugfile.py', 'dict_store')
simple_execute()
assert len(jug.value(space['classified'])) == 2
assert len(jug.value(space['classified_wpred'])) ==3
@task_reset
def test_kmeans():
store, space = jug.jug.init('milk/tests/data/jugparallel_kmeans_jugfile.py', 'dict_store')
simple_execute()
assert len(value(space['clustered'])) == 2
milk-0.5/milk/tests/test_svm.py 0000664 0001750 0001750 00000015503 12043771003 020667 0 ustar luispedro luispedro 0000000 0000000 import milk.supervised.svm
from milk.supervised.svm import svm_learn_smo, svm_learn_libsvm, _svm_apply, learn_sigmoid_constants, svm_sigmoidal_correction
import numpy
import random
import numpy as np
import milk.supervised.svm
import pickle
eps=1e-3
def approximate(a,b):
a=numpy.asanyarray(a)
b=numpy.asanyarray(b)
return numpy.abs(a-b).max() < eps
def assert_kkt(SVM):
X,Y,Alphas,b,C,kernel=SVM
N=len(Alphas)
for i in xrange(N):
if Alphas[i] == 0.:
assert Y[i]*_svm_apply(SVM,X[i])+eps >= 1
elif Alphas[i] == C:
assert Y[i]*_svm_apply(SVM,X[i])-eps <= 1
else:
assert abs(Y[i]*_svm_apply(SVM,X[i])-1) <= eps
def assert_all_correctly_classified(SVM,X,Y):
N = len(X)
for i in xrange(N):
assert _svm_apply(SVM,X[i]) * Y[i] > 0
def assert_more_than_50(SVM,X,Y):
N = len(X)
correct = 0
for i in xrange(N):
correct += (_svm_apply(SVM,X[i]) * Y[i] > 0)
assert correct > N/2
def test_simplest():
X=numpy.array([
[1],
[2],
])
Y=numpy.array([1,-1])
C=4.
kernel=numpy.dot
Alphas,b=svm_learn_smo(X,Y,kernel,C)
SVM=(X,Y,Alphas,b,C,kernel)
Alphas_,b_=svm_learn_libsvm(X,Y,kernel,C)
assert approximate(Alphas,Alphas_)
assert approximate(b,b_)
assert approximate(Alphas,[2.,2.])
assert approximate(b,-3)
assert_kkt(SVM)
assert_all_correctly_classified(SVM,X,Y)
def test_more_complex_kkt():
X=numpy.array([
[1,0],
[2,1],
[2,0],
[4,2],
[0,1],
[0,2],
[1,2],
[2,8]])
Y=numpy.array([1,1,1,1,-1,-1,-1,-1])
C=4.
kernel=numpy.dot
Alphas,b=svm_learn_smo(X,Y,kernel,C)
def verify():
SVM=(X,Y,Alphas,b,C,kernel)
sv=numpy.array([1,1,0,0,1,0,1,0])
nsv=~sv
computed_sv = (Alphas > 0) & (Alphas < C)
computed_nsv = ~computed_sv
assert_kkt(SVM)
assert numpy.all((sv-computed_sv) >= 0) # computed_sv in sv
assert numpy.all((computed_nsv-nsv) >= 0) # nsv in computed_nsv
assert_all_correctly_classified(SVM,X,Y)
verify()
Alphas,b=svm_learn_libsvm(X,Y,kernel,C)
verify()
def rbf(xi,xj):
return numpy.exp(-((xi-xj)**2).sum())
def test_rbf():
X=numpy.array([
[0,0,0],
[1,1,1],
])
Y=numpy.array([ 1, -1 ])
C=10
Alphas,b=svm_learn_smo(X,Y,rbf,C)
SVM=(X,Y,Alphas,b,C,rbf)
assert_all_correctly_classified(SVM,X,Y)
Alphas,b=svm_learn_libsvm(X,Y,rbf,C)
SVM=(X,Y,Alphas,b,C,rbf)
assert_all_correctly_classified(SVM,X,Y)
def test_random():
R=numpy.random.RandomState(123)
X=R.rand(10,3)
X[:5]+=1.
C=2
Y=numpy.ones(10)
Y[5:] *= -1
Alphas,b=svm_learn_smo(X,Y,rbf,C)
SVM=(X,Y,Alphas,b,C,rbf)
assert_more_than_50(SVM,X,Y)
Alphas,b=svm_learn_libsvm(X,Y,rbf,C)
SVM=(X,Y,Alphas,b,C,rbf)
assert_more_than_50(SVM,X,Y)
def test_platt_correction():
F=numpy.ones(10)
L=numpy.ones(10)
F[:5] *= -1
L[:5] *= 0
A,B=learn_sigmoid_constants(F,L)
assert 1./(1.+numpy.exp(+10*A+B)) > .99
assert 1./(1.+numpy.exp(-10*A+B)) <.01
def test_platt_correction_class():
F=numpy.ones(10)
L=numpy.ones(10)
F[:5] *= -1
L[:5] *= 0
corrector = svm_sigmoidal_correction()
A,B = learn_sigmoid_constants(F,L)
model = corrector.train(F,L)
assert model.A == A
assert model.B == B
assert model.apply(10) > .99
assert model.apply(-10) < .01
def test_perfect():
data = numpy.zeros((10,2))
data[5:] = 1
labels = numpy.zeros(10)
labels[5:] = 1
classifier=milk.supervised.svm.svm_raw(kernel=milk.supervised.svm.rbf_kernel(1),C=4)
model = classifier.train(data,labels)
assert numpy.all( (numpy.array([model.apply(data[i]) for i in xrange(10)]) > 0) == labels )
def test_smart_rbf():
import milksets.wine
features,labels = milksets.wine.load()
labels = (labels == 1)
kernel = milk.supervised.svm.rbf_kernel(2.**-4)
C = milk.supervised.svm.svm_raw(C=2.,kernel=kernel)
model = C.train(features,labels)
smartkernel = [model.apply(f) for f in features]
del kernel.kernel_nr_
del kernel.kernel_arg_
C = milk.supervised.svm.svm_raw(C=2.,kernel=kernel)
model = C.train(features,labels)
dumbkernel = [model.apply(f) for f in features]
np.allclose(smartkernel, dumbkernel)
def test_preprockernel():
def test_preproc(kernel):
X = np.random.rand(200,20)
Qs = np.random.rand(100, 20)
prekernel = kernel.preprocess(X)
Qs = np.random.rand(10, 20)
for q in Qs:
preprocval = prekernel(q)
procval = np.array([kernel(q,x) for x in X])
assert np.allclose(preprocval, procval)
yield test_preproc, milk.supervised.svm.rbf_kernel(2.)
def test_svm_to_binary():
base = milk.supervised.svm.svm_raw(kernel=np.dot, C=4.)
bin = milk.supervised.svm.svm_to_binary(base)
X = np.array([
[1,0],
[2,1],
[2,0],
[4,2],
[0,1],
[0,2],
[1,2],
[2,8]])
Y = np.array([1,1,1,1,-1,-1,-1,-1])
model = bin.train(X,Y)
modelbase = base.train(X,Y)
assert np.all(modelbase.Yw == model.models[0].Yw)
for x in X:
assert model.apply(x) in set(Y)
def test_fast_dotkernel():
base = milk.supervised.svm.svm_raw(kernel=np.dot, C=4.)
X = np.array([
[1,0],
[2,1],
[2,0],
[4,2],
[0,1],
[0,2],
[1,2],
[2,8]])
Y = np.array([1,1,1,1,-1,-1,-1,-1])
model = base.train(X,Y)
base = milk.supervised.svm.svm_raw(kernel = milk.supervised.svm.dot_kernel(), C = 4.)
model2 = base.train(X,Y)
assert np.all( model2.Yw == model.Yw )
def count_common_elements(s0,s1):
return len(s0.intersection(s1))
def test_not_ndarray_input():
elems = [
set([1,2,3]),
set([2,3]),
set([2,]),
set([0,1]),
set([0,2,3]),
set([1,2,3]),
set([2,3,45]),
set([1,2,100])]
has_one = [(1 in s) for s in elems]
has_one[0] = not has_one[0]
c = milk.supervised.svm.svm_raw(kernel = count_common_elements, C = 2.)
model = c.train(elems, has_one)
trainpreds = [model.apply(e) for e in elems]
assert np.mean((np.array(trainpreds) > 0 ) == has_one ) > .5
model = pickle.loads(pickle.dumps(model))
trainpreds = [model.apply(e) for e in elems]
assert np.mean((np.array(trainpreds) > 0 ) == has_one ) > .5
def test_rbf_kernel_call_many():
from milk.supervised.svm import rbf_kernel
np.random.seed(232)
X = np.random.random((32,16))
k = rbf_kernel(1.)
pre = k.preprocess(X)
qs = np.random.random((8,16))
mapped = np.array(map(pre, qs))
manyed = pre.call_many(qs)
assert np.allclose(manyed, mapped)
milk-0.5/milk/tests/test_som.py 0000664 0001750 0001750 00000003743 11723217323 020667 0 ustar luispedro luispedro 0000000 0000000 import numpy as np
import random
from milk.unsupervised import som
from milk.unsupervised.som import putpoints, closest
def _slow_putpoints(grid, points, L=.2):
for point in points:
dpoint = grid-point
y,x = np.unravel_index(np.abs(dpoint).argmin(), dpoint.shape)
for dy in xrange(-4, +4):
for dx in xrange(-4, +4):
ny = y + dy
nx = x + dx
if ny < 0 or ny >= grid.shape[0]:
continue
if nx < 0 or nx >= grid.shape[1]:
continue
L2 = L/(1+np.abs(dy)+np.abs(dx))
grid[ny,nx] *= 1. - L2
grid[ny,nx] += point*L2
def data_grid():
np.random.seed(22)
data = np.arange(100000, dtype=np.float32)
grid = np.array([data.flat[np.random.randint(0, data.size)] for i in xrange(64*64)]).reshape((64,64,1))
data = data.reshape((-1,1))
return grid, data
def test_putpoints():
grid, points = data_grid()
points = points[:100]
grid2 = grid.copy()
putpoints(grid, points, L=0., R=1)
assert np.all(grid == grid2)
putpoints(grid, points, L=.5, R=1)
assert not np.all(grid == grid2)
def test_against_slow():
grid, points = data_grid()
grid2 = grid.copy()
putpoints(grid, points[:10], shuffle=False)
_slow_putpoints(grid2.reshape((64,64)), points[:10])
assert np.allclose(grid, grid2)
def test_som():
N = 10000
np.random.seed(2)
data = np.array([np.arange(N), N/4.*np.random.randn(N)])
data = data.transpose().copy()
grid = som(data, (8,8), iterations=3, R=4)
assert grid.shape == (8,8,2)
y,x = closest(grid, data[0])
assert 0 <= y < grid.shape[0]
assert 0 <= x < grid.shape[1]
grid2 = grid.copy()
np.random.shuffle(grid2)
full = np.abs(np.diff(grid2[:,:,0], axis=0)).mean()
obs = np.abs(np.diff(grid[:,:,0], axis=0)).mean()
obs2 = np.abs(np.diff(grid[:,:,0], axis=1)).mean()
assert obs + 4*np.abs(obs-obs2) < full
milk-0.5/milk/tests/test_regression_constant_features.py 0000664 0001750 0001750 00000000373 11723217323 026054 0 ustar luispedro luispedro 0000000 0000000 import milk
import numpy as np
def test_constant_features():
learner = milk.defaultclassifier()
features = np.ones(20).reshape((-1,1))
labels = np.zeros(20)
labels[10:] += 1
features[10:] *= -1
learner.train(features, labels)
milk-0.5/milk/tests/test_fisher.py 0000664 0001750 0001750 00000002330 11723217323 021340 0 ustar luispedro luispedro 0000000 0000000 import milk.supervised.svm
import milk.supervised.normalise
import numpy as np
import milk.supervised.svm
def _slow_f(features,labels,kernel_or_sigma):
try:
kernel = kernel_or_sigma
kernel(features[0],features[1])
except:
kernel = milk.supervised.svm.rbf_kernel(kernel_or_sigma)
N1 = (labels == 0).sum()
N2 = (labels == 1).sum()
x1 = features[labels == 0]
x2 = features[labels == 1]
dm = 0
for i in xrange(N1):
for j in xrange(N1):
dm += kernel(x1[i],x1[j])/N1/N1
for i in xrange(N2):
for j in xrange(N2):
dm += kernel(x2[i],x2[j])/N2/N2
for i in xrange(N1):
for j in xrange(N2):
dm -= 2*kernel(x1[i],x2[j])/N1/N2
s1 = N1
for i in xrange(N1):
for j in xrange(N1):
s1 -= kernel(x1[i],x1[j])/N1
s2 = N2
for i in xrange(N2):
for j in xrange(N2):
s2 -= kernel(x2[i],x2[j])/N2
return (s1 + s2)/dm
def test_fisher_approx():
from milksets import wine
features,labels = wine.load()
f = milk.supervised.svm.sigma_value_fisher(features,labels)
for sigma in (2.**-4,2.,16.,32.):
assert abs(f(sigma) - _slow_f(features,labels,sigma)) < 1e-6
milk-0.5/milk/tests/test_affinity.py 0000664 0001750 0001750 00000000556 11723217323 021701 0 ustar luispedro luispedro 0000000 0000000 import milk.unsupervised.affinity
import numpy as np
def test_affinity():
np.random.seed(22)
X = np.random.randn(100,10)
X[:40] += .4
S = milk.unsupervised.pdist(X)
clusters, labels = milk.unsupervised.affinity.affinity_propagation(S)
assert labels.max()+1 == len(clusters)
assert len(labels) == len(X)
assert clusters.max() < len(X)
milk-0.5/milk/tests/test_logistic.py 0000664 0001750 0001750 00000000604 11723217323 021677 0 ustar luispedro luispedro 0000000 0000000 import milk.supervised.logistic
import milksets.wine
import numpy as np
def test_better_than_random():
learner = milk.supervised.logistic.logistic_learner()
features, labels = milksets.wine.load()
model = learner.train(features, labels == 0)
error = np.array([np.abs(model.apply(f)-(l == 0))
for f,l in zip(features, labels)])
assert error.mean() < .1
milk-0.5/milk/tests/test_nfoldcrossvalidation_regression.py 0000664 0001750 0001750 00000023507 11723217323 026560 0 ustar luispedro luispedro 0000000 0000000 import numpy as np
from milk.measures.nfoldcrossvalidation import nfoldcrossvalidation, foldgenerator
# Regression test in 2011-01-31
def test_getfoldgenerator():
labels = np.array([
7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,
7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,
7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,
7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,
7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,
6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6,
6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6,
6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6,
6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6,
6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6,
6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6,
6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6,
6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6,
6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6,
6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6,
6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6,
6, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
5, 5, 5, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
])
origins = np.array([
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 6,
6, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 8, 8, 8, 8, 8, 8,
8, 8, 8, 8, 8, 8, 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 10, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13,
13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 14, 14, 14, 14, 14, 14,
14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14,
14, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15,
15, 15, 15, 15, 15, 15, 15, 15, 15, 16, 16, 16, 16, 16, 16, 16, 16,
16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16,
17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17,
17, 17, 17, 17, 17, 17, 17, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18,
18, 18, 18, 18, 18, 18, 18, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19,
19, 19, 19, 19, 19, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21,
21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 22, 22, 22, 22,
22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22,
22, 22, 22, 22, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23,
23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 24, 24, 24, 24, 24,
24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 25, 25, 25, 25, 25, 25, 25,
25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25,
26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26,
26, 26, 26, 26, 26, 26, 26, 26, 27, 27, 27, 27, 27, 27, 27, 27, 27,
27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 28, 28,
28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28,
28, 28, 28, 28, 28, 28, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29,
29, 29, 29, 29, 29, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30,
30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 31, 31, 31, 31, 31,
31, 31, 31, 31, 31, 31, 31, 31, 31, 32, 32, 32, 32, 32, 32, 32, 32,
33, 33, 33, 33, 33, 33, 33, 33, 33, 33, 33, 33, 33, 34, 34, 34, 34,
34, 34, 34, 34, 34, 34, 34, 34, 34, 34, 35, 35, 35, 35, 35, 35, 35,
35, 35, 35, 35, 35, 35, 35, 35, 35, 35, 35, 35, 35, 35, 35, 35, 36,
36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 37, 37, 37, 37,
37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 38, 38, 38, 38, 38, 38, 38,
38, 38, 38, 38, 38, 38, 38, 38, 39, 39, 39, 39, 39, 39, 39, 39, 39,
39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 40, 40,
40, 40, 40, 40, 40, 40, 40, 40, 40, 41, 41, 41, 41, 41, 41, 41, 41,
41, 41, 41, 41, 41, 41, 41, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42,
42, 42, 42, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43,
43, 43, 43, 43, 43, 43, 43, 43, 44, 44, 44, 44, 44, 44, 44, 44, 44,
44, 44, 44, 44, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45,
45, 45, 46, 46, 46, 46, 46, 46, 46, 46, 46, 46, 46, 46, 46, 46, 46,
47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47,
47, 47, 47, 47, 47, 47, 47, 47, 48, 48, 48, 48, 48, 48, 48, 48, 48,
48, 48, 49, 49, 49, 49, 49, 49, 49, 49, 49, 49, 49, 50, 50, 50, 50,
50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50,
50, 50, 50, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51,
51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 52, 52, 52, 52, 52, 52, 52,
52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52,
52, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53,
53, 53, 53, 53, 53, 53, 53, 53, 53, 54, 54, 54, 54, 54, 54, 54, 54,
54, 54, 54, 54, 54, 54, 54, 54, 54, 54, 54, 54, 54, 54, 54, 54, 55,
55, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55,
55, 55, 55, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56,
56, 56, 56, 56, 56, 56, 56, 56, 56, 56
])
for Tr,Te in foldgenerator(labels, 3, origins):
assert (np.array(labels)[Te] == 2).any()
for Tr,Te in foldgenerator(labels[::2], 3, origins[::2]):
assert (np.array(labels[::2])[Te] == 2).any()
for Tr,Te in foldgenerator(labels[::3], 3, origins[::3]):
assert (np.array(labels[::3])[Te] == 2).any()
def test_getfoldgenerator_simplified():
# This is a cut-down version of the above
labels = np.zeros(45, bool)
labels[35:] = 1
origins = np.array([0, 0, 1, 2, 2, 2, 3, 4, 4, 4, 5, 6, 6, 6,
6, 7, 7, 7, 8, 8, 8, 8, 9, 9, 9, 10, 10, 10, 10, 11, 11,
11, 12, 12, 12, 13, 13, 14, 14, 14, 14, 15, 15, 15, 16])
for Tr,Te in foldgenerator(labels, 3, origins):
assert np.any(labels[Te])
for Tr,Te in foldgenerator(labels, 4, origins):
assert np.any(labels[Te])
def test_getfoldgenerator_simplified_2():
# This is a cut-down version of the above
labels = np.zeros(44, bool)
labels[35:] = 1
origins = np.array([0, 0, 1, 2, 2, 2, 3, 4, 4, 4, 5, 6, 6, 6,
6, 7, 7, 7, 8, 8, 8, 8, 9, 9, 9, 10, 10, 10, 10, 11, 11,
11, 12, 12, 12, 13, 13, 14, 14, 14, 14, 15, 15, 15])
for Tr,Te in foldgenerator(labels, 3, origins):
assert np.any(labels[Te])
milk-0.5/milk/tests/test_gridsearch.py 0000664 0001750 0001750 00000007171 11723217323 022203 0 ustar luispedro luispedro 0000000 0000000 import milk.supervised.gridsearch
import milk.supervised.svm
from milk.supervised.gridsearch import gridminimise, _allassignments, gridsearch
from milk.tests.fast_classifier import fast_classifier
from nose.tools import raises
import numpy as np
def slow_gridminimise(learner, features, labels, params, measure=None):
from ..measures.nfoldcrossvalidation import nfoldcrossvalidation
if measure is None:
measure = np.trace
best_val = initial_value
best = None
for assignement in _allassignments(params):
_set_assignment(learner, assignement)
S,_ = nfoldcrossvalidation(features, labels, classifier=learner)
cur = measure(S)
if cur > best_val:
best = assignement
best_val = cur
return best
def test_gridsearch():
from milksets import wine
features, labels = wine.load()
selected = (labels < 2)
features = features[selected]
labels = labels[selected]
G = milk.supervised.gridsearch(
milk.supervised.svm.svm_raw(),
params={'C':[.01,.1,1.,10.],
'kernel':[milk.supervised.svm.rbf_kernel(0.1),milk.supervised.svm.rbf_kernel(1.)]
})
model = G.train(features,labels)
reslabels = [model.apply(f) for f in features]
assert len(reslabels) == len(features)
test_gridsearch.slow = True
def test_all_assignements():
assert len(list(_allassignments({'C': [0,1], 'kernel' : ['a','b','c']}))) == 2 * 3
class error_learner(object):
def train(self, features, labels, **kwargs):
raise ValueError('oops')
def set_option(self, k, v):
pass
@raises(Exception)
def test_with_error():
from milksets.wine import load
features, labels = load()
learner = error_learner()
G = milk.supervised.gridsearch(
error_learner(),
params = { 'error' : range(3), 'error2' : range(5) }
)
G.train(features,labels)
class simple_model:
def __init__(self, c):
self.c = c
def apply(self, f):
return self.c
def f(a,b,c):
return a**2 + b**3 + c
class simple_learner:
def set_option(self, k, v):
setattr(self, k, v)
def train(self, fs, ls, normalisedlabels=False):
return simple_model(f(self.a, self.b, self.c))
def test_gridminimise():
features = np.arange(100)
labels = np.tile((0,1), 50)
paramspace = { 'a': np.arange(4), 'b' : np.arange(-3,3), 'c' : np.linspace(2., 10) }
best,value = gridminimise(simple_learner(), features, labels, paramspace, measure=(lambda _, p: p[0]), return_value=True)
best = dict(best)
val = f(best['a'], best['b'], best['c'])
assert value == val*100
for a in np.arange(4):
for b in np.arange(-3,3):
for c in np.linspace(2., 10):
assert val <= f(a,b,c)
gs = gridsearch(simple_learner(), paramspace, measure=(lambda _, p: p[0]), annotate=True)
model = gs.train(features, labels)
assert model.value == value
assert model.arguments == val
def test_gridminimise():
from milksets.wine import load
features, labels = load()
x = gridminimise(milk.supervised.svm_simple(kernel=np.dot, C=2.), features[::2], labels[::2] == 0, {'C' : (0.5,) })
cval, = x
assert cval == ('C', .5)
def test_gridminimise_return():
from milksets.wine import load
features,labels = load()
learner = fast_classifier()
gridminimise(learner, features, labels, { 'ignore' : [0] })
_,error = gridminimise(learner, features, labels, { 'ignore' : [0] }, return_value=True, nfolds=5)
cmat,_ = milk.nfoldcrossvalidation(features, labels, learner=learner, nfolds=5)
assert error == cmat.sum()-cmat.trace()
milk-0.5/milk/tests/test_grouped.py 0000664 0001750 0001750 00000004624 11723217323 021535 0 ustar luispedro luispedro 0000000 0000000 import numpy as np
import milk.supervised.svm
from milk.supervised.svm import rbf_kernel
import milk.supervised.multi
import milk.supervised.grouped
from milk.supervised.classifier import ctransforms
import milksets.wine
def group(features, labels, step):
N = len(labels)
i = 0
gfeatures = []
glabels = []
while i < N:
next = i + step
while next > N or labels[next-1] != labels[i]: next -= 1
gfeatures.append(features[i:next])
glabels.append(labels[i])
i = next
return gfeatures, glabels
def test_voting():
base = ctransforms(milk.supervised.svm.svm_raw(C=2.,kernel=milk.supervised.svm.rbf_kernel(2.**-3)),milk.supervised.svm.svm_binary())
base = milk.supervised.multi.one_against_rest(base)
features,labels = milksets.wine.load()
gfeatures, glabels = group(features, labels, 3)
learner = milk.supervised.grouped.voting_classifier(base)
learner.train(gfeatures, glabels)
model = learner.train(gfeatures, glabels)
assert ([model.apply(f) for f in gfeatures] == np.array(glabels)).mean() > .8
def test_filter_outliers():
np.random.seed(22)
features = [np.random.randn(10,10) for i in xrange(20)]
for f in features:
f[0] *= 10
trainer = milk.supervised.grouped.filter_outliers(.9)
model = trainer.train(features, [0] * len(features))
for f in features:
ff = model.apply(f)
assert np.all(ff == f[1:])
def test_nfoldcrossvalidation():
np.random.seed(22)
features = np.array([np.random.rand(8+(i%3), 12)*(i//20) for i in xrange(40)], dtype=object)
labels = np.zeros(40, int)
labels[20:] = 1
classifier = milk.supervised.grouped.voting_classifier(milk.supervised.svm_simple(C=1., kernel=rbf_kernel(1./12)))
cmat, names = milk.nfoldcrossvalidation(features, labels, classifier=classifier)
assert cmat.shape == (2,2)
assert sorted(names) == range(2)
class identity_classifier(object):
def train(self, features, labels):
return identity_model()
class identity_model(object):
def apply(self, f):
return f
def test_meanclassif():
gfeatures = [np.arange(10), np.arange(10)%2]
glabels = [0,1]
meanclassif = milk.supervised.grouped.mean_classifier(identity_classifier())
model = meanclassif.train(gfeatures, glabels)
assert model.apply(gfeatures[0]) == np.arange(10).mean()
assert model.apply(gfeatures[1]) == .5
milk-0.5/milk/tests/test_rf.py 0000664 0001750 0001750 00000000606 11723217323 020473 0 ustar luispedro luispedro 0000000 0000000 from milk.supervised import randomforest
import numpy as np
def test_rf():
from milksets import wine
features, labels = wine.load()
features = features[labels < 2]
labels = labels[labels < 2]
learner = randomforest.rf_learner()
model = learner.train(features[::5], labels[::5])
test = [model.apply(f) for f in features]
assert np.mean(labels == test) > .7
milk-0.5/milk/tests/test_pca.py 0000664 0001750 0001750 00000001405 11734654046 020636 0 ustar luispedro luispedro 0000000 0000000 import numpy.random
import milk.unsupervised.pca
import numpy as np
def test_pca():
numpy.random.seed(123)
X = numpy.random.rand(10,4)
X[:,1] += numpy.random.rand(10)**2*X[:,0]
X[:,1] += numpy.random.rand(10)**2*X[:,0]
X[:,2] += numpy.random.rand(10)**2*X[:,0]
Y,V = milk.unsupervised.pca(X)
Xn = milk.unsupervised.normalise.zscore(X)
assert X.shape == Y.shape
assert ((np.dot(V[:4].T,Y[:,:4].T).T-Xn)**2).sum()/(Xn**2).sum() < .3
def test_mds():
from milk.unsupervised import pdist
np.random.seed(232)
for _ in xrange(12):
features = np.random.random_sample((12,4))
X = milk.unsupervised.mds(features,4)
D = pdist(features)
D2 = pdist(X)
assert np.mean( (D - D2) ** 2) < 10e-4
milk-0.5/milk/tests/test_normalise.py 0000664 0001750 0001750 00000006246 12032626111 022054 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2012, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
from __future__ import division
import numpy
import numpy as np
from milk.supervised.normalise import sample_to_2min
import milk.supervised.normalise
def test_zscore_normalise():
I=milk.supervised.normalise.zscore_normalise()
numpy.random.seed(1234)
features = numpy.random.rand(20,100)
L = numpy.zeros(100)
model = I.train(features, L)
transformed = np.array([model.apply(f) for f in features])
assert np.all( transformed.mean(0)**2 < 1e-7 )
assert np.all( np.abs(transformed.std(0) - 1) < 1e-3 )
def test_sample_to_2min():
A = np.zeros(256, np.int32)
def test_one(A):
selected = sample_to_2min(A)
ratios = []
for l0 in set(A):
for l1 in set(A):
ratios.append( (A[selected] == l0).sum() / (A[selected] == l1).sum() )
assert np.max(ratios) <= 2.001
A[20:] = 1
yield test_one, A
A[21:] = 1
yield test_one, A
A[129:] = 2
yield test_one, A
def test_sample_to_2min_list():
from collections import defaultdict
def count(xs):
counts = defaultdict(int)
for x in xs:
counts[x] += 1
return counts
labels = ["A"]*8 + ["B"]*12 + ["C"]*16 + ["D"] * 24 + ["E"] * 1000
selected = sample_to_2min(labels)
before = count(labels)
after = count(np.array(labels)[selected])
assert max(after.values()) == min(before.values())*2
def test_interval_normalise():
interval = milk.supervised.normalise.interval_normalise()
np.random.seed(105)
features = np.random.randn(100, 5)
model = interval.train(features, features[0] > 0)
transformed = np.array([model.apply(f) for f in features])
assert np.allclose(transformed.min(0), -1)
assert np.allclose(transformed.max(0), +1)
def test_nanstd():
from milk.unsupervised.normalise import _nanstd
np.random.seed(234)
for i in xrange(8):
x = np.random.rand(200,231)
np.allclose(_nanstd(x,0), x.std(0))
np.allclose(_nanstd(x,1), x.std(1))
milk-0.5/milk/tests/test_adaboost.py 0000664 0001750 0001750 00000002011 12014503604 021642 0 ustar luispedro luispedro 0000000 0000000 import numpy as np
import milk.supervised.tree
import milk.supervised.adaboost
def test_learner():
from milksets import wine
learner = milk.supervised.adaboost.boost_learner(milk.supervised.tree.stump_learner())
features, labels = wine.load()
features = features[labels < 2]
labels = labels[labels < 2] == 0
labels = labels.astype(int)
model = learner.train(features[::2], labels[::2])
train_out = np.array(map(model.apply, features))
assert (train_out == labels).mean() > .9
def test_too_many_boolean_indices_regression():
import milk.supervised.randomforest
import milk.supervised.adaboost
import milksets.wine
from milk.supervised.multi import one_against_one
weak = milk.supervised.randomforest.rf_learner()
learner = milk.supervised.adaboost.boost_learner(weak)
learner = one_against_one(learner)
features, labels = milksets.wine.load()
# sample features so that the test is faster (still gives error):
learner.train(features[::16], labels[::16])
milk-0.5/milk/tests/test_multi_label.py 0000664 0001750 0001750 00000002372 11723217323 022357 0 ustar luispedro luispedro 0000000 0000000 from milk.tests.fast_classifier import fast_classifier
import milk.supervised.multi_label
import milk
import numpy as np
def test_one_by_one():
np.random.seed(23)
r = np.random.random
ps = np.array([.7,.5,.8,.3,.8])
learner = milk.supervised.multi_label.one_by_one(fast_classifier())
universe = range(len(ps))
for _ in xrange(10):
labels = []
features = []
bases = [np.random.rand(20) for pj in ps]
for i in xrange(256):
cur = []
curf = np.zeros(20,float)
for j,pj in enumerate(ps):
if r() < pj:
cur.append(j)
curf += r()*bases[j]
if not cur: continue
labels.append(cur)
features.append(curf)
model = learner.train(features, labels)
predicted = model.apply_many(features)
matrix = np.zeros((2,2), int)
for t,p in zip(labels, predicted):
for ell in universe:
row = (ell in t)
col = (ell in p)
matrix[row,col] += 1
Tn,Fp = matrix[0]
Fn,Tp = matrix[1]
prec = Tp/float(Tp+Fp)
recall = Tp/float(Tp+Fn)
F1 = 2*prec*recall/(prec + recall)
assert F1 > .3
milk-0.5/milk/tests/test_svm_sigmoidal.py 0000664 0001750 0001750 00000004741 11723217323 022725 0 ustar luispedro luispedro 0000000 0000000 import numpy
from milk.supervised import svm
import numpy
import numpy as np
def old_learn_sigmoid_constants(F,Y,
max_iters=None,
min_step=1e-10,
sigma=1e-12,
eps=1e-5):
'''
Old version. Direct C-like implementation
'''
# the deci[i] array is called F[i] in this code
F = np.asanyarray(F)
Y = np.asanyarray(Y)
assert len(F) == len(Y)
assert numpy.all( (Y == 1) | (Y == 0) )
from numpy import log, exp
N=len(F)
if max_iters is None: max_iters = 1000
prior1 = Y.sum()
prior0 = N-prior1
small_nr = 1e-4
hi_t = (prior1+1.)/(prior1+2.)
lo_t = 1./(prior0+2.)
T = Y*hi_t + (1-Y)*lo_t
A = 0.
B = log( (prior0+1.)/(prior1+1.) )
def target(A,B):
fval = 0.
for i in xrange(N):
fApB = F[i]*A+B
if fApB >= 0:
fval += T[i]*fApB+log(1+exp(-fApB))
else:
fval += (T[i]-1.)*fApB+log(1+exp(fApB))
return fval
fval = target(A,B)
for iter in xrange(max_iters):
h11=sigma
h22=sigma
h21=0.
g1=0.
g2=0.
for i in xrange(N):
fApB = F[i]*A+B
if (fApB >= 0):
p = exp(-fApB)/(1.+exp(-fApB))
q = 1./(1.+exp(-fApB))
else:
p = 1./(1.+exp(fApB))
q = exp(fApB)/(1.+exp(fApB))
d2 = p * q
h11 += F[i]*F[i]*d2
h22 += d2
h21 += F[i]*d2
d1 = T[i] - p
g1 += F[i]*d1
g2 += d1
if abs(g1) < eps and abs(g2) < eps: # Stopping criteria
break
det = h11*h22 - h21*h21
dA = - (h22*g1 - h21*g2)/det
dB = - (h21*g1 + h11*g2)/det
gd = g1*dA + g2*dB
stepsize = 1.
while stepsize >= min_step:
newA = A + stepsize*dA
newB = B + stepsize*dB
newf = target(newA,newB)
if newf < fval+eps*stepsize*gd:
A = newA
B = newB
fval = newf
break
stepsize /= 2
if stepsize < min_step:
break
return A,B
def test_learn_sigmoid_contants():
Y = np.repeat((0,1),100)
np.random.seed(3)
for i in xrange(10):
F = np.random.rand(200)-.3
F[100:] *= -1
old = old_learn_sigmoid_constants(F,Y)
new = svm.learn_sigmoid_constants(F,Y)
assert np.allclose(old, new)
milk-0.5/milk/tests/test_curves.py 0000664 0001750 0001750 00000001106 11723217323 021367 0 ustar luispedro luispedro 0000000 0000000 from milk.measures.curves import precision_recall
import numpy as np
def test_precision_recall():
labels = [0,1]*10
values = np.linspace(0,1,len(labels))
precision, recall = precision_recall(values, labels)
assert np.min(recall) >= 0.
assert np.max(recall) <= 1.
assert np.max(precision) <= 1.
assert np.min(precision) >= 0.
labels = [0]*10 + [1] * 10
values = np.linspace(0,1.,20)
precision,recall = precision_recall(values, labels, 'steps', 10)
assert min(precision) >= .5
assert max(precision) == 1.
assert max(recall) == 1.
milk-0.5/milk/tests/test_basic.py 0000664 0001750 0001750 00000000043 11723217323 021140 0 ustar luispedro luispedro 0000000 0000000 def test_import():
import milk
milk-0.5/milk/tests/test_regression.py 0000664 0001750 0001750 00000003050 11723217323 022240 0 ustar luispedro luispedro 0000000 0000000 import pickle
import numpy as np
import milk.supervised._svm
from gzip import GzipFile
from os import path
import numpy as np
from milksets.wine import load
from milk.supervised import defaultclassifier
import milk
def test_svm_crash():
X,Y,kernel, C, eps ,tol, = pickle.load(GzipFile(path.dirname(__file__) + '/data/regression-2-Dec-2009.pp.gz'))
X = X[2:-2,:].copy()
Y = Y[2:-2].copy()
N = len(Y)
Y = Y.astype(np.int32)
p = -np.ones(N,np.double)
params = np.array([0,C,eps,tol],np.double)
Alphas0 = np.zeros(N, np.double)
cache_size = (1<<20)
# The line below crashed milk:
milk.supervised._svm.eval_LIBSVM(X,Y,Alphas0,p,params,kernel,cache_size)
# HASN'T CRASHED!
def test_nov2010():
# Bug submitted by Mao Ziyang
# This was failing in 0.3.5 because SDA selected no features
np.random.seed(222)
features = np.random.randn(100,20)
features[:50] *= 2
labels = np.repeat((0,1), 50)
classifier = milk.defaultclassifier()
model = classifier.train(features, labels)
new_label = model.apply(np.random.randn(20)*2)
new_label2 = model.apply(np.random.randn(20))
assert new_label == 0
assert new_label2 == 1
def test_default_small():
features, labels = load()
selected = np.concatenate( [np.where(labels < 2)[0], np.where(labels == 2)[0][:6]] )
features = features[selected]
labels = labels[selected]
learner = defaultclassifier('fast')
# For version 0.3.8, the line below led to an error
milk.nfoldcrossvalidation(features, labels, classifier=learner)
milk-0.5/milk/tests/test_multi_view.py 0000664 0001750 0001750 00000002166 11723217323 022253 0 ustar luispedro luispedro 0000000 0000000 import milk.supervised.multi_view
import numpy as np
import milk.supervised.svm
from milk.supervised.defaultclassifier import feature_selection_simple
def test_multi_view():
from milksets.wine import load
features, labels = load()
features0 = features[::10]
features1 = features[1::10]
features2 = features[2::10]
labels0 = labels[::10]
labels1 = labels[1::10]
labels2 = labels[2::10]
assert np.all(labels0 == labels1)
assert np.all(labels1 == labels2)
labels = labels0
train_features = zip(features0,features1,features2)
test_features = zip(features[3::10], features[4::10], features[5::10])
base = milk.supervised.classifier.ctransforms(
feature_selection_simple(),
milk.supervised.svm.svm_raw(C=128, kernel=milk.supervised.svm.rbf_kernel(4.)),
milk.supervised.svm.svm_sigmoidal_correction()
)
classifier = milk.supervised.multi_view.multi_view_classifier([base,base,base])
model = classifier.train(train_features, labels == 0)
assert ([model.apply(f) for f in test_features] == (labels == 0)).mean() > .9
milk-0.5/milk/tests/test_measures_clusters.py 0000664 0001750 0001750 00000001025 11723217323 023630 0 ustar luispedro luispedro 0000000 0000000 import milk.measures.cluster_agreement
import numpy as np
def test_rand_arand_jaccard():
np.random.seed(33)
labels = np.repeat(np.arange(4),10)
clusters = np.repeat(np.arange(4),10)
a0,b0,c0= milk.measures.cluster_agreement.rand_arand_jaccard(clusters, labels)
assert a0 == 1.
assert b0 == 1.
np.random.shuffle(clusters)
a1,b1,c1= milk.measures.cluster_agreement.rand_arand_jaccard(clusters, labels)
assert a1 >= 0.
assert a1 < 1.
assert b1 < 1.
assert b1 >= 0.
assert c1 < c0
milk-0.5/milk/tests/test_lasso.py 0000664 0001750 0001750 00000004004 12030616107 021174 0 ustar luispedro luispedro 0000000 0000000 from milk.supervised.lasso import lasso_learner
import milk.supervised.lasso
import numpy as np
def test_lasso_smoke():
np.random.seed(3)
for i in xrange(8):
X = np.random.rand(100,10)
Y = np.random.rand(5,10)
B = np.random.rand(5,100)
before = np.linalg.norm(Y - np.dot(B,X))
B = milk.supervised.lasso(X,Y)
after = np.linalg.norm(Y - np.dot(B,X))
assert after < before
assert np.all(~np.isnan(B))
def test_lasso_nans():
np.random.seed(3)
for i in xrange(8):
X = np.random.rand(100,10)
Y = np.random.rand(5,10)
B = np.random.rand(5,100)
for j in xrange(12):
Y.flat[np.random.randint(0,Y.size-1)] = float('nan')
B = milk.supervised.lasso(X,Y)
assert np.all(~np.isnan(B))
def test_lam_zero():
np.random.seed(2)
for i in xrange(8):
X = np.random.rand(24,2)
Y = np.random.rand(1,2)
B = milk.supervised.lasso(X,Y, lam=0.0)
R = Y - np.dot(B,X)
R = R.ravel()
assert np.dot(R,R) < .01
def test_lasso_walk():
np.random.seed(5)
for i in xrange(4):
X = np.random.rand(100,10)
Y = np.random.rand(5,10)
Bs = milk.supervised.lasso_walk(X,Y, start=.0001, nr_steps=3)
B0 = milk.supervised.lasso(X,Y, lam=.0001)
assert np.all(Bs[0] == B0)
assert not np.all(Bs[0] == Bs[-1])
assert len(Bs) == 3
def test_lasso_walk_nans():
np.random.seed(5)
for i in xrange(3):
X = np.random.rand(100,10)
Y = np.random.rand(5,10)
B = np.random.rand(5,100)
for j in xrange(12):
Y.flat[np.random.randint(0,Y.size-1)] = float('nan')
B = milk.supervised.lasso_walk(X,Y, nr_steps=6)
assert np.all(~np.isnan(B))
def test_learner():
np.random.seed(334)
learner = lasso_learner()
X = np.random.rand(100,10)
Y = np.random.rand(5,10)
model = learner.train(X,Y)
test = model.apply(np.random.rand(100))
assert len(test) == len(Y)
milk-0.5/milk/tests/test_knn.py 0000664 0001750 0001750 00000001551 11723217323 020652 0 ustar luispedro luispedro 0000000 0000000 import numpy as np
import milk.supervised.knn
import numpy
def test_simple():
X=np.array([
[0,0,0],
[1,1,1],
])
Y=np.array([ 1, -1 ])
kNN = milk.supervised.knn.kNN(1)
kNN = kNN.train(X,Y)
assert kNN.apply(X[0]) == Y[0]
assert kNN.apply(X[1]) == Y[1]
assert kNN.apply([0,0,1]) == Y[0]
assert kNN.apply([0,1,1]) == Y[1]
def test_nnclassifier():
labels=[0,1]
data=[[0.,0.],[1.,1.]]
C = milk.supervised.knn.kNN(1)
model = C.train(data,labels)
assert model.apply(data[0]) == 0
assert model.apply(data[1]) == 1
assert model.apply([.01,.01]) == 0
assert model.apply([.99,.99]) == 1
assert model.apply([100,100]) == 1
assert model.apply([-100,-100]) == 0
assert model.apply([.9,.9]) == 1
middle = model.apply([.5,.5])
assert (middle == 0) or (middle == 1)
milk-0.5/milk/tests/test_gaussianmixture.py 0000664 0001750 0001750 00000002217 11723217323 023314 0 ustar luispedro luispedro 0000000 0000000 import numpy as np
from milk.unsupervised import gaussianmixture
def _sq(x):
return x*x
def test_gm():
np.random.seed(22)
centroids = np.repeat(np.arange(4), 4).reshape((4,4))
fmatrix = np.concatenate([(np.random.randn(12,4)+c) for c in centroids])
assignments = np.repeat(np.arange(4), 12)
rss = sum(np.sum(_sq(fmatrix[i*12:(i+1)*12]-i)) for i in xrange(4))
assert np.abs(gaussianmixture.residual_sum_squares(fmatrix, assignments, centroids) - rss) < 1.e-12
assert gaussianmixture.BIC(fmatrix, assignments, centroids) > 0
assert gaussianmixture.AIC(fmatrix, assignments, centroids) > 0
assert gaussianmixture.BIC(fmatrix, assignments, centroids, model='full_covariance') > \
gaussianmixture.BIC(fmatrix, assignments, centroids, model='diagonal_covariance') > \
gaussianmixture.BIC(fmatrix, assignments, centroids, model='one_variance')
assert gaussianmixture.AIC(fmatrix, assignments, centroids, model='full_covariance') > \
gaussianmixture.AIC(fmatrix, assignments, centroids, model='diagonal_covariance') > \
gaussianmixture.AIC(fmatrix, assignments, centroids, model='one_variance')
milk-0.5/milk/tests/fast_classifier.py 0000664 0001750 0001750 00000001471 12043765776 022207 0 ustar luispedro luispedro 0000000 0000000 import numpy as np
from milk.supervised.base import supervised_model
class fast_classifier(object):
def __init__(self):
pass
def set_option(self, _k, _v):
pass
def train(self, features, labels, **kwargs):
examples = {}
for f,lab in zip(features, labels):
if lab not in examples:
examples[lab] = f
return fast_model(examples)
class fast_model(supervised_model):
def __init__(self, examples):
self.examples = examples
assert len(self.examples)
def apply(self, f):
best = None
best_val = +np.inf
for k,v in self.examples.iteritems():
d = v-f
dist = np.dot(d,d)
if dist < best_val:
best = k
best_val = dist
return best
milk-0.5/milk/tests/__init__.py 0000664 0001750 0001750 00000000664 11723217323 020570 0 ustar luispedro luispedro 0000000 0000000 try:
import milksets
del milksets
except ImportError:
import sys
sys.stderr.write('''\
Could not import milksets.
This companion package does not provide any functionality, but
is necessary for some of the testing.''')
def run():
import nose
from os import path
currentdir = path.dirname(__file__)
updir = path.join(currentdir, '..')
nose.run('milk', argv=['', '--exe', '-w', updir])
milk-0.5/milk/tests/test_utils.py 0000664 0001750 0001750 00000001760 11723217323 021226 0 ustar luispedro luispedro 0000000 0000000 import numpy as np
from milk.utils.utils import get_nprandom, get_pyrandom
def test_nprandom():
assert get_nprandom(None).rand() != get_nprandom(None).rand()
assert get_nprandom(1).rand() != get_nprandom(2).rand()
assert get_nprandom(1).rand() == get_nprandom(1).rand()
r = get_nprandom(1)
assert get_nprandom(r).rand() != r.rand()
def test_pyrandom():
assert get_pyrandom(None).random() != get_pyrandom(None).random()
assert get_pyrandom(1).random() != get_pyrandom(2).random()
assert get_pyrandom(1).random() == get_pyrandom(1).random()
r = get_pyrandom(1)
assert get_pyrandom(r).random() != r.random()
def test_cross_random():
assert get_pyrandom(get_nprandom(1)).random() == get_pyrandom(get_nprandom(1)).random()
assert get_nprandom(get_pyrandom(1)).rand() == get_nprandom(get_pyrandom(1)).rand()
def test_recursive():
def recurse(f):
R = f(None)
assert f(R) is R
yield recurse, get_pyrandom
yield recurse, get_nprandom
milk-0.5/milk/tests/test_nfoldcrossvalidation.py 0000664 0001750 0001750 00000012100 11723217323 024303 0 ustar luispedro luispedro 0000000 0000000 import milk.measures.nfoldcrossvalidation
from milk.measures.nfoldcrossvalidation import nfoldcrossvalidation, foldgenerator
import milk.supervised.tree
import numpy as np
from fast_classifier import fast_classifier
def test_foldgenerator():
labels = np.array([1]*20+[2]*30+[3]*20)
for nf in [None,2,3,5,10,15,20]:
assert np.array([test.copy() for _,test in foldgenerator(labels,nf)]).sum(0).max() == 1
assert np.array([test.copy() for _,test in foldgenerator(labels,nf)]).sum() == len(labels)
assert np.array([(test&train).sum() for train,test in foldgenerator(labels,nf)]).sum() == 0
def test_foldgenerator_not_empty():
for nf in (None, 2, 3, 5, 10, 15, 20):
for Tr,Te in foldgenerator([0] * 10 + [1] *10, nf, None):
assert not np.all(Tr)
assert not np.all(Te)
def test_nfoldcrossvalidation_simple():
from milksets import wine
features, labels = wine.load()
features = features[::2]
labels = labels[::2]
cmat,clabels = nfoldcrossvalidation(features, labels, classifier=fast_classifier())
assert cmat.shape == (3,3)
assert len(clabels) == 3
def test_nfoldcrossvalidation_simple_list():
from milksets import wine
features, labels = wine.load()
features = features[::2]
labels = labels[::2]
cmat,clabels = nfoldcrossvalidation(list(features), list(labels), classifier=fast_classifier())
assert cmat.shape == (3,3)
assert len(clabels) == 3
class test_classifier(object):
def __init__(self,N):
self.tested = np.zeros(N,bool)
self.N = N
def apply(self, features):
cur = np.zeros(self.N,bool)
cur[features] = True
assert not np.any(cur & self.tested)
self.tested |= cur
return np.zeros_like(features)
def train(self,f,l):
return self
def test_nfoldcrossvalidation_testall():
N = 121
C = test_classifier(N)
features = np.arange(N)
labels = np.zeros(N)
cmat,clabels = nfoldcrossvalidation(features, labels, classifier=C)
assert np.all(C.tested)
def test_getfold():
A = np.zeros(20)
A[:10] = 1
t,s = milk.measures.nfoldcrossvalidation.getfold(A,0,10)
tt,ss = milk.measures.nfoldcrossvalidation.getfold(A,1,10)
assert not np.any((~t)&(~tt))
def test_nfoldcrossvalidation_defaultclassifier():
np.random.seed(2233)
X = np.random.rand(60,5)
X[:20] += 4.
X[-20:] -= 4.
Y = np.ones(60)
Y[:20] = 0
Y[-20:] = 2
Y += 100
cmat,clabels = milk.measures.nfoldcrossvalidation.nfoldcrossvalidation(X,Y)
assert cmat.shape == (3,3)
clabels.sort()
assert np.all(clabels == [100,101,102])
def test_foldgenerator_origins():
def test_origins(labels, origins):
for nf in (2,3,5,7):
assert np.array([test.copy() for _,test in foldgenerator(labels, nf, origins)]).sum(0).max() == 1
assert np.array([test.copy() for _,test in foldgenerator(labels, nf, origins)]).sum() == len(labels)
assert np.min([test.sum() for _,test in foldgenerator(labels, nf, origins)]) > 0
assert np.min([train.sum() for train,_ in foldgenerator(labels, nf, origins)]) > 0
for Tr,Te in foldgenerator(labels, nf, origins):
assert not np.any(Tr&Te)
in_test = set(origins[Te])
in_train = set(origins[Tr])
assert len(in_train.intersection(in_test)) == 0
tested = np.zeros(len(labels))
for Tr,Te in foldgenerator(labels, nf, origins):
tested[Te] += 1
assert np.all(tested == 1)
labels = np.zeros(120, np.uint8)
labels[39:] += 1
labels[66:] += 1
origins = np.repeat(np.arange(40), 3)
yield test_origins, labels, origins
reorder = np.argsort(np.random.rand(len(labels)))
labels = labels[reorder]
origins = origins[reorder]
yield test_origins, labels, origins
def test_stringlabels():
np.random.seed(222)
D = np.random.rand(100,10)
D[:40] += np.random.rand(40,10)**2
labelnames = ['one'] * 40 + ['two'] * 60
cmat,Lo = nfoldcrossvalidation(D, labelnames, classifier=fast_classifier())
assert Lo[0] in labelnames
assert Lo[1] in labelnames
assert Lo[0] != Lo[1] in labelnames
def test_predictions():
np.random.seed(222)
D = np.random.rand(100,10)
D[:40] += np.random.rand(40,10)**2
labels = [0] * 40 + [1] * 60
cmat,_,predictions = nfoldcrossvalidation(D, labels, classifier=fast_classifier(), return_predictions=1)
assert np.all((predictions == 0)|(predictions == 1))
assert cmat.trace() == np.sum(predictions == labels)
def test_multi():
np.random.seed(30)
r = np.random.random
for _ in xrange(10):
labels = []
p = np.array([.24,.5,.1,.44])
for i in xrange(100):
cur = [j for j in xrange(4) if r() < p[j]]
if not cur: cur = [0]
labels.append(cur)
seen = np.zeros(100, int)
for Tr,Te in foldgenerator(labels, 5, multi_label=True):
assert np.sum(Tr & Te) == 0
seen[Te] += 1
assert np.sum(seen) == 100
assert np.ptp(seen) == 0
milk-0.5/milk/tests/test_defaultclassifier.py 0000664 0001750 0001750 00000002367 11723217323 023563 0 ustar luispedro luispedro 0000000 0000000 import numpy as np
import milk
import milk.supervised.defaultclassifier
import pickle
def test_defaultclassifier():
from milksets import wine
features, labels = wine.load()
C = milk.supervised.defaultclassifier()
model = C.train(features,labels)
labelset = set(labels)
for f in features:
assert model.apply(f) in labelset
test_defaultclassifier.slow = True
def test_pickle():
np.random.seed(23232432)
X = np.random.rand(100,10)
labels = np.zeros(100)
X[50:] += .5
labels[50:] = 1
classifier = milk.supervised.defaultclassifier()
model = classifier.train(X, labels)
s = pickle.dumps(model)
model = pickle.loads(s)
test = [model.apply(x) for x in X]
test = np.array(test)
assert (test == labels).mean() > .6
def test_pickle_learner():
learner = milk.defaultlearner()
assert len(pickle.dumps(learner))
def test_expandend():
np.random.seed(23232432)
X = np.random.rand(100,10)
labels = np.zeros(100)
X[50:] += .5
labels[50:] = 1
learners = milk.defaultlearner(expanded=True)
for learner in learners:
model = learner.train(X, labels)
test = [model.apply(x) for x in X]
test = np.array(test)
assert set(test) == set(labels)
milk-0.5/milk/tests/test_pdist.py 0000664 0001750 0001750 00000001421 12017466466 021216 0 ustar luispedro luispedro 0000000 0000000 import numpy as np
from milk.unsupervised import pdist, plike
def test_pdist():
np.random.seed(222)
X = np.random.randn(100,23)
Y = np.random.randn(80,23)
Dxx = pdist(X)
for i in xrange(X.shape[0]):
for j in xrange(X.shape[1]):
assert np.allclose(Dxx[i,j], np.sum((X[i]-X[j])**2))
Dxy = pdist(X,Y)
for i in xrange(X.shape[0]):
for j in xrange(Y.shape[1]):
assert np.allclose(Dxy[i,j], np.sum((X[i]-Y[j])**2))
Dxye = pdist(X, Y, 'euclidean')
assert np.allclose(Dxye, np.sqrt(Dxy))
def test_plike():
np.random.seed(222)
X = np.random.randn(100,23)
Lxx = plike(X)
assert len(Lxx) == len(Lxx.T)
Lxx2 = plike(X, sigma2=.001)
assert Lxx[0,1] != Lxx2[0,1]
assert Lxx[0,0] == Lxx2[0,0]
milk-0.5/milk/tests/test_featureselection.py 0000664 0001750 0001750 00000005101 11723217323 023420 0 ustar luispedro luispedro 0000000 0000000 import milk.supervised.featureselection
from milk.supervised.featureselection import select_n_best, rank_corr
import numpy as np
def test_sda():
from milksets import wine
features, labels = wine.load()
selected = milk.supervised.featureselection.sda(features,labels)
for sel in selected:
assert sel <= features.shape[1]
def test_linear_independent_features():
np.random.seed(122)
X3 = np.random.rand(20,3)
X = np.c_[X3,X3*2+np.random.rand(20,3)/20.,-X3*2+np.random.rand(20,3)/10.]
X2 = np.c_[X3,X3*2,-X3*3e-3]
assert len(milk.supervised.featureselection.linear_independent_features(X)) == 9
assert len(milk.supervised.featureselection.linear_independent_features(X2)) == 3
assert np.all (np.sort(milk.supervised.featureselection.linear_independent_features(X2) % 3) == np.arange(3))
def _rank(A,tol=1e-8):
s = np.linalg.svd(A,compute_uv=0)
return (s > tol).sum()
def _slow_linear_independent_features(featmatrix):
'''
Returns the indices of a set of linearly independent features (columns).
indices = linear_independent_features(features)
'''
independent = [0,]
rank = 1
feat = [featmatrix[:,0]]
for i,col in enumerate(featmatrix.T):
feat.append(col)
nrank = _rank(np.array(feat))
if nrank == rank:
del feat[-1]
else:
rank = nrank
independent.append(i)
return np.array(independent)
def test_select_n():
from milksets.wine import load
features,labels = load()
for n in (1,2,4,8):
select = select_n_best(n, rank_corr)
model = select.train(features,labels)
f = model.apply(features[3])
assert len(f) == n
def test_select_n():
from milksets.wine import load
features,labels = load()
for n in (1,2,4,8):
select = select_n_best(n, rank_corr)
model = select.train(features,labels)
f = model.apply(features[3])
assert len(f) == n
def slow_rank_corr(features, labels):
features = np.asanyarray(features)
labels = np.asanyarray(labels)
binlabels = [(labels == ell) for ell in set(labels)]
rs = []
for feat in features.T:
ranks = feat.argsort()
corrcoefs = [np.corrcoef(ranks, labs)[0,1] for labs in binlabels]
corrcoefs = np.array(corrcoefs)
corrcoefs **= 2
rs.append(np.max(corrcoefs))
return np.array(rs)
def test_compare_rank_corr():
from milksets.wine import load
features,labels = load()
r0 = rank_corr(features,labels)
r1 = slow_rank_corr(features,labels)
assert np.allclose(r0,r1)
milk-0.5/milk/tests/test_multi.py 0000664 0001750 0001750 00000005230 12043765776 021234 0 ustar luispedro luispedro 0000000 0000000 import numpy as np
import random
import milk.supervised.svm
import milk.supervised.multi
from milk.supervised.classifier import ctransforms
from fast_classifier import fast_classifier
import milksets.wine
features,labels = milksets.wine.load()
A = np.arange(len(features))
random.seed(9876543210)
random.shuffle(A)
features = features[A]
labels = labels[A]
labelset = set(labels)
base = ctransforms(milk.supervised.svm.svm_raw(C=2.,kernel=milk.supervised.svm.rbf_kernel(2.**-3)),milk.supervised.svm.svm_binary())
def test_one_against_rest():
M = milk.supervised.multi.one_against_rest(base)
M = M.train(features[:100,:],labels[:100])
tlabels = [M.apply(f) for f in features[100:]]
for tl in tlabels:
assert tl in labelset
def test_one_against_one():
M = milk.supervised.multi.one_against_one(base)
M = M.train(features[:100,:],labels[:100])
tlabels = [M.apply(f) for f in features[100:]]
for tl in tlabels:
assert tl in labelset
tlabels_many = M.apply_many(features[100:])
assert np.all(tlabels == tlabels_many)
def test_two_thirds():
np.random.seed(2345)
C = milk.supervised.defaultclassifier('fast')
X = np.random.rand(120,4)
X[:40] += np.random.rand(40,4)
X[:40] += np.random.rand(40,4)
X[40:80] -= np.random.rand(40,4)
X[40:80] -= np.random.rand(40,4)
Y = np.repeat(np.arange(3), 40)
model = C.train(X,Y)
Y_ = np.array([model.apply(x) for x in X])
assert (Y_ == Y).mean() * 3 > 2
def test_multi_labels():
clabels = [[lab, lab+7] for lab in labels]
multi_label = milk.supervised.multi.one_against_rest_multi(base)
model = multi_label.train(features[::2], clabels[::2])
test_vals = [model.apply(f) for f in features[1::2]]
for ts in test_vals:
if 0.0 in ts: assert 7.0 in ts
if 1.0 in ts: assert 8.0 in ts
if 2.0 in ts: assert 9.0 in ts
def test_classifier_no_set_options():
# Basically these should not raise an exception
milk.supervised.multi.one_against_rest_multi(fast_classifier())
milk.supervised.multi.one_against_rest(fast_classifier())
milk.supervised.multi.one_against_one(fast_classifier())
def test_tree():
mtree = milk.supervised.multi.multi_tree_learner(fast_classifier())
labels = [0,1,2,2,3,3,3,3]
features = np.random.random_sample((len(labels), 8))
model = mtree.train(features, labels)
counts = np.zeros(4)
for ell in labels:
counts[ell] += 1
g0,g1 = milk.supervised.multi.split(counts)
assert np.all(g0 == [3]) or np.all(g1 == [3])
def r(m):
if len(m) == 1: return int(m[0])
else: return sorted([r(m[1]), r(m[2])])
assert r(model.model) == [3,[2,[0,1]]]
milk-0.5/milk/tests/test_kmeans.py 0000664 0001750 0001750 00000003450 11723217323 021342 0 ustar luispedro luispedro 0000000 0000000 import numpy as np
import milk.unsupervised
from milk.unsupervised.kmeans import assign_centroids, repeated_kmeans
def test_kmeans():
np.random.seed(132)
features = np.r_[np.random.rand(20,3)-.5,.5+np.random.rand(20,3)]
def test_distance(dist, kwargs={}):
centroids, _ = milk.unsupervised.kmeans(features, 2, distance=dist, **kwargs)
positions = [0]*20 + [1]*20
correct = (centroids == positions).sum()
assert correct >= 38 or correct <= 2
yield test_distance, 'euclidean'
yield test_distance, 'seuclidean'
yield test_distance, 'mahalanobis', { 'icov' : np.eye(3) }
def test_kmeans_centroids():
np.random.seed(132)
features = np.random.rand(201,30)
for k in [2,3,5,10]:
indices,centroids = milk.unsupervised.kmeans(features, k)
for i in xrange(k):
if np.any(indices == i):
assert np.allclose(centroids[i], features[indices == i].mean(0))
def test_assign_cids():
from milksets.wine import load
features,_ = load()
assigns, centroids = milk.unsupervised.kmeans(features, 3, R=2, max_iters=10)
assert np.all(assign_centroids(features, centroids) == assigns)
def test_non_contiguous_fmatrix():
from milksets.wine import load
features,_ = load()
features = features[:,::2]
assigns, centroids = milk.unsupervised.kmeans(features, 3, R=2, max_iters=10)
assert np.all(assign_centroids(features, centroids) == assigns)
features = features.astype(np.int32)
assigns, centroids = milk.unsupervised.kmeans(features, 3, R=2, max_iters=10)
assert np.all(assign_centroids(features, centroids) == assigns)
def test_repeated_kmeans():
np.random.seed(132)
features = np.random.rand(201,30)
cids,cs = repeated_kmeans(features, 3, 2)
assert len(cids) == len(features)
milk-0.5/milk/tests/test_parzen.py 0000664 0001750 0001750 00000001556 11723217323 021370 0 ustar luispedro luispedro 0000000 0000000 from __future__ import division
import milk.supervised.normalise
from milk.supervised.parzen import get_parzen_rbf_loocv
import numpy as np
import milksets
def _slow_parzen(features, labels, sigma):
correct = 0
N = len(features)
labels = 2*labels - 1
def kernel(fi, fj):
return np.exp(-((fi-fj)**2).sum()/sigma)
for i in xrange(N):
C = 0.
for j in xrange(N):
if i == j: continue
C += labels[j] * kernel(features[i],features[j])
if (C*labels[i] > 0): correct += 1
return correct/N
def test_parzen():
features,labels = milksets.wine.load()
labels = (labels == 1)
features = milk.supervised.normalise.zscore(features)
f = get_parzen_rbf_loocv(features, labels)
sigmas = 2.**np.arange(-4,4)
for s in sigmas:
assert abs(_slow_parzen(features, labels, s) - f(s)) < 1e-6
milk-0.5/milk/tests/test_precluster_learner.py 0000664 0001750 0001750 00000003113 11723217323 023760 0 ustar luispedro luispedro 0000000 0000000 import numpy as np
from milk.supervised.precluster import precluster_learner, select_precluster
from milk.tests.fast_classifier import fast_classifier
def c0():
return np.random.rand(8)
def c1():
return c0()+2.*np.ones(8)
def gen_data(seed, with_nums=False):
np.random.seed(seed)
features = []
labels =[]
for i in xrange(200):
f = []
for j in xrange(40):
use_0 = (i < 100 and j < 30) or (i >= 100 and j >= 30)
if use_0: f.append(c0())
else: f.append(c1())
labels.append((i < 100))
if with_nums:
features.append((f,[]))
else:
features.append(f)
return features, labels
def test_precluster():
learner = precluster_learner([2], base=fast_classifier(), R=12)
features, labels = gen_data(22)
model = learner.train(features,labels)
assert model.apply([c0() for i in xrange(35)])
assert not model.apply([c1() for i in xrange(35)])
def test_codebook_learner():
learner = select_precluster([2,3,4], base=fast_classifier())
learner.rmax = 3
features, labels = gen_data(23, 1)
model = learner.train(features,labels)
assert model.apply(([c0() for i in xrange(35)],[]))
assert not model.apply(([c1() for i in xrange(35)],[]))
def test_codebook_learner_case1():
learner = select_precluster([2], base=fast_classifier())
learner.rmax = 1
features, labels = gen_data(23, 1)
model = learner.train(features,labels)
assert model.apply(([c0() for i in xrange(35)],[]))
assert not model.apply(([c1() for i in xrange(35)],[]))
milk-0.5/milk/nfoldcrossvalidation.py 0000664 0001750 0001750 00000000130 11723217323 022102 0 ustar luispedro luispedro 0000000 0000000 from .measures.nfoldcrossvalidation import foldgenerator, getfold, nfoldcrossvalidation
milk-0.5/milk/supervised/ 0000775 0001750 0001750 00000000000 12045730202 017472 5 ustar luispedro luispedro 0000000 0000000 milk-0.5/milk/supervised/featureselection.py 0000664 0001750 0001750 00000021460 12026076202 023412 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2012, Luis Pedro Coelho
#
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division
import numpy as np
from numpy.linalg import det
from . classifier import normaliselabels
__all__ = [
'sda',
'linearly_independent_subset',
'linear_independent_features',
'filterfeatures',
'featureselector',
'sda_filter',
'rank_corr',
'select_n_best',
]
def _sweep(A, k, flag):
Akk = A[k,k]
if Akk == 0:
Akk = 1.e-5
# cross[i,j] = A[i,k] * A[k,j]
cross = (A[:,k][:, np.newaxis] * A[k])
B = A - cross/Akk
# currently: B[i,j] = A[i,j] - A[i,k]*A[k,j]/Akk
# Now fix row k and col k, followed by Bkk
B[k] = flag * A[k]/A[k,k]
B[:,k] = flag * A[:,k]/A[k,k]
B[k,k] = -1./Akk
return B
def sda(features, labels, tolerance=.01, significance_in=.05, significance_out=.05, loose=False):
'''
features_idx = sda(features, labels, tolerance=.01, significance_in=.05, significance_out=.05)
Stepwise Discriminant Analysis for feature selection
Pre-filter the feature matrix to remove linearly dependent features
before calling this function. Behaviour is undefined otherwise.
This implements the algorithm described in Jennrich, R.I. (1977), "Stepwise
Regression" & "Stepwise Discriminant Analysis," both in Statistical Methods
for Digital Computers, eds. K. Enslein, A. Ralston, and H. Wilf, New York;
John Wiley & Sons, Inc.
Parameters
----------
features : ndarray
feature matrix. There should not be any perfectly correlated features.
labels : 1-array
labels
tolerance : float, optional
significance_in : float, optional
significance_out : float, optional
Returns
-------
features_idx : sequence
sequence of integer indices
'''
from scipy import stats
assert len(features) == len(labels), 'milk.supervised.featureselection.sda: length of features not the same as length of labels'
N, m = features.shape
labels,labelsu = normaliselabels(labels)
q = len(labelsu)
df = features - features.mean(0)
T = np.dot(df.T, df)
dfs = [(features[labels == i] - features[labels == i].mean(0)) for i in xrange(q)]
W = np.sum(np.dot(d.T, d) for d in dfs)
ignoreidx = ( W.diagonal() == 0 )
if ignoreidx.any():
idxs, = np.where(~ignoreidx)
if not len(idxs):
return np.arange(m)
selected = sda(features[:,~ignoreidx],labels)
return idxs[selected]
output = []
D = W.diagonal()
df1 = q-1
last_enter_k = -1
while True:
V = W.diagonal()/T.diagonal()
W_d = W.diagonal()
V_neg = (W_d < 0)
p = V_neg.sum()
if V_neg.any():
V_m = V[V_neg].min()
k, = np.where(V == V_m)
k = k[0]
Fremove = (N-p-q+1)/(q-1)*(V_m-1)
df2 = N-p-q+1
PrF = 1 - stats.f.cdf(Fremove,df1,df2)
if PrF > significance_out:
#print 'removing ',k, 'V(k)', 1./V_m, 'Fremove', Fremove, 'df1', df1, 'df2', df2, 'PrF', PrF
if k == last_enter_k:
# We are going into an infinite loop.
import warnings
warnings.warn('milk.featureselection.sda: infinite loop detected (maybe bug?).')
break
W = _sweep(W,k,1)
T = _sweep(T,k,1)
continue
ks = ( (W_d / D) > tolerance)
if ks.any():
V_m = V[ks].min()
k, = np.where(V==V_m)
k = k[0]
Fenter = (N-p-q)/(q-1) * (1-V_m)/V_m
df2 = N-p-q
PrF = 1 - stats.f.cdf(Fenter,df1,df2)
if PrF < significance_in:
#print 'adding ',k, 'V(k)', 1./V_m, 'Fenter', Fenter, 'df1', df1, 'df2', df2, 'PrF', PrF
W = _sweep(W,k,-1)
T = _sweep(T,k,-1)
if loose or (PrF < 0.0001):
output.append((Fenter,k))
last_enter_k = k
continue
break
output.sort(reverse=True)
return np.array([idx for _,idx in output])
def linearly_independent_subset(V, threshold=1.e-5, return_orthogonal_basis=False):
'''
subset = linearly_independent_subset(V, threshold=1.e-5)
subset,U = linearly_independent_subset(V, threshold=1.e-5, return_orthogonal_basis=True)
Discover a linearly independent subset of `V`
Parameters
----------
V : sequence of input vectors
threshold : float, optional
vectors with 2-norm smaller or equal to this are considered zero
(default: 1e.-5)
return_orthogonal_basis : Boolean, optional
whether to return orthogonal basis set
Returns
-------
subset : ndarray of integers
indices used for basis
U : 2-array
orthogonal basis into span{V}
Implementation Reference
------------------------
Use Gram-Schmidt with a check for when the v_k is close enough to zero to ignore
See http://en.wikipedia.org/wiki/Gram-Schmidt_process
'''
V = np.array(V, copy=True)
orthogonal = []
used = []
for i,u in enumerate(V):
for v in orthogonal:
u -= np.dot(u,v)/np.dot(v,v) * v
if np.dot(u,u) > threshold:
orthogonal.append(u)
used.append(i)
if return_orthogonal_basis:
return np.array(used),np.array(orthogonal)
return np.array(used)
def linear_independent_features(features, labels=None):
'''
indices = linear_independent_features(features, labels=None)
Returns the indices of a set of linearly independent features (columns).
Parameters
----------
features : ndarray
labels : ignored
This argument is only here to conform to the learner interface.
Returns
-------
indices : ndarray of integers
indices of features to keep
See Also
--------
`linearly_independent_subset` :
this function is equivalent to `linearly_independent_subset(features.T)`
'''
return linearly_independent_subset(features.T)
class filterfeatures(object):
'''
selector = filterfeatures(idxs)
Returns a transformer which selects the features given by idxs. I.e.,
``apply(features)`` is equivalent to ``features[idxs]``
Parameters
----------
idxs : ndarray
This can be either an array of integers (positions) or an array of booleans
'''
def __init__(self, idxs):
self.idxs = idxs
def apply(self, features):
return features[self.idxs]
def apply_many(self, features):
features = np.asanyarray(features)
return features[:,self.idxs]
def __repr__(self):
return 'filterfeatures(%s)' % self.idxs
class featureselector(object):
'''
selector = featureselector(function)
Returns a transformer which selects features according to
selected_idxs = function(features,labels)
'''
def __init__(self, selector):
self.selector = selector
def train(self, features, labels, **kwargs):
idxs = self.selector(features, labels)
if len(idxs) == 0:
import warnings
warnings.warn('milk.featureselection: No features selected! Using all features as fall-back.')
idxs = np.arange(len(features[0]))
return filterfeatures(idxs)
def __repr__(self):
return 'featureselector(%s)' % self.selector
def sda_filter():
return featureselector(sda)
def rank_corr(features, labels):
'''
rs = rank_corr(features, labels)
Computes the following expression::
rs[i] = max_e COV²(rank(features[:,i]), labels == e)
This is appropriate for numeric features and categorical labels.
Parameters
----------
features : ndarray
feature matrix
labels : sequence
Returns
-------
rs : ndarray of float
rs are the rank correlations
'''
features = np.asanyarray(features)
labels = np.asanyarray(labels)
n = len(features)
ranks = features.argsort(0)
ranks = ranks.astype(float)
binlabels = np.array([(labels == ell) for ell in set(labels)], dtype=float)
mx = ranks.mean(0)
my = binlabels.mean(1)
sx = ranks.std(0)
sy = binlabels.std(1)
r = np.dot(binlabels,ranks)
r -= np.outer(n*my, mx)
r /= np.outer(sy, sx)
r /= n # Use n [instead of n-1] to match numpy's corrcoef
r **= 2
return r.max(0)
class select_n_best(object):
'''
select_n_best(n, measure)
Selects the `n` features that score the highest in `measure`
'''
def __init__(self, n, measure):
self.n = n
self.measure = measure
def train(self, features, labels, **kwargs):
values = self.measure(features, labels)
values = values.argsort()
return filterfeatures(values[:self.n])
milk-0.5/milk/supervised/perceptron.py 0000664 0001750 0001750 00000002350 11723217323 022233 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2011, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# License: MIT. See COPYING.MIT file in the milk distribution
import numpy as np
from .classifier import normaliselabels
from .base import supervised_model
from . import _perceptron
class perceptron_model(supervised_model):
def __init__(self, w):
self.w = w
def apply(self, f):
f = np.asanyarray(f)
v = self.w[0] + np.dot(f, self.w[1:])
return v > 0
class perceptron_learner(object):
def __init__(self, eta=.1, max_iters=128):
self.eta = eta
self.max_iters = max_iters
def train(self, features, labels, normalisedlabels=False, **kwargs):
if not normalisedlabels:
labels, _ = normaliselabels(labels)
features = np.asanyarray(features)
if features.dtype not in (np.float32, np.float64):
features = features.astype(np.float64)
weights = np.zeros(features.shape[1]+1, features.dtype)
for i in xrange(self.max_iters):
errors = _perceptron.perceptron(features, labels, weights, self.eta)
if not errors:
break
return perceptron_model(weights)
milk-0.5/milk/supervised/multi_view.py 0000664 0001750 0001750 00000003645 11723217323 022246 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2011, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# License: MIT. See COPYING.MIT file in the milk distribution
import numpy as np
__all__ = [
'multi_view_learner',
]
class multi_view_model(object):
def __init__(self, models):
self.models = models
def apply(self, features):
if len(features) != len(self.models):
raise ValueError('milk.supervised.two_view: Nr of features does not match training data (got %s, expected %s)' % (len(features) ,len(self.models)))
Ps = np.array([model.apply(f) for model,f in zip(self.models, features)])
if np.any(Ps <= 0.): return False
if np.any(Ps >= 1.): return True
# This is binary only:
# if \prod Pi > \prod (1-Pi) return 1
# is equivalent to
# if \prod Pi/(1-Pi) > 1. return 1
# if \sum \log( Pi/(1-Pi) ) > 0. return 1
return np.sum( np.log(Ps/(1-Ps)) ) > 0
class multi_view_learner(object):
'''
Multi View Learner
This learner learns different classifiers on multiple sets of features and
combines them for classification.
'''
def __init__(self, bases):
self.bases = bases
def train(self, features, labels, normalisedlabels=False):
features = zip(*features)
if len(features) != len(self.bases):
raise ValueError('milk.supervised.multi_view_learner: ' +
'Nr of features does not match classifiser construction (got %s, expected %s)'
% (len(features) ,len(self.bases)))
models = []
for basis,f in zip(self.bases, features):
try:
f = np.array(f)
except:
f = np.array(f, dtype=object)
models.append(basis.train(f, labels))
return multi_view_model(models)
multi_view_classifier = multi_view_learner
milk-0.5/milk/supervised/adaboost.py 0000664 0001750 0001750 00000004205 12014503604 021641 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2012, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division
import numpy as np
from .normalise import normaliselabels
from .base import supervised_model
'''
AdaBoost
Simple implementation of Adaboost
Learner
-------
boost_learner
'''
__all__ = [
'boost_learner',
]
def _adaboost(features, labels, base, max_iters):
m = len(features)
D = np.ones(m, dtype=float)
D /= m
Y = np.ones(len(labels), dtype=float)
names = np.array([-1, +1])
Y = names[labels]
H = []
A = []
for t in xrange(max_iters):
Ht = base.train(features, labels, weights=D)
train_out = np.array(map(Ht.apply, features))
train_out = names[train_out.astype(int)]
Et = np.dot(D, (Y != train_out))
if Et > .5:
# early return
break
At = .5 * np.log((1. + Et) / (1. - Et))
D *= np.exp((-At) * Y * train_out)
D /= np.sum(D)
A.append(At)
H.append(Ht)
return H, A
class boost_model(supervised_model):
def __init__(self, H, A, names):
self.H = H
self.A = A
self.names = names
def apply(self, f):
v = sum((a*h.apply(f)) for h,a in zip(self.H, self.A))
v /= np.sum(self.A)
return self.names[v > .5]
class boost_learner(object):
'''
learner = boost_learner(weak_learner_type(), max_iters=100)
model = learner.train(features, labels)
test = model.apply(f)
AdaBoost learner
Attributes
----------
base : learner
Weak learner
max_iters : integer
Nr of iterations (default: 100)
'''
def __init__(self, base, max_iters=100):
self.base = base
self.max_iters = max_iters
def train(self, features, labels, normalisedlabels=False, names=(0,1), weights=None, **kwargs):
if not normalisedlabels:
labels,names = normaliselabels(labels)
H,A = _adaboost(features, labels, self.base, self.max_iters)
return boost_model(H, A, names)
milk-0.5/milk/supervised/defaultclassifier.py 0000664 0001750 0001750 00000000121 11723217323 023535 0 ustar luispedro luispedro 0000000 0000000 from milk.supervised.defaultlearner import *
defaultclassifier = defaultlearner
milk-0.5/milk/supervised/lasso.py 0000664 0001750 0001750 00000012455 12032625516 021203 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
import numpy as np
import _lasso
from .base import supervised_model
from milk.unsupervised import center
def lasso(X, Y, B=None, lam=1., max_iter=None, tol=None):
'''
B = lasso(X, Y, B={np.zeros()}, lam=1. max_iter={1024}, tol={1e-5})
Solve LASSO Optimisation
B* = arg min_B ½/n || Y - BX ||₂² + λ||B||â‚
where $n$ is the number of samples.
Milk uses coordinate descent, looping through the coordinates in order
(with an active set strategy to update only non-zero βs, if possible). The
problem is convex and the solution is guaranteed to be optimal (within
floating point accuracy).
Parameters
----------
X : ndarray
Design matrix
Y : ndarray
Matrix of outputs
B : ndarray, optional
Starting values for approximation. This can be used for a warm start if
you have an estimate of where the solution should be. If used, the
solution might be written in-place (if the array has the right format).
lam : float, optional
λ (default: 1.0)
max_iter : int, optional
Maximum nr of iterations (default: 1024)
tol : float, optional
Tolerance. Whenever a parameter is to be updated by a value smaller
than ``tolerance``, that is considered a null update. Be careful that
if the value is too small, performance will degrade horribly.
(default: 1e-5)
Returns
-------
B : ndarray
'''
X = np.ascontiguousarray(X, dtype=np.float32)
Y = np.ascontiguousarray(Y, dtype=np.float32)
if B is None:
B = np.zeros((Y.shape[0],X.shape[0]), np.float32)
else:
B = np.ascontiguousarray(B, dtype=np.float32)
if max_iter is None:
max_iter = 1024
if tol is None:
tol = 1e-5
if X.shape[0] != B.shape[1] or \
Y.shape[0] != B.shape[0] or \
X.shape[1] != Y.shape[1]:
raise ValueError('milk.supervised.lasso: Dimensions do not match')
if np.any(np.isnan(X)) or np.any(np.isnan(B)):
raise ValueError('milk.supervised.lasso: NaNs are only supported in the ``Y`` matrix')
W = np.ascontiguousarray(~np.isnan(Y), dtype=np.float32)
Y = np.nan_to_num(Y)
n = Y.size
_lasso.lasso(X, Y, W, B, max_iter, float(2*n*lam), float(tol))
return B
def lasso_walk(X, Y, B=None, nr_steps=None, start=None, step=None, tol=None, return_lams=False):
'''
Bs = lasso_walk(X, Y, B={np.zeros()}, nr_steps={64}, start={automatically inferred}, step={.9}, tol=None, return_lams=False)
Bs,lams = lasso_walk(X, Y, B={np.zeros()}, nr_steps={64}, start={automatically inferred}, step={.9}, tol=None, return_lams=True)
Repeatedly solve LASSO Optimisation
B* = arg min_B ½/n || Y - BX ||₂² + λ||B||â‚
for different values of λ.
Parameters
----------
X : ndarray
Design matrix
Y : ndarray
Matrix of outputs
B : ndarray, optional
Starting values for approximation. This can be used for a warm start if
you have an estimate of where the solution should be.
start : float, optional
first λ to use (default is ``np.abs(Y).max()``)
nr_steps : int, optional
How many steps in the path (default is 64)
step : float, optional
Multiplicative step to take (default is 0.9)
tol : float, optional
This is the tolerance parameter. It is passed to the lasso function
unmodified.
return_lams : bool, optional
Whether to return the values of λ used (default: False)
Returns
-------
Bs : ndarray
'''
if nr_steps is None:
nr_steps = 64
if step is None:
step = .9
if start is None:
n = Y.size
start = 0.5/n*np.nanmax(np.abs(Y))*np.abs(X).max()
lam = start
lams = []
Bs = []
for i in xrange(nr_steps):
# The central idea is that each iteration is already "warm" and this
# should be faster than starting from zero each time
B = lasso(X, Y, B, lam=lam, tol=tol)
lams.append(lam)
Bs.append(B.copy())
lam *= step
if return_lams:
return np.array(Bs), np.array(lams)
return np.array(Bs)
def _dict_subset(mapping, keys):
return dict(
[(k,mapping[k]) for k in keys])
class lasso_model(supervised_model):
def __init__(self, betas, mean):
self.betas = betas
self.mean = mean
def retrain(self, features, labels, lam, **kwargs):
features, mean = center(features)
betas = lasso(features, labels, self.betas.copy(), lam=lam, **_dict_subset(kwargs, ['tol', 'max_iter']))
return lasso_model(betas, mean)
def apply(self, features):
return np.dot(self.betas, features) + self.mean
class lasso_learner(object):
def __init__(self, lam=1.0):
self.lam = lam
def train(self, features, labels, betas=None, **kwargs):
labels, mean = center(labels, axis=1)
betas = lasso(features, labels, betas, lam=self.lam)
return lasso_model(betas, mean)
def lasso_model_walk(X, Y, B=None, nr_steps=64, start=None, step=.9, tol=None, return_lams=False):
Y, mean = center(Y, axis=1)
Bs,lams = lasso_walk(X,Y, B, nr_steps, start, step, tol, return_lams=True)
models = [lasso_model(B, mean) for B in Bs]
if return_lams:
return models, lams
return models
milk-0.5/milk/supervised/normalise.py 0000664 0001750 0001750 00000010306 12043773252 022047 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2012, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division
import numpy as np
from .base import supervised_model
from ..unsupervised.normalise import zscore
__all__ = [
'zscore',
'zscore_normalise',
'interval_normalise',
'chkfinite',
'sample_to_2min',
'normaliselabels'
]
class subtract_divide_model(supervised_model):
def __init__(self, shift, factor):
factor[factor == 0] = 1 # This makes the division a null op.
self.shift = shift
self.factor = factor
def apply_many(self, features):
return (features - self.shift)/self.factor
def apply(self, features):
return (features - self.shift)/self.factor
def __repr__(self):
return 'subtract_divide_model(%s, %s)' % (self.shift, self.factor)
class zscore_normalise(object):
'''
Normalise to z-scores
A preprocessor that normalises features to z scores.
'''
def train(self, features, labels, **kwargs):
shift = features.mean(0)
factor = np.std(features,0)
return subtract_divide_model(shift, factor)
class interval_normalise(object):
'''
Linearly scale to the interval [-1,1] (per libsvm recommendation)
'''
def train(self, features, labels, **kwargs):
ptp = features.ptp(0)
shift = features.min(0) + ptp/2.
factor = ptp/2.
return subtract_divide_model(shift, factor)
def __repr__(self):
return 'interval_normalise()'
def sample_to_2min(labels):
'''
selected = sample_to_2min(labels)
Select examples so that the ratio of size of the largest
class to the smallest class is at most two (i.e.,
min_label_count = min { (labels == L).sum() | for L in set(labels) }
for L' in set(labels):
assert (labels == L').sum() <= 2 * min_label_count
)
Parameters
----------
labels : sequence of labels
Returns
-------
selected : a Boolean numpy.ndarray
'''
from collections import defaultdict
counts = defaultdict(int)
for n in labels:
counts[n] += 1
labels = np.asanyarray(labels)
max_entries = np.min(counts.values())*2
selected = np.zeros(len(labels), bool)
for c in counts.iterkeys():
p, = np.where(labels == c)
p = p[:max_entries]
selected[p] = 1
return selected
class chkfinite(supervised_model):
'''
Fill NaN & Inf values
Replaces NaN & Inf values with zeros.
'''
def __init__(self):
pass
def train(self, features, labels, **kwargs):
return self
def apply(self, features):
nans = np.isnan(features) + np.isinf(features)
if nans.any():
features = features.copy()
features[nans] = 0
return features
def __repr__(self):
return 'chkfinite()'
def normaliselabels(labels, multi_label=False):
'''
normalised, names = normaliselabels(labels, multi_label=False)
If not ``multi_label`` (the default), normalises the labels to be integers
from 0 through N-1. Otherwise, assume that each label is actually a
sequence of labels.
``normalised`` is a np.array, while ``names`` is a list mapping the indices to
the old names.
Parameters
----------
labels : any iterable of labels
multi_label : bool, optional
Whether labels are actually composed of multiple labels
Returns
------
normalised : a numpy ndarray
If not ``multi_label``, this is an array of integers 0 .. N-1;
otherwise, it is a boolean array of size len(labels) x N
names : list of label names
'''
if multi_label:
names = set()
for ell in labels: names.update(ell)
names = list(sorted(names))
normalised = np.zeros( (len(labels), len(names)), bool)
for i,ls in enumerate(labels):
for ell in map(names.index, ls):
normalised[i,ell] = True
return normalised, names
else:
names = sorted(set(labels))
normalised = map(names.index, labels)
return np.array(normalised), names
milk-0.5/milk/supervised/_perceptron.cpp 0000664 0001750 0001750 00000005663 11723217323 022536 0 ustar luispedro luispedro 0000000 0000000 // Copyright (C) 2011, Luis Pedro Coelho
// License: MIT
#include
#include
#include
#include
extern "C" {
#include
#include
}
namespace {
template
int perceptron(PyArrayObject* data_arr, const long* labels, PyArrayObject* weights_arr, double eta) {
const T* data = reinterpret_cast(PyArray_DATA(data_arr));
T* weights = reinterpret_cast(PyArray_DATA(weights_arr));
const int N0 = PyArray_DIM(data_arr, 0);
const int N1 = PyArray_DIM(data_arr, 1);
int nr_errors = 0;
for (int i = 0; i != N0; ++i, data += N1, ++labels) {
T val = weights[0];
for (int j = 0; j != N1; ++j) {
val += weights[j+1] * data[j];
}
int ell = (val > 0);
if (ell != *labels) {
int pm = (*labels ? +1 : -1);
++nr_errors;
T error = pm * eta * std::abs(pm-val);
weights[0] += error;
for (int j = 0; j != N1; ++j) {
weights[j+1] += error*data[j];
}
}
}
return nr_errors;
}
PyObject* py_perceptron(PyObject* self, PyObject* args) {
const char* errmsg = "Arguments were not what was expected for perceptron.\n"
"This is an internal function: Do not call directly unless you know exactly what you're doing.\n";
PyArrayObject* data;
PyArrayObject* labels;
PyArrayObject* weights;
double eta;
if (!PyArg_ParseTuple(args, "OOOd", &data, &labels, &weights, &eta)) {
PyErr_SetString(PyExc_RuntimeError,errmsg);
return 0;
}
if (!PyArray_Check(data) || !PyArray_ISCONTIGUOUS(data) ||
!PyArray_Check(weights) || !PyArray_ISCONTIGUOUS(weights) ||
!PyArray_Check(labels) || !PyArray_ISCONTIGUOUS(labels) || !PyArray_EquivTypenums(PyArray_TYPE(labels), NPY_LONG) ||
PyArray_TYPE(data) != PyArray_TYPE(weights)||
PyArray_NDIM(data) != 2 || PyArray_NDIM(weights) != 1 || PyArray_DIM(data,1) + 1 != PyArray_DIM(weights,0)) {
PyErr_SetString(PyExc_RuntimeError,errmsg);
return 0;
}
int nr_errors;
if (PyArray_TYPE(data) == NPY_FLOAT) {
nr_errors = perceptron(data, reinterpret_cast(PyArray_DATA(labels)), weights, eta);
} else if (PyArray_TYPE(data) == NPY_DOUBLE) {
nr_errors = perceptron(data, reinterpret_cast(PyArray_DATA(labels)), weights, eta);
} else {
PyErr_SetString(PyExc_RuntimeError, errmsg);
return 0;
}
return PyLong_FromLong(nr_errors);
}
PyMethodDef methods[] = {
{"perceptron", py_perceptron, METH_VARARGS , "Do NOT call directly.\n" },
{NULL, NULL,0,NULL},
};
const char * module_doc =
"Internal Module.\n"
"\n"
"Do NOT use directly!\n";
} // namespace
extern "C"
void init_perceptron()
{
import_array();
(void)Py_InitModule3("_perceptron", methods, module_doc);
}
milk-0.5/milk/supervised/set2binary_array.py 0000664 0001750 0001750 00000001574 11723217323 023341 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2011, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division
import numpy as np
__all__ = [
'set2binary_array',
]
class set2binary_array_model(object):
def __init__(self, universe):
self.universe = list(universe)
def apply(self, elems):
res = np.zeros(len(self.universe) + 1, bool)
for e in elems:
try:
res[self.universe.index(e)] = True
except :
res[-1] = True
return res
class set2binary_array(object):
def train(self, features, labels, normalisedlabels=False):
allfeatures = set()
for f in features:
allfeatures.update(f)
return set2binary_array_model(allfeatures)
milk-0.5/milk/supervised/parzen.py 0000664 0001750 0001750 00000001361 11723217323 021352 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2011, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division
import numpy as np
def get_parzen_rbf_loocv(features,labels):
xij = np.dot(features,features.T)
f2 = np.sum(features**2,1)
d = f2-2*xij
d = d.T + f2
d_argsorted = d.argsort(1)
d_sorted = d.copy()
d_sorted.sort(1)
e_d = np.exp(-d_sorted)
labels_sorted = labels[d_argsorted].astype(np.double)
labels_sorted *= 2
labels_sorted -= 1
def f(sigma):
k = e_d ** (1./sigma)
return (((k[:,1:] * labels_sorted[:,1:]).sum(1) > 0) == labels).mean()
return f
milk-0.5/milk/supervised/multi_label.py 0000664 0001750 0001750 00000002540 11723217323 022344 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2011, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division
import numpy as np
from .base import supervised_model, base_adaptor
class one_by_one_model(supervised_model):
def __init__(self, models):
self.models = models
def apply(self, fs):
result = []
for ell,model in self.models.iteritems():
if model.apply(fs):
result.append(ell)
return result
class one_by_one(base_adaptor):
'''
Implements 1-vs-all multi-label classifier by transforming a base (binary)
classifier.
Example
-------
features = [....]
labels = [
(0,),
(1,2),
(0,2),
(0,3),
(1,2,3),
(2,0),
...
]
learner = one_by_one(milk.defaultlearner())
model = learner.train(features, labels)
'''
def train(self, features, labels, **kwargs):
universe = set()
for ls in labels:
universe.update(ls)
models = {}
for ell in universe:
contained = np.array([int(ell in ls) for ls in labels])
models[ell] = self.base.train(features, contained, normalisedlabels=True)
return one_by_one_model(models)
milk-0.5/milk/supervised/gridsearch.py 0000664 0001750 0001750 00000023024 12043765776 022206 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2011, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division
import numpy as np
from .classifier import normaliselabels
import multiprocessing
__all__ = [
'gridminimise',
'gridsearch',
]
def _allassignments(options):
try:
from itertools import product
except ImportError:
def product(*args, **kwds):
# from http://docs.python.org/library/itertools.html#itertools.product
pools = map(tuple, args) * kwds.get('repeat', 1)
result = [[]]
for pool in pools:
result = [x+[y] for x in result for y in pool]
for prod in result:
yield tuple(prod)
from itertools import repeat, izip
for ks,vs in izip(repeat(options.keys()), product(*options.values())):
yield zip(ks,vs)
def _set_options(learner, options):
for k,v in options:
learner.set_option(k,v)
class Grid1(multiprocessing.Process):
def __init__(self, learner, features, labels, measure, train_kwargs, options, folds, inq, outq):
self.learner = learner
self.features = features
self.labels = labels
self.measure = measure
self.train_kwargs = train_kwargs
self.options = options
self.folds = folds
self.inq = inq
self.outq = outq
super(Grid1, self).__init__()
def execute_one(self, index, fold):
_set_options(self.learner, self.options[index])
train, test = self.folds[fold]
model = self.learner.train(self.features[train], self.labels[train], normalisedlabels=True, **self.train_kwargs)
preds = model.apply_many(self.features[test])
error = self.measure(self.labels[test], preds)
return error
def run(self):
try:
while True:
index,fold = self.inq.get()
if index == 'shutdown':
self.outq.close()
self.outq.join_thread()
return
error = self.execute_one(index, fold)
self.outq.put( (index, error) )
except Exception, e:
import traceback
errstr = r'''\
Error in milk.gridminimise internal
Exception was: %s
Original Traceback:
%s
(Since this was run on a different process, this is not a real stack trace).
''' % (e, traceback.format_exc())
self.outq.put( ('error', errstr) )
def gridminimise(learner, features, labels, params, measure=None, nfolds=10, return_value=False, train_kwargs=None, nprocs=None, origins=None):
'''
best = gridminimise(learner, features, labels, params, measure={0/1 loss}, nfolds=10, return_value=False, nprocs=None)
best, value = gridminimise(learner, features, labels, params, measure={0/1 loss}, nfolds=10, return_value=True, nprocs=None)
Grid search for the settings of parameters that maximises a given measure
This function is equivalent to searching the grid, but does not actually
search the whole grid.
Parameters
----------
learner : a classifier object
features : sequence of features
labels : sequence of labels
params : dictionary of sequences
keys are the options to change,
values are sequences of corresponding elements to try
measure : function, optional
a function that takes labels and outputs and returns the loss.
Default: 0/1 loss. This must be an *additive* function.
nfolds : integer, optional
nr of folds to run, default: 10
return_value : boolean, optional
Whether to return the error value as well. Default False
train_kwargs : dict, optional
Options that are passed to the train() method of the classifier, using
the ``train(features, labels, **train_kwargs)`` syntax. Defaults to {}.
nprocs : integer, optional
Number of processors to use. By default, uses the
``milk.utils.parallel`` framework to check the number of
processors.
Returns
-------
best : a sequence of assignments
value : float
Only returned if ``return_value`` is true
'''
# The algorithm is as follows:
#
# for all assignments: error = 0, next_iteration = 0
#
# at each iteration:
# look for assignment with smallest error
# if that is done: return it
# else: perform one more iteration
#
# When the function returns, that assignment has the lowest error of all
# assignments and all the iterations are done. Therefore, other assignments
# could only be worse even if we never computed the whole error!
from ..measures.nfoldcrossvalidation import foldgenerator
from ..utils import parallel
if measure is None:
from ..measures.measures import zero_one_loss
measure = zero_one_loss
if train_kwargs is None:
train_kwargs = {}
try:
features = np.asanyarray(features)
except:
features = np.array(features, dtype=object)
labels,_ = normaliselabels(labels)
options = list(_allassignments(params))
iteration = np.zeros(len(options), int)
error = np.zeros(len(options), float)
folds = [(Tr.copy(), Te.copy()) for Tr,Te in foldgenerator(labels, nfolds=nfolds, origins=origins)]
# foldgenerator might actually decide on a smaller number of folds,
# depending on the distribution of class sizes:
nfolds = len(folds)
assert nfolds
if nprocs is None:
nprocs = len(options)
else:
nprocs = min(nprocs, len(options))
assert nprocs > 0, 'milk.supervised.gridminimise: nprocs <= 0!!'
nprocs = parallel.get_procs(nprocs, use_current=True)
executing = set()
workers = []
if nprocs > 1:
inqueue = multiprocessing.Queue()
outqueue = multiprocessing.Queue()
for i in xrange(nprocs):
inqueue.put((i,0))
executing.add(i)
w = Grid1(learner, features, labels, measure, train_kwargs, options, folds, inqueue, outqueue)
w.start()
workers.append(w)
getnext = outqueue.get
queuejob = lambda next, fold: inqueue.put( (next, fold) )
else:
worker = Grid1(learner, features, labels, measure, train_kwargs, options, folds, None, None)
queue = []
def queuejob(index,fold):
queue.append((index,fold))
def getnext():
index,fold = queue.pop()
return index, worker.execute_one(index,fold)
queuejob(0,0)
executing.add(0)
try:
while True:
p,err = getnext()
if p == 'error':
raise RuntimeError(err)
executing.remove(p)
iteration[p] += 1
error[p] += err
for best in np.where(error == error.min())[0]:
if iteration[best] == nfolds:
if return_value:
return options[best], error[best]
return options[best]
for next in error.argsort():
if iteration[next] < nfolds and next not in executing:
executing.add(next)
queuejob(next, iteration[next])
break
finally:
assert np.max(iteration) <= nfolds
if len(workers):
for w in workers:
inqueue.put( ('shutdown', None) )
inqueue.close()
inqueue.join_thread()
for w in workers:
w.join()
parallel.release_procs(nprocs, count_current=True)
class gridsearch(object):
'''
G = gridsearch(base, measure=accuracy, nfolds=10, params={ param1 : [...], param2 : [...]}, annotate=False)
Perform a grid search for the best parameter values.
When G.train() is called, then for each combination of p1 in param1, p2 in
param2, ... it performs (effectively)::
base.param1 = p1
base.param2 = p2
...
value[p1, p2,...] = measure(nfoldcrossvalidation(..., learner=base))
it then picks the highest set of parameters and re-learns a model on the
whole data.
Parameters
-----------
base : classifier to use
measure : function, optional
a function that takes labels and outputs and returns the loss.
Default: 0/1 loss. This must be an *additive* function.
nfolds : integer, optional
Nr of folds
params : dictionary
annotate : boolean
Whether to annotate the returned model with ``arguments`` and ``value``
fields with the result of cross-validation. Defaults to False.
All of the above can be *passed as parameters to the constructor or set as
attributes*.
See Also
--------
gridminimise : function
Implements the basic functionality behind this object
'''
def __init__(self, base, measure=None, nfolds=10, params={}, annotate=False):
self.params = params
self.base = base
self.nfolds = 10
self.measure = measure
self.annotate = annotate
def is_multi_class(self):
return self.base.is_multi_class()
def train(self, features, labels, normalisedlabels=False, **kwargs):
best,value = gridminimise(self.base, features, labels, self.params, self.measure, self.nfolds, return_value=True, train_kwargs=kwargs)
_set_options(self.base, best)
model = self.base.train(features, labels, normalisedlabels=normalisedlabels, **kwargs)
if self.annotate:
model.arguments = best
model.value = value
return model
milk-0.5/milk/supervised/_lasso.cpp 0000664 0001750 0001750 00000012575 12045727111 021475 0 ustar luispedro luispedro 0000000 0000000 // Copyright (C) 2012, Luis Pedro Coelho
// License: MIT
#include
#include
#include
#include
#include
#include
#include
using namespace Eigen;
extern "C" {
#include
#include
}
namespace {
template
int random_int(T& random, const int max) {
std::uniform_int_distribution dist(0, max - 1);
return dist(random);
}
inline
float soft(const float val, const float lam) {
return std::copysign(std::fdim(std::fabs(val), lam), val);
}
typedef Map, Aligned> MapXAf;
bool has_nans(const MapXAf& X) {
for (int i = 0; i != X.rows(); ++i) {
for (int j = 0; j != X.cols(); ++j) {
if (std::isnan(X(i,j))) return true;
}
}
return false;
}
struct lasso_solver {
lasso_solver(const MapXAf& X, const MapXAf& Y, const MapXAf& W, MapXAf& B, const int max_iter, const float lam, const float eps)
:X(X)
,Y(Y)
,W(W)
,B(B)
,max_iter(max_iter)
,lam(lam)
,eps(eps)
{ }
void next_coords(int& i, int& j) {
++j;
if (j == B.cols()) {
j = 0;
++i;
if (i == B.rows()) {
i = 0;
}
}
}
int solve() {
Matrix residuals;
int i = 0;
int j = -1;
// sweep is whether we are doing a whole-B sweep (i.e., even looking at
// non-zero Bs)
// changed is whether something has changed
// We set this up to trigger computation of residuals and a sweep on
// the first iteration
bool sweep = false;
bool changed = false;
assert(!has_nans(X));
assert(!has_nans(Y));
assert(!has_nans(B));
for (int it = 0; it != max_iter; ++it) {
this->next_coords(i, j);
if (i == 0 && j == 0) {
if (sweep && !changed) {
return it;
}
if (!changed) {
sweep = true;
// Reset the residuals matrix to the best values to avoid
// drift due to successive rounding:
residuals = Y - B*X;
} else {
sweep = false;
}
changed = false;
}
if (!sweep && B(i,j) == 0.0) continue;
// We now set βᵢⱼ holding everything else fixed. This comes down
// to a very simple 1-dimensional problem.
// We remember the current value in order to compute update below
const float prev = B(i,j);
assert(Y.cols() == W.cols());
assert(Y.cols() == X.cols());
assert(Y.cols() == residuals.cols());
const float x2 = (W.row(i).array()*X.row(j).array() * X.row(j).array()).sum();
const float xy = (W.row(i).array()*X.row(j).array() * residuals.row(i).array()).sum();
const float raw_step = (x2 == 0.0 ? 0.0 : xy/x2);
const float best = soft(prev + raw_step, lam);
const float step = best - prev;
if (std::fabs(step) > eps) {
assert(!std::isnan(best));
B(i,j) = best;
residuals.row(i) -= step*X.row(j);
changed = true;
}
}
return max_iter;
}
std::mt19937 r;
const MapXAf& X;
const MapXAf& Y;
const MapXAf& W;
MapXAf& B;
const int max_iter;
const float lam;
const float eps;
};
MapXAf as_eigen(PyArrayObject* arr) {
assert(PyArray_EquivTypenums(PyArray_TYPE(arr), NPY_FLOAT32));
assert(PyArray_ISCARRAY_RO(arr));
return MapXAf(
static_cast(PyArray_DATA(arr)),
PyArray_DIM(arr, 0),
PyArray_DIM(arr, 1));
}
const char* errmsg = "INTERNAL ERROR";
PyObject* py_lasso(PyObject* self, PyObject* args) {
PyArrayObject* X;
PyArrayObject* Y;
PyArrayObject* W;
PyArrayObject* B;
int max_iter;
float lam;
float eps;
if (!PyArg_ParseTuple(args, "OOOOiff", &X, &Y, &W, &B, &max_iter, &lam, &eps)) return NULL;
if (!PyArray_Check(X) || !PyArray_ISCARRAY_RO(X) ||
!PyArray_Check(Y) || !PyArray_ISCARRAY_RO(Y) ||
!PyArray_Check(W) || !PyArray_ISCARRAY_RO(W) ||
!PyArray_Check(B) || !PyArray_ISCARRAY(B) ||
!PyArray_EquivTypenums(PyArray_TYPE(X), NPY_FLOAT32) ||
!PyArray_EquivTypenums(PyArray_TYPE(Y), NPY_FLOAT32) ||
!PyArray_EquivTypenums(PyArray_TYPE(W), NPY_FLOAT32) ||
!PyArray_EquivTypenums(PyArray_TYPE(B), NPY_FLOAT32)) {
PyErr_SetString(PyExc_RuntimeError,errmsg);
return 0;
}
MapXAf mX = as_eigen(X);
MapXAf mY = as_eigen(Y);
MapXAf mW = as_eigen(W);
MapXAf mB = as_eigen(B);
max_iter *= mB.size();
lasso_solver solver(mX, mY, mW, mB, max_iter, lam, eps);
const int iters = solver.solve();
return Py_BuildValue("i", iters);
}
PyMethodDef methods[] = {
{"lasso", py_lasso, METH_VARARGS , "Do NOT call directly.\n" },
{NULL, NULL,0,NULL},
};
const char * module_doc =
"Internal Module.\n"
"\n"
"Do NOT use directly!\n";
} // namespace
extern "C"
void init_lasso()
{
import_array();
(void)Py_InitModule3("_lasso", methods, module_doc);
}
milk-0.5/milk/supervised/classifier.py 0000664 0001750 0001750 00000005736 12043765776 022231 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2012, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
from __future__ import division
import numpy as np
from .normalise import normaliselabels
from .base import supervised_model
__all__ = ['normaliselabels', 'ctransforms']
class threshold_model(object):
'''
threshold_model
Attributes
----------
threshold : float
threshold value
'''
def __init__(self, threshold=.5):
self.threshold = .5
def apply(self, f):
return f >= self.threshold
class fixed_threshold_learner(object):
def __init__(self, threshold=.5):
self.threshold = threshold
def train(self, features, labels, **kwargs):
return threshold_model(self.threshold)
class ctransforms_model(supervised_model):
'''
model = ctransforms_model(models)
A model that consists of a series of transformations.
See Also
--------
ctransforms
'''
def __init__(self, models):
self.models = models
def apply_many(self, features):
for m in self.models:
features = m.apply_many(features)
return features
def apply(self,features):
for T in self.models:
features = T.apply(features)
return features
class ctransforms(object):
'''
ctransf = ctransforms(c0, c1, c2, ...)
Concatenate transforms.
'''
def __init__(self,*args):
self.transforms = args
def train(self, features, labels, **kwargs):
models = []
model = None
for T in self.transforms:
if model is not None:
features = np.array([model.apply(f) for f in features])
model = T.train(features, labels, **kwargs)
models.append(model)
return ctransforms_model(models)
def set_option(self, opt, val):
idx, opt = opt
self.transforms[idx].set_option(opt,val)
milk-0.5/milk/supervised/multi.py 0000664 0001750 0001750 00000023737 12043765776 021240 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2011, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division
from .classifier import normaliselabels
from .base import supervised_model, base_adaptor
import numpy as np
__all__ = [
'one_against_rest',
'one_against_one',
'one_against_rest_multi',
'ecoc_learner',
'multi_tree_learner',
]
def _asanyarray(f):
try:
return np.asanyarray(f)
except:
return np.array(f, dtype=object)
class one_against_rest(base_adaptor):
'''
Implements one vs. rest classification strategy to transform
a binary classifier into a multi-class classifier.
classifier = one_against_rest(base)
base must obey the classifier interface
Example
-------
::
multi = one_against_rest(milk.supervised.simple_svm())
model = multi.train(training_features,labels)
print model.apply(testing_features)
See Also
--------
one_against_one
'''
def train(self, features, labels, normalisedlabels=False):
labels, names = normaliselabels(labels)
nclasses = labels.max() + 1
models = []
for i in xrange(nclasses):
model = self.base.train(features, (labels == i).astype(int), normalisedlabels=True)
models.append(model)
return one_against_rest_model(models, names)
class one_against_rest_model(supervised_model):
def __init__(self, models, names):
self.models = models
self.nclasses = len(self.models)
self.names = names
def apply(self, feats):
vals = np.array([c.apply(feats) for c in self.models])
(idxs,) = np.where(vals)
if len(idxs) == 1:
(label,) = idxs
elif len(idxs) == 0:
label = 0
else:
label = idxs[0]
return self.names[label]
class one_against_one(base_adaptor):
'''
Implements one vs. one classification strategy to transform
a binary classifier into a multi-class classifier.
classifier = one_against_one(base)
base must obey the classifier interface
Example
-------
::
multi = one_against_one(milk.supervised.simple_svm())
multi.train(training_features,labels)
print multi.apply(testing_features)
See Also
--------
one_against_rest
'''
def train(self, features, labels, weights=None, **kwargs):
'''
one_against_one.train(objs,labels)
'''
labels, names = normaliselabels(labels)
if weights is not None:
weights = np.asanyarray(weights)
features = _asanyarray(features)
nclasses = labels.max() + 1
models = [ [None for i in xrange(nclasses)] for j in xrange(nclasses)]
child_kwargs = kwargs.copy()
child_kwargs['normalisedlabels'] = True
for i in xrange(nclasses):
for j in xrange(i+1, nclasses):
idxs = (labels == i) | (labels == j)
if not np.any(idxs):
raise ValueError('milk.multi.one_against_one: Pair-wise classifier has no data')
if weights is not None:
child_kwargs['weights'] = weights[idxs]
model = self.base.train(features[idxs], (labels[idxs]==i).astype(int), **child_kwargs)
models[i][j] = model
return one_against_one_model(models, names)
class one_against_one_model(supervised_model):
def __init__(self, models, names):
self.models = models
self.names = _asanyarray(names)
self.nclasses = len(models)
def apply_many(self, features):
nc = self.nclasses
votes = np.zeros((nc, len(features)))
for i in xrange(nc):
for j in xrange(i+1,nc):
vs = self.models[i][j].apply_many(features)
vs = _asanyarray(vs)
votes[i] += (vs > 0)
votes[j] += (vs <= 0)
return self.names[votes.argmax(0)]
def apply(self,feats):
'''
one_against_one.apply(objs)
Classify one single object.
'''
nc = self.nclasses
votes = np.zeros(nc)
for i in xrange(nc):
for j in xrange(i+1,nc):
c = self.models[i][j].apply(feats)
if c:
votes[i] += 1
else:
votes[j] += 1
return self.names[votes.argmax(0)]
class one_against_rest_multi_model(supervised_model):
def __init__(self, models):
self.models = models
def apply(self, feats):
return [lab for lab,model in self.models.iteritems() if model.apply(feats)]
class one_against_rest_multi(base_adaptor):
'''
learner = one_against_rest_multi()
model = learner.train(features, labels)
classes = model.apply(f_test)
This for multi-label problem (i.e., each instance can have more than one label).
'''
def train(self, features, labels, normalisedlabels=False, weights=None):
'''
'''
import operator
all_labels = set()
for ls in labels:
all_labels.update(ls)
models = {}
kwargs = { 'normalisedlabels': True }
if weights is not None:
kwargs['weights'] = weights
for label in all_labels:
nlabels = np.array([int(label in ls) for ls in labels])
models[label] = self.base.train(features, nlabels, **kwargs)
return one_against_rest_multi_model(models)
def _solve_ecoc_model(codes, p):
try:
import scipy.optimize
except ImportError:
raise ImportError("milk.supervised.ecoc: fitting ECOC probability models requires scipy.optimize")
z,_ = scipy.optimize.nnls(codes.T, p)
z /= z.sum()
return z
class ecoc_model(supervised_model):
def __init__(self, models, codes, return_probability):
self.models = models
self.codes = codes
self.return_probability = return_probability
def apply(self, f):
word = np.array([model.apply(f) for model in self.models])
if self.return_probability:
return _solve_ecoc_model(self.codes, word)
else:
word = word.astype(bool)
errors = (self.codes != word).sum(1)
return np.argmin(errors)
class ecoc_learner(base_adaptor):
'''
Implements error-correcting output codes for reducing a multi-class problem
to a set of binary problems.
Reference
---------
"Solving Multiclass Learning Problems via Error-Correcting Output Codes" by
T. G. Dietterich, G. Bakiri in Journal of Artificial Intelligence
Research, Vol 2, (1995), 263-286
'''
def __init__(self, base, probability=False):
base_adaptor.__init__(self, base)
self.probability = probability
def train(self, features, labels, normalisedlabels=False, **kwargs):
if normalisedlabels:
labelset = np.unique(labels)
else:
labels,names = normaliselabels(labels)
labelset = np.arange(len(names))
k = len(labelset)
n = 2**(k-1)
codes = np.zeros((k,n),bool)
for k_ in xrange(1,k):
codes[k_].reshape( (-1, 2**(k-k_-1)) )[::2] = 1
codes = ~codes
# The last column of codes is not interesting (all 1s). The array is
# actually of size 2**(k-1)-1, but it is easier to compute the full
# 2**(k-1) and then ignore the last element.
codes = codes[:,:-1]
models = []
for code in codes.T:
nlabels = np.zeros(len(labels), int)
for ell,c in enumerate(code):
if c:
nlabels[labels == ell] = 1
models.append(self.base.train(features, nlabels, normalisedlabels=True, **kwargs))
return ecoc_model(models, codes, self.probability)
def split(counts):
groups = ([],[])
weights = np.zeros(2, float)
in_order = counts.argsort()
for s in in_order[::-1]:
g = weights.argmin()
groups[g].append(s)
weights[g] += counts[s]
return groups
class multi_tree_model(supervised_model):
def __init__(self, model):
self.model = model
def apply(self, feats):
def ap_recursive(smodel):
if len(smodel) == 1:
return smodel[0]
model,left,right = smodel
if model.apply(feats): return ap_recursive(left)
else: return ap_recursive(right)
return ap_recursive(self.model)
class multi_tree_learner(base_adaptor):
'''
Implements a multi-class learner as a tree of binary decisions.
At each level, labels are split into 2 groups in a way that attempt to
balance the number of examples on each side (and not the number of labels
on each side). This mean that on a 4 class problem with a distribution like
[ 50% 25% 12.5% 12.5%], the "optimal" splits are
o
/ \
/ \
[0] o
/ \
[1] o
/ \
[2][3]
where all comparisons are perfectly balanced.
'''
def train(self, features, labels, normalisedlabels=False, **kwargs):
if not normalisedlabels:
labels,names = normaliselabels(labels)
labelset = np.arange(len(names))
else:
labels = np.asanyarray(labels)
labelset = np.arange(labels.max()+1)
def recursive(labelset, counts):
if len(labelset) == 1:
return labelset
g0,g1 = split(counts)
nlabels = np.array([(ell in g0) for ell in labels], int)
model = self.base.train(features, nlabels, normaliselabels=True, **kwargs)
m0 = recursive(labelset[g0], counts[g0])
m1 = recursive(labelset[g1], counts[g1])
return (model, m0, m1)
counts = np.zeros(labels.max()+1)
for ell in labels:
counts[ell] += 1
return multi_tree_model(recursive(np.arange(labels.max()+1), counts))
milk-0.5/milk/supervised/_tree.cpp 0000664 0001750 0001750 00000011627 11723217323 021311 0 ustar luispedro luispedro 0000000 0000000 // Copyright (C) 2010, Luis Pedro Coelho
// License: MIT
#include
#include
#include
#include
extern "C" {
#include
#include
}
namespace {
template
double set_entropy(const T* data, const int N, double* counts, long clen) {
for (int i = 0; i != clen; ++i) counts[i] = 0.;
for (int i = 0; i != N; ++i) {
int value = data[i];
if (value >= clen) {
PyErr_SetString(PyExc_RuntimeError, "milk.supervised.tree.set_entropy: label value too large. aborting");
return 0;
}
counts[value] += 1.;
}
// Here is the formula we use:
//
// H = - \sum px \log(px)
// = - \sum (cx/N) \log( cx / N)
// = - 1/N \sum { cx [ \log cx - \log N ] }
// = - 1/N { (\sum cx \log cx ) - ( \sum cx \log N ) }
// = - 1/N { (\sum cx \log cx ) - N \log N }
// = ( - 1/N \sum cx \log cx ) + \log N
double entropy = 0.;
for (int i = 0; i != clen; ++i) {
double cx = counts[i];
if (cx) entropy += cx * std::log(cx);
}
entropy /= -N;
entropy += std::log(double(N));
return entropy;
}
double set_entropy(PyArrayObject* points, double* counts, long clen) {
const int N = PyArray_SIZE(points);
#define HANDLE(type_marker, type) \
if (PyArray_TYPE(points) == type_marker) return set_entropy(reinterpret_cast(PyArray_DATA(points)), N, counts, clen);
HANDLE(NPY_INT, int)
HANDLE(NPY_LONG, long)
#undef HANDLE
assert(false);
return 0.;
}
PyObject* py_set_entropy(PyObject* self, PyObject* args) {
const char* errmsg = "Arguments were not what was expected for set_entropy.\n"
"This is an internal function: Do not call directly unless you know exactly what you're doing.\n";
PyArrayObject* labels;
PyArrayObject* counts;
if (!PyArg_ParseTuple(args, "OO", &labels, &counts)) {
PyErr_SetString(PyExc_RuntimeError,errmsg);
return 0;
}
if (!PyArray_Check(labels) || (PyArray_TYPE(labels) != NPY_INT && PyArray_TYPE(labels) != NPY_LONG) || !PyArray_ISCONTIGUOUS(labels) ||
!PyArray_Check(counts) || PyArray_TYPE(counts) != NPY_DOUBLE || !PyArray_ISCONTIGUOUS(counts)) {
PyErr_SetString(PyExc_RuntimeError,errmsg);
return 0;
}
double* cdata = reinterpret_cast(PyArray_DATA(counts));
const int nlabels = PyArray_DIM(counts, 0);
double res = set_entropy(labels, cdata, nlabels);
return PyFloat_FromDouble(res);
}
PyObject* py_information_gain(PyObject* self, PyObject* args) {
const char* errmsg = "Arguments were not what was expected for information_gain.\n"
"This is an internal function: Do not call directly unless you know exactly what you're doing.\n";
PyArrayObject* labels0;
PyArrayObject* labels1;
if (!PyArg_ParseTuple(args, "OO", &labels0, &labels1)) {
PyErr_SetString(PyExc_RuntimeError,errmsg);
return 0;
}
if (!PyArray_Check(labels0) || (PyArray_TYPE(labels0) != NPY_INT && PyArray_TYPE(labels0) != NPY_LONG) || !PyArray_ISCONTIGUOUS(labels0) ||
!PyArray_Check(labels1) || (PyArray_TYPE(labels1) != NPY_INT && PyArray_TYPE(labels1) != NPY_LONG) || !PyArray_ISCONTIGUOUS(labels1)) {
PyErr_SetString(PyExc_RuntimeError,errmsg);
std::cout << PyArray_TYPE(labels0) << " " << PyArray_TYPE(labels1) << '\n';
return 0;
}
double* counts;
double counts_array[8];
long clen = 0;
#define GET_MAX(index) \
const int N ## index = PyArray_DIM(labels ## index, 0); \
{ \
for (int i = 0; i != N ## index; ++i) { \
long val = 0; \
if (PyArray_TYPE(labels ## index) == NPY_INT) val = *reinterpret_cast(PyArray_GETPTR1(labels ## index, i)); \
else if (PyArray_TYPE(labels ## index) == NPY_LONG) val = *reinterpret_cast(PyArray_GETPTR1(labels ## index, i)); \
if (val > clen) clen = val; \
} \
}
GET_MAX(0);
GET_MAX(1);
++clen;
if (clen > 8) {
counts = new(std::nothrow) double[clen];
if (!counts) {
PyErr_NoMemory();
return 0;
}
} else {
counts = counts_array;
}
const double N = N0 + N1;
double H = - N0/N * set_entropy(labels0, counts, clen) - N1/N * set_entropy(labels1, counts, clen);
if (clen > 8) delete [] counts;
return PyFloat_FromDouble(H);
}
PyMethodDef methods[] = {
{"set_entropy", py_set_entropy, METH_VARARGS , "Do NOT call directly.\n" },
{"information_gain", py_information_gain, METH_VARARGS , "Do NOT call directly.\n" },
{NULL, NULL,0,NULL},
};
const char * module_doc =
"Internal Module.\n"
"\n"
"Do NOT use directly!\n";
} // namespace
extern "C"
void init_tree()
{
import_array();
(void)Py_InitModule3("_tree", methods, module_doc);
}
milk-0.5/milk/supervised/precluster_learner.py 0000664 0001750 0001750 00000000032 11723217323 023745 0 ustar luispedro luispedro 0000000 0000000 from .precluster import *
milk-0.5/milk/supervised/_svm.cpp 0000664 0001750 0001750 00000102347 11723217323 021157 0 ustar luispedro luispedro 0000000 0000000 // Copyright (C) 2008, Luis Pedro Coelho
// Copyright (c) 2000-2008 Chih-Chung Chang and Chih-Jen Lin (LIBSVM Code)
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in
// all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
// THE SOFTWARE.
#include
#include
#include
#include
#include
#include
#include
#include
//#include
//#include
//using __gnu_debug::vector;
//using __gnu_debug::list;
using std::vector;
using std::list;
extern "C" {
#include
#include
}
namespace {
const double INF = HUGE_VAL;
// This is a boost function
// Copied here for convenience.
template
inline Iter prior(Iter it) {
--it;
return it;
}
using std::max;
using std::min;
template
T median(T bot, T mid, T hi) {
if (mid < bot) return bot;
if (mid > hi) return hi;
return mid;
}
struct SMO_Exception {
SMO_Exception(const char* msg): msg(msg) { }
const char* msg;
};
struct Python_Exception { };
void check_for_interrupts() {
PyErr_CheckSignals();
if (PyErr_Occurred()) {
throw Python_Exception();
}
}
class KernelComputation {
public:
virtual ~KernelComputation() { }
virtual double do_kernel(int i1, int i2) const = 0;
};
class KernelCache {
public:
KernelCache(std::auto_ptr computation_, int N, int cache_nr_doubles);
virtual ~KernelCache();
double* get_kline(int idx, int size = -1);
double* get_diag();
double kernel_apply(int i1, int i2) const;
protected:
virtual double do_kernel(int i1, int i2) const {
return computation_->do_kernel(i1,i2);
}
const int N_;
private:
std::auto_ptr computation_;
double ** cache_;
double * dcache_;
int cache_free_;
list cache_lru;
vector::iterator> cache_iter;
};
KernelCache::KernelCache(std::auto_ptr computation, int N, int cache_nr_floats):
N_(N),
computation_(computation),
dcache_(0) {
cache_ = new double*[N_];
for (int i = 0; i != N_; ++i) cache_[i] = 0;
cache_free_ = (cache_nr_floats/N_);
cache_iter.resize(N_,cache_lru.end());
}
KernelCache::~KernelCache() {
for (int i = 0; i != N_; ++i) delete [] cache_[i];
delete [] cache_;
delete [] dcache_;
}
double* KernelCache::get_kline(int idx, int s) {
if (s == -1) s = N_;
assert (s <= N_);
if (!cache_[idx]) {
if (!cache_free_) {
int to_remove = cache_lru.front();
cache_lru.pop_front();
cache_[idx] = cache_[to_remove];
cache_[to_remove] = 0;
} else {
cache_[idx] = new double[N_];
--cache_free_;
}
for (int i = 0; i != N_; ++i) {
if (i == idx && dcache_) cache_[i][i] = dcache_[i];
else if(i != idx && cache_[i]) cache_[idx][i] = cache_[i][idx];
else cache_[idx][i] = do_kernel(idx,i);
}
} else {
cache_lru.erase(cache_iter[idx]);
}
cache_lru.push_back(idx);
cache_iter[idx]=prior(cache_lru.end());
return cache_[idx];
}
double* KernelCache::get_diag() {
if (!dcache_) {
dcache_ = new double[N_];
for (int i = 0; i != N_; ++i) {
if (cache_[i]) dcache_[i] = cache_[i][i];
else dcache_[i] = do_kernel(i,i);
}
}
return dcache_;
}
/***
* Returns the value of Kernel(X_i1, X_i2).
* Uses the cache if possible, but does not update it.
*/
double KernelCache::kernel_apply(int i1, int i2) const {
if (cache_[i1]) {
assert(do_kernel(i1,i2) == cache_[i1][i2]);
return cache_[i1][i2];
}
if (cache_[i2]) {
assert(do_kernel(i1,i2) == cache_[i2][i1]);
return cache_[i2][i1];
}
if (i1 == i2 && dcache_) {
assert(do_kernel(i1,i2) == dcache_[i1]);
return dcache_[i1];
}
return do_kernel(i1,i2);
}
class PyKernel : public KernelComputation {
PyKernel(const PyKernel&);
public:
PyKernel(PyObject* X, PyObject* kernel, int N);
~PyKernel();
protected:
virtual double do_kernel(int i1, int i2) const;
private:
PyObject* const X_;
PyObject* const pykernel_;
const int N_;
};
PyKernel::PyKernel(PyObject* X, PyObject* kernel, int N):
X_(X),
pykernel_(kernel),
N_(N)
{
Py_INCREF(X);
Py_INCREF(kernel);
}
PyKernel::~PyKernel() {
Py_DECREF(X_);
Py_DECREF(pykernel_);
}
double PyKernel::do_kernel(int i1, int i2) const {
assert(i1 < N_);
assert(i2 < N_);
PyObject* obj1 = PySequence_GetItem(X_,i1);
PyObject* obj2 = PySequence_GetItem(X_,i2);
if (!obj1 || !obj2) {
Py_XDECREF(obj1);
Py_XDECREF(obj2);
throw SMO_Exception("svm.eval_SMO: Unable to access element in X array");
}
PyObject* arglist = Py_BuildValue("(OO)",obj1,obj2);
PyObject* result = PyEval_CallObject(pykernel_,arglist);
Py_XDECREF(obj1);
Py_XDECREF(obj2);
Py_DECREF(arglist);
if (!result) {
check_for_interrupts();
throw SMO_Exception("svm.eval_SMO: Unable to call kernel");
}
double val = PyFloat_AsDouble(result);
Py_DECREF(result);
return val;
}
class RBFKernel : public KernelComputation {
public:
RBFKernel(PyArrayObject* X, double gamma);
~RBFKernel();
protected:
virtual double do_kernel(int i1, int i2) const;
private:
PyArrayObject* X_;
double ngamma_;
const int N_;
const int N1_;
};
RBFKernel::RBFKernel(PyArrayObject* X, double gamma):
X_(reinterpret_cast(X)),
ngamma_(-1./gamma),
N_(PyArray_DIM(X,0)),
N1_(PyArray_DIM(X,1))
{
if (!PyArray_Check(X)) {
throw SMO_Exception("RBF Kernel used, but not with numpy array.");
}
if (!PyArray_ISCARRAY(X)) {
throw SMO_Exception("RBF Kernel used but not with CARRAY.");
}
Py_INCREF(X);
}
RBFKernel::~RBFKernel() {
Py_DECREF(X_);
}
double RBFKernel::do_kernel(int i1, int i2) const {
assert(i1 < N_);
assert(i2 < N_);
const double* data1 = static_cast(PyArray_GETPTR1(X_,i1));
const double* data2 = static_cast(PyArray_GETPTR1(X_,i2));
double sumdiff = 0.;
for (int i = 0; i != N1_; ++i) {
double diff = data1[i]-data2[i];
sumdiff += diff * diff;
}
sumdiff *= ngamma_;
double res = std::exp(sumdiff);
return res;
}
class DotKernel : public KernelComputation {
public:
explicit DotKernel(PyArrayObject* X);
~DotKernel();
protected:
virtual double do_kernel(int i1, int i2) const;
private:
PyArrayObject* X_;
const int N_;
const int N1_;
};
DotKernel::DotKernel(PyArrayObject* X):
X_(reinterpret_cast(X)),
N_(PyArray_DIM(X,0)),
N1_(PyArray_DIM(X,1))
{
if (!PyArray_Check(X)) {
throw SMO_Exception("Dot Kernel used, but not with numpy array.");
}
if (!PyArray_ISCARRAY(X)) {
throw SMO_Exception("Dot Kernel used but not with CARRAY.");
}
Py_INCREF(X);
}
DotKernel::~DotKernel() {
Py_DECREF(X_);
}
double DotKernel::do_kernel(int i1, int i2) const {
assert(i1 < N_);
assert(i2 < N_);
const double* data1 = static_cast(PyArray_GETPTR1(X_,i1));
const double* data2 = static_cast(PyArray_GETPTR1(X_,i2));
double dotsum = 0.;
for (int i = 0; i != N1_; ++i) {
dotsum += data1[i] * data2[i];
}
return dotsum;
}
class PrecomputedKernel : public KernelComputation {
public:
PrecomputedKernel(PyArrayObject* X);
~PrecomputedKernel();
protected:
virtual double do_kernel(int i1, int i2) const;
private:
PyArrayObject* X_;
};
PrecomputedKernel::PrecomputedKernel(PyArrayObject* X):
X_(X)
{
if (!PyArray_Check(X)) {
throw SMO_Exception("PrecomputedKernel used, but not with numpy array.");
}
if (!PyArray_ISCARRAY(X)) {
throw SMO_Exception("Precomputed used but not with CARRAY.");
}
Py_INCREF(X);
}
PrecomputedKernel::~PrecomputedKernel() {
Py_DECREF(X_);
}
double PrecomputedKernel::do_kernel(int i1, int i2) const {
const double* data = static_cast(PyArray_GETPTR2(X_,i1,i2));
return *data;
}
std::auto_ptr get_kernel(PyObject* X, PyObject* kernel) {
typedef std::auto_ptr res_type;
if (PyCallable_Check(kernel)) return res_type(new PyKernel(X, kernel, PySequence_Length(X)));
if (!PyTuple_Check(kernel) || PyTuple_Size(kernel) != 2) throw SMO_Exception("Cannot parse kernel.");
PyObject* type = PyTuple_GET_ITEM(kernel,0);
PyObject* arg = PyTuple_GET_ITEM(kernel,1);
if (!PyInt_Check(type) || !PyFloat_Check(arg)) throw SMO_Exception("Cannot parse kernel (wrong types)");
long type_nr = PyInt_AsLong(type);
double arg_value = PyFloat_AsDouble(arg);
switch (type_nr) {
case 0:
return res_type(new RBFKernel(reinterpret_cast(X), arg_value));
case 1:
return res_type(new PrecomputedKernel(reinterpret_cast(X)));
case 2:
return res_type(new DotKernel(reinterpret_cast(X)));
default:
throw SMO_Exception("Unknown kernel type!");
}
}
class LIBSVM_KernelCache : public KernelCache {
public:
LIBSVM_KernelCache(const int* Y, std::auto_ptr kernel, int N, int cache_nr_doubles)
:KernelCache(kernel,N,cache_nr_doubles),
Y_(Y)
{ }
private:
double do_kernel(int i, int j) const {
double res = KernelCache::do_kernel(i,j);
return res * Y_[i] * Y_[j];
}
const int* Y_;
};
class SMO {
public:
SMO(PyObject* X, int* Y, double* Alphas, double& b, double C, int N, PyObject* kernel, double eps, double tol, int cache_size):
Alphas(Alphas),
Y(Y),
b(b),
C(C),
N(N),
cache_(get_kernel(X,kernel),N,cache_size),
eps(eps),
tol(tol) { }
void optimise();
double apply(int) const;
bool take_step(int,int);
bool examine_example(int);
double get_error(int) const;
private:
double* Alphas;
int* Y;
double& b;
double C;
int N;
mutable KernelCache cache_;
const double eps;
const double tol;
};
double SMO::get_error(int j) const {
return apply(j) - Y[j];
}
double SMO::apply(int j) const {
double sum = -b;
double* Kernel_Line = cache_.get_kline(j);
for (int i = 0; i != N; ++i) {
if (Alphas[i] != C) {
sum += Y[i] * Alphas[i] * Kernel_Line[i];
}
}
//std::cout << "SMO::apply( " << j << " ): " << sum << '\n';
return sum;
}
bool SMO::take_step(int i1, int i2) {
//std::cout << "take_step( " << i1 << ", " << i2 << " );\n";
if (i1 == i2) return false;
const double alpha1 = Alphas[i1];
const double alpha2 = Alphas[i2];
const int y1 = Y[i1];
const int y2 = Y[i2];
double L, H;
if (y1 != y2) {
L = max(0.,alpha2-alpha1);
H = min(C,C+alpha2-alpha1);
} else {
L = max(0.,alpha1+alpha2-C);
H = min(C,alpha1+alpha2);
}
if (L == H) return false;
const int s = y1*y2;
const double E1 = get_error(i1);
const double E2 = get_error(i2);
const double k11 = cache_.kernel_apply(i1,i1);
const double k12 = cache_.kernel_apply(i1,i2);
const double k22 = cache_.kernel_apply(i2,i2);
const double eta = 2*k12-k11-k22;
double a1,a2;
if (eta < 0) {
a2 = alpha2-y2*(E1-E2)/eta;
a2 = median(L,a2,H);
} else {
double gamma = alpha1+s*alpha2; // Eq. (12.22)
double v1=E1+y1+b-y1*alpha1*k11-y2*alpha2*k12; // Eq. (12.21) // Note that f(x1) = E1 + y1
double v2=E2+y2+b-y1*alpha1*k12-y2*alpha2*k22; // Eq. (12.21)
double L_obj = gamma-s*L+L-.5*k11*(gamma-s*L)*(gamma-s*L)-.5*k22*L*L-s*k12*(gamma-s*L)*L-y1*(gamma-s*L)*v1-y2*L*v2; // # + W_const # Eq. (12.23)
double H_obj = gamma-s*H+H-.5*k11*(gamma-s*H)*(gamma-s*L)-.5*k22*H*H-s*k12*(gamma-s*H)*H-y1*(gamma-s*H)*v1-y2*H*v2; // # + W_const # Eq. (12.23)
if (L_obj > H_obj + eps) {
a2 = L;
} else if (L_obj < H_obj - eps) {
a2 = H;
} else {
a2 = alpha2;
}
}
if (a2 < tol) a2 = 0;
else if (a2 > C-tol) a2 = C;
if (std::abs(a2-alpha2) < eps*(a2+alpha2+eps)) return false;
a1 = alpha1+s*(alpha2-a2);
if (a1 < tol) a1 = 0;
if (a1 > C-tol) a1 = C;
// update everything
Alphas[i1]=a1;
Alphas[i2]=a2;
double b1 = E1 + Y[i1]*(a1-alpha1)*k11+Y[i2]*(a2-alpha2)*k12+b; // Eq. (12.9)
double b2 = E2 + Y[i1]*(a1-alpha1)*k12+Y[i2]*(a2-alpha2)*k22+b; // Eq. (12.10)
const double new_b = (b1+b2)/2.;
/*
#for i in xrange(N):
# if Alphas[i] in (0,C):
# continue
# elif i == i1 or i == i2:
# E[i] = 0
# else:
# E[i] += y1*(a1-alpha1)*kernel_apply(i1,i)+y2*(a2-alpha2)*kernel_apply(i2,i) + (b-new_b) # Eq. (12.11)
#E[i1]=f_at(i1)-y1
#E[i2]=f_at(i2)-y2
*/
b = new_b;
return true;
}
bool SMO::examine_example(int i2) {
//std::cout << "examine_example( " << i2 << " ) " << std::endl;
const int y2 = Y[i2];
const double alpha2 = Alphas[i2];
const double E2 = get_error(i2);
const double r2 = E2 * y2;
//#print 'alpha2', alpha2, 'E2', E2, 'r2', r2
if ( ( (r2 < -tol) && (alpha2 < C) ) ||
( (r2 > tol) && (alpha2 > 0) )){
int best_i1 = -1;
double bestE = -1;
for (int i = 0; i != N; ++i) {
if (Alphas[i] != 0 && Alphas[i] != C) {
double dE = E2-get_error(i);
if (dE < 0.) dE = -dE;
if (dE > bestE) {
bestE=dE;
best_i1 = i;
}
}
}
if (best_i1 != -1 && take_step(best_i1,i2)) return true;
for (int i1 = 0; i1 != N; ++i1) {
if (Alphas[i1] && Alphas[i1] != C && take_step(i1,i2)) return true;
}
for (int i1 = 0; i1 != N; ++i1) {
if (take_step(i1,i2)) return true;
}
}
return false;
}
void SMO::optimise() {
b = 0;
for (int i = 0; i != N; ++i) Alphas[i] = 0;
int changed = 0;
bool examineAll = true;
//int iter = 0;
while (changed || examineAll) {
//std::cout << "SMO::optimize loop: " << iter++ << "\n";
check_for_interrupts();
changed = 0;
for (int i = 0; i != N; ++i) {
if (examineAll || (Alphas[i] != 0 && Alphas[i] != C)) {
changed += examine_example(i);
}
}
if (examineAll) examineAll = false;
else if (!changed) examineAll = true;
}
}
void assert_type_contiguous(PyArrayObject* array,int type) {
if (!PyArray_Check(array) ||
PyArray_TYPE(array) != type ||
!PyArray_ISCONTIGUOUS(array)) {
throw SMO_Exception("Arguments to eval_(SMO|LIBSVM) don't conform to expectation. Are you calling this directly? This is an internal function!");
}
}
PyObject* eval_SMO(PyObject* self, PyObject* args) {
try {
PyObject* X;
PyArrayObject* Y;
PyArrayObject* Alphas0;
PyArrayObject* params;
PyObject* kernel;
int cache_size;
if (!PyArg_ParseTuple(args, "OOOOOi", &X, &Y, &Alphas0, ¶ms,&kernel,&cache_size)) {
const char* errmsg = "Arguments were not what was expected for eval_SMO.\n"
"This is an internal function: Do not call directly unless you know what you're doing.\n";
PyErr_SetString(PyExc_RuntimeError,errmsg);
return 0;
}
assert_type_contiguous(Y,NPY_INT32);
assert_type_contiguous(Alphas0,NPY_DOUBLE);
assert_type_contiguous(params,NPY_DOUBLE);
if (PyArray_DIM(params,0) < 4) throw SMO_Exception("eval_SMO: Too few parameters");
int * Yv = static_cast(PyArray_DATA(Y));
double* Alphas = static_cast(PyArray_DATA(Alphas0));
unsigned N = PyArray_DIM(Y,0);
double b = *static_cast(PyArray_GETPTR1(params,0));
double C = *static_cast(PyArray_GETPTR1(params,1));
double eps = *static_cast(PyArray_GETPTR1(params,2));
double tol = *static_cast(PyArray_GETPTR1(params,3));
SMO optimiser(X,Yv,Alphas,b,C,N,kernel,eps,tol,cache_size);
optimiser.optimise();
*static_cast(PyArray_GETPTR1(params,0)) = b; // Write back b
Py_RETURN_NONE;
} catch (const Python_Exception&) {
// if Python_Exception was thrown, then PyErr is already set.
return 0;
} catch (const SMO_Exception& exc) {
PyErr_SetString(PyExc_RuntimeError,exc.msg);
return 0;
} catch (...) {
PyErr_SetString(PyExc_RuntimeError,"Some sort of exception in eval_SMO.");
return 0;
}
}
// The code for LIBSVM_Solver is taken from LIBSVM and adapted to work well in milk
// Copyright (c) 2000-2008 Chih-Chung Chang and Chih-Jen Lin
// Changes were made to make it more similar to our formulation.
// An SMO algorithm in Fan et al., JMLR 6(2005), p. 1889--1918
// Solves:
//
// min 0.5(\alpha^T Q \alpha) + p^T \alpha
//
// y^T \alpha = \delta
// y_i = +1 or -1
// 0 <= alpha_i <= Cp for y_i = 1
// 0 <= alpha_i <= Cn for y_i = -1
//
// Given:
//
// Q, p, y, Cp, Cn, and an initial feasible point \alpha
// l is the size of vectors and matrices
// eps is the stopping tolerance
//
// solution will be put in \alpha, objective value will be put in obj
//
#define info printf
void info_flush() { }
class LIBSVM_Solver {
public:
LIBSVM_Solver(PyObject* X, int* Y, double* Alphas, double* p, double& b, double C, int N, PyObject* kernel, double eps, double tol, int cache_size, bool shrinking):
Alphas(Alphas),
Y(Y),
b(b),
C(C),
N(N),
cache_(Y,get_kernel(X,kernel),N,cache_size),
eps(eps),
tol(tol),
alpha_status(N),
active_size(N),
active_set(N),
p(p),
shrinking(shrinking),
tau(eps)
{ }
virtual ~LIBSVM_Solver() { }
struct SolutionInfo {
double obj;
double rho;
double upper_bound_p;
double upper_bound_n;
double r; // for LIBSVM_Solver_NU
};
void optimise();
protected:
double* Alphas;
int* Y;
double& b;
double C;
const int N;
LIBSVM_KernelCache cache_;
const double eps;
const double tol;
vector G; // gradient of objective function
enum alpha_status_e { lower_bound, upper_bound, free};
vector alpha_status;
int active_size;
vector active_set;
double *p;
vector G_bar; // gradient, if we treat free variables as 0
bool unshrinked; // XXX
bool shrinking;
double tau;
double get_C(int i) const {
return C;
//return (Y[i] > 0)? Cp : Cn;
}
void update_alpha_status(int i)
{
if(Alphas[i] >= get_C(i))
alpha_status[i] = upper_bound;
else if(Alphas[i] <= 0)
alpha_status[i] = lower_bound;
else alpha_status[i] = free;
}
bool is_upper_bound(int i) const { return alpha_status[i] == upper_bound; }
bool is_lower_bound(int i) const { return alpha_status[i] == lower_bound; }
bool is_free(int i) const { return alpha_status[i] == free; }
void swap_index(int i, int j);
void reconstruct_gradient();
virtual bool select_working_set(int &i, int &j);
virtual double calculate_rho();
virtual void do_shrinking();
private:
bool be_shrunken(int i, double Gmax1, double Gmax2);
void print_status() const;
};
void LIBSVM_Solver::swap_index(int i, int j)
{
// We *do not* swap in the cache or kernel
// Therefore *all acesses* to the cache or kernel
// must be of the form cache_.get_kernel(active_set[i])
// instead of cache_.get_kernel(i)!
std::swap(Y[i],Y[j]);
std::swap(G[i],G[j]);
std::swap(alpha_status[i],alpha_status[j]);
std::swap(Alphas[i],Alphas[j]);
std::swap(p[i],p[j]);
std::swap(active_set[i],active_set[j]);
std::swap(G_bar[i],G_bar[j]);
}
void LIBSVM_Solver::reconstruct_gradient()
{
// reconstruct inactive elements of G from G_bar and free variables
if(active_size == N) return;
for(int i=active_size;i= 0) && (i < active_size));
assert((j >= 0) && (j < active_size));
// update alpha[i] and alpha[j], handle bounds carefully
const double *Q_i = cache_.get_kline(active_set[i], active_size);
const double *Q_j = cache_.get_kline(active_set[j], active_size);
const double C_i = get_C(i);
const double C_j = get_C(j);
const double old_alpha_i = Alphas[i];
const double old_alpha_j = Alphas[j];
if(Y[i]!=Y[j]) {
double quad_coef = Q_i[active_set[i]]+Q_j[active_set[j]]+2.0*Q_i[active_set[j]];
if (quad_coef <= 0) quad_coef = tau;
const double delta = (-G[i]-G[j])/quad_coef;
const double diff = Alphas[i] - Alphas[j];
Alphas[i] += delta;
Alphas[j] += delta;
if(diff > 0) {
if(Alphas[j] < 0) {
Alphas[j] = 0;
Alphas[i] = diff;
}
} else {
if(Alphas[i] < 0) {
Alphas[i] = 0;
Alphas[j] = -diff;
}
}
if(diff > C_i - C_j) {
if(Alphas[i] > C_i) {
Alphas[i] = C_i;
Alphas[j] = C_i - diff;
}
} else {
if(Alphas[j] > C_j) {
Alphas[j] = C_j;
Alphas[i] = C_j + diff;
}
}
//print_status();
} else {
double quad_coef = Q_i[active_set[i]]+Q_j[active_set[j]]-2*Q_i[active_set[j]];
if (quad_coef <= 0) quad_coef = tau;
const double delta = (G[i]-G[j])/quad_coef;
double sum = Alphas[i] + Alphas[j];
Alphas[i] -= delta;
Alphas[j] += delta;
if(sum > C_i) {
if(Alphas[i] > C_i) {
Alphas[i] = C_i;
Alphas[j] = sum - C_i;
}
} else {
if(Alphas[j] < 0) {
Alphas[j] = 0;
Alphas[i] = sum;
}
}
if(sum > C_j) {
if(Alphas[j] > C_j) {
Alphas[j] = C_j;
Alphas[i] = sum - C_j;
}
} else {
if(Alphas[i] < 0) {
Alphas[i] = 0;
Alphas[j] = sum;
}
}
}
// update G
const double delta_Alphas_i = Alphas[i] - old_alpha_i;
const double delta_Alphas_j = Alphas[j] - old_alpha_j;
// print_status();
for(int k=0;krho = calculate_rho();
b = calculate_rho();
// calculate objective value
double v = 0;
for(int i=0;iobj = v/2;
// put back the solution
for (int i = 0; i != N; ++i) {
while (active_set[i] != i) {
int j = active_set[i];
std::swap(Y[i],Y[j]); // It's not polite to clobber Y, so put it back
std::swap(Alphas[i],Alphas[j]);
std::swap(active_set[i],active_set[j]);
}
}
//si->upper_bound_p = Cp;
//si->upper_bound_n = Cn;
}
void LIBSVM_Solver::print_status() const {
std::cout << " active_set_size: " << active_size << "\n";
for (int i = 0; i != N; ++i) {
std::cout << " active_set[i]: " << active_set[i] << "\n";
std::cout << " p[i]: " << p[i] << "\n";
std::cout << " G[i]: " << G[i] << "\n";
std::cout << " G_bar[i]: " << G_bar[i] << "\n";
std::cout << " alpha_status[i]: " << alpha_status[i] << "\n";
std::cout << " Y[i]: " << Y[i] << "\n";
std::cout << " Alphas[i]: " << Alphas[i] << "\n";
std::cout << "\n";
}
}
// return true if already optimal, return false otherwise
bool LIBSVM_Solver::select_working_set(int &out_i, int &out_j) {
// return i,j such that
// i: maximizes -y_i * grad(f)_i, i in I_up(\alpha)
// j: minimizes the decrease of obj value
// (if quadratic coefficeint <= 0, replace it with tau)
// -y_j*grad(f)_j < -y_i*grad(f)_i, j in I_low(\alpha)
//print_status();
double Gmax = -INF;
double Gmax2 = -INF;
int Gmax_idx = -1;
int Gmin_idx = -1;
double obj_diff_min = INF;
for(int t=0;t= Gmax) {
Gmax = -Y[t]*G[t];
Gmax_idx = t;
}
}
}
if (Gmax_idx == -1) return true;
const int i = Gmax_idx;
const double* Q_i = cache_.get_kline(active_set[i], N);
const double* QDiag = cache_.get_diag();
for(int j=0;j= Gmax2) Gmax2 = YGj;
if (grad_diff > 0) {
const double quad_coef = Q_i[active_set[i]] + QDiag[active_set[j]] - 2*Y[j]*Y[i]*Q_i[active_set[j]];
const double obj_diff_factor = (quad_coef > 0) ? quad_coef : tau;
const double obj_diff = -(grad_diff*grad_diff)/obj_diff_factor;
if (obj_diff <= obj_diff_min) {
Gmin_idx = j;
obj_diff_min = obj_diff;
}
}
}
}
if (Gmin_idx == -1) return true;
if(Gmax+Gmax2 < eps) return true;
out_i = Gmax_idx;
out_j = Gmin_idx;
return false;
}
bool LIBSVM_Solver::be_shrunken(int i, const double Gmax1, const double Gmax2)
{
if(is_upper_bound(i)) {
if(Y[i]==+1) return -G[i] > Gmax1;
return -G[i] > Gmax2;
} else if(is_lower_bound(i)) {
if (Y[i]==+1) return G[i] > Gmax2;
return G[i] > Gmax1;
}
return false;
}
void LIBSVM_Solver::do_shrinking()
{
double Gmax1 = -INF; // max { -y_i * grad(f)_i | i in I_up(\alpha) }
double Gmax2 = -INF; // max { y_i * grad(f)_i | i in I_low(\alpha) }
// find maximal violating pair first
for(int i=0;i i) {
if (!be_shrunken(active_size, Gmax1, Gmax2)) {
swap_index(i,active_size);
break;
}
--active_size;
}
}
}
// unshrink, check all variables again before final iterations
if(unshrinked || Gmax1 + Gmax2 > eps*10) return;
unshrinked = true;
reconstruct_gradient();
for(int i= N-1; i >= active_size; --i) {
if (!be_shrunken(i, Gmax1, Gmax2)) {
while (active_size < i) {
if (be_shrunken(active_size, Gmax1, Gmax2)) {
swap_index(i,active_size);
break;
}
++active_size;
}
++active_size;
}
}
}
double LIBSVM_Solver::calculate_rho() {
int nr_free = 0;
double ub = INF, lb = -INF, sum_free = 0;
for(int i=0;i0) return sum_free/nr_free;
return (ub+lb)/2;
}
PyObject* eval_LIBSVM(PyObject* self, PyObject* args) {
try {
PyObject* X;
PyArrayObject* Y;
PyArrayObject* Alphas0;
PyArrayObject* p;
PyArrayObject* params;
PyObject* kernel;
int cache_size;
if (!PyArg_ParseTuple(args, "OOOOOOi", &X, &Y, &Alphas0, &p, ¶ms,&kernel,&cache_size)) {
const char* errmsg = "Arguments were not what was expected for eval_LIBSVM.\n"
"This is an internal function: Do not call directly unless you know exactly what you're doing.\n";
PyErr_SetString(PyExc_RuntimeError,errmsg);
return 0;
}
assert_type_contiguous(Y,NPY_INT32);
assert_type_contiguous(Alphas0,NPY_DOUBLE);
assert_type_contiguous(p,NPY_DOUBLE);
assert_type_contiguous(params,NPY_DOUBLE);
if (PyArray_DIM(params,0) < 4) throw SMO_Exception("eval_LIBSVM: Too few parameters");
int * Yv = static_cast(PyArray_DATA(Y));
double* Alphas = static_cast(PyArray_DATA(Alphas0));
double* pv = static_cast(PyArray_DATA(p));
unsigned N = PyArray_DIM(Y,0);
double& b = *static_cast(PyArray_GETPTR1(params,0));
double C = *static_cast(PyArray_GETPTR1(params,1));
double eps = *static_cast(PyArray_GETPTR1(params,2));
double tol = *static_cast(PyArray_GETPTR1(params,3));
LIBSVM_Solver optimiser(X,Yv,Alphas,pv,b,C,N,kernel,eps,tol,cache_size, true);
optimiser.optimise();
Py_RETURN_NONE;
} catch (const Python_Exception&) {
// if Python_Exception was thrown, then PyErr is already set.
return 0;
} catch (const SMO_Exception& exc) {
PyErr_SetString(PyExc_RuntimeError,exc.msg);
return 0;
} catch (...) {
PyErr_SetString(PyExc_RuntimeError,"Some sort of exception in eval_LIBSVM.");
return 0;
}
}
PyMethodDef methods[] = {
{"eval_SMO",eval_SMO, METH_VARARGS , "Do NOT call directly.\n" },
{"eval_LIBSVM",eval_LIBSVM, METH_VARARGS , "Do NOT call directly.\n" },
{NULL, NULL,0,NULL},
};
const char * module_doc =
"Internal SVM Module.\n"
"\n"
"Do NOT use directly!\n";
} // namespace
extern "C"
void init_svm()
{
import_array();
(void)Py_InitModule3("_svm", methods, module_doc);
}
milk-0.5/milk/supervised/base.py 0000664 0001750 0001750 00000001462 11723217323 020767 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2011, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division
class supervised_model(object):
def apply_many(self, fs):
'''
labels = model.apply_many( examples )
This is equivalent to ``map(model.apply, examples)`` but may be
implemented in a faster way.
Parameters
----------
examples : sequence of training examples
Returns
-------
labels : sequence of labels
'''
return map(self.apply, fs)
class base_adaptor(object):
def __init__(self, base):
self.base = base
def set_option(self, k, v):
self.base.set_option(k, v)
milk-0.5/milk/supervised/defaultlearner.py 0000664 0001750 0001750 00000013450 12026076213 023050 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2012, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division
import numpy as np
from milk.supervised.base import supervised_model
__all__ = [
'defaultlearner',
'svm_simple',
'feature_selection_simple',
]
class default_learner(object):
def __init__(self, preproc, classifier, multi_adapter):
self.preproc = preproc
self.classifier = classifier
self.multi_adapter = preproc
def train(self, features, labels):
model = self.preproc.train(features, labels)
nfeatures = model.apply_many(features)
classifier = self.classifier
if len(set(labels)) > 2:
classifier = self.multi_adapter(classifier)
cmodel = classifier.train(nfeatures, labels)
return default_model(classifier, cmodel)
class default_model(supervised_model):
def __init__(self, preproc, classify):
self.preproc = preproc
self.classify = classify
def apply(self, features):
features = self.preproc.apply(features)
return self.classify.apply(features)
def defaultlearner(mode='medium', multi_strategy='1-vs-1', expanded=False):
'''
learner = defaultlearner(mode='medium')
Return the default classifier learner
This is an SVM based classifier using the 1-vs-1 technique for multi-class
problems (by default, see the ``multi_strategy`` parameter). The features
will be first cleaned up (normalised to [-1, +1]) and go through SDA
feature selection.
Parameters
-----------
mode : string, optional
One of ('fast','medium','slow', 'really-slow'). This defines the speed
accuracy trade-off. It essentially defines how large the SVM parameter
range is.
multi_strategy : str, optional
One of ('1-vs-1', '1-vs-rest', 'ecoc'). This defines the strategy used
to convert the base binary classifier to a multi-class classifier.
expanded : boolean, optional
If true, then instead of a single learner, it returns a list of
possible learners.
Returns
-------
learner : classifier learner object or list
If `expanded`, then it returns a list
See Also
--------
feature_selection_simple : Just perform the feature selection
svm_simple : Perform classification
'''
# These cannot be imported at module scope!
# The reason is that they introduce a dependency loop:
# gridsearch depends on nfoldcrossvalidation
# nfoldcrossvalidation depends on defaultlearner
# which cannot depend on gridsearch
#
# Importing at function level keeps all these issues at bay
#
from .classifier import ctransforms
from .gridsearch import gridsearch
from . import svm
from .normalise import chkfinite, interval_normalise
from .featureselection import sda_filter, featureselector, linear_independent_features
from .multi import one_against_one, one_against_rest, ecoc_learner
assert mode in ('really-slow', 'slow', 'medium', 'fast'), \
"milk.supervised.defaultlearner: mode must be one of 'fast','slow','medium'."
if multi_strategy == '1-vs-1':
multi_adaptor = one_against_one
elif multi_strategy == '1-vs-rest':
multi_adaptor = one_against_rest
elif multi_strategy == 'ecoc':
multi_adaptor = ecoc_learner
else:
raise ValueError('milk.supervised.defaultlearner: Unknown value for multi_strategy: %s' % multi_strategy)
if mode == 'fast':
c_range = np.arange(-2,4)
sigma_range = np.arange(-2,3)
elif mode == 'medium':
c_range = np.arange(-2,4)
sigma_range = np.arange(-4,4)
elif mode == 'really-slow':
c_range = np.arange(-4,10)
sigma_range = np.arange(-7,7)
else: # mode == 'slow'
c_range = np.arange(-9,5)
sigma_range = np.arange(-7,4)
kernels = [svm.rbf_kernel(2.**i) for i in sigma_range]
Cs = 2.**c_range
if expanded:
return [ctransforms(feature_selection_simple(),
multi_adaptor(svm.svm_to_binary(svm.svm_raw(C=C, kernel=kernel))))
for C in Cs for kernel in kernels]
return ctransforms(feature_selection_simple(),
gridsearch(multi_adaptor(svm.svm_to_binary(svm.svm_raw())),
params={ 'C': Cs, 'kernel': kernels, }))
def feature_selection_simple():
'''
selector = feature_selection_simple()
Standard feature normalisation and selection
This fills in NaNs and Infs (to 0 and large numbers), normalises features
to [-1, +1] and uses SDA for feature selection.
Returns
-------
selector : supervised learner
See Also
--------
defaultlearner : perform feature selection *and* classification
'''
from .classifier import ctransforms
from .normalise import chkfinite, interval_normalise
from .featureselection import sda_filter, featureselector, linear_independent_features
return ctransforms(
chkfinite(),
interval_normalise(),
featureselector(linear_independent_features),
sda_filter(),
)
def svm_simple(C, kernel):
'''
learner = svm_simple(C, kernel)
Returns a one-against-one SVM based classifier with `C` and `kernel`
Parameters
----------
C : double
C parameter
kernel : kernel
Kernel to use
Returns
-------
learner : supervised learner
See Also
--------
feature_selection_simple : Perform feature selection
defaultlearner : feature selection and gridsearch for SVM parameters
'''
from . import svm
from .multi import one_against_one
return one_against_one(svm.svm_to_binary(svm.svm_raw(C=C, kernel=kernel)))
milk-0.5/milk/supervised/__init__.py 0000664 0001750 0001750 00000003556 12030633375 021623 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2012, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# License: MIT. See COPYING.MIT file in the milk distribution
'''
milk.supervised
This hold the supervised classification modules:
Submodules
----------
- defaultclassifier: contains a default "good enough" classifier
- svm: related to SVMs
- grouped: contains objects to transform single object classifiers into group classifiers
by voting
- multi: transforms binary classifiers into multi-class classifiers (1-vs-1 or 1-vs-rest)
- featureselection: feature selection
- knn: k-nearest neighbours
- tree: decision tree classifiers
Classifiers
-----------
All classifiers have a `train` function which takes 2 arguments:
- features : sequence of features
- labels : sequence of labels
They return a `model` object, which has an `apply` function which takes a
single input and returns its label.
Note that there are always two objects: the learned and the model and they are
independent. Every time you call learner.train() you get a new model.
Both classifiers and models are pickle()able.
Example
-------
::
features = np.random.randn(100,20)
features[:50] *= 2
labels = np.repeat((0,1), 50)
classifier = milk.defaultclassifier()
model = classifier.train(features, labels)
new_label = model.apply(np.random.randn(100))
new_label2 = model.apply(np.random.randn(100)*2)
'''
from .defaultclassifier import defaultclassifier, svm_simple
from .classifier import normaliselabels
from .gridsearch import gridsearch
from .tree import tree_learner
from .lasso import lasso, lasso_learner, lasso_model_walk, lasso_walk
__all__ = [
'normaliselabels',
'defaultclassifier',
'svm_simple',
'gridsearch',
'lasso',
'lasso_learner',
'lasso_model_walk',
'lasso_walk',
'tree_learner',
]
milk-0.5/milk/supervised/logistic.py 0000664 0001750 0001750 00000007127 11723217323 021676 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2011, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division
import numpy as np
from .normalise import normaliselabels
from .base import supervised_model
__all__ = [
'logistic_learner',
]
@np.vectorize
def _sigmoidal(z):
if (z > 300): return 1.
if z < -300: return 0.
return 1./(1+np.exp(-z))
class logistic_model(supervised_model):
def __init__(self, bs):
self.bs = bs
def apply(self, fs):
return _sigmoidal(self.bs[0] + np.dot(fs, self.bs[1:]))
class logistic_learner(object):
'''
learner = logistic_learner(alpha=0.0)
Logistic regression learner
There are two implementations:
1. One which depends on ``scipy.optimize``. This is the default and is
extremely fast.
2. If ``import scipy`` fails, then we fall back to a Python only
gradient-descent. This gives good results, but is many times slower.
Properties
----------
alpha : real, optional
penalty for L2-normalisation. Default is zero, for no penalty.
'''
def __init__(self, alpha=0.0):
self.alpha = alpha
def train(self, features, labels, normalisedlabels=False, names=None, **kwargs):
def error(bs):
response = bs[0] + np.dot(features, bs[1:])
response = _sigmoidal(response)
diff = response - labels
log_like = np.dot(diff, diff)
L2_penalty = self.alpha * np.dot(bs, bs)
return log_like + L2_penalty
def error_prime(bs):
fB = np.dot(features, bs[1:])
response = _sigmoidal(bs[0] + fB)
sprime = response * (1-response)
ds = (response - labels) * sprime
b0p = np.sum(ds)
b1p = np.dot(features.T, ds)
bp = np.concatenate( ([b0p], b1p) )
return 2.*(bp + self.alpha*bs)
features = np.asanyarray(features)
if not normalisedlabels:
labels, _ = normaliselabels(labels)
N,f = features.shape
bs = np.zeros(f+1)
try:
from scipy import optimize
# Some testing revealed that this was a good combination
# call fmin_cg twice first and then fmin
# I do not understand why 100%, but there it is
bs = optimize.fmin_cg(error, bs, error_prime, disp=False)
bs = optimize.fmin_cg(error, bs, error_prime, disp=False)
bs = optimize.fmin(error, bs, disp=False)
except ImportError:
import warnings
warnings.warn('''\
milk.supervised.logistic.train: Could not import scipy.optimize.
Fall back to very simple gradient descent (which is slow).''')
bs = np.zeros(f+1)
cur = 1.e-6
ebs = error(bs)
for i in xrange(1000000):
dir = error_prime(bs)
step = (lambda e : bs - e *dir)
enbs = ebs + 1
while enbs > ebs:
cur /= 2.
if cur == 0.:
break
nbs = step(cur)
enbs = error(nbs)
while cur < 10.:
cur *= 2
nnbs = step(cur)
ennbs = error(nnbs)
if ennbs < enbs:
nbs = nnbs
enbs = ennbs
else:
break
bs = nbs
ebs = enbs
return logistic_model(bs)
milk-0.5/milk/supervised/grouped.py 0000664 0001750 0001750 00000007440 11723217323 021524 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2010-2011, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# License: MIT. See COPYING.MIT file in the milk distribution
# -*- coding: utf-8 -*-
from __future__ import division
import numpy as np
from collections import defaultdict
from .classifier import normaliselabels
from .base import base_adaptor, supervised_model
__all__ = [
'voting_learner',
'mean_learner',
'remove_outliers',
'filter_outliers',
]
def _concatenate_features_labels(gfeatures, glabels):
if type(gfeatures) == np.ndarray and gfeatures.dtype == object:
gfeatures = list(gfeatures)
features = np.concatenate(gfeatures)
labels = []
for feats,label in zip(gfeatures, glabels):
labels.extend( [label] * len(feats) )
return features, labels
class voting_learner(base_adaptor):
'''
Implements a voting scheme for multiple sub-examples per example.
classifier = voting_learner(base)
base should be a binary classifier
Example
-------
::
voterlearn = voting_learner(milk.supervised.simple_svm())
voter = voterlearn.train(training_groups, labeled_groups)
res = voter.apply([ [f0, f1, f3] ])
'''
def train(self, gfeatures, glabels, normalisedlabels=False):
features, labels = _concatenate_features_labels(gfeatures, glabels)
return voting_model(self.base.train(features, labels))
voting_classifier = voting_learner
class voting_model(supervised_model):
def __init__(self, base):
self.base = base
def apply(self, gfeatures):
votes = defaultdict(int)
for feats in gfeatures:
votes[self.base.apply(feats)] += 1
best = None
most_votes = 0
for k,v in votes.iteritems():
if v > most_votes:
best = k
most_votes = v
return best
class mean_learner(base_adaptor):
'''
Implements a mean scheme for multiple sub-examples per example.
classifier = mean_learner(base)
`base` should be a classifier that returns a numeric confidence value
`classifier` will return the **mean**
Example
-------
::
meanlearner = mean_learner(milk.supervised.raw_svm())
model = meanlearner.train(training_groups, labeled_groups)
res = model.apply([ [f0, f1, f3] ])
'''
def train(self, gfeatures, glabels, normalisedlabels=False):
features, labels = _concatenate_features_labels(gfeatures, glabels)
return mean_model(self.base.train(features, labels))
mean_classifier = mean_learner
class mean_model(supervised_model):
def __init__(self, base):
self.base = base
def apply(self, gfeatures):
return np.mean([self.base.apply(feats) for feats in gfeatures])
def remove_outliers(features, limit, min_size):
'''
features = remove_outliers(features, limit, min_size)
'''
nsize = int(limit * len(features))
if nsize < min_size:
return features
normed = features - features.mean(0)
std = normed.std(0)
std[std == 0] = 1
normed /= std
f2_sum1 = (normed**2).mean(1)
values = f2_sum1.copy()
values.sort()
top = values[nsize]
selected = f2_sum1 < top
return features[selected]
class filter_outliers_model(supervised_model):
def __init__(self, limit, min_size):
self.limit = limit
self.min_size = min_size
def apply(self, features):
return remove_outliers(features, self.limit, self.min_size)
class filter_outliers(object):
def __init__(self, limit=.9, min_size=3):
self.limit = limit
self.min_size = min_size
def train(self, features, labels, normalisedlabels=False):
return filter_outliers_model(self.limit, self.min_size)
milk-0.5/milk/supervised/weighted_voting_adaboost.py 0000664 0001750 0001750 00000005636 11723217323 025126 0 ustar luispedro luispedro 0000000 0000000 from math import exp, log
from operator import itemgetter
'''
AdaBoost implementation with weighted voting as a decision procedure
'''
class weighted_voting_adaboost(object):
# initializes with already built classifiers and corresponding
def __init__(self, in_classifiers, in_coefficients):
self.classifiers = in_classifiers
self.coefficients = in_coefficients
# decision by weighted voting
def apply(self, in_features):
# a "class number" => "votes value" mapping
answers = {}
for classifier, coefficient in zip(self.classifiers, self.coefficients):
answer = classifier.apply(in_features)
if answer in answers:
answers[answer] += coefficient
else:
answers[answer] = coefficient
# dict maximum by value
result = max(answers.iteritems(), key=itemgetter(1))
return result[0]
class weighted_voting_ada_learner(object):
def __init__(self, in_composition_size, in_learner):
self.learner = in_learner
self.composition_size = in_composition_size
def reset(self, in_features):
self.classifiers = []
# linear coefficients for the classifiers in composition
self.coefficients = []
self.weights = [1. / float(len(in_features))] * len(in_features)
def train(self, in_features, in_labels):
self.reset(in_features)
for iteration in xrange(self.composition_size):
self.classifiers.append(self.learner.train(in_features, in_labels, weights=self.weights))
# new classifier initially gets weight 1
self.coefficients.append(1)
answers = []
for obj in in_features:
answers.append(self.classifiers[-1].apply(obj))
err = self.compute_weighted_error(in_labels, answers)
if abs(err) < 1e-6:
return weighted_voting_adaboost(self.classifiers, self.coefficients)
alpha = 0.5 * log((1.0 - err) / err)
# updating the coefficient of the last added classifier
self.coefficients[-1] = alpha
self.update_weights(in_labels, answers, alpha)
self.normalize_weights()
return weighted_voting_adaboost(self.classifiers, self.coefficients)
def compute_weighted_error(self, in_labels, in_answers):
error = 0.
w_sum = sum(self.weights)
for ind in xrange(len(in_labels)):
error += (in_answers[ind] != in_labels[ind]) * self.weights[ind] / w_sum
return error
def update_weights(self, in_labels, in_answers, in_alpha):
for ind in xrange(len(in_labels)):
self.weights[ind] *= exp(in_alpha * (in_answers[ind] != in_labels[ind]))
def normalize_weights(self):
w_sum = sum(self.weights)
for ind in xrange(len(self.weights)):
self.weights[ind] /= w_sum
milk-0.5/milk/supervised/knn.py 0000664 0001750 0001750 00000003124 11723217323 020640 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2011, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division
from collections import defaultdict
import numpy as np
from .base import supervised_model
__all__ = [
'kNN',
]
class kNN(object):
'''
k-Nearest Neighbour Classifier
Naive implementation of a k-nearest neighbour classifier.
C = kNN(k)
Attributes:
-----------
k : integer
number of neighbours to consider
'''
def __init__(self, k=1):
self.k = k
def train(self, features, labels, normalisedlabels=False, copy_features=False):
features = np.asanyarray(features)
labels = np.asanyarray(labels)
if copy_features:
features = features.copy()
labels = labels.copy()
features2 = np.sum(features**2, axis=1)
return kNN_model(self.k, features, features2, labels)
class kNN_model(supervised_model):
def __init__(self, k, features, features2, labels):
self.k = k
self.features = features
self.f2 = features2
self.labels = labels
def apply(self, features):
features = np.asanyarray(features)
diff2 = np.dot(self.features, (-2.)*features)
diff2 += self.f2
neighbours = diff2.argsort()[:self.k]
labels = self.labels[neighbours]
votes = defaultdict(int)
for L in labels:
votes[L] += 1
v,L = max( (v,L) for L,v in votes.items() )
return L
milk-0.5/milk/supervised/tree.py 0000664 0001750 0001750 00000017445 11723217323 021024 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2011, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# License: MIT. See COPYING.MIT file in the milk distribution
'''
================
Tree Classifier
================
Decision tree based classifier
------------------------------
'''
from __future__ import division
import numpy as np
from .classifier import normaliselabels
from .base import supervised_model
__all__ = [
'tree_learner',
'stump_learner',
]
class Leaf(object):
'''
v : value
w : weight
'''
def __init__(self, v, w):
self.v = v
self.w = w
def __repr__(self):
return 'Leaf(%s,%s)' % (self.v, self.w)
class Node(object): # This could be replaced by a namedtuple
def __init__(self, featid, featval, left, right):
self.featid = featid
self.featval = featval
self.left = left
self.right = right
def _split(features, labels, weights, criterion, subsample, R):
N,f = features.shape
if subsample is not None:
samples = np.array(R.sample(xrange(features.shape[1]), subsample))
features = features[:, samples]
f = subsample
best = None
best_val = float('-Inf')
for i in xrange(f):
domain_i = sorted(set(features[:,i]))
for d in domain_i[1:]:
cur_split = (features[:,i] < d)
if weights is not None:
value = criterion(labels[cur_split], labels[~cur_split], weights[cur_split], weights[~cur_split])
else:
value = criterion(labels[cur_split], labels[~cur_split])
if value > best_val:
best_val = value
if subsample is not None:
ti = samples[i]
else:
ti = i
best = ti,d
return best
from ._tree import set_entropy
from ._tree import information_gain as _information_gain
def information_gain(labels0, labels1, include_entropy=False):
'''
ig = information_gain(labels0, labels1, include_entropy=False)
Information Gain
See http://en.wikipedia.org/wiki/Information_gain_in_decision_trees
The function calculated here does not include the original entropy unless
you explicitly ask for it (by passing include_entropy=True)
'''
if include_entropy:
return set_entropy(np.concatenate( (labels0, labels1) )) + \
_information_gain(labels0, labels1)
return _information_gain(labels0, labels1)
def z1_loss(labels0, labels1, weights0=None, weights1=None):
'''
z = z1_loss(labels0, labels1)
z = z1_loss(labels0, labels1, weights0, weights1)
zero-one loss
'''
def _acc(labels, weights):
c = (labels.mean() > .5)
if weights is not None:
return np.dot((labels != c), weights)
return np.sum(labels != c)
return _acc(labels0, weights0) + _acc(labels1, weights1)
def neg_z1_loss(labels0, labels1, weights0=None, weights1=None):
'''
z = neg_z1_loss(labels0, labels1)
z = neg_z1_loss(labels0, labels1, weights0, weights1)
zero-one loss, with the sign reversed so it can be *maximised*.
'''
return -z1_loss(labels0,labels1,weights0,weights1)
def build_tree(features, labels, criterion, min_split=4, subsample=None, R=None, weights=None):
'''
tree = build_tree(features, labels, criterion, min_split=4, subsample=None, R=None, weights={all 1s})
Parameters
----------
features : sequence
features to use
labels : sequence
labels
criterion : function {labels} x {labels} -> float
function to measure goodness of split
min_split : integer
minimum size to split on
subsample : integer, optional
if given, then, at each step, choose
R : source of randomness, optional
See `milk.util.get_pyrandom`
weights : sequence, optional
weight of instance (default: all the same)
Returns
-------
tree : Tree
'''
assert len(features) == len(labels), 'build_tree: Nr of labels does not match nr of features'
features = np.asanyarray(features)
labels = np.asanyarray(labels, dtype=np.int)
if subsample is not None:
if subsample <= 0:
raise ValueError('milk.supervised.tree.build_tree: `subsample` must be > 0.\nDid you mean to use None to signal no subsample?')
from ..utils import get_pyrandom
R = get_pyrandom(R)
def recursive(features, labels):
N = float(len(labels))
if N < min_split:
return Leaf(labels.sum()/N, N)
S = _split(features, labels, weights, criterion, subsample, R)
if S is None:
return Leaf(labels.sum()/N, N)
i,thresh = S
split = features[:,i] < thresh
return Node(featid=i,
featval=thresh,
left =recursive(features[ split], labels[ split]),
right=recursive(features[~split], labels[~split]))
return recursive(features, labels)
def apply_tree(tree, features):
'''
conf = apply_tree(tree, features)
Applies the decision tree to a set of features.
'''
if type(tree) is Leaf:
return tree.v
if features[tree.featid] < tree.featval:
return apply_tree(tree.left, features)
return apply_tree(tree.right, features)
class tree_learner(object):
'''
tree = tree_learner()
model = tree.train(features, labels)
model2 = tree.train(features, labels, weights=weights)
predicted = model.apply(testfeatures)
A decision tree classifier (currently, implements the greedy ID3
algorithm without any pruning).
Attributes
----------
criterion : function, optional
criterion to use for tree construction,
this should be a function that receives a set of labels
(default: information_gain).
min_split : integer, optional
minimum size to split on (default: 4).
'''
def __init__(self, criterion=information_gain, min_split=4, return_label=True, subsample=None, R=None):
self.criterion = criterion
self.min_split = min_split
self.return_label = return_label
self.subsample = subsample
self.R = R
def train(self, features, labels, normalisedlabels=False, weights=None):
if not normalisedlabels:
labels,names = normaliselabels(labels)
tree = build_tree(features, labels, self.criterion, self.min_split, self.subsample, self.R, weights)
return tree_model(tree, self.return_label)
tree_classifier = tree_learner
class tree_model(supervised_model):
'''
tree model
'''
def __init__(self, tree, return_label):
self.tree = tree
self.return_label = return_label
def apply(self,feats):
value = apply_tree(self.tree, feats)
if self.return_label:
return value > .5
return value
class stump_model(supervised_model):
def __init__(self, idx, cutoff, names):
self.names = names
self.idx = idx
self.cutoff = cutoff
def apply(self, f):
value = f[self.idx] > self.cutoff
if self.names is not None:
return self.names[value]
return value
def __repr__(self):
return '' % (self.idx, self.cutoff)
class stump_learner(object):
def __init__(self):
pass
def train(self, features, labels, normalisedlabels=False, weights=None, **kwargs):
if not normalisedlabels:
labels,names = normaliselabels(labels)
else:
names = kwargs.get('names')
split = _split(features, labels, weights, neg_z1_loss, subsample=None, R=None)
if split is None:
raise ValueError('milk.supervised.stump_learner: Unable to find split (all features are the same)')
idx,cutoff = split
return stump_model(idx, cutoff, names)
milk-0.5/milk/supervised/precluster.py 0000664 0001750 0001750 00000012515 12043766022 022247 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2011, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division
import numpy as np
from milk.unsupervised.kmeans import select_best_kmeans, assign_centroids
from .base import supervised_model, base_adaptor
import multiprocessing
from milk.utils import parallel
from milk import defaultlearner
class precluster_model(supervised_model):
def __init__(self, centroids, base):
self.centroids = centroids
self.base = base
self.normalise = True
def apply(self, features):
histogram = assign_centroids(features, self.centroids, histogram=True, normalise=self.normalise)
return self.base.apply(histogram)
class precluster_learner(object):
'''
This learns a classifier by clustering the input features
'''
def __init__(self, ks, base=None, R=None):
if base is None:
base = defaultlearner()
self.ks = ks
self.R = R
self.base = base
self.normalise = True
def set_option(self, k, v):
if k in ('R', 'ks'):
setattr(self, k, v)
else:
self.base.set_option(k,v)
def train(self, features, labels, **kwargs):
allfeatures = np.vstack(features)
assignments, centroids = select_best_kmeans(allfeatures, self.ks, repeats=1, method="AIC", R=self.R)
histograms = [assign_centroids(f, centroids, histogram=True, normalise=self.normalise) for f in features]
base_model = self.base.train(histograms, labels, **kwargs)
return precluster_model(centroids, base_model)
class codebook_model(supervised_model):
def __init__(self, centroids, base, normalise):
self.centroids = centroids
self.base = base
self.normalise = normalise
def apply(self, features):
from milk.unsupervised.kmeans import assign_centroids
f0,f1 = features
features = assign_centroids(f0, self.centroids, histogram=True, normalise=self.normalise)
if f1 is not None and len(f1):
features = np.concatenate((features, f1))
return self.base.apply(features)
class codebook_learner(base_adaptor):
def set_option(self, k, v):
if k != 'codebook':
raise KeyError('milk.precluster.codebook_learner: unknown option `%s`' % k)
self.codebook = v
def train(self, features, labels, **kwargs):
from milk.unsupervised.kmeans import assign_centroids
tfeatures = np.array([ assign_centroids(f, self.codebook, histogram=True, normalise=self.normalise)
for f,_ in features])
tfeatures = np.hstack((tfeatures, np.array([f for _,f in features])))
base_model = self.base.train(tfeatures, labels, **kwargs)
return codebook_model(self.codebook, base_model, self.normalise)
class kmeans_cluster(multiprocessing.Process):
def __init__(self, features, inq, outq):
self.features = features
self.inq = inq
self.outq = outq
def execute(self):
import milk
while True:
k,ri = self.inq.get()
if k == 'shutdown':
return
_,centroids = milk.kmeans(self.features, k=k, R=(k*1024+ri))
self.outq.put(centroids)
def run(self):
try:
self.execute()
except Exception, e:
errstr = r'''\
Error in milk.supervised.precluster.learn_codebook internal
Exception was: %s
Original Traceback:
%s
(Since this was run on a different process, this is not a real stack trace).
''' % (e, traceback.format_exc())
self.outq.put( ('error', errstr) )
class select_precluster(object):
def __init__(self, ks, base, normalise=True, rmax=16):
self.base = base
self.ks = ks
self.rmax = rmax
self.sample = 16
self.nfolds = 5
self.normalise = normalise
def train(self, features, labels, **kwargs):
from milk.supervised.gridsearch import gridminimise
c_features = np.concatenate([f for f,_ in features if len(f)])
c_features = c_features[::self.sample]
nprocs = parallel.get_procs(use_current=True)
tow = multiprocessing.Queue()
fromw = multiprocessing.Queue()
for k in self.ks:
for ri in xrange(self.rmax):
tow.put((k,ri))
for i in xrange(nprocs):
tow.put(('shutdown',None))
workers = [kmeans_cluster(c_features, tow, fromw) for i in xrange(nprocs)]
for w in workers:
if nprocs > 1:
w.start()
else:
w.execute()
try:
codebooks = [fromw.get() for i in xrange(len(self.ks)*self.rmax)]
finally:
tow.close()
tow.join_thread()
if nprocs > 1:
for w in workers:
w.join()
parallel.release_procs(len(workers), count_current=True)
base = codebook_learner(self.base)
base.normalise = self.normalise
if len(codebooks) > 1:
(best,) = gridminimise(base, features, labels, { 'codebook' : codebooks }, nfolds=self.nfolds)
_,codebook = best
else:
(codebook,) = codebooks
base.codebook = codebook
return base.train(features, labels)
milk-0.5/milk/supervised/svm.py 0000664 0001750 0001750 00000036645 12043771003 020671 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2012, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division
from .classifier import normaliselabels, ctransforms_model
from .base import supervised_model
from collections import deque
import numpy
import numpy as np
import random
from . import _svm
__all__ = [
'rbf_kernel',
'polynomial_kernel',
'precomputed_kernel',
'dot_kernel',
'svm_raw',
'svm_binary',
'svm_to_binary',
'svm_sigmoidal_correction',
'sigma_value_fisher',
'fisher_tuned_rbf_svm',
]
def _svm_apply(SVM, q):
'''
f_i = _svm_apply(SVM, q)
@internal: This is mostly used for testing
'''
X,Y,Alphas,b,C,kernel=SVM
N = len(X)
s = 0.0
for i in xrange(N):
s += Alphas[i] * Y[i] * kernel(q, X[i])
return s - b
def svm_learn_smo(X,Y,kernel,C,eps=1e-4,tol=1e-2,cache_size=(1<<20)):
'''
Learn a svm classifier
X: data
Y: labels in SVM format (ie Y[i] in (1,-1))
This is a very raw interface. In general, you should use a class
like svm_classifier.
Implements the Sequential Minimum Optimisation Algorithm from Platt's
"Fast training of support vector machines using sequential minimal optimization"
in Advances in kernel methods: support vector learning
Pages: 185 - 208
Year of Publication: 1999
ISBN:0-262-19416-3
'''
assert numpy.all(numpy.abs(Y) == 1)
assert len(X) == len(Y)
N = len(Y)
Y = Y.astype(numpy.int32)
params = numpy.array([0,C,1e-3,1e-5],numpy.double)
Alphas0 = numpy.zeros(N, numpy.double)
_svm.eval_SMO(X,Y,Alphas0,params,kernel,cache_size)
return Alphas0, params[0]
def svm_learn_libsvm(features, labels, kernel, C, eps=1e-4, tol=1e-2, cache_size=(1<<20), alphas=None):
'''
Learn a svm classifier using LIBSVM optimiser
This is a very raw interface. In general, you should use a class
like svm_classifier.
This uses the LIBSVM optimisation algorithm
Parameters
----------
X : ndarray
data
Y : ndarray
labels in SVM format (ie Y[i] in (1,-1))
kernel : kernel
C : float
eps : float, optional
tol : float, optional
cache_size : int, optional
alphas : ndarray, optional
Returns
-------
alphas : ndarray
b : float
'''
if not np.all(np.abs(labels) == 1):
raise ValueError('milk.supervised.svm.svm_learn_libsvm: Y[i] != (-1,+1)')
assert len(features) == len(labels)
n = len(labels)
labels = labels.astype(np.int32)
p = -np.ones(n, np.double)
params = np.array([0,C,eps,tol], dtype=np.double)
if alphas is None:
alphas = np.zeros(n, np.double)
elif alphas.dtype != np.double or len(alphas) != n:
raise ValueError('milk.supervised.svm_learn_libsvm: alphas is in wrong format')
_svm.eval_LIBSVM(features, labels, alphas, p, params, kernel, cache_size)
return alphas, params[0]
class preprocessed_rbf_kernel(object):
def __init__(self, X, sigma, beta):
self.X = X
self.Xsum = (X**2).sum(1)
self.sigma = sigma
self.beta = beta
def call_many(self, qs):
from milk.unsupervised import pdist
dists = pdist(self.X, qs, 'euclidean2')
dists /= -self.sigma
np.exp(dists, dists)
dists *= self.beta
return dists.T
def __call__(self, q):
minus_d2_sigma = np.dot(self.X,q)
minus_d2_sigma *= 2.
minus_d2_sigma -= self.Xsum
minus_d2_sigma -= np.dot(q,q)
minus_d2_sigma /= self.sigma
return self.beta * np.exp(minus_d2_sigma)
class rbf_kernel(object):
'''
kernel = rbf_kernel(sigma,beta=1)
Radial Basis Function kernel
Returns a kernel (ie, a function that implements)
beta * exp( - ||x1 - x2|| / sigma)
'''
def __init__(self, sigma, beta=1):
self.sigma = sigma
self.beta = beta
self.kernel_nr_ = 0
self.kernel_arg_ = float(sigma)
def __call__(self, x1, x2):
d2 = x1 - x2
d2 **= 2
d2 = d2.sum()
res = self.beta*np.exp(-d2/self.sigma)
return res
def preprocess(self, X):
return preprocessed_rbf_kernel(X, self.sigma, self.beta)
class polynomial_kernel(object):
'''
kernel = polynomial_kernel(d,c=1)
returns a kernel (ie, a function) that implements:
(+c)**d
'''
def __init__(self, d, c=1):
self.d = d
self.c = c
def __call__(self,x1,x2):
return (np.dot(x1,x2)+self.c)**self.d
class precomputed_kernel(object):
'''
kernel = precomputed_kernel(kmatrix)
A "fake" kernel which is precomputed.
'''
def __init__(self, kmatrix, copy=False):
kmatrix = np.ascontiguousarray(kmatrix, np.double, copy=copy)
self.kernel_nr_ = 1
self.kernel_arg_ = 0.
def __call__(self, x0, x1):
return kmatrix[x0,x1]
class _call_kernel(object):
def __init__(self, k, svs):
self.svs = svs
self.kernel = k
def __call__(self, q):
return np.array([self.kernel(s, q) for s in self.svs])
class preprocessed_dot_kernel(object):
def __init__(self, svs):
self.svs = svs
def __call__(self, x1):
return np.dot(self.svs, x1)
class dot_kernel(object):
def __init__(self):
self.kernel_nr_ = 2
self.kernel_arg_ = 0.
def __call__(self, x0, x1):
return np.dot(x0, x1)
def preprocess(self, svs):
return preprocessed_dot_kernel(svs)
class svm_raw_model(supervised_model):
def __init__(self, svs, Yw, b, kernel):
self.svs = svs
self.Yw = Yw
self.b = b
self.kernel = kernel
try:
self.kernelfunction = self.kernel.preprocess(self.svs)
except AttributeError:
self.kernelfunction = _call_kernel(self.kernel, self.svs)
def apply_many(self, qs):
try:
qs = self.kernelfunction.call_many(qs)
except AttributeError:
qs = np.array(map(self.kernelfunction, qs))
return np.dot(qs, self.Yw) - self.b
def apply(self, q):
Q = self.kernelfunction(q)
return np.dot(Q, self.Yw) - self.b
class svm_raw(object):
'''
svm_raw: classifier
classifier = svm_raw(kernel, C, eps=1e-3, tol=1e-8)
Parameters
----------
kernel : callable
the kernel to use. This should be a function that takes two data
arguments see rbf_kernel and polynomial_kernel.
C : float
the C parameter
eps : float, optional
the precision to which to solve the problem (default 1e-3)
tol : float, optional
(|x| < tol) is considered zero
'''
def __init__(self, kernel=None, C=1., eps=1e-3, tol=1e-8):
self.C = C
self.kernel = kernel
self.eps = eps
self.tol = tol
self.algorithm = 'libsvm'
def train(self, features, labels, normalisedlabels=False, **kwargs):
assert self.kernel is not None, 'milk.supervised.svm_raw.train: kernel not set!'
assert self.algorithm in ('libsvm','smo'), 'milk.supervised.svm_raw: unknown algorithm (%s)' % self.algorithm
assert not (np.isinf(self.C) or np.isnan(self.C)), 'milk.supervised.svm_raw: setting C to NaN or Inf causes problems.'
features = np.asanyarray(features)
if normalisedlabels:
Y = labels.copy()
else:
Y,_ = normaliselabels(labels)
assert Y.max() == 1, 'milk.supervised.svm_raw can only handle binary problems'
Y *= 2
Y -= 1
kernel = self.kernel
try:
kernel = (self.kernel.kernel_nr_, self.kernel.kernel_arg_)
features = np.ascontiguousarray(features, np.double)
except AttributeError:
pass
if self.algorithm == 'smo':
alphas,b = svm_learn_smo(features,Y,kernel,self.C,self.eps,self.tol)
else:
alphas,b = svm_learn_libsvm(features,Y,kernel,self.C,self.eps,self.tol)
svsi = (alphas != 0)
svs = features[svsi]
w = alphas[svsi]
Y = Y[svsi]
Yw = w * Y
return svm_raw_model(svs, Yw, b, self.kernel)
def get_params(self):
return self.C, self.eps,self.tol
def set_params(self,params):
self.C,self.eps,self.tol = params
def set_option(self, optname, value):
setattr(self, optname, value)
def learn_sigmoid_constants(F,Y,
max_iters=None,
min_step=1e-10,
sigma=1e-12,
eps=1e-5):
'''
A,B = learn_sigmoid_constants(F,Y)
This is a very low-level interface look into the svm_classifier class.
Parameters
----------
F : Values of the function F
Y : Labels (in boolean format, ie, in (0,1))
Other Parameters
----------------
max_iters : Maximum nr. of iterations
min_step : Minimum step
sigma : sigma
eps : A small number
Reference for Implementation
----------------------------
Implements the algorithm from "A Note on Platt's Probabilistic Outputs for
Support Vector Machines" by Lin, Lin, and Weng.
Machine Learning, Vol. 68, No. 3. (23 October 2007), pp. 267-276
'''
# Below we use safe constructs to avoid using the overflown values, but we
# must compute them because of the way numpy works.
errorstate = np.seterr(over='ignore')
# the deci[i] array is called F in this code
F = np.asanyarray(F)
Y = np.asanyarray(Y)
assert len(F) == len(Y)
assert numpy.all( (Y == 1) | (Y == 0) )
if max_iters is None:
max_iters = 1000
prior1 = Y.sum()
prior0 = len(F)-prior1
small_nr = 1e-4
hi_t = (prior1+1.)/(prior1+2.)
lo_t = 1./(prior0+2.)
T = Y*hi_t + (1-Y)*lo_t
A = 0.
B = np.log( (prior0+1.)/(prior1+1.) )
def target(A,B):
fApB = F*A + B
lef = np.log1p(np.exp(fApB))
lemf = np.log1p(np.exp(-fApB))
fvals = np.choose(fApB >= 0, ( T*fApB + lemf, (T-1.)*fApB + lef))
return np.sum(fvals)
fval = target(A,B)
for iter in xrange(max_iters):
fApB = F*A + B
ef = np.exp(fApB)
emf = np.exp(-fApB)
p = np.choose(fApB >= 0, ( emf/(1.+emf), 1./(1.+ef) ))
q = np.choose(fApB >= 0, ( 1/(1.+emf), ef/(1.+ef) ))
d2 = p * q
h11 = np.dot(F*F,d2) + sigma
h22 = np.sum(d2) + sigma
h21 = np.dot(F,d2)
d1 = T - p
g1 = np.dot(F,d1)
g2 = np.sum(d1)
if abs(g1) < eps and abs(g2) < eps: # Stopping criteria
break
det = h11*h22 - h21*h21
dA = - (h22*g1 - h21*g2)/det
dB = - (h21*g1 + h11*g2)/det
gd = g1*dA + g2*dB
stepsize = 1.
while stepsize >= min_step:
newA = A + stepsize*dA
newB = B + stepsize*dB
newf = target(newA,newB)
if newf < fval+eps*stepsize*gd:
A = newA
B = newB
fval = newf
break
stepsize /= 2
else:
print 'Line search fails'
break
np.seterr(**errorstate)
return A,B
class svm_binary_model(supervised_model):
def __init__(self, classes):
self.classes = classes
def apply(self,f):
return self.classes[f >= 0.]
class svm_binary(object):
'''
classifier = svm_binary()
model = classifier.train(features, labels)
assert model.apply(f) in labels
'''
def train(self, features, labels, normalisedlabels=False, **kwargs):
if normalisedlabels:
return svm_binary_model( (0,1) )
assert len(labels) >= 2, 'Cannot train from a single example'
names = sorted(set(labels))
assert len(names) == 2, 'milk.supervised.svm.svm_binary.train: Can only handle two class problems'
return svm_binary_model(names)
class svm_to_binary(object):
'''
svm_to_binary(base_svm)
A simple wrapper so that
svm_to_binary(base_svm)
is a model that takes the base_svm classifier and then binarises its model output.
NOTE: This class does the same job as::
ctransforms(base_svm, svm_binary())
'''
def __init__(self, svm_base):
'''
binclassifier = svm_to_binary(svm_base)
a classifier that binarises the output of svm_base.
'''
self.base = svm_base
def train(self, features, labels, **kwargs):
model = self.base.train(features, labels, **kwargs)
binary = svm_binary()
binary_model = binary.train(features, labels, **kwargs)
return ctransforms_model([model, binary_model])
def set_option(self, opt, value):
self.base.set_option(opt, value)
class svm_sigmoidal_correction_model(supervised_model):
def __init__(self, A, B):
self.A = A
self.B = B
def apply(self,features):
return 1./(1.+numpy.exp(features*self.A+self.B))
class svm_sigmoidal_correction(object):
'''
svm_sigmoidal_correction : a classifier
Sigmoidal approximation for obtaining a probability estimate out of the output
of an SVM.
'''
def __init__(self):
self.max_iters = None
def train(self, features, labels, **kwargs):
A,B = learn_sigmoid_constants(features,labels,self.max_iters)
return svm_sigmoidal_correction_model(A, B)
def get_params(self):
return self.max_iters
def set_params(self,params):
self.max_iters = params
def sigma_value_fisher(features,labels):
'''
f = sigma_value_fisher(features,labels)
value_s = f(s)
Computes a function which computes how good the value of sigma
is for the features. This function should be *minimised* for a
good value of sigma.
Parameters
-----------
features : features matrix as 2-ndarray.
Returns
-------
f : a function: float -> float
this function should be minimised for a good `sigma`
Reference
----------
Implements the measure in
"Determination of the spread parameter in the
Gaussian kernel for classification and regression"
by Wenjian Wanga, Zongben Xua, Weizhen Luc, and Xiaoyun Zhanga
'''
features = np.asanyarray(features)
xij = np.dot(features,features.T)
f2 = np.sum(features**2,1)
d = f2-2*xij
d = d.T + f2
N1 = (labels==0).sum()
N2 = (labels==1).sum()
C1 = -d[labels == 0][:,labels == 0]
C2 = -d[labels == 1][:,labels == 1]
C12 = -d[labels == 0][:,labels == 1]
C1 = C1.copy()
C2 = C2.copy()
C12 = C12.copy()
def f(sigma):
sigma = float(sigma)
N1 = C1.shape[0]
N2 = C2.shape[0]
if C12.shape != (N1,N2):
raise ValueError
C1v = np.sum(np.exp(C1/sigma))/N1
C2v = np.sum(np.exp(C2/sigma))/N2
C12v = np.sum(np.exp(C12/sigma))/N1/N2
return (N1 + N2 - C1v - C2v)/(C1v/N1+C2v/N2 - 2.*C12v)
return f
class fisher_tuned_rbf_svm(object):
'''
F = fisher_tuned_rbf_svm(sigmas, base)
Returns a wrapper classifier that uses RBF kernels automatically
tuned using sigma_value_fisher.
'''
def __init__(self, sigmas, base):
self.sigmas = sigmas
self.base = base
def train(self, features, labels, **kwargs):
f = sigma_value_fisher(features, labels)
fs = [f(s) for s in self.sigmas]
self.sigma = self.sigmas[np.argmin(fs)]
self.base.set_option('kernel',rbf_kernel(self.sigma))
return self.base.train(features, labels, **kwargs)
milk-0.5/milk/supervised/randomforest.py 0000664 0001750 0001750 00000005276 11723217323 022567 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2010-2011, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# License: MIT. See COPYING.MIT file in the milk distribution
'''
Random Forest
-------------
Main elements
-------------
rf_learner : A learner object
'''
from __future__ import division
import numpy as np
import milk.supervised.tree
from .normalise import normaliselabels
from .base import supervised_model
from ..utils import get_nprandom
__all__ = [
'rf_learner',
]
def _sample(features, labels, n, R):
'''
features', labels' = _sample(features, labels, n, R)
Sample n element from (features,labels)
Parameters
----------
features : sequence
labels : sequence
Same size as labels
n : integer
R : random object
Returns
-------
features' : sequence
labels' : sequence
'''
N = len(features)
sfeatures = []
slabels = []
for i in xrange(n):
idx = R.randint(N)
sfeatures.append(features[idx])
slabels.append(labels[idx])
return np.array(sfeatures), np.array(slabels)
class rf_model(supervised_model):
def __init__(self, forest, names, return_label = True):
self.forest = forest
self.names = names
self.return_label = return_label
def apply(self, features):
rf = len(self.forest)
votes = sum(t.apply(features) for t in self.forest)
if self.return_label:
return (votes > (rf//2))
return votes / rf
class rf_learner(object):
'''
Random Forest Learner
learner = rf_learner(rf=101, frac=.7)
Attributes
----------
rf : integer, optional
Nr of trees to learn (default: 101)
frac : float, optional
Sample fraction
R : np.random object
Source of randomness
'''
def __init__(self, rf=101, frac=.7, R=None):
self.rf = rf
self.frac = frac
self.R = get_nprandom(R)
def train(self, features, labels, normalisedlabels=False, names=None, return_label=True, **kwargs):
N,M = features.shape
m = int(self.frac*M)
n = int(self.frac*N)
R = get_nprandom(kwargs.get('R', self.R))
tree = milk.supervised.tree.tree_learner(return_label=return_label)
forest = []
if not normalisedlabels:
labels,names = normaliselabels(labels)
elif names is None:
names = (0,1)
for i in xrange(self.rf):
forest.append(
tree.train(*_sample(features, labels, n, R),
**{'normalisedlabels' : True})) # This syntax is necessary for Python 2.5
return rf_model(forest, names, return_label)
milk-0.5/milk/unsupervised/ 0000775 0001750 0001750 00000000000 12045730202 020035 5 ustar luispedro luispedro 0000000 0000000 milk-0.5/milk/unsupervised/normalise.py 0000664 0001750 0001750 00000007535 12031431355 022414 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2012, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
from __future__ import division
import numpy as np
__all__ = [
'center',
'zscore',
]
def _nanmean(arr, axis=None):
nancounts = np.sum(~np.isnan(arr), axis=axis)
return np.nansum(arr,axis=axis)/nancounts
def _nanstd(arr, axis=None):
if axis == 1:
return _nanstd(arr.T, axis=0)
mu = _nanmean(arr,axis=axis)
return np.sqrt(_nanmean((arr-mu)**2, axis=axis))
def zscore(features, axis=0, can_have_nans=True, inplace=False):
"""
features = zscore(features, axis=0, can_have_nans=True, inplace=False)
Returns a copy of features which has been normalised to zscores
Parameters
----------
features : ndarray
2-D input array
axis : integer, optional
which axis to normalise (default: 0)
can_have_nans : boolean, optional
whether ``features`` is allowed to have NaNs (default: True)
inplace : boolean, optional
Whether to operate inline (i.e., potentially change the input array).
Default is False
Returns
-------
features : ndarray
zscored version of features
"""
if features.ndim != 2:
raise('milk.unsupervised.zscore: Can only handle 2-D arrays')
if not inplace:
features = features.copy()
if can_have_nans:
mu = _nanmean(features, axis)
sigma = _nanstd(features, axis)
else:
mu = features.mean(axis)
sigma = np.std(features, axis)
sigma[sigma == 0] = 1.
if axis == 0:
features -= mu
features /= sigma
elif axis == 1:
features -= mu[:,None]
features /= sigma[:,None]
return features
def center(features, axis=0, can_have_nans=True, inplace=False):
'''
centered, mean = center(features, axis=0, inplace=False)
Center data
Parameters
----------
features : ndarray
2-D input array
axis : integer, optional
which axis to normalise (default: 0)
can_have_nans : boolean, optional
whether ``features`` is allowed to have NaNs (default: True)
inplace : boolean, optional
Whether to operate inline (i.e., potentially change the input array).
Default is False
Returns
-------
features : ndarray
centered version of features
mean : ndarray
mean values
'''
if can_have_nans:
meanfunction = _nanmean
else:
meanfunction = np.mean
features = np.array(features, copy=(not inplace), dtype=float)
mean = meanfunction(features, axis=axis)
if axis == 0:
features -= mean
elif axis == 1:
features -= mean[:,None]
else:
raise ValueError('milk.unsupervised.center: axis ∉ { 0, 1}')
return features, mean
milk-0.5/milk/unsupervised/parzen.py 0000664 0001750 0001750 00000005522 11723217323 021720 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2010, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
from __future__ import division
import numpy as np
def get_parzen_1class_rbf_loocv(features):
'''
f,fprime = get_parzen_1class_rbf_loocv(features)
Leave-one-out crossvalidation value for 1-class Parzen window evaluator for
features.
Parameters
----------
features : ndarray
feature matrix
Returns
-------
f : function: double -> double
function which evaluates the value of a window value. Minize to get the
best window value.
fprime : function: double -> double
function: df/dh
'''
from milk.unsupervised.pdist import pdist
D2 = -pdist(features)
n = len(features)
sumD2 = D2.sum()
D2.flat[::(n+1)] = -np.inf
def f(h):
D2h = D2 / (2.*h)
np.exp(D2h, D2h)
val = D2h.sum()
return val/np.sqrt(2*h*np.pi)
def fprime(h):
D2h = D2 / (2.*h)
D2h.flat[::(n+1)] = 1.
D2h *= np.exp(D2h)
val = D2h.sum() - D2h.trace()
val /= np.sqrt(2*h*np.pi)
return -1./(4*np.pi*h)*f(h) + val
return f,fprime
def parzen(features, h):
'''
f = parzen(features, h)
Parzen window smoothing
Parameters
----------
features : ndarray
feature matrix
h : double
bandwidth
Returns
-------
f : callable (double^N -> double)
density function
'''
sum2 = np.array([np.dot(f,f) for f in features])
N = len(features)
beta = np.sqrt(2*h*np.pi)/N
def f(x):
dist = np.dot(features, -2*x)
dist += sum2
dist += np.dot(c,c)
dist /= 2.*h
np.exp(dist, dist)
val = dist.sum()
return val*beta
return f
milk-0.5/milk/unsupervised/affinity.py 0000664 0001750 0001750 00000010506 11723217323 022230 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
# Copyright (C) 2010-2011,
# Luis Pedro Coelho ,
# Alexandre Gramfort ,
# Gael Varoquaux
#
# License: MIT. See COPYING.MIT file in the milk distribution
"""Affinity propagation
Original Authors (for scikits.learn):
Alexandre Gramfort alexandre.gramfort@inria.fr
Gael Varoquaux gael.varoquaux@normalesup.org
Luis Pedro Coelho made the implementation more careful about allocating
intermediate arrays.
"""
import numpy as np
__all__ = [
'affinity_propagation',
]
def affinity_propagation(S, p=None, convit=30, maxit=200, damping=0.5, copy=True, R=0):
"""Perform Affinity Propagation Clustering of data
Parameters
----------
S : array [n_points, n_points]
Matrix of similarities between points
p : array [n_points,] or float, optional
Preferences for each point
damping : float, optional
Damping factor
copy : boolean, optional
If copy is False, the affinity matrix is modified inplace by the
algorithm, for memory efficiency
R : source of randomness
Returns
-------
cluster_centers_indices : array [n_clusters]
index of clusters centers
labels : array [n_points]
cluster labels for each point
Notes
-----
See examples/plot_affinity_propagation.py for an example.
Reference:
Brendan J. Frey and Delbert Dueck, "Clustering by Passing Messages
Between Data Points", Science Feb. 2007
"""
if copy:
# Copy the affinity matrix to avoid modifying it inplace
S = S.copy()
n_points = S.shape[0]
assert S.shape[0] == S.shape[1]
if p is None:
p = np.median(S)
if damping < 0.5 or damping >= 1:
raise ValueError('damping must be >= 0.5 and < 1')
random_state = np.random.RandomState(R)
# Place preferences on the diagonal of S
S.flat[::(n_points+1)] = p
A = np.zeros((n_points, n_points))
R = np.zeros((n_points, n_points)) # Initialize messages
# Remove degeneracies
noise = random_state.randn(n_points, n_points)
typeinfo = np.finfo(S.dtype)
noise *= typeinfo.tiny*100
S += noise
del noise
# Execute parallel affinity propagation updates
e = np.zeros((n_points, convit))
ind = np.arange(n_points)
for it in range(maxit):
Aold = A.copy()
Rold = R.copy()
A += S
I = np.argmax(A, axis=1)
Y = A[ind, I]#np.max(A, axis=1)
A[ind, I] = typeinfo.min
Y2 = np.max(A, axis=1)
R = S - Y[:, np.newaxis]
R[ind, I[ind]] = S[ind, I] - Y2
Rold *= damping
R *= (1-damping)
R += Rold
# Compute availabilities
Rd = R.diagonal().copy()
np.maximum(R, 0, R)
R.flat[::n_points+1] = Rd
A = np.sum(R, axis=0)[np.newaxis, :] - R
dA = np.diag(A)
A = np.minimum(A, 0)
A.flat[::n_points+1] = dA
Aold *= damping
A *= (1-damping)
A += Aold
# Check for convergence
E = (np.diag(A) + np.diag(R)) > 0
e[:, it % convit] = E
K = np.sum(E, axis=0)
if it >= convit:
se = np.sum(e, axis=1);
unconverged = np.sum((se == convit) + (se == 0)) != n_points
if (not unconverged and (K>0)) or (it==maxit):
print "Converged after %d iterations." % it
break
else:
print "Did not converge"
I = np.where(np.diag(A+R) > 0)[0]
K = I.size # Identify exemplars
if K > 0:
c = np.argmax(S[:, I], axis=1)
c[I] = np.arange(K) # Identify clusters
# Refine the final set of exemplars and clusters and return results
for k in range(K):
ii = np.where(c==k)[0]
j = np.argmax(np.sum(S[ii, ii], axis=0))
I[k] = ii[j]
c = np.argmax(S[:, I], axis=1)
c[I] = np.arange(K)
labels = I[c]
# Reduce labels to a sorted, gapless, list
cluster_centers_indices = np.unique(labels)
labels = np.searchsorted(cluster_centers_indices, labels)
else:
labels = np.empty((n_points, 1))
cluster_centers_indices = None
labels.fill(np.nan)
return cluster_centers_indices, labels
milk-0.5/milk/unsupervised/kmeans.py 0000664 0001750 0001750 00000025633 12043765776 021724 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2012, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
from __future__ import division
import numpy as np
from numpy import linalg
from . import _kmeans
from ..utils import get_pyrandom
from .normalise import zscore
__all__ = [
'kmeans',
'select_best_kmeans',
'repeated_kmeans',
]
try:
# This tests for the presence of 3-argument np.dot
# with the 3rd argument being the output argument
_x = np.array([
[1., 1.],
[0.,1.]] )
_y = np.array([2., 4.])
_r = np.array([0.,0.])
np.dot(_x, _y, _r)
if _r[0] != 6 or _r[1] != 4:
raise NotImplementedError
_dot3 = np.dot
except:
def _dot3(x, y, _):
return np.dot(x,y)
finally:
del _x
del _y
del _r
def _mahalanobis2(fmatrix, x, icov):
diff = fmatrix-x
# The expression below seems to be faster than looping over the elements and summing
return np.dot(diff, np.dot(icov, diff.T)).diagonal()
def centroid_errors(fmatrix, assignments, centroids):
'''
errors = centroid_errors(fmatrix, assignments, centroids)
Computes the following::
for all i, j:
ci = assignments[i]
errors[i,j] = fmatrix[ci, j] - centroids[ci, j]
Parameters
----------
fmatrix : 2D ndarray
feature matrix
assignments : 1D ndarray
Assignments array
centroids : 2D ndarray
centroids
Returns
-------
errors : float
Difference between fmatrix and corresponding centroid
'''
errors = []
for k,c in enumerate(centroids):
errors.append(fmatrix[assignments == k] - c)
return np.concatenate(errors)
def residual_sum_squares(fmatrix,assignments,centroids,distance='euclidean',**kwargs):
'''
rss = residual_sum_squares(fmatrix, assignments, centroids, distance='euclidean', **kwargs)
Computes residual sum squares
Parameters
----------
fmatrix : 2D ndarray
feature matrix
assignments : 1D ndarray
Assignments array
centroids : 2D ndarray
centroids
Returns
-------
rss : float
residual sum squares
'''
if distance != 'euclidean':
raise NotImplemented("residual_sum_squares only implemented for 'euclidean' distance")
rss = 0.0
for k, c in enumerate(centroids):
diff = fmatrix[assignments == k] - c
diff = diff.ravel()
rss += np.dot(diff, diff)
return rss
def assign_centroids(fmatrix, centroids, histogram=False, normalize=False, normalise=None):
'''
cids = assign_centroids(fmatrix, centroids, histogram=False, normalize=False)
Assigns a centroid to each element of fmatrix
Parameters
----------
fmatrix : 2D ndarray
feature matrix
centroids : 2D ndarray
centroids matrix
histogram : boolean, optional
If True, then the result is actually a histogram
normalize : boolean, optional
If True and ``histogram``, then the histogram is normalized to sum to
one.
Returns
-------
cids : sequence
``cids[i]`` is the index of the centroid closes to ``fmatrix[i];`` or,
if ``histogram``, then ``cids[i]`` is the number of points that were
assigned to centroid ``i.``
'''
dists = np.dot(fmatrix, (-2)*centroids.T)
dists += np.array([np.dot(c,c) for c in centroids])
cids = dists.argmin(1)
if histogram:
hist = np.array(
[np.sum(cids == ci) for ci in xrange(len(centroids))],
np.float)
if (normalize or normalise) and len(fmatrix):
hist /= hist.sum()
return hist
return cids
def _pycomputecentroids(fmatrix, centroids, assignments, counts):
k, Nf = centroids.shape
bins = np.arange(k+1)
ncounts,_ = np.histogram(assignments, bins)
counts[:] = ncounts
any_empty = False
mean = None
for ci,count in enumerate(counts):
if count:
where = (assignments.T == ci)
mean = _dot3(where, fmatrix, mean) # mean = dot(fmatrix.T, where.T), but it is better to not cause copies
mean /= count
centroids[ci] = mean
else:
any_empty = True
return any_empty
def kmeans(fmatrix, k, distance='euclidean', max_iter=1000, R=None, **kwargs):
'''
assignments, centroids = kmean(fmatrix, k, distance='euclidean', max_iter=1000, R=None, icov=None, covmat=None)
k-Means Clustering
Parameters
----------
fmatrix : ndarray
2-ndarray (Nelements x Nfeatures)
distance: string, optional
one of:
- 'euclidean' : euclidean distance (default)
- 'seuclidean' : standartised euclidean distance. This is equivalent to first normalising the features.
- 'mahalanobis' : mahalanobis distance.
This can make use of the following keyword arguments:
+ 'icov' (the inverse of the covariance matrix),
+ 'covmat' (the covariance matrix)
If neither is passed, then the function computes the covariance from the feature matrix
max_iter : integer, optional
Maximum number of iteration (default: 1000)
R : source of randomness, optional
Returns
-------
assignments : ndarray
An 1-D array of size `len(fmatrix)`
centroids : ndarray
An array of `k'` centroids
'''
fmatrix = np.asanyarray(fmatrix)
if not np.issubdtype(fmatrix.dtype, np.float):
fmatrix = fmatrix.astype(np.float)
if distance == 'seuclidean':
fmatrix = zscore(fmatrix)
distance = 'euclidean'
if distance == 'euclidean':
def distfunction(fmatrix, cs, dists):
dists = _dot3(fmatrix, (-2)*cs.T, dists)
dists += np.array([np.dot(c,c) for c in cs])
# For a distance, we'd need to add the fmatrix**2 components, but
# it doesn't matter because we are going to perform an argmin() on
# the result.
return dists
elif distance == 'mahalanobis':
icov = kwargs.get('icov', None)
if icov is None:
covmat = kwargs.get('covmat', None)
if covmat is None:
covmat = np.cov(fmatrix.T)
icov = linalg.inv(covmat)
def distfunction(fmatrix, cs, _):
return np.array([_mahalanobis2(fmatrix, c, icov) for c in cs]).T
else:
raise ValueError('milk.unsupervised.kmeans: `distance` argument unknown (%s)' % distance)
if k < 2:
raise ValueError('milk.unsupervised.kmeans `k` should be >= 2.')
if fmatrix.dtype in (np.float32, np.float64) and fmatrix.flags['C_CONTIGUOUS']:
computecentroids = _kmeans.computecentroids
else:
computecentroids = _pycomputecentroids
R = get_pyrandom(R)
centroids = np.array(R.sample(fmatrix,k), fmatrix.dtype)
prev = np.zeros(len(fmatrix), np.int32)
counts = np.empty(k, np.int32)
dists = None
for i in xrange(max_iter):
dists = distfunction(fmatrix, centroids, dists)
assignments = dists.argmin(1)
if np.all(assignments == prev):
break
if computecentroids(fmatrix, centroids, assignments.astype(np.int32), counts):
(empty,) = np.where(counts == 0)
centroids = np.delete(centroids, empty, axis=0)
k = len(centroids)
counts = np.empty(k, np.int32)
# This will cause new matrices to be allocated in the next iteration
dists = None
prev[:] = assignments
return assignments, centroids
def repeated_kmeans(fmatrix,k,iterations,distance='euclidean',max_iter=1000,R=None,**kwargs):
'''
assignments,centroids = repeated_kmeans(fmatrix, k, repeats, distance='euclidean',max_iter=1000,**kwargs)
Runs kmeans repeats times and returns the best result as evaluated
according to distance
See Also
--------
kmeans : runs kmeans once
Parameters
----------
fmatrix : feature matrix
k : nr of centroids
iterations : Nr of repetitions
distance : 'euclidean' (default) or 'seuclidean'
max_iter : Max nr of iterations per kmeans run
R : random source
Returns
-------
assignments : 1-D array of assignments
centroids : centroids
These are the same returns as the kmeans function
'''
kwargs['max_iter'] = max_iter
return select_best_kmeans(fmatrix, [k], repeats=iterations, method='loglike', distance=distance, **kwargs)
def select_best_kmeans(fmatrix, ks, repeats=1, method='AIC', R=None, **kwargs):
'''
assignments,centroids = select_best_kmeans(fmatrix, ks, repeats=1, method='AIC', R=None, **kwargs)
Runs kmeans repeats times and returns the best result as evaluated
according to distance
See Also
--------
kmeans : runs kmeans once
Parameters
----------
fmatrix : feature matrix
ks : sequence of integers
nr of centroids to try
iterations : integer, optional
Nr of repetitions for each value of k
R : random source, optional
Returns
-------
assignments : 1-D array of assignments
centroids : 2-D ndarray
centroids
These are the same returns as the kmeans function
'''
best = None
best_val = np.inf
R = get_pyrandom(R)
from milk.unsupervised.gaussianmixture import AIC, BIC, log_likelihood
if method == 'AIC':
method = AIC
elif method == 'BIC':
method = BIC
elif method == 'loglike':
method = log_likelihood
else:
raise ValueError('milk.kmeans.select_best_kmeans: unknown method: %s' % method)
if 'distance' in kwargs and kwargs['distance'] == 'seuclidean':
fmatrix = zscore(fmatrix)
for k in ks:
for i in xrange(repeats):
As,Cs = kmeans(fmatrix, k, R=R, **kwargs)
value = method(fmatrix, As, Cs)
if value < best_val:
best_val = value
best = As,Cs
return best
milk-0.5/milk/unsupervised/__init__.py 0000664 0001750 0001750 00000001553 12030344666 022163 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2012, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# License: MIT. See COPYING.MIT file in the milk distribution
'''
milk.unsupervised
Unsupervised Learning
---------------------
- kmeans: This is a highly optimised implementation of kmeans
- PCA: Simple implementation
- Non-negative matrix factorisation: both direct and with sparsity constraints
'''
from kmeans import kmeans,repeated_kmeans, select_best_kmeans
from gaussianmixture import *
from pca import pca, mds
import nnmf
from nnmf import *
from pdist import pdist, plike
from .som import som
from .normalise import zscore, center
__all__ = [
'center',
'kmeans',
'mds'
'pca',
'pdist',
'plike',
'repeated_kmeans',
'select_best_kmeans',
'som',
'zscore',
] + \
nnmf.__all__
milk-0.5/milk/unsupervised/nnmf/ 0000775 0001750 0001750 00000000000 12045730202 020773 5 ustar luispedro luispedro 0000000 0000000 milk-0.5/milk/unsupervised/nnmf/lee_seung.py 0000664 0001750 0001750 00000005740 11723217323 023327 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2010, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
from __future__ import division
import numpy as np
from numpy import dot
from ...utils import get_nprandom
__all__ = ['nnmf']
def nnmf(V, r, cost='norm2', max_iter=int(1e4), tol=1e-8, R=None):
'''
A,S = nnmf(X, r, cost='norm2', tol=1e-8, R=None)
Implement Lee & Seung's algorithm
Parameters
----------
V : 2-ndarray
input matrix
r : integer
nr of latent features
cost : one of:
'norm2' : minimise || X - AS ||_2 (default)
'i-div' : minimise D(X||AS), where D is I-divergence (generalisation of K-L divergence)
max_iter : integer, optional
maximum number of iterations (default: 10000)
tol : double
tolerance threshold for early exit (when the update factor is with tol
of 1., the function exits)
R : integer, optional
random seed
Returns
-------
A : 2-ndarray
S : 2-ndarray
Reference
---------
"Algorithms for Non-negative Matrix Factorization"
by Daniel D Lee, Sebastian H Seung
(available at http://citeseer.ist.psu.edu/lee01algorithms.html)
'''
# Nomenclature in the function follows lee & seung, while outside nomenclature follows
eps = 1e-8
n,m = V.shape
R = get_nprandom(R)
W = R.standard_normal((n,r))**2
H = R.standard_normal((r,m))**2
for i in xrange(max_iter):
if cost == 'norm2':
updateH = dot(W.T,V)/(dot(dot(W.T,W),H)+eps)
H *= updateH
updateW = dot(V,H.T)/(dot(W,dot(H,H.T))+eps)
W *= updateW
elif cost == 'i-div':
raise NotImplementedError,'I-Div not implemented in lee_seung.nnmf'
if True or (i % 10) == 0:
max_update = max(updateW.max(),updateH.max())
if abs(1.-max_update) < tol:
break
return W,H
milk-0.5/milk/unsupervised/nnmf/__init__.py 0000664 0001750 0001750 00000002473 11723217323 023120 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2010, Luis Pedro Coelho
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
import lee_seung as ls
from hoyer import sparse_nnmf
import hoyer
lee_seung = ls.nnmf
__all__ = ['lee_seung','sparse_nnmf']
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
milk-0.5/milk/unsupervised/nnmf/hoyer.py 0000664 0001750 0001750 00000010240 11723217323 022476 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2010, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
from __future__ import division
import numpy as np
from ...utils import get_nprandom
__all__ = ['hoyer_sparse_nnmf']
def sp(s):
L2 = np.sqrt(np.dot(s,s))
L1 = np.abs(s).sum()
sn = np.sqrt(len(s))
return (sn-L1/L2)/(sn-1)
def _solve_alpha(s,m,L2):
sm = s-m
s2 = np.dot(s,s)
sm2 = np.dot(sm, sm)
m2 = np.dot(m, m)
dot = np.dot(m, sm)
alpha = (-dot + np.sqrt(dot**2 - sm2*(m2-L2**2)))/sm2
return alpha
def _project(x,L1,L2):
'''
Implement projection onto sparse space
'''
x = np.asanyarray(x)
n = len(x)
s = x + (L1 - x.sum())/n
Z = np.zeros(n,bool)
while True:
m = (~Z) * L1/(n-Z.sum())
alpha = _solve_alpha(s,m,L2)
s = m + alpha * (s - m)
negs = (s < 0)
if not negs.any():
return s
Z |= negs
s[Z] = 0
c = (s.sum() - L1)/(~Z).sum()
s -= c*(~Z)
def _L1for(s,x,L2):
'''
Solve for L1 in
s = [ sqrt(n) - L1/L2] / [sqrt(n) - 1]
'''
sn = np.sqrt(len(x))
return L2*(s+sn-s*sn)
def sparse_nnmf(V, r, sparsenessW=None, sparsenessH=None, max_iter=10000, R=None):
'''
W,H = hoyer.sparse_nnmf(V, r, sparsenessW = None, sparsenessH = None, max_iter=10000, R=None)
Implement sparse nonnegative matrix factorisation.
Parameters
----------
V : 2-D matrix
input feature matrix
r : integer
number of latent features
sparsenessW : double, optional
sparseness contraint on W (default: no sparsity contraint)
sparsenessH : double, optional
sparseness contraint on H (default: no sparsity contraint)
max_iter : integer, optional
maximum nr of iterations (default: 10000)
R : integer, optional
source of randomness
Returns
-------
W : 2-ndarray
H : 2-ndarray
Reference
---------
"Non-negative Matrix Factorisation with Sparseness Constraints"
by Patrik Hoyer
in Journal of Machine Learning Research 5 (2004) 1457--1469
'''
n,m = V.shape
R = get_nprandom(R)
mu_W = .15
mu_H = .15
eps = 1e-8
W = R.standard_normal((n,r))**2
H = R.standard_normal((r,m))**2
def fix(X, sparseness):
for i in xrange(r):
row = X[i]
L2 = np.sqrt(np.dot(row, row))
row /= L2
X[i] = _project(row, _L1for(sparseness, row, 1.), 1.)
def fixW():
fix(W.T, sparsenessW)
def fixH():
fix(H, sparsenessH)
if sparsenessW is not None: fixW()
if sparsenessH is not None: fixH()
for i in xrange(max_iter):
if sparsenessW is not None:
W -= mu_W * np.dot(np.dot(W,H)-V,H.T)
fixW()
else:
updateW = np.dot(V,H.T)/(np.dot(W,np.dot(H,H.T))+eps)
W *= updateW
if sparsenessH is not None:
H -= mu_H * np.dot(W.T,np.dot(W,H)-V)
fixH()
else:
updateH = np.dot(W.T,V)/(np.dot(np.dot(W.T,W),H)+eps)
H *= updateH
return W,H
milk-0.5/milk/unsupervised/som.py 0000664 0001750 0001750 00000006304 11723217323 021216 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2010, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division
import numpy as np
from ..utils import get_pyrandom
from . import _som
def putpoints(grid, points, L=.2, radius=4, iterations=1, shuffle=True, R=None):
'''
putpoints(grid, points, L=.2, radius=4, iterations=1, shuffle=True, R=None)
Feeds elements of `points` into the SOM `grid`
Parameters
----------
grid : ndarray
Self organising map
points : ndarray
data to feed to array
L : float, optional
How much to influence neighbouring points (default: .2)
radius : integer, optional
Maximum radius of influence (in L_1 distance, default: 4)
iterations : integer, optional
Number of iterations
shuffle : boolean, optional
Whether to shuffle the points before each iterations
R : source of randomness
'''
if radius is None:
radius = 4
if type(L) != float:
raise TypeError("milk.unsupervised.som: L should be floating point")
if type(radius) != int:
raise TypeError("milk.unsupervised.som: radius should be an integer")
if grid.dtype != np.float32:
raise TypeError('milk.unsupervised.som: only float32 arrays are accepted')
if points.dtype != np.float32:
raise TypeError('milk.unsupervised.som: only float32 arrays are accepted')
if len(grid.shape) == 2:
grid = grid.reshape(grid.shape+(1,))
if shuffle:
random = get_pyrandom(R)
for i in xrange(iterations):
if shuffle:
random.shuffle(points)
_som.putpoints(grid, points, L, radius)
def closest(grid, f):
'''
y,x = closest(grid, f)
Finds the coordinates of the closest point in the `grid` to `f`
::
y,x = \\argmin_{y,x} { || grid[y,x] - f ||^2 }
Parameters
----------
grid : ndarray of shape Y,X,J
self-organised map
f : ndarray of shape J
point
Returns
-------
y,x : integers
coordinates into `grid`
'''
delta = grid - f
delta **= 2
delta = delta.sum(2)
return np.unravel_index(delta.argmin(), delta.shape)
def som(data, shape, iterations=1000, L=.2, radius=4, R=None):
'''
grid = som(data, shape, iterations=1000, L=.2, radius=4, R=None):
Self-organising maps
Parameters
----------
points : ndarray
data to feed to array
shape : tuple
Desired shape of output. Must be 2-dimensional.
L : float, optional
How much to influence neighbouring points (default: .2)
radius : integer, optional
Maximum radius of influence (in L_1 distance, default: 4)
iterations : integer, optional
Number of iterations
R : source of randomness
Returns
-------
grid : ndarray
Map
'''
R = get_pyrandom(R)
d = data.shape[1]
if data.dtype != np.float32:
data = data.astype(np.float32)
grid = np.array(R.sample(data, np.product(shape))).reshape(shape + (d,))
putpoints(grid, data, L=L, radius=radius, iterations=iterations, shuffle=True, R=R)
return grid
milk-0.5/milk/unsupervised/gaussianmixture.py 0000664 0001750 0001750 00000012370 12043765776 023670 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2011, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division
import numpy as np
from numpy import log, pi, array
from numpy.linalg import det, inv
from .kmeans import residual_sum_squares, centroid_errors
__all__ = [
'BIC',
'AIC',
'log_likelihood',
'nr_parameters',
]
def log_likelihood(fmatrix,assignments,centroids,model='one_variance',covs=None):
'''
log_like = log_likelihood(feature_matrix, assignments, centroids, model='one_variance', covs=None)
Compute the log likelihood of feature_matrix[i] being generated from centroid[i]
'''
N,q = fmatrix.shape
k = len(centroids)
if model == 'one_variance':
Rss = residual_sum_squares(fmatrix,assignments,centroids)
#sigma2=Rss/N
return -N/2.*log(2*pi*Rss/N)-N/2
elif model == 'diagonal_covariance':
errors = centroid_errors(fmatrix,assignments,centroids)
errors *= errors
errors = errors.sum(1)
Rss = np.zeros(k)
counts = np.zeros(k)
for i in xrange(fmatrix.shape[0]):
c = assignments[i]
Rss[c] += errors[i]
counts[c] += 1
sigma2s = Rss/(counts+(counts==0))
return -N/2.*log(2*pi) -N/2. -1/2.*np.sum(counts*np.log(sigma2s+(counts==0)))
elif model == 'full_covariance':
res = -N*q/2.*log(2*pi)
for k in xrange(len(centroids)):
diff = (fmatrix[assignments == k] - centroids[k])
if covs is None:
covm = np.cov(diff.T)
else:
covm = covs[k]
if covm.shape == ():
covm = np.matrix([[covm]])
icov = np.matrix(inv(covm))
diff = np.matrix(diff)
Nk = diff.shape[0]
res += -Nk/2.*log(det(covm)) + \
-.5 * (diff * icov * diff.T).diagonal().sum()
return res
raise ValueError("log_likelihood: cannot handle model '%s'" % model)
def nr_parameters(fmatrix,k,model='one_variance'):
'''
nr_p = nr_parameters(fmatrix, k, model='one_variance')
Compute the number of parameters for a model of k clusters on
Parameters
----------
fmatrix : 2d-array
feature matrix
k : integer
nr of clusters
model : str
one of 'one_variance' (default), 'diagonal_covariance', or 'full_covariance'
Returns
-------
nr_p : integer
Number of parameters
'''
N,q = fmatrix.shape
if model == 'one_variance':
return k*q+1
elif model == 'diagonal_covariance':
return k*(q+1)
elif model == 'full_covariance':
return k*+q*q
raise ValueError("milk.unsupervised.gaussianmixture.nr_parameters: cannot handle model '%s'" % model)
def _compute(type, fmatrix, assignments, centroids, model='one_variance', covs=None):
N,q = fmatrix.shape
k = len(centroids)
log_like = log_likelihood(fmatrix, assignments, centroids, model, covs)
n_param = nr_parameters(fmatrix,k,model)
if type == 'BIC':
return -2*log_like + n_param * log(N)
elif type == 'AIC':
return -2*log_like + 2 * n_param
else:
assert False
def BIC(fmatrix, assignments, centroids, model='one_variance', covs=None):
'''
B = BIC(fmatrix, assignments, centroids, model='one_variance', covs={From Data})
Compute Bayesian Information Criterion
Parameters
----------
fmatrix : 2d-array
feature matrix
assignments : 2d-array
Centroid assignments
centroids : sequence
Centroids
model : str, optional
one of
'one_variance'
All features share the same variance parameter sigma^2. Default
'full_covariance'
Estimate a full covariance matrix or use covs[i] for centroid[i]
covs : sequence or matrix, optional
Covariance matrices. If None, then estimate from the data. If scalars
instead of matrices are given, then s stands for sI (i.e., the diagonal
matrix with s along the diagonal).
Returns
-------
B : float
BIC value
See Also
--------
AIC
'''
return _compute('BIC', fmatrix, assignments, centroids, model, covs)
def AIC(fmatrix,assignments,centroids,model='one_variance',covs=None):
'''
A = AIC(fmatrix,assignments,centroids,model)
Compute Akaike Information Criterion
Parameters
----------
fmatrix : 2d-array
feature matrix
assignments : 2d-array
Centroid assignments
centroids : sequence
Centroids
model : str, optional
one of
'one_variance'
All features share the same variance parameter sigma^2. Default
'full_covariance'
Estimate a full covariance matrix or use covs[i] for centroid[i]
covs : sequence, optional
Covariance matrices. If None, then estimate from the data. If scalars
instead of matrices are given, then s stands for sI (i.e., the diagonal
matrix with s along the diagonal).
Returns
-------
B : float
AIC value
See Also
--------
BIC
'''
return _compute('AIC', fmatrix, assignments, centroids, model, covs)
milk-0.5/milk/unsupervised/_som.cpp 0000664 0001750 0001750 00000007502 11723217323 021510 0 ustar luispedro luispedro 0000000 0000000 #include
#include
#include
extern "C" {
#include
#include
}
namespace {
struct SOM_Exception {
SOM_Exception(const char* msg): msg(msg) { }
const char* msg;
};
void assert_type_contiguous(PyArrayObject* array,int type) {
if (!PyArray_Check(array) ||
PyArray_TYPE(array) != type ||
!PyArray_ISCONTIGUOUS(array)) {
throw SOM_Exception("Arguments to putpoints don't conform to expectation. Are you calling this directly? This is an internal function!");
}
}
void putpoints(PyArrayObject* grid, PyArrayObject* points, float L, int radius) {
if (PyArray_NDIM(grid) != 3) throw SOM_Exception("grid should be three dimensional");
if (PyArray_NDIM(points) != 2) throw SOM_Exception("points should be two dimensional");
const int rows = PyArray_DIM(grid, 0);
const int cols = PyArray_DIM(grid, 1);
const int d = PyArray_DIM(grid, 2);
const int n = PyArray_DIM(points, 0);
if (PyArray_DIM(points, 1) != d) throw SOM_Exception("second dimension of points is not third dimension of grid");
Py_BEGIN_ALLOW_THREADS
for (int i = 0; i != n; i++){
const float* p = static_cast(PyArray_GETPTR1(points, i));
int min_y = 0;
int min_x = 0;
float best = std::numeric_limits::max();
for (int y = 0; y != rows; ++y) {
for (int x = 0; x != cols; ++x) {
float dist = 0.;
const float* gpoint = static_cast(PyArray_GETPTR2(grid, y, x));
for (int j = 0; j != d; ++j) {
dist += (p[j] - gpoint[j])*(p[j] - gpoint[j]);
}
if (dist < best) {
best = dist;
min_y = y;
min_x = x;
}
}
}
const int start_y = std::max(0, min_y - radius);
const int start_x = std::max(0, min_x - radius);
const int end_y = std::min(rows, min_y + radius);
const int end_x = std::min(rows, min_x + radius);
for (int y = start_y; y != end_y; ++y) {
for (int x = start_x; x != end_x; ++x) {
const float L2 = L /(1 + std::abs(min_y - y) + std::abs(min_x - x));
float* gpoint = static_cast(PyArray_GETPTR2(grid, y, x));
for (int j = 0; j != d; ++j) {
gpoint[j] *= (1.-L2);
gpoint[j] += L2 * p[j];
}
}
}
}
Py_END_ALLOW_THREADS
}
PyObject* py_putpoints(PyObject* self, PyObject* args) {
try {
PyArrayObject* grid;
PyArrayObject* points;
float L;
int radius;
if (!PyArg_ParseTuple(args, "OOfi", &grid, &points, &L, &radius)) {
const char* errmsg = "Arguments were not what was expected for putpoints.\n"
"This is an internal function: Do not call directly unless you know exactly what you're doing.\n";
PyErr_SetString(PyExc_RuntimeError,errmsg);
return 0;
}
assert_type_contiguous(grid, NPY_FLOAT);
assert_type_contiguous(points, NPY_FLOAT);
putpoints(grid, points, L, radius);
Py_RETURN_NONE;
} catch (const SOM_Exception& exc) {
PyErr_SetString(PyExc_RuntimeError,exc.msg);
return 0;
} catch (...) {
PyErr_SetString(PyExc_RuntimeError,"Some sort of exception in putpoints.");
return 0;
}
}
PyMethodDef methods[] = {
{"putpoints", py_putpoints, METH_VARARGS , "Do NOT call directly.\n" },
{NULL, NULL,0,NULL},
};
const char * module_doc =
"Internal SOM Module.\n"
"\n"
"Do NOT use directly!\n";
} // namespace
extern "C"
void init_som()
{
import_array();
(void)Py_InitModule3("_som", methods, module_doc);
}
milk-0.5/milk/unsupervised/pca.py 0000664 0001750 0001750 00000003437 11734651656 021204 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2012, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division
import numpy as np
from numpy import linalg
from . import normalise
from .pdist import pdist
__all__ = [
'pca',
'mds',
]
def pca(X, zscore=True):
'''
Y,V = pca(X, zscore=True)
Principal Component Analysis
Performs principal component analysis. Returns transformed
matrix and principal components
Parameters
----------
X : 2-dimensional ndarray
data matrix
zscore : boolean, optional
whether to normalise to zscores (default: True)
Returns
-------
Y : ndarray
Transformed matrix (of same dimension as X)
V : ndarray
principal components
'''
if zscore:
X = normalise.zscore(X)
C = np.cov(X.T)
w,v = linalg.eig(C)
Y = np.dot(v,X.T).T
return Y,v
def mds(features, ndims, zscore=False):
'''
X = mds(features, ndims, zscore=False)
Euclidean Multi-dimensional Scaling
Parameters
----------
features : ndarray
data matrix
ndims : int
Number of dimensions to return
zscore : boolean, optional
Whether to zscore the features (default: False)
Returns
-------
X : ndarray
array of size ``(m, ndims)`` where ``m = len(features)``
'''
if zscore:
X = normalise.zscore(features)
P2 = pdist(features)
n = len(P2)
J = np.eye(n) - (1./n)* np.ones((n,n))
B = -.5 * np.dot(J,np.dot(P2,J))
w,v = np.linalg.eig(B)
w = w[:ndims]
s = np.sign(w)
w = np.abs(w).real
w = np.diag(np.sqrt(s * w))
X = np.dot(v[:,:ndims], w)
return X.real
milk-0.5/milk/unsupervised/_kmeans.cpp 0000664 0001750 0001750 00000011510 11723217323 022162 0 ustar luispedro luispedro 0000000 0000000 #include
extern "C" {
#include
#include
}
namespace {
struct Kmeans_Exception {
Kmeans_Exception(const char* msg): msg(msg) { }
const char* msg;
};
void assert_type_contiguous(PyArrayObject* array,int type) {
if (!PyArray_Check(array) ||
PyArray_TYPE(array) != type ||
!PyArray_ISCONTIGUOUS(array)) {
throw Kmeans_Exception("Arguments to kmeans don't conform to expectation. Are you calling this directly? This is an internal function!");
}
}
template
int computecentroids(ftype* f, ftype* centroids, PyArrayObject* a_assignments, PyArrayObject* a_counts, const int N, const int Nf, const int k) {
int zero_counts = 0;
Py_BEGIN_ALLOW_THREADS
const npy_int32* assignments = static_cast(PyArray_DATA(a_assignments));
npy_int32* counts = static_cast(PyArray_DATA(a_counts));
std::fill(counts, counts + k, 0);
std::fill(centroids, centroids + k*Nf, 0.0);
for (int i = 0; i != N; i++){
const int c = assignments[i];
if (c >= k) continue; // throw Kmeans_Exception("wrong value in assignments");
ftype* ck = centroids + Nf*c;
for (int j = 0; j != Nf; ++j) {
*ck++ += *f++;
}
++counts[c];
}
for (int i = 0; i != k; ++i) {
ftype* ck = centroids + Nf*i;
const ftype c = counts[i];
if (c == 0) {
++zero_counts;
} else {
for (int j = 0; j != Nf; ++j) {
*ck++ /= c;
}
}
}
Py_END_ALLOW_THREADS
return zero_counts;
}
PyObject* py_computecentroids(PyObject* self, PyObject* args) {
try {
PyArrayObject* fmatrix;
PyArrayObject* centroids;
PyArrayObject* assignments;
PyArrayObject* counts;
if (!PyArg_ParseTuple(args, "OOOO", &fmatrix, ¢roids, &assignments, &counts)) { throw Kmeans_Exception("Wrong number of arguments for computecentroids."); }
if (!PyArray_Check(fmatrix) || !PyArray_ISCONTIGUOUS(fmatrix)) throw Kmeans_Exception("fmatrix not what was expected.");
if (!PyArray_Check(centroids) || !PyArray_ISCONTIGUOUS(centroids)) throw Kmeans_Exception("centroids not what was expected.");
if (!PyArray_Check(counts) || !PyArray_ISCONTIGUOUS(counts)) throw Kmeans_Exception("counts not what was expected.");
if (!PyArray_Check(assignments) || !PyArray_ISCONTIGUOUS(assignments)) throw Kmeans_Exception("assignments not what was expected.");
if (PyArray_TYPE(counts) != NPY_INT32) throw Kmeans_Exception("counts should be int32.");
//if (PyArray_TYPE(assignments) != NPY_INT32) throw Kmeans_Exception("assignments should be int32.");
if (PyArray_TYPE(fmatrix) != PyArray_TYPE(centroids)) throw Kmeans_Exception("centroids and fmatrix should have same type.");
if (PyArray_NDIM(fmatrix) != 2) throw Kmeans_Exception("fmatrix should be two dimensional");
if (PyArray_NDIM(centroids) != 2) throw Kmeans_Exception("centroids should be two dimensional");
if (PyArray_NDIM(assignments) != 1) throw Kmeans_Exception("assignments should be two dimensional");
const int N = PyArray_DIM(fmatrix, 0);
const int Nf = PyArray_DIM(fmatrix, 1);
const int k = PyArray_DIM(centroids, 0);
if (PyArray_DIM(centroids, 1) != Nf) throw Kmeans_Exception("centroids has wrong number of features.");
if (PyArray_DIM(assignments, 0) != N) throw Kmeans_Exception("assignments has wrong size.");
if (PyArray_DIM(counts, 0) != k) throw Kmeans_Exception("counts has wrong size.");
switch (PyArray_TYPE(fmatrix)) {
#define TRY_TYPE(code, ftype) \
case code: \
if (computecentroids( \
static_cast(PyArray_DATA(fmatrix)), \
static_cast(PyArray_DATA(centroids)), \
assignments, \
counts, \
N, Nf, k)) { \
Py_RETURN_TRUE; \
} \
Py_RETURN_FALSE;
TRY_TYPE(NPY_FLOAT, float);
TRY_TYPE(NPY_DOUBLE, double);
}
throw Kmeans_Exception("Cannot handle this type.");
} catch (const Kmeans_Exception& exc) {
PyErr_SetString(PyExc_RuntimeError,exc.msg);
return 0;
} catch (...) {
PyErr_SetString(PyExc_RuntimeError,"Some sort of exception in computecentroids.");
return 0;
}
}
PyMethodDef methods[] = {
{"computecentroids", py_computecentroids, METH_VARARGS , "Do NOT call directly.\n" },
{NULL, NULL,0,NULL},
};
const char * module_doc =
"Internal _kmeans Module.\n"
"\n"
"Do NOT use directly!\n";
} // namespace
extern "C"
void init_kmeans()
{
import_array();
(void)Py_InitModule3("_kmeans", methods, module_doc);
}
milk-0.5/milk/unsupervised/pdist.py 0000664 0001750 0001750 00000003605 12017466466 021557 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2012, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# License: MIT. See COPYING.MIT file in the milk distribution
from __future__ import division
import numpy as np
__all__ = [
'pdist',
'plike',
]
def pdist(X, Y=None, distance='euclidean2'):
'''
D = pdist(X, Y={X}, distance='euclidean2')
Compute distance matrix::
D[i,j] == np.sum( (X[i] - Y[j])**2 )
Parameters
----------
X : feature matrix
Y : feature matrix (default: use `X`)
distance : one of 'euclidean' or 'euclidean2' (default)
Returns
-------
D : matrix of doubles
'''
# Use Dij = np.dot(Xi, Xi) + np.dot(Xj,Xj) - 2.*np.dot(Xi,Xj)
if Y is None:
D = np.dot(X, X.T)
x2 = D.diagonal()
y2 = x2
else:
D = np.dot(X, Y.T)
x2 = np.array([np.dot(x,x) for x in X])
y2 = np.array([np.dot(y,y) for y in Y])
D *= -2.
D += x2[:,np.newaxis]
D += y2
# Because of numerical imprecision, we might get negative numbers
# (which cause problems down the road, e.g., when doing the sqrt):
np.maximum(D, 0, D)
if distance == 'euclidean':
np.sqrt(D, D)
return D
def plike(X, sigma2=None):
'''
L = plike(X, sigma2={guess based on X})
Compute likelihood that any two objects come from the same distribution
under a Gaussian distribution hypothesis::
L[i,j] = exp( ||X[i] - X[j]||^2 / sigma2 )
Parameters
----------
X : ndarray
feature matrix
sigma2 : float, optional
bandwidth
Returns
-------
L : ndarray
likelihood matrix
See Also
--------
pdist : function
Compute distances between objects
'''
L = pdist(X)
if sigma2 is None:
sigma2 = np.median(L)
L /= -sigma2
np.exp(L, L)
return L
milk-0.5/milk/ext/ 0000775 0001750 0001750 00000000000 12045730202 016101 5 ustar luispedro luispedro 0000000 0000000 milk-0.5/milk/ext/jugparallel.py 0000664 0001750 0001750 00000011454 12026575303 020772 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2011-2012, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
# License: MIT. See COPYING.MIT file in the milk distribution
'''
============
Jug Parallel
============
These are some functions that make it easier to take advantage of `jug
`__ to perform tasks in parallel.
All of the functions in this module return a jug.Task object and are usable
with the ``CompoundTask`` interface in jug.
None of this will make sense unless you understand and have used jug.
'''
from __future__ import division
import numpy as np
import milk
try:
from jug import TaskGenerator, value
from jug.utils import identity
from jug.mapreduce import mapreduce
from jug.mapreduce import reduce as jug_reduce
except ImportError:
raise ImportError('milk.ext.jugparallel requires jug (http://luispedro.org/software/jug)')
def _nfold_reduce(a,b):
cmat = a[0] + b[0]
names = a[1]
assert a[1] == b[1]
if len(a) == 2:
return cmat, names
predictions = np.array([a[2],b[2]])
return cmat, names, predictions.max(0)
def nfoldcrossvalidation(features, labels, **kwargs):
'''
jug_task = nfoldcrossvalidation(features, labels, **kwargs)
A jug Task that perform n-foldcrossvalidation
N-fold cross validation is inherently parallel. This function returns a
``jug.Task`` which performs n-fold crossvalidation which jug can
parallelise.
Parameters
----------
features : sequence of features
labels : sequence
kwargs : any
This will be passed down to ``milk.nfoldcrossvalidation``
Returns
-------
jug_task : a jug.Task
A Task object
See Also
--------
milk.nfoldcrossvalidation : The same functionality as a "normal" function
jug.CompoundTask : This function can be used as argument to CompoundTask
'''
nfolds = kwargs.get('nfolds', 10)
features,labels = map(identity, (features,labels))
kwargs = dict( (k,identity(v)) for k,v in kwargs.iteritems())
nfold_one = TaskGenerator(milk.nfoldcrossvalidation)
mapped = [nfold_one(features, labels, folds=[i], **kwargs) for i in xrange(nfolds)]
return jug_reduce(_nfold_reduce, mapped)
def _select_min(s0, s1):
if s0[0] < s1[0]: return s0
else: return s1
def _evaluate_solution(args):
features, results, method = args
from milk.unsupervised.gaussianmixture import AIC, BIC
if method == 'AIC':
method = AIC
elif method == 'BIC':
method = BIC
else:
raise ValueError('milk.ext.jugparallel.kmeans_select_best: unknown method: %s' % method)
assignments, centroids = results
value = method(features, assignments, centroids)
return value, results
def _select_best(features, results, method):
features = identity(features)
return mapreduce(_select_min, _evaluate_solution, [(features,r,method) for r in results], reduce_step=32, map_step=8)
def kmeans_select_best(features, ks, repeats=1, method='AIC', R=None, **kwargs):
'''
assignments_centroids = kmeans_select_best(features, ks, repeats=1, method='AIC', R=None, **kwargs)
Perform ``repeats`` calls to ``kmeans`` for each ``k`` in ``ks``, select
the best one according to ``method.``
Note that, unlike a raw ``kmeans`` call, this is *always deterministic*
even if ``R=None`` (which is interpreted as being equivalent to setting it
to a fixed value). Otherwise, the jug paradigm would be broken as different
runs would give different results.
Parameters
----------
features : array-like
2D array
ks : sequence of integers
These will be the values of ``k`` to try
repeats : integer, optional
How many times to attempt each k (default: 1).
method : str, optional
Which method to use. Must be one of 'AIC' (default) or 'BIC'.
R : random number source, optional
Even you do not pass a value, the result will be deterministic. This is
different from the typical behaviour of ``R``, but, when using jug,
reproducibility is often but, when using jug, reproducibility is often
a desired feature.
kwargs : other options
These are passed transparently to ``kmeans``
Returns
-------
assignments_centroids : jug.Task
jug.Task which is the result of the best (as measured by ``method``)
kmeans clustering.
'''
from milk import kmeans
from milk.utils import get_pyrandom
kmeans = TaskGenerator(kmeans)
if R is not None:
start = get_pyrandom(R).randint(0,1024*1024)
else:
start = 7
results = []
for ki,k in enumerate(ks):
for i in xrange(repeats):
results.append(kmeans(features, k, R=(start+7*repeats*ki+i), **kwargs))
return _select_best(features, results, method)[1]
milk-0.5/milk/ext/__init__.py 0000664 0001750 0001750 00000000264 11723217323 020222 0 ustar luispedro luispedro 0000000 0000000 '''
===============
Milk Extensions
===============
These are modules whose functionality is not really part of the core
functionality of milk, but which are useful with it.
'''
milk-0.5/milk/__init__.py 0000664 0001750 0001750 00000005161 11723217323 017423 0 ustar luispedro luispedro 0000000 0000000 # -*- coding: utf-8 -*-
# Copyright (C) 2008-2011, Luis Pedro Coelho
# vim: set ts=4 sts=4 sw=4 expandtab smartindent:
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
'''
Milk
Machine learning in Python
Toplevel functions
------------------
- nfoldcrossvalidation: n-fold crossvalidation
- defaultclassifier: get a general purpose classifier
- kmeans: kmeans clustering
Modules
-------
- supervised
- unsupervised
- measures
Example
-------
::
features = np.random.randn(100,20)
features[:50] *= 2
labels = np.repeat((0,1), 50)
classifier = milk.defaultclassifier()
model = classifier.train(features, labels)
new_label = model.apply(np.random.randn(100))
new_label2 = model.apply(np.random.randn(100)*2)
'''
try:
from .nfoldcrossvalidation import nfoldcrossvalidation
from .supervised.defaultclassifier import defaultclassifier
from .supervised.defaultlearner import defaultlearner
from .unsupervised.kmeans import kmeans
from milk_version import __version__
except ImportError, e:
import sys
print >>sys.stderr, '''\
Could not import submodules (exact error was: %s).
There are many reasons for this error the most common one is that you have
either not built the packages or have built (using `python setup.py build`) or
installed them (using `python setup.py install`) and then proceeded to test
milk **without changing the current directory**.
Try installing and then changing to another directory before importing milk.
''' % e
__all__ = [
'__version__',
'kmeans',
'defaultclassifier',
'defaultlearner',
'nfoldcrossvalidation',
]
milk-0.5/milk/milk_version.py 0000664 0001750 0001750 00000000024 12045730063 020355 0 ustar luispedro luispedro 0000000 0000000 __version__ = '0.5'
milk-0.5/milk.egg-info/ 0000775 0001750 0001750 00000000000 12045730202 016773 5 ustar luispedro luispedro 0000000 0000000 milk-0.5/milk.egg-info/PKG-INFO 0000664 0001750 0001750 00000014661 12045730202 020100 0 ustar luispedro luispedro 0000000 0000000 Metadata-Version: 1.0
Name: milk
Version: 0.5
Summary: Machine Learning Toolkit
Home-page: http://luispedro.org/software/milk
Author: Luis Pedro Coelho
Author-email: luis@luispedro.org
License: MIT
Description: ==============================
MILK: MACHINE LEARNING TOOLKIT
==============================
Machine Learning in Python
--------------------------
Milk is a machine learning toolkit in Python.
Its focus is on supervised classification with several classifiers available:
SVMs (based on libsvm), k-NN, random forests, decision trees. It also performs
feature selection. These classifiers can be combined in many ways to form
different classification systems.
For unsupervised learning, milk supports k-means clustering and affinity
propagation.
Milk is flexible about its inputs. It optimised for numpy arrays, but can often
handle anything (for example, for SVMs, you can use any dataype and any kernel
and it does the right thing).
There is a strong emphasis on speed and low memory usage. Therefore, most of
the performance sensitive code is in C++. This is behind Python-based
interfaces for convenience.
To learn more, check the docs at `http://packages.python.org/milk/
`_ or the code demos included with the source
at ``milk/demos/``.
Examples
--------
Here is how to test how well you can classify some ``features,labels`` data,
measured by cross-validation::
import numpy as np
import milk
features = np.random.rand(100,10) # 2d array of features: 100 examples of 10 features each
labels = np.zeros(100)
features[50:] += .5
labels[50:] = 1
confusion_matrix, names = milk.nfoldcrossvalidation(features, labels)
print 'Accuracy:', confusion_matrix.trace()/float(confusion_matrix.sum())
If want to use a classifier, you instanciate a *learner object* and call its
``train()`` method::
import numpy as np
import milk
features = np.random.rand(100,10)
labels = np.zeros(100)
features[50:] += .5
labels[50:] = 1
learner = milk.defaultclassifier()
model = learner.train(features, labels)
# Now you can use the model on new examples:
example = np.random.rand(10)
print model.apply(example)
example2 = np.random.rand(10)
example2 += .5
print model.apply(example2)
There are several classification methods in the package, but they all use the
same interface: ``train()`` returns a *model* object, which has an ``apply()``
method to execute on new instances.
Details
-------
License: MIT
Author: Luis Pedro Coelho (with code from LibSVM and scikits.learn)
API Documentation: `http://packages.python.org/milk/ `_
Mailing List: `http://groups.google.com/group/milk-users
`__
Features
--------
- SVMs. Using the libsvm solver with a pythonesque wrapper around it.
- LASSO
- K-means using as little memory as possible. It can cluster millions of
instances efficiently.
- Random forests
- Self organising maps
- Stepwise Discriminant Analysis for feature selection.
- Non-negative matrix factorisation
- Affinity propagation
Recent History
--------------
The ChangeLog file contains a more complete history.
New in 0.5 (05 Nov 2012)
~~~~~~~~~~~~~~~~~~~~~~~~
- Add coordinate-descent based LASSO
- Add unsupervised.center function
- Make zscore work with NaNs (by ignoring them)
- Propagate apply_many calls through transformers
- Much faster SVM classification with means a much faster defaultlearner()
[measured 2.5x speedup on yeast dataset!]
New in 0.4.3 (17 Sept 2012)
~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Add select_n_best & rank_corr to featureselection
- Add Euclidean MDS
- Add tree multi-class strategy
- Fix adaboost with boolean weak learners (issue #6, reported by audy
(Austin Richardson))
- Add ``axis`` arguments to zscore()
New in 0.4.2 (16 Jan 2012)
~~~~~~~~~~~~~~~~~~~~~~~~~~
- Make defaultlearner able to take extra arguments
- Make ctransforms_model a supervised_model (adds apply_many)
- Add expanded argument to defaultlearner
- Fix corner case in SDA
- Fix repeated_kmeans
- Fix parallel gridminimise on Windows
- Add multi_label argument to normaliselabels
- Add multi_label argument to nfoldcrossvalidation.foldgenerator
- Do not fork a process in gridminimise if nprocs == 1 (makes for easier
debugging, at the cost of slightly more complex code).
- Add milk.supervised.multi_label
- Fix ext.jugparallel when features is a Task
- Add milk.measures.bayesian_significance
New in 0.4.1
~~~~~~~~~~~~
- Fix important bug in multi-process gridsearch
New in 0.4.0
~~~~~~~~~~~~
- Use multiprocessing to take advantage of multi core machines (off by
default).
- Add perceptron learner
- Set random seed in random forest learner
- Add warning to milk/__init__.py if import fails
- Add return value to ``gridminimise``
- Set random seed in ``precluster_learner``
- Implemented Error-Correcting Output Codes for reduction of multi-class
to binary (including probability estimation)
- Add ``multi_strategy`` argument to ``defaultlearner()``
- Make the dot kernel in svm much, much, faster
- Make sigmoidal fitting for SVM probability estimates faster
- Fix bug in randomforest (patch by Wei on milk-users mailing list)
For older versions, see ``ChangeLog`` file
Platform: UNKNOWN
milk-0.5/milk.egg-info/dependency_links.txt 0000664 0001750 0001750 00000000001 12045730202 023041 0 ustar luispedro luispedro 0000000 0000000
milk-0.5/milk.egg-info/top_level.txt 0000664 0001750 0001750 00000000005 12045730202 021520 0 ustar luispedro luispedro 0000000 0000000 milk
milk-0.5/milk.egg-info/SOURCES.txt 0000664 0001750 0001750 00000006575 12045730202 020674 0 ustar luispedro luispedro 0000000 0000000 COPYING.MIT
INSTALL.rst
MANIFEST.in
README.rst
setup.py
milk/__init__.py
milk/milk_version.py
milk/nfoldcrossvalidation.py
milk.egg-info/PKG-INFO
milk.egg-info/SOURCES.txt
milk.egg-info/dependency_links.txt
milk.egg-info/top_level.txt
milk/active/__init__.py
milk/active/eimpact.py
milk/active/uncertainty.py
milk/demos/__init__.py
milk/demos/adaboost.py
milk/demos/rf_wine_2d.py
milk/demos/svm-decision-boundary.py
milk/ext/__init__.py
milk/ext/jugparallel.py
milk/measures/__init__.py
milk/measures/cluster_agreement.py
milk/measures/curves.py
milk/measures/measures.py
milk/measures/nfoldcrossvalidation.py
milk/supervised/__init__.py
milk/supervised/_lasso.cpp
milk/supervised/_perceptron.cpp
milk/supervised/_svm.cpp
milk/supervised/_tree.cpp
milk/supervised/adaboost.py
milk/supervised/base.py
milk/supervised/classifier.py
milk/supervised/defaultclassifier.py
milk/supervised/defaultlearner.py
milk/supervised/featureselection.py
milk/supervised/gridsearch.py
milk/supervised/grouped.py
milk/supervised/knn.py
milk/supervised/lasso.py
milk/supervised/logistic.py
milk/supervised/multi.py
milk/supervised/multi_label.py
milk/supervised/multi_view.py
milk/supervised/normalise.py
milk/supervised/parzen.py
milk/supervised/perceptron.py
milk/supervised/precluster.py
milk/supervised/precluster_learner.py
milk/supervised/randomforest.py
milk/supervised/set2binary_array.py
milk/supervised/svm.py
milk/supervised/tree.py
milk/supervised/weighted_voting_adaboost.py
milk/tests/__init__.py
milk/tests/fast_classifier.py
milk/tests/test_adaboost.py
milk/tests/test_affinity.py
milk/tests/test_basic.py
milk/tests/test_curves.py
milk/tests/test_defaultclassifier.py
milk/tests/test_defaultlearner.py
milk/tests/test_ecoc_learner.py
milk/tests/test_ext_jugparallel.py
milk/tests/test_featureselection.py
milk/tests/test_fisher.py
milk/tests/test_gaussianmixture.py
milk/tests/test_gridsearch.py
milk/tests/test_grouped.py
milk/tests/test_kmeans.py
milk/tests/test_knn.py
milk/tests/test_lasso.py
milk/tests/test_logistic.py
milk/tests/test_measures.py
milk/tests/test_measures_clusters.py
milk/tests/test_multi.py
milk/tests/test_multi_label.py
milk/tests/test_multi_view.py
milk/tests/test_nfoldcrossvalidation.py
milk/tests/test_nfoldcrossvalidation_regression.py
milk/tests/test_nnmf.py
milk/tests/test_normalise.py
milk/tests/test_normaliselabels.py
milk/tests/test_parzen.py
milk/tests/test_pca.py
milk/tests/test_pdist.py
milk/tests/test_perceptron.py
milk/tests/test_precluster_learner.py
milk/tests/test_regression.py
milk/tests/test_regression_constant_features.py
milk/tests/test_rf.py
milk/tests/test_set2binary_array.py
milk/tests/test_som.py
milk/tests/test_svm.py
milk/tests/test_svm_sigmoidal.py
milk/tests/test_tree.py
milk/tests/test_utils.py
milk/tests/data/__init__.py
milk/tests/data/jugparallel_jugfile.py
milk/tests/data/jugparallel_kmeans_jugfile.py
milk/tests/data/regression-2-Dec-2009.pp.gz
milk/unsupervised/__init__.py
milk/unsupervised/_kmeans.cpp
milk/unsupervised/_som.cpp
milk/unsupervised/affinity.py
milk/unsupervised/gaussianmixture.py
milk/unsupervised/kmeans.py
milk/unsupervised/normalise.py
milk/unsupervised/parzen.py
milk/unsupervised/pca.py
milk/unsupervised/pdist.py
milk/unsupervised/som.py
milk/unsupervised/nnmf/__init__.py
milk/unsupervised/nnmf/hoyer.py
milk/unsupervised/nnmf/lee_seung.py
milk/utils/__init__.py
milk/utils/parallel.py
milk/utils/utils.py
milk/wrapper/__init__.py
milk/wrapper/wraplibsvm.py milk-0.5/PKG-INFO 0000664 0001750 0001750 00000014661 12045730202 015452 0 ustar luispedro luispedro 0000000 0000000 Metadata-Version: 1.0
Name: milk
Version: 0.5
Summary: Machine Learning Toolkit
Home-page: http://luispedro.org/software/milk
Author: Luis Pedro Coelho
Author-email: luis@luispedro.org
License: MIT
Description: ==============================
MILK: MACHINE LEARNING TOOLKIT
==============================
Machine Learning in Python
--------------------------
Milk is a machine learning toolkit in Python.
Its focus is on supervised classification with several classifiers available:
SVMs (based on libsvm), k-NN, random forests, decision trees. It also performs
feature selection. These classifiers can be combined in many ways to form
different classification systems.
For unsupervised learning, milk supports k-means clustering and affinity
propagation.
Milk is flexible about its inputs. It optimised for numpy arrays, but can often
handle anything (for example, for SVMs, you can use any dataype and any kernel
and it does the right thing).
There is a strong emphasis on speed and low memory usage. Therefore, most of
the performance sensitive code is in C++. This is behind Python-based
interfaces for convenience.
To learn more, check the docs at `http://packages.python.org/milk/
`_ or the code demos included with the source
at ``milk/demos/``.
Examples
--------
Here is how to test how well you can classify some ``features,labels`` data,
measured by cross-validation::
import numpy as np
import milk
features = np.random.rand(100,10) # 2d array of features: 100 examples of 10 features each
labels = np.zeros(100)
features[50:] += .5
labels[50:] = 1
confusion_matrix, names = milk.nfoldcrossvalidation(features, labels)
print 'Accuracy:', confusion_matrix.trace()/float(confusion_matrix.sum())
If want to use a classifier, you instanciate a *learner object* and call its
``train()`` method::
import numpy as np
import milk
features = np.random.rand(100,10)
labels = np.zeros(100)
features[50:] += .5
labels[50:] = 1
learner = milk.defaultclassifier()
model = learner.train(features, labels)
# Now you can use the model on new examples:
example = np.random.rand(10)
print model.apply(example)
example2 = np.random.rand(10)
example2 += .5
print model.apply(example2)
There are several classification methods in the package, but they all use the
same interface: ``train()`` returns a *model* object, which has an ``apply()``
method to execute on new instances.
Details
-------
License: MIT
Author: Luis Pedro Coelho (with code from LibSVM and scikits.learn)
API Documentation: `http://packages.python.org/milk/ `_
Mailing List: `http://groups.google.com/group/milk-users
`__
Features
--------
- SVMs. Using the libsvm solver with a pythonesque wrapper around it.
- LASSO
- K-means using as little memory as possible. It can cluster millions of
instances efficiently.
- Random forests
- Self organising maps
- Stepwise Discriminant Analysis for feature selection.
- Non-negative matrix factorisation
- Affinity propagation
Recent History
--------------
The ChangeLog file contains a more complete history.
New in 0.5 (05 Nov 2012)
~~~~~~~~~~~~~~~~~~~~~~~~
- Add coordinate-descent based LASSO
- Add unsupervised.center function
- Make zscore work with NaNs (by ignoring them)
- Propagate apply_many calls through transformers
- Much faster SVM classification with means a much faster defaultlearner()
[measured 2.5x speedup on yeast dataset!]
New in 0.4.3 (17 Sept 2012)
~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Add select_n_best & rank_corr to featureselection
- Add Euclidean MDS
- Add tree multi-class strategy
- Fix adaboost with boolean weak learners (issue #6, reported by audy
(Austin Richardson))
- Add ``axis`` arguments to zscore()
New in 0.4.2 (16 Jan 2012)
~~~~~~~~~~~~~~~~~~~~~~~~~~
- Make defaultlearner able to take extra arguments
- Make ctransforms_model a supervised_model (adds apply_many)
- Add expanded argument to defaultlearner
- Fix corner case in SDA
- Fix repeated_kmeans
- Fix parallel gridminimise on Windows
- Add multi_label argument to normaliselabels
- Add multi_label argument to nfoldcrossvalidation.foldgenerator
- Do not fork a process in gridminimise if nprocs == 1 (makes for easier
debugging, at the cost of slightly more complex code).
- Add milk.supervised.multi_label
- Fix ext.jugparallel when features is a Task
- Add milk.measures.bayesian_significance
New in 0.4.1
~~~~~~~~~~~~
- Fix important bug in multi-process gridsearch
New in 0.4.0
~~~~~~~~~~~~
- Use multiprocessing to take advantage of multi core machines (off by
default).
- Add perceptron learner
- Set random seed in random forest learner
- Add warning to milk/__init__.py if import fails
- Add return value to ``gridminimise``
- Set random seed in ``precluster_learner``
- Implemented Error-Correcting Output Codes for reduction of multi-class
to binary (including probability estimation)
- Add ``multi_strategy`` argument to ``defaultlearner()``
- Make the dot kernel in svm much, much, faster
- Make sigmoidal fitting for SVM probability estimates faster
- Fix bug in randomforest (patch by Wei on milk-users mailing list)
For older versions, see ``ChangeLog`` file
Platform: UNKNOWN
milk-0.5/COPYING.MIT 0000664 0001750 0001750 00000002077 11723217323 016044 0 ustar luispedro luispedro 0000000 0000000 Copyright (c) 2008-2011 Luis Pedro Coelho
Permission is hereby granted, free of charge, to any person
obtaining a copy of this software and associated documentation
files (the "Software"), to deal in the Software without
restriction, including without limitation the rights to use,
copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the
Software is furnished to do so, subject to the following
conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.
milk-0.5/setup.cfg 0000664 0001750 0001750 00000000073 12045730202 016166 0 ustar luispedro luispedro 0000000 0000000 [egg_info]
tag_build =
tag_date = 0
tag_svn_revision = 0
milk-0.5/INSTALL.rst 0000664 0001750 0001750 00000000571 12045024232 016207 0 ustar luispedro luispedro 0000000 0000000 =============
Building milk
=============
To install dependencies in Ubuntu::
sudo apt-get install python-numpy python-scipy libeigen3-dev
The following should work::
python setup.py install
A C++ compiler is required. On Windows, you might need to specify the compiler.
For example::
python setup.py install --compiler=mingw32
If you have mingw installed.