SHRIPHANI PALAKODETY: Posts tagged 'dimension-reduction'

The Smallest Eigenvalues of a Graph Laplacian

Tue, 07 Apr 2015 02:54:43 UT

Given a graph $ G = (V, E) $, its adjacency matrix $ A $ contains an entry at $ A_{ij} $ if vertices $ i $ and $ j $ have an edge between them. The degree matrix $ D $ contains the degree of each vertex along its diagonal.

The graph laplacian of $ G $ is given by $ D - A $.

Several popular techniques leverage the information contained in this matrix. This blog post focuses on the two smallest eigenvalues.

First, we look at the eigenvalue 0 and its eigenvectors. A very elegant result about its multiplicity forms the foundation of spectral clustering.

Then we look at the second smallest eigenvalue and the corresponding eigenvector. A slightly more involved result (YMMV) allows us to partition the graph in question. A recent publication by John Urschel (who apparently moonlights as a sportsperson) focused on this quantity.

The insights provided here sacrifice some rigor for the sake of brevity. I find such descriptions help me study without getting bogged down too much with details. A bibliography provided at the end contains links to actual proofs.

Eigenvalue 0

The Insight

The zero-th eigenvalue tells us whether the graph is connected or not.

In particular, if a graph has $ k $ connected components, then eigenvalue 0 has multiplicity $ k $ (i.e. $ k $ distinct non-trivial eigenvectors).

A blueprint for the proof looks like this (detailed proof provided later):

The eigenvector corresponding to eigenvalue 0 (known hereafter as $ \lambda_0 $) must contain some non-zero entries - (this is established by showing that $ L $ is positive semi-definite (psd)).
In fact, if vertices $ i $ and $ j $ are connected, then components $ i $ and $ j $ in $ \lambda_0 $ must be equal.
If the graph is connected, apply transitive property and you get an eigenvector where all the components are equal (or all components set to 1).

This looks like so:

If the graph is not connected, then consider each connected component separately and run this procedure on it again. For $ k $ connected portions of the graph, we should have $ k $ distinct eigenvectors, each of which contains a distinct, disjoint set of components set to 1.

So, if the graph has 2 connected components, then the eigenvalue 0 has 2 non-trivial eigenvectors:

In the diagram above, the vertices 1, 2, and 3 form one connected component and vertices 0, 4, and 5 form the other component.

A toy example illustrates this nicely.

In this example, we have a graph with 6 vertices. Say the graph is connected.

Let us see what the value of the desired eigenvector is $ < 1, 1, 1, 1, 1, 1 > $.

For 2 connected components, this value possibly is $ < 1, 1, 1, 0, 0, 0 > and < 0, 0, 0, 1, 1, 1 > $

And so on.

From Eigenvectors to Clustering

The gist of the previous section is:

(i) The multiplicity of eigenvalue 0 equals the number of connected components. (ii) If vertices $ i $ and $ j $ are connected, then in a certain eigenvector corresponding to the eigenvalue zero, the $ i^{th} $ and $ j^{th} $ components are set to 1.

The next task is to use these insights for clustering a set of points.

A simple strategy seems to be (i) constructing a graph from this point set, (ii) ensuring that different clusters become different connected components in this graph, and (iii) look at the eigenvectors of the first eigenvalue for some assistance in discovering these clusters.

The first two steps are trivial. We can build a $ k $-NN graph - a graph where each point in the dataset is a vertex and an edge exists between this point and the $ k $ points closest to it.

So this diagram illustrates this setting. Let us say we have 6 points in the dataset and 2 clusters.

Let us construct a $ k $-NN graph - I’m going to set $ k \leftarrow 2 $ (this is for convenience). The $ k $-NN graph looks like:

Thus there are 2 connected components.

The insight from the previous section tells us that the eigenvalue 0 has multiplicity 2 and the 2 distinct eigenvectors look like:

Consider a matrix with these eigenvectors as its columns:

$$ \begin{bmatrix} 1 & 0\\ 1 & 0\\ 1 & 0\\ 0 & 1\\ 0 & 1\\ 0 & 1\\ \end{bmatrix} $$

This matrix has as many rows as the original dataset.

It also has the effect of causing the clusters in the graph to pop out.

In this example, I can tell that the first three points belong in the same cluster. The next three points form the second cluster.

If you supply this matrix to any classic clustering algorithm (say $ k $-means), it should have no issues clustering this and assigning points to the correct clusters.

This is exactly what spectral clustering does.

Thus the steps involved are:

Construct a $ k $-NN graph (or indeed any other graph - say one that uses a threshold test on euclidean distances).
Obtain the laplacian of this graph.
Obtain the eigendecomposition of the laplacian, retain the first $ k $ columns of the eigenvector matrix.
Supply this matrix to $ k $-means (or your favorite clustering algorithm).

Spectral clustering deals well with non-convex cluster shapes because of the underlying graph constructed. The manifold considered as a result captures the shape of the clusters reasonably well - something we cannot accomplish if only euclidean distances are used:

This is a neat trick exploited by the isomap algorithm which was covered in a previous post.

Constructing the graph tends to be a bit involved - often there isn’t a clear way to build one. The performance of spectral clustering depends on how the connected components in the graph reflect clusters in the dataset.

A connected graph (which you can produce quite easily by picking a large $ k $) will yield poor results.

Despite these issues, spectral clustering is a very powerful and well-studied technique and belongs in any practitioner’s toolbox (IMO).

Second Smallest Eigenvalue of the Laplacian

For the sake of brevity, I will call this quantity $ \lambda_1 $. I will call the associated eigenvector $ v_1 $.

M. Fiedler in his landmark monograph called this quantity the algebraic connectivity of a graph. $ \lambda_1 $ and its eigenvectors provide amazing insights.

One of the insights is:

If $ \lambda_1 = 0 $ clearly eigenvalue 0 has multiplicity greater than 1. Thus the graph is not connected.

This is fairly trivial to establish - the insight from the previous section covers it.

The next insight, my favorite, involves partitioning a graph.

When we partition a graph (into say 2 partitions), we desire 2 reasonably large groups of vertices with very few edges between them.

Observe that this exercise is a waste of time if the graph isn’t connected (there are already two distinct components with no edges between them). Thus it makes sense to only consider connected graphs.

Let us try to give a formal shape to the partitioning problem.

Partitioning can be defined as assigning a value of $ +1 $ or $ –1 $ to each vertex. Vertices with different assignments are in different partitions.

Say vertex $ v_i $ gets assigned value $ x_i $.

Let us assume a perfect paritioning. Exactly $ |V|/2 $ points are assigned and $ x_i = +1 $ and the other half assigned $ x_i = –1 $.

Now, a pair of vertices $ v_i $ and $ v_j $ that belong to different partitions are assigned values $ x_i $ and $ x_j $ where $ x_i \neq x_j $. Thus, the only possible value for $ (x_i - x_j)^{2} $ is $ 4 $.

For each edge from one partition to the other, we have a value of 4. Thus, the number of edges from one partition to the other is given by:

$$ \sum_{i=1, j=i}^{i=n, j=n} \frac{(x_i - x_j)^2}{4} $$

Also, assuming a perfect paritioning, an equal number of vertices are assigned values $ +1 $ and $ –1 $. Thus we have:

$$ \sum_{i} x_i = 0 $$

Our objective is to minimize the number of edges from one partition to the other while achieving a reasonable size for each partition.

This can be expressed as the following optimization function:

Minimize

$$ \sum_{i=1, j=i}^{i=n, j=n} \frac{(x_i - x_j)^2}{4} $$

with the constraint

$$ \sum_{i} x_i = 0 $$

This unfortunately has a trivial solution. Set all $ x_i $ to 0.

An additional constraint eliminates this problem. The new constraint is:

$$ \sum_{i} {x_i}^{2} = |V| $$

We work with matrix variants of these equations. Clearly, the component responsible for the number of edges looks like $ \frac{x^T \mathcal{L} x}{4} $.

The constraint that enforces reasonable partition sizes is $ x^T\mathcal{1} = 0$. Here $ \mathcal{1} $ is a vector of all ones.

Finally, the term responsible for ensuring a non-trivial solution is $ x^Tx = |V| $.

And the lagrangian looks like:

$$ \nabla \frac{x^T \mathcal{L} x}{4} - \nabla \eta_1 (x^Tx - |V|) - \nabla \eta_2 (x^Te) = 0 $$

which becomes:

$$ \mathcal{L} x - \eta_1 x - \eta_2 \mathcal{1} = 0 $$

Multiply by $ \mathcal{1}^T $ on both sides:

$$ \mathcal{1}^T \mathcal{L} x - \eta_1 \mathcal{1}^T x - \eta_2 \mathcal{1}^T \mathcal{1} = 0 $$

This essentially becomes:

$$ \mathcal{L}x - \eta_1x = 0 $$

Thus $ x $ is clearly an eigenvector of the graph laplacian and $ \eta_1 $ is an eigenvalue.

Obviously, the eigenvector of the eigenvalue 0 doesn’t work (it assigns the value 1 to all points).

Clearly $ v_1 $ (the eigenvector of the second smallest eigenvalue) is a solution (the intuition is that the smaller the eigenvalue, the fewer the edges between the two partitions).

Thus, the eigenvector $ v_1 $ (a.k.a the Fiedler vector) provides an assignment to each vertex in the graph. This assignment can be used to partition the graph.

There is just one issue here. The eigenvector contains real values, not necessarily $ 1 $ and $ –1 $.

A whole host of tricks can be applied to convert the entries in the eigenvector to $ +1 $ and $ –1 $:

$ sgn(v_{1_{i}}) $ i.e. vertex $ i $ gets assigned a value $ x_i = $ the sign of the $ i^{th} $ component of the eigenvector $ v_1 $.
$ x_i = +1 $ if $ v_{1_i} > m $, $ -1 $ otherwise. Here $ m $ is the median of all the eigenvalue components (or the mean or zero or whatever).

And this is how the Fiedler vector helps with graph partitioning.

The bibliography attached to this post contains some amazing literature that I had a lot of fun reading. I have mirrored these documents in github and provided the github link.

Bibliography

A Tutorial on Spectral Clustering - Ulrike von Luxburg - A self-contained and elaborate tutorial on spectral clustering.
Algebraic Connectivity of Graphs - Miroslav Fiedler - A landmark paper on the properties of the second smallest eigenvalue and its associated eigenvector
Partitioning Sparse Matrices with Eigenvectors of Graphs - Alex Pothen, Horst Simon, Kang-Pu Paul Liu - An algorithm to leverage the Fiedler vector for graph Partitioning

Dimension Analysis: A Recap

Thu, 22 Jan 2015 03:31:29 UT

In the past few blog posts, I covered some details of popular dimension-reduction techniques and showed some common themes. In this post, I will collect all these materials and tie them together.

Dimension?

The best definition I’ve seen for the topic comes from Benoit Mandelbrot’s work on fractal geometry. The fractal dimension is associated with the ability of a pattern to fill space. Here’s a good example to illustrate what we mean.

Consider a curve viewed at three different scales (image stolen from Chris Burges’s document on dimension reduction):

Now, at a microscopic level, we begin observing. How do we observe? Assume that a there’s a sphere around the observer. Now, let this sphere expand a bit. At the microscopic level, your sphere encounters more of the curve’s material in 2 dimensions. This is illustrated in the rightmost figure.

Now, at a slightly different scale, when our sphere expands, we observe more material along just 1 dimension. This is illustrated in the middle figure.

On a scale like the one in the leftmost pic, we encounter no material at all. This is akin to a zero-dimension figure (a point).

An intuitive explanation of why scale matters is provided in this Wikipedia example. Using a ruler of different lengths, we obtain different measures for the coastline of Great Britain. At various levels of scale, we acquire various measures of the coastline - using a ruler that is as long as the diameter of the earth, the coastline of britain is a negligible fraction of our instrument.

There’s a neat formula that can be used to estimate the fractal dimension of a dataset:

$ n $: The number of pairs of points in our data.
$ r $: The radius of a sphere centered around the observer.
$ p $: The number of pairs of points in a sphere of radius $ r $.

The estimate of the fractal dimension is given by the slope of $ \log(p) $ vs $ \log(r) $.

For the curve in the example above, this value is some real number between 1 and 2 (so the points on the curve have more freedom than those on a line but less freedom than those on a 2D-plane).

A First Stab at Dimension Reduction

Working with intuitions we developed in the first section, we can develop a greedy algorithm:

Estimate the fractal-dimension of the dataset.
Choose a dimension (column) to drop, drop it and recompute the fractal-dimension. If the dimension doesn’t change too much (stays within a certain tolerance), consider this dimension dropped.
Repeat until no more dimensions can be dropped without significantly altering the fractal-dimension.

This is the Grassberger-Procaccia algorithm.

It is intuitive to grasp.

However, in a high-dimension setting, our technique for estimating fractal dimensions falls apart.

In a high-dimensional setting, pairwise distances between the points in a dataset are tightly clustered about a mean. Essentially, the points seem to be equidistant from each other. A Hoeffding bound is provided in this blog post that illustrates this point.

The PCA

One common attempt at reducing dimensions is capturing directions of maximum variance. The PCA projects points in the dataset along the eigenvectors of the covariance matrix. Since this technique is well-known, I’ll just point to this Wikipedia article.

From Proximities to Datasets

A family of techniques I like a lot operate on proximity matrices. A proximity matrix is a symmetric matrix containing similarity scores between the points in a dataset (thus this matrix contains $ n $ rows and $ n $ columns where $ n $ is the number of points in the dataset).

A simple argument demonstrates that proximity matrices are gram matrices (a gram matrix is a close cousin of the covariance matrix). One can retrieve a collection of points for a given gram matrix - see this blog post for a proof.

This family of techniques formulates the dimension-reduction problem as such: “Find a configuration of points in a lower-dimensional place that preserves the proximities in the proximity matrix”.

The standard MDS algorithm uses euclidean distances between points to populate the proximity matrix. This blog post contains more info about this algorithm.

A variant of this algorithm uses path-weights in a $k$-NN graph. This is the Isomap algorithm - covered in this post.

The Kernel Trick

The Kernel trick is leveraged in settings where we transform our points to a higher-dimensional space to make the desired insight pop out. This desired insight is a hyperplane to separate two different classes when working with a classifier. In Kernel PCA, the desired insight is capturing variances so you can run a PCA on the newer dataset in a higher-dimension.

Interestingly, MDS and Isomap are all variants of the Kernel PCA - a topic explored in this blog post.

Up Next

In future blog posts, I will discuss scaling issues with spectral algorithms, insights that can be transferred to other domains and so on.

Multidimensional Scaling and PCA are the Same Thing

Fri, 16 Jan 2015 02:13:37 UT

There is a very simple argument that shows that MDS and PCA achieve the same results.

This argument has 2 important components. The first of these shows that an eigendecomposition of a gram matrix can be used for dimension-reduction.

PCA leverages the singular value decomposition (SVD). Given a matrix $ X $, the SVD is $ X = USV^{T} $.

We drop columns from X by using $ US_{t} $ where we drop some rows and columns from S.

This is also conviently obtained using an eigendecomposition of the covariance matrix.

Working with the gram matrix, we have $ XX^{T} $ and when expressed in terms of $ U $, $ S $ and $ V $, we have $ XX^{T} $ = $ (USV^{T})(VSU^{T}) $.

Simple algebra tells us that this is equal to $ US^{2}U^{T} $. The spectral theorem tells us that this the eigendecomposition of the gram matrix will return this decomposition. $ U $ and $ S $ can be retrieved and a dataset with fewer dimensions can be obtained.

The second part of the argument involves proving that a matrix of distances is indeed a gram matrix. This argument was discussed in a previous post.

Implementing Truncated Matrix Decompositions for Core.Matrix

Fri, 28 Nov 2014 11:16:15 UT

Eigendecompositions and Singular Value Decompositions appear in a variety of settings in machine learning and data mining. The eigendecomposition looks like so:

$$ \mathbf{A}=\mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^{-1} $$

$ \mathbf{Q} $ contains the eigenvectors of $ \mathbf{A} $ and $ \mathbf{\Lambda} $ is a diagonal matrix containing the eigenvalues.

The singular value decomposition looks like:

$$ \mathbf{A} = \mathbf{U} \boldsymbol{\Sigma} \mathbf{V}^* $$

$ \mathbf{U} $ contains the eigenvectors of the covariance matrix $ \mathbf{A}\mathbf{A^T} $. $ \mathbf{V} $ contains the eigenvectors of the gram matrix $ \mathbf{A^T}\mathbf{A} $.

The truncated variants of these decompositions allow us to compute only a few eigenvalues(vectors) or singular values (vectors).

This is important since (i) a lot of times, the smaller eigenvalues are discarded, and (ii) you don’t want to compute the entire decomposition and retain only a few of the rows and columns of the computed matrices each time.

For core.matrix, I implemented these truncated decompositions in Kublai. Details below.

For large matrices , on consumer grade hardware (like your laptop), it is near impossible to compute a full decomposition. If your algorithm performs a decomposition per iteration, then things get worse.

The excellent ARPACK library implements an efficient truncated SVD that is leveraged by several popular numerical libraries like the popular Python library scikit learn and Apache Spark.

ARPACK’s eigendecomposition is not tied to any particular matrix library. You only need to supply a routine that multiplies a vector with the input matrix (which ARPACK uses for its power-iteration step).

Clojure’s budding matrix library core.matrix implements both the eigen and singular value decompositions but doesn’t contain a truncated version.

In this post, I will describe how I used the ARPACK library to implement these truncated decompositions for core.matrix. The resulting implementation is independent of the particular implementation of core.matrix being used.

While the details of the post are tied to core.matrix, the lessons can be transferred to other numerical libraries in other languages.

The excellent netlib library provides a clean java interface to the ARPACK library (which is implemented in FORTRAN).

I will assume that you have ARPACK installed on your machine. If so, netlib will be able to invoke ARPACK.

Truncated Eigendecomposition:

There are a symmetric eigendecomposition that I have currently implemented (and a non-symmetric version is on the way).

Invoking the truncated eigendecompositon routine is trival:

Call the DSAUPD routine (for symmetric matrices. Use DNEUPD for the opposite) till the IDO flag is set to 99.
Once this is done, the eigen vectors and values can be retrieved by a call to DSEUPD. The eigenpairs are returned in ascending order of eigenvalues.

And that’s it. There are a few flags that tell you if there are fatal errors which you need to check that the docs for DSAUPD and DSEUPD contain.

Truncated Singular Value Decomposition:

The truncated SVD can just invoke the eigendecomposition on the gram and covariance matrices. No ARPACK calls are needed here.

The implementation for both the decompositions is available in this github repository.

Usage

This module can be used in the following fashion:

For computing symmetric eigendecompositions:

user> (def M (matrix [[1 2 3 4]
                   [2 5 6 7]
                   [3 6 8 9]
                   [4 7 9 10]]))
#'user/M
user> (use 'kublai.core :reload-all)
nil
user> (eigs M 2 :symmetric) ;; compute 2 eigenvectors for this matrix
{:Q [[-0.22593827269074584 -0.4432218615090191 -0.5727878807113498 -0.6514754961809404] [0.7253136654558885 0.3184697313242928 0.1424607347013554 -0.5934661371986827]], :A [[24.06253512439672 0.0] [0.0 -0.8054849155764637]]}
user>

For computing a truncated SVD:

user> (def M (matrix [[1 2 3 4]
                   [5 6 7 8]
                   [9 10 11 12]
                   [13 14 15 16]]))
#'user/M
user> (svd M 2)
{:U [[-0.13472212372225584 0.8257420598345273] [-0.3407576960799602 0.4288172018031381] [-0.5467932684376645 0.03189234377176592] [-0.7528288407953688 -0.365032514259624]], :V* [[0.4284123959267892 0.4743725155726848 0.5203326352185806 0.5662927548644766] [0.7186534763126667 0.27380780936493887 -0.17103785758268963 -0.6158835245304229]], :S [[38.62265683187287 0.0] [0.0 2.0713230668787377]]}
user> (clojure.pprint/pprint (svd M 2))
{:U
 [[-0.13472212372225592 -0.825742059834525]
  [-0.34075769607996026 -0.42881720180314464]
  [-0.5467932684376648 -0.03189234377175876]
  [-0.7528288407953688 0.3650325142596215]],
 :V*
 [[-0.4284123959267895
   -0.4743725155726852
   -0.5203326352185804
   -0.5662927548644764]
  [-0.7186534763126535
   -0.27380780936497917
   0.1710378575827312
   0.615883524530409]],
 :S [[38.62265683187287 0.0] [0.0 2.0713230668787403]]}
nil

Efficiency

In this example I shall demonstrate how valuable a truncated SVD is. Say we have a very large matrix and we only need 10 singular values/vectors. We will be using the vectorz-clj implementation of core.matrix.

(let [M1 (reshape (matrix (range 500000)) [10000 50])
        M2 (reshape (matrix (range 5000000)) [10000 500])
        M3 (reshape (matrix (range 5000000)) [1000 5000])]
    (time (kublai/svd M1 10))
    (time (kublai/svd M2 10))
    (time (kublai/svd M3 10)))

Here, M1 is a 10000 x 50 matrix, M2 is a 10000 x 500 matrix, and M3 is a 1000 x 5000 matrix.

And the results are:

"Elapsed time: 17372.943 msecs"
"Elapsed time: 78085.404 msecs"
"Elapsed time: 41511.266 msecs"

Now say we had to run a full decomposition on these matrices (using the standard stuff that ships with core.matrix), the results look like:

"Elapsed time: 19617.777 msecs"
"Elapsed time: 157861.467 msecs"
"Elapsed time: 97627.002 msecs"

For the larger matrices this is nearly twice as long as the truncated versions.

(Experiments on a Macbook Air with 8 GB of memory and a 1.8 Ghz i5).

The Isomap Algorithm

Wed, 12 Nov 2014 21:24:54 UT

This is part of a series on a family of dimension-reduction algorithms called non-linear dimension reduction. The goal here is to reduce the dimensions of a dataset (i.e. discard some columns in your data)

In previous posts, I discussed the MDS algorithm and presented some key ideas. In this post, I will describe how those ideas are leveraged in the Isomap algorithm. A clojure implementation based on core.matrix is also included.

Intuition

Isomap uses the same core ideas as the MDS algorithm:

Obtain a matrix of proximities (distances between points in a dataset).
This distance matrix is a matrix of inner products.
An eigendecomposition of this matrix gives us the lower dimension embedding.

Isomap differs from MDS in one vital way - the construction of the distance matrix. In MDS, the distance between two points is just the euclidean distance.

In Isomap, the distances between points are the weight of the shortest path in a point-graph.

The point graph is constructed by placing an edge between two points if the euclidean distance between them falls under a certain threshold or between a point and its top $ k $ neighbors.

This distance matrix captures the underlying manifold more accurately than one constructed using euclidean distances. The following toy example demonstrates this:

The data shown here looks like a swirl that starts at point 1 and ends at point 9. We would like to recover this phenomenon in our lower-dimension embedding.

The first step is to build a distance matrix. Say we use euclidean distances between two points as the corresponding entry in the distance matrix.

In this figure, it is clear that euclidean_distance(1, 6) = euclidean_distance(1, 8) and euclidean_distance(1, 5) = euclidean_distance(1, 9).

Clearly the distances computed here miss the “swirl” in the data entirely. Working with the point graph mentioned above helps us get around this problem.

Let us build a point graph by adding an edge between a node and its nearest neighbor (so $ 1-NN $). The weight on the edge is the euclidean distance between the nodes. The distance between two points is the weight of the shortest path between these points. The point graph is shown below:

Observe that when we use this newer distance metric, distance(1, 6) is indeed less than distance(1, 8) and distance(1, 5) is indeed less than distance(1, 9)

This distance function is clearly doing a better job of capturing the “swirl” in the data.

The Isomap algorithm uses a distance matrix constructed like this in place of one constructed with euclidean distances. This distance matrix is then plugged into the MDS framework and an eigendecomposition is run on the double-centered matrix.

Implementation

Let us do a clojure implementation.

We have a point-set as a core.matrix matrix. First, we compute the point-graph. I am going to place edges between a point and its 3 nearest neighbors (so $ 3-NN $). This routines expects a map of the type {point-index point-vector, …}

(defn build-point-graph
  "A point graph is a k-NN graph. Edges between
   a point and its 3 nearest neigbors"
  ([indexed-points]
     (build-point-graph indexed-points 3))

  ([indexed-points num-neighbors]
     (reduce
      (fn [acc pt]
        (let [other-points (filter
                            (fn [x]
                              (not= (first x)
                                    (first pt)))
                            indexed-points)]
          (merge
           acc
           {(first pt) (map
                        first
                        (take num-neighbors
                              (sort-by
                               #(distance (first pt)
                                          (first %))
                               other-points)))})))
      {}
      indexed-points)))

Then a simple Floyd Warshall algorithm implementation that computes the weights on the shortest paths. It takes the graph built in the previous step and the original indexed points and builds the graph.

(defn floyd-warshall-distance
  "Expected graph representation:
    {V -> neighboring-points}"
  [a-graph indexed-points]
  (let [indexed-points-dict (into {} indexed-points)
        edges    (reduce
                  (fn [acc [x neighbors]]
                    (concat acc (map (fn [n] [x n])
                                     neighbors)))
                  []
                  a-graph)

        inf-matrix (+ Double/POSITIVE_INFINITY
                      (zero-matrix (count indexed-points)
                                   (count indexed-points)))

        zero-diag  (reduce
                    (fn [acc i]
                      (mset acc i i 0))
                    inf-matrix
                    (-> indexed-points count range))

        weights-init (reduce
                      (fn [acc [x y]]
                        (mset acc
                              x
                              y
                              (distance
                               (indexed-points-dict x)
                               (indexed-points-dict y))))
                      zero-diag
                      edges)]

    (reduce
     (fn [old-distances [k i j]]
       (if (< (+ (mget old-distances i k)
                 (mget old-distances k j))
              (mget old-distances i j))
         (mset old-distances
               i
               j
               (+ (mget old-distances i k)
                  (mget old-distances k j)))
         old-distances))
     weights-init
     (for [k (-> indexed-points count range)
           i (-> indexed-points count range)
           j (-> indexed-points count range)]
       [k i j]))))

Once we have a distance matrix, we can simply feed it to MDS:

(defn isomap
  "Takes indexed-points and the target dimension"
  [points n]
  (let [indexed-points (map-indexed (fn [i x] [i x]) points)
        graph (build-point-graph indexed-points)
        distances (floyd-warshall-distance graph indexed-points)]
    (mds/distances->points distances n)))

And that’s it!

Examples

I will use word-vectors from word2vec for these 10 words: river lake city town actor doctor dog cat animal home

The word vectors for these words are available in foo.csv.

Let us reduce these to two dimensions. We get:

ISOMAP Embeddings

The embeddings produced by the MDS algorithm are:

MDS Embeddings

Compared to the plot produced by MDS, we have more separation between terms - for instance cat and dog are place close by but they don’t overlap (unlike the MDS plot). This is a qualitative analysis, it is pretty hard to gauge which embedding is better.

Full Source

See this repo: https://github.com/shriphani/clojure-manifold

Powerful Ideas in Manifold Learning

Sun, 02 Nov 2014 10:10:47 UT

In a previous post, I described the MDS (multidimensional scaling) algorithm. This algorithm operates on a proximity matrix which is a matrix of distances between the points in a dataset. From this matrix, a configuration of points is retrieved in a lower dimension.

The MDS strategy is:

We have a matrix $ D $ for distances between points in the data. This matrix is symmetric.
We express distances as dot-products (using a proof from Schonberg). This means that $ D $ is expressed as $ X^T X $. (Observe that $ X^T X $ is a matrix of dot-products).
Once we have $ X^T X $, dimension reduction is trivial. Running an eigendecomposition on this matrix will produced centered coordinates. The low-dimension embedding is recovered by discarding eigenvalues (eigenvectors).

Thus, assuming that we work with euclidean distances between points, we retrieve an embedding that PCA itself would produce. Thus, MDS with euclidean distances is identical to PCA.

Then what exactly is the value of running MDS on a dataset?

First, the PCA is not the most powerful approach. For certain datasets, euclidean distances do not capture the shape of the underlying manifold. Running the steps of the MDS on a different distance matrix (at least one that doesn’t contain euclidean distances) can lead to better results - a technique that the Isomap algorithm exploits.

Second, the PCA requires a vector-representation for points. In several situations, the objects in the dataset are not points in a metric space (like strings). We can retrieve distances between objects (say edit-distance for strings) and then obtain a vector-representation for the objects using MDS.

In the next blog post, I will describe and implement the Isomap algorithm that leverages the ideas in the MDS strategy. Isomap constructs a distance matrix that attempts to do a better job at recovering the underlying manifold.

PROOFS:

Distances are dot-products: These notes from Peking U are easy to follow. I have mirrored them here in case that link 404s.

A Comment on Dimension-Estimation

Sat, 30 Nov 2013 04:49:33 UT

I saw this neat comment in a paper I was recently reading. If you have all i.i.d features and you want to estimate its dimension using Grassberger-Procaccia (which estimates dimension using a distance-based metric) or want to classify using a k-NN classifier, it is bad if the data points are mostly pairwise equidistant (for instance, a correlation integral plot will look like a step function and thus will be useless; a k-NN classifier will break because the test point ends up equidistant from all the existing points).

There is a trivial argument using the Hoeffding bound in Chris Burges’ paper that suggests that if the features are all i.i.d, a majority of pairwise distances will end up clustered tightly around a mean which means that k-NN or Grassberger-Procaccia won’t work well. I am going to repeat this argument here so I can remember it for later:

Our vectors are of dimension $ d $ and the components are $ \pm1 $. Assuming all the components are $ iid $, the Hoeffding bound gives us:

$$ P(||| x_{1} - x_{2} ||^{2} – 2d| > d\epsilon) = P(| x_{1} \cdot x_{2} | > d\epsilon/2) \le 2exp(-\frac{d\epsilon^2}{8})$$

and this shows us that most pairwise distances will end up clustered very tightly around a mean and this means that a majority of pairs of points in the dataset will end up equidistant and thus a $ k-NN $ classifier will fail.

This also means that the correlation integral is a good way to determine if a k-NN classifier will work well. If the plot resembles a spike, the distance function needs to change.

The correlation-integral is an immensely powerful tool and here’s an implementation