SHRIPHANI PALAKODETY: Posts tagged 'data-science'

The Isomap Algorithm

Wed, 12 Nov 2014 21:24:54 UT

This is part of a series on a family of dimension-reduction algorithms called non-linear dimension reduction. The goal here is to reduce the dimensions of a dataset (i.e. discard some columns in your data)

In previous posts, I discussed the MDS algorithm and presented some key ideas. In this post, I will describe how those ideas are leveraged in the Isomap algorithm. A clojure implementation based on core.matrix is also included.

Intuition

Isomap uses the same core ideas as the MDS algorithm:

Obtain a matrix of proximities (distances between points in a dataset).
This distance matrix is a matrix of inner products.
An eigendecomposition of this matrix gives us the lower dimension embedding.

Isomap differs from MDS in one vital way - the construction of the distance matrix. In MDS, the distance between two points is just the euclidean distance.

In Isomap, the distances between points are the weight of the shortest path in a point-graph.

The point graph is constructed by placing an edge between two points if the euclidean distance between them falls under a certain threshold or between a point and its top $ k $ neighbors.

This distance matrix captures the underlying manifold more accurately than one constructed using euclidean distances. The following toy example demonstrates this:

The data shown here looks like a swirl that starts at point 1 and ends at point 9. We would like to recover this phenomenon in our lower-dimension embedding.

The first step is to build a distance matrix. Say we use euclidean distances between two points as the corresponding entry in the distance matrix.

In this figure, it is clear that euclidean_distance(1, 6) = euclidean_distance(1, 8) and euclidean_distance(1, 5) = euclidean_distance(1, 9).

Clearly the distances computed here miss the “swirl” in the data entirely. Working with the point graph mentioned above helps us get around this problem.

Let us build a point graph by adding an edge between a node and its nearest neighbor (so $ 1-NN $). The weight on the edge is the euclidean distance between the nodes. The distance between two points is the weight of the shortest path between these points. The point graph is shown below:

Observe that when we use this newer distance metric, distance(1, 6) is indeed less than distance(1, 8) and distance(1, 5) is indeed less than distance(1, 9)

This distance function is clearly doing a better job of capturing the “swirl” in the data.

The Isomap algorithm uses a distance matrix constructed like this in place of one constructed with euclidean distances. This distance matrix is then plugged into the MDS framework and an eigendecomposition is run on the double-centered matrix.

Implementation

Let us do a clojure implementation.

We have a point-set as a core.matrix matrix. First, we compute the point-graph. I am going to place edges between a point and its 3 nearest neighbors (so $ 3-NN $). This routines expects a map of the type {point-index point-vector, …}

(defn build-point-graph
  "A point graph is a k-NN graph. Edges between
   a point and its 3 nearest neigbors"
  ([indexed-points]
     (build-point-graph indexed-points 3))

  ([indexed-points num-neighbors]
     (reduce
      (fn [acc pt]
        (let [other-points (filter
                            (fn [x]
                              (not= (first x)
                                    (first pt)))
                            indexed-points)]
          (merge
           acc
           {(first pt) (map
                        first
                        (take num-neighbors
                              (sort-by
                               #(distance (first pt)
                                          (first %))
                               other-points)))})))
      {}
      indexed-points)))

Then a simple Floyd Warshall algorithm implementation that computes the weights on the shortest paths. It takes the graph built in the previous step and the original indexed points and builds the graph.

(defn floyd-warshall-distance
  "Expected graph representation:
    {V -> neighboring-points}"
  [a-graph indexed-points]
  (let [indexed-points-dict (into {} indexed-points)
        edges    (reduce
                  (fn [acc [x neighbors]]
                    (concat acc (map (fn [n] [x n])
                                     neighbors)))
                  []
                  a-graph)

        inf-matrix (+ Double/POSITIVE_INFINITY
                      (zero-matrix (count indexed-points)
                                   (count indexed-points)))

        zero-diag  (reduce
                    (fn [acc i]
                      (mset acc i i 0))
                    inf-matrix
                    (-> indexed-points count range))

        weights-init (reduce
                      (fn [acc [x y]]
                        (mset acc
                              x
                              y
                              (distance
                               (indexed-points-dict x)
                               (indexed-points-dict y))))
                      zero-diag
                      edges)]

    (reduce
     (fn [old-distances [k i j]]
       (if (< (+ (mget old-distances i k)
                 (mget old-distances k j))
              (mget old-distances i j))
         (mset old-distances
               i
               j
               (+ (mget old-distances i k)
                  (mget old-distances k j)))
         old-distances))
     weights-init
     (for [k (-> indexed-points count range)
           i (-> indexed-points count range)
           j (-> indexed-points count range)]
       [k i j]))))

Once we have a distance matrix, we can simply feed it to MDS:

(defn isomap
  "Takes indexed-points and the target dimension"
  [points n]
  (let [indexed-points (map-indexed (fn [i x] [i x]) points)
        graph (build-point-graph indexed-points)
        distances (floyd-warshall-distance graph indexed-points)]
    (mds/distances->points distances n)))

And that’s it!

Examples

I will use word-vectors from word2vec for these 10 words: river lake city town actor doctor dog cat animal home

The word vectors for these words are available in foo.csv.

Let us reduce these to two dimensions. We get:

ISOMAP Embeddings

The embeddings produced by the MDS algorithm are:

MDS Embeddings

Compared to the plot produced by MDS, we have more separation between terms - for instance cat and dog are place close by but they don’t overlap (unlike the MDS plot). This is a qualitative analysis, it is pretty hard to gauge which embedding is better.

Full Source

See this repo: https://github.com/shriphani/clojure-manifold

Powerful Ideas in Manifold Learning

Sun, 02 Nov 2014 10:10:47 UT

In a previous post, I described the MDS (multidimensional scaling) algorithm. This algorithm operates on a proximity matrix which is a matrix of distances between the points in a dataset. From this matrix, a configuration of points is retrieved in a lower dimension.

The MDS strategy is:

We have a matrix $ D $ for distances between points in the data. This matrix is symmetric.
We express distances as dot-products (using a proof from Schonberg). This means that $ D $ is expressed as $ X^T X $. (Observe that $ X^T X $ is a matrix of dot-products).
Once we have $ X^T X $, dimension reduction is trivial. Running an eigendecomposition on this matrix will produced centered coordinates. The low-dimension embedding is recovered by discarding eigenvalues (eigenvectors).

Thus, assuming that we work with euclidean distances between points, we retrieve an embedding that PCA itself would produce. Thus, MDS with euclidean distances is identical to PCA.

Then what exactly is the value of running MDS on a dataset?

First, the PCA is not the most powerful approach. For certain datasets, euclidean distances do not capture the shape of the underlying manifold. Running the steps of the MDS on a different distance matrix (at least one that doesn’t contain euclidean distances) can lead to better results - a technique that the Isomap algorithm exploits.

Second, the PCA requires a vector-representation for points. In several situations, the objects in the dataset are not points in a metric space (like strings). We can retrieve distances between objects (say edit-distance for strings) and then obtain a vector-representation for the objects using MDS.

In the next blog post, I will describe and implement the Isomap algorithm that leverages the ideas in the MDS strategy. Isomap constructs a distance matrix that attempts to do a better job at recovering the underlying manifold.

PROOFS:

Distances are dot-products: These notes from Peking U are easy to follow. I have mirrored them here in case that link 404s.

Low Dimension Embeddings for Visualization

Thu, 30 Oct 2014 03:15:52 UT

Representation learning is a hot area in machine learning. In natural language processing (for example), learning long vectors for words has proven quite effective on several tasks. Often, these representations have several hundred dimensions. To perform a qualitative analysis of the learned representations, it helps to visualize them. Thus, we need a principled approach to drop from this high dimensional space to a lower dimensional space (like $ \mathbb{R}^2 $ for instance).

In this blog post, I will discuss the Multidimensional scaling (MDS) algorithm - a manifold learning algorithm that recovers (or at least tries to recover) the underlying manifold that contains the data.

MDS aims to find a configuration in a lower-dimension space that preserves distances between points in the data. In this post, I will provide intuition for this algorithm and an implementation for clojure (incanter).

MDS works by preserving distances between points in the data. The end result of running this algorithm on a dataset is a spatial arrangement that preserves distances (to the best of its ability).

For example, say you learn a representation for words - like word2vec. These representations are continuous vectors of 300 elements (at least the default vectors are). These representations preserve (or exhibit) some striking linguistic phenomena. For instance, $ R(man) - R(woman) \approx R(king) - R(queen) $. Words that are semantically similar tend to have similar representations.

So how does one go about visualizing phenomena like these? Clearly, you want to be able to view these points in 2 dimensions (on a plot say) and also preserve some of the properties (i.e. similarity of semantically similar words).

MDS operates on pairwise distances between data points. It retrieves an embedding of the data-points in a lower-dimensional space that preserves these distances. Thus, we can visualize the representations learned by using MDS to drop down to 2 dimensions. Ideally the distances should be preserved. Perfect.

Here is an implementation for clojure (using core.matrix - I had to set the implementation to vectorz since svd seems to not be available by default):

(ns clojure-manifold.mds
  "Implementation of MDS"
  (:refer-clojure :exclude [* - + == /])
  (:require [clojure.core.matrix :refer :all]
            [clojure.core.matrix.operators :refer :all]
            [clojure.core.matrix.linear :refer :all]))

(set-current-implementation :vectorz)

(defn double-center
  "Double centering:
   0.5 * (matrix - row-mean - col-mean + grand-mean)"
  [a-matrix]
  (let [row-sum (apply + a-matrix)
        col-sum (->> a-matrix transpose (apply +))

        [num-rows num-cols]  (shape a-matrix)
        
        row-mean (->> (/ row-sum num-rows)
                      (repeat num-rows)
                      matrix)
        col-mean (->> (/ col-sum num-cols)
                      (repeat num-cols)
                      matrix
                      transpose)

        grand-mean (+ (/ (reduce + row-sum)
                         (* num-rows num-cols))
                      (new-matrix num-rows num-cols))]

    (* -0.5
       (-> a-matrix
           (- row-mean)
           (- col-mean)
           (+ grand-mean)))))

(defn proximity-matrix
  "Construct a proximity matrix (of euclidean distances)"
  [a-matrix]
  (map
   (fn [x]
     (map
      (fn [y]
        (Math/pow (distance x y)
                  2))
      a-matrix))
   a-matrix))

(defn mds
  "Take a matrix and find a configuration in n dimensions
   that preserves directions"
  [a-matrix n]
  (let [D (proximity-matrix a-matrix)
        centered-D (double-center D)
        {U :U
         S :S} (svd centered-D)]
    (* (submatrix U 1 [0 n])
       (submatrix S 0 n 0 n))))

I will test these on a few word-vectors from word2vec. The words in question are:

river
lake
city
town
actor
doctor
dog
cat
animal
home

The word vectors for these words are available in foo.csv.

I then ran MDS on this matrix like so:

user> (use 'clojure-manifold.mds :reload)
nil
user> (def M (load-data "foo.csv"))
#'user/M
user> (save-csv (mds M 2) "foo-reduced.csv") ; reduce to 2 dimensions

The plot looks like so:

Looking at this plot, we observe that the resulting configuration places semantically similar words close by. For instance {dog, cat, animal}, {home, actor, doctor}, {lake, river} and {city, town} are groups that are clustered together and the individual clusters are pretty far away from each other.

In a later blog post, I will work through the proof.

The source code for MDS is available in this repo: https://github.com/shriphani/clojure-manifold