{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5b5d5593-2ba6-4d5e-a559-f9627d5a4990",
   "metadata": {},
   "source": [
    "# Topic Modelling using NMF and SVD"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64387628-d3d1-4278-8faa-f765124fe744",
   "metadata": {},
   "source": [
    "## Term Document Matrix\n",
    "- Matrix Factorization\n",
    "- Bag of Words => no word order or sentence structure\n",
    "- Johnson-Lindenstrauss Lemma: (from wikipedia) a small set of points in a high-dimensional space can be embedded into a space of much lower dimension in such a way that distances between the points are nearly preserved.\n",
    "\n",
    "- It is desirable to be able to reduce dimensionality of data in a way that preserves relevant structure. The Johnson–Lindenstrauss lemma is a classic result of this type.\n",
    "\n",
    "## References \n",
    "- https://nbviewer.org/github/fastai/numerical-linear-algebra/blob/master/nbs/2.%20Topic%20Modeling%20with%20NMF%20and%20SVD.ipynb\n",
    "- https://blog.insightdatascience.com/explicit-matrix-factorization-als-sgd-and-all-that-jazz-b00e4d9b21ea"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a1e6b51-93ab-44d3-af09-dfab7bbb0668",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "698683e7-8917-496f-8ad4-1657d813b809",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import scipy as sp \n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from matplotlib import rcParams, animation, rc\n",
    "from sklearn import decomposition\n",
    "import pandas as pd\n",
    "from sklearn.datasets import fetch_20newsgroups\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
    "from ipywidgets.widgets import interact\n",
    "from IPython.display import HTML\n",
    "from fastcore.all import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25f60535-51d4-4d04-8780-71d7259c6a6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "# %matplotlib notebook\n",
    "np.set_printoptions(suppress=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b53864c9-90df-49a3-85fc-79c971644a4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "categories = ['alt.atheism', 'talk.religion.misc', 'comp.graphics', 'sci.space']\n",
    "remove = ('headers', 'footers', 'quotes')\n",
    "newsgroups_train = fetch_20newsgroups(subset='train', categories=categories, remove=remove)\n",
    "newsgroups_test = fetch_20newsgroups(subset='test', categories=categories, remove=remove)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "264dfa1f-f944-4602-9c69-ca28510e6ee3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['data', 'filenames', 'target_names', 'target', 'DESCR'])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "newsgroups_test.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f1f59e2-2e32-4ac7-ae4c-fb52fcad4875",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data 2034\n",
      "filenames 2034\n",
      "target_names 4\n",
      "target 2034\n",
      "DESCR 10617\n"
     ]
    }
   ],
   "source": [
    "for i in newsgroups_test.keys():\n",
    "    print(i, len(newsgroups_train[i]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efb08380-8172-4f91-aacc-eb89600f007b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# newsgroups_train['target_names']\n",
    "newsgroups_train.target_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6927c87c-09a8-47aa-b074-0b9bb3935514",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 2, 3])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.unique(newsgroups_train['target'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23df3fbd-817b-4868-a879-11ef21313e33",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "newsgroups_train.target_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e86c7f7-8afe-442b-85b6-9f0e58a7405e",
   "metadata": {},
   "outputs": [],
   "source": [
    "num_topics, num_top_words = 6, 8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3105986-69de-44a6-82ae-bd25e5ad6fa6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CountVectorizer(stop_words='english')"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorizer = CountVectorizer(stop_words='english')\n",
    "vectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "346ac583-2c80-4bec-a075-5d889c2b98fb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2034, 26576)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectors = vectorizer.fit_transform(newsgroups_train.data).todense()\n",
    "vectors.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "240a422d-6939-4d5a-8bf8-d5789532db4e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda/envs/aiking/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
      "  warnings.warn(msg, category=FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(26576,)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab = np.array(vectorizer.get_feature_names())\n",
    "vocab.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1af85e39-d59a-4314-a801-3fb1e652b92d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['cosmonauts', 'cosmos', 'cosponsored', 'cost', 'costa', 'costar',\n",
       "       'costing', 'costly', 'costruction', 'costs', 'cosy', 'cote',\n",
       "       'couched', 'couldn', 'council', 'councils', 'counsel',\n",
       "       'counselees', 'counselor', 'count'], dtype='<U80')"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab[7000:7020]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a3fbbe1-a834-4f87-aa02-d5243a5986d0",
   "metadata": {},
   "source": [
    "## SVD"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6951172d-9020-4ff8-ba0d-b6787236c53c",
   "metadata": {},
   "source": [
    "- Good words to separate topics - word which appear frequently in one topic doesn't appear that frequently in other topic => Topics are orthogonal\n",
    "\n",
    "- SVD \n",
    "    - Matrix = Orthogonal rows (Embedding of # by the words they cooccur with)\n",
    "              + diagnoal rows( relative importance of each factor) \n",
    "              + orthogonal columns ( embedding of words by # they cooccure with "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ac3f6c9-423a-4acb-98c0-1e553fc2bb9f",
   "metadata": {},
   "source": [
    "- [Facebook Randomized SVD](https://research.fb.com/fast-randomized-svd/)\n",
    "- Exact Decomposition\n",
    "- Applications\n",
    "    - Semanic Analysis\n",
    "    - Collaborative filtering/ recommendations\n",
    "    - Moore-Penrose pseudoinverse\n",
    "    - Data Compression\n",
    "    - Principal Component Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "34361459-9100-4dd5-bfd2-7ef3226a5ac6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 35s, sys: 9.72 s, total: 1min 44s\n",
      "Wall time: 28 s\n"
     ]
    }
   ],
   "source": [
    "%time U, s, Vh = sp.linalg.svd(vectors, full_matrices=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cac64cb-df49-4553-8124-6a30f37ba8e6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((2034, 2034), (2034,), (2034, 26576))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "U.shape, s.shape, Vh.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80ccb560-0d4a-4d90-8a8d-ed92f1ee4e59",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2034, 26576)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(U@Vh).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8aaabcd-fa43-4a3c-9a21-d9ff24fdcc56",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1., -0.,  0., ...,  0.,  0.,  0.],\n",
       "       [-0.,  1.,  0., ...,  0., -0.,  0.],\n",
       "       [ 0.,  0.,  1., ...,  0., -0.,  0.],\n",
       "       ...,\n",
       "       [ 0.,  0.,  0., ...,  1., -0.,  0.],\n",
       "       [ 0., -0., -0., ..., -0.,  1.,  0.],\n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  1.]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "U@U.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "184b2342-ed4b-404f-803a-1c7800bf7b24",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.,  0., -0., ..., -0.,  0.,  0.],\n",
       "       [ 0.,  1., -0., ..., -0., -0.,  0.],\n",
       "       [-0., -0.,  1., ..., -0., -0., -0.],\n",
       "       ...,\n",
       "       [-0., -0., -0., ...,  1., -0., -0.],\n",
       "       [ 0., -0., -0., ..., -0.,  1., -0.],\n",
       "       [ 0.,  0., -0., ..., -0., -0.,  1.]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Vh@Vh.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b74c5209-96d7-4e0a-94a8-94a5795d3eb5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0xaee5b6b9f70>]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAZH0lEQVR4nO3de5Ac51nv8e/TPXuTdnVfy4qkRHaiXFwUsR0lCEhSEB+M40BkLkmZwzkWOT5xUSSQVKCCIXU4UEUVMZyTHFxQBoNdkVOGJCSkrEoZEuM4mFMVX1a+23KstWNFkmVpraulvc7Mwx/9zmzPbq/2orn16Pepmpqet3tmHvWsfv3O2z3d5u6IiEhniVpdgIiI1J/CXUSkAyncRUQ6kMJdRKQDKdxFRDpQodUFAKxbt863bNnS6jJERHJlz549r7n7YNa8tgj3LVu2MDQ01OoyRERyxcz2zzVPwzIiIh1I4S4i0oEU7iIiHUjhLiLSgRTuIiIdSOEuItKBFO4iIh0o1+H+6MvH+cJ3fsBksdzqUkRE2kquw33P/hPc+t1himWFu4hIWq7D3cK9rjciIlIr3+Ee0l3ZLiJSK9/hXu27i4hIWq7DvULXgRURqZXrcNewjIhItlyHe4U67iIitXId7qauu4hIpnyHe7h3pbuISI18h3ul465sFxGpke9wD/fKdhGRWvkOd9Nx7iIiWXId7hU6zl1EpNaCw93MYjN73My+FR5fYmYPm9mwmX3VzLpDe094PBzmb2lQ7TrOXURkDovpuX8K2Jt6fAvwRXd/C3ACuDG03wicCO1fDMs1hE4cJiKSbUHhbmabgA8Bfx8eG/AB4OthkV3AdWF6R3hMmH+VNWpwPLysDoUUEam10J77/wM+C1ROnL4WOOnuxfD4ILAxTG8EDgCE+afC8jXM7CYzGzKzoZGRkSUVX91iKNtFRGrMG+5m9gvAUXffU883dvfb3X2bu28bHBxc0mtozF1EJFthAcv8NPBhM7sW6AVWAH8JrDKzQuidbwIOheUPAZuBg2ZWAFYCx+peOdOn/NWYu4hIrXl77u7+B+6+yd23ANcD33X3XwceAH41LLYTuCdM7w6PCfO/6w06VnG65650FxFJO5/j3H8f+IyZDZOMqd8R2u8A1ob2zwA3n1+Jc9NPmEREsi1kWKbK3b8HfC9MvwS8J2OZceAjdahtEXU1891ERNpfrn+hqh2qIiLZ8h3u1R2qincRkbRchzs65a+ISKZch7t2qIqIZMt3uJuOcxcRyZLvcA/3Os5dRKRWvsNd4zIiIplyHe4VGpYREamV63DXce4iItnyHe46zl1EJFO+w109dxGRTLkO9wp13EVEauU63Kev3qd0FxFJy3e4h3v13EVEauU73HWcu4hIplyHe4U67iIitXId7rqGqohItnyHu66hKiKSKd/hHu7VcxcRqZXvcNfFOkREMuU63Ct9dw3LiIjUynW4q+cuIpIt3+He6gJERNpUrsNdRESy5TrcdQ1VEZFs+Q73cK8dqiIitfId7tqhKiKSqTPCvbVliIi0nXyHuy6zJyKSKdfhjnruIiKZch3uOreMiEi2fIe7rtYhIpIp1+E+TV13EZG0XIe7hmVERLLlO9y1Q1VEJFO+w12X2RMRyTRvuJtZr5k9YmZPmtmzZvYnof0SM3vYzIbN7Ktm1h3ae8Lj4TB/S6OKn/6FqtJdRCRtIT33CeAD7v5O4HLgGjPbDtwCfNHd3wKcAG4My98InAjtXwzLNcT0uWVERCRt3nD3xJnwsCvcHPgA8PXQvgu4LkzvCI8J86+yRh2zqHPLiIhkWtCYu5nFZvYEcBS4D3gROOnuxbDIQWBjmN4IHAAI808BazNe8yYzGzKzoZGRkSUVb7pch4hIpgWFu7uX3P1yYBPwHuDt5/vG7n67u29z922Dg4Pn91oamBERqbGoo2Xc/STwAPCTwCozK4RZm4BDYfoQsBkgzF8JHKtHsTOZBt1FRDIt5GiZQTNbFab7gJ8D9pKE/K+GxXYC94Tp3eExYf53vUGHsyjbRUSyFeZfhA3ALjOLSTYGX3P3b5nZc8BXzOxPgceBO8LydwBfNrNh4DhwfQPqBnSZPRGRucwb7u7+FHBFRvtLJOPvM9vHgY/Upbp5TP9CVekuIpKW81+oJtRzFxGple9w17llREQy5Trc0XHuIiKZch7uCZ1bRkSkVq7DXcMyIiLZ8h3ulQmlu4hIjXyHe+U4d6W7iEiNfId7uNeQu4hIrXyHu075KyKSKd/hXrnMXovrEBFpN/kOdx3mLiKSKdfhXqHj3EVEanVGuLe6ABGRNpPrcNcOVRGRbPkOd12uQ0QkU77DXT13EZFMnRHurS1DRKTt5Dvc0WX2RESy5DvcdZk9EZFM+Q73VhcgItKmch3uFRqWERGpletwr5zyt6x0FxGpketwL0RJuJfKCncRkbRch3scwr2ocBcRqZHrcC/E6rmLiGTJd7hHSfnquYuI1Mp5uIeee6nc4kpERNpLrsM9jjXmLiKSJdfhrqNlRESy5TrcdbSMiEi2fId7+BFTsaRwFxFJy3e4V4Zl9AtVEZEauQ53MyMyKGtYRkSkRq7DHZLeu3ruIiK1ch/ukZl67iIiM+Q+3AuR6WgZEZEZch/uUWQ6zl1EZIZ5w93MNpvZA2b2nJk9a2afCu1rzOw+M9sX7leHdjOzW81s2MyeMrMrG/kPiCPT+dxFRGZYSM+9CPyuu18GbAc+YWaXATcD97v7VuD+8Bjgg8DWcLsJuK3uVacU1HMXEZll3nB398Pu/liYfh3YC2wEdgC7wmK7gOvC9A7gLk88BKwysw31LrwiMoW7iMhMixpzN7MtwBXAw8B6dz8cZr0KrA/TG4EDqacdDG0zX+smMxsys6GRkZHF1l0Vq+cuIjLLgsPdzPqBbwCfdvfT6Xnu7sCiEtbdb3f3be6+bXBwcDFPrRGZjnMXEZlpQeFuZl0kwX63u/9zaD5SGW4J90dD+yFgc+rpm0JbQxRiHecuIjLTQo6WMeAOYK+7fyE1azewM0zvBO5Jtd8QjprZDpxKDd/UXWw6zl1EZKbCApb5aeC/A0+b2ROh7Q+BzwNfM7Mbgf3AR8O8e4FrgWFgFPhYPQueKdKhkCIis8wb7u7+/wGbY/ZVGcs78InzrGvBYh0tIyIyS+5/oZocLdPqKkRE2kuHhLvSXUQkLffhHkWGLsQkIlIr9+Ee62IdIiKz5D7cC1GkHaoiIjPkPtyjCIW7iMgMuQ93XWZPRGS23Ie7zgopIjJb7sNd53MXEZkt9+EeR5HOLSMiMkPuw70r1o+YRERmyn24x5HOCikiMlPuw11j7iIis+U+3OMooqjzD4iI1Mh9uHfFRlFj7iIiNXIf7rpAtojIbLkP94J2qIqIzJL7cNeYu4jIbLkP94LG3EVEZsl/uGvMXURklo4Id425i4jUyn24x1GEu87pLiKSlvtwL8QGoHF3EZGU/Id7lIS7eu4iItNyH+5xVOm5K9xFRCpyH+6VnruOdRcRmZb7cF/WUwDg9NhUiysREWkfuQ/3jav6ADh8arzFlYiItI/ch/tAb9JzPztRbHElIiLtI/fhvjwMy5ydVLiLiFTkPtz7Q7ifUc9dRKQq9+Fe7bkr3EVEqnIf7su6YgDOTJRaXImISPvIfbhHkbF+RQ8/fO1sq0sREWkbuQ93gE2rl3Hi7GSryxARaRsdEe7LumNGdbSMiEjVvOFuZnea2VEzeybVtsbM7jOzfeF+dWg3M7vVzIbN7Ckzu7KRxVck4a4xdxGRioX03L8EXDOj7WbgfnffCtwfHgN8ENgabjcBt9WnzHNb3l3Qce4iIinzhru7Pwgcn9G8A9gVpncB16Xa7/LEQ8AqM9tQp1rn1NcdM6aeu4hI1VLH3Ne7++Ew/SqwPkxvBA6kljsY2mYxs5vMbMjMhkZGRpZYRmJ5T4GzOhRSRKTqvHeoursDiz7frrvf7u7b3H3b4ODgedWwrDtmbKqkC3aIiARLDfcjleGWcH80tB8CNqeW2xTaGmpZd/JDprEp9d5FRGDp4b4b2BmmdwL3pNpvCEfNbAdOpYZvGmZZd3IKAh0OKSKSKMy3gJn9I/AzwDozOwj8b+DzwNfM7EZgP/DRsPi9wLXAMDAKfKwBNc+yvCfpuY9OlGCgGe8oItLe5g13d/+1OWZdlbGsA58436IWq69Lp/0VEUnriF+oVnruOhxSRCTREeFeGXN/Xaf9FREBOiTc37CqF4BXTo61uBIRkfbQEeG+fqCX7kLEj46NtroUEZG20BHhHkXG5tV97Fe4i4gAHRLuABcN9HLs7ESryxARaQsdE+467a+IyLTOCfeegsJdRCTomHBf19/N4VNjjOv8MiIinRPuP/u2ixifKvP9F4+1uhQRkZbrmHD/8U0rAXhx5EyLKxERab2OCfeVfV0M9BT40XEdDiki0jHhbmZsXrNM4S4iQgeFO8A7NqzgkR8ep6wrMonIBa6jwn37pWsYnSzx6Mszr+ctInJh6ahw/8DbLwLgiQMnW1uIiEiLdVS4r+3v4bINK/j6noMk1w0REbkwdVS4A3z8/Zew7+gZdj/5SqtLERFpmY4L9w+/cyPv2LCCW/7leV2ZSUQuWB0X7nFk/K8PvYNXTo3zR/c80+pyRERaouPCHeCn3rKOj7/vEv5pz0EeeP5oq8sREWm6jgx3gN+9+m1svaif37r7Mb797KutLkdEpKk6Ntx7u2Lu/p8/wVsvHuC3/+Fx7nvuSKtLEhFpmo4Nd4CLVvTypd94N5cOLufjdw1x2/debHVJIiJN0dHhDrB6eTe7P/lerr5sPbf86/P82b17GZ0strosEZGG6vhwB+guRNz2397F9e/ezN8++BI/+3++x9cePUBJ56ARkQ51QYQ7JIdIfv5Xfpyv/+ZPsmFlH5/9xlN86Nb/4MEXRlpdmohI3V0w4V6xbcsavvlbP8Vf/dcrODtZ5IY7H+GGOx/h+y8eo1gqt7o8EZG6sHY4B8u2bdt8aGio6e87USzx5e/v59b793F6vMja5d3suHwjH9m2ibdfPICZNb0mEZGFMrM97r4tc96FHO4VZyaKPPjCCN966hXue+4IUyVncKCHqy9bz7vetJrtl65lw8pehb2ItBWF+yIcOzPBd547wn/sG+GB50cYm0rOT7Ouv5t3bFgRbgO8/eIVvHmwn+7CBTeyJSJtQuG+RKWys/fwafbsP8HTh07x/KuneeHIGSaLydh8V2y8ebCfresH2LS6L9yWsWl1HxtX9dHbFbf4XyAinexc4V5odjF5EkfGj21cyY9tXFltK5bKvPTaWfYePs3ew6+z9/Bpnjxwkn995jBTpdoN5drl3Vy8speLV/SyfmUv6wd6uWhFD+tX9HDRQC8XDfSwenk3XbF6/yJSXwr3RSrEEW9dP8Bb1w+w4/Lp9lLZOXJ6nEMnxzh4YpQDx8c4fGqcV0+N8cqpcR4/cJLjZyczX3NFb4G1/T2sXd7NmuXdrO1P7lcv62ZlXxerwn3ltqKvQF9XrH0AIjInhXudxJHxhlV9vGFVH+/esiZzmclimZEzExw9Pc6R0xOMvD7OsbOTHD87mdyfmWT/sVEe+9EJToxOnfNHVl2x0d9ToL+3QH9PFwNhenlPgf6eAgO9yQZgWXdy6+suhPuYZV0xy7oL9HVHSXtX0t5TiLTBEOkQCvcm6i5EbFyVjMfPp1x2Xp8ocmp0ilNjye3k2CSnx4qcHk8en50o8vp4cjszMcXI6xP88LWznJkocma8WN0ZvFBxZPQWIvq6Y3q7krDviiN6ChHdYbq7ENEdR3QVInriGe1hXvq+dp6F+zjMSx73ZLx2d5zcokgbG5GlULi3qSiy6jDMUpXLzthUidHJEmOTJUanitPTkyVGJ4vV6WS5IuNTZcamSoxPlpgolpkslZksTt/OTBST6VT7VGW6VJ613+F8dcVWG/xzbGy6Z22IpjckXQWjJzy3EEcUIqMQG4XIiKOoOl2ZF0dGV5zM6wqPk2Ui4srjyIgiqy4/3R4RmxGH14xselmRZmpIuJvZNcBfAjHw9+7++Ua8j5xbFBnLe5KhmmYpl52pcjr4PQR/icmiz9ooVDYgU6kNRrU9Y96sjUp4PDpaZLLkTBZL4Tk+6zmtZEYS+rM2DBFxRM2Go7qMJRuVOLLqc9Pz0xuUmg1NxsYlDu8Z2/R9HJGaTpZNLxdHVNvi1GvXPKfaxqy29GumXyv9mtXpymvMatNGcanq/r/ezGLgr4GfAw4Cj5rZbnd/rt7vJe0nioyeKKan0F6Hgbo7xbJTKjtTpXK4Tx4Xy2WKpdnzK+2lsjNVdoqhvZx6rWS56enK43K1vUypDKVyOXnsTim8V/V1SqG95rnlzNefLJannxtqm35umXKZWc+t1FIOy+XtfHlxxgZk5sYhythwVjaO6W9P8Yy29LeyQmx0Ve7j5NtfIU6+vRXi2vmV9q44SvZr9RTo70n2ZfWHDtXqZV0t3YfViC7de4Bhd38JwMy+AuwAFO7SMmYWhni44H9/4J4EfCkV+CVPNgLT01Q3RKXqRmF6I1OuaZv7tarzZ73+zPek2lYqz5gfXiP9/tk1J69TSm3cqhu4sDEslsuMF6dft7JRL4YhxWI5uZ8qlavLL3WoccPKXvoX8K35d67ayi++8w1Leo9zaUS4bwQOpB4fBH5i5kJmdhNwE8Ab3/jGBpQhIlnMjNiSHrHMz1PfqiqhPxW+1U2VytX9V2cmSoxOFDk7WeLE2UmePHiS8gJ+JHo++9XOpWU7VN39duB2SH6h2qo6RETOxcJQTiFn3/oa8dPIQ8Dm1ONNoU1ERJqkEeH+KLDVzC4xs27gemB3A95HRETmUPdhGXcvmtkngW+THAp5p7s/W+/3ERGRuTVkzN3d7wXubcRri4jI/HQ6QhGRDqRwFxHpQAp3EZEOpHAXEelAbXGZPTMbAfYv8enrgNfqWE69tGNd7VgTtGddqmnh2rGudqwJ6l/Xm9x9MGtGW4T7+TCzobmuIdhK7VhXO9YE7VmXalq4dqyrHWuC5talYRkRkQ6kcBcR6UCdEO63t7qAObRjXe1YE7RnXapp4dqxrnasCZpYV+7H3EVEZLZO6LmLiMgMCncRkQ6U63A3s2vM7AdmNmxmNzfxfTeb2QNm9pyZPWtmnwrtf2xmh8zsiXC7NvWcPwh1/sDMfr6Btb1sZk+H9x8KbWvM7D4z2xfuV4d2M7NbQ11PmdmVDajnban18YSZnTazT7diXZnZnWZ21MyeSbUtet2Y2c6w/D4z29mAmv7CzJ4P7/tNM1sV2reY2Vhqnf1N6jnvCp/7cKh7yZdZmqOmRX9e9f7/OUddX03V9LKZPRHam7Wu5sqClv5dAcklpPJ4Izmd8IvApUA38CRwWZPeewNwZZgeAF4ALgP+GPi9jOUvC/X1AJeEuuMG1fYysG5G258DN4fpm4FbwvS1wL8ABmwHHm7CZ/Yq8KZWrCvg/cCVwDNLXTfAGuClcL86TK+uc01XA4UwfUuqpi3p5Wa8ziOhTgt1f7DONS3q82rE/8+sumbM/7/AHzV5Xc2VBS39u3L3XPfcqxfidvdJoHIh7oZz98Pu/liYfh3YS3Lt2LnsAL7i7hPu/kNgmKT+ZtkB7ArTu4DrUu13eeIhYJWZbWhgHVcBL7r7uX6N3LB15e4PAscz3m8x6+bngfvc/bi7nwDuA66pZ03u/h13L4aHD5FczWxOoa4V7v6QJ0lxV+rfUZeazmGuz6vu/z/PVVfofX8U+MdzvUYD1tVcWdDSvyvI97BM1oW4zxWwDWFmW4ArgIdD0yfD1607K1/FaG6tDnzHzPZYchFygPXufjhMvwqsb0FdkFyVK/2fr9XrCha/bppd3/8g6elVXGJmj5vZv5vZ+1K1HmxCTYv5vJq9nt4HHHH3fam2pq6rGVnQ8r+rPId7y5lZP/AN4NPufhq4DXgzcDlwmORrYrO9192vBD4IfMLM3p+eGXorTT/+1ZJLLn4Y+KfQ1A7rqkar1s1czOxzQBG4OzQdBt7o7lcAnwH+wcxWNKmctvu8Zvg1ajsOTV1XGVlQ1aq/qzyHe0svxG1mXSQf5t3u/s8A7n7E3UvuXgb+junhhKbV6u6Hwv1R4JuhhiOV4ZZwf7TZdZFsbB5z9yOhvpavq2Cx66Yp9ZnZbwC/APx6CAfC0MexML2HZEz7reH900M3da9pCZ9X0z5HMysAvwx8NVVv09ZVVhbQBn9XeQ73ll2IO4zv3QHsdfcvpNrT49W/BFT26u8GrjezHjO7BNhKslOn3nUtN7OByjTJjrlnwvtX9r7vBO5J1XVD2IO/HTiV+ipZbzU9q1avq5TFrptvA1eb2eowNHF1aKsbM7sG+CzwYXcfTbUPmlkcpi8lWTcvhbpOm9n28Ld5Q+rfUa+aFvt5NfP/538Bnnf36nBLs9bVXFlAO/xdnc/e2FbfSPY8v0CyVf5cE9/3vSRfs54Cngi3a4EvA0+H9t3AhtRzPhfq/AHnsXd+nrouJTkq4Ung2co6AdYC9wP7gH8D1oR2A/461PU0sK1BdS0HjgErU21NX1ckG5fDwBTJmOaNS1k3JOPgw+H2sQbUNEwy/lr52/qbsOyvhM/1CeAx4BdTr7ONJHBfBP6K8OvzOta06M+r3v8/s+oK7V8CfnPGss1aV3NlQUv/rtxdpx8QEelEeR6WERGROSjcRUQ6kMJdRKQDKdxFRDqQwl1EpAMp3EVEOpDCXUSkA/0nv01Kyro8oL4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5be3f66d-fc80-4745-a274-2fc2ff5faa3d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "760445d31fc14892aed5366782bb05c1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=20, description='i', max=20, min=1), Output()), _dom_classes=('widget-in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# @interact(i=(1,s.shape[0]))\n",
    "@interact(i=(1,20))\n",
    "def plot_features(i=20):\n",
    "    plt.plot(s[:i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6a399f5-992a-4dc1-957f-2fff9f0b11be",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.00940972, -0.0114532 , -0.00002169, ..., -0.00000572,\n",
       "        -0.00001144, -0.00109243],\n",
       "       [-0.00356688, -0.01769167, -0.00003045, ..., -0.00000773,\n",
       "        -0.00001546, -0.0018549 ],\n",
       "       [ 0.00094971, -0.02282845, -0.00002339, ..., -0.0000122 ,\n",
       "        -0.0000244 ,  0.00150538],\n",
       "       ...,\n",
       "       [-0.00218087, -0.04322025, -0.00012552, ...,  0.00003759,\n",
       "         0.00007518,  0.00160907],\n",
       "       [-0.00039196,  0.00494894,  0.00000309, ..., -0.00001321,\n",
       "        -0.00002643, -0.00015038],\n",
       "       [ 0.00306552, -0.01437264, -0.00000405, ..., -0.00003597,\n",
       "        -0.00007193,  0.00056218]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_topics, num_top_words = 6, 8\n",
    "\n",
    "def show_topics(v_mat,vocab,n_docs=10, n_top_words=8):\n",
    "    v_docs = v_mat[:n_docs]\n",
    "    return [\" \".join(gw) for gw in vocab[np.argsort(v_docs)[:,-(n_top_words+1):-1]]]\n",
    "\n",
    "n_docs = 10\n",
    "n_top_words=8\n",
    "v_docs = Vh[:n_docs]\n",
    "v_docs.shape\n",
    "v_docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06295650-057c-4514-ae55-2757ade53bc6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([13816, 12642,  8956, ..., 19210,  7181,  8462])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# vocab[np.argsort(v_docs)[0,:][-(n_top_words+1):-1]]\n",
    "np.argsort(v_docs[0,:])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f700645-389b-46c7-9667-a07623fa33da",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['salvadorans imaginative surreal kindergarten galacticentric surname propagandist critus',\n",
       " 'bit format jfif image quality color file gif',\n",
       " 'send ftp ray 3d 128 mail pub edu',\n",
       " 'religious graphics does atheism atheists people matthew god',\n",
       " 'tool display tools available software analysis processing data',\n",
       " 'atheist true argument religion believe religious atheism atheists',\n",
       " 'surface probes missions moon probe mars lunar nasa',\n",
       " 'mariner orbit moon probes mars lunar surface probe',\n",
       " 'false premises argumentum ad true example conclusion fallacy',\n",
       " 'star material nasa physical universe theory image larson']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_topics(Vh, vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8549d4fd-9bb1-430c-9c70-b437ec1ddee7",
   "metadata": {},
   "source": [
    "```{note}\n",
    "We have many tools that allow us to filter matrix exactly.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77cbb141-a1de-4489-a1b2-667da188009d",
   "metadata": {},
   "source": [
    "## Non-negative Matrix Factorization(NMF)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ed947c7-5c09-41f9-9941-eb317ad6e5e7",
   "metadata": {},
   "source": [
    "- NMF - constraining factors to be non-negative instead of orthogonal.\n",
    "- World is inherently non negative.\n",
    "- Applications\n",
    "    - Face Decompositions\n",
    "    - Collaborative Filtering / Movie Recommendation\n",
    "    - Audio Source Seperation\n",
    "    - Chemistry \n",
    "    - Bioinformatics \n",
    "    - Topic Modelling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d451888f-fdfb-46f4-b7d2-9a5e047e49a1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2034, 26576)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m, n = vectors.shape # documents, words\n",
    "m, n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "406e6baa",
   "metadata": {},
   "outputs": [],
   "source": [
    "d = 5 # num of topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12c01dd4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NMF(n_components=5, random_state=1)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = decomposition.NMF(n_components=d, random_state=1)\n",
    "clf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d67e9362",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2034, 26576)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectors.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc2f92b6",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda/envs/aiking/lib/python3.9/site-packages/sklearn/utils/validation.py:585: FutureWarning: np.matrix usage is deprecated in 1.0 and will raise a TypeError in 1.2. Please convert to a numpy array with np.asarray. For more information see: https://numpy.org/doc/stable/reference/generated/numpy.matrix.html\n",
      "  warnings.warn(\n",
      "/opt/anaconda/envs/aiking/lib/python3.9/site-packages/sklearn/decomposition/_nmf.py:289: FutureWarning: The 'init' value, when 'init=None' and n_components is less than n_samples and n_features, will be changed from 'nndsvd' to 'nndsvda' in 1.1 (renaming of 0.26).\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[0.08858936, 0.02984714, 0.        , 0.04220515, 0.        ],\n",
       "       [0.        , 0.00074146, 0.0037713 , 0.02133068, 0.00096886],\n",
       "       [0.        , 0.01650813, 0.00026615, 0.02582674, 0.00294897],\n",
       "       ...,\n",
       "       [0.00897787, 0.03011237, 0.0033981 , 0.01453643, 0.00331353],\n",
       "       [0.01862327, 0.        , 0.00123129, 0.21812662, 0.        ],\n",
       "       [0.        , 0.        , 0.        , 0.        , 0.        ]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "W1 = clf.fit_transform(vectors)\n",
    "W1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f34bb520",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2034, 5)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "W1.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d4a0158",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.12126083, 0.        , 0.        , ..., 0.00003202, 0.00006403,\n",
       "        0.00028996],\n",
       "       [0.11928575, 0.12085267, 0.00016592, ..., 0.0001165 , 0.000233  ,\n",
       "        0.05186412],\n",
       "       [0.05766391, 0.48160644, 0.0007993 , ..., 0.00028762, 0.00057525,\n",
       "        0.        ],\n",
       "       [0.        , 0.14794335, 0.        , ..., 0.00006585, 0.0001317 ,\n",
       "        0.        ],\n",
       "       [0.12742192, 0.19356417, 0.00050472, ..., 0.        , 0.        ,\n",
       "        0.        ]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "H1 = clf.components_\n",
    "H1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cad6278b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5, 26576)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "H1.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6576df4c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['version quality format images color file gif image',\n",
       " '3d send ftp ray 128 mail pub graphics',\n",
       " 'data market year satellites commercial nasa satellite launch',\n",
       " 'just said atheism does atheists matthew people god',\n",
       " 'images analysis edu ftp processing software available data']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_topics(H1, vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecd81fec",
   "metadata": {},
   "source": [
    "## TFIDF"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdbd6572",
   "metadata": {},
   "source": [
    "[Topic Frequency-Inverse Document Frequency (TF-IDF)](http://www.tfidf.com/) is a way to normalize term counts by taking into account how often they appear in a document, how long the document is, and how commmon/rare the term is.\n",
    "\n",
    "TF = (# occurrences of term t in document) / (# of words in documents)\n",
    "\n",
    "IDF = log(# of documents / # documents with term t in it)\n",
    "\n",
    "Example:\n",
    "_Consider a document containing 100 words wherein the word cat appears 3 times. The term frequency (i.e., tf) for cat is then (3 / 100) = 0.03. Now, assume we have 10 million documents and the word cat appears in one thousand of these. Then, the inverse document frequency (i.e., idf) is calculated as log(10,000,000 / 1,000) = 4. Thus, the Tf-idf weight is the product of these quantities: 0.03 * 4 = 0.12_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f98cbe84",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mSignature:\u001b[0m \u001b[0mvectorizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit_transform\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mraw_documents\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "Learn vocabulary and idf, return document-term matrix.\n",
       "\n",
       "This is equivalent to fit followed by transform, but more efficiently\n",
       "implemented.\n",
       "\n",
       "Parameters\n",
       "----------\n",
       "raw_documents : iterable\n",
       "    An iterable which generates either str, unicode or file objects.\n",
       "\n",
       "y : None\n",
       "    This parameter is ignored.\n",
       "\n",
       "Returns\n",
       "-------\n",
       "X : sparse matrix of (n_samples, n_features)\n",
       "    Tf-idf-weighted document-term matrix.\n",
       "\u001b[0;31mFile:\u001b[0m      /opt/anaconda/envs/aiking/lib/python3.9/site-packages/sklearn/feature_extraction/text.py\n",
       "\u001b[0;31mType:\u001b[0m      method\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "vectorizer = TfidfVectorizer(stop_words=\"english\")\n",
    "vectorizer.fit_transform?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3da0715-3d06-4ee1-bb4d-716ae64a9174",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectors = vectorizer.fit_transform(newsgroups_train.data).todense(); vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a51478a-b819-4eed-8032-21792e0305f3",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda/envs/aiking/lib/python3.9/site-packages/sklearn/utils/validation.py:585: FutureWarning: np.matrix usage is deprecated in 1.0 and will raise a TypeError in 1.2. Please convert to a numpy array with np.asarray. For more information see: https://numpy.org/doc/stable/reference/generated/numpy.matrix.html\n",
      "  warnings.warn(\n",
      "/opt/anaconda/envs/aiking/lib/python3.9/site-packages/sklearn/decomposition/_nmf.py:289: FutureWarning: The 'init' value, when 'init=None' and n_components is less than n_samples and n_features, will be changed from 'nndsvd' to 'nndsvda' in 1.1 (renaming of 0.26).\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "W1 = clf.fit_transform(vectors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8226f526-a38b-443f-bb97-7e514b8a80ef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5, 26576)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.components_.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1927bfbc-64e3-4c2f-8a59-866897a90d8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "H1 = clf.components_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2846874-3d67-41ee-a703-748dda1543f4",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda/envs/aiking/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
      "  warnings.warn(msg, category=FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array(['00', '000', '0000', ..., 'zware', 'zwarte', 'zyxel'], dtype='<U80')"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab = np.array(vectorizer.get_feature_names())\n",
    "vocab.shape\n",
    "\n",
    "vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c92c3961-71c3-4695-8c40-7afbb219d167",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['know morality say objective like just think don',\n",
       " 'format know windows program file image files thanks',\n",
       " 'station earth lunar moon orbit shuttle launch nasa',\n",
       " 'vice queens sank manhattan bronx beauchaine tek bobbe',\n",
       " 'faith belief does atheism christian believe bible jesus']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_topics(H1, vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18bf5431-716b-4366-919c-6f4f662dcc9a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['cosmonauts', 'cosmos', 'cosponsored', 'cost', 'costa', 'costar',\n",
       "       'costing', 'costly', 'costruction', 'costs', 'cosy', 'cote',\n",
       "       'couched', 'couldn', 'council', 'councils', 'counsel',\n",
       "       'counselees', 'counselor', 'count'], dtype='<U80')"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab[7000:7020]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c36a4505-dbf4-4e44-8204-b3279d9d100d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0xaee5b12b250>]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAlB0lEQVR4nO3de3gdZb0v8O+PYtmPiEew1eMDSJFTj7tuETQiZ3vZ3tDipRVvFC8bPLo5euyj+6jbXUAqFhBEQQWKm0KLCEIpIiW0kdKWAr2QNmmbpm3SNGmatkkvSZM0aXO//M4fa1Y7WZlZ6521ZtbMmvX9PE+frjVrMvPOmlm/eee9iqqCiIgK32lhJ4CIiPzBgE5EFBMM6EREMcGATkQUEwzoREQxcXpYO540aZJOmTIlrN0TERWkzZs3H1XVyU6fhRbQp0yZgsrKyrB2T0RUkERkn9tnLHIhIooJBnQiophgQCciigkGdCKimGBAJyKKCQZ0IqKYYEAnIooJo4AuItNFpE5EGkRkjss6XxORGhHZKSJP+JtMIsrV1v2d2NHSFXYyKEAZOxaJyAQA8wFcAaAZQIWIlKpqjW2dqQBuAPAhVe0UkbcElWAiys5VD2wAADTd+bmQU0JBMcmhXwagQVUbVXUQwGIAM1PW+TcA81W1EwBUtdXfZBIRUSYmAf1cAAds75utZXbvBPBOEVkvIuUiMt1pQyJyvYhUikhlW1tbdikmIiJHflWKng5gKoCPAbgGwEMi8qbUlVR1gaqWqGrJ5MmOY8sQEVGWTAJ6C4Dzbe/Ps5bZNQMoVdUhVd0LYDcSAZ5iSlXx2xV1aDraE3ZSiMhiEtArAEwVkQtFZCKAWQBKU9ZZikTuHCIyCYkimEb/kklR09zZh/vXNODbf6oIOylEZMkY0FV1GMBsACsA1AJYoqo7RWSeiMywVlsBoF1EagCsAfAfqtoeVKIpOoZGRsNOAhFZjMZDV9UyAGUpy+baXiuAH1v/iIgoBOwpSkQUEwzoRFR0RkcV3f1DYSfDdwzoRFR0frdqNy6+5UV09gyGnRRfMaATUdFZXn0IANDRy4BOREQRxIBORBQTDOhERDHBgE5EFBMM6EREMcGATkQUEwzoREQxwYBORBQTDOhEVLRUw06BvxjQiaj4SNgJCAYDOhFRTDCgE3mwo6ULGrfndIoNBnQiQ6/ubsPn71uHx8v3hZ0UIkcM6ESG9nX0AgB2HT4eckqInDGgExHFBAM6EVFMMKATEcUEAzoRUUwwoBMRxQQDOhFRTBgFdBGZLiJ1ItIgInMcPr9ORNpEpMr6913/k0pEROmcnmkFEZkAYD6AKwA0A6gQkVJVrUlZ9SlVnR1AGomIyIBJDv0yAA2q2qiqgwAWA5gZbLKIiMgrk4B+LoADtvfN1rJUXxaRahH5q4ic77QhEbleRCpFpLKtrS2L5BIRkRu/KkWfBzBFVS8GsBLAo04rqeoCVS1R1ZLJkyf7tGsiIgLMAnoLAHuO+zxr2Umq2q6qA9bbhwG835/kERGRKZOAXgFgqohcKCITAcwCUGpfQUTeZns7A0Ctf0kkihYOnktRlbGVi6oOi8hsACsATACwSFV3isg8AJWqWgrghyIyA8AwgA4A1wWYZqJQxHSSG4qRjAEdAFS1DEBZyrK5ttc3ALjB36QREZEX7ClKREUsXgVoDOhEVHTiWnzGgE5EFBMM6EREMcGATkQUEwzoREQxwYBORBQTDOhERDHBgE5EFBMM6EREMcGATkQUEwzoREQxwYBO5JHGa/gPihEGdCJDEtcBQCg2GNCJiGKCAZ2IKCYY0ImIYoIBnYgoJhjQiYhiggGdiCgmGNCJiGKCAZ2IilbcOokxoBNR0ZGY9hIzCugiMl1E6kSkQUTmpFnvyyKiIlLiXxKJiMhExoAuIhMAzAdwJYBpAK4RkWkO650F4EcANvqdSCIiyswkh34ZgAZVbVTVQQCLAcx0WO9WAL8G0O9j+oiIyJBJQD8XwAHb+2Zr2Uki8j4A56vqch/TRhRRMatJo9jIuVJURE4DcA+Anxise72IVIpIZVtbW667JsorQTwr0ig+TAJ6C4Dzbe/Ps5YlnQXgnwC8LCJNAC4HUOpUMaqqC1S1RFVLJk+enH2qiYhoHJOAXgFgqohcKCITAcwCUJr8UFW7VHWSqk5R1SkAygHMUNXKQFJMRESOMgZ0VR0GMBvACgC1AJao6k4RmSciM4JOIBERmTndZCVVLQNQlrJsrsu6H8s9WURE5BV7ihIRxQQDOhFRTDCgExHFBAM6EVFMMKATEcUEAzoRUUwwoBMRxQQDOhFRTDCgExHFBAM6EVFMMKATUdGK28j2DOhEVHTiOrI9AzoRUUwwoBMRxQQDOpGPRkYV/3zHapRuOxh2UqgIMaAT+ahncBgHu/px09+2h50UKkIM6EREMcGATkQUEwzoREQxwYBOFIC4dVihwsCATuSjfHdYuXnpDkyZszzPe6WoYkAnKmCPle8LOwkUIQzoREQxwYBO5JGygJwiyiigi8h0EakTkQYRmePw+fdEZLuIVInIOhGZ5n9SicIlHgrIlVGfQpAxoIvIBADzAVwJYBqAaxwC9hOq+h5VvQTAXQDu8TuhRIVAvER9Ip+Z5NAvA9Cgqo2qOghgMYCZ9hVUtdv29kyw1RYRUd6dbrDOuQAO2N43A/hg6koi8gMAPwYwEcAnnDYkItcDuB4A3v72t3tNKxERpeFbpaiqzlfViwD8J4Cfu6yzQFVLVLVk8uTJfu2aiIhgFtBbAJxve3+etczNYgBfzCFNRAWPZY4UBpOAXgFgqohcKCITAcwCUGpfQUSm2t5+DkC9f0kkKhysEqUwZQzoqjoMYDaAFQBqASxR1Z0iMk9EZlirzRaRnSJShUQ5+rVBJZiI4uFARy8Wrtsbahri1rrUpFIUqloGoCxl2Vzb6x/5nC4iirlvLtyIfe29uOrSc3HOmRPzuu+4ti5lT1GiAMQt5xeE4/3DANgJy08M6EQ+imvOjwoDAzoRUUwwoBNFUHNnL+oOHw87GXnBAhf/GFWKElF+ffjXawAATXd+LuSUBIelU/5jDp3II5M6PGW+k0LAgE5kyJ6j3Lq/EyW3rURX71DKOsx3UngY0ImycO/qehw9MYjN+zvCTkrBY6tF/zCgE1Eo2MTTfwzoREQxwYBOFIBiLUboGRhGz8Bw2MkoWmy2SOSjYi9GePcvVgDw1tySLYL8wxw6EYWkyO9+AWBAp4zW1R/F5b9ajb7BkbCTQkRpMKBTRr8qq8Xh7n7saTsRdlKIKA0GdKIAsFTYA35ZvmFAJ6JQFHsFchAY0ImIYoIBnYjIo2XVB/Hx376M0dFolRexHToVpKGRUQDA6ybkP09i0m6apQnmwgyJ2baB/4+nq9E3NIL+4RG8fmJ0wihz6FSQPnD7Klx8y4t53aeXMt9o5duiKcybXnJUzKFhxbCVOYgDBnQqSMd6h9A3FH67+GLt4h8XX7h/HT79u1fDToZvGNCJsiBsohEbjUd7wk6CbxjQifLgxMAwHl7bCGWWfpw4fSUbG9uxcN3e0PZvFNBFZLqI1IlIg4jMcfj8xyJSIyLVIrJaRC7wP6lEBSQlSN36fA1uW16LNXWt4aQnIFfc8wq+86eKrP42jg85Vy8ox63LakLbf8aALiITAMwHcCWAaQCuEZFpKattBVCiqhcD+CuAu/xOKFEh6+pLTFU3MBSfCjgAqG89gdW74nWTKmQmOfTLADSoaqOqDgJYDGCmfQVVXaOqvdbbcgDn+ZtMoniIUemCsYbWE3h4bWPYySgKJgH9XAAHbO+brWVuvgPg704fiMj1IlIpIpVtbW3mqSQqcHEsXjB11QPrcdvyWtfmgRwP3T++VoqKyDcBlAD4jdPnqrpAVUtUtWTy5Ml+7pooEuJUweeX5AxGqS2DhN2vfGfSxakFwPm29+dZy8YQkU8BuAnAv6jqgD/JIyosiyv2AwAG3XKjRRzwEy18GMSDZJJDrwAwVUQuFJGJAGYBKLWvICKXAngQwAxVZQ0JFa1VtUcclxdzkYtbm30WtfgvY0BX1WEAswGsAFALYImq7hSReSIyw1rtNwDeAOBpEakSkVKXzREVpWLOmRea/qER9EegF3I2jEaVUdUyAGUpy+baXn/K53QRUcyk3tOiWob+rptfwNmvfx22zv102EnxjD1FiQzlkssu6iKXsBOQhc7eIaP1ovbkxYBOWTl4rC/sJERSVHOd5K+o3qAZ0LPQNziC1S6VX8Xi6gXlAIDhkYhlUQLkx484ihWBh7r60H4i+IZpbrnZqOVyCxkDehZufm4HvvNoJWoOdoedlIJy3+p6PFc1rsVrUYhyzv1/3fES3n/bqsC273YjjGout5BFZ6qNArKvPTHc5gmrwwSZuXvlbgDAzEvSdTQubPYg9cruNvzj287CW876h/ASRKE70NGLkVHFlElnBr4v5tBzwKFQKZ1rF23C1/7rtTHLivmSiWJxU9DW7GrFR+5ag4/99uW87I8BPQtRfnwm//QNjuCZzc053bib2nszrxRzTr+Xf/nNGhzq6gcQ7wHL8j2ULgM6kYt5y2rwk6e3obyxI+ykxIL9vrgv5BtdGOX3NQe7UbrtYKD7YEDPgVvO4uiJAXxr4UZ09gzmNT3kr9buRA6yx4+6kmJ+qCvmY7f57L1r8cMntwa6Dwb0bGS4QBeu24u19UfxxKb9+UkPUZ6oKjbsOVr09UdRPXwG9BxE9aSSv/w8zcltqSrKG9sLLjA+uekAvv7QRjxffci3bQb1HbSfGMCBjmCLduxFN1EY/4UBPQuZniDz+RvdsOcofvCXLaEFhmJsuZCN1GtmaVULZi0oxzNbCqtd/r6ORJPdlk7znsJhlbhcfsdqfOSuNYFsu88heN/4t+2B7MsLBvQcZApm+ah4uW5RBZZvP+Q6/jblzstpdBsqNlWyUnB/wDnIYjYUUC/mtfXOs61t2d8ZyP68YEDPAnu4kW8KrMglF3E51LrDx8NOgisG9AJRe6gbU+YsR9WBY46fx+XHUiyi3pdBVfF/HqvEK7udc6NeitoyZYB47fqHAd1QR88gHl7bOLas2m2wIcOLfU1dK6bMWY6G1hMZ131pV2IiqBU7D4/9INpxIZbSBaC4nI5RBVbsPIJvP7JpzHL7jci03ibOAdt+bKbFbUHiWC6GfrKkCmvq2lAy5Rzj3FWm9ZZbLQW27O/E/3jLG4y2GecfR9Q5nU+v5yM1CBby6Qy6kwx5xxy6oe7+ROeSYVvlo+uP0euv1GB91xHrPO4qKlQVN/xteyQqkjJxOj1u37vbqRw3432IJ+6+1fW+bKftuNmQuxHIuBYNBvQs+HWBZrMZt+KcQsu59w2N4MlN+/GNhzaGnZRQhXHekqNe5mLxpgOe/yYuTVyjULTihgE9JOsbjuLpzc0AzC50t+KbfFxbw6OJp5JmD22PTcXlR57kdjpSl29v6Qo6KYFIXm9emlv6VQH8wo7DONYb3eE0ohDmGdBzkCl3lS7YLly319+0BBgYdx9JVNre9Kx/HSei3srDT05BaGVNYsaruN3QgtLa3Y/vPb4Z3398S9hJifSVy4AeAJOfqP2i8PTYnbJuoQfGQisqSmd4ZBTH+sZPLhxUB5dCk8u5HhhOPCVGoSNWhEtcGNCDlO682y8KoxtAhC+ibCSPJ06h7udLd2CbQz+BuJ07rzId/0fuWoPu/vE3Qj/1DAyjd3AYLcf60OVw0/VFBM6zUUAXkekiUiciDSIyx+Hzj4rIFhEZFpGv+J/M6FCY/0AVwJKKA/jQnS85fJrd2XcLgIWU0z14zP+y+Ch4PotmfIV03oDgYtbW/cfS7zfHHb/7FytwyS9X4kN3voRP3fMKRkaz/+IjELddZQzoIjIBwHwAVwKYBuAaEZmWstp+ANcBeMLvBEbBvOdrsHlf+uZ1Ow924Qv3rUPv4Nixs3/2TDVaHALYmBy6SbNFt+VZXF2fuPtlfDxPU2I5GbKPO1NgAS0bJsVr25u78Fj5vrykx82idXux+0hw3drTnepk+/z7X6rHlDnLM66XjeR4R23HBzBaaHdSQyY59MsANKhqo6oOAlgMYKZ9BVVtUtVqALEcIWrReucKTHuF1h1lu7C9pWtM4E8Xa0+zfXjjs9vHzT3pZmhkFMurD7l2UPn6Q+W4LqV3X6rGth7sPdpjtD87v4oOVO1FLt5/WF29wT6eezH3uR1G67l9d8mj/8L963DzUrNtBWXeshp8/t51adfJJgyaXDbJ7f72Recmlcmmgge7+sdMzj48MoolFQcwauW4b11Wg8/87tUsUumSLlW8XNdqdCOJQs7dJKCfC8De6LTZWuaZiFwvIpUiUtnW5jxGRCFwqohM3vFV3XMRtYe68cX569E7ODxuG5ua0k9zlgwIj6xvwg+e2IIXdhy20jLWhj3teLkumO/Wz0xNtpW5e9pO4L3zXvQvIWm4BeHu/iGstoZiOGjNixkXgyOj6BscwUU3loWdlDHsp+I7f6o4+XrR+r342TPVeKoyEaIWrtuLOh+fMp7d2oLrHqnAk7Z292yHblHVBapaoqolkydPzueuA5EMcIlZXNoTr22fp57325fXourAMWze15lzbrc9wOnt+gZHsGmv2TyaR7rNegvalTe2Y6vVQ9TrTWJv29gni9bj/ZgyZzkWW7NDfePhclx8ywrPacpkTV0rhqz2+Ct2HvH0t/YAUAjNFDvTtPXO5bJNm8v18LXYn4KTv4NjAT21Jet7Wo4lWtcsXLcXr7oOWBY+k7FcWgCcb3t/nrWsaKUG42zqV7wG9FyaJ+463I1JbzgDk95whtH6c/5WjeeqDmLtzz6O8895fdb7dd/+qfbsisQP9MwzJuBd//2NnrfVdDTxQ3tmSzMuefubsL6h3XG9ZdUH0dk7hPX1R/GRd07CNz54gfE+Nuw5ikfWN2HyWWbfX6ro5uec+R2YTHK0CsWObDpb2RI77GFOgGxz2bcuq/G0fmMWRZu5MMmhVwCYKiIXishEALMAlAabrMLgdOGblLWp5t5+XJEowukZHMm43+m/X4tP3v2K8bZ3HUo8svakVPAG8aQ5Mqr48h83YPrv12b19/bjTreN2U9sxc1Ld+CFnYdx07PeyqqPnkjkAt3GLllb34bLf7Ua/cOZA4rX8/79xzd7Wv+Zzc05j49j/06DmH7PzefvG19+//DaRhzo6DW69noDnALO7ec15ok8sL2byxjQVXUYwGwAKwDUAliiqjtFZJ6IzAAAEfmAiDQD+CqAB0VkZ5CJjpoxrRjgfvLtF6XnHHrK+oeO9eHKP5gHwcDa3uZZ6vfwqjV7TJAdrDJt+fbltTjc3W/UFC61yCXT/f/vOw6nXyHFT57ehi89sMHT3wTNqFJUU98rWrv7cdvyWlz7yKbQOtBFubzciVEZuqqWqeo7VfUiVb3dWjZXVUut1xWqep6qnqmqb1bVdweZ6KjbaJU/p3sCzPVCeeDlPZ7/5odPbvW0fiG07GqypnIL8vee6VSZnMtck/eVP27ADR7mrLS3BPHK7by/XNeKVsMRFnPdp+qposyegeGMHfGCjrsF8FMAwJ6iGaUrykh+NmYNPTXw0tETzhe/wv/4Y3LBmY5fHZVMyZb9nVhT1zpm2bi0WQfuVhyyZldr2nbNfsj0daXrc2BSSfpUxX5U7uvEk5v2o6tvyKiseHm1/2OVX/dIBf5qDSiXDb8yCCOjmvM0cG6/zVxEITfPgJ7B/3uqatyydCfOtBXDa43jK+8GhkfGdUwqZl96YAO+/UhF2nWS37dbu/qlVc7196OjatxJxUvAdv578fRjT81d/+czp3Lm7/3li2MqlZOO95sF+lAYHHrqmUh9n7qJz/w+0dZ8W/OxxPrqLZMUVKuYsDGgZ7C0yj2nkykcuI2oeO2iTY45yiv/sBbT5jo3ucvUjNCP3M89K3fjH29+IfcN5VG64968rwPPuZy/d9xYZlyEkSkYe82YZSoOuf+lhrSfL906/ib1nltexE+e3uYtIS6iUNRmerMtb0z8LoLOCLnWiwW6V+8Y0LOQ7iSmnW8ywy+/sc29idOLNd7aPmfj3tX16LO1FIjCDztVauVYui7cVz9YnnZbiytOdRY5MTCMdfVHXfaZ3mkZzqv9qU0VuO+levuH42Sb03a7eXkVWFv5dM3QM45F7W9STEWgFMUTBvSYqDDsCOQmlzEywpSuLNXLEf34qSp8c+HGrAYOM/nNJ5+wXtndhqHhwvyus2Xy/Tz6WtOY916/oSiMzRKF2M+A7tGY6yb8a+ik7/65Mqe/P9x9qgv7vna3MaejcMmO1eSaVm83qfrWxCQefU5tmXMtRAdODtD2cl3rmKFis7mEhkcVH/zVKry0K7entkwDYQHpD73DsLeyyTFubBybIUk9dYU+7n++MKBn4Uj32PE7xnTEcLl6R0fVtctwFNjT7RjUkBjD5KIby7C2Pv/H8YHbV+H9t6709DcmgcQkKGUKJplCjf277e4fzqmlSNKR7gHcuqzWaN3Kpg40tI5/knEb3dH0PphuZMYDDhNRJItyDjuMf5Oaw7YX+xzpHkB7T/pWKRHIoEeCSdd/SrHLesw/2DX+8dxpGQC8tKvVcXlUmJQVDlo9ITNV2gUhWYlc3WzWPbyjZ9DoR97ZO4hzzpyYS9JyKmfNpajL9G+/YjiSp/f9u3/Wb2UKuvqGxtzwNu3twNceNEtP7eHuk6/nPJO+AjvoeG40728EHiKYQ8/B6Kji5qU7TjadAoC5zzl3kh0w6BZeKILudXr1g6/hOZfmhi/vNrsxmg5rm6zQTAbHbH6T2zPcZIIKNrlu1+3Jw4/0KoDVtUfw3l++iO7+RAsU1cRwFU5SO9mqAsurD518X+UwE1Tq+rn4+kPOFeiZns7mPrcz8H4OXjCg56B3cASPle/DNQs2hp2UnDlduG65kl0pFZF+z8S+cW8HfrS4yvGzTDPbJJneQE9LOewZ968HADxX1YJVtYmbR6YKt+EcZr95aO1eNKW0oTfdWq5BzHWMdh/KL0ZVT/aYzmbbe9pOjDs36Sg0p449ydFSvXpmS+7FZ35iQPeo9fj48r9CGBLVyTJbb0K3npYmLpm3EgPDwQ2MlA3TKcZe2HEYs584NZN8so24/YbyrEO7bz/9W5YV2vmcMNkkEA/abqJuHX1M732/fqEuY3NQuwdfacQ//cLfYZOnzFl+cpjnQvmJM6B7NPuJU+OhnJx1J4In++G1jY7L7bP92I9lZY23QaBSDabkiLfu78xuOFSfmNZZ3PH3XVhme7QPQqYcfrKFTVK+imLd9pPNbFbA2KElVIGalOIVhZenj9xy3Knbylay/8ezW1s8zZQ1MqquxYZBYkDPQbKYwqQNbLbX5sio4pfPex+88g+r6h2Xu872k+OPJzXnddUDGxyHQy1Gj25oCjsJnvxu1alp4JKn1ekST12U2iFqbUpHrf6hEePg2nKsD/U+zTx0s2F9Smu3++xTrccH8O9PuQ9ul3pcX3+o3LXYMEgM6D7IoQg1o+TkCl6ZJungsT68sMM5h+opY5Nm3Z6BYdzmcWKAfLMXOa30uVeuUxO+KMs2Q2v/HTgVQ970rPlokY1tPajMMDH7qOEP7/Hy/UbrfTVD65t0s3OlJsWp/iAf2GwxB0GXnZduO+h5yFu7rfs7T8496uZLD2zA4e5+/OiTU7PeD5B4Slm6tQUNrSfw08/8zzGf/XzpjpzKoec9H/zNIDlRCJB9mbabXIayzcXmfZ2urUoA96EonOKkyZVuf1J1uinUHjqOf75oksGWzPx+tfNTaGL/3n+b+9p7s+opDCSepKNQ9Fp0AX3T3g6UXHA2TvNShR6SJzY6d/wwcWJgGFcZTHSQ7CGaa8uGS22dflIDumkwP9Ldj7e+8R/GLV+03nmQs0KRbC2Tb1/+Y3YTXaR2lBscHj1VOehicHjU6Brys4v+vWkCerayKd4EEscVheEHiqrIZW19G7724GtY4FJh6JWX7sju3emd3bas5uRIcqaO92ffPvxeh85ClU0d+KyHWZGceLlRfPBXq3PaV1y0ZJlLtMs0vteSygPG+/np09tw9YLx7bTt5/baRZvGtCxyOu2Hu/uxpPLA+A8CsPvIicwrOUjXsiZ1SkY71Wg0jiiqgH7I6nLc0JrdyU7l5Y786xd2Ga/72xV1eNhl6N10/rLRrKzQ1C3P14xrqeDVhTeU+ZSa4uF12jknN2Yor/7ZX6tdP0vtZ+A2McqaulNDQLzW2J6xDB0A9qQZUdRPyfHSvUoX0NNlykY0Go2XCy6gH+sdxKosK62SJ8uvR6M7/m4epL24f012XevvDCg9+TTj/uJuGXOpWyukAtBjqytIbeFSKHJpjWba9yFIBRfQv//4Fnz3z5VZTSF1WoTbjVOC6VgtcdVZwDPp3L3yVHPH36yoCzEl2fPSmclOVY07epm2zslGwQX0fe2JR7YdLV2eO0DYc+g7D3Zhypzl2OgwFRyQGDvi6TyV98VNtW1sG6JCMpJlbs9L7vwdN5YFlpsvuFYuydYp11lzTTbd+Tmjv9t24NjJ3oOjCqxvSDwSrqo9gg++483j1v/i/PV+JLcoJcdDISo0y7PsNWza1j1paGQUE06bkNW+0im4HPqENM0Nh0dG8Vj5PscpvGbOX3+ycmd0VE9W4Di1xXUaO5qIyI29d62JoIbTLriAnlrGdbx/CGXbD6H1eD++9/hm3Lx0x8nJmVUVe4/2jHu86e4fOlmOvuDVxpNdkpNlW5+6J7saciIiE7kMhpeOUZGLiEwH8AcAEwA8rKp3pnx+BoA/A3g/gHYAV6tqk79JTUjNoL/nlvGtAqoOHEPptoN4cedhLKs+hEvf/qYxn6+tP4qag6ea473LNtP9LV+Y5mt6iYhS5TLkcjqSqeOHiEwAsBvAFQCaAVQAuEZVa2zr/F8AF6vq90RkFoCrVPXqdNstKSnRykrvXayvuOeVcaPTEREVknPOnIgtN1+R1d+KyGZVLXH6zKTI5TIADaraqKqDABYDmJmyzkwAj1qv/wrgk+LX2Jcp0pWhExEVAtMJtr0yCejnArC332u2ljmuo6rDALoAjGs6IiLXi0iliFS2tWU30fA3Lr8gq78jIoqKu7/63kC2m9dmi6q6AMACIFHkks02vnX5BfgWgzoR0TgmOfQWAOfb3p9nLXNcR0ROB/DfkKgcJSKiPDEJ6BUAporIhSIyEcAsAKUp65QCuNZ6/RUAL6kfM80SEZGxjEUuqjosIrMBrECi2eIiVd0pIvMAVKpqKYCFAB4TkQYAHUgEfSIiyiOjMnRVLQNQlrJsru11P4Cv+ps0IiLyouB6ihIRkTMGdCKimGBAJyKKCQZ0IqKYyDiWS2A7FmkDkO209pMAFOYcV94Uw3HyGOOhGI4RiMZxXqCqk50+CC2g50JEKt0Gp4mTYjhOHmM8FMMxAtE/Tha5EBHFBAM6EVFMFGpAXxB2AvKkGI6TxxgPxXCMQMSPsyDL0ImIaLxCzaETEVEKBnQiopgouIAuItNFpE5EGkRkTtjp8UpEmkRku4hUiUiltewcEVkpIvXW/2dby0VE7rWOtVpE3mfbzrXW+vUicq3b/vJBRBaJSKuI7LAt8+2YROT91nfWYP1t3uchdDnGW0SkxTqXVSLyWdtnN1jprRORz9iWO16/1vDUG63lT1lDVeeViJwvImtEpEZEdorIj6zlcTuXbsdZ+OdTVQvmHxLD9+4B8A4AEwFsAzAt7HR5PIYmAJNSlt0FYI71eg6AX1uvPwvg7wAEwOUANlrLzwHQaP1/tvX67BCP6aMA3gdgRxDHBGCTta5Yf3tlRI7xFgA/dVh3mnVtngHgQuuanZDu+gWwBMAs6/V/Afh+CMf4NgDvs16fhcTk8NNieC7djrPgz2eh5dBNJqwuRPZJth8F8EXb8j9rQjmAN4nI2wB8BsBKVe1Q1U4AKwFMz3OaT1LVV5EYB9/Ol2OyPnujqpZr4tfxZ9u28sblGN3MBLBYVQdUdS+ABiSuXcfr18qlfgKJCdaBsd9X3qjqIVXdYr0+DqAWifmC43Yu3Y7TTcGcz0IL6CYTVkedAnhRRDaLyPXWsreq6iHr9WEAb7Veux1vIXwPfh3Tudbr1OVRMdsqbliULIqA92N8M4Bjmphg3b48NCIyBcClADYixucy5TiBAj+fhRbQ4+DDqvo+AFcC+IGIfNT+oZVziVVb0jgek+WPAC4CcAmAQwDuDjU1PhGRNwB4BsC/q2q3/bM4nUuH4yz481loAd1kwupIU9UW6/9WAM8i8dh2xHochfV/q7W62/EWwvfg1zG1WK9Tl4dOVY+o6oiqjgJ4CIlzCXg/xnYkiitOT1medyLyOiSC3F9U9W/W4tidS6fjjMP5LLSAbjJhdWSJyJkiclbyNYBPA9iBsZNsXwvgOet1KYB/tVoTXA6gy3r0XQHg0yJytvVY+GlrWZT4ckzWZ90icrlVNvmvtm2FKhnkLFchcS6BxDHOEpEzRORCAFORqAx0vH6tXO8aJCZYB8Z+X3ljfb8LAdSq6j22j2J1Lt2OMxbnMx81r37+Q6JmfTcStcs3hZ0ej2l/BxI14dsA7EymH4kyt9UA6gGsAnCOtVwAzLeOdTuAEtu2/jcSlTMNAL4d8nE9icQj6hAS5YXf8fOYAJQg8ePaA+B+WD2cI3CMj1nHUI3Ej/5ttvVvstJbB1tLDrfr17o2NlnH/jSAM0I4xg8jUZxSDaDK+vfZGJ5Lt+Ms+PPJrv9ERDFRaEUuRETkggGdiCgmGNCJiGKCAZ2IKCYY0ImIYoIBnYgoJhjQiYhi4v8DKhk/GkpnZu4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(clf.components_[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a51cc97-291e-4866-8db2-9bbb673aa7b1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "43.71292605795258"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.reconstruction_err_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0d621ec-0793-4ea5-9148-c3bceee3c526",
   "metadata": {},
   "source": [
    "## NMF in summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11633b09-8915-43f4-be49-905c91df4517",
   "metadata": {},
   "source": [
    "- Benefit fast and easy to use\n",
    "- Took years of research to master\n",
    "- For NMF, matrix need to be as tall as wide - otherwise we will have error in fit_transform ($m \\ge n$ for matrix $V^{mxn}$)\n",
    "- Can use df_min in CountVectorizer to only look at words that were in at least k of the split texts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0de20658-dab2-4b8c-b149-04aca524a363",
   "metadata": {},
   "source": [
    "### Gradient Descent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af6f7eae-9166-40f1-a553-d9d25b71a692",
   "metadata": {},
   "outputs": [],
   "source": [
    "def lin(a, b, x): return a*x+b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22a9ecd9-6e2a-40ec-902e-97ce71f8ec9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "a=3\n",
    "b=8\n",
    "n=30\n",
    "x = np.random.random(n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0470ce9e-b596-4512-a56c-862af5c2304f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 8.66536449,  8.79541699, 10.56126547,  8.24715391, 10.51300969,\n",
       "        9.43275675,  8.04751459,  8.29722369, 10.10790904,  9.93738814,\n",
       "        8.31807423,  9.26202454,  8.13544973, 10.5840327 ,  9.73784168,\n",
       "        9.98504435,  8.1977378 ,  9.50929408, 10.18216117,  8.48690698,\n",
       "        8.66043837,  8.57549578, 10.60328607,  9.00900207,  9.9869936 ,\n",
       "        8.31974839, 10.52028891,  9.37556759,  8.68929477,  9.58876118])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAUPklEQVR4nO3df5DcdX3H8deLyyp7VblITkoOQmjFDEJKozuIMloqSCxVyETagQ5T6CAZrVarTsYwdYTBziQ2tnU6dkpTZMA6UgEzZ1p/BEZQOlYyXjxiiEyEigILygk5WpsTl/DuH/u9eNns3m32u9n97vf7fMzc3O73+7n9fvjO3YtvPt/39/NxRAgAkF/H9LsDAICji6AHgJwj6AEg5wh6AMg5gh4Acm5RvzvQaMmSJbF8+fJ+dwMABsrOnTt/HhGjzfZlLuiXL1+uiYmJfncDAAaK7Z+02sfQDQDkHEEPADlH0ANAzhH0AJBzBD0A5Fzmqm4AoGjGJ6vavH2vnpye0dKRstavXqE1q8a69vkEPQD00fhkVddu3a2Z2gFJUnV6Rtdu3S1JXQt7hm4AoE/GJ6v6yO27Dob8rJnaAW3evrdrx+GKHgB6aHaYpjo9I0tqtSLIk9MzXTsmQQ8APdI4TDPfsk9LR8pdOy5DNwDQI5u37z1smKaZcmlI61ev6NpxCXoA6JF2hmOGbG1cu5KqGwDIuvHJqq7ftkfTMzVJ0uLhko4rlw6+b6ZcGup6yEttXNHbvtn207YfnLPtlbbvtv1w8n1xi589YPuB5GtbNzsOAFk1PlnV+jt2HRLq+/bX9L/Pv6DSMT6k7ey7sZHyUQl5qb0r+lskfUbS5+Zs2yDpGxGxyfaG5P1Hm/zsTET8btpOAsAg2bx9r2ovHn6r9cCLoVcMlzT8kkVH7eGoZhYM+oi4z/byhs2XSDoveX2rpG+qedADQOHMNxY/vb+myY9f2MPedH4z9oSIeCp5/VNJJ7Rod6ztCdv3217T6sNsr0vaTUxNTXXYJQDIhvlKI7tZNtmu1FU3ERFqXQ56SkRUJP2JpE/b/u0Wn7ElIioRURkdbboSFgAMjPWrVxw2Fi9JpSF3tWyyXZ0G/c9snyhJyfenmzWKiGry/UeqD++s6vB4ADAw1qwa0+Y/Oksj5dLBbYuHS9p86VlHfTy+mU7LK7dJulLSpuT7lxsbJJU4+yPiedtLJJ0r6W867SgADJI1q8b6EurNtFNeeZuk70haYfsJ21erHvBvs/2wpAuS97JdsX1T8qOnS5qwvUvSvZI2RcQPjsZ/BACgtXaqbi5vsev8Jm0nJL07ef1fklam6h0AIDWmQACAnCPoASDnCHoAyDmCHgByjqAHgJwj6AEg5wh6AMg5gh4Aco6gB4CcI+gBIOcIegDIOYIeAHKOoAeAnCPoASDnCHoAyDmCHgByrtOlBAGg78Ynq9q8fa+enJ7R0pGy1q9ekZnl+7KEoAcwkMYnq7p2627N1A5IkqrTM7p2625JIuwbMHQDYCBt3r73YMjPmqkd0Obte/vUo+wi6AEMpCenZ45oe5ER9AAG0tKR8hFtLzKCHsBAWr96hcqloUO2lUtDWr96RZ96lF3cjAWQSQtV1My+pupmYQQ9gMxpt6Jmzaoxgr0NDN0AyBwqarqLoAeQOVTUdBdBDyBzqKjpLoIeQOZQUdNd3IwFkDlU1HQXQQ8gk6io6Z4Fh25s32z7adsPztn2Stt32344+b64xc9embR52PaV3ew4AKA97VzR3yLpM5I+N2fbBknfiIhNtjck7z8694dsv1LSdZIqkkLSTtvbImJfNzoOIPvGJ6u6ftseTc/UJEmLh0u67p1ncKXeYwte0UfEfZKebdh8iaRbk9e3SlrT5EdXS7o7Ip5Nwv1uSW/vvKsABsn4ZFXr79h1MOQlad/+mtbfuUvjk9U+9qx4Oq26OSEinkpe/1TSCU3ajEl6fM77J5Jth7G9zvaE7YmpqakOuwQgSzZv36vai3HY9tqB4MGnHktdXhkRofrQTJrP2BIRlYiojI6Opu0SgAyY7+EmHnzqrU6rbn5m+8SIeMr2iZKebtKmKum8Oe9PkvTNDo8HIOMaJyEbGS5p3/5a07Y8+NRbnV7Rb5M0W0VzpaQvN2mzXdKFthcnVTkXJtsA5MzsJGTV6RmF6pOQ/eKXL+gYH962NGQefOqxdsorb5P0HUkrbD9h+2pJmyS9zfbDki5I3st2xfZNkhQRz0r6hKTvJl83JNsA5EyzSchqL4ZecWxJI+XSwW2Lh0vafOlZVN302IJDNxFxeYtd5zdpOyHp3XPe3yzp5o57B2AgtBpzf26mpkc3/WGPe4NGzHUDIDUmIcs2gh5AakxClm3MdQMgNSYhyzaCHkBXMAlZdjF0AwA5R9ADQM4R9ACQcwQ9AOQcQQ8AOUfQA0DOEfQAkHPU0QMF0TiNMA80FQdBDxTA7DTCszNMVqdndO3W3ZJE2BcAQzdAATSbRnimdoAl/QqCoAcKoNU0wizpVwwEPVAATCNcbAQ9UABMI1xs3IwFCoBphIuNoAcKgmmEi4uhGwDIOYIeAHKOoAeAnCPoASDnuBkLZBxz1CAtgh7IMOaoQTcwdANkGHPUoBsIeiDDmKMG3UDQAxnGHDXoBoIeyDDmqEE3pLoZa/uDkq6RZEn/EhGfbth/nqQvS3o02bQ1Im5Ic0wgTxaqqGGOGnRDx0Fv+0zVQ/5sSb+S9HXb/xERjzQ0/c+IeEeKPgK51G5FDXPUIK00QzenS9oREfsj4gVJ35K0tjvdAvKPihr0Spqgf1DSm20fb3tY0kWSTm7S7o22d9n+mu0zUhwPyBUqatArHQ/dRMRDtj8p6S5J/yfpAUkHGpp9T9IpEfEL2xdJGpd0WuNn2V4naZ0kLVu2rNMuAQNl6UhZ1SahTkUNui1V1U1EfDYiXh8Rb5G0T9IPG/b/T0T8Inn9VUkl20uafM6WiKhERGV0dDRNl4CBQUUNeiVt1c2rIuJp28tUH58/p2H/b0r6WUSE7bNV/x/LM2mOCQwKKmqQFWnnuvmS7eMl1SS9LyKmbb9HkiLiRkmXSnqv7RckzUi6LCIi5TGBzKOiBlnirOVupVKJiYmJfncDSOXcTfc0HX8fGynr2xve2oceIe9s74yISrN9PBkLHAVU1CBLmKYYSKHVODwVNcgSruiBDs2Ow1enZxT69Tj8+GSVihpkCkEPdGi+J1vXrBrTxrUrNTZSllUfm9+4diU3XtEXDN0AHVpoHJ6KGmQFV/RAh5grHoOCoAc6xDg8BgVDN0CHeLIVg4KgB1JgHB6DgKEbAMg5rugBLTwBGTDICHoUXrsTkAGDiqEbFB5L+iHvCHoUHhOQIe8IehQeDz4h7wh6FB4PPiHvuBmLwuPBJ+QdQQ+IB5+QbwzdAEDOEfQAkHMEPQDkHEEPADlH0ANAzhH0AJBzBD0A5BxBDwA5R9ADQM4R9ACQcwQ9AOQcQQ8AOZcq6G1/0PaDtvfY/ssm+237H2w/Yvv7tl+X5njIv/HJqs7ddI9O3fAVnbvpHo1PVvvdJWDgdRz0ts+UdI2ksyWdJekdtl/d0OwPJJ2WfK2T9E+dHg/5N7t2a3V6RqFfr91K2APppLmiP13SjojYHxEvSPqWpLUNbS6R9Lmou1/SiO0TUxwTOcbarcDRkSboH5T0ZtvH2x6WdJGkkxvajEl6fM77J5Jth7C9zvaE7YmpqakUXcIgY+1W4OjoOOgj4iFJn5R0l6SvS3pA0oH5fmaez9oSEZWIqIyOjnbaJQw41m4Fjo5UN2Mj4rMR8fqIeIukfZJ+2NCkqkOv8k9KtgGHYe1W4OhIW3XzquT7MtXH57/Q0GSbpD9Nqm/OkfRcRDyV5pjIrzWrxrRx7UqNjZRlSWMjZW1cu5Il/oCU0q4Z+yXbx0uqSXpfREzbfo8kRcSNkr6q+tj9I5L2S/qzlMdDzrF2K9B9qYI+It7cZNuNc16HpPelOQYAIB2ejAWAnCPoASDnCHoAyDmCHgByjqAHgJwj6AEg5wh6AMi5tA9MoUDGJ6vavH2vnpye0dKRstavXsHDTcAAIOjRltm54menEZ6dK14SYQ9kHEM3aAtzxQODi6BHW5grHhhcBD3awlzxwOAi6NEW5ooHBhc3Y9GW2RuuVN0Ag4egR9uYKx4YTAR9QVETDxQHQV9A1MQDxULQF8jsVXy1SUnkbE08QQ/kD0FfEOOTVa2/Y5dqL0bLNtTEA/lEeWVBXL9tz7whL1ETD+QVQV8Q0zO1efdTEw/kF0M30BhVN0CuEfQFsXi4pH37D7+qXzxc0rc3vLUPPQLQKwzdFMR17zxDpSEfsq00ZF33zjP61CMAvcIVfQ608/ATUxgAxUXQD7gjefiJKQyAYmLoZsCxIAiAhRD0A44FQQAshKAfcCwIAmAhBP2AY0EQAAtJFfS2P2R7j+0Hbd9m+9iG/VfZnrL9QPL17nTdRaM1q8a0ce1KjY2UZdUfftq4diU3XQEc1HHVje0xSR+Q9NqImLF9u6TLJN3S0PSLEfH+zrtYTEcyXzzVNADmk7a8cpGksu2apGFJT6bvEpgvHkA3dTx0ExFVSZ+S9JikpyQ9FxF3NWn6Ltvft32n7ZObfZbtdbYnbE9MTU112qXcoGQSQDd1HPS2F0u6RNKpkpZK+g3bVzQ0+3dJyyPidyTdLenWZp8VEVsiohIRldHR0U67lBuUTALopjQ3Yy+Q9GhETEVETdJWSW+a2yAinomI55O3N0l6fYrjFQYlkwC6KU3QPybpHNvDti3pfEkPzW1g+8Q5by9u3I/mKJkE0E0d34yNiB2275T0PUkvSJqUtMX2DZImImKbpA/YvjjZ/6ykq9J3Of+YgAxANzli/uXleq1SqcTExES/uwEAA8X2zoioNNvHk7EAkHMEPQDkHEEPADlH0ANAzhH0AJBzLCXYJR8b363bdjyuAxEasnX5G07WX69Z2e9uAQBB3w0fG9+tz9//2MH3ByIOvifsAfQbQzcpjE9Wde6mew4J+blu2/F4j3sEAIfjir5DjVMJN3MgYw+jASgmrug71Gwq4UZDdo96AwCtcUXfhmarPbUzZfDlb2g6/T4A9BRBv4BWqz2NDJe0b3+t6c9QdQMgSwj6BbRa7emli45RuTR0yL5yaYiFuQFkDmP0C2g1RPPcTE0b167U2EhZljQ2UibkAWQSV/QNGsfjjyuXND1z+BDN0pGy1qwaI9gBZB5BP0ez8fjSkFU6xqq9+OtSSVZ7AjBIGLqZo9l4fO1A6GXHLmKIBsDA4op+jlbj8dP7a5r8+IU97g0AdAdX9HMsHSkf0XYAGASFC/rZ+WlO3fAVnbvpHo1PVg/uW796hcqloUPaMx4PYNAVauim1cNPkg6poGl8CpbxeACDrFBB3+rhp83b9x4Mc0omAeRNoYZuWt1sbWfeGgAYVIUKem62AiiiQgU9N1sBFFGhxui52QqgiHIT9M3mjG8W4NxsBVA0uQj6hcomAaDIcjFGP1/ZJAAUXaqgt/0h23tsP2j7NtvHNux/qe0v2n7E9g7by1P1tgXKJgGgtY6D3vaYpA9IqkTEmZKGJF3W0OxqSfsi4tWS/l7SJzs93nwomwSA1tIO3SySVLa9SNKwpCcb9l8i6dbk9Z2SzrftlMc8DGWTANBax0EfEVVJn5L0mKSnJD0XEXc1NBuT9HjS/gVJz0k6vtNjtrJm1RjL+gFACx1X3dherPoV+6mSpiXdYfuKiPh8B5+1TtI6SVq2bFlH/aFsEgCaSzN0c4GkRyNiKiJqkrZKelNDm6qkkyUpGd45TtIzjR8UEVsiohIRldHR0RRdAgA0ShP0j0k6x/ZwMu5+vqSHGtpsk3Rl8vpSSfdERAgA0DNpxuh3qH6D9XuSdieftcX2DbYvTpp9VtLxth+R9GFJG1L2FwBwhJy1C+xKpRITExP97gYADBTbOyOi0mxfLp6MBQC0lrkrettTkn7SRtMlkn5+lLszqDg38+P8tMa5mV+Wz88pEdG0miVzQd8u2xOt/plSdJyb+XF+WuPczG9Qzw9DNwCQcwQ9AOTcIAf9ln53IMM4N/Pj/LTGuZnfQJ6fgR2jBwC0Z5Cv6AEAbSDoASDnMh/0tt9ue2+yStVhUyj0ahWrLGrj3HzY9g9sf9/2N2yf0o9+9stC52dOu3fZDtsDVzbXqXbOje0/Tn5/9tj+Qq/72E9t/G0ts32v7cnk7+uifvSzbRGR2S/VV636b0m/JeklknZJem1Dmz+XdGPy+jJJX+x3vzN0bn5f0nDy+r1FOTftnp+k3csl3SfpftVXS+t737NwbiSdJmlS0uLk/av63e+MnZ8tkt6bvH6tpB/3u9/zfWX9iv5sSY9ExI8i4leS/k31OfDn6skqVhm04LmJiHsjYn/y9n5JJ/W4j/3Uzu+OJH1C9SUuf9nLzvVZO+fmGkn/GBH7JCkinu5xH/upnfMTkl6RvD5Oh6+ulylZD/qDK1Qlnki2NW0TR3EVqwxq59zMdbWkrx3VHmXLgufH9usknRwRX+llxzKgnd+d10h6je1v277f9tt71rv+a+f8XC/pCttPSPqqpL/oTdc60/EKUxgctq+QVJH0e/3uS1bYPkbS30m6qs9dyapFqg/fnKf6vwTvs70yIqb72akMuVzSLRHxt7bfKOlfbZ8ZES/2u2PNZP2K/uAKVYmTkm1N28y3ilUOtXNuZPsCSX8l6eKIeL5HfcuChc7PyyWdKembtn8s6RxJ2wpyQ7ad350nJG2LiFpEPCrph6oHfxG0c36ulnS7JEXEdyQdq/qEZ5mU9aD/rqTTbJ9q+yWq32zd1tCmqKtYLXhubK+S9M+qh3yRxlilBc5PRDwXEUsiYnlELFf9HsbFEVGExRDa+bsaV/1qXraXqD6U86Me9rGf2jk/j6m+qp5sn6560E/1tJdHINNBn4y5v1/SdtWXKbw9IvawilXb52azpJepvnD7A7Ybf1lzq83zU0htnpvtkp6x/QNJ90paHxFF+Jdyu+fnI5Kusb1L0m2SrsryBSZTIABAzmX6ih4AkB5BDwA5R9ADQM4R9ACQcwQ9AOQcQQ8AOUfQA0DO/T9etAakt0tdLgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(x, lin(a, b, x))\n",
    "y = lin(a, b, x)\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15b1e772-7487-4372-b5c9-0ff94fdc50fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "def sse(y, y_pred) : return ((y-y_pred)**2).sum()\n",
    "def loss(y, a, b, x): return sse(y, lin(a, b, x))\n",
    "def avg_loss(y, a, b, x):\n",
    "    n = x.shape[0]\n",
    "    return np.sqrt(loss(y, a, b, x)/n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88a08919-ee27-4dff-ba3e-309a0161aa05",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "avg_loss(y, a, b, x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b442eec2-7555-4eff-9013-0bba6a1cd314",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8.7778840202002"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a_guess=-1.\n",
    "b_guess=1.\n",
    "avg_loss(y, a_guess, b_guess, x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a1d64e6-1d1f-42b3-9ffe-d6386060d288",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.01"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = 0.01; lr\n",
    "# d[(y-(a*x+b))**2,b] = 2*(y-(a+bx))*(-1) = 2*(y_pred - y)\n",
    "# d[(y-(a*x+b))**2,a] = 2 x (b + a x - y) = x * dy/db"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68e1a289-9795-4007-aa8d-67807a5b824f",
   "metadata": {},
   "outputs": [],
   "source": [
    "a_guess=-1.\n",
    "b_guess=-1.\n",
    "def upd():\n",
    "    global a_guess, b_guess\n",
    "    y_pred = lin(a_guess, b_guess, x)\n",
    "    dydb = 2*(y_pred - y)\n",
    "    dyda = x*dydb\n",
    "    a_guess -= lr*dyda.mean()\n",
    "    b_guess -= lr*dydb.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29a0aa51-270a-4abf-93fa-d3bb4bff51d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(dpi=100, figsize=(5, 4))\n",
    "plt.scatter(x,y)\n",
    "line, = plt.plot(x,lin(a_guess,b_guess,x))\n",
    "plt.close()\n",
    "\n",
    "def animate(i):\n",
    "    line.set_ydata(lin(a_guess, b_guess, x))\n",
    "    for i in range(10): upd()\n",
    "    return line,\n",
    "\n",
    "ani = animation.FuncAnimation(fig, animate, np.arange(0, 40), interval=100)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45b986d1-1763-4821-b704-83010131c65a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video width=\"500\" height=\"400\" controls autoplay loop>\n",
       "  <source type=\"video/mp4\" src=\"data:video/mp4;base64,AAAAHGZ0eXBNNFYgAAACAGlzb21pc28yYXZjMQAAAAhmcmVlAABPrG1kYXQAAAKuBgX//6rcRem9\n",
       "5tlIt5Ys2CDZI+7veDI2NCAtIGNvcmUgMTU1IHIyOTE3IDBhODRkOTggLSBILjI2NC9NUEVHLTQg\n",
       "QVZDIGNvZGVjIC0gQ29weWxlZnQgMjAwMy0yMDE4IC0gaHR0cDovL3d3dy52aWRlb2xhbi5vcmcv\n",
       "eDI2NC5odG1sIC0gb3B0aW9uczogY2FiYWM9MSByZWY9MyBkZWJsb2NrPTE6MDowIGFuYWx5c2U9\n",
       "MHgzOjB4MTEzIG1lPWhleCBzdWJtZT03IHBzeT0xIHBzeV9yZD0xLjAwOjAuMDAgbWl4ZWRfcmVm\n",
       "PTEgbWVfcmFuZ2U9MTYgY2hyb21hX21lPTEgdHJlbGxpcz0xIDh4OGRjdD0xIGNxbT0wIGRlYWR6\n",
       "b25lPTIxLDExIGZhc3RfcHNraXA9MSBjaHJvbWFfcXBfb2Zmc2V0PS0yIHRocmVhZHM9NiBsb29r\n",
       "YWhlYWRfdGhyZWFkcz0xIHNsaWNlZF90aHJlYWRzPTAgbnI9MCBkZWNpbWF0ZT0xIGludGVybGFj\n",
       "ZWQ9MCBibHVyYXlfY29tcGF0PTAgY29uc3RyYWluZWRfaW50cmE9MCBiZnJhbWVzPTMgYl9weXJh\n",
       "bWlkPTIgYl9hZGFwdD0xIGJfYmlhcz0wIGRpcmVjdD0xIHdlaWdodGI9MSBvcGVuX2dvcD0wIHdl\n",
       "aWdodHA9MiBrZXlpbnQ9MjUwIGtleWludF9taW49MTAgc2NlbmVjdXQ9NDAgaW50cmFfcmVmcmVz\n",
       "aD0wIHJjX2xvb2thaGVhZD00MCByYz1jcmYgbWJ0cmVlPTEgY3JmPTIzLjAgcWNvbXA9MC42MCBx\n",
       "cG1pbj0wIHFwbWF4PTY5IHFwc3RlcD00IGlwX3JhdGlvPTEuNDAgYXE9MToxLjAwAIAAABLTZYiE\n",
       "AD///vdonwKbWkN6gOSVxSXbT4H/q2dwfI/pAwAAAwAAAwAkAvzHv6EHQzQgAABQic/gslcQVuwh\n",
       "+MaREpo9+FkePJq8QSINwOId5fFl+UVBnNKG4UaFYvdMs8zThSv1gbg8nHvh4Bo8IKAV87SmZNK9\n",
       "PXdpm2Pgtkww8rPKOEzf1HOFz2QV3BxQRN+UMSLA0MiUErHS+PI0/dG/oJAHvrS4fz/0KVq8KheQ\n",
       "5givZ3/VDSrfKHjYWft3Wnv1BBqtW3OgM5CRLn578lkDzRzTawIDgOm+I6IvyVit6s1Eumt0hafh\n",
       "SGeGw/0QSxbnpv7mT+VOo6Sc4fuhLxozcXEtBf8UtIu6Kqf+jfx7oYJK5JVPyeNU98Z2NocMpg4e\n",
       "wWP6yw4Efv77Q0m21Wi8Oh/9HKvkNK82iLwd9PNNQ06ADSYI5WdUXAj+oakqHfhyTXvK+/elrMqa\n",
       "znWNGAXYt1ZSgpIhyxsDPHtte7Bk0RNhxfc1Czzd2s/wlhSqQCE0QHHxf3tf0YZNPubLpEn3XvLC\n",
       "zGklYkkbAaAblzWkuJU70Nqi7K/4vi+OOfswIL67U7ZNfHE0sElc0eer2E/wSFz5UfAmMhcbpo8X\n",
       "yi8YFdIzQSkvZ4iABg03KIjVOPEOPowyTy1z+E/0wAuB+lK5dqRif5JxSrF1E/ia6+wc6kCueuUj\n",
       "q7KyAwm75hUjRs+q0cTRLWIZ35V/99U1t7T1cKGIccW+ZHE5D3nku5R5xDX9B8I1usR5pZz9Py0Z\n",
       "jyk+PkTSp8MJx6YzSeYE35FJWKlZGHcE3seepRXYTJNQoaJkcObLKwGV82LjewJZsGuR1wk0Pjae\n",
       "ZhTLQe8NeqwkkrXEF2ViUPuBdPHeW+pDKggYjrApgLfD6dH/yfgGr/YuzR2U6rR0hlssYz4sDGN0\n",
       "y7SbmO/23YmUjoqBEPSLsOVDvN9wK4l1e840cMc04QTchc+4oaxB0Y4DHw26wiVfBnnY2k5f94O2\n",
       "WCS2bFI5Ej3dIHJ3Pl1mPojMjGgfFWOnVkOlcvxqILhF3ZUMuvtdw2FjaNS3kBYYDbGGpcMdW/xL\n",
       "G2txJIs97MwYsFF7Y/+5rKQkctkxsfthvytRLrS1xZbe8HAlq4mxJfI1FBg1W4sdDPZkqg9PP8ZP\n",
       "ANlvmfop72URfIRiBs13u7JoiPXyJ905OiQQHg54hHAsstui8mgHEzNYt0yUkFjYwcguy8RpcpK6\n",
       "4QYO02qc6C0FdYeqeEkR4iopEhAsp7wVXHOHvQDGGXDAr7uU95n/S69VIDNwINLunMyjo8/P7nVa\n",
       "6BbUwNGroZyECdqbW+ax2BsyPha9vUqUfajCYjWcguAL6esp6CYPUiq4pTUP/NHRrQAtlf7o6WLt\n",
       "8YUsNa0OIUXb2hgA5GiwxaFvB5nicNajNYlr/70eorE+ccVIzkcvNO4HB8Xi7nOupbTXOugwuFUx\n",
       "aPaGAiNn2a/1/IvCqUqyO6Zjmy4pB9zD0fieOxnXryK+xB+8cm1YMDVvNNGCIAlOR5ZCCEnEGamq\n",
       "LZrIxokttzNHQC6An7KrS0crapnsQHL2zKuOD2ew+8Ir4Bq2CYIwZp4PaDNtpmsZmU6cAvrmS5CK\n",
       "rwmek2LTrF/xoqPrsBUhM9myhovKQ8pb4FQNdJyQ58T9yC9kaL8y36CWnMLwzSEb5s12g7jIHiEY\n",
       "zmzb1O7Smo+Y1xljjzfpg/6w9y9UUixg7ZErwGT5mwXeamKTWAOzfJ9wheJG43LqWYcPZvbKAZWd\n",
       "OOaLVnKiHRAaj2dEYNQhO7hWmHjPNR1UaRGmdAJW6VRJjDRNv6GwUoMzxNy7FMvXzmw+X4cc4RkU\n",
       "vxh6bBxPqImXkr4PsT9iWvS5fte+8E7ULg4scvRqYmSpvmQfxl+Un+bwa+ZX2HFS5aN0Ajs08X4W\n",
       "aZ5Nbp5I1YCPSBR50E2RH7aSw/l8MvhYBBH8cCvxEJmyzXjvh6gfFzT2uwSwjHC2hBc4TazHjn6P\n",
       "vq2TeDKgksT+94TLpcKdlgSU0jfKoAsCXRY1kB36Cnunez8HSXOY8n+hmh1HLvme7j7RNjgE9HoO\n",
       "mIVW/KiUBL1wjNGgSVOhnfxxOuIXGLKZaUWw8Aez5+0zrOKaZteUW5VzKgEvkP/nS5oeQWlfmz0W\n",
       "a/tbJgK+5tswIB5hwwxqWZiGbkfH6FXVgrF1xRR2kgDZWUMR5LMQeue5YN2S1ScS6wSBkIsVqs4k\n",
       "dlsI4BW3WUVZQ/WBY45r8mvk2A7ZAkFlIxndHoxeshtMIYumCt4yZTS8UkFrf2Uoym/B2AE7tydB\n",
       "qZEF3PGwm6Kk989niGxKmLy82vtWYIVXIK+lAwJg7ZyIbR1Z41TSPTWhtNahgSKHYpKfGI8p26XI\n",
       "8fU024MWygjTmRzIadyfsXAvjMlS66vK+OVLGH3JtoFWnyYX5jJoCG+mvhdyooD+ntm18s7P4RPL\n",
       "dlXt0Lk0SD6jyoNFlAbGcs6M8103cH1Cmku8lCu340m2TZpqeIDV9Lc+IhhbV/aOvAease3lwm8U\n",
       "I4p6rUN9or4sJ1YA0/xV4h7PA17wUq79QW5n0/a+sTrfx0xW/6MLZ4Wc0RuLF3vuCL4nBIaR5D9K\n",
       "klYszFu0CP9oAzPrCAv46fUpEKjA4GTBC/Y5PltQyDFSo0sGBUJMlLh5bfzdU9Pcx2SZfNTnnOHB\n",
       "m+QMHKyFNv/DWyB2EORz5iyxbmVKFnuvbqDIH0+eVcD9z32zvJsgqhnjG+IlKCDCmXAHwDPgYK8N\n",
       "cNDQszaxjPrG3ZOBAQkT6JJUaSaB6VXA9R8u13+GAwNxNw/Gcgb5mxg33jmp6b3EM6zzFJ4vLlRc\n",
       "ZI0fOxSYJUAxzv9+5Xs7Fjkdo8WyHsaM2N/G+7fTNo8SkPFCHc5/6b+/Vhjac3G2ePkrTfopAhEB\n",
       "PvBYujNu+HTC8B2nElh5IST1CDodfESHY7ixhJ3xVnN8C9MZomstlxyKKC5LaydS3DiAV/7fFSSB\n",
       "xq3mGtYI3amaWO7bKeHEw1BLR1Qe2uOtIdSwEN1HGcRhAAahlk9Gic0VwkHEKnYCo6h+0sEUqfCC\n",
       "zQd9N60a962AeBiYvvhSn3TRyMQRs8QtnCt7b6lRKn3m3ztfNe70sYLrrfTC/OyL9AEn69GFUSh1\n",
       "U5BEdv2DuyeeO2B7xS7MJCH6odiVQHOR/ZVVu8xRUD5A/1H9uBCa88W7JrRZ579jSNbowElcL3jf\n",
       "P3/47rD+TbkxvHUc+Y7+rX+hu95wDRzT1yQ8wch4CRsNqoxpE5lutLh0pBApTNoZZYkfJgL/ri2k\n",
       "5jfPnLiZ/U5iNq54BLMG3w+icrkfwHW1Is/b6hlqQpeykg36Q4WmjzvezeJuiOCLluHJM6Ttffbw\n",
       "TZpFGTFdCiH//MI87ygIsPDpGK4RBQ8mqOxJCBYiJgZRoE8gxF1MMqO5QdAnDnACg6FJ2I323buW\n",
       "Dd8u97O12nuiG/PWSMprXl6EMV1jjAfRGWeEXievt3bDVz5JNi/Mi679seDJjRqBI4si/fUqXfzW\n",
       "cN8GepFr0XiIBvLJiQn0XQSvjTKss9R/b68p/DX5FqpQ0PE4fZEBFt9Km5433OtyPmluot8kZSbB\n",
       "P7MyM7wQkiCueosH3vK2IjVo+ge/OfWcdpT36lXDjxtzxm+KdGun4nDhmglTG1GSvltTVBzGLe5B\n",
       "zXzIbHf83xkM6jgC0vYKap74TaNKmYroBKu6BgbqmwnjRtAaZp4GCIkpDqNvg+KaF54UJJKi/ZfM\n",
       "POV7avnmi2KPQAADqb6eJhZEtyrIa1d8AlXBNLwVCpXCbzZe3vJ4uyZIfRYNIGHwAHXmBsHIvlN9\n",
       "/J7rLO28jUR5N4J8nQ6XHAELNDk/hPeXam3/6s4fp5DU0Wkk7XIrjl9uN1m7mUehXKm6C1IdZUXT\n",
       "/O7hiNf5pZue1h9jMatwFniBtQ/viVqDnBw61wY6f5W54mMAvDMc46QHK8Q72PZVyuKyqIRM4V9A\n",
       "T0O3vKJWV7HkS9k6M+nw0haPHEe756ub4CnAUeg0hzd2+g4+1s14d5KW7QmwGOsPdLBoHFgfZKf9\n",
       "YNYqmmAbwPWFF63A2ofrqODXgTcnQxgyz1Ov19/D1aPIkep36ttxXN8Fa2ozP9w15BYNGnGLUvU2\n",
       "189/Jv2FHMv5OLvrA309y/Sej7s4h05hTtl1vKU/Sg7USqa3XOYf5tqKijCzdav0Oh+OONLfA1XO\n",
       "hiMObCK05o0O/IYwv10c2XhLiyJIZhft1DMr7vDG4TeXQcxhDdFqlA6YTli4VnXWVWtQPtU7A3+s\n",
       "7VuwJKolcLsWKM3l0th1XUElUaOX8dD/LtGQRUGjlBNNXf1cKV1U5+APgllvO0ZY9maNziym2HlO\n",
       "HTYxwnL5IgvNmruMBQQ8VtcmW7A65hDwT7R+WZUZqdf8nvvxYYZNiuSi02WgO/0OmXunP//0ZlHm\n",
       "I2ZYL2ZnZ/FU1jmFgxRuAHHlx7O8T/31ib/ddtv6OjFiY2yhun7xdxD9groOgTTrKn23pS4G0+/A\n",
       "Ek5E2UXolh2R+t6dFvYNdsUBKozURRF24c7eAPAio/gg04nYEGm4+53rDlF7yNplgK4x1ceM/Cod\n",
       "J9vznPtaGIxnAFAhJQi2YyWJV8gIp01gciPIDP2xTEcGrNthdxt+UZIAMDecSfW3R38otOkfDd5o\n",
       "uUvGXzkLAlEuc8dJCMsas2nsqToX9SCVKqcMzN3Qlu0gKyBLKYp86EgJBLOFFKIR2ugR5mlo1qcE\n",
       "lPdxDLQQqBUC8GqTy4fATnAJTV3U3AZQNy5pQCeb16Chvucqf/ST3zY9lgtWuUlLSO9AT9gm2cmF\n",
       "uP4K2MYQZlVUXaV+sgWfkD1AAARHNAEv5pgSHn/ucyx7KJHRkh2XLPAqtM+5g57/oSWhJSkeFEzO\n",
       "Cddibem7DKEgXqXOj6zcC7qn4xaLUDnXX5QlHVtbzLyvxIhvsyWA3vW5Y6/pH8vFuMNCLb8IYbIg\n",
       "AQfMlNX0hyNuBNPQKI/KBvLtvq6v8pHqBAa9Sfx24TDJsZAHlcTm3KSyA6RnKugryKKFOCUfRCk+\n",
       "kACNSYFFSE/K8jPBvo6NBt124Tre/1BYsIAoso7503XlNOk0r0r4zRX3K4RgerD27Ti5P6nUzkoC\n",
       "jg/sZAzBqAriwNTR65TWRDtqixTORDl+n/pwWlzXl93Pe+JrupBT1GUHuPWG/+yeSUsHbykmAPw8\n",
       "z/aQ4tZNLVHZ0656i1is6o7rTbafhjVeaXAf4l7yofrES7Lkhw2FAcsgsIWYtrY4pOtUCnQgjSOs\n",
       "m/xKnKmUf+2SdRFjnTg6jssJRuNssgyOwSo5zhwBiZACU3B3MzGK2lql/g1IA4mlRr0GJ2WS129/\n",
       "Rxa8Ym7rZXB97fDNaKJYIqgaixGZhu7firVMS13W5zxFQxAA/CV0uR5NfiPOKFfIINxSN1BAoPYw\n",
       "McL4M4z8JUeYCF3/Wycn30v6GGB/1CeZO+TDPhi4rZ4F4eScGZ58U0uA1FaMZ8f87H40tTZYLZuC\n",
       "VW0Z7IeekV/cwm3HcI2vXZagrK/+EsA9f97H6zs2FhML1cJbHT2t0LTDIhjLTp3JeHo5QQLw7Tnx\n",
       "UusX8xqV/ZlE3k1uW2A+sJfo7yQfsATxuHeFNzBQGk6JztIewYXzIjrsrbCGEg5mjy8AxqexStuz\n",
       "qOR77CMrFDBLe8Mm4QKNsx7rZXeKc1HvsyqoAM6SwymkGofbk4dmFL2pRi/WeTNoGqTL5WJ5KHAp\n",
       "I/479ECyyMn8eeBOJs9Ls7hDkNixoU1Q5LD0NS+rUzFyp0lChab2qJuo4SbB9OQQD4ueQI8RIyDv\n",
       "pQy0vefR0n6KogQZB2gGaNvjuPln9SzuNJAUeViI/aY8axErnH65Lw3xshnwQHFVkJ4G2vZ1gh6Q\n",
       "cR+NgWhblrp49xz7e8TSS9UVVGgK8GCuVJ4qpTaujxErxfJvYpSPH3kChZM0X8dWVosz+zFLnxOS\n",
       "TarnQwncd/+l6FyDVxE7SDLGnA23Cld49P1SJk5C7VXvlsol9ZyrckLP1T03up2cGUkEQoF65Xo6\n",
       "13EaOEGX+0uEqXeEAGIYnerCfI8xShOxbIgdlSNqP8yDMX/DOeRb5SACdmsgmxCLYlYtS9zRR7VJ\n",
       "J9ojSZ9On7gNujBZoISTjFRwsE7DsZ5bEdpJ2XV1oiywIKoM6tUKwoyFvRLQqZ/vtj0Py8cJSM/c\n",
       "6GqyLsPKUhFvr/vNmLXZmBg3fbEOpR+IsOsrqi927aY44t8Ik21u8dwcDrL3vm13mannVbPhJ242\n",
       "dcBx4pvgkYUs58/wqjSk4jF5bo9e1AU2PgDapByXQM65NDsgpGx7Igjmu9Fc9ZSv+PUpn9chqoAC\n",
       "Ni4mhie6p3JRzhYC6ggFn3IBlWBdcIAE1nnnOG8BFfDLBIxToMki5l+n+Z6YsYrlKwc4jHpEHCZ8\n",
       "jd7Srt1lcAo2llkO66ys9c1Rr2UCHLyOLM8BZQAAAzBBmiFsQ//+qZYAIgx/YAv92V53t7p9U6pN\n",
       "tKL/W8+/xZNbeACmXf9b+fdw+h30AhdoVzN3wo9//VHDwj2ngdHT7qRBmU40sDjYmcDaCGsnbtoc\n",
       "C5urHObfbYZ7IynZN3pDwCPyy76y795oG0Hx+ZjKC79CbgVnANdlI1tWK8YeK9SipF3LBfgp7WcS\n",
       "w3YOlaoaRG7cXop38paYOnXXQOANw+LWyroUg0N/YsNeX7g9mYUEHAvYOlrse8WEWppJ4/AlK5AY\n",
       "opq2VVb9vN34SxntO5FpiozQl4252DmWOilSGAS/ZetmgBUOCu67A4l8Li8aOdJhcAY3HpQVGPuc\n",
       "rF4BDx+vbagidcrydAzjUdaxV2pFbD5nd6IwVBVZNZcKLrQyAF8v7vHxczUn76jyJLVosj8DGI3q\n",
       "BkyIhHROS89tFrzNh8wW8PMEx08n41EyvFkJSGDMLm/05x7/BWGeX4NLLFpgCC+yzGjGcwSXv1t9\n",
       "2vLkNnwAohN42STUORAVOOndeS/TD6IDWYxOCLTUkX7xpf3hN7VD8uBeP24dyrimy/QcnAxqHiW0\n",
       "mx8S89uUP5CN+kC6FsF8P2NBarOL/ISoY3zvtXz41F0Ko/BRopzHLu9t3di2Xgunq/SeWUX7GpuW\n",
       "NTDWAZasrJ3dYDN9iQrgeuxi4C30iVVG7mJcAgRY0h0A0bYkey0cQloZRNLkaktXGVoI2/UXEnNB\n",
       "K8GgXoxaDVtJS7OcZ4p/Nm+wAvBE5Sy1P729/qQ6QqAmuRoZzao8kroySSvI2ZpX4NG8TUAVZzGV\n",
       "ZpXT2QtOtDtb7gQFcyyy/Ehohnd0wyiOkUgzjAKYU9bWTdlIJvqV00kU0W3n6yDi2O4aNEny78Ye\n",
       "b/ljE0gAAAMAAB9QYuiY4PHdV/3aKgD7FUvZjxMsEeR/0tBnlqb6/ASHFwTXZ0vbtWIKNvqPk0fm\n",
       "/RboGhsbBxCvmPFJ3+y29ugryUF6qccKwHJ6xCfgKxJlumFN5mPskj/174MSIunC0g8n8Koc+LuI\n",
       "OpSj3D1cynPVCRTwU/4cj1cotoqOaYySyyg9zJBaRVbyRVTD8cP6x3+RCqTFvl/FKZgAAAJ3QZpC\n",
       "PCGTKYQ///6plgAFTCbPdl3K/8PMQQAJ2+K19s60sSMCEM5knRq3dGDdMYALBA9kdxeTTaWRIjE3\n",
       "H4Tzn8B3uygsycZ6hef2KBMuKFxAyjISVqg3piBXbsDHP7taVnVWSHjnGRzd+Trc65x0HenatRzg\n",
       "HhM0uJEN6kO2+6e5hxbbS5A7eBfJ8F1vy16ZSvqiBoGt0eCM+opw5v3kANZ+ADFbFiOL+fzWPIku\n",
       "FlfA/EV+nqEMmM6PNt6E7787BMkIaKg2n8kM1d845+GM0NRgR0ja90+8AqNTa62IcpSvdksVKojY\n",
       "IgoCLHJ+hQr9evPTUaAO/b+Gsp5PD6VbVbTrr5Dz4DKK6ZW1U+iuzHhWQzKKhCnNMoSx5PfRxNqP\n",
       "PrzDZ3OYyIoXjyXalMuwNwj85stWGULGazZ/G8aakoky4MQRiyXebg89mB7Z2TXXk2VrpLOGnDiH\n",
       "P2o17OKE/XESTsqlcpMbeArlC33DjaHnFtT2u6A+kkwyU84oZHhKW3ZnDp6lYT5+xFSwEJpdKDMw\n",
       "D2/f7y5gdD4fVAO2+z3t5a1YIFUxSRcYH0AqRqoSSN/iUmeuX2UBWjnUEQbD3N7jnRsmA1VwwIAR\n",
       "3l/ed/WfiONc0K6a7r7rfKWfryWt0xQqAUvGPC+js/fmGWLg3++sgM2p4cYE6IvDpYUNSH7KlbwE\n",
       "DOR9HCy4c5r2BQ7Qn/yQNc1E0KxOg7gQC4Y5D8wFzganWhSpbeO5kC7nOqTvY29+qCn8Mis/MG+N\n",
       "qYfBGhkBr1cPyTtkCuqi7H4hYwI4e3eb3bUnC7zk1dHDUffwBQZYhzH8EADjn/4ABL9dLlUAD5QS\n",
       "FwAAA7xBmmNJ4Q8mUwIf//6plgAALJzm8KlC29yVZCE6J23eB8AO75/ICnSfCd1AFq0evTf4eHCQ\n",
       "FmfIldh+ncAI0g7Jb91ZGiTu93bDKX33Mr5uJ1hHz55PtHj5N0TWzE3bWbiMZ2PdzbAYArgfmEMZ\n",
       "/uLoqYM2adAzIPsGAyrrn7dL3FW5rOiI4qbbrrOtR2+4vFGEBshPmSvImId63AdDcEYWyweyR1oF\n",
       "S7/mdSiohBVKIe5NmOfcqCc69WfN1kRpew78QP8LwGn0aDuXA8d91jBUMm15b//SFbw2vTwpgY14\n",
       "az4ooNujAvXqoQMA4ssxgpL1NXRk2AWDTSIH0HlgyMMtRwOwD0RUoPneot4rOCLndMsXPL2yc7BX\n",
       "nX8ZEHW/LSHfNPGMg+dR3CLWpubpdLZKE2UTWjf1DBJCICZk0UtMEQ/0skAtEcat3dqs1mP9wbBP\n",
       "PN/xS78QHCtrnXQoiFe5AXY908wqhxBSqJBBnWloPvH8cTfcy9BXzilv46fwPKJ7FDHdqIpcSQHL\n",
       "sJbuaA0wcKnA8kGDfptUDLc1N0BgJDDjhGdLZmxcW0EMeJvwVW/3ccujYjDOdpZkSKSYcaQYd9dG\n",
       "lZFLa8lRwUt93iVZ2N0ycXmOZgjIyXUFy+I8nRGh/o9d5qWpuVWigW1t7MT/pyfzCBPGkJy9eVFe\n",
       "qBJsAlz7K5mZ73XbIYEhbCARlJ0kVDHeEJdYWtlY3WDG8BNeT6bFWr6IZ+Xv6R0vw2E0qOSzTYlH\n",
       "KVyGu2j64ZEo5rLmGAvFosQQ0Rb7OT7dvA8kLSyJC8GGJ/VzFzG5tMa6Ev5VsFs3Q+EtuPVb4DtC\n",
       "t9gkn4iLk0X4uawJCMRX78HEqeem4klUL+IqPnlWW+2IRKjhwNo3cbWxihkL1zxlEeX3EYhIlYcm\n",
       "N4NUDnDEU57EnORJVKGJAzpRqFMnamiY9S7mQ6GT43d9XSygEGZctJ9AlWDET5lXiBHUcBp8hvv8\n",
       "LUYJzTS17a4wST+6Z1RdkHQPTrsoTLBqI2k6iK8EFjaAcRElqPzuMnt+63yAqRLouSBurBP+bpzN\n",
       "TKJPFYpZCnLAR4yw2th8h/gf65E1Y9+BRyQmt5dr/iV4b9VHrMQ30RttiKi61cZsqZ04FBd9sOqa\n",
       "aYqc54QeyCwTI52bTE5QCRewCYxuDVYbLWtE7wbGGb/8r3/X0/U/izvVQS+b6EEoAj7DJUz6fKbe\n",
       "c6yx6Waxt6BCZxuGeKUP7GimRg5G7Xd0FIECssurvKBGsPlH8g5AAkThwhrk8sAChgAAAmBBmoRJ\n",
       "4Q8mUwIf//6plgAAtvTbQAcb48qWTAr6+k6SzemuMsj//p1yieuhzCvCrwZdy6Zh52UMATHpFAb8\n",
       "wzgmqQxdOD7Yby+qkKsJC/nzmyKY79SCP0ZDlgxqk80mJQUp4Wza3gny1/lqB0o8Hl7G9Fq6Ieid\n",
       "HgAmB9qaSqHnvC3ItYmCwUS1TLtxuEcAFt8yyapwpN1q9jfFAueuV+fff06gb1FbQPA4PEJejPEj\n",
       "6rvX154lbyK62Ly5Tf5OHaD9Vn4koB/tFRH8l+zkxAjKAAIkLsGdGXzxbwdpI+TqmAMRumieqHAR\n",
       "RzVRDoEBXwtYX431vdaKKRIbbBll9ybrOjqDcZNPTjAUOOmlHXJCYjdn1oRr3klGcYrErGk/WzDf\n",
       "i3F+HTWe1NgyfsO8DLKi2K4catgK2Neqrw8myXTI3IHetZeaqfqDvaGb8lRbnjPF34Rn2wAbBjst\n",
       "3m3Mv441rSDyGiEaV2zKKlwdx/vIPCMRAgkxsbHearlW+rY9t6yO/RXB353TsdLzRR6tqCmQ/aPp\n",
       "qmfM20UcEP0+Q3MMKXL+YJ3EUHDFJOJ4TM83m2Zf/bM/w/IYuVxmxRSQHVdavSEYUoLw3k5Xe8d1\n",
       "xY6wsrczRVNOZmgKF8v7clflfpIt2jkOX1RsRPP3TJWuAUcZNL8Eh7RfScxhUZwug/vKH5Kj8e1H\n",
       "1zE2Np/7WfKKrjSU97XgSlcWO4e7CoCsZwWQnXDwoWflKVYK7egT9dEGB7wI87bwQy90CiAKORg0\n",
       "BTBi/De61LMyUzO6oQChKcSPf0XSfEpsn1KwAABpnwhSQQAAAcxBmqVJ4Q8mUwIIf/6qVQAEwHv/\n",
       "u9ItH2KWCXBiJ5U/3c/c+5IcbSp6fWj84Ze15pEpf3HXPYBjhHKHyylwUCnkL4z4uzbjJWkbyvBr\n",
       "d5iveuaypRq6Lfn8GQLw4XBk3FisvEPE4yJAPUJsLbOGFv7Fcv+fqtmTK+La9tt1MOenVUJNo9wV\n",
       "T1+YlcvJZEaXlsnG5ffeWnpk/5THyX0ax8hcMA0xNXXAhDD1lLxykEH+91U8UzpUfaewc9wWQAY2\n",
       "rnvzB7WgFBGQaTyo3obcNTlpR9UEEl1o9fAUMLlid8U6apSYXE5WJiPL0qamXP5aCe3jv83fwUx3\n",
       "WvxnTvkQKQIhWhT7XQoDpIrTkb9KK6ZLbTzo6OjWSxzaoCrMvneRTDjoZk40cdXfr7JmHfWBxOaN\n",
       "cjetmwl633H7BYmak/8e1xMZV7hm38sSiTx+mDhdOdc7oOx6hEnpPXwYPsmXu27PeHr/YHZ1ujla\n",
       "I48lFjg1eXwKcMo8TiZSxi6DrzLqnSMqoC+osH9FOELpEk3DrLRlFYOHfnAn2fgQ2Ag9dJx5hpzh\n",
       "TIQOx9K8hsxUst3RgSSqO5KfobdYgOjhDt4Rh9AjYd9uYpAAAisaoFbBAAAB7kGaxknhDyZTAgh/\n",
       "/qpVAADungn6cIMoX7pfQJStkUU9JDe7feY2/w8K1AGwXlfvGWed9XBPBxrrEXfbGK1RUErYNgSS\n",
       "cNwIa6cDrCUqTtV0TDO8TIXcINfRRJO+JMCx0FHdF1Hq9CuZyxhPiRMAeLqFjtEGmBOdsbk7gm7i\n",
       "nF7ELco+GaMPTSoaG1r06UpBgEetFUKlQ9dTYM98CLNqtJyisqQPmrGuzpTLuAPdFSAHLvquAUdH\n",
       "+OF2W3WkKctYdk8jlRtLhK6Xfx0K89LWr6HFd5yjT+UwTPUcL6j/+i1P4szvX/Oj43okPX46QV92\n",
       "2lO1QTJVrLGVT2gOYeNL+LLMzLgP1YffhMXIhc5CF+E0+mQqlL6bOUyZ5OWzMWoDxC64QgdZT+h7\n",
       "l2FB8D/0u+tlLG1uuL8/biau3IRecyY1pLUcdqr00Lz7sPf5HZj6cSXMyKZuU35ccBmOjgoObOxJ\n",
       "02vGT2rpuTw4uGbCGXF46cZhRVaFcUdulWqJnbjb8nin2Rd47V7oiw9509mgy03Xk69izv004BPS\n",
       "+JmEBGDU7zXodjgEGZ2bcN6zpNorxjH6h7BrP7cuXHTBMhZyxMRzuOXaOwQqhErHabdvBFduZ3b1\n",
       "1K1o7G5Zh5VoKWcksd3kZSn+4ltBcd4eEM+BAAABtUGa50nhDyZTAgj//rUqgACINoPcavmq4L+Q\n",
       "HbJlD8N3vpLoOf5fiHLJSPpwWe9dRQECK1U8XygBIXp14dLa7G9VYn7jKhg4ltgfyq0Dd1DgowHc\n",
       "YHaf4dJjFEEF4QB++ghaMvuPmoI64NKvfiQEtQkKYkgwP5d30fBhm3NyAYAXPdGR0sgrxOq7/9lV\n",
       "2A2Ce4P/pmYrIriFm67valkXvGQbb1ak2rVBZNHxpRqcMDKOGtJuHHp97n9i8eYAmnG2dAkir0BV\n",
       "9mUNKGPbU0PQJVdUzYeWNH/Sb/ziBIDgDe4yaIYKGz4/TvH6IgvjTxDjnWj1kby59JP1uZm1NAKs\n",
       "T8l3Urg4wVJL4BhYnydov/5DPlfgBeZMt7QYziRGkMYv2yGbgNbxL/v+AOU8fkuSH1Ikmgj7bQWR\n",
       "CThO0/n5PHcFF0NvJVkw+q0q6KROiTLasxqmSZiEBKvn1ixmmPm/g0tRhVrwH9RSnT77tYscdwvw\n",
       "zW7RqL0SnHyVFuyizme6nN8jZv/UlP0rUDbdIs9bGP4m5JpE8kCtL7+vT0eDroTPnGOth3LWwppJ\n",
       "oxoZJg7VGUkaCxsxAAABnEGbCUnhDyZTBRE8Ef/+tSqAASj2GQ02ZhF/kdmrbT+K/E76WzBuLMSa\n",
       "by2wehC5jH0pEAC3rz59DnuYntiJum9NQgVrgnd0MtRvsenLOeNcM9eevaQZJ3CvJYBRpmhOHqBW\n",
       "55bMmMENy2X3PEt/nOUXh8wO516flJeH24eAtmhSxsgRaAuAbUiRZUz9B02xPeMssCNOKyfsEL5t\n",
       "pgT9HQXT8ETuK3TGJ0PTR+pJFMsldc7Zktj1hAY5klnZhTslfDhjKBDO63FzOMP/TlAmT5v3LEMr\n",
       "3LR6cl8x9ONwS5yTV3GGkyZCpzOn3FpHvgxxwunlQDOSZaEXRWD3K7RkcIliHky2qy8sleKNxguY\n",
       "GLOpAyVxzzzO2qcTDM25JrlMKdI6vR+4vBciB+KqvD4qpuMp6WYtqGlyq3Z/wozzMPlZHbzttyu6\n",
       "y36LX4rAQayPpqfcFC+qxrQAzilCVDifwbdQjwQ9LdQqHMTREEPH1ZfPV7+pe4zMb7K4FqEe08RK\n",
       "1urcF0cOcL4OxtOoWHicjBQAAPfiTiouRvggS8AAAACsAZ8oakN/AAZsuF71ikNXPt0vMWkb1x9+\n",
       "vkB97YYmcAETcsrY8+LUowwXV7rCRXSbpcVLFmSiEr3H4L5J8NSuTayofxVwAwpespplPHf+aN3u\n",
       "tLq6toOhChNbVz335aExG1dolHM8NsajyyLNvVxtictZ1p1a1gTFEn0m0jyb/oQvvsMuvxOcq52r\n",
       "1eCDvtzN13Mja1cqxqHi7+EAToyACDsk7JXAADdEYHhVJAAAA+dBmy1J4Q8mUwII//61KoACpk/S\n",
       "ABEGwL/2ZcZMAKENvLjctt4+t04N1Ekb/PzPLtkR7TJdaLtL5WQqCqGWvR+PrHyHHqXYECmxE0mk\n",
       "sNLRsaP44axAaIeGMEWuUkWVAH+TGQF4lwAJ2UiJaPiY6IWaI9YrBb26KARf0eq7hh2i+lzsGmvM\n",
       "+HFHuIm/gklOtzM8p9mEFhyBFaSTwimP2DMy7XY6GD5oJ/HitbCWYeBpNeq7PLrR30hJspN4+J23\n",
       "hFSOZuA1D9Ia4gZZ3UHvQvZnGgGbnYHiaaHghgRERni1XdeucrAxv1zqQf67yFHmdtURDq0Pl/1/\n",
       "PMBNliWJy8LIIqblRs7lO5KPKB5RBnupyIv/uBdxPLnZMAYTspniviVQ2f1g/VfgT7zLEQ1oAr4P\n",
       "uryd4fiWpSUAIbb3QpTZWOj5RZOo0HMwEd1+Ut9+ZTWnuCkaZ4rRPFhwquF/Z+K5ggVHPO0uyQMm\n",
       "g0mj8GUaVzluHBsf0BiJShJlWrbdY+nGa2xwhy9XBPnbWNpt6K4a/hjUNJpptA3NDCUXvb7v1fll\n",
       "nAOz59yzeZm6y4eLWdHzokl1I7Z+Hgu6ylNk6AF8C2Okge9hQrZBReXM6gdWs1MelJk0w8azdzn8\n",
       "yF12TYVVaeInImC5FXIeV0VOkb9QXbIgOQMScSjWZaSp3hXjO5PyIWdpGSWoHj6imP/3lk2sIL9e\n",
       "7K9S7csV/7njtR+VJH8Ts/BuTcl35b3IBpDbWOWoaJb+RaPaGWTV2Mlfx5a9KLssGWB1xII0YQbN\n",
       "nEZuttGm+1VSr6eyyGnW90S2eG5Xc8wxrg/T3rUxhA0bMrBjFjVRgxl9VZrJZuLWnVh4kdmnymGp\n",
       "sni6Cl7VMfYRw71ZZYouOv4TzR8+1/YO8lENJHEtbHf1nONpoXT//+0hs7mKDyhhv8EnX26dfnzR\n",
       "RkgEE6/tnak/9XFkzPHJUiVw0XmR6q4NXGsEWs3P0TNZe+IG0e2STltzV+rbR5xP5WmRwNdylesY\n",
       "puFAZRMr7S1E5yxafE0qhipnDki3gBBb5UO4dO5/10I0Rw7Pox2iHQNASbmaV/YSyAiK/RmdOob8\n",
       "vL9Im5iBS7LNcpfGVmgGOyP0q20x8TqG8+jY5qjwSW5KqNiLczqimzkSnyuE7u9gUjVtE4CtSEid\n",
       "quvjzXwHECCNN+gNjqKv8+xuJo/Z7zUChq/5NS8IzMrijFtQDiZL/9e2pHnau7zorI6P/cjdiq8p\n",
       "9Gc7uTE173ydYCjX7sEVuk3ttrnldO/L5TaWSyi/Pqh6mzfsP4ufnu1EDkTtDulpQsVffWxRmTcE\n",
       "ahx+9GuVhbsSHhEAAAHRQZ9LRRE8O/8AClKvXmyu/0CsusYepfTROWqxVn8iRZPN7NYNj8k5xflI\n",
       "QA3Fajpu5hXPtbk6kQ4zY81k6ghQtLlIOGx6SHtT/hZvSDhaVJb8JPKDHqu27gcne9fwDxPj0H5j\n",
       "ccYjNyhVLPGqHATgDeeMJADh9Q0xQuNAqDlwT3KTyOXTfRT0qReHy+ExmXHvvHyo9hPN22clZo3a\n",
       "2GscGADLPcWBXD4em6cesefNhNNHdUtKiDJ7yPWiH5ClXnZb0aNKURS+bjYg6JhEjgd37uJjw5N6\n",
       "iC7cev95qEaq1bVh0VyMZjJ60McJwt/BUjpmn2vksYJ8dHwiuYS2TmSik8EHKIMNoEnovPBd/Av2\n",
       "/HQNgqTk6vB6jPMlpG5ukEyUntuCjLcVjF9ugWm1UVugOUJqeFucABI+KOQYBqKCW84N00UKrhlv\n",
       "/h2G+LTvTwp2k4M0sDh4b3wmMU49HK+sePGf2KEF999Ja//r8UOf4AuCkhJsjX5TTwwO0PYD+wNw\n",
       "z5vzITsgw5vkPWOndhHUIpkZEG56GK28O2eWN6beXVqQRgt9mRandyGgkKhv7yBrZLJ5FK7YyuOy\n",
       "+mA6T+DvkuMkFly7NAA9Lfo8e0Cio4j4AAAAvgGfanRDfwAN0hM0KQgVeffbgAC6UBnKC5rGYlf6\n",
       "Dxw77taniZ3x9tbAl8aVeL1gPrAu3gbxqVX11dT6SCn3ugRsFfADfk+ksTiwbwMkvRFPvm9CCq+M\n",
       "wrraNg8RrycV2mBHt+ztvA4wPO4+/MYYmF08oaETzDNlpOfjaOLNv4hU+dpXmY2l7OnZoaUNfBjF\n",
       "mwkAk6q0vAUOSC5Zac1jTVfI5xLqY1eSYvR5bhIynw8z/T0qNvJQ2QAAN8HfGfAAAAFTAZ9sakN/\n",
       "AA6BQgtyABEGfoeHBsVGP0HdHnyJP6sn/S1ml/Tl/m7Xs4REV3O6061uflNRkISN4E2NbuO+3ClX\n",
       "dxD5wv0aDAthz0KiJFz+AXggJyUY5h3amDMoZlBtStGtUInsinQ8b2ZCQ9U9jXWsJtSc7GKqzq3+\n",
       "/fDnhJi7Mr665/vrrja79VwHTr9pq/LITcjDN2ulOWX3eOxGH3Aw8Dplj9jNYh70MooBf1EsUXqs\n",
       "3DjAnxT5VzOuXS9rnxVUDG5nCZ3Mz80DS8GyXrzMRUfO2Jv4iO6lWWX2ucevayj5UGVQDL9It3xn\n",
       "3K5ASv7A4lFsk8/i3wf+0uMCrTJpCnDD72kxOXnoshc6w6L8LxnDvfzBVNJNHrJj62TfBZjEv0JH\n",
       "YLVODKLyxElO3gfCU+uyCpeWCAz3LrWvJn6LaUyici3VFrhuxzHXRTMwAGMDCjBhAAAEdkGbcUmo\n",
       "QWiZTAgj//61KoACzcR/wjl5ACD22HYUXXpXJUX+8HdFr3AAaq2ePe+jWBP9J4DwWpfd2Fd5dXQz\n",
       "jtuz1s9vmyk7E2LWgQFOG5oxzr3z8uZr936NUCGtol9pDZRzGUQhg6cvrXwRHDKAPByXm9Kf83At\n",
       "JCN7I6JhtrAbdgOl200MQuyHdyqv+9WKefJrraXDazpQ3S7Bd8vf39S1rp/l9nhHMMjo62D+R1Du\n",
       "6u4t7yRERpb/vLDjXb638j29L6SCOwdYm59WTvU0a68+0bq5/cauDoISEzhTD0ATXjawH8IJkq+e\n",
       "HVjmtRXGjz+dTTgW0fRjNOocl7PTBaJbVLWNY3eOYOznLUBCyT1XkmLXd6VewyWBcTZj7aml+mqZ\n",
       "IQMJEEsPtzJnaIjJx3U1tQBNEbhjzyBb+oe8qFojmIsWET4LJmEbX9jFjbFcfTImS67sUieJq7HO\n",
       "GJ6KYCf3/mzw1opFwqLO0XT5JmrDALaynzlqltuV4cra+lkTj8J1GpqWPJXgVpuCH1TBwHO9UmfR\n",
       "1qYIOFgSAsX/fLLSlCcuLX0MTOSHa0S4GjArrTgj6M2eUmt4VfcgMaH7XWQBUZj4qNxCprr3O/ww\n",
       "WZ0Czr07+s/QgOcaogXXnPc4TRRjl6Nx83/IlOKlXZqCDfHypYYT+ZrBF0ij0Ycijaba6+MsSVBZ\n",
       "lgBZ3PwStDy07QAn5Hv9E6LMIYp+XLJH6z4hxp558wg+6p2iWorbKsMuF3dSgOovsrjw6Q2a0CWU\n",
       "NxWg9yQfR5WWdD0d+B0SoyH8KJenjZvQ5nGmwHTsmrnNoLURcdlLVx0JYx7whTOngFlsutIWY6wn\n",
       "7A2VUfMLqb27H4mCCgcwCMwNi1D7L06wgAIBE7URMjlYFto7ABG1G90/2TjKM3zmd9gRVE6GRsRv\n",
       "xWpU+BGYKq1L8EzSxc2M6wPyiIAy2OPucedN5o24oU7YOqazbn2/XFo+KWjS3q9zW25BHITLXlNa\n",
       "yzVYBqQIX2G3d02S9t1g3wKbB+XDEb46BWOAFpFxDAf5nWS3nczTUH5+kRe3mCOOXBfHIVCBVDzL\n",
       "3qajUL4dg+y73UXIquk+SeVMtPRPLpSx0vqwYWKhRDiuOhqffpfCR3n6ECq3pf73Uxut2KcqO+ZV\n",
       "CAvQl3X5ozQHB3INnb94QGip4h91bb809qGTFfSqSGthn3+GasnOmAmDzfkLm2XfTMnYxrmRra94\n",
       "LEf+ASwpd6HhdH8FxwKmmiIQQ7UftLeIO8Mm7qcFS5Vw1EPapC1u29pgC4XXWyaysJ1ChBnWMP2I\n",
       "lflssvj52nAejYl5cFLPhdRGDMZEsgtzOfsaumG4W/nHK6y8cRoL/AWcUpsd5AHVIq4GA2Tn5gJm\n",
       "ZogjAkc/u2TRyCCpClOknCuHkI/KCqHXHMcyyXGFEdeRhklrG245/S99wAegpMG2VevypauoHGcb\n",
       "FGzHKMAn/avxLcDWE4/CE/JuiKt5i3JFTx7QLVXbl6HBz2R9DOfGLoZ3nxhVc8gBbFjhVDbhAAAC\n",
       "NUGfj0URLDv/AAueMIdyxxQFpDwACHEpHqZAVwK2/lneLsY8uCCu98gNu/18Qq+ExRBmAt5gRjbo\n",
       "ghTV9nIBBe29h6eQVGD8qJOzhSzVgAyyOKKWZMly4JXFwwWwyC6sq9JE2Gz9vyheXpoYOG98PV5g\n",
       "FxiXXiLSNpTp++nV4Gbu23J118V//1QbDUEe7ZWecBM+n5QyOH+MSiuTJm8OJ6QLCKPdOJUHWPQv\n",
       "+qNIRLdwQTn+7vl7it3ETJZJVkySlei6rg1AqcET2vfFcpaM/jj87FAo60OXsiDcsC1MNlGBAPm4\n",
       "oxStYjAJf0GmjCRTqE0b4AlhITiKD1kDnbOJcumSOvk4kSsJy2CoQMRRyT8Ac8LTQlA5S7QHL/Zh\n",
       "ftlKmxNgJFzTkMaRTwu5OLv6sBQrKBBTs+NxfLW1wT5eewc68kwaES85RJipjk/NlVoUsC2IzrYY\n",
       "TeXu+4OZ2bAGd9YrTAEdjrzfW/NZspOeAsmMN2xWvlvK3ROvZqpPNyAUkrMTIOIhOLmF5Q7UtUpa\n",
       "wq+KGK3elL1fRQE2NkoWUyzDXC3FHxKFHpNFiZQo0hcY5dvNmsj/FQ5SWPiAuZr9BHgpwXLJgC5F\n",
       "9qlvIF6QAx0ywCd0fuOXODZ8OzohymHXNbn5KmlGHhJ7a59zz3DALmtfDcpuWwrOZSvSMY6E7Uxe\n",
       "yO8vR7N00FzIsZg1Zau6v6wq2F9Wsr5AFHE6AAADA4b3jgqygbXl+rWg/WQeMdJA8lBY/bEAAAHX\n",
       "AZ+udEN/ABApAG5SWa1aQAIdoVwlXu42aQVgEHMuO9qoH2cMtAkFt6ekTGDLpDSBQxTkzYA8QPKi\n",
       "vBcNi4onfxsk5kDzAldw8+6K0W217FbX5+S17Jvy89qgCOD99J8bq9HlwVPCv0M8z+NqAD26lCZm\n",
       "p3z8eUMz9FY4ICPN25Sbdm5buCMgPEr0sBJgqr6VSLjrZvTYvr1t1/3Sos8S68H1V2udDdn7C4OZ\n",
       "Xf2VWuBsK0iVW6Y8kmLbk7QIzMSwkwCpthQiV33mc+2WPpcvXFagZAzsd1mjDEOIOuZjE3HWLARE\n",
       "3kaMhHx4AfKBq12vtMf81mVNJDrIVvnkWGIOwgYgeIuhQBnXRjKB76v9IAFdGzONARhnwZE4iQfc\n",
       "BAyoAz69XBbhagr9HzFHlXUDJYbaKJFMwlU2wYRejz3eFrqxXVgefUVpqgTAGEvbe+GZwPJUJOSg\n",
       "yETSBu/hVijtpFRbDkyf1x4d37fR6C/YJSpl/TFtFEPK+EkmXkJjRFzAzM5S6dAR+ms0tfVILmfh\n",
       "07519tJZHCNHD/BgChJqyvCVOxYv9tAqNevkuzhzixfmCLbGdtdQLmM9uElAalXYkUNbG2lVP1co\n",
       "h62mC0zkFACKQcEBFAZcAAABFwGfsGpDfwAQKQBuY7kgVRB8ZEInuOF7HdOFu4WvtONLXC4w3ouD\n",
       "jcHI/sXpOm3ROHRmU/BmGSTN5M6kOm/slqMMa3RWC8xHPgJvDnJk+/qqSXECnlkS8uXWgAs3KnZ2\n",
       "Buyc4FPcC4LIxUpfx/hAs5Ow2Wzmwvn2HGlVBX+lHParaOErWL3wwPZR5RQUkdY0vqrSy3blkfap\n",
       "tqMtqt91sqL6j+rfQ//z6oBhm/13ZrzYGS30Q4ED7f43RLGcLdoHJUtkBGT/vl/iNT6JBFJouKo5\n",
       "zIKA78Clj4ytJnAiuXPV/sEVO5fS/RjbSv7JsnupETWO4GA2Vo10bODMxO8qmoNTM6a74BiJbPCA\n",
       "DcB9ViDiDvx+CCT/QAAABLxBm7VJqEFsmUwIIf/+qlUABbfE3gB0rwztLh+c8AIkNgYE6x+BnWvq\n",
       "kbyWsoLElTWjgdz7YSw9/MMAmHNwqKrkNMHZsrJ+VXRLHFcnxfP+eX8cCt7OMmZlm0Mf/a6FqDd5\n",
       "pV0qYmXNC7jeNqqOI1WKemtV8C456DQONF7NtFJV0pWko9BqolD8bTw4vY/1/BwBOEfLbRvBNwIJ\n",
       "estHxE0hP8sbm89Rha2xXnQ0Y5fYtnSzcBnh1xLC4K01J6JZGqvORiNiJwaBHPAPkSY/hvqBCOT8\n",
       "XNSzFqidUytGlbgiKfADMv3yDkMklK8IJfftpLf/oPmQu78iVLnciAsz0YLBF50v8yMiAaTgKWh2\n",
       "FHb0M4LiR4CJqm9OcJMBVT5aXADsk/3XhFMnKv70RuJ19WojTktqrrFB3mWa5uXQxSAABhNT5QXM\n",
       "RYTirvKF+aS923k2WmHY0AV/jJo9K5bOccPh93oxYw7q6EmvuiMINschMDiLP7TPpDwYyhc706lj\n",
       "OEh1Bku0YwTFFivKMFUkGmwvLB2wIvy+72noox1UjUhe1NZegpWUu4HAu0tmCr/Ionqk4dbktROk\n",
       "RWUWKoYWCazW5BsfbcSlQeknIUJxwq8GKltamwlNC6naht5RwFIxyOCGrJTXPh42KteGFfAf3VJC\n",
       "RgYZGNf+N14P2i40uJGhJyaIOLpMftVA4a+kSB1hQpGzJ/fBvG7fp5ist+3UE/4dwX7bccZG00Ti\n",
       "Lhx0iJJXYTz8WAQivGXsWIoJ96Jh4BrOp8ifX+eFXjPQG+/lT8bP707bTciy07M/rP1Fa0WuMTWC\n",
       "ZzIHCNpD5j8r8EUre++q+hC/hugJ7UvJdWIa9VFY+8yjtQqHtAWvWRNSniuPIv0ms3CjbLuj3Vaj\n",
       "7vQvoYWIYzmECeyV0EUEtY7Gy1DbO7e+KRNT9p25U0WCs6gX7lqriZOQngWb3XJmAe4lO7odypwL\n",
       "rGaXgLm0+37rsm6iUmjktzUfQ8pYRKTwqcauEtm4O2pQwW2WjoXQWiguNMsjZT2Q2Dm672dDsyCz\n",
       "hMAB9R5DF8+PqsigmQAuohoMmrvqYSORTtR/1ETlLSvUu4w3unyo/ewCD3zfCkbxC2taWiXzSMDk\n",
       "ABs9hvvmIM++c6J2PMy4KZaO9EBu5uXPDlfIHKhAGigQvvsY4n2gA0nIyvLIkpfudA49VY3KyvpP\n",
       "H++iPbz8r0BgzLNBw8iRWNlOPPWemL8/wyIOfqBV0Q62+mV8l+XnG6CeAajWD0yFIHK1RtFWYtY1\n",
       "KVtLM6TlvnHI1DkQ1LOv2aawwQ9XRlUe96WZ8N2OmGx+Yn8jMWtO5ZaDO2fl7OcD/DnGJapCgwJZ\n",
       "+fgQVtWSQJrThouCO/SP2qtqs7JhkcgI0MCXo9YHTIQ4VFtr2WjeJ4QI4r1KBVI7Xzs+Y47Trck7\n",
       "P0Y+G5rfKE1eF+IBzdrCj7+qzLa8/R8NvPKMRLf+XyMKhJah+gV14D690BxXH+cxIkH8CIO/EVzn\n",
       "9Tm0RpqO4Up0QN/gZ/68kmhn1iCu6TKP3v2EdFtKSxSTp9FUG2VGQus+7iSJwOyEAn/ECyE8IRfq\n",
       "3FH21umCjt5IyXfDdK9wlhWX99xluwoTOJhFA5Bq7KkAAAGdQZ/TRRUsO/8AC54wjFdmGz2adjZH\n",
       "ViAAXSfedLyIVqyWcmw0PIzT4dbXL5YF+fAc0PrX8HevVg0J/982TTs2RgGG+K5j85PWwrwLKFj2\n",
       "jjK2H+DVcmLtHjIP2vzAWd/Ym4/ojoq402QEaTN2xM8fpMZm8shG+yPa8FBrNjntLfOX0isnHRc2\n",
       "RYyOBsM2rCG85ufShRyM8bk+KXadttxBPrSCy7mJ4j9dg2SoPLZiIvwjoU8DA1XU8RHp0XrteevG\n",
       "z3Tp+bh13ZtoaAOlLl6oV7KteH1W7fLhDcve1584ejtBFCARkHtR2d47poeiygD7uBxanhqIlVS+\n",
       "ra1lZLYX3PTRnjMG9SxJKn8QhN/ZMR4AouxEHmtZ8OnkcvtYCjHPWDbgqUW8z6pMBYxgCiApAn17\n",
       "bnJyMqiz57d5M3eJFabdfjjRhTxHHcZRMqRCGbFmzweBP//j0GQUMffsQWoAiL7gCzJ2Hq1DTjbp\n",
       "q3j9l3xL+w4AFubE9jIf+R5mlceId9//gdG0XDGnXSY1a8wjcQvLJegBZh4Jnik6HBgAAAFRAZ/y\n",
       "dEN/ABApAG4hEcoilj1PpABqYvMs/Urh2pgp12ne5tJuOYJ9wPugRw+OWjL+GDAZmuh9Y2mhpmDS\n",
       "N3MPljP2/r6gvMMsE6DaZCcBSkM/eerQHPN/rfFuu1PRAVqWL/P4UyuWkpDN9NoebWZUTRetQvZj\n",
       "X6727BwTItr7kQ4u4211wW7qy1xuHsHsptBy92aa9A2wcRdquHgR6b979F5kSdvWyAqTi/QeVoJn\n",
       "SFwqK+OzMaGo2aGXYfwmxxVldgyibTYMSuQ8qdOW3Mall3T4s636ciKAulHXhsiSnp9lMSjpqn+R\n",
       "Lwl4AzxUUjBLYJSSHPp1Yb06LGI0bNqlS1EMP1kcg8YMHEVEV3T/JVphHqDVWefsFI4DoJ72APED\n",
       "ZhhDx66p3nKff0ymAbHXnh/vwBObjh0m9TPI/hQMKnMn+db81ZyH0x13CfKRYAAfkAAAANEBn/Rq\n",
       "Q38AECkAeOLWFjjXNOm+sNgA/bNDXcktIoueVoRgPaTHoMndBxIfSRW7T9rjTxIoxKsobAqHFYAL\n",
       "svrdquNJ3cVmBbn9oDSYkY5uaNmPnBRlkWKM/DvBFS9rXaQIXUB4NfGtDEmeiWH0JcvvDVPZi8Gy\n",
       "SEi2pL+if9xZLzOL0Qc4QvnFeTGckPhqlUSvg6/qZFczN/dHsKaQKB57pqhayLvCiOOzjod2yp6x\n",
       "pgQjYX5igYTI97rsYCamxDTE+j0kYQISAzueuuTaMAAXcQAAAl9Bm/dJqEFsmUwUTBD//qpVAAsx\n",
       "09u6BAAHG3LmzK1WrwKV1wXendcFqfQV97C+clX29CNDEP06r3Xa8Ej4Qtq8KFnBfQKgvIxiZjBp\n",
       "5vWx9v+xiWvMnrvu9V+8WOV3rzSPjcyyYe5+IizRSP6PV8ECL1BHhN3NWnCmadoTOngpEJWVDCpB\n",
       "jyYfbK+ALzE0ZonawuJr55CVkECL/zU6JuH/ImrpyT6aq+T1K56trabROh272aP76OC1sVa5/gBt\n",
       "npWb9EWMQHaNhOoFmwgg2qKQfsLPOSDRIY3U3787+W4m2MOUVTYmQ7qHqOeMn1yPGHYEvulECDFq\n",
       "U+fALO0VOu+xTrWnvx+XCxwExQ1vJDU+tMnDV++eZZlUv0l6+UY++FC+2GssC23t+NqvNd139kdt\n",
       "sgwns2hE0yRiSikU6xUEUkd1eusT61Hd7mO0vHbLJAXG4DKL0e7Z4GX0cjqewc8zWcz+H1pR76R/\n",
       "5ZNqhGteag1oRNZJwhGO4CP+moyLI0yZeyAp8ap36OzW2gTxw7DKSz2SMAQKjc7IrDXySGMvn5Tc\n",
       "dhK/1X+7PCDcNPFTKcUInJyKuo+MrLbhL6R1IkFjhSCE+Q/dsZrC5IJFPNViVVZZJiM4C0IgFq4M\n",
       "l2SrB4Tk0TTZi8chNKv0x/URP/iYrzP0J6hLLLxR1nNPPQ6F4vacftSHEZtYA8MHD3pIvw084W0m\n",
       "ydP1usizNrVlNRkkQB/sO6VXSQGSplq6egZyOKFhgwwFky7lzSYhQjpVqbpukqtgDwxqW6ziadvc\n",
       "dNIBWIRjmLkibhHqpHUqu9bMAAAAnwGeFmpDfwAeXbpLIEbG59cC34NEgf4AAfyJ0J/D5EQT8PAw\n",
       "VVY4vfUG3V1UOo77g+ZDGbVdKj6yt0x/PjA5SnZ+bm0jGoRCshnqjGMEopRX806Cj+4qODhp12iG\n",
       "yof4zBAni4v/2FPvJlMGbq1ivAYDodQ8YeSIIWsgywkkBJSsU0FIbl/hgc82ovyQQbdVPHKsklJS\n",
       "H3ABRe5U6/lwgQAAAVNBmhhJ4QpSZTAgh//+qlUACyePgACMrwzsjUBwSCnq4BLtKGDU8/rUNkGo\n",
       "LYRDzpiIriInsX6/vS+SvjV+DrskB36PEaEO5hEXNUwFKY1AWvopTKRxfE8WtQsBakyH5Jb3bFan\n",
       "oYHLjgOuFW5/jvzROoclntZL1twPwrIPYq5f0sJL+I+5r7gtNIS3/RoFXDLwUTwRNirfI7okEayX\n",
       "EEqIyUaA8sFs1EsWp5/TbUAlTJimOO0Rn7YT8RXIPM6gjddcBfvzUBYD0CBLrkhxGYvL4wpfW9H0\n",
       "Q6JrkxJIakzVI6/XS2flsdaxOOrXnicnLzNEf1lFb7tPldTqg0EZAi9EeGQe4XpmR/Jiwy2V3TH0\n",
       "pB/YnppDMEh4N/b9XW9eRAQR6qshH6MODLCS+Ps2jg7h6zCWSDUOs8NhN0JzkWeX6vl97knJbHVr\n",
       "3ZORRHs6EAAAi4EAAAHTQZo5SeEOiZTAgj/+tSqABZSIDMAImsepfPf9EXV8wfUciamDgfZ3OzBg\n",
       "2jXKxRVsM0m+LvL3WocCVQeL8W8bcREJfokPey2Je+NxzfG7kwGPOpiqVSdcsoWGSc4KyY6s3MzB\n",
       "8l21J30fmSxWssIMZgsVOP+4vwT4iUPgklbS39fDEcNjgKpdQWeyciETDLfbFk0gASEk+Qur1FAc\n",
       "ansYAI2/2v8Z3wIc2EptPwC/rLnMTklHE/mV1nxONyPIduzH9KJ7e9iqszQSllenEvMouUwj95Ka\n",
       "N6VUcv9G13s3g85QLLRu4xB0Lr0pWmcSJhIOLe2bD787GcbFzliveJkoxmdqOUndDXxIAgmJ6u9s\n",
       "azHuBxxxHIvS2Bgvs06w/z1gXnEFcTS0R9rh750jrwExxzzjbSIFgJb4JjyGtrKe++8FBtQMQFF/\n",
       "QJdZDYo4agN1iVEOJNnWLi2i8HpEHqqcDdgjmKXPyKmemV02RPFbGjokytT6/5vW/NA12WtwgE98\n",
       "rZVuGljridnPBoSYidNFJgPN5pmISBVPIxbfI/nUNdHVPMbZ46ylp9zxi8WNMRJl2nQ97hYDuJuy\n",
       "Fw9JazZB6c1TJ8sAFlPmmGNidVe04b/UBgQAAAHcQZpbSeEPJlMFETwR//61KoAFlDT6ABbCMCp4\n",
       "Y5wxtQIV2IehuR68+yShm04PIXNLN8RCNNNnwbQU1Rr6sS4UzkSNNSGJkdHZWgGktdMFa4N2JmB4\n",
       "OI05Kp7/3LQ0NcypnHl/QZrKDzsWApdJN0fr/8HzWlcoIu8GlqRjbPOVvKXD01/YOufUnu4KusmY\n",
       "TL1QMEkHptWBrapAvPTDXjjPNcVByGeotq690v8kO1DxCh5ONYf9BYZESoQxaKuTxWeD2LEOyoyZ\n",
       "k5zF7oSTAiZ4462+z6PvbP1QYkSThZoyN6qZ3YbP4XMi9hOy8ytUptPWgaoK5twr8x2vg/OPc0SE\n",
       "Ex8CdbBITTsBq7eRMJmGRDVeoydCPDL9HsyJqWY6gRCBd7FP2YguwB8EF9RHZ/+C1keDi9TinBmz\n",
       "06SLCnPBqjQxrZ4m3VNXzzhvS6G++3juaZfJVBesGaqya+qsh2kmiN5+lPhJPnfAt5e2VTxZRMYP\n",
       "oW4+IzGlBxW9w9aeFCVM0ytq9HvcvhYklP07NhINj64PjuDHPSvk36b58x41UAfEay+Lq8H0Ozyd\n",
       "YK9PnWBddT7ugBxRwb76p4iW0dP/92YDnwM5e7Pfw71tQReYJwjDkWaMkNeVfm4AHzEAAAAlAZ56\n",
       "akN/AB5cZ/wt5KaSXBji4XLygUwsDbWK2AFaa/360AAH+AAAAWlBmn9J4Q8mUwIIf/6qVQALKx9q\n",
       "AANzz594U1LrzbZFJvpm/HpJ9/7ytuao12zTVrwfYkDynT21y/A/eEE8uqF1f71KpV6DM8SI6aU3\n",
       "ezTNaqlf1aEhdVfXdZdFsiHo8+urLiYvCWDLpfEcOsXwB1HrpnB3lpTNE2iJONWsxAjNO8+2l2hy\n",
       "21aMB7BG2sYKMx5OPFU8U7tmzNmxmYkX85V9PbHpn6hPV1RrXHSkyGaIa5tCBMbgp8q7h81iHLLR\n",
       "FQsneY0cXPdWwhcv5T7zSXFi9NXNPIWek3QG/yQ8wpBS1D/+gkVTyj2dyPWEIpczCRRDX88ELHa/\n",
       "GFOmwXKMnOrwDgi29tl2OJjK9aRurslYNiHk16rpqEUAVdRtdEJiTxJOAlR+7kXm1bGzMtiKiIc2\n",
       "pndPID02TBmSQ/VRNayXD2etR6+NpCwCBwh9MCzmn/uYp4uSP4zD1AxzbTo35lYmvojpw2e6bEed\n",
       "AAAAbEGenUURPDv/ABXBde2HTft8veuh5+SBb+04gTvAzmEAJoiuxi59gEBPFLnXCIul4ssF3L2N\n",
       "/X5bBf0OBEwjW/HgtLV/CJIc+wvPgcbRl0X17bF504eHPA42b+e5NwMWI5RgAmyW2zCPtOIbcQAA\n",
       "ADoBnrx0Q38AECkAePRL3PXIATp8ir/itGAg/c/gHPTkkpp8QcaZrnfv/NBYiM0d0uNz0TMMf1AA\n",
       "JnFVAAAAKAGevmpDfwAQKQBjiWvYMP+NurHrnGDAIodNVobsgQnKZRKoAAXPUVAAAADVQZqjSahB\n",
       "aJlMCCH//qpVAAsnT+LgGlqABKtLWF3DiqWWxnqiNZitJvGar3furtNvov+8apYRDV4fARcatkLV\n",
       "Iw1f9tVYc6IOTH1IivMLPY1fINzl4Qm+w+o33wLMLgmfZjycgDiF1PKhntGu9J4L4io7b7GzBfEL\n",
       "UcWT33nW1H8f+KgVjAu4Q/jWG/RYh599gC6csTYzoZ2EmJBh8bYVhCN0Y1ilViSCnQIBT2OEi1qO\n",
       "IAQgwWOTKvMD2Fmr572qp6GSaLnYU0so/TUWhZt3/X8AAAelAAAAMEGewUURLDv/AAueMITmvoRh\n",
       "PIkldJVhxndHGJT22UKpeB3nXrSXIAAUYp7V2LMbaAAAACABnuB0Q38AEFfbgHQm8WmU3zSzn78o\n",
       "eLNNQAD8q+uO4QAAAB4BnuJqQ38AEFr2Y60vy927OpYn3cSsu/ABWrfY4BAAAAAjQZrnSahBbJlM\n",
       "CG///qeEABfbULyUlZBSUWpvCP3xVxtAAeMAAAAoQZ8FRRUsO/8AC7uGEqwnc/2A6IPzO1jtK8e/\n",
       "G1LQnFgAFyBHBkhBwQAAAB8BnyR0Q38AEFfZg2Tlbj+CZlbqNGaaHI2gAJNW3gERAAAAHwGfJmpD\n",
       "fwAQWvZjrdsmvOTyVND2z5qdf4AAOxo0GRUAAATPbW9vdgAAAGxtdmhkAAAAAAAAAAAAAAAAAAAD\n",
       "6AAAD6AAAQAAAQAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAA\n",
       "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAAA/l0cmFrAAAAXHRraGQAAAADAAAAAAAAAAAA\n",
       "AAABAAAAAAAAD6AAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAA\n",
       "AABAAAAAAfQAAAGQAAAAAAAkZWR0cwAAABxlbHN0AAAAAAAAAAEAAA+gAAAIAAABAAAAAANxbWRp\n",
       "YQAAACBtZGhkAAAAAAAAAAAAAAAAAAAoAAAAoABVxAAAAAAALWhkbHIAAAAAAAAAAHZpZGUAAAAA\n",
       "AAAAAAAAAABWaWRlb0hhbmRsZXIAAAADHG1pbmYAAAAUdm1oZAAAAAEAAAAAAAAAAAAAACRkaW5m\n",
       "AAAAHGRyZWYAAAAAAAAAAQAAAAx1cmwgAAAAAQAAAtxzdGJsAAAAtHN0c2QAAAAAAAAAAQAAAKRh\n",
       "dmMxAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAfQBkABIAAAASAAAAAAAAAABAAAAAAAAAAAAAAAA\n",
       "AAAAAAAAAAAAAAAAAAAAAAAAAAAAGP//AAAAMmF2Y0MBZAAW/+EAGWdkABas2UCAM+fhAAADAAEA\n",
       "AAMAFA8WLZYBAAZo6+PLIsAAAAAcdXVpZGtoQPJfJE/FujmlG88DI/MAAAAAAAAAGHN0dHMAAAAA\n",
       "AAAAAQAAACgAAAQAAAAAFHN0c3MAAAAAAAAAAQAAAAEAAAEQY3R0cwAAAAAAAAAgAAAACAAACAAA\n",
       "AAABAAAMAAAAAAEAAAQAAAAAAQAAFAAAAAABAAAIAAAAAAEAAAAAAAAAAQAABAAAAAABAAAUAAAA\n",
       "AAEAAAgAAAAAAQAAAAAAAAABAAAEAAAAAAEAABQAAAAAAQAACAAAAAABAAAAAAAAAAEAAAQAAAAA\n",
       "AQAADAAAAAABAAAEAAAAAAIAAAgAAAAAAQAADAAAAAABAAAEAAAAAAEAABQAAAAAAQAACAAAAAAB\n",
       "AAAAAAAAAAEAAAQAAAAAAQAAFAAAAAABAAAIAAAAAAEAAAAAAAAAAQAABAAAAAABAAAUAAAAAAEA\n",
       "AAgAAAAAAQAAAAAAAAABAAAEAAAAABxzdHNjAAAAAAAAAAEAAAABAAAAKAAAAAEAAAC0c3RzegAA\n",
       "AAAAAAAAAAAAKAAAFYkAAAM0AAACewAAA8AAAAJkAAAB0AAAAfIAAAG5AAABoAAAALAAAAPrAAAB\n",
       "1QAAAMIAAAFXAAAEegAAAjkAAAHbAAABGwAABMAAAAGhAAABVQAAANUAAAJjAAAAowAAAVcAAAHX\n",
       "AAAB4AAAACkAAAFtAAAAcAAAAD4AAAAsAAAA2QAAADQAAAAkAAAAIgAAACcAAAAsAAAAIwAAACMA\n",
       "AAAUc3RjbwAAAAAAAAABAAAALAAAAGJ1ZHRhAAAAWm1ldGEAAAAAAAAAIWhkbHIAAAAAAAAAAG1k\n",
       "aXJhcHBsAAAAAAAAAAAAAAAALWlsc3QAAAAlqXRvbwAAAB1kYXRhAAAAAQAAAABMYXZmNTguMjku\n",
       "MTAw\n",
       "\">\n",
       "  Your browser does not support the video tag.\n",
       "</video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML(ani.to_html5_video())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7dddcf4-f5bf-485c-bcf0-7fd258d8e1f9",
   "metadata": {},
   "source": [
    "## Numpy NMF Implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "144741a0-a8bf-402d-8f1f-54738cfef899",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2034, 26576)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectors.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53dfd1fc-1cb8-44c6-ae32-27be6007e119",
   "metadata": {},
   "outputs": [],
   "source": [
    "m, n = vectors.shape\n",
    "d = 5 #num of topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "444fdd06",
   "metadata": {},
   "outputs": [],
   "source": [
    "# W = np.random.randn?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9b17939-7cbe-4ac3-8245-b03081737829",
   "metadata": {},
   "outputs": [],
   "source": [
    "# W = np.random.randn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a95ca9d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((2034, 5), (5, 26576))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "W = np.abs(np.random.randn(m, d))\n",
    "H = np.abs(np.random.randn(d,n))\n",
    "W.shape, H.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee5891a5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2034, 26576)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "V = vectors\n",
    "V.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffdc9e08",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.0"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(np.eye(3)*2)\n",
    "np.trace(np.eye(3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "897120a8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 1., 1.])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.diag(np.eye(3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dae3ef8f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "26304.31003242264 8.189374727876224e-05 2.087590827597957e-06 0 0\n"
     ]
    }
   ],
   "source": [
    "# Frobenius norm\n",
    "# np.sum(np.abs(V-W@H))\n",
    "def frobenius(R): return np.sqrt(np.trace(R@R.T))\n",
    "\n",
    "def grads(V, W, H, lam=1e3, mu=1e-6):\n",
    "    R = W@H - V\n",
    "    return R@H.T + penalty(W,mu)*lam, \\\n",
    "           W.T@R+penalty(H,mu)*lam\n",
    "\n",
    "def penalty(V, mu=1e-6):\n",
    "    return np.where(V>=mu, 0, np.min(V-mu, 0))\n",
    "\n",
    "def upd(V, W, H, lr=1e-2):\n",
    "    dW, dH = grads(V,W,H)\n",
    "    W -=lr*dW; H -=lr*dH\n",
    "    \n",
    "def report(V, W, H):\n",
    "    R = V-W@H\n",
    "    print(#frobenius(R), \n",
    "          np.linalg.norm(R), \n",
    "          W.min(), H.min(),\n",
    "          (W<0).sum(), (H<0).sum())\n",
    "    \n",
    "R =  V-W@H\n",
    "frobenius(R)\n",
    "report(V, W, H)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2f62417-1c7f-486b-8218-f482793289a6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1.43858379, -0.7124614 ],\n",
       "       [ 1.6209956 , -0.1244286 ]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "T=np.random.randn(2,2);T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4913b7d1-1c0e-4192-aa7c-6992e3ff2a83",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1438.58479296,  -712.46239837],\n",
       "       [    0.        ,  -712.46239837]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "penalty(T)*1e3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2776f09-20a6-4710-bae9-67ac87f52ddd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.01"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1e-2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd5aa3ef-9c4a-4a48-a073-53a1e2062315",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.4385837929618248"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "T.min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6d0b074-1081-4b2b-840a-67e370782cb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "upd(V,W,H)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a92ab8a7-e9f9-46d0-87e7-bdbc2518f708",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1684343703.54675 -1864.8739728368212 -152.72994046790748 10170 132880\n"
     ]
    }
   ],
   "source": [
    "report(V, W, H)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53d8a629-3d44-4194-a6cc-7dc1043db8da",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "44.42709338245277 2.0082546532603725e-06 2.5518343448689233e-07 0 0\n"
     ]
    }
   ],
   "source": [
    "W = np.abs(np.random.normal(scale=0.01, size=(m,d)))\n",
    "H = np.abs(np.random.normal(scale=0.01, size=(d,n)))\n",
    "report(V, W, H)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fcb9cdb-9885-4d6f-b6d7-26ff6819754e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "44.41913949000226 -0.001166507815926114 -7.840976535951406e-05 156 280\n",
      "CPU times: user 1.61 s, sys: 1.78 s, total: 3.38 s\n",
      "Wall time: 1.06 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "upd(V,W,H)\n",
    "report(V,W,H)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b246c703-ac39-48f6-8a41-e3cc38dfbbd5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "44.413605139935 -0.0007657338245319698 -7.283834887410182e-05 124 295\n",
      "44.3753063735921 -0.0003872330392145567 -4.58065321347774e-05 38 514\n",
      "44.34679042913337 -0.00023315718246128998 -6.448792691161058e-05 30 977\n",
      "44.315143320648524 -0.0001187202450460459 -0.00011718420987392124 28 1549\n",
      "44.28030523243511 -0.00014848956987308697 -0.00012402489019729026 32 2224\n",
      "CPU times: user 59.8 s, sys: 48.7 s, total: 1min 48s\n",
      "Wall time: 32.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "for i in range(50):\n",
    "    upd(V,W,H)\n",
    "    if i % 10 == 0 : report(V,W,H)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "554716e3-dcea-4b80-83c7-96c7eb7dab46",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['does like just know think don god space',\n",
       " 'does think know god like just people space',\n",
       " 'know does think like just don space people',\n",
       " 'does think know like people god don just',\n",
       " 'good know just like don people think space']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_topics(H, vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec16d454-7e98-406a-acc2-4c07baa2ad52",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "c73638f3-48b2-4fd1-9bec-6d3dc7b3ce73",
   "metadata": {},
   "source": [
    "## Pytorch Implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da9e6896-29ec-46fc-acfc-3be902bd4045",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "import torch.cuda as tc\n",
    "from torch.autograd import Variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d409d78-f515-4454-a998-235ee923f2d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def V(M): return Variable(M, requires_grad=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "faeb3884-172c-4d89-970e-9ad44465db96",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectors = vectorizer.fit_transform(newsgroups_train.data).todense(); vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6be08b98-752b-494c-b105-c0450ead4242",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v = vectors.copy()\n",
    "v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8751d13c-241c-4714-81d1-937c32650ccd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0.,  ..., 0., 0., 0.],\n",
       "        [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "        [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "        [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "        [0., 0., 0.,  ..., 0., 0., 0.]], device='cuda:0')"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t_vectors = torch.Tensor(v.astype(np.float32)).cuda()\n",
    "t_vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e478cfa-1323-429b-9fa4-340ddfbf15f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def grads_t(M, W, H, lam=1e2, mu=1e-5):\n",
    "    R = W.mm(H)-M\n",
    "    return (R.mm(H.t())) + penalty_t(W,mu)*lam,\\\n",
    "            W.t().mm(R) + penalty_t(H,mu)*lam\n",
    "           # (W.t().mm(R)) +penalty_t(H,mu)*lam\n",
    "\n",
    "def penalty_t(M, mu=1e-5):\n",
    "    return (M<mu).type(tc.FloatTensor)\\\n",
    "            *torch.clamp(M-mu, max=0.)\n",
    "    # return np.where(V>=mu, 0, np.min(V-mu, 0))\n",
    "\n",
    "def upd_t(M, W, H, lr=1e-2):\n",
    "    dW, dH = grads_t(M,W,H)\n",
    "    W.sub_(lr*dW); H.sub_(lr*dH)\n",
    "    \n",
    "def report_t(M, W, H):\n",
    "    R = M-W.mm(H)\n",
    "    print(#frobenius(R), \n",
    "          R.norm(2),\n",
    "          W.min(), H.min(),\n",
    "          (W<0).sum(), (H<0).sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d28a5926-447e-4101-a3b2-5f4bdbd41120",
   "metadata": {},
   "outputs": [],
   "source": [
    "m, n = v.shape\n",
    "d = 6\n",
    "t_W = tc.FloatTensor(m, d)\n",
    "t_H = tc.FloatTensor(d, n)\n",
    "t_W.normal_(std=0.01).abs_();\n",
    "t_H.normal_(std=0.01).abs_();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2fa823f3-28f6-4a85-bc68-d6bb35984159",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([2034, 6]), torch.Size([6, 26576]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t_W.shape, t_H.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38e9f745-57b8-4b10-916d-a81911e93c1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 0.05"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fd26043-418b-407f-8031-f0f949e4887d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(44.3929, device='cuda:0') tensor(-0.0065, device='cuda:0') tensor(-0.0005, device='cuda:0') tensor(1579, device='cuda:0') tensor(2166, device='cuda:0')\n",
      "tensor(43.7250, device='cuda:0') tensor(-0.0070, device='cuda:0') tensor(-0.0132, device='cuda:0') tensor(2066, device='cuda:0') tensor(21653, device='cuda:0')\n",
      "tensor(43.7064, device='cuda:0') tensor(-0.0061, device='cuda:0') tensor(-0.0116, device='cuda:0') tensor(2179, device='cuda:0') tensor(24659, device='cuda:0')\n",
      "tensor(43.6964, device='cuda:0') tensor(-0.0086, device='cuda:0') tensor(-0.0084, device='cuda:0') tensor(2396, device='cuda:0') tensor(25231, device='cuda:0')\n",
      "tensor(43.6673, device='cuda:0') tensor(-0.0049, device='cuda:0') tensor(-0.0051, device='cuda:0') tensor(2482, device='cuda:0') tensor(24241, device='cuda:0')\n",
      "tensor(43.6627, device='cuda:0') tensor(-0.0043, device='cuda:0') tensor(-0.0045, device='cuda:0') tensor(3044, device='cuda:0') tensor(25558, device='cuda:0')\n",
      "tensor(43.6616, device='cuda:0') tensor(-0.0050, device='cuda:0') tensor(-0.0038, device='cuda:0') tensor(3151, device='cuda:0') tensor(26004, device='cuda:0')\n",
      "tensor(43.6601, device='cuda:0') tensor(-0.0047, device='cuda:0') tensor(-0.0044, device='cuda:0') tensor(2952, device='cuda:0') tensor(26325, device='cuda:0')\n",
      "tensor(43.6583, device='cuda:0') tensor(-0.0041, device='cuda:0') tensor(-0.0042, device='cuda:0') tensor(2852, device='cuda:0') tensor(26202, device='cuda:0')\n",
      "tensor(43.6558, device='cuda:0') tensor(-0.0050, device='cuda:0') tensor(-0.0040, device='cuda:0') tensor(3208, device='cuda:0') tensor(28970, device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "for i in range(1000):\n",
    "    upd_t(t_vectors, t_W, t_H, lr)\n",
    "    if i%100 == 0:\n",
    "        report_t(t_vectors, t_W, t_H)\n",
    "        lr *= 0.9  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6bb6712e-8b82-46d0-9b1a-8ad94d91108f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['sci gov orbit data station launch shuttle nasa',\n",
       " 'real ve know time people think like don',\n",
       " 'christians belief does atheism christian believe bible jesus',\n",
       " 'know format windows program file image files graphics',\n",
       " 'don does religion values think people moral morality',\n",
       " 'vice queens sank manhattan beauchaine bronx tek bobbe']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_topics(t_H.cpu().numpy(), vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91c0e31f-2ce6-4d01-8058-cd987d8e2236",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0xdd9826a2a90>]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAhBUlEQVR4nO3de5wcZZ3v8c8PAugqXhMvC+gghKO4ArJzWFCOuHgLuEvOvliVuLug4mb1AILoWaNIwLi7iiAgaxQTBCRAuIiQcBIIIQkXc58kk8vkOpmE3DOT2+Q2k0wmv/NHV/d091Rfp2amu/J9v155pbuquvqp7p5vPfXU81SZuyMiItXvuP4ugIiIREOBLiISEwp0EZGYUKCLiMSEAl1EJCYG9NcbDxw40Gtqavrr7UVEqtKCBQt2uPugsHn9Fug1NTXU1dX119uLiFQlM3sj1zw1uYiIxIQCXUQkJhToIiIxoUAXEYkJBbqISEwo0EVEYkKBLiISEwp0Eakauw8cZvLSrf1djIqlQBeRqvFv4xbwfx5bSPPe9v4uSkVSoItI1di8pw2Aw51H+7kklUmBLiISEwp0EZGYUKCLiMSEAl1EJCYU6CIiMaFAFxGJCQW6iEhMKNBFRGJCgS4iEhMKdBGpOu79XYLKpEAXEYkJBbqISEwo0EVEYqJgoJvZg2bWbGbLcsz/JzNbYmZLzWyWmZ0bfTFFRKSQYmroDwND8sxfB1zi7h8DfgqMiaBcIiJSogGFFnD318ysJs/8WWlP5wCnRlAuEREpUdRt6NcCL+SaaWbDzazOzOpaWloifmsRkWNbZIFuZn9LItB/kGsZdx/j7rXuXjto0KCo3lpERCiiyaUYZnYO8ABwmbvvjGKdIiJSmh7X0M3sA8CfgH9x99U9L5KIiJSjYA3dzMYDnwYGmtkm4DbgBAB3vx8YCbwb+I2ZARxx99reKrCIiIQrppfLsALzvwl8M7ISiYjkkKgzSi4aKSoiEhMKdBGRmFCgi4jEhAJdRCQmFOgiIjGhQBcRiQkFuohITCjQRURiQoEuIhITCnQRqTru/V2CyqRAF5GqoaH/+SnQRURiQoEuIhITCnQRkZhQoIuIxIQCXUQkJhToIiIxoUAXEYkJBbqISEwo0EVEYkKBLiJVx9HY/zAFA93MHjSzZjNblmO+mdl9ZtZoZkvM7PzoiykiAobG/udTTA39YWBInvmXAYODf8OB3/a8WCIiUqqCge7urwG78iwyFHjEE+YA7zCz90dVQBERKU4UbeinABvTnm8KpnVjZsPNrM7M6lpaWiJ4axERSerTk6LuPsbda929dtCgQX351iIisRdFoG8GTkt7fmowTURE+lAUgT4RuDro7XIh0OruWyNYr4iIlGBAoQXMbDzwaWCgmW0CbgNOAHD3+4HJwOVAI3AQ+HpvFVZERHIrGOjuPqzAfAeui6xEIiJSFo0UFRGJCQW6iFQd18j/UAp0EakappH/eSnQRURiQoEuIhITCnQRkZhQoIuIxIQCXUQkJhToIiIxoUAXEYkJBbqIVA0NKMpPgS4iVUcDjMIp0EWk6qimHk6BLiJVQzXz/BToIiIxoUAXEYkJBbqISEwo0EVEYkKBLiISEwp0EZGYUKCLiMREUYFuZkPMbJWZNZrZiJD5HzCzGWa2yMyWmNnl0RdVRETyKRjoZnY8MBq4DDgbGGZmZ2ct9mPgKXf/OHAV8JuoCyoiIvkVU0O/AGh09yZ3Pww8AQzNWsaBtwWP3w5sia6IIiKZNPI/XDGBfgqwMe35pmBautuBfzazTcBk4IawFZnZcDOrM7O6lpaWMoorIscyjfzPL6qTosOAh939VOByYJyZdVu3u49x91p3rx00aFBEby0iIlBcoG8GTkt7fmowLd21wFMA7j4beBMwMIoCiohIcYoJ9PnAYDM73cxOJHHSc2LWMhuAzwCY2UdIBLraVERE+lDBQHf3I8D1wBRgBYneLA1mNsrMrggW+x7wr2a2GBgPfM1dVywWEelLA4pZyN0nkzjZmT5tZNrj5cAnoy2aiIiUQiNFRURiQoEuIhITCnQRkZhQoItI1VGfi3AKdBGpGqa7ROelQBcRiQkFuohITCjQRURiQoEuIhITCnQRkZhQoIuIxIQCXUQkJhToIiIxoUAXEYkJBbqIVB0N/A+nQBeRqqGB//kp0EVEYkKBLiISEwp0EZGYUKCLiMSEAl1EJCaKCnQzG2Jmq8ys0cxG5Fjmy2a23MwazOzxaIspIiKFDCi0gJkdD4wGPgdsAuab2UR3X562zGDgh8An3X23mb2ntwosIiLhiqmhXwA0unuTux8GngCGZi3zr8Bod98N4O7N0RZTREQKKSbQTwE2pj3fFExLdxZwlpnNNLM5ZjYkbEVmNtzM6sysrqWlpbwSi4hIqKhOig4ABgOfBoYBY83sHdkLufsYd69199pBgwZF9NYicqxxjf0PVUygbwZOS3t+ajAt3SZgort3uPs6YDWJgBcRiY7G/udVTKDPBwab2elmdiJwFTAxa5nnSNTOMbOBJJpgmqIrpsix7d/G1TGhPrseJZKpYKC7+xHgemAKsAJ4yt0bzGyUmV0RLDYF2Glmy4EZwP919529VWiRY82Uhu3c+ER9fxdDKlzBbosA7j4ZmJw1bWTaYwduDv6JiEg/0EhREZGYUKCLiMSEAl1EJCYU6CIiMaFAFxGJCQW6iEhMKNBFpApp7H8YBbqIVA2N/M9PgS4iEhMKdBGRmFCgi4jEhAJdRCQmFOgiIjGhQBcRiQkFuohITCjQRaRqaDhRfgp0EalCGmIURoEuIlVIdfUwCnQRqRqql+enQBcRiQkFuohITBQV6GY2xMxWmVmjmY3Is9yVZuZmVhtdEUVEpBgFA93MjgdGA5cBZwPDzOzskOVOBm4E5kZdSBERKayYGvoFQKO7N7n7YeAJYGjIcj8F7gDaIyyfiIgUqZhAPwXYmPZ8UzAtxczOB05z90n5VmRmw82szszqWlpaSi6siIjk1uOTomZ2HHA38L1Cy7r7GHevdffaQYMG9fStRUQkTTGBvhk4Le35qcG0pJOBvwJeMbP1wIXARJ0YFRHpW8UE+nxgsJmdbmYnAlcBE5Mz3b3V3Qe6e4271wBzgCvcva5XSiwiIqEKBrq7HwGuB6YAK4Cn3L3BzEaZ2RW9XUARkWyukf+hBhSzkLtPBiZnTRuZY9lP97xYIiLdmWnwfz4aKSoiEhMKdBGRmFCgi4jEhAJdRCqOu3P/q2vZuf9QfxelqijQRaTiLNq4h5+/sJLvPb24v4tSVRToIlJxjnQm+iUeOHSkn0tSXRToIlKx1N+8NAp0Eak46m5eHgW6iFQsVdBLo0AXkYpTqIKuoA+nQBeRiuVZjehqiclPgS7SQ+t3HOCWZ5fSeVT1xqioDb08CnSRHrp+/EIem7uBhi2t/V2U2NEusjQKdBGpQKqil0OBLtJDpvDpNeqHXhoFukhEFD7RURt6eRToIj2k8Olfizfuob2js7+LUREU6CIRUQU9eoU+0+172xk6eib//sclfVKeSqdAF+mhZAU9u8+0lK/Yg579wcW7lm1WDyNQoEtg5/5D3DdtjUKpHGpz6T36PZZEgS4A/OCZpdw9dTVz1+3q76JUrUqInkdmr2feMfAdZud8JXz2laCoQDezIWa2yswazWxEyPybzWy5mS0xs2lm9sHoiyq96eDhxKGrRjuWrpLq5yMnNPDl383u72L0muyDoUr67CtBwUA3s+OB0cBlwNnAMDM7O2uxRUCtu58D/BH4RdQFlb5RTUe4a7bv6+8iZKimz67SmZqxylJMDf0CoNHdm9z9MPAEMDR9AXef4e4Hg6dzgFOjLab0tp7+/Ty/eEufnpiauHgLn7vnNaYu395n75lLvs9u1todfHTki+xt7+i7AlWRxuZ91IyYRFPL/tD52keWpphAPwXYmPZ8UzAtl2uBF8JmmNlwM6szs7qWlpbiSykV74bxi/i7//5zn73fiq17AVhdUbX07vFz78trOHC4k+Vb9vZDeSpbe0cnT9VtAmDy0q0Z81Q/L8+AKFdmZv8M1AKXhM139zHAGIDa2lrtfCuQV0mdqJK6Cip8yvPhW19MPc71NVbA11tVign0zcBpac9PDaZlMLPPArcAl7j7oWiKJ32l2q5Hkmzm0B98PJXaBFgJO/ZKUEyTy3xgsJmdbmYnAlcBE9MXMLOPA78DrnD35uiLKUlHOo/SdljDnJM7oEr6M1amlC/XR1foiFEnTzMVDHR3PwJcD0wBVgBPuXuDmY0ysyuCxe4E3go8bWb1ZjYxx+qkh657fCEfGfli4QXLVC2hVEk19GJCpZhydh51akZM4s4pK+k86hzpPBpB6crX3tHJ0NEzqd+4p9ffK/vzqbYjxkpRVD90d5/s7me5+xnu/p/BtJHuPjF4/Fl3f6+7nxf8uyL/GqVcUxp6p1dHtVV0Um3oFVRHDytJKR/r4SOJAH/g9XVc+stXOOvHoX0L+kzDlr0s3riH2yc2lPza/YeOcOMTi9h94HAvlExy0UhRqU7BHqgiauhRr8/gjZ0H6e8xXj3ZyY+fu4EJ9VsYPaOxqOVz7Zij/n63tbZHu8IKo0A/BjW17K/6EaFdNfSec3deathGRw+bOHoaPpV0tJGuJ6Uq97VhO5PZa3eyevv+YL2lr3ny0q1c+LNpzGzcUWapKp8C/RjT1LKfS3/5Kve+vLpX1t/a1jcDaKyr32KP1/Xq6haGj1vAr15e06Oy9LSnRfLlldJ+nCpFGdtVei+VwtOHjZ2T+/VFvMfCN3YDxPrer7EK9Bmrmnlmwab+LkZF27Y3ccjZWxdwOvcnL/XKerNF2ctlV9DOu3lPW4/K0lPJbamU8xnJk709qqH38kFHKR9VJZ1I7y2xCvSvPzSf7z29uL+LEWr73nZmVcChXqEgrJbfem/8cfa4hh3R+1dInhPFQVBPm5Gi/D1GsYPqiaWbWpm0ZGvhBXsgVoFeCQ4d6eSG8YvYuOtgxvTLf/U6X31gbj+VqosVaHy+5sF5fVaWnoiyl0uPa8Qlvn7Gqua87fWV0rc6tdMs4zPO3obOo8642etTPXmyZb9Db3wEEbbSleXvf/1nrnt8Ya++hwI9YjMbd/D84i2MnLAsY/rOCum+VYnd/cpRiYfPxZRlbtNOvv7QfO6asqr768t+X+fOKSvZsPNg4YWzXvfI7PXs2B8+sLuUpqRNuw9y6Ej3AW/Jz+SPCzZy64QGfvfq2pLLGJnK2E/2KgV6P2vv6OShmes4GlGvE3dn+srtOQelWI7ufpVSK6wZMYl/eiD3ya+k3jh8LrtHRgnLJtvr1+88kHOZ5G3Vwnzml6/wL7/PPNJb23KA0TPWMnxcXQklgTXN+xk5oYHvjF8EwPIte/nS/bO6jUQulKntHZ1cfMeMjPt6Zn8m+9oT27Qn10lzd9o7Onu13/q42W8EZYnvlS9jHejuXvHd8+6ZupqfPL+c55dsiWR9r6xq4RsP1zF6RnhN6LjUYXRlWLltL43NmVdMnNm4s0/LkF0TnVC/mTdCArd5b3vesM131JOcl+/IopjK6NqWA7y+JvNczNHghUdK/K0nmz92H0wE3Kj/18D89btZtHF3RlkbtuxlQdBDJMyhjsR6ZqzsuupHrvrBUXfaDndyIOtzdBK9WD7+06klbUPq9UVs+sFgR7V9b3wvNVW1gf7DPy3l7wtcrvWul1Zxxo8mhx4Khlm/4wB16/v29l17gj+mgxFdn6UlOHzesCv88LucLnY7cxyS7z90hGFj5oSGXz67DhzmkdnrcXeG3Ps6n737tZJeDz1rcjlw6Ahzm3LvNG58op4v3tf9t3XBf03jC/d0L2u+g5vu8/IsXOZeNvkZHFfiQVbO30JIOa787azC5QgtW3JH1nVkeO6ol/jobVO6Lbtow56ushV53FPOgeXRSmqni1jVBvr4eRtYWuCGCo/O2QBQ9MWsPn3XK/zj/dHevuvmp+r5aJ5rr6RqbhG9X7Hr6XYSKmSZ9o5Orn5wHn/9Hy+HDsaYtmI7s5t2cmdIe3A+Nz1Zz8gJDazYWvha5m/sPEDNiElMX5l5yYNUb508f5xbW9tCTz7e/FQ9Xxkzh+a9uUcN5qqJl9q1sTlHbTDKSDma6h1T2q8oe/luz4tdXchy2ZPSn4edGO3LjFWgV7j2jvDATtZYerPVZWtrGzUjJvHnrMPgGatacHf+tHAzB4rYoUTdhJ378L/4IfP3TF3Na6sTNyIJu0DTcTnasdftOJC6R2mYPQcT7aTFjMy85M5XALoN+rECTUetBzu46GfTQ69Dsjy4OUZbR2ltxbmkAirt9bPX7qTtcCdNOzKPXvI2uZQZ86kBSRH9hlL94bNiuWbEpPxHu3mKX+qRYRTbMmvtDr7+0Lxu56fKyYP563excddBfj19DWNfawJg+srtFdceH+kNLvrCss2tGSPGbnl2KY/N3cCLN/2v1DR3x8xSgRNlO3pH51Hc4cQBiX1h3fpE2+L4eRv4xBnv5vG5XTd3KuZ9C/2+3R13OK7I4+lCJzcLBWG6QqM+U4GetRF/e9crXHzmQB795t8U8S5FsuzaJKHvnbTvUKLs6e26kLjl2cZdbcE6LGzVJVuY1lQAsHHXQYaNncPQ8/4yNe2+aWv49fRGvvaJmpzrKXeHktwRHMrRJRC6dp4nHN9Vh+t2w+Ws5+Pnbei2nvbDRzlpwPF5X5dZtmCZ3IsEy5V2LRf3xI70nqm5Rzx/+9GFtLZ18Maug/w57QhzexnXc/lS1pH7ZR97H994uI7PfuQ9PHDN/0xNX7VtH29/8wm87+1vKvk9olB1NfSxrzelzpgDPDY38aNbsqmr+eXROYmz2ZYjcMrV0XmUwbe8wFk/fiHVRSx9dN/kZVt5eUVX00Cud60ZMYkbgp4FSbkOl786di4f+tHk0gub481zDecOu2hRoZ1I6ggoJEf+HPEgqm6H8GW2oae312cHUWtbR1lXFkxKFiX5+5xQ33Wie07TLmat7Wq3X7mt+y3pyu+2mPh/a2tXc1B7Ryff/MN81gVHCBf9bBofHdm93Tr99enlAxgX/B1lKHLnl6tiUew2FrOTfWzuGzwXfMZhO4Tk7/OmJ+u59bmubsTz0s6TvbhsG0/O777jKiTZKpB9BPaFe1/jwp9NK3l9Uam6QM/1PTc2d91kdlJwf8JkZSSqCvrgW7ouZ3rrhGXMWrsj1XywYddBrn88M6Tzhc3zi5M/xHDff3oxNSMmMTvPybswr69JNJHsS2sDdnduemIRk5ZsTfWEyH7fVSH35ix0UNCTgSel2pLVdh3F0P9DR46yZvs+ng7ua/nq6hYenrW+7PUVs3NJBt2m3W3devf0tOLR3tG1Z53TtJOXVzRzW7CD2rH/MIezmrhyfX/5ilFMbTxsXcnKQTGbmN0DJpewtvitrW28uroldZQOuU/qA3zr0QX84JmlRb1fpsro5put6ppccu35xwTtWpCoDX35/tmpJoFcJ0HW7zjAxMVbuOHSM/M2VWzafbDb/FdXt/Dq6q4bXacfISQVc6f3VNGC1V/32MLUDqkcyVrh1OWZJxGfq9/Cc/VbeNdbTkyV969um8Li2z7P8TmS+7iC1aT8f6Q1IyaxfNQX+IsTw39mQ0fPzLnmIfdm9iZp3neImhGTePTav+HiwQNTwbK1tY297R287U0nZJasiCrej55dWtY1bWpGTOL7nz+L6y8dHDo/31tPSuue2rzvEGe+5+TU8+8+VfplKzqPOveGXFQsV3NYRjlLOJ/S9ZrEb/+aB+fx8s2f4sz3nJwRba0HO2jr6EztSLJP+uf6W0yf/NHbpqSaUJOvTz/6SAr73V70s+kAfPh9J6f6/Bezfa+ubuGSswYVXjBLU0tpPbx6W2xq6Nnmrd/V7Uf0k+cbqBkxKbXMNQ/N4+6pqwv2S734jhl88ufTSy5r7X+8XPSyybLmCvNpKzID+rrHFnLpXa8UXO+WPW0Zo1R3pT3ef+gIizbs5mO3dz8U37H/UEa/5uRoxPS71xdz0jl5jqGQ+6atYdzs9annK7ftY+W27kcNv3ttLS8u6/qMJi/dxjm3v5TzBGu+v+VSwjx7/Xe9tJoJ9ZvZntZTppgjlefSmmHSm9lmrd2ROgFdiqfrNmY082Ur5/xRY/M+Tv/hpJzzfz09sQPJ/m73HzrCuaNeymhycE+cU8h1ZJj0VN3GjOfJz2b19v3UjJiUCupihf120n33yfqM51v2tLHgjV1FHx1UyDi8bqquhn7W+04uvFBgS9Au7J7ouvjQzPUZ85PtYP05DD79vVds7d6umjR83ALW/tflqefF1OK3trbxiQI7ogdnrss4J5GUvTNqbetg7OvreGjmepaPGgKk1+Bzf34PzVzHJ854N1f+dhZnvfdk7vzSuaHL3Z3n5Fa619fs6DawBuDW55bx8yvPyfm6YWPm8MVz3l/Ue2Qb9fxyHpy5rtv0G5+oL2t9SQ+83sSwsXN4/vqL+erY0q7zs7ZlP+7OiD+FNxckKzG5asTbWttD2/EBnlm4OWettnnfIeYHQZ5swsl3NPTY3A2p81z57Nhf2gjR6SubefMJmSdncx2NhH0Gzy7azD1fOS/1fEL9ZuY07eJzZ7+XsVfXAjBu9noen7eRyd+5uNvrs7fY3TMqQN99sp5T3/lmAP57eiPrf/7FYjarx6ou0P9HCYGe5A6PpNX+9h86wltPGtAv151O72KZ7NYIiQFGl/3q9Zyv6zzq3PJs9z/e0TMauXPKKs497R388VsXZcwrplYzeem2btPSj2KSxr6+LlUOgBvGL0qdB8ju/pduxqoWbn++gcWbWlm8qZWrLjgttHmqp2as6urNct+0NZxz6tsB2Nrazk+eb2B2086Sz0fUjJjENRd9kD/MDjk5GCKZG9ldWHOZFvTAKXXIPsBnfvlq3vnJkJ3TtIs1aedHFm3YTWtbB197aH5qWnbc5WtzvvPFrjEHIyc08NULPlDSYLzsi9blUqgGfOeUVXzn0jPT1tvGi8u6/5Yhd5NLepPpnKbENjSkjW25dUKi2WhtEc0qF98xI2OMwrOLNhd8TW+oukAvJ4I7jh7llVVdX94Njy/k7i+fl3q+ZU8bLfu6fsRthzt584mJvX/rwWj7mX741q5BRr95pWt4/n9OXlHwtWE1neSgnsUb92SctO0tyT+OZJhD4aH6yQFeAFf+NtqBW0nb9x5iZuMOTh/4lm61/ewjs1IUG+YAV5d5pcqtebrRhe1ci5H+d/K5tNGt//Cb7iM+G5v3Z7zPljzlebEhMzTPLPE3F3Z0Febbjy4ouMx90zNvb5frImO5jsDDriy6pbWdI51HM2rbYX3vL03bod78ZH3BAWfpzYl7Dh7mHX9xYt7ly2WRXs2sBLW1tV5XV3rN5LXVLWX/4RRr6Hl/ya+u+jhQ/h+USH/6xZXn8O/PLCm84DHArLQTv9+65AymLt+Wqpnf85Vz+e6TpZ+wzqcnTTBmtsDda8PmFXVS1MyGmNkqM2s0sxEh808ysyeD+XPNrKbs0hZQuOdFz02o38LLy7fnvd6HSCVTmHcptc56/6trM5pZog7z3lSwycXMjgdGA58DNgHzzWyiuy9PW+xaYLe7n2lmVwF3AF/pjQIX0xUwCt98pPSjBxGRYsxeu5OLznh35OstpoZ+AdDo7k3ufhh4AhiatcxQ4A/B4z8Cn7FeusD27LWqNYtIdct3w+ueKCbQTwHSO4luCqaFLuPuR4BWoNvux8yGm1mdmdW1tJTe5xbgx3/3kbJeJyJSKabc9KleWW+f9nJx9zHAGEicFC1nHScNOL7P+nSKiFSTYmrom4HT0p6fGkwLXcbMBgBvB9Q2IiLSh4oJ9PnAYDM73cxOBK4CJmYtMxG4Jnj8j8B076/+kCIix6iCTS7ufsTMrgemAMcDD7p7g5mNAurcfSLwe2CcmTUCu0iEvoiI9KGi2tDdfTIwOWvayLTH7cCXoi2aiIiUouqutigiIuEU6CIiMaFAFxGJCQW6iEhM9NvVFs2sBSj+2qSZBgLR3oW4Mh0L26ltjIdjYRuhMrbzg+4eer+8fgv0njCzulyXj4yTY2E7tY3xcCxsI1T+dqrJRUQkJhToIiIxUa2BPqa/C9BHjoXt1DbGw7GwjVDh21mVbegiItJdtdbQRUQkiwJdRCQmqi7QC92wutKZ2XozW2pm9WZWF0x7l5lNNbM1wf/vDKabmd0XbOsSMzs/bT3XBMuvMbNrcr1fXzCzB82s2cyWpU2LbJvM7K+Dz6wxeG3v3yk8RI7tvN3MNgffZ72ZXZ4274dBmVeZ2RfSpof+hoNLVM8Npj8ZXK66T5nZaWY2w8yWm1mDmd0YTI/N95lnG6v/u3T3qvlH4vK9a4EPAScCi4Gz+7tcJW7DemBg1rRfACOCxyOAO4LHlwMvAAZcCMwNpr8LaAr+f2fw+J39uE2fAs4HlvXGNgHzgmUteO1lFbSdtwPfD1n27OD3eRJwevC7PT7fbxh4CrgqeHw/8O1+2Mb3A+cHj08GVgfbEpvvM882Vv13WW019GJuWF2N0m+y/Qfgf6dNf8QT5gDvMLP3A18Aprr7LnffDUwFhvRxmVPc/TUS18FPF8k2BfPe5u5zPPHX8UjauvpUju3MZSjwhLsfcvd1QCOJ32/obziopV5K4ibrkPmZ9Rl33+ruC4PH+4AVJO4ZHJvvM8825lI132W1BXoxN6yudA68ZGYLzGx4MO297r41eLwNeG/wONf2VsPnENU2nRI8zp5eSa4PmhseTDZFUPp2vhvY44mbrKdP7zdmVgN8HJhLTL/PrG2EKv8uqy3Q4+Bidz8fuAy4zswybv8d1Fpi1Zc0jtuU5rfAGcB5wFbgl/1amoiY2VuBZ4Cb3H1v+ry4fJ8h21j132W1BXoxN6yuaO6+Ofi/GXiWxGHb9uBQlOD/5mDxXNtbDZ9DVNu0OXicPb0iuPt2d+9096PAWBLfJ5S+nTtJNFcMyJre58zsBBJB95i7/ymYHKvvM2wb4/BdVlugF3PD6oplZm8xs5OTj4HPA8vIvMn2NcCE4PFE4OqgJ8GFQGtw2DsF+LyZvTM4LPx8MK2SRLJNwby9ZnZh0DZ5ddq6+l0y5AL/QOL7hMR2XmVmJ5nZ6cBgEicDQ3/DQa13BombrEPmZ9Zngs/498AKd787bVZsvs9c2xiL77IvzrxG+Y/EWfXVJM4u39Lf5Smx7B8icSZ8MdCQLD+JNrdpwBrgZeBdwXQDRgfbuhSoTVvXN0icnGkEvt7P2zWexCFqB4n2wmuj3CaglsQf11rg1wQjnCtkO8cF27GExB/++9OWvyUo8yrSenLk+g0Hv495wfY/DZzUD9t4MYnmlCVAffDv8jh9n3m2seq/Sw39FxGJiWprchERkRwU6CIiMaFAFxGJCQW6iEhMKNBFRGJCgS4iEhMKdBGRmPj/inly1u/9hPIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(t_H.cpu().numpy()[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35f35719-7ee9-4b69-8095-12e65490db28",
   "metadata": {},
   "source": [
    "## Python Autograd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d32d77ca-5946-4697-ba87-a5faa34e1709",
   "metadata": {},
   "source": [
    "### Autograd Intro"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd2b6c2c-e660-42e4-bd2d-f1cc4703d30f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1.],\n",
       "        [1., 1.]], requires_grad=True)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = Variable(torch.ones(2,2), requires_grad=True)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa85a599-e9ac-45c4-9339-c447803faf2d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1.],\n",
      "        [1., 1.]], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3aec7b8c-a62a-4044-be68-1ae9975bfbaf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1.],\n",
      "        [1., 1.]])\n"
     ]
    }
   ],
   "source": [
    "print(x.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5feaf9fb-4171-499e-89f0-efb32dd874f8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None\n"
     ]
    }
   ],
   "source": [
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c7def39-0cf9-4820-b30e-b82a6a726fa8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[3., 3.],\n",
      "        [3., 3.]], grad_fn=<AddBackward0>)\n"
     ]
    }
   ],
   "source": [
    "y = x+2\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "765f902e-b880-4d0b-9dc5-613b229c8bea",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[27., 27.],\n",
      "        [27., 27.]], grad_fn=<MulBackward0>) tensor(108., grad_fn=<SumBackward0>)\n"
     ]
    }
   ],
   "source": [
    "z = y*y*3\n",
    "out = z.sum()\n",
    "print(z, out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "109d9b72-f758-4b3c-9f4a-0e50747fe167",
   "metadata": {},
   "outputs": [],
   "source": [
    "out.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c6d1e8a-429b-4583-8069-94c0ff19d966",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[18., 18.],\n",
      "        [18., 18.]])\n"
     ]
    }
   ],
   "source": [
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70f7b022-fa60-48ba-8165-e9d2af68f143",
   "metadata": {},
   "source": [
    "## NMF with Autograd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fa79bac-5f0f-462b-96ff-531a91e72fe0",
   "metadata": {},
   "outputs": [],
   "source": [
    "lam = 1e-6\n",
    "m, n = vectors.shape\n",
    "d = 6 # num of topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8aafc650-f7b9-4ec3-ba12-995c00e8bb7e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 9.5471e-09, -4.3698e-07, -3.2861e-06,  9.2815e-07,  8.1901e-08,\n",
       "         -4.0887e-07],\n",
       "        [-1.3723e-07, -1.4769e-05,  9.6822e-06,  5.6049e-08,  2.2556e-05,\n",
       "          4.2913e-08],\n",
       "        [-9.5589e-07, -7.9720e-06,  1.0199e-07,  1.3454e-06,  1.5050e-05,\n",
       "          8.0139e-07],\n",
       "        ...,\n",
       "        [-3.3975e-07,  1.7489e-05,  4.3336e-08, -5.0231e-06, -3.6682e-07,\n",
       "         -3.6701e-07],\n",
       "        [ 1.4516e-07, -5.8843e-05, -4.0165e-05, -6.0871e-08,  8.3824e-05,\n",
       "         -1.0180e-06],\n",
       "        [ 5.1754e-11,  1.0262e-10,  9.8942e-12, -4.1353e-11, -3.6190e-10,\n",
       "         -1.0846e-11]], device='cuda:0', requires_grad=True)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pW = Variable(tc.FloatTensor(m,d), requires_grad=True)\n",
    "pH = Variable(tc.FloatTensor(d,n), requires_grad=True)\n",
    "pW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "582e1413-1f22-478f-b9e3-a9885a297979",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.0004, 0.0047, 0.0057, 0.0086, 0.0132, 0.0113],\n",
       "        [0.0095, 0.0050, 0.0014, 0.0197, 0.0099, 0.0050],\n",
       "        [0.0082, 0.0080, 0.0024, 0.0041, 0.0083, 0.0057],\n",
       "        ...,\n",
       "        [0.0051, 0.0159, 0.0061, 0.0081, 0.0110, 0.0224],\n",
       "        [0.0148, 0.0090, 0.0080, 0.0089, 0.0073, 0.0135],\n",
       "        [0.0046, 0.0009, 0.0085, 0.0034, 0.0006, 0.0071]], device='cuda:0',\n",
       "       requires_grad=True)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pW.data.normal_(std=0.01).abs_()\n",
    "pH.data.normal_(std=0.01).abs_()\n",
    "pW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "540bb4dc-28f5-4af4-b7ee-404c9f5c2a84",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0.,  ..., 0., 0., 0.],\n",
       "        [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "        [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "        [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "        [0., 0., 0.,  ..., 0., 0., 0.]], device='cuda:0')"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t_vectors = torch.Tensor(v.astype(np.float32)).cuda()\n",
    "t_vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65f5aafe-74de-48a3-a34b-db8bb3d83b5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def report():\n",
    "    W, H = pW.data, pH.data\n",
    "    print((M-pW.mm(pH)).norm(2).data,\n",
    "          W.min(), H.min(), \n",
    "          (W<0).sum(), (H<0).sum())\n",
    "    \n",
    "def penalty(A):\n",
    "    return torch.pow((A<0).type(tc.FloatTensor)*torch.clamp(A, max=0), 2)\n",
    "\n",
    "def penalize():\n",
    "    return penalty(pW).mean() + penalty(pH).mean()\n",
    "\n",
    " \n",
    "def loss():\n",
    "    return (M -pW.mm(pH)).norm(2)+penalize()*lam"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7bade0e7-4b89-4056-8cdd-113b23543cc6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0.,  ..., 0., 0., 0.],\n",
       "        [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "        [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "        [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "        [0., 0., 0.,  ..., 0., 0., 0.]], device='cuda:0')"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "M = Variable(t_vectors).cuda()\n",
    "M"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46fab992-5422-4877-902f-292815091fd1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on class Adam in module torch.optim.adam:\n",
      "\n",
      "class Adam(torch.optim.optimizer.Optimizer)\n",
      " |  Adam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False)\n",
      " |  \n",
      " |  Implements Adam algorithm.\n",
      " |  \n",
      " |  .. math::\n",
      " |     \\begin{aligned}\n",
      " |          &\\rule{110mm}{0.4pt}                                                                 \\\\\n",
      " |          &\\textbf{input}      : \\gamma \\text{ (lr)}, \\beta_1, \\beta_2\n",
      " |              \\text{ (betas)},\\theta_0 \\text{ (params)},f(\\theta) \\text{ (objective)}          \\\\\n",
      " |          &\\hspace{13mm}      \\lambda \\text{ (weight decay)},  \\: amsgrad                      \\\\\n",
      " |          &\\textbf{initialize} :  m_0 \\leftarrow 0 \\text{ ( first moment)},\n",
      " |              v_0\\leftarrow 0 \\text{ (second moment)},\\: \\widehat{v_0}^{max}\\leftarrow 0\\\\[-1.ex]\n",
      " |          &\\rule{110mm}{0.4pt}                                                                 \\\\\n",
      " |          &\\textbf{for} \\: t=1 \\: \\textbf{to} \\: \\ldots \\: \\textbf{do}                         \\\\\n",
      " |          &\\hspace{5mm}g_t           \\leftarrow   \\nabla_{\\theta} f_t (\\theta_{t-1})           \\\\\n",
      " |          &\\hspace{5mm}\\textbf{if} \\: \\lambda \\neq 0                                           \\\\\n",
      " |          &\\hspace{10mm} g_t \\leftarrow g_t + \\lambda  \\theta_{t-1}                            \\\\\n",
      " |          &\\hspace{5mm}m_t           \\leftarrow   \\beta_1 m_{t-1} + (1 - \\beta_1) g_t          \\\\\n",
      " |          &\\hspace{5mm}v_t           \\leftarrow   \\beta_2 v_{t-1} + (1-\\beta_2) g^2_t          \\\\\n",
      " |          &\\hspace{5mm}\\widehat{m_t} \\leftarrow   m_t/\\big(1-\\beta_1^t \\big)                   \\\\\n",
      " |          &\\hspace{5mm}\\widehat{v_t} \\leftarrow   v_t/\\big(1-\\beta_2^t \\big)                   \\\\\n",
      " |          &\\hspace{5mm}\\textbf{if} \\: amsgrad                                                  \\\\\n",
      " |          &\\hspace{10mm}\\widehat{v_t}^{max} \\leftarrow \\mathrm{max}(\\widehat{v_t}^{max},\n",
      " |              \\widehat{v_t})                                                                   \\\\\n",
      " |          &\\hspace{10mm}\\theta_t \\leftarrow \\theta_{t-1} - \\gamma \\widehat{m_t}/\n",
      " |              \\big(\\sqrt{\\widehat{v_t}^{max}} + \\epsilon \\big)                                 \\\\\n",
      " |          &\\hspace{5mm}\\textbf{else}                                                           \\\\\n",
      " |          &\\hspace{10mm}\\theta_t \\leftarrow \\theta_{t-1} - \\gamma \\widehat{m_t}/\n",
      " |              \\big(\\sqrt{\\widehat{v_t}} + \\epsilon \\big)                                       \\\\\n",
      " |          &\\rule{110mm}{0.4pt}                                                          \\\\[-1.ex]\n",
      " |          &\\bf{return} \\:  \\theta_t                                                     \\\\[-1.ex]\n",
      " |          &\\rule{110mm}{0.4pt}                                                          \\\\[-1.ex]\n",
      " |     \\end{aligned}\n",
      " |  \n",
      " |  For further details regarding the algorithm we refer to `Adam: A Method for Stochastic Optimization`_.\n",
      " |  \n",
      " |  Args:\n",
      " |      params (iterable): iterable of parameters to optimize or dicts defining\n",
      " |          parameter groups\n",
      " |      lr (float, optional): learning rate (default: 1e-3)\n",
      " |      betas (Tuple[float, float], optional): coefficients used for computing\n",
      " |          running averages of gradient and its square (default: (0.9, 0.999))\n",
      " |      eps (float, optional): term added to the denominator to improve\n",
      " |          numerical stability (default: 1e-8)\n",
      " |      weight_decay (float, optional): weight decay (L2 penalty) (default: 0)\n",
      " |      amsgrad (boolean, optional): whether to use the AMSGrad variant of this\n",
      " |          algorithm from the paper `On the Convergence of Adam and Beyond`_\n",
      " |          (default: False)\n",
      " |  \n",
      " |  .. _Adam\\: A Method for Stochastic Optimization:\n",
      " |      https://arxiv.org/abs/1412.6980\n",
      " |  .. _On the Convergence of Adam and Beyond:\n",
      " |      https://openreview.net/forum?id=ryQu7f-RZ\n",
      " |  \n",
      " |  Method resolution order:\n",
      " |      Adam\n",
      " |      torch.optim.optimizer.Optimizer\n",
      " |      builtins.object\n",
      " |  \n",
      " |  Methods defined here:\n",
      " |  \n",
      " |  __init__(self, params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False)\n",
      " |      Initialize self.  See help(type(self)) for accurate signature.\n",
      " |  \n",
      " |  __setstate__(self, state)\n",
      " |  \n",
      " |  step(self, closure=None)\n",
      " |      Performs a single optimization step.\n",
      " |      \n",
      " |      Args:\n",
      " |          closure (callable, optional): A closure that reevaluates the model\n",
      " |              and returns the loss.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from torch.optim.optimizer.Optimizer:\n",
      " |  \n",
      " |  __getstate__(self)\n",
      " |  \n",
      " |  __repr__(self)\n",
      " |      Return repr(self).\n",
      " |  \n",
      " |  add_param_group(self, param_group)\n",
      " |      Add a param group to the :class:`Optimizer` s `param_groups`.\n",
      " |      \n",
      " |      This can be useful when fine tuning a pre-trained network as frozen layers can be made\n",
      " |      trainable and added to the :class:`Optimizer` as training progresses.\n",
      " |      \n",
      " |      Args:\n",
      " |          param_group (dict): Specifies what Tensors should be optimized along with group\n",
      " |          specific optimization options.\n",
      " |  \n",
      " |  load_state_dict(self, state_dict)\n",
      " |      Loads the optimizer state.\n",
      " |      \n",
      " |      Args:\n",
      " |          state_dict (dict): optimizer state. Should be an object returned\n",
      " |              from a call to :meth:`state_dict`.\n",
      " |  \n",
      " |  state_dict(self)\n",
      " |      Returns the state of the optimizer as a :class:`dict`.\n",
      " |      \n",
      " |      It contains two entries:\n",
      " |      \n",
      " |      * state - a dict holding current optimization state. Its content\n",
      " |          differs between optimizer classes.\n",
      " |      * param_groups - a list containing all parameter groups where each\n",
      " |          parameter group is a dict\n",
      " |  \n",
      " |  zero_grad(self, set_to_none: bool = False)\n",
      " |      Sets the gradients of all optimized :class:`torch.Tensor` s to zero.\n",
      " |      \n",
      " |      Args:\n",
      " |          set_to_none (bool): instead of setting to zero, set the grads to None.\n",
      " |              This will in general have lower memory footprint, and can modestly improve performance.\n",
      " |              However, it changes certain behaviors. For example:\n",
      " |              1. When the user tries to access a gradient and perform manual ops on it,\n",
      " |              a None attribute or a Tensor full of 0s will behave differently.\n",
      " |              2. If the user requests ``zero_grad(set_to_none=True)`` followed by a backward pass, ``.grad``\\ s\n",
      " |              are guaranteed to be None for params that did not receive a gradient.\n",
      " |              3. ``torch.optim`` optimizers have a different behavior if the gradient is 0 or None\n",
      " |              (in one case it does the step with a gradient of 0 and in the other it skips\n",
      " |              the step altogether).\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors inherited from torch.optim.optimizer.Optimizer:\n",
      " |  \n",
      " |  __dict__\n",
      " |      dictionary for instance variables (if defined)\n",
      " |  \n",
      " |  __weakref__\n",
      " |      list of weak references to the object (if defined)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(torch.optim.Adam)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "337fb06c-6077-4b61-b74d-78422bf9c57c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# (torch.optim.Adam/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf758355-58c8-4a67-8724-9207283f43fc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(44.4393, device='cuda:0') tensor(7.9982e-07, device='cuda:0') tensor(2.1715e-08, device='cuda:0') tensor(0, device='cuda:0') tensor(0, device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "opt = torch.optim.Adam([pW,pH], lr=1e-3, betas=(0.9,0.9))\n",
    "lr=0.05\n",
    "report()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80a18521-1aaa-4416-a8d0-bcadf6f781fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(43.8757, device='cuda:0') tensor(-0.0856, device='cuda:0') tensor(-0.0843, device='cuda:0') tensor(3068, device='cuda:0') tensor(62686, device='cuda:0')\n",
      "tensor(43.6954, device='cuda:0') tensor(-0.1680, device='cuda:0') tensor(-0.1618, device='cuda:0') tensor(3800, device='cuda:0') tensor(49265, device='cuda:0')\n",
      "tensor(43.6364, device='cuda:0') tensor(-0.2488, device='cuda:0') tensor(-0.2549, device='cuda:0') tensor(3845, device='cuda:0') tensor(46335, device='cuda:0')\n",
      "tensor(43.6285, device='cuda:0') tensor(-0.2613, device='cuda:0') tensor(-0.2972, device='cuda:0') tensor(3631, device='cuda:0') tensor(49044, device='cuda:0')\n",
      "tensor(43.6285, device='cuda:0') tensor(-0.2609, device='cuda:0') tensor(-0.2967, device='cuda:0') tensor(3660, device='cuda:0') tensor(49023, device='cuda:0')\n",
      "tensor(43.6285, device='cuda:0') tensor(-0.2608, device='cuda:0') tensor(-0.2964, device='cuda:0') tensor(3643, device='cuda:0') tensor(49037, device='cuda:0')\n",
      "tensor(43.6285, device='cuda:0') tensor(-0.2608, device='cuda:0') tensor(-0.2963, device='cuda:0') tensor(3677, device='cuda:0') tensor(49003, device='cuda:0')\n",
      "tensor(43.6285, device='cuda:0') tensor(-0.2606, device='cuda:0') tensor(-0.2962, device='cuda:0') tensor(3651, device='cuda:0') tensor(49054, device='cuda:0')\n",
      "tensor(43.6285, device='cuda:0') tensor(-0.2606, device='cuda:0') tensor(-0.2960, device='cuda:0') tensor(3668, device='cuda:0') tensor(49392, device='cuda:0')\n",
      "tensor(43.6285, device='cuda:0') tensor(-0.2604, device='cuda:0') tensor(-0.2960, device='cuda:0') tensor(3663, device='cuda:0') tensor(49160, device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "for i in range(1000):\n",
    "    opt.zero_grad()\n",
    "    l = loss()\n",
    "    l.backward()\n",
    "    opt.step()\n",
    "    if i % 100 ==99:\n",
    "        report()\n",
    "        lr *= 0.9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8559ca10-76ea-4bc9-9537-670a0dec190d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['does say moral just don morality people think',\n",
       " 'manhattan beauchaine bronx morality tek bobbe ico god',\n",
       " 'know space said graphics people think like don',\n",
       " 'launch shuttle people bible believe jesus nasa god',\n",
       " 'file does image know files jesus graphics thanks',\n",
       " 'morality data files thanks image program nasa objective']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h = pH.data.cpu().numpy()\n",
    "show_topics(h, vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23d49571-da1b-4456-9e7e-f73275317008",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAwk0lEQVR4nO3dd3wUZf4H8M83CQm9V2kBBQEFKREpCiIdVM6ze6ecjTtPznbqYe/K2fXkPDn1Z0VUFOUEpAmINAm9Q4DQDEkInQBpz++Pnd3M7s7Wmd2d3f28Xy9ezM5MZp5p33nmeZ55RpRSICKixJcS6wQQEVF0MOATESUJBnwioiTBgE9ElCQY8ImIkkRarBPgS8OGDVVmZmask0FEFFdWrlx5UCnVyGiabQN+ZmYmsrOzY50MIqK4IiK7fU1jkQ4RUZJgwCciShIM+ERESYIBn4goSTDgExElCQZ8IqIkwYBPRJQkGPCJEphSCt+s3IdTJeWxTgrZAAM+UQJbsqMIf/96LV6YsSnWSSEbYMAnSmDHT5cBAAqOnYlxSsgOGPCJiJIEAz4RUZJgwCciShIM+ERESYIBn4goSTDgExElCQZ8IqIkwYBPRJQkGPCJiJIEAz4RUZJgwCciShKWBHwRGSYiW0UkR0TGGUz/i4isF5E1IvKLiHSyYr1ERBQ80wFfRFIBTAAwHEAnADcaBPRJSqnOSqmuAF4G8LrZ9RIRUWisyOH3BJCjlNqplCoBMBnAKP0MSqljup81ACgL1ktEFDF7DxWjpKwi1smwlBUBvzmAvbrf+7RxbkTkbhHZAUcO/x6jBYnIGBHJFpHswsJCC5JGRBS646dLccnL8/HIt+tjnRRLRa3SVik1QSl1NoB/AHjcxzwTlVJZSqmsRo0aRStpRATgVEk5FuccjHUybMH5hbCftydWxtOKgL8fQEvd7xbaOF8mA/idBesloiAFU4b6yLfr8If3lyP34MmIp4diw4qAvwJAOxFpIyLpAG4AME0/g4i00/0cCWC7BeslogBEgp93W/4JAMCJM2URSg3FWprZBSilykRkLIBZAFIBfKiU2igizwLIVkpNAzBWRAYBKAVwGMBos+slIqLQmA74AKCUmgFghse4J3XD91qxHiKiaFIJ1p6Qb9oSEXkKoSgsnjDgE8Wh4pIyvDl3G8rKE6udOEUWAz5RHHpjzja8OXc7vl3lr0EckTsGfKI4VKy1Ez/DHD6FgAGfiACE1oST4hMDPhFRkmDAJyJKEgz4RPEs0RqKU0Qx4FPEHDh6mv2yRAjL2ykclrxpS2Sk10vzAAC540fGOCWJhxn7aEmsHc0cPlE8Y1Y/IiRBX7VlwCciN3x6SFwM+EQEgA8LyYABnyieBciOM7dOegz4RHGIuXEKBwM+UQLjjYH0GPCJLPTAl2vw9jx+wZPsie3wiSz07WpHd8X3DGwXYE5zWDZP4WAOn4jcqAR72YgqMeATxaFIlM0n6stGZiTakxQDPhGRh0St7LYk4IvIMBHZKiI5IjLOYPoDIrJJRNaJyDwRaW3FeomIKHimA76IpAKYAGA4gE4AbhSRTh6zrQaQpZTqAmAKgJfNrpeIiEJjRQ6/J4AcpdROpVQJgMkARulnUErNV0oVaz+XAWhhwXqJiCgEVgT85gD26n7v08b5cjuAmUYTRGSMiGSLSHZhYaEFSSMiIqeoVtqKyB8BZAF4xWi6UmqiUipLKZXVqFGjaCaNKOmxOWbis+LFq/0AWup+t9DGuRGRQQAeA9BfKXXGgvUSEVEIrMjhrwDQTkTaiEg6gBsATNPPICLdALwH4EqlVIEF6yQii7EdfuIzHfCVUmUAxgKYBWAzgK+UUhtF5FkRuVKb7RUANQF8LSJrRGSaj8UREVGEWNKXjlJqBoAZHuOe1A0PsmI9ROSOpe6RlWj7l2/aEsUhFr9EVqLuXQZ8IqIkwYBPFIdCbUKZaJ2AUXgY8IniWKCih3CKJnhzSFwM+EQEIHF7iKRKDPhEcYyZcQoFAz5RHGIrHQoHAz4RUZJgwCciShIM+ERESYIBnyJCsW2fT/sOF2Nz3jFTy0iGroyPnipF5rjp+GRpbszSkGjnMQM+RcTqvUdinQTbuvif8zH8rUWxToZPdglx+cdOAwA+Xbo76uuWBG2jyoBPEVFRYZewkZgi0UrHriEuHs+kCfNz8L+1v8U6GV4s6S2TiMhqdr0BBeOVWVsBAFdccFaMU+KOOXyKiAR9IqYYSLRy9FhiwCciW2KmwXoM+EQRsrPwBE6eKYvoOuIt87v3UDEe/249ykOo44mzTbQ1BnyKEGbPLnttIW758NdYJ8NW7p28Gp8t24M1ew8HMTfPIasx4BNF0MrdwQS28AUq9ggndxzJMvPKJYcQzJnFtwwDPkUEy1/tJajjEdWDFjiK8xyyHgM+WWLC/BzsKDwR62SQD3Yp6w/rgyyWpyI+1h0JDPgRsjnvGE5EuMLOLo6fLsUrs7bi+veWxjop5MGumeRgbkCxTLtd95tZlgR8ERkmIltFJEdExhlM7yciq0SkTESusWKddlZRoTD8rUW4/aMVsU5KVDiv3dOlFa5xiXrB2I1dcu7BCqfLAru1wy8rr8D+I6fC/vu8o+H/rVmmA76IpAKYAGA4gE4AbhSRTh6z7QHwJwCTzK4vHjhPz19zD8U0HdHC4B598V6+HUwIt2t/Ni/N3IK+439CwfHTIf/t0h1F6P3ST/h+zf4IpCwwK3L4PQHkKKV2KqVKAEwGMEo/g1IqVym1DkCF0QISjT1P08jT58RifbGWVyjWKdhQvJXhG/l5WyEA4Ehxach/6+wldfWeI1YmKWhWBPzmAPbqfu/TxoVMRMaISLaIZBcWFlqQtNiy2ZNoxBgF91g/hv/rp+0Y+NpCbM8/HtN0REqynFvxrrjEXvV4tqq0VUpNVEplKaWyGjVqFOvk2MKOwhP4aUu+qWV8nb0XR0+FnhsJlZ1iUHauo/37gWOhP3bHE5uWegQUyxvWbybK30P1j2/WR21dwbAi4O8H0FL3u4U2jiww8LWFuO2j7LD+Nv/YaWSOm46HpqzDQ1+vtThllYxiTqyLdJJFvOX0QzktInEGzducjz7jf8K8zeYyUcHaXXQyKusJlhUBfwWAdiLSRkTSAdwAYJoFyyWT9G95Fhw/E8OUxE68BcRgRfN+eqS4BHM3WRsgQynys/IYrt13FACwfv9R6xYaR0wHfKVUGYCxAGYB2AzgK6XURhF5VkSuBAARuVBE9gG4FsB7IrLR7HopsGgHO/36Yp2/5wOGdcZ8uhJ3fJKNohPRzTTY4RgeKS7F3Z+vCqmzNz27ZTgsKcNXSs1QSrVXSp2tlHpBG/ekUmqaNrxCKdVCKVVDKdVAKXWeFeuNZ/uPnIp4ubr+u6eRPO/scGH6YrPrza+eL8zFf3/eGetkeHEWS5SUR66RXe7Bkz6Daqy/3zt9fZ7rc4uRtuXAsYg2eLBVpW0y6Tv+Jwx5Y2FE1+F23kQhq2H2wlyy4yDu+HgFtuUfR2kEg4tdFRw/gxdmbI51MryY+Zziuwt24OYPlvtd1u6ik7j01QV4fc5Wy9YbSKxy3v4yR/O3FmDYm4vw9cp9EVs/A75JFRUKny7NxenScr/zHT1V6vWGXf6xxChXN7oww8n1//mTlZi7uQBD3vgZL83Ygk+X7cbj39mrlYMvX2fvxeGTJZYus9uzs/HAV2sMp0UyYPnq4TOcdf7zxy1YtP2g97J0w87rYPlO4xcV7VAsEskk5BScwPXvLcUGrX5hS17kmhIz4Js0a+MBPPH9Rtc3LAHjk2PAqwvQ+6Wfopcwj3RE45qx8sJcufsQnvhuAz5btsfUciL9PsDK3YewOe8YHpqyDn/7YrWlyz5cXIpvV0W/wdvz0zdjSU5lkHbevIPdk9+v2Y/1+3xUihpkBHwtP5pFhYdPlmDyr+bONSOBnnorKhQemrIWy3cdwuId3jdGq/Ej5iYd1zpIC/TW3SGLc3/BiPbLT2fK9H3phHG16v7EbMqj0Sz0+OlSXP3uUjSpnQEAKPTREkophfEzt+Cq7s3RoWltS9Ydic3TL1LfV0yoq7p38hoAQO74kSGt19f5uu/wKWzYfxTnN68TYkqCd/9Xa7BgayG6taqHc5vW8pquT1uFNhxOsaPnJr45b3tU37plDt8s7QDaoeJy3Dfr8PGSXMNpdngsDuT46cq3EoNJb1l5BeZtzjcMFNG42TkrGQMVzR07VYb3ft6J6/7j3pvojsIT+MWguCMcSimUlFlX72G096zcp24tuoJ4grj8X79Ytm4jB7UWSGfKygOmZUehoxL7vYWhVbCfLi3Hsz9schs3fd1vIS3DLAZ8k5yPbCk2CPiTV+zFU9MqW7zGQ5D3JZgK4HcX7MDtH2fjpy0Flqwz0s0OPbdo4GsL8UddhaYZr87eivaPz8SpEv91SUFzC8iOk9vzfPo6ey+e+V9wLaxzChzl0saXifHyPTNRSinM31KAijCbSHquv6y8At+u2oeKCuV6Ig3lmjlcHPipXf+ka1QKcPRUaJkcsxjwTXKee19lR65mPVxGQTOn4Dju/nyVpblBI2afeII5+XcfKgYAFJ3wvvCMKgr9uf69pejx/FxX2fPuopNYnBN+7rtM/7jvKrMwnjfcNt56X65wdGd1/Iw1TX2DueE+NGUd/m9xblDLW7jNfV/qlx9sHcHU1ftx60cr0PbRGVi9x/enI8vKKzD51z0B9+v7v+zCA1+txTer9gVMQ7jB2Gg7fc8beQz4JtkxF51/7DSKS8rc0uY88R6asg7T1+fZ/k3DYParax4Lnq6W73K0EHluuuORu/8rC/CH98PPfZ/z2EzXsD6YGBWLvDxrS8DlKaUwbe1vXjdq5bkTfO630E5Uq89rZzqNgp5UzuQ+3mPmvKOVbeFfn7PNazlf/LoHb8/bjk+X7ca4b9fj06W5vtMD4KBW53K4uCRgPUKkuG1iFFbNSluTKiJwgpwqKcfOgydw3lnhVVJd9OI8dGpWGxe1re8a553MyhFl5RU4XVaBmhn2OR2C2avOm5i/eB/q0VkTgQq0QMFk7qZ8PDK8o99lzNtcgHu+WI2/Xno2Hh7WwWubU3zE+3ArryMeewyKjJwenboemQ2qY2SXs9z/RLf/jHblI9+6N+E9rBWhFJ04g2rpqaie7n5+u27EqvJHRF9QDDA9Gi+YMYdvoZdmbsa6fUdML+feyasx8u1fcOx0+I/nm/KOGT5uG510f/tiNc5/albQyy4rr7CurNgHZ7/hfrkqzN23au3eI9YnKAi+Llhn+krLFUb/X3hfQTuklRc7+0Ty1YQxEhkQKxokON9T8dd6y5nyScv34MUZWwLcyB1zz9mUj0nL/Ten7PH8XFz5zmKv8frzxjm0JOcgHoxUR4M2qOdjwDdJf3m9t3Anfv/vJQH/JlBzLueLL6M//NVM0tx4xgH975kbDoS0rNs/zkbHJ3+sXFaMXn33VaIzasJit5m25x/HNj/94rd/fKbPaX7XH+RmO9NXUl7h+nhGIDkF7h9vcRWJeC5bC1rhVDp6WqO7URotx8yyPbup1i+q8gnI/zL00yu0S+jOT7Lx6FTjl/MUgIXa/vbcn57zOWP/q7O3YYqJN13/b/EuZI6bbphmwxcU9elgpa39eT6mB5Mb2n/Yf3/czlxaJNrnip9H18teW4CTQXx43XkRRaOPfQAY/PpCv00vUwKcxYPf+BlD3vjZ5/RIV2CHY9Dr7t1uODc/xeME+3DxLvzx/eVB9/tfVl4R1HEzrlQ1H5GW7izyGhfs8l/TldsHlRalMO6bdT4n+7vRhFuR/sz/NvmcZoem2wz4JoVzV04N0IbTaJHOVgnhfEfT1zI97Sw8GVJl7phPvPvpn7R8Dz5btjuElFXydUFsLzjh9lKXk/Oa9FdM8PXKvT6nAd43rZLyirC/N6r/iLueFRe6c1s9b267i4rxi641kfM4z1yfhy9XeBd13P/VWlzwzGy37S46cQazN7o/5RnlTFfvOYIfN+T5TOMHv+zC7R8ZF1l5Nbl0Gw796eREEBkTz/X4m+g539NBNjcFHF1g3OPnLeuiE2e8MiyGGZig1xg+BnyTPA9cabnyW9N/urTcrblfcUkZHpu63vVm45xN+YbtdfccKsaM9Xno+cI8LN3hnUvaddD/hxYi0frAqJz90anr8fh3G8IKcp65V70OT/yI41qdRv6x09iw/2hlkY6fdc1YXxnIcgpOYOmOImSOm46cghPIO3oKFzwz2+tvnG+KAo7it8+X7w7YVxLgOEZGrOgEzPnUF6gS1nmc7/p8leHXlv631vGiz1W6Yq9bP1qBMZ+udF+Obti5yvu+XIO/fLYKgHHG47kfNmGej3ciPM8+o+UHaoevt7PQ/XwvMHjCUQB+OxpcBslzv05f535j83f5HC4uxbS1xi9QbS84gR7Pz8Vny/cErrSNQpkOA75JoT75PfX9RozTtSbo9OQsfL58D/6udZJ1p0GuGQC+X/MbVuQ6mg5u/M07F379e0u9xhkJJvQcLS7FR4t3eZ2AOQUn3Msng1pj8AI9Rh8+6Qj4/V+Zj8v/9YtX+o4Ul/gtkvps2W48+f0GAMCynUXYF6BoDQAufWUBHpu6AU9+vyFqXy8yquPxLMP3dRzHTjLOae48eBJjJ61y++0aLjTYLj/BRymFni/M8znd+G+Mx+8oPOF6gSmU88mzcvqGicuCWudCgzoUBRXwuhj+1s9hBWRnceEiH3U30S7msU87vDgValnfl9nGRQyBlvPTlgJ0MOjjwynYR1wn57nr2dpGKWDct+swc8MBbM0/jpd+38U1zVe5crQ4L3Jn0UllDl9w8kwZuj47x9WvjZEVuYewXau8m7MpH+0a1wy4TueT11fZ+/BV9r6g+4cx4/K3vbsR8Hya+XipcbHZmr1HsPWAdwX1zsKTxoEdQFmFwQ1GN+wZk46HeK55L9Fx7qzecxhX6Ro5eNWH+QnDnpeLr6crT/qKaWfXCEoB2T56CHU6WVKOCgWkmgjQbq2CDCI9i3TigFX9lxuVUXs65qeyLdB5WFkk4Ph98wfLkXf0lFtrGydn+e4Xv+71W6HpvECNAr/+YrWqf3vP1Th7dDxSXOIqQ/XXr43+IxYLtxWG/HQGAANfW4C9h4q9yrz92Xs4uGDktNWgRZGzOwF/xV5OQ9/0XUGt5+yqoLQ8tB2R5lEHpZRye/IzXpf3uIen+K5QDaSkrAJHdF0bBLMFPZ6bYzg+GkUpCoGv0eIzzn58IpceBvwI0Fei/WPKOlzycuBukdf56k5Wx1eAOnGmDCcDtIvflu/I2ToD8ZmyCuwo8M7xzVifh91FlQHqrXnebzQ6KTjqJH4J0AXBkDd+xnMenUblHzuNfy/IgVIKczblBwwYgPeF6Xyx5snvN/osO9Y76NEFg1HONpAdhSdxycvzMebTlejmI4B48tdCCAicO73gmdmVlbYhlgH4a1P+44YDuPWjFYZPl5OW70HmuOkoLilDbpF7+jwbHZQFcedUCtigaxAwe9MBlHsczy0eTyaBGhB0fbZy/xtXgrqPK/LRY+0Xv/qv2LfCnE35bsU3y3cdglLKLYPivNEvyfGuo7MKi3QiQN+Bma8inHD4ano36p3gehL0DKpGHXd96tHCZkvecfy4IQ/92zf2mre4pBzP/bAJnxu8+OIZlz5ZuhtXdWuO1XuO4JberXHXZyuxas8RDOrYxLA1iRGr8z03f2Ddew5mBMphHz1VGva2+2uC6e/D9s7gW2DwxDTirUVuv31lkPVB+MvsvW7Xwic+iqT0fNVnGa7LYFxxkC8H6ruC9mdF7iH0atvA53TPIk9/NucdQ5tHZhhOO1kSTpFZcBjwIyDctx2DyeUCjg9U7Dp4EjkFJ3Bz79au7lojYd6WAszbUoDrs1oaTjcK9r44y2urpKXgmNYV8tX/XoIW9asH9fc3TFyG33drHvT64kmgHiCdT0jbC47j113GX4YKlbMRgD+XvrrAa5zn+ebrfA/nMpi6OryXnozWFWzHbsG6YeIytzqcRdsPurVW8vdyFxB8a61IljAx4Hs4droUaSni1e+GEV8fvND36x4pzkC73KKLP5BQn1Su/Y/vVkM7Ck64yvSPnykLrhsFOPb3ezb8yLcVPly8K6j5FucUYXFOcC2yAvlhne829aHo8IR3PRDgaAkVqvu/jFC3Bh7enrc9rL/L9rhJhtRaKcjSuGCfOMIhVlRYiMgwAG8BSAXwvlJqvMf0DACfAOgBoAjA9UqpXH/LzMrKUtnZwT/SWSVz3HTUr5GOVU8MxsETZ5CRloJaVat4zbck5yBuMtGbIhElly4t6gRVVwcA/70lC4M7NQlrPSKyUimVZTTNdKWtiKQCmABgOIBOAG4UkU4es90O4LBS6hwAbwD4p9n1BnL0VKnrjUmlKl+G+mjxLrePiW/YfxSHTpZg7d4jOKC9pOH8HGHW83PR+enZbn8PODrnYrAnolAEG+wB4KEpkXnSMZ3DF5HeAJ5WSg3Vfj8CAEqpl3TzzNLmWSoiaQAOAGik/Kw83By+UgrD31rkVeNPRBRPwn3nI6I5fADNAegLePdp4wznUUqVATgKwHd1twlFJ0sY7ImIDNiqHb6IjBGRbBHJLiwMrhtZT6G2UyYiShZWBPz9APRt9lpo4wzn0Yp06sBReetGKTVRKZWllMpq1KhRWIkJt0nkmH5tXcO39W2D+Q9e6vpdu2oabuxZuYmvX3cBcsePRO74kch5YbjPZV7dvUVYaVn08ADX8K6XRuD/br0wrOUAwJJxl7mGrwqhSeO5Tdy7cWhYM901nDt+JBbrlutp6HlN8NDQc4Naz6MjOmDuA/2DTle4RnU9yzDdDwxuj1eu6eLjr/zr1qpuyH9za9/MkObv374RPr6tZ8jrmXVfPzwwuL3P7jg6NavtGv710YEBl/fMlechIy1y+cMtzw2L2LKd3v1Dd69xAzs0drvWfWnbqIbb79zxI91igtVevfaCiCzXiiO4AkA7EWkjIukAbgAwzWOeaQBGa8PXAPjJX/m9GakiaNuoBm6/uA02P1t5Et094GzXcOfmdQAAfc52lCp9/ZfeeHRER2Q/PggPDT0Xj47ogDYNa2DqX/sAAJ684jy89PsuSNdO+MyGlQc/LTUFU//aBzPvvcQrLYEubl8XY0tdu3QRwYBzvV968uWKC9w/C1e1SioAoFZGGt64vqtr/G1927iGNzwz1O1vbundGk9dWVnvnp6W4hV0mtethpb1qwEA3rmpm9u0927Owt0DzkGDGunwJ3f8SIzpdzaa1akaYKvMG35+M8Px9wxsh2uzWrpu4EZuv7iN4fhv/tIHueNHYtdLI/DNXX3cpl3epRmm/rUPVj4+yG18ldTgL7mBHRrj49t6on/7RvjhbxfjuVHnuaZt95PR+FOfTJzbtBbuGdgOP97XD9/d3ddrngm64NeoVgYu7+K+f359dCB2vDjC9Xt0n0yMG94hYJovaFnXNfzatY6MkWdan76iE/7zxx6u3ylSeZ4CwLU9AmeUwinfHt65GXLHj3Rta8Oa6fjgTxeiTcMaPpeXO34kpv61D2bcc4nXPC9e1Rk//O1i1+//3pKFF6/qDACYfX8/n++uOK17eojrvPv3H7rjicsrr7mru0fmfRPTAV8rkx8LYBaAzQC+UkptFJFnReRKbbYPADQQkRwADwAYZ3a9vtSrkY6f/n4pnri8E6qlV55Efx98LlppgfS16y7AB6OzMOnOXsgdPxIXZjq+/dqwZgbuHnAO0rSLslurelj+6EBco52Agzs6mkl55nS6taqHjs1qu052ZyAPVLykzyGeE0RHXkYuadfQ7Xfdau5NSFN9pOHJKypPLv23bG/s2RLPjjrf7XX77/7a1/UyiD5nOPPefljx2CBkta78dq5eh2aO/aC/uI3UyEhDzgvDsfqJwX7nC+TZUefh/kHtAbhvU8/M+hh6nuPY6V9w+iiIJ6cdL45wC/hrnqxMY4rWxYCIoEfreq7x/xjWAe/c1B3dWtVDg5oZbk8Q+sDmy/nNa+PWvpl46feddePq4Obema7fVVJTUD3de1m540fi6SvPcxvX1eO8BIDMBtVxpZY5EBGUaW/7OjM1VdNTvbpQuLWv8Y3PLe1nVZ4fI7XA6nmT+1PfNhh2flPXE/BlHRwZmi4t6uCVa7rglQC5W88MRqj+0t+R+WtY03dHe3rdWtVzO24XtXGc7yKC87XMIwAM7tQEN13UCrnjR6J9k1r45zVd0LiW73XU1jX3HtG5mdt5Fu63iAOx5BlNKTVDKdVeKXW2UuoFbdyTSqlp2vBppdS1SqlzlFI9lVJRe3vmm7v6YNkjA10XJwCkp6ZgYMfg2rg2qV2Z+xx/dWe8eX1Xnx8Xn3THRVj08ABXsVKgLzHVqZbuyjl31eWMQvHade4Xh2f/JBlVHInIyqwHf+poN4rLOjj2S1br+shqXQ8z770Enc6q7Qr4+vOwZkYaGvk5od/9Yw9MHtMLw85vGnA70lJTUC/AE4GTPuh6csaWOrob36huZ3ldQM3rVsOlQTw5paaI22v7daunY+eLI/wW5TWo6b4dZ+tu5n/p7yg6rJ6einsHtsMjBrnm/u0b4akrzkPj2v6ffH75x2VuxX/+bH9hOKbfU/kUKiJ4/boLXE93d/Zrg7QUwYIHL8VPf+/vFoyqhNlFpD5IGhWbvHbdBVj08AC8c5PjaWPa2ItxbYBcMQDUr+6+f33lzu+57By0b+I7IxVOUN310gh8+efeQc/vPHdu6d065HVFQsK/aavPeVV+Si08tapWwe/8lIPXyEhDjYw0V0dXqSJoXrca9h85hapVUnC6tAINa6bj5JlynCotR93qVdC/fSNHrkzX/04oGteqilVPDMaE+Tm4sWdLr46gqlZJxY/3XeJ6uvFl7VNDcLq03HWRVktPxRRdMYWzIyqja6RWVePTqHbVKq6+Rx4d0QEvztgScHvmPtAPYyet9tnSqmZGGurqLvgLWtTBWl37ZudFPLhTE3y0JBeAe++Ork8FhpDV8ez2ICVFkOLntUnPm2D3VvXwyW090attA6SnpeCHv12MxrUy0Lh2VZRXODrQmr+1wPURm2Bfwa9fIx31g7xJOnPZ1/RogVVaV8BpqSmoqY3v0bo+cnRFOE6Lx12GakE8lTj5iqFtGtYwHN8yyG41nD78Uxb6nNPQ5/Q+ZzfAkh1FGNSxMR4Yci425R1zdRzo5MzEjfDIiCx6eAAueXk+AEcR6N8GnuO1fKObxCvXdMGynf7feB874Jyg+g+KtIQP+HrN61bD7qJi12NrpOi/TjT7/n44XVqOGyYuw/aCE5hwU3d0b10PnyzdjZt7Vd71Q6nSyB0/0q3fnfo10l3lfw8Mbo861aogPS0FE+bnAAA6NK1tuJxJd1yEDN3F7K+4wdc3VQHHjW7d00PQ68V5bjlrvTH9znYF/J/+3t/nTfecxrXw4339XNu3+onBqJ6Rit//ewk2/nbMVaRUKyMNx8+UefUgesOFLbFwWyFu69vGFfD1N+nGWn/5Dw4JrlIZqNz2FvWq+Z1v3dNDMG9zvmGdS7/2lY0Q9MUAqSmCJ6/ohLPqVsXz0x1dbUeyoVmolYHN67pv8//GXoyftxfi0MkSfPCLozuIWhlpeHDouXhq2ka/N6u3b+yG6iHcPIw4n0B9Gd0nE0t2FMHZj4HR50Qb1crAuqeHoFaGe/jT33zWe9Rr+XNtVsugnkycGtZMx409WwU9v5WSKuBPuKk7fsk56HUSW62y7/LKXL+zqKVBzXRUSU3xWRFoVo2MNNwzsB2AyrJKX/zllDwF+sRe7apVsP7poUHlTds2Cr6+wlnMM/S8ptj42zH01MpPFz48AMdPl3p94alBzQx89eferk8SpqelICPN/aYWbIXfB6Md7644n24C1cnUrloFV3ULr2XW7Re3Qd7R064galedW9RB5xZ1UFpegYMnzmBgxybo365RUJ2eXenRoCAanv9dZ8zamO81vrZBdymRVjMjDSfOlCH7cXN1VWYkVcCvVyPdqxVLJFx6bmN8tCQX9XRFD/5yyEB0vnZjhuuLS37mCfRxdjOcOfvzmzv+dxZnBPxYRJg7tmZGmquex6j+wmoi4vPpyI6qpKbgrRsqK089v8hlpT/3axtyp3nOdPirY4q2OQ/0w54i/98+eGjouejXLrwm6cFIqoAfLY+P7Ig/92/rVgn56rVd8NrsbSGXWdpFNIKe3js3dXNrETSoUxP8/NAAtGrgvv/0JWGB6ilCoX+PIJibXbT8eN8lrm/7OjnLrWPJdX5EYNmPjOgYdMCP9mc3Q9GsTjU0q+O/dOHuAd71Blay1Zu2iSItNcXrwPZoXR+T7uzlsx22nU9UoLKOwcybzE9d0Qk1DJoSGrm8y1leRT+ewR6o/ArYP6/u7NbqxllRe2Eb/62TfBndJ1O3Dv/FWdHUoWlt9D7bvVeSSXf2ilFqKl13YUuM7NwMYy9rF+ukALDHzdmOmMO3iUh+x9IKFRbk4G7t2yaottyhcN6I9BWhgOOmO/PewK2TgluH4/9Ix3u73/T9qZmRhgl/6B6V78Ma+e7uvpiz6QDsVjhqt2PKgE9BOVd7aSfSj5zhMmod0rGZceukUDnLgW+KUssK5k5D17VlXXRtWRc/bnB81MUGD2PubJIeBnybsFtOwFOdalXC7q41ksJpVx8qu247ebP7dRRrLMO3iUFBvvlL7lzl63bJQpEt8HwwxoBvEwM6BN9BGlW6WOtLqF71+GnSSJHDDL5/LNKhuPbYiI64rW+bgP3OEBFz+BTn0lJT4vbdBk92b6llN766FyffmMMnshvbNTGxpy//3Bt5R08ZTuMuNMaAT5QAZt/fD6dKymOdjKiqU62KV3cU9mulY68EMeATJYD2TVi8oWe3HL5dWg2xDJ+IKEkw4BNRwmDFt38M+ESUcOxShGI3DPhENmG/Csf4w33oHwM+kc0wb2oB7kRDDPhElDDslsG32xOHqYAvIvVFZI6IbNf+N/zahIj8KCJHROQHM+sjIgqG3TL4dmkmajaHPw7APKVUOwDztN9GXgFws8l1ESUFm2UK40qsPsASiF2SZTbgjwLwsTb8MYDfGc2klJoH4LjJdRElNLvkAs2ww2cgATulI9YpcGc24DdRSuVpwwcAmOrUXUTGiEi2iGQXFhaaTBpRfLFLLpASV8CuFURkLoCmBpMe0/9QSikRMXXKKqUmApgIAFlZWTz9KSnZLFMYl7gPjQUM+EqpQb6miUi+iDRTSuWJSDMABZamjoiILGO2SGcagNHa8GgA35tcHhFRwrBbMZ3ZgD8ewGAR2Q5gkPYbIpIlIu87ZxKRRQC+BjBQRPaJyFCT6yUi8mK3AOtkl8pbU90jK6WKAAw0GJ8N4A7d70vMrIeIKBR2CbB2wzdtiWzCppnTuMLeMv1jwCeyGeZOzeMuNMaAT0QJw65l+HbBgE9ECccub9raDQM+ESUM5vD9Y8AnItt5bERHU39vl/y93e4/DPhEZDt39msb1t/ZLcA62eUGxIBPZBcsj7COXSKszTDgE9kMP8AdPrv2h28XDPhElHB40zTGgE9ECYP5e/8Y8Iko4bAZvjEGfCKiCLFbnQIDPpFN2Cs0xKdh5zdFVut6uOeydrFOihu7vPlrqntkIrKeTWJDXKpdtQqm3NUn1smwLebwiYiSBAM+EVGSYMAnIkoSDPhEREmCAZ/IJmzWgo8SEAM+kc2wkU7isNs93FSzTBGpD+BLAJkAcgFcp5Q67DFPVwDvAqgNoBzAC0qpL82sN1FNHtMLVaukxjoZRJSgzObwxwGYp5RqB2Ce9ttTMYBblFLnARgG4E0RqWtyvQmpV9sG6NqybqyTQTFmt1whhc9uT2tmA/4oAB9rwx8D+J3nDEqpbUqp7drwbwAKADQyuV6ihMMXrijSzAb8JkqpPG34AIAm/mYWkZ4A0gHsMLleIiIKUcAyfBGZC6CpwaTH9D+UUkpEfD6NikgzAJ8CGK2UqvAxzxgAYwCgVatWgZJGlFDYSociLWDAV0oN8jVNRPJFpJlSKk8L6AU+5qsNYDqAx5RSy/ysayKAiQCQlZXF05+SEkt2KFLMFulMAzBaGx4N4HvPGUQkHcBUAJ8opaaYXB8RUdywW67VbMAfD2CwiGwHMEj7DRHJEpH3tXmuA9APwJ9EZI32r6vJ9RIRxQ27PLWZaoevlCoCMNBgfDaAO7ThzwB8ZmY9RERkHt+0JSJKEgz4RERJggGfyCaU7ar4KNEw4BPZDN+4pUhhwCciShIM+EREEWK3t6cZ8ImIIswuxXQM+EREScLUi1dEZB27Pf4niiXjLsOhkyWxToYtMOAT2YzY5fk/QZxVtxrOqlst1smwBRbpEBElCQb8JFOnWpVYJ4GIYoRFOklm/oOX4nAxyzOJokHZrGKGAT/J1K+Rjvo10mOdDKKkIjbpIJlFOkQ2Ya+8ICUi5vCJyPY+u/0iZFRh/tQsBnwisr2L2zWMdRISAm+ZRERJggE/TnRoWivWSSCiOMeAHye+H9s31kkgohDVyLBXqbm9UkM+ZaSlxjoJREHr3qpurJNgC1/c2QuzNx1Aner2eOGRAT+ONa1dNdZJIAtd3b053l2wAyM7N4t1UkyZcc8laFnffN81//5Dd7RpWMOCFMVOZsMaGNPv7Fgnw8VUwBeR+gC+BJAJIBfAdUqpwx7ztAYwFY7ioyoA/qWU+o+Z9SaDRQ8PwJmycq/xA85tBACYNrYvO4RKMOc0roXc8SNjnQzTOp1V25LljIjBje+bu3qjXvXEfTHRbA5/HIB5SqnxIjJO+/0Pj3nyAPRWSp0RkZoANojINKXUbybXndBa1q/uNU4fDLq0qBvF1BAlhx6t68c6CRFlNuCPAnCpNvwxgAXwCPhKKX3HLRlgRTHZ2Ge3X4Sik2dinQyiiDAb8JsopfK04QMAmhjNJCItAUwHcA6Ah3zl7kVkDIAxANCqVSuTSSMKHV/woUQWMOCLyFwATQ0mPab/oZRSImLYHYhSai+ALiJyFoDvRGSKUirfYL6JACYCQFZWFrsWISKyUMCAr5Qa5GuaiOSLSDOlVJ6INANQEGBZv4nIBgCXAJgScmqJiChsZsvTpwEYrQ2PBvC95wwi0kJEqmnD9QBcDGCryfUSEVGIzAb88QAGi8h2AIO03xCRLBF5X5unI4DlIrIWwEIAryql1ptcLxERhchUpa1SqgjAQIPx2QDu0IbnAOhiZj1ERGQem0gSESUJBnwioiTBgE9ElCTEbl9VdxKRQgC7TSyiIYCDFiXHrriNiSEZthFIju20wza2Vko1Mppg24BvlohkK6WyYp2OSOI2JoZk2EYgObbT7tvIIh0ioiTBgE9ElCQSOeBPjHUCooDbmBiSYRuB5NhOW29jwpbhExGRu0TO4RMRkQ4DPhFRkki4gC8iw0Rkq4jkaJ9djCsikisi60VkjYhka+Pqi8gcEdmu/V9PGy8i8ra2retEpLtuOaO1+beLyGhf64sWEflQRAq07rGd4yzbLhHpoe23HO1vJbpb6HMbnxaR/drxXCMiI3TTHtHSu1VEhurGG57DItJGRJZr478Ukah/fFVEWorIfBHZJCIbReRebXzCHEs/2xj/x1IplTD/AKQC2AGgLYB0AGsBdIp1ukLchlwADT3GvQxgnDY8DsA/teERAGYCEAC9ACzXxtcHsFP7v542XC/G29UPQHcAGyKxXQB+1eYV7W+H22QbnwbwoMG8nbTzMwNAG+28TfV3DgP4CsAN2vB/ANwVg21sBqC7NlwLwDZtWxLmWPrZxrg/lomWw+8JIEcptVM5vqU7GY7v7sa7UXB8Mxja/7/Tjf9EOSwDUFccH6IZCmCOUuqQUuowgDkAhkU5zW6UUj8DOOQx2pLt0qbVVkotU44r6BPdsqLGxzb6MgrAZKXUGaXULgA5cJy/huewlsu9DJUfDtLvr6hRSuUppVZpw8cBbAbQHAl0LP1soy9xcywTLeA3B7BX93sf/B8oO1IAZovISnF84xfw/e1gX9sbL/vBqu1qrg17jreLsVpxxofOog6Evo0NABxRSpV5jI8ZEckE0A3AciTosfTYRiDOj2WiBfxEcLFSqjuA4QDuFpF++olarifh2tIm6nYBeBfA2QC6AsgD8FpMU2MREakJ4BsA9ymljumnJcqxNNjGuD+WiRbw9wNoqfvdQhsXN5RS+7X/CwBMheOxMF971IW4fzvY1/bGy36warv2a8Oe42NOKZWvlCpXSlUA+C8cxxMIfRuL4CgOSfMYH3UiUgWOQPi5UupbbXRCHUujbUyEY5loAX8FgHZaDXg6gBvg+O5uXBCRGiJSyzkMYAiADfD97eBpAG7RWkL0AnBUe6yeBWCIiNTTHjuHaOPsxpLt0qYdE5FeWvnoLTD4vnIsOIOg5io4jifg2MYbRCRDRNoAaAdHZaXhOazlmucDuEb7e8NvSEeatn8/ALBZKfW6blLCHEtf25gQxzIaNcPR/AdHq4BtcNSOPxbr9ISY9rZw1OSvBbDRmX44yvzmAdgOYC6A+tp4ATBB29b1ALJ0y7oNjsqjHAC32mDbvoDjMbgUjjLL263cLgBZcFyAOwC8A+0tchts46faNqyDIzA0083/mJberdC1RPF1Dmvnx6/atn8NICMG23gxHMU16wCs0f6NSKRj6Wcb4/5YsmsFIqIkkWhFOkRE5AMDPhFRkmDAJyJKEgz4RERJggGfiChJMOATESUJBnwioiTx/yZd1VCe3TSwAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(h[0]);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "675da5fc-df58-4789-97a6-13b1ecef9db2",
   "metadata": {},
   "source": [
    "## Truncated SVD\n",
    "\n",
    "- Big Matrices\n",
    "- Missing / Inaccurate Data - why spend compute resources when impression of input limits precision of output\n",
    "- Data transfer is often bottleneck - Algorithms with fewer passes are faster ( even if they require more floating point operations / flops)\n",
    "- GPU advantage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21a26d68-667f-452d-abe5-55014d9ffd2a",
   "metadata": {},
   "source": [
    "### Full SVD vs Randomized SVD\n",
    "- Randomized often inherently stable\n",
    "- performance guarantees do not depend on subtle spectral properties\n",
    "- Matrix-vector products can be done in parallel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e8004b0-0592-4366-b448-7a3023b144e4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<2034x26576 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 133634 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectors = vectorizer.fit_transform(newsgroups_train.data); vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "197cda74-61e5-4d58-bcb9-fd21ed2cf21e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2034, 26576)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectors.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8855cf7c-df4e-4207-9a2b-8def785d20fe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 14min 28s, sys: 19.3 s, total: 14min 48s\n",
      "Wall time: 3min 47s\n"
     ]
    }
   ],
   "source": [
    "%time U, s, Vh = sp.linalg.svd(vectors.todense(), full_matrices=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acbbb611-1142-4178-972c-527b16b0627e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 32s, sys: 7.18 s, total: 1min 39s\n",
      "Wall time: 26 s\n"
     ]
    }
   ],
   "source": [
    "%time U, s, Vh = sp.linalg.svd(vectors.todense(), full_matrices=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25d4fa72-6808-4502-8243-fd12e37e7ed1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2034, 2034) (2034,) (2034, 26576)\n"
     ]
    }
   ],
   "source": [
    "print(U.shape, s.shape, Vh.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "942fdec6-5741-4725-8340-ced770fc0471",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 282 ms, sys: 229 ms, total: 511 ms\n",
      "Wall time: 131 ms\n"
     ]
    }
   ],
   "source": [
    "%time u, s, v = decomposition.randomized_svd(vectors, 5, random_state=23)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f89cadf-14e4-4478-89a9-27c50cdf6683",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2034, 2034) (5,) (2034, 26576)\n"
     ]
    }
   ],
   "source": [
    "print(U.shape, s.shape, Vh.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c34ead8a-68bf-4c83-930c-30b570c5d6d1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 10.7 s, sys: 3.66 s, total: 14.3 s\n",
      "Wall time: 3.76 s\n"
     ]
    }
   ],
   "source": [
    "%time u, s, v = decomposition.randomized_svd(vectors.todense(), 5, random_state=23)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "328ef31e-ae46-4df3-9161-2ae3aa70db4b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['does like know space think just don people',\n",
       " 'file ftp nasa image files program thanks graphics',\n",
       " 'earth station lunar orbit moon shuttle launch nasa',\n",
       " 'vice queens sank manhattan bronx beauchaine tek bobbe',\n",
       " 'christ nasa believe atheism satan bible space jesus']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_topics(v, vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6001ce2-7348-45cf-b9a8-c9fd546bee93",
   "metadata": {},
   "source": [
    "### Implementing Randomized "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "371ff6dd-fc70-4110-8d8e-e5a61a3bc341",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.78584373, -1.15584006,  1.02165028],\n",
       "       [ 1.30688763, -2.40217237, -1.33608905],\n",
       "       [ 0.18773331,  0.2357126 , -1.24552412]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Q = np.random.randint(1,10, 9).reshape([3,3]); Q\n",
    "Q = np.random.randn(3,3); Q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb48c04e-1870-47e0-ab52-987fc1e1c7a8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.78584373,  1.30688763,  0.18773331],\n",
       "       [-1.15584006, -2.40217237,  0.2357126 ],\n",
       "       [ 1.02165028, -1.33608905, -1.24552412]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Q.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc2bda87-e6dc-41d3-a1d5-efe4e270d85e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 2.99728591,  0.38450186, -1.69246517],\n",
       "       [ 0.38450186,  9.26352133,  1.34325519],\n",
       "       [-1.69246517,  1.34325519,  1.64213456]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Q@Q.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66e6c903-e198-4248-aeb3-9c2e18182987",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10, 9)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A = np.random.normal(size=(10,9))\n",
    "A.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00f22a6b-7d59-482d-a653-21f3fe15e4e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9, 3)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = 3\n",
    "size = (A.shape[1], s)\n",
    "size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd6a0bd8-0864-45ee-aa05-61437838206f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9, 3)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Q = np.random.normal(size=size);Q.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe8738a1-3cff-4fbf-ae23-65c3448e155c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10, 3)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(A@Q).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2d628f8-a58f-4beb-9f1a-3423dd485ea0",
   "metadata": {},
   "outputs": [],
   "source": [
    "Q, tmp = sp.linalg.lu(A@Q, permute_l=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d77f49f8-3029-40a6-bf26-793b6ca8fbc3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10, 3)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Q.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa9be558-3599-4b1b-bb00-0fbb4da447a9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 3)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10e05fc4-162c-4415-a827-3de3400eb2f6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((10, 9), (9, 3))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A.shape, (A.T@Q).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "638cb19b-a8ce-42d6-bfdb-45f9367d0a6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "Q, tmp = sp.linalg.lu(A.T@Q, permute_l=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09ddbb1f-f0a9-4511-8dec-dcd19818d40e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9, 3)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Q.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bdc4af8d-1ad6-4ace-be2f-b6f724e60f96",
   "metadata": {},
   "outputs": [],
   "source": [
    "Q, tmp = sp.linalg.qr(A@Q, mode='economic')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aae966a1-acbc-47e8-bb85-34294d52fefa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9, 3)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Q.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "407d78f8-1164-45c6-97dd-b59aa45bd7cd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 3)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ec8f78c-8ce7-4e06-b466-f020879a3882",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 3.95035136, -3.38329669, -1.37453569],\n",
       "       [ 0.        ,  3.6106873 ,  4.23593696],\n",
       "       [ 0.        ,  0.        , -5.28262572]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "743ccf84-5fb5-43bc-a6d0-29562d0e6ad6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.10075212,  0.21837398,  0.11792651],\n",
       "       [ 1.        ,  0.        ,  0.        ],\n",
       "       [-0.03682198,  0.44183267,  1.        ],\n",
       "       [-0.44080867, -0.37256568,  0.02324834],\n",
       "       [-0.70322551, -0.15386372,  0.21592698],\n",
       "       [ 0.08350268, -0.0905467 ,  0.14158361],\n",
       "       [ 0.50762928,  0.84231332,  0.8683797 ],\n",
       "       [ 0.3939241 ,  1.        ,  0.        ],\n",
       "       [-0.96173764,  0.53149523,  0.20955641]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35890771-e374-4f11-b8b7-e8c8b033e6dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "def randomized_range_finder(A, size, n_iter=5):\n",
    "    Q = np.random.normal(size=(A.shape[1], size))\n",
    "    for i in range(n_iter):\n",
    "        Q, _ = sp.linalg.lu(A@Q, permute_l=True)\n",
    "        Q, _ = sp.linalg.lu(A.T@Q, permute_l=True)\n",
    "    Q, _ = sp.linalg.qr(A@Q, mode='economic')\n",
    "    return Q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b1fcc51-ef7b-48e5-9950-19a4ae865510",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2034, 26576)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_components=5\n",
    "n_oversamples=10\n",
    "n_iter=4\n",
    "\n",
    "vectors.shape\n",
    "# M.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfc5dc51-1a42-44ef-b101-3f6fd5b8de80",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.02137672,  0.02867907,  0.00147528, ...,  0.01530008,\n",
       "         0.0131687 , -0.00112568],\n",
       "       [-0.01125351, -0.007019  , -0.00373134, ...,  0.00326252,\n",
       "         0.02130407,  0.01448943],\n",
       "       [-0.01073123, -0.00256461, -0.00537675, ...,  0.00307844,\n",
       "        -0.00575456,  0.0005117 ],\n",
       "       ...,\n",
       "       [-0.0231287 ,  0.03040804,  0.00351839, ...,  0.01126928,\n",
       "         0.05252739, -0.01553131],\n",
       "       [-0.03342604, -0.02780798, -0.02251785, ..., -0.07096806,\n",
       "        -0.00142386, -0.04975683],\n",
       "       [-0.        , -0.        ,  0.        , ..., -0.        ,\n",
       "        -0.        ,  0.        ]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_random = n_components + n_oversamples\n",
    "Q = randomized_range_finder(vectors, n_random, n_iter)\n",
    "Q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55db4bf7-8c67-4a93-837b-d61f28492bb8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2034, 15)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Q.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "190130d0-e936-4391-9582-873b8b959c9b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2034, 26576)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectors.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5794af07-3e2c-44ff-a14b-58aab686daec",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(15, 26576)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "M = vectors.copy()\n",
    "B = Q.T@M\n",
    "B.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db179d14-bdae-4a6c-975a-8bbff9ad4856",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 54.3 ms, sys: 24.1 ms, total: 78.4 ms\n",
      "Wall time: 24.8 ms\n"
     ]
    }
   ],
   "source": [
    "%time Uhat, s, V =  sp.linalg.svd(B, full_matrices=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ab34fc0-d53d-4c06-be4c-539031d7ddcf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((15, 15), (15,), (15, 26576))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Uhat.shape, s.shape, V.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93e861dd-3b4a-40b1-ad53-03d23507ceda",
   "metadata": {},
   "outputs": [],
   "source": [
    "del B\n",
    "U = Q@Uhat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15bee320-4439-4794-9624-fca68684d37d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2034, 15)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "U.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66c1efe6-f115-4880-9305-0730f5318844",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2034, 5)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "U[:,:n_components].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14327004-471d-4424-8403-08d949cbc56e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5,)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s[:n_components].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f9b2a96-7bc0-4126-9ec0-90c7ceb17dd0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5, 26576)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "V[:n_components,:].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2fd4e0b-0cc6-4f63-9dd9-d090ff571e63",
   "metadata": {},
   "outputs": [],
   "source": [
    "def randomized_svd(M, n_components, n_oversamples=10, n_iter=4):\n",
    "    n_random = n_components + n_oversamples\n",
    "    Q = randomized_range_finder(M, n_random, n_iter)\n",
    "    B = Q.T@M\n",
    "    Uhat, s, V = sp.linalg.svd(B, full_matrices=False)\n",
    "    del B\n",
    "    U = Q@Uhat\n",
    "    return U[:,:n_components], s[:n_components], V[:n_components, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e3b3e5a-fd06-4160-a77c-7b26ab58983c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 224 ms, sys: 131 ms, total: 355 ms\n",
      "Wall time: 103 ms\n"
     ]
    }
   ],
   "source": [
    "%time u, s, v = randomized_svd(vectors, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91194c4e-f3fc-4651-864d-e361b680a5c3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((2034, 5), (5,), (5, 26576))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "u.shape, s.shape, v.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b01470c7-58b2-4de9-8f99-04fbf5c7e7a4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['does like know space think just don people',\n",
       " 'file ftp nasa image files program thanks graphics',\n",
       " 'earth station lunar orbit moon shuttle launch nasa',\n",
       " 'vice queens sank manhattan bronx beauchaine tek bobbe',\n",
       " 'like values people moral just don morality think']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_topics(v, vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13c82ade-1b1f-440a-9c22-307819273815",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2034,)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "UV=u@s\n",
    "# M_recon= US@V\n",
    "# M_recon.shape\n",
    "UV.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63be9b41-50fa-4cbe-9302-beca7a225b67",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5, 26576)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43d611d0-f223-46cb-9d14-99d9a5b627ed",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2034, 26576)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "M_recon = u@np.diag(s)@v\n",
    "M_recon.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f0eea74-8cc3-4337-929c-d9f082251438",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "43.70003415792525"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.linalg.norm(M-M_recon)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0691be3-0d8a-47bb-a2ed-3b45e496c8f9",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}