TechieYan Technologies

Converting words to numbers using gensim word2vec (sports and outdoors dataset)


Computers do not understand words; we need to convert them to numbers to make sure computer understands them. We use gensim library to convert word to vector form. These word embeddings on the sports and outdoors dataset t will help us get similar words. After generating the word embeddings we can check similarity between words, most similar words to a given word. These embeddings are the basic building blocks for building higher level NLP projects in the future.


We used gensim library to generate the word embeddings. Gensim is a great library to do NLP tasks. To read more about gensim and word embeddings you can refer to this article –

Word embeddings – Representation of word into a vector format where words with similar meanings have similar vector representation.

To give you a simple explanation of word embeddings, let us take 2 words. 1- India , 2 – China. Both are countries, have strong military, high population and hence similar in some way. So the word embeddings generated would look like India – [0,0.9,0.2,0.85,0.7] and China [0,0.8,0.3,0.80,065].

We get these word embeddings after training the model on the data and before training we preprocess the data, clean it, remove punctuations and stop words. Convert all words to lower case and in a standardized way. After training, we get the individual word embeddings.


Data Description

The dataset contains reviews on amazon products. It has 9 columns like reviewerID, reviewerName, reviewText. “reviewText” column is what we will use to generate the word embeddings and contains the reviews given by users on a specific product be it good or bad.

Converting words to numbers data description

How to Execute?

So, before execution we have some pre-requisites that we need to download or install i.e., anaconda environment, python and a code editor. 

Anaconda: Anaconda is like a package of libraries and offers a great deal of information which allows a data engineer to create multiple environments and install required libraries easy and neat.

  1. Refer to this, if you are just starting and want to know how to install anaconda.

    If you already have anaconda and want to check on how to create anaconda environment, refer to this article set up jupyter notebook. You can skip the article if you have knowledge of installing anaconda, setting up environment and installing requirements.txt

    Install necessary libraries from requirements.txt file provided.

Install libraries

     2.Go to the directory where your requirement.txt file is present.

       e.g. cd C:\Users\Hi\word2vecgensim, this is just  an example. Set appropriate path as in your computer.

path of program
  1. Run command pip install -r requirements.txt or conda install requirements.txt (Requirements.txt is a text file consisting of all the necessary libraries required for executing this python file. If it gives any error while installing libraries, you might need to install them individually.)
conda installation

All the  necessary files will get downloaded.

To run the code, start jupyter notebook, open folder where your code is present.

jupyter notebook

When you run the sports_ word2vec.ipynb file, you get the appropriate results.

result of the program


Converting words to numbers
Converting words to numbers using gensim word2vec

Issues you may face while executing the code

  1. Make sure to change the path in the code. Give full path of dataset/csv file you want to use.
  2. Make sure you have the appropriate versions of the given libraries.

          Click here to download the code and associated files.

+91 7075575787