Learn to Use Markov Chains by Building a Lyrics Generator That Mixes Gospel and Brazilian Gangsta Funk

Using Python and a bit of creativity, you can generate some pretty funny lyrics.

A little nerd humor:

I was studying Markov Chains and how to use Python to work with them.

I ended up building a song lyrics generator.

Naturally, I decided to mix funk proibidão (Brazilian gangsta funk) with gospel music to see what comes out.

(Note: If you don't find this as funny as I do, you're probably a normal person and I just have a really weird sense of humor.)

This was one of the lyrics:

It's not very coherent (I couldn't get it to rhyme!), but it was great for laughing and for learning Markov Chains.

How I did it: I built a scraper that downloaded the lyrics from Vagalume, then used a simple chain to generate the lyrics.

Then I found out there was a library that does it 10x better. (obviously!)/

What is a Markov Chain?

A Markov Chain is a model that computes a sequence of events.

That sequence is computed in such a way that the probability of an event happening depends on the previous one.

This model is brilliant for working with text: it generates sentences by choosing words based on the previous ones, computing which words are most likely to come next.

How to build a Markov Chain in Python

Now, on to our song lyrics generator.

The first part of the process involves having lots of lyrics in a text file, to train our chain on the probabilities of word sequences.

For that, I built a scraper for the Vagalume website, which downloads all of an artist's lyrics into a text file. I'll write a detailed post on how to do that soon. In the meantime, you can check out the final code.

The second part of the process is writing the Markov Chain itself:

We start by generating a random word.

After that, every other word is generated based on the probability of it following the previous one.

Shall we?

import random
import os

#Abrindo e lendo um arquivo de texto com as letras
lyrics = open("letras.txt","r")
lyrics = lyrics.read()

## Criando uma lista de palavras baseado no texto. 
## Aqui nós: substituímos as quebras de linha por espaços
## separamos o string pelos espaços, 
## usamos um filtro para remover valores vazios da lista

wordList = lyrics.replace('\n', ' ') 
wordList = wordList.split(' ') 
wordList = list(filter(None, wordList))

We now have a cleaned-up list with every word in our text file. The next step is turning it into a probability dictionary:

index = 1
chain = {} ## nosso dicionário
wordCount = 5 ## o número de palavras geradas

## um loop que busca todas as palavras na nossa lista, e coloca no dicionário uma nova chave com cada palavra, e seu valores, todas as palavras que as sucedem 
for word in wordList[index:]:
    key = wordList[index - 1]
    if key in chain:
        chain[key].append(word)
    else:
        chain[key] = [word]
    index += 1

Now, we start generating the lyrics:

#Uma palavra inicial aleatória
notCapitalized = True

while notCapitalized:
    firstWord = random.choice(list(chain.keys()))
    if firstWord[0].isupper():
        notCapitalized = False

We build a Loop that picks a random word based on the successors:

iterations = 0
while len(line.split(' ')) < wordCount:
    capitalized = True
    while capitalized:
        iterations += 1
        nextWord = random.choice(chain[firstWord])
        if nextWord[0].islower():
            capitalized = False
        if iterations > 15:
            capitalized = False
            iterations = 0
    firstWord = nextWord
    line += ' ' + nextWord

Tying it all together in a function:

def generateLine():
    wordCount = 5 ## o número de palavras geradas
    notCapitalized = True

    while notCapitalized:
        firstWord = random.choice(list(chain.keys()))
        if firstWord[0].isupper():
            notCapitalized = False

    line = firstWord
    iterations = 0

    while len(line.split(' ')) < wordCount:
        capitalized = True
        while capitalized:
            iterations += 1
            nextWord = random.choice(chain[firstWord])
            if nextWord[0].islower():
                capitalized = False
            if iterations > 15:
                capitalized = False
                iterations = 0
        firstWord = nextWord
        line += ' ' + nextWord
    return line

Now, we write a Loop that generates verses based on those lines:

index = 0
verseList = list()

while index < 12:
    verseList.append(generateLine())
    if index == 5:
        firstVerse = verseList
        verseList = list()
        print("\n")
    if index == 11:
        for y in firstVerse:
            print(y)
        print("\n")
        for y in verseList:
            print(y)
        print("\n")        
        for y in firstVerse:
            print(y)
        print("\n")
        for y in verseList:
            print(y)
    index += 1

Code

In the end, I found a library called pymarkovchain, which made the lyrics more coherent.

If you're only interested in using Markov chains to learn, it's much more productive to build one from scratch, but if you want to create something more coherent, it's worth taking a look at the existing libraries.

Markov Chain

## Esse método é baseado no código de Mehrab Jamee 
import random
import os

#Abrindo e lendo um arquivo de texto com as letras
lyrics = open("letras.txt","r")
lyrics = lyrics.read()

## Criando uma lista de palavras baseado no texto. 
## Aqui nós: substituímos as quebras de linha por espaços
## separamos o string pelos espaços, 
## usamos um filtro para remover valores vazios da lista

wordList = lyrics.replace('\n', ' ') 
wordList = wordList.split(' ') 
wordList = list(filter(None, wordList))

index = 1
chain = {} ## nosso dicionário
wordCount = 5 ## o número de palavras geradas

## um loop que busca todas as palavras na nossa lista, e coloca no dicionário uma nova chave com cada palavra, e seu valores, todas as palavras que as sucedem 
for word in wordList[index:]:
    key = wordList[index - 1]
    if key in chain:
        chain[key].append(word)
    else:
        chain[key] = [word]
    index += 1

## nossa função que encontra as palavras seguintes aleatoriamente, baseado no dicionário e nas anteriores 
def generateLine():
    wordCount = 5 ## o número de palavras geradas
    notCapitalized = True

    while notCapitalized:
        firstWord = random.choice(list(chain.keys()))
        if firstWord[0].isupper():
            notCapitalized = False

    line = firstWord
    iterations = 0

    while len(line.split(' ')) < wordCount:
        capitalized = True
        while capitalized:
            iterations += 1
            nextWord = random.choice(chain[firstWord])
            if nextWord[0].islower():
                capitalized = False
            if iterations > 15:
                capitalized = False
                iterations = 0
        firstWord = nextWord
        line += ' ' + nextWord
    return line


## a função que gera estrofes:
index = 0
verseList = list()

while index < 12:
    verseList.append(generateLine())
    if index == 5:
        firstVerse = verseList
        verseList = list()
        print("\n")
    if index == 11:
        for y in firstVerse:
            print(y)
        print("\n")
        for y in verseList:
            print(y)
        print("\n")        
        for y in firstVerse:
            print(y)
        print("\n")
        for y in verseList:
            print(y)
    index += 1

BeautifulSoup as a Scraper and pymarkovchain

```python
import os
from bs4 import BeautifulSoup
from pymarkovchain import MarkovChain
import urllib.request
```

## Vamos buscar o nome de todas as músicas do Mc Daleste

htmlArtista = urllib.request.urlopen("https://www.vagalume.com.br/mc-daleste/")
soup = BeautifulSoup(htmlArtista, 'html.parser')
nameBox = soup.find("ul", attrs={"class": "tracks"})
nameBox = nameBox.find_all("a")


## Colocando em uma lista somente as urls para facilitar o trabalho
nameList = list()
for i in nameBox:   
    nameList.append(i['href'])

## Agora, nós usamos a urllib para fazer o download de todas as páginas. (você também pode usar requests)
htmlList = list()
for name in nameList:
    htmlList.append(urllib.request.urlopen("https://www.vagalume.com.br" + name))    


## Aqui, extraímos letra das páginas 
songList = list()
for page in htmlList:
    soup = BeautifulSoup(page, 'html.parser')
    nameBox = soup.find("div", attrs={"itemprop": "description"})
    lines = str(name_box).replace("<br/>", "\n")
    lines = lines.replace('<div itemprop="description">', '')
    lines = lines.replace('</div>', '')
    songList.append(lines)

## e finalmente salvamos o resultado em um arquivo de texto, para facilitar a manipulação posterior
file = open("daleste.txt","w") 
for letra in song_list:
    file.write(letra)

## Agora começamos com a Markov Chain
mc = MarkovChain("./markov")

## eu juntei algumas letras diferentes usando o scraper de cima.

songlist = open("todas_letras_gospel.txt","r")
songlist2 = open("todas_letras.txt","r")
songlist3 = open("todas_letras_bk.txt","r")
songlist4 = open("mr-catra.txt","r")
songlist5 = open("daleste.txt","r")

## eu sei que isso é feio..

read = songlist.read()
read2 = songlist2.read()
read3 = songlist3.read()
read4 = songlist4.read()
read5 = songlist5.read()

read6 = read + read2 + read3 + read4 + read5

#geramos uma database usando todo esse texto que está junto:
mc.generateDatabase(read6)

# isso deve gerar uma estrofe - pra mim saiu: 'Ou me mama porque tu me abraça forte e não solte jamais' 
mc.generateString()

#aí eu faço esse loopzinho pra gerar uma letra completa:

i = 0
verseList = list()
while i < 12:
    verseList.append(mc.generateString())
    if i == 5:
        firstVerse = verseList
        verseList = list()
        print("\n")
    if i == 11:
        for y in firstVerse:
            print(y)
        print("\n")
        for y in verseList:
            print(y)
        print("\n")        
        for y in firstVerse:
            print(y)
        print("\n")
        for y in verseList:
            print(y)
    i = i + 1

One more bonus lyric for those who made it this far: