There are many ways to estimate the readability of text, often of interest to educators when determining the appropriate age/grade level for various books etc.

In this assignment you will implement several automated readability tests on English text. We will work with the following definitions:

• A word shall be any whitespace-delimited token that remains non-empty after removing all characters other than the following (case-insensitive) and stripping single-quotes and hyphens from both ends:

  abcdefghijklmnopqrstuvwxyz'-

• A sentence shall be denoted by any whitespace-delimited token that ends with any of the following, ignoring any single or double quotes at the end of the token:

  .?!

• A character shall be any character in a word, as defined above.
• The number of syllables in a word shall be:
  • The number of syllables indicated in file /srv/datasets/syllables.txt, if the file contains an entry for the word.
    • Some words have multiple syllable counts. In that case, opt for the highest count.
  • If the above file does not contain an entry for the word, estimate that each sequence of five characters in a word represents a syllable, and round to the nearest syllable count for the word.

If you are unsure of how a segment of text should be classified with regard to the above definitions, consider running the sample executable with the -v flag as described below.

You will be implementing the following readability tests, which I have slightly simplified in some cases.

The automated readability index (ARI) produces an estimate of the US grade level needed to comprehend a text. Here is the formula:

$$ 4.71 \left (\frac{\mbox{characters}}{\mbox{words}} \right) + 0.5 \left (\frac{\mbox{words}}{\mbox{sentences}} \right) - 21.43 $$

characters is the total number of characters in all words, words is the total number of words, and sentences is the total number of sentences.

The Coleman-Liau index produces an estimate of the US grade level needed to comprehend a text. Here is the formula:

$$ 0.0588{L} - 0.296{S} - 15.8\,\! $$

L is the average number of characters per 100 words and S is the average number of sentences per 100 words.

The Dale-Chall readability score produces a numeric measure of how difficult a text is to comprehend. Here is the formula:

$$ 0.1579 \left (\frac{\mbox{difficult words}}{\mbox{words}}\times 100 \right) + 0.0496 \left (\frac{\mbox{words}}{\mbox{sentences}} \right) $$

difficult words is the number of words that are not present in a list of 3000 “easy” words (available in file /srv/datasets/dale-chall_familiar_words.txt), words is the total number of words, and sentences is the total number of sentences.

The Flesch-Kincaid grade level produces an estimate of the US grade level needed to comprehend a text. Here is the formula:

$$ 0.39 \left ( \frac{\mbox{words}}{\mbox{sentences}} \right ) + 11.8 \left ( \frac{\mbox{syllables}}{\mbox{words}} \right ) - 15.59 $$

words is the total number of words, sentences is the total number of sentences, and syllables is the total number of syllables.

The Gunning fog index produces an estimate of the number of years of education needed to comprehend a text. Here is the formula:

$$ 0.4\left[ \left(\frac{\mbox{words}}{\mbox{sentences}}\right) + 100\left(\frac{\mbox{complex words}}{\mbox{words}}\right) \right] $$

words is the total number of words, sentences is the total number of sentences, and complex words is the total number of words consisting of three or more syllables.

The SMOG ("Simple Measure of Gobbledygook") grade produces an estimate of the number of years of education needed to comprehend a text. Here is the formula:

$$ \mbox{grade} = 1.0430 \sqrt{\mbox{complex words}\times{30 \over \mbox{sentences}} } + 3.1291 $$

sentences is the total number of sentences, and complex words is the total number of words consisting of three or more syllables.

#include <algorithm>
#include <cmath>
#include <fstream>
#include <iostream>
#include <string>
#include <tuple>
#include <unordered_map>
#include <unordered_set>
#include <vector>

std::cerr
std::cin
std::cout
std::fixed
std::getline
std::ifstream
std::max
std::ofstream
std::round
std::sqrt
std::string
std::transform
std::tuple
std::unordered_map
std::unordered_set
std::vector

Executable cs19_readability exists on the server, and demonstrates the expected behavior of your program, along with an option to help you check your work. Your program (when compiled with -Ofast) will be expected to terminate in no more than three times the runtime duration of the sample executable, given the same input.

% cs19_readability smog </srv/datasets/dewey-moral-principles-education.txt 
14.566
 
% cs19_readability smog </srv/datasets/cat-in-the-hat.txt
3.647

Adding a -v flag as the second command-line argument will print the relevant counts in the text to stderr. You don't need to implement this in your program, but feel free to use it for testing purposes, e.g.:

% cs19_readability fkgl -v <<<"This is a *relatively* short text with nine words."
9 words 1 sentence 10 syllables
1.031
 
% cs19_readability gfi -v </srv/datasets/cat-in-the-hat.txt
1620 words 243 sentences 2 complex words
2.716

head -500 /srv/datasets/shakespeare-macbeth.txt | cs19_readability ari

As submissions are received, these leaderboards will be updated with the top-performing fully functional/near-perfect solutions for each readability test, with regard to execution speed.

<html>

<script> function updateLeaderboards() {

document.getElementById('leaderboards-updated').innerText = `(Last updated: ${new Date()})`;

let leaderboards = document.getElementById('leaderboards-content');
while (leaderboards.firstChild)
  leaderboards.removeChild(leaderboards.firstChild);

['ari', 'cli', 'dcrs', 'fkgl', 'gfi', 'smog'].forEach(readabilityTest => {

  let testHeader = document.createElement('h3');
  testHeader.textContent = readabilityTest;
  leaderboards.appendChild(testHeader);

  let table = document.createElement('table');
  let tHead = document.createElement('thead');
  let headRow = document.createElement('tr');
  let thRank = document.createElement('th');
  thRank.textContent = 'Rank';
  let thTestTime = document.createElement('th');
  thTestTime.textContent = 'Test Time (s)';
  let thMemUsage = document.createElement('th');
  thMemUsage.textContent = 'Memory Usage (kB)';
  let thSloc = document.createElement('th');
  thSloc.textContent = 'SLOC (lines)';
  let thUser = document.createElement('th');
  thUser.textContent = 'User';
  headRow.appendChild(thRank);
  headRow.appendChild(thTestTime);
  headRow.appendChild(thMemUsage);
  headRow.appendChild(thSloc);
  headRow.appendChild(thUser);
  tHead.appendChild(headRow);
  table.appendChild(tHead);
  let tBody = document.createElement('tbody');
  table.appendChild(tBody);

  window.fetch(`/~turnin/leaderboard-cs19f22as05-${readabilityTest}.txt?t=${new Date().getTime()}`, {
    method: 'get'
  }).then(response =>
    response.text()
  ).then(text => {
    let lines = text.split('\n');
    for (let i = 0; i < lines.length; ++i) {
      let tokens = lines[i].split(' ').filter(token => token.length > 0);
      if (tokens.length < 2)
        continue;
      let tdRank = document.createElement('td');
      tdRank.textContent = i + 1;
      let tdTime = document.createElement('td');
      tdTime.textContent = Number(tokens[0]).toFixed(4);
      let tdMemUsage = document.createElement('td');
      tdMemUsage.textContent = tokens[1];
      let tdSloc = document.createElement('td');
      tdSloc.textContent = tokens[2];
      let tdUser = document.createElement('td');
      let userLink = document.createElement('a');
      userLink.href = `/~${tokens[3]}/`;
      userLink.target = '_blank';
      userLink.textContent = tokens[3];
      tdUser.appendChild(userLink);
      let tr = document.createElement('tr');
      tr.appendChild(tdRank);
      tr.appendChild(tdTime);
      tr.appendChild(tdMemUsage);
      tr.appendChild(tdSloc);
      tr.appendChild(tdUser);
      tBody.appendChild(tr);
    }
  }).catch(err => {
    console.log('Something bad happened: ' + err);
  });
  leaderboards.appendChild(table);
});
window.setTimeout(updateLeaderboards, 60000);

} updateLeaderboards(); </script> </html>

Submit your source-code file(s) via turnin. If you submit multiple source-code files, make sure there is only one main() function.