# Minimal Word Search

Last week, we worked to create the shortest 1-D string using the top 10,000 words in the English language. Now, lets try the same challenge in 2D!

What you need to do is to take all of the above words, and put them in a rectangle as small as possible, allowing for overlaps. For example, if your words were ["ape","pen","ab","be","pa"], then a possible rectangle would be:

.b..
apen


The above rectangle would give a score of 5.

Rules:

• Overlapping multiple letters in a word is allowed
• Words can go in any of the 8 directions
• Words cannot wrap around
• You can use any character for the empty locations

You need to create a word search that contains these top 10,000 words in English (according to Google). Your score is equal to the number of characters in your word search (excluding unused characters). If there is a tie, or if a submission is proven to be optimal, then the submission that is first posted wins.

• I'd like to note that I am aware of this previous word search challenge, but given that none of the answers there will run in a reasonable amount of time for this challenge, I don't believe its a duplicate. Commented Aug 12, 2016 at 14:52
• Related. Commented Aug 12, 2016 at 15:58
• I fear the optimal solution will turn out to be an n x 1 grid, making this problem ultimately the same as the last one (reasoning: tangent intersections will rarely save many characters but will often introduce "holes", wasting space). Maybe you should score it on the width + height, rather than width * height, so that it strongly favours square solutions (more interesting).
– Dave
Commented Aug 13, 2016 at 18:04
• Hmmm... I fear that solutions will simply be word strings stacked on top of each other, then. I think not scoring empty locations might be a good idea Commented Aug 13, 2016 at 18:10
• Risk with that is there's no need to keep the grid size small; a 1000x1000 grid with a sprawling horizontal and vertical list would score the same as a tightened spiral pattern / similar. Maybe try width+height, then letters-excluding-blanks as a tie-breaker? Might need a bit more thought. Edit: or maybe letters-excluding-blanks first then width+height as a tie-breaker would work better.
– Dave
Commented Aug 13, 2016 at 18:25

# Rust, 31430 30081 used characters

This is a greedy algorithm of sorts: we start with an empty grid, and repeatedly add the word that can be added with the fewest new letters, with ties broken by preferring longer words. To make this run quickly, we maintain a priority queue of candidate word placements (implemented as a vector of vectors of deques, with a vector for each number of new letters, containing a deque for each word length). For each newly added letter, we enqueue all candidate placements that run through that letter.

Compile and run with rustc -O wordsearch.rs; ./wordsearch < google-10000-english.txt. On my laptop, this runs in 70 seconds, using 531 MiB RAM.

The output fits in a rectangle with 248 columns and 253 rows.

### Code

use std::collections::{HashMap, HashSet, VecDeque};
use std::io::prelude::*;
use std::iter::once;
use std::vec::Vec;

type Coord = i16;
type Pos = (Coord, Coord);
type Dir = u8;
type Word = u16;

struct Placement { word: Word, dir: Dir, pos: Pos }

static DIRS: [Pos; 8] =
[(1, 0), (1, 1), (0, 1), (-1, 1), (-1, 0), (-1, -1), (0, -1), (1, -1)];

fn fit(grid: &HashMap<Pos, u8>, (x, y): Pos, d: Dir, word: &String) -> Option<usize> {
let (dx, dy) = DIRS[d as usize];
let mut n = 0;
for (i, c) in word.bytes().enumerate() {
if let Some(c1) = grid.get(&(x + (i as Coord)*dx, y + (i as Coord)*dy)) {
if c != *c1 {
return None;
}
} else {
n += 1;
}
}
return Some(n)
}

struct PlacementQueue { queue: Vec<Vec<VecDeque<Placement>>>, extra: usize }

impl PlacementQueue {
fn new() -> PlacementQueue {
return PlacementQueue { queue: Vec::new(), extra: std::usize::MAX }
}

fn enqueue(self: &mut PlacementQueue, extra: usize, total: usize, placement: Placement) {
while self.queue.len() <= extra {
self.queue.push(Vec::new());
}
while self.queue[extra].len() <= total {
self.queue[extra].push(VecDeque::new());
}
self.queue[extra][total].push_back(placement);
if self.extra > extra {
self.extra = extra;
}
}

fn dequeue(self: &mut PlacementQueue) -> Option<Placement> {
while self.extra < self.queue.len() {
let mut subqueue = &mut self.queue[self.extra];
while !subqueue.is_empty() {
let total = subqueue.len() - 1;
if let Some(placement) = subqueue[total].pop_front() {
return Some(placement);
}
subqueue.pop();
}
self.extra += 1;
}
return None
}
}

fn main() {
let stdin = std::io::stdin();
let all_words: Vec<String> =
stdin.lock().lines().map(|l| l.unwrap()).collect();
let words: Vec<&String> = {
let subwords: HashSet<&str> =
all_words.iter().flat_map(|word| {
(0..word.len() - 1).flat_map(move |i| {
(i + 1..word.len() - (i == 0) as usize).map(move |j| {
&word[i..j]
})
})
}).collect();
all_words.iter().filter(|word| !subwords.contains(&word[..])).collect()
};
let letters: Vec<Vec<(usize, usize)>> =
(0..128).map(|c| {
words.iter().enumerate().flat_map(|(w, word)| {
word.bytes().enumerate().filter(|&(_, c1)| c == c1).map(move |(i, _)| (w, i))
}).collect()
}).collect();

let mut used = vec![false; words.len()];
let mut remaining = words.len();
let mut grids: Vec<HashMap<Pos, u8>> = Vec::new();

while remaining != 0 {
let mut grid: HashMap<Pos, u8> = HashMap::new();
let mut queue = PlacementQueue::new();
for (w, word) in words.iter().enumerate() {
if used[w] {
continue;
}
queue.enqueue(0, word.len(), Placement {
pos: (0, 0),
dir: 0,
word: w as Word
});
}

while let Some(placement) = queue.dequeue() {
if used[placement.word as usize] {
continue;
}
let word = words[placement.word as usize];
if let None = fit(&grid, placement.pos, placement.dir, word) {
continue;
}
let (x, y) = placement.pos;
let (dx, dy) = DIRS[placement.dir as usize];
let new_letters: Vec<(usize, u8)> = word.bytes().enumerate().filter(|&(i, _)| {
!grid.contains_key(&(x + (i as Coord)*dx, y + (i as Coord)*dy))
}).collect();
for (i, c) in word.bytes().enumerate() {
grid.insert((x + (i as Coord)*dx, y + (i as Coord)*dy), c);
}
used[placement.word as usize] = true;
remaining -= 1;

for (i, c) in new_letters {
for &(w1, j) in &letters[c as usize] {
if used[w1] {
continue;
}
let word1 = words[w1];
for (d1, &(dx1, dy1)) in DIRS.iter().enumerate() {
let pos1 = (
x + (i as Coord)*dx - (j as Coord)*dx1,
y + (i as Coord) - (j as Coord)*dy1);
if let Some(extra1) = fit(&grid, pos1, d1 as Dir, word1) {
queue.enqueue(extra1, word1.len(), Placement {
pos: pos1,
dir: d1 as Dir,
word: w1 as Word
});
}
}
}
}
}
grids.push(grid);
}

let width = grids.iter().map(|grid| {
grid.iter().map(|(&(x, _), _)| x).max().unwrap() -
grid.iter().map(|(&(x, _), _)| x).min().unwrap() + 1
}).max().unwrap();
print!(
"{}",
grids.iter().flat_map(|grid| {
let x0 = grid.iter().map(|(&(x, _), _)| x).min().unwrap();
let y0 = grid.iter().map(|(&(_, y), _)| y).min().unwrap();
let y1 = grid.iter().map(|(&(_, y), _)| y).max().unwrap();
(y0..y1 + 1).flat_map(move |y| {
(x0..x0 + width).map(move |x| {
*grid.get(&(x, y)).unwrap_or(&('.' as u8)) as char
}).chain(once('\n').take(1))
})
}).collect::<String>()
);
}

• I haven't read the code yet, but do you do anything to encourage non-linear placements? I would have expected an algorithm like this to end up with a handful of crossing super-strings, but it looks like you're getting some pretty good space-filling.
– Dave
Commented Aug 14, 2016 at 8:24
• @Dave Nothing specific, it just works out that way. The super-strings never get so long that better non-linear placements can never be found, probably because there are so many more non-linear placements to choose from. Commented Aug 14, 2016 at 8:43
• starts with "congratulations", ends with "extraordinary"
– YOU
Commented Aug 14, 2016 at 8:52
• I didn´t catch that you can go diagonal too. thanks for the picture. I don´t know if I should wish for comments on the code blocks. :) Commented Aug 14, 2016 at 9:45

# C++, 27243 character grid (248x219, 50.2% filled)

(Posting this as a new answer because I'd like to keep the 1D bound I originally posted as a reference)

This blatantly rips off is heavily inspired by @AndersKaseorg's answer in its main structure, but has a couple of tweaks. First, I use my original program to merge strings until the best available overlap is just 3 characters. Then I use the method AndersKaseorg describes to progressively fill a 2D grid using these generated strings. The constraints are a little different too: it still tries to add the fewest characters each time, but ties are broken by favouring square grids first, then small grids, and finally by adding the longest word.

The behaviour it displays is to alternate between periods of filling in space and rapidly expanding the grid (unfortunately it ran out of words just after a rapid expansion stage, so there's a lot of blank space around the edges). I suspect with some tweaking of the cost function it could be made to get better than 50% space filling.

There are 2 executables here (to avoid the need to re-run the whole process when iteratively improving the algorithm). The output of one can be piped directly into the other:

#include <iostream>
#include <string>
#include <vector>
#include <algorithm>
#include <cstdlib>

std::size_t calcOverlap(const std::string &a, const std::string &b, std::size_t limit, std::size_t minimal) {
std::size_t la = a.size();
for(std::size_t p = std::min(std::min(la, b.size()), limit + 1); -- p > minimal; ) {
if(a.compare(la - p, p, b, 0, p) == 0) {
return p;
}
}
return 0;
}

bool isSameReversed(const std::string &a, const std::string &b) {
std::size_t l = a.size();
if(b.size() != l) {
return false;
}
for(std::size_t i = 0; i < l; ++ i) {
if(a[i] != b[l-i-1]) {
return false;
}
}
return true;
}

int main(int argc, const char *const *argv) {
// Usage: prog [<stop_threshold>]

std::size_t stopThreshold = 3;

if(argc >= 2) {
char *check;
long v = std::strtol(argv[1], &check, 10);
if(check == argv[1] || v < 0) {
std::cerr
<< "Invalid stop threshold. Should be an integer >= 0"
<< std::endl;
return 1;
}
stopThreshold = v;
}

std::vector<std::string> words;

// Load all words from input and their reverses (words can be backwards now)
while(true) {
std::string word;
std::getline(std::cin, word);
if(word.empty()) {
break;
}
words.push_back(word);
std::reverse(word.begin(), word.end());
words.push_back(std::move(word));
}

std::cerr
<< "Input word count: " << words.size() << std::endl;

// Remove all fully subsumed words

for(auto p = words.begin(); p != words.end(); ) {
bool subsumed = false;
for(auto i = words.begin(); i != words.end(); ++ i) {
if(i == p) {
continue;
}
if(i->find(*p) != std::string::npos) {
subsumed = true;
break;
}
}
if(subsumed) {
p = words.erase(p);
} else {
++ p;
}
}

std::cerr
<< "After subsuming checks: " << words.size()
<< std::endl;

// Sort words longest-to-shortest (not necessary but doesn't hurt. Makes finding maxlen a tiny bit easier)
std::sort(words.begin(), words.end(), [](const std::string &a, const std::string &b) {
return a.size() > b.size();
});

std::size_t maxlen = words.front().size();

// Repeatedly combine most-compatible words until we reach the threshold
std::size_t bestPossible = maxlen - 1;
while(words.size() > 2) {
auto bestA = words.begin();
auto bestB = -- words.end();
std::size_t bestOverlap = 0;
for(auto p = ++ words.begin(), e = words.end(); p != e; ++ p) {
if(p->size() - 1 <= bestOverlap) {
continue;
}
for(auto q = words.begin(); q != p; ++ q) {
std::size_t overlap = calcOverlap(*p, *q, bestPossible, bestOverlap);
if(overlap > bestOverlap && !isSameReversed(*p, *q)) {
bestA = p;
bestB = q;
bestOverlap = overlap;
}
overlap = calcOverlap(*q, *p, bestPossible, bestOverlap);
if(overlap > bestOverlap && !isSameReversed(*p, *q)) {
bestA = q;
bestB = p;
bestOverlap = overlap;
}
}
if(bestOverlap == bestPossible) {
break;
}
}
if(bestOverlap <= stopThreshold) {
break;
}
std::string newStr = std::move(*bestA);
newStr.append(*bestB, bestOverlap, std::string::npos);

if(bestA == -- words.end()) {
words.pop_back();
*bestB = std::move(words.back());
words.pop_back();
} else {
*bestB = std::move(words.back());
words.pop_back();
*bestA = std::move(words.back());
words.pop_back();
}

// Remove any words which are now in the result (forward or reverse)
// (would not be necessary if we didn't have the reversed forms too)
std::string newRev = newStr;
std::reverse(newRev.begin(), newRev.end());
for(auto p = words.begin(); p != words.end(); ) {
if(newStr.find(*p) != std::string::npos || newRev.find(*p) != std::string::npos) {
std::cerr << "Now subsumes: " << *p << std::endl;
p = words.erase(p);
} else {
++ p;
}
}

std::cerr
<< "Words remaining: " << (words.size() + 1)
<< " Latest combination: (" << bestOverlap << ") " << newStr
<< std::endl;

words.push_back(std::move(newStr));
words.push_back(std::move(newRev));
bestPossible = bestOverlap; // Merging existing words will never make longer merges possible
}

std::cerr
<< "After merging: " << words.size()
<< std::endl;

// Remove all fully subsumed words (i.e. reversed words)

for(auto p = words.begin(); p != words.end(); ) {
bool subsumed = false;
std::string rev = *p;
std::reverse(rev.begin(), rev.end());
for(auto i = words.begin(); i != words.end(); ++ i) {
if(i == p) {
continue;
}
if(i->find(*p) != std::string::npos || i->find(rev) != std::string::npos) {
subsumed = true;
break;
}
}
if(subsumed) {
p = words.erase(p);
} else {
++ p;
}
}

std::cerr
<< "After subsuming: " << words.size()
<< std::endl;

// Sort words longest-to-shortest for display
std::sort(words.begin(), words.end(), [](const std::string &a, const std::string &b) {
return a.size() > b.size();
});

std::size_t len = 0;
for(const auto &word : words) {
std::cout
<< word
<< std::endl;
len += word.size();
}
std::cerr
<< "Total size: " << len
<< std::endl;
return 0;
}

#include <iostream>
#include <string>
#include <vector>
#include <unordered_map>
#include <unordered_set>
#include <limits>

class vec2 {
public:
int x;
int y;

vec2(void) : x(0), y(0) {};
vec2(int x, int y) : x(x), y(y) {}

bool operator ==(const vec2 &b) const {
return x == b.x && y == b.y;
}

vec2 &operator +=(const vec2 &b) {
x += b.x;
y += b.y;
return *this;
}

vec2 &operator -=(const vec2 &b) {
x -= b.x;
y -= b.y;
return *this;
}

vec2 operator +(const vec2 b) const {
return vec2(x + b.x, y + b.y);
}

vec2 operator *(const int b) const {
return vec2(x * b, y * b);
}
};

class box2 {
public:
vec2 tl;
vec2 br;

box2(void) : tl(), br() {};
box2(vec2 a, vec2 b)
: tl(std::min(a.x, b.x), std::min(a.y, b.y))
, br(std::max(a.x, b.x) + 1, std::max(a.y, b.y) + 1)
{}

void grow(const box2 &b) {
if(b.tl.x < tl.x) {
tl.x = b.tl.x;
}
if(b.br.x > br.x) {
br.x = b.br.x;
}
if(b.tl.y < tl.y) {
tl.y = b.tl.y;
}
if(b.br.y > br.y) {
br.y = b.br.y;
}
}

bool intersects(const box2 &b) const {
return (
((tl.x >= b.br.x) != (br.x > b.tl.x)) &&
((tl.y >= b.br.y) != (br.y > b.tl.y))
);
}

box2 &operator +=(const vec2 b) {
tl += b;
br += b;
return *this;
}

int width(void) const {
return br.x - tl.x;
}

int height(void) const {
return br.y - tl.y;
}

int maxdim(void) const {
return std::max(width(), height());
}
};

template <> struct std::hash<vec2> {
std::size_t operator ()(const vec2 &o) const {
return std::hash<int>()(o.x) + std::hash<int>()(o.y) * 997;
}
};

template <class A,class B> struct std::hash<std::pair<A,B>> {
std::size_t operator ()(const std::pair<A,B> &o) const {
return std::hash<A>()(o.first) + std::hash<B>()(o.second) * 31;
}
};

class word_placement {
public:
vec2 start;
vec2 dir;
box2 bounds;
const std::string *word;

word_placement(vec2 start, vec2 dir, const std::string *word)
: start(start)
, dir(dir)
, bounds(start, start + dir * (word->size() - 1))
, word(word)
{}

word_placement(vec2 start, const word_placement &copy)
: start(copy.start + start)
, dir(copy.dir)
, bounds(copy.bounds)
, word(copy.word)
{
bounds += start;
}

word_placement(const word_placement &copy)
: start(copy.start)
, dir(copy.dir)
, bounds(copy.bounds)
, word(copy.word)
{}
};

public:
std::unordered_set<word_placement*> placements;
std::unordered_set<std::pair<char,word_placement*>> relativePlacements;
};

class grid {
public:
std::vector<std::string> wordCache; // Just a block of memory for our pointers to reference
std::unordered_map<vec2,char> state;
std::unordered_set<word_placement*> placements;
std::unordered_map<char,std::unordered_set<word_placement*>> relativeWordPlacements;
box2 bound;

grid(const std::vector<std::string> &words) {
wordCache = words;
std::vector<vec2> directions;
directions.emplace_back(+1,  0);
directions.emplace_back(+1, +1);
directions.emplace_back( 0, +1);
directions.emplace_back(-1, +1);
directions.emplace_back(-1,  0);
directions.emplace_back(-1, -1);
directions.emplace_back( 0, -1);
directions.emplace_back(+1, -1);

wordPlacements.reserve(wordCache.size());
placements.reserve(wordCache.size());
relativeWordPlacements.reserve(64);

std::size_t total = 0;
for(const std::string &word : wordCache) {
p.placements.reserve(8);
auto &rp = p.relativePlacements;
std::size_t l = word.size();
rp.reserve(l * directions.size());
for(int i = 0; i < l; ++ i) {
for(const vec2 &d : directions) {
word_placement *rwp = new word_placement(d * -i, d, &word);
rp.emplace(word[i], rwp);
relativeWordPlacements[word[i]].insert(rwp);
}
}
total += l;
}
state.reserve(total);
}

const std::string *find_word(const std::string &word) const {
for(const std::string &w : wordCache) {
if(w == word) {
return &w;
}
}
throw std::string("Failed to find word in cache");
}

void remove_word(const std::string *word) {
placements.erase(p);
delete p;
}
relativeWordPlacements[p.first].erase(p.second);
delete p.second;
}
wordPlacements.erase(word);
}

void remove_placement(word_placement *placement) {
wordPlacements[placement->word].placements.erase(placement);
placements.erase(placement);
delete placement;
}

bool check_placement(const word_placement &placement) const {
vec2 p = placement.start;
for(const char c : *placement.word) {
auto i = state.find(p);
if(i != state.end() && i->second != c) {
return false;
}
p += placement.dir;
}
return true;
}

int check_new(const word_placement &placement) const {
int n = 0;
vec2 p = placement.start;
for(const char c : *placement.word) {
n += !state.count(p);
p += placement.dir;
}
return n;
}

void check_placements(const box2 &b) {
for(auto i = placements.begin(); i != placements.end(); ) {
if(!b.intersects((*i)->bounds) || check_placement(**i)) {
++ i;
} else {
i = placements.erase(i);
}
}
}

void add_placement(const vec2 p, const word_placement &relative) {
word_placement check(p, relative);
if(check_placement(check)) {
word_placement *wp = new word_placement(check);
placements.insert(wp);
wordPlacements[relative.word].placements.insert(wp);
}
}

void place(word_placement placement) {
remove_word(placement.word);
int overlap = 0;
for(const char c : *placement.word) {
char &g = state[placement.start];
if(g == '\0') {
g = c;
for(const word_placement *rp : relativeWordPlacements[c]) {
}
} else if(g != c) {
throw std::string("New word changes an existing character!");
} else {
++ overlap;
}
placement.start += placement.dir;
}
bound.grow(placement.bounds);
check_placements(placement.bounds);

std::cerr
<< draw('.', "\n")
<< "Added " << *placement.word << " (overlap: " << overlap << ")"
<< ", Grid: " << bound.width() << "x" << bound.height() << " of " << state.size() << " chars"
<< ", Words remaining: " << wordPlacements.size()
<< std::endl;
}

int check_cost(box2 b) const {
b.grow(bound);
return (
((b.maxdim() - bound.maxdim()) << 16) |
(b.width() + b.height() - bound.width() - bound.height())
);
}

int bestNew = std::numeric_limits<int>::max();
int bestCost = std::numeric_limits<int>::max();
int bestLen = 0;
word_placement *best = nullptr;
for(word_placement *p : placements) {
int n = check_new(*p);
if(n <= bestNew) {
int l = p->word->size();
int cost = check_cost(box2(p->start, p->start + p->dir * l));
if(n < bestNew || cost < bestCost || (cost == bestCost && l < bestLen)) {
bestNew = n;
bestCost = cost;
bestLen = l;
best = p;
}
}
}
if(best == nullptr) {
throw std::string("Failed to find join to existing blob");
}
place(*best);
}

void fill(void) {
while(!placements.empty()) {
}
}

std::string draw(char blank, const std::string &linesep) const {
std::string result;
result.reserve((bound.width() + linesep.size()) * bound.height());
for(int y = bound.tl.y; y < bound.br.y; ++ y) {
for(int x = bound.tl.x; x < bound.br.x; ++ x) {
auto c = state.find(vec2(x, y));
result.push_back((c == state.end()) ? blank : c->second);
}
result.append(linesep);
}
return result;
}

box2 bounds(void) const {
return bound;
}

int chars(void) const {
return state.size();
}
};

int main(int argc, const char *const *argv) {
std::vector<std::string> words;

// Load all words from input
while(true) {
std::string word;
std::getline(std::cin, word);
if(word.empty()) {
break;
}
words.push_back(std::move(word));
}

std::cerr
<< "Input word count: " << words.size() << std::endl;

// initialise grid
grid g(words);

// add first word (order of input file means this is longest word)
g.place(word_placement(vec2(0, 0), vec2(1, 0), g.find_word(words.front())));

g.fill();

std::cout << g.draw('.', "\n");

int w = g.bounds().width();
int h = g.bounds().height();
int n = g.chars();
std::cerr
<< "Final grid: " << w << "x" << h
<< " with " << n << " characters"
<< " (" << (n * 100.0 / (w * h)) << "% filled)"
<< std::endl;
return 0;
}


And finally, the result:

Alternative result (after fixing a couple of bugs in the program which were biasing certain directions and tweaking the cost function, I got a more compact but less optimal solution): 29275 chars, 198x195 (75.8% filled):

Again I haven't done much to optimise these programs, so it takes a while. But you can watch as it fills in the grid, which is quite hypnotic.

# C++, 34191 character "grid" (with minimal human intervention, 6 or 7 can easily be saved)

This should be taken more as a bound for the 2D case, because the answer is still a 1D string. It's just my code from the previous challenge, but with the new ability to reverse any string. This gives us a lot more scope for combining words (particularly because it caps the worst case of non-overlapping superstrings to 26; one for each letter of the alphabet).

For some slight 2D visual appeal, it puts linebreaks in the result if it can do so for free (i.e. between 0-overlap words).

Pretty slow (still no caching). Here's the code:

#include <iostream>
#include <string>
#include <vector>
#include <algorithm>

std::size_t calcOverlap(const std::string &a, const std::string &b, std::size_t limit, std::size_t minimal) {
std::size_t la = a.size();
for(std::size_t p = std::min(std::min(la, b.size()), limit + 1); -- p > minimal; ) {
if(a.compare(la - p, p, b, 0, p) == 0) {
return p;
}
}
return 0;
}

bool isSameReversed(const std::string &a, const std::string &b) {
std::size_t l = a.size();
if(b.size() != l) {
return false;
}
for(std::size_t i = 0; i < l; ++ i) {
if(a[i] != b[l-i-1]) {
return false;
}
}
return true;
}

int main() {
std::vector<std::string> words;

// Load all words from input and their reverses (words can be backwards now)
while(true) {
std::string word;
std::getline(std::cin, word);
if(word.empty()) {
break;
}
words.push_back(word);
std::reverse(word.begin(), word.end());
words.push_back(std::move(word));
}

std::cerr
<< "Input word count: " << words.size() << std::endl;

// Remove all fully subsumed words

for(auto p = words.begin(); p != words.end(); ) {
bool subsumed = false;
for(auto i = words.begin(); i != words.end(); ++ i) {
if(i == p) {
continue;
}
if(i->find(*p) != std::string::npos) {
subsumed = true;
break;
}
}
if(subsumed) {
p = words.erase(p);
} else {
++ p;
}
}

std::cerr
<< "After subsuming checks: " << words.size()
<< std::endl;

// Sort words longest-to-shortest (not necessary but doesn't hurt. Makes finding maxlen a tiny bit easier)
std::sort(words.begin(), words.end(), [](const std::string &a, const std::string &b) {
return a.size() > b.size();
});

std::size_t maxlen = words.front().size();

// Repeatedly combine most-compatible words until we have only 1 word left (+ its reverse)
std::size_t bestPossible = maxlen - 1;
while(words.size() > 2) {
auto bestA = words.begin();
auto bestB = -- words.end();
std::size_t bestOverlap = 0;
for(auto p = ++ words.begin(), e = words.end(); p != e; ++ p) {
if(p->size() - 1 <= bestOverlap) {
continue;
}
for(auto q = words.begin(); q != p; ++ q) {
std::size_t overlap = calcOverlap(*p, *q, bestPossible, bestOverlap);
if(overlap > bestOverlap && !isSameReversed(*p, *q)) {
bestA = p;
bestB = q;
bestOverlap = overlap;
}
overlap = calcOverlap(*q, *p, bestPossible, bestOverlap);
if(overlap > bestOverlap && !isSameReversed(*p, *q)) {
bestA = q;
bestB = p;
bestOverlap = overlap;
}
}
if(bestOverlap == bestPossible) {
break;
}
}
std::string newStr = std::move(*bestA);
if(bestOverlap == 0) {
newStr.push_back('\n');
}
newStr.append(*bestB, bestOverlap, std::string::npos);

if(bestA == -- words.end()) {
words.pop_back();
*bestB = std::move(words.back());
words.pop_back();
} else {
*bestB = std::move(words.back());
words.pop_back();
*bestA = std::move(words.back());
words.pop_back();
}

// Remove any words which are now in the result (forward or reverse)
// (would not be necessary if we didn't have the reversed forms too)
std::string newRev = newStr;
std::reverse(newRev.begin(), newRev.end());
for(auto p = words.begin(); p != words.end(); ) {
if(newStr.find(*p) != std::string::npos || newRev.find(*p) != std::string::npos) {
std::cerr << "Now subsumes: " << *p << std::endl;
p = words.erase(p);
} else {
++ p;
}
}

std::cerr
<< "Words remaining: " << (words.size() + 1)
<< " Latest combination: (" << bestOverlap << ") " << newStr
<< std::endl;

words.push_back(std::move(newStr));
words.push_back(std::move(newRev));
bestPossible = bestOverlap; // Merging existing words will never make longer merges possible
}

std::cerr
<< "After non-trivial merging: " << words.size()
<< std::endl;

if(words.size() == 2 && !isSameReversed(words.front(), words.back())) {
// must be 2 palindromes, so just join them
words.front().append(words.back());
}

std::string result = words.front();

std::cout
<< result
<< std::endl;
std::cerr
<< "Word size: " << result.size() // Note this number includes newlines, so to get the grid size according to the rules, subtract newlines manually
<< std::endl;
return 0;
}


Result: http://pastebin.com/UTe2WMcz (4081 characters less than the previous challenge)

It's pretty clear that some trivial savings can be made by putting the xd and wv lines vertical, intersecting the monster line. Then hhidetautisbneudui can intersect with the d, and lxwwwowaxocnnaesdda with w. This saves 4 characters. nbcllilhn can be substituted in to an existing s overlap (if one can be found) to save another 2 (or just 1 if no such overlap exists and it must be added vertically instead). Finally mjjrajaytq can be added vertically somewhere to save 1. This means with minimal human intervention, 6–7 characters can be saved from the result.

I would like to get this into 2D with the following method, but I'm struggling to find a way to implement it without making the algorithm O(n^4), which is quite impractical to compute!

1. Run the algorithm as above, but stop short when the overlaps reach 1 character
2. Repeatedly:
1. Find a group of 4 words which can be arranged into a rectangle
2. Add as many words as possible on top of this rectangle where each word overlaps at least 2 characters of the current shape (check all 8 directions) — this is the only stage where we can actually get an advantage over the current code
3. Combine the resulting grids and lone-words looking for single-letter overlaps each time

# PHP

this one does the job theroratically; but 10000 are probably too many words for recursion. The script is running now. (still ran 24 hours later)
works fine on small directories, but I may make an iterative version next week.

$f=array("pen","op","po","ne","pro","aaa","abcd","dcba"); will output abcd apen arop ao.. although this is not an optimal result (scoring was changed ... I´m working on a generator). One optimal result is this: open .r.a .o.a dcba It is also not very fast; only removes substrings and sorts the remains by length, the rest is brute force: try to fit the words into a rectangle, try on a larger rectangle if it fails. btw: The substring part needs 4.5 minutes on my machine for the large directory and cuts it down to 6,190 words; the sort on them takes 11 seconds. $f=file('https://raw.githubusercontent.com/first20hours/google-10000-english/master/google-10000-english.txt');
// A: remove substrings - forward or reversed
$s=join(' ',$f);
$haystack="$s ".strrev($s); foreach($f as$w) {$r=strrev($w=trim($w)); // remove trailing line break and create reverse word
if(!preg_match("%$w\w|\w$w%",$haystack) // no substr match ... now: is the reverse word in the list? // if so, keep only the lower one (ascii values) &!($w>$r&&strstr($s,$r)) // strstr does NOT render the reverse substr regex obsolete: // this is only executed for$w=abc, not for $w=bca! )$g[]=$w ; } // B: sort the words by length usort($g,function($a,$b){return strlen($a)-strlen($b);});

// C1: function to fit $words into$map
function gomap($words,$map)
{
$h=count($map);$w=strlen($map[0]);
$len=strlen($word=array_pop($words)); //$x,$y=position;$d=0:horizontal, $d=1:vertical;$r=0: word, $r=1: reverse word for($x=$w-$len;$x>=0;$x--)for($y=$h-$len;$y>=0;$y--)for($d=0;$d<2;$d++)for($r=0;$r<2;$r++) { // does the word fit there?$drow=$r?strrev($word):$word; for($ok=1,$i=0;$ok&$i<$len;$i++)$ok=in_array($map[$y+$d*$i][$x+$i-$d*$i], [' ',$drow[$i]])
;
// it does, paint it
if($ok) { for($i=0;$i<$len;$i++)$map[$y+$d*$i][$x+$i-$d*$i]=$drow[$i]; if(!count($words))      // this was the last word: return map
return $map; else // there are more words: recurse if ($ok=gomap($words,$map))
return $ok; // no fit, try next position } } return 0; } // C2: rectangle loop for($h=0;++$h;)for($w=0;$w++<$h;)   // define a rectangle
{
// and try to fit the words in there
if($map=gomap($g,
array_fill(0,$h,str_repeat(' ',$w))
))
{
// words fit; output and break loops
echo '<pre>',implode("\n",\$map),'</pre>';
break 2;
}
}
`
• Could you include an example when the program is run on a smaller dictionary? Commented Aug 13, 2016 at 16:53
• I've actually changed the scoring (sorry!). The number of unused characters is not included in your score. Commented Aug 13, 2016 at 18:13
• The looping here means this is ~O((w*h)^n). We know the solution will have something like 35k letters (from the last challenge), so it will end up calling gomap about 35000^6000 times. My calculator tells me that's "infinity". A better calculator tells me the actual number (wolframalpha.com/input/?i=35000%5E6000). Now, if we assume every atom in the universe is a 3 terrahertz processor dedicated to running this program, the universe will need to exist for 10^27154 times longer than it has so far before it completes. What I'm saying is: don't wait for it to finish!
– Dave
Commented Aug 13, 2016 at 23:05