Nibbles, 161 words, 400 bytes.
.=\`\<159\`D 6+~"o y"`D 21*"sh"`%!|;@`#$4+~:`/@5`<+$~;@=" "$$45f31d2e733373ab092b79e7b544370bc7a0e362d8beeb825fe11e496df679b92e76991f03d6bc7f4389125c5b3648e326c221e31340945b6a79d28e6cbafaf177dccffd45c3e0e544365a8c973aed312eb485f57c8d861d18c62044842f41b7db23a106ede98d24c314c9d979ba388abeed82b04e848d8a9b6e44aafbb4bc0a7eae60687b13ecba3c0c240b05f7a0b2910a1843ca0af8f46849d9a565d695255ae19b2c954869ba9662723e903f422af11ef182dc00343885921657571bd40757f46ba10eda6a60e78c37ba913f26c93db1c2e1b9714d6c2601f504eba9cb0fff65d060cf08156986b079ae1046cd3dc1415a1daad9d57a79d13a271c9bc788631440d1ee118a40099517ce55c1389854cf0afa17ad73765d017de916abb93abfa49ff8d2d77382ab8f89db95a4ff8ce929bccd1435a1c98f9af7e67fe1847e6cdb348fad50d8c2c7ed5efd26bf7b838a985bc0bffe0a5168fcad28060f974b40c7dcf00b562ed1cbc3937ce387e71f
That's 64 nibbles of code and 736 hex digits of data at half a byte for each.
Nibbles isn't on TIO yet, but run it like this:
nibbles words.nbl < /dev/null
Ungolfed version:
.=\
`\ <159 \`D 6 +
~"o y" `D 21 \c unused dat
*"sh"`%!
|fstLine `# $ 4 #" abdfghiklmnoprstuvyz"
+~:`/c 5 sets m `<+m~ dat sets w
=
" "
w
$
(hex data is the same)
The method is essentially to encode each letter in base 21 data, but with 1 key trick.
Notice these two inefficiencies with that trivial encoding.
- If the list is sorted then the next first letter is the same or 1 more.
- Spaces are by far the most common
With the trick, a space and the next first letter combined can be encoded with a single base 6 digit! That's a savings of 125 bytes of data.
First sort the words alphabetically by first character and length as the tie breaker.
Now compute difference in length of words, with 6 added if they differ in starting letter.
These numbers now encode what the next words length and starting letters are.
So if the last word was "ats" and the digit is 2, it means the next word also starts with 'a' but has length 5. Now if the next digit is 2, it means the next word starts with 'b' and is length 3. That is because there are no 7+ letter words and no 1 letter words encoded (o and y are handled specially for this reason).
One problem I ran into is there are only 160 words < 7 in length. But if we handle even one 7 letter word the above method would need base 7 digits which would add about 5 bytes of data. To get around this I encode all "sh" as spaces, which effectively makes words with sh in them shorter. That's the most common letter combo (used 23 times in the words used), but this trick doesn't really save much because it increases the base that the letters are encoded as and requires code to convert back. Each use saves one base 21 digit (-12.5 bytes), but requires base 21 instead of 20 (+5 bytes), and the decoding code is *"sh"`% WORD_GOES_HERE " " which is 5 bytes. The next most common combo (15ish occurrences) isn't common enough to save anything unless the decoding process becomes more efficient.
The way the base 21 list and base 6 list are encoded is using nibbles data feature which lets you encode a single number after your program with no terminators possible (normal numbers would use 1 bit of every 4 to terminate or not). This is a pure number not a list, but can be extracted to a desired base using `D <base>. But since the data needs to be used for both lists, we reverse it for one of them and truncate it. The non truncated list will have garbage data at the end, but this will never be consumed the way the program is written.
There's another useful trick that saves about 8 bytes which is how the alphabet is encoded which is: " abdfghiklmnoprstuvyz"
Rather than encode this directly, we use luck. The Nibbles hash function `# also uses data as salt, so if we have enough degrees of freedom in our encoding then we can create a hash that will deterministically map:
" abcdefghijklmnopqrstuvwxyz" to
"111010111101111110111110011" so that after filtering it, it becomes
" ab d fghi klmnop rstuv yz" (without the spaces)
The probability that this mapping occurs is (1/4)**7*(3/4)**20 if we take the hash value mod 4 and use 0 as false and 1 2 3 as true. This is about 1 in 5 million. Notice that we can permute sets of words that have the same starting letter and length. This will change the encoded data, but will still give a correct output. There are also multiple 6 letter words that aren't used (now that sh is 1 letter), so that is another degree of freedom.
Luckily this only takes about 1 hour to find with brute force search on random permutations of the starting words.
Here's the code to find it:
# removed o y
let uwords |.%"abail abikis ak akniton alfabit atirsin atom ats aypal baka baklait banl bgil bianto bit bo boraky borsh bot br brbrno byt daosik diga digiz dogo dolmuk dongso douv drarkak fa fafasok fasmao fidr floria fridka friv fully fylo fyrus gilmaf gogu gogzo gongul gorlb gosio gratbt gutka hisi horrib hurigo inglish iomshuksor isa ispikar isplo ita iushb izlash kaikina kat katisas katlani kfaso khalling kisongso kivsk klingt kod koik kolbat komy konma korova koss kotsh kris kromir ksik ksuz ktaf ktomish ktomizo la lan lgra loshad lyly mak makanto makav makina marinolus marta matiks mim mimhis mingso mitsh molo mosdob motif motkop movogal muratsh mutka naita ngoirfo nidriok nilo nin nini ninibr nirus noguna nolif nors ogig omkop onion opru orilha orni ozon papin paslo pasrus pirishy pluban polr posodo pouk produty rak raznitsa rigiks rilifior riomr rno ronta rus ryby sabbia sai sakar saman sandvitsh sasa sfiny shatris shidoso shilan shimshoom shor shotiya shuan shubhu shuksor shuriok shutis shutrak shyba sibas sibin simriomr sod sokit spamra spamt srirad sutka taika tipstor tizingso tksu tobl tokvod tomp tonfa toto totobr tshans tshimiso tshomp tura tuvung uboiak umub vaf vintz voda vodl vontark yntsh zimdo zivod zombit zorliplak"~
\w
?==w "yntsh" w *" "`%w"sh" # don't convert this word because the jump to zimdo would be 6 then
- 7 ,$
hex / |.,~ \i
let words +=~ uwords \w `# :w i 4273 # this essentially permutes the words randomly
# permuting randomly is more efficient than searching through permutations in order as small changes will actually likely map to the same encoding (i.e. if you swap words of different lengths that will get undone during the encode process).
let takewords 159
let bestchars -" abdfghiklmnoprstuvyz" ""
let selectedwords | | words ~ \w | w ~ \l ? bestchars l - 7,$
let nc ,bestchars
let bestwords < takewords +=~ selectedwords ,$
let sortedwords +=~ +=~ bestwords ,$ =1$
let tailed .sortedwords>>$
let lens \:1 ! sortedwords >>sortedwords ~ \w1 w2
let lengthdiff -,w2,w1
let chardiff - ?bestchars =1w2 ?bestchars=1w1
+lengthdiff *5 chardiff
let data . sortedwords \word
let wd ! bestchars word ?
>>. wd
\c1 -c1~
let d1 `@ nc +data
let d2 `@ 6 lens
let d1p *d1 ^nc ,`@ nc d2
+d1p % -d2 `@ 6 \<,lens \`@ 6 d1p ^6 takewords
\fd
* ==| " abcdefghijklmnopqrstuvwxyz" \c `#~ :o c `@ 256 fd 4
" abdfghiklmnoprstuvyz"
- 737 ,hex fd # max size 736 (sometimes it is 737 for unknown reasons)
$
Other ideas:
Remove z, this was a complete wash because z is in some short words.
I considered using something like huffman encoding since some letters are common and others rare. It would save about 30 bytes of data, but now you have to encode at least the order of commonness of the alphabet (and then assume some function of probability on it, it is actually pretty linear though). That would take at least 10 bytes. I don't think that leaves enough to decode and save anything.
I'm kind of surprised this method did so well without really finding any other patterns in the data. But it doesn't seem like there are any other obvious patterns that are so strong to be worth the extraction code.
zlibone. And anything more than 400 bytes wouldn't make inbuilt compression weaker, it would just make it so that more answers could fit the whole dictionary. \$\endgroup\$