The number of distinct words in a sentence

Challenge: Find the number of distinct words in a sentence

Your task is to write a program that takes a sentence as input and returns the number of distinct words in it. For this challenge, words are defined as consecutive sequences of letters, digits, and underscores ('_').

Input:

• A string containing a sentence. The length of the sentence will not exceed 1000 characters.

Output:

• An integer indicating the number of distinct words in the sentence.

Examples:

Input: "The quick brown fox jumps over the lazy dog"
Output: 8


Explanation: The sentence contains 9 words, 8 of which are distinct: "The", "quick", "brown", "fox", "jumps", "over", "lazy", and "dog"

Input: "To be, or not to be, that is the question"
Output: 8


Explanation: The sentence contains 8 distinct words: "To", "be", "or", "not", "that", "is", "the", and "question".

Input: "Hello, World!"
Output: 2


Explanation: The sentence contains 2 distinct words: "Hello" and "World".

Input: "hello-world2"
Output: 2


Explanation: Two distinct words, "hello" and "world2" separated by hyphen.

Input: "Hello, World! Hello!"
Output: 2


Explanation: Only "Hello" and "World" are two distinct words here.

Scoring: This is a code golf challenge, so the goal is to minimize the size of your code while still producing correct output for all test cases. In case of a tie, the earliest submission wins. Good luck!

• Suggested test cases:  (0 words), 1 (1 word), hello-world2 (2 words), code_._golf. (2 words) -- assuming I'm interpreting the question correctly. Feb 28 at 17:38
• Suggested test case: Hello, World! Hello!, to catch solutions that try to split on spaces. Feb 28 at 17:51
• Can we assume the input will only contain printable ASCII (' '-'~' : codepoints 32-126)? Mar 1 at 8:39
• Is it guaranteed that the letters will only be from the English alphabet? Mar 1 at 9:52
• How about diacritics? This is in line with comments above from @KevinCruijssen and EzioMercer
– JvdV
Mar 1 at 12:16

Bash + GNU utils, 28

grep -Eo \\w+|sort -fu|wc -l


Try it online!

• this solution is swear-word adjacent! bad naughty coder. sort -fu|wc -l Mar 4 at 5:52

Python, 54 bytes

lambda s:len({*re.findall('\w+',s.lower())})
import re


Attempt This Online!

jq, 53 52 50 bytes

-1 byte by Neil for recognizing splitting on just \W works since we handle empty strings anyway. -2 bytes by me for realizing I could ascii_downcase first, saving a . and a |

ascii_downcase+":"|[splits("\\W")]|unique|length-1


Equivalent 50 byte answer:

ascii_downcase+":"|split("\\W";"")|unique|length-1


Thanks to chune's answer for inspiring me to try golfing splits("\\W+") instead of match("\\W+";"g"). Turns out, despite having to work around the empty string being matched in some cases, it is two bytes shorter!

For those curious, here's the match method:

[match("\\w+";"g").string|ascii_downcase]|unique|length

• You don't need the + in the first version.
– Neil
Mar 1 at 9:46

Perl-p, 24 bytes

$_=grep!$s{+lc}++,/\w+/g


Attempt This Online!

-1 byte thanks to Kjetil S

• You could replace lc$_ with +lc to shave a byte. Mar 1 at 11:20 JavaScript, 46 bytes s=>new Set(s.toLowerCase().match(/\w+/g)).size  Try it online! • you can use .split(" ") instead to save 3 characters: Try it online! Mar 1 at 18:23 • No, @foerno, that would fail on punctuation - e.g., the last test case. Mar 1 at 21:07 Zsh, 36 bytes <<<${#${(u)=${1:l}//[^[:IDENT:]]/ }}


Try it online!

Fortunately, the [:IDENT:] character class is exactly the words we should keep.

<<<${#${(u)=${1:l}//[^[:IDENT:]]/ }}${1:l}                    # lowercase string
${ //[^[:IDENT:]]/ } # // replace non-[:IDENT:] with spaces${   =                      }   # = split on $IFS (space/tab/newline${(u)                       }   # keep first occurance of each word
|%{$_|% *l*r} # Calls ToLower() on each of the words |sort|gu # Get unique words, Get-Unique needs the list to be sorted to remove all duplicates ( ).Count # Return the count  Thunno, $$\ 11 \log_{256}(96) \approx \$$ 9.05 bytes u"\w+"AfZUL  Attempt This Online! Same approach as basically every other answer. u"\w+"AfZUL # Implicit input u # Uppercase "\w+"Af # Regex findall "\w+" ZU # Uniquify L # Length  Excel, 120 bytes =LET( x,MID(UPPER(A1),ROW(A:A),1), ROWS( UNIQUE( TEXTSPLIT(A1,, IF((ABS(77.5-CODE(x&"Z"))<13)+1-ISERR(0+x)+(x="_")=0,x),1 ) ) ) )  • Great to see you around this place! For what it's worth, =ROWS(UNIQUE(LOWER(TEXTSPLIT(A1,,TEXTSPLIT(A1&" ",CHAR(VSTACK(ROW(48:57),ROW(65:90),ROW(97:122),95)),,1),1)))) would be 110 bytes if one would assume only characters in the \w character class as per basically all other answers. If only the case-insensitive split would not be bugged... – JvdV Mar 1 at 12:40 • @JvdV Thanks, likewise! So glad I stumbled upon this site - love the challenges and so useful for keeping oneself formula-sharp and for really getting stuck into the newer Excel functions. As you'll have noticed, here I borrowed one of the techniques you pioneered with TEXTSPLIT + an array of parameters for parsing a string. Will study your posted formula now. Cheers. Mar 1 at 12:41 • @JvdV Ok, but for Hello, World! Hello!, for example, you've got an issue with your extra space, I think. You'll return 3, since the second TEXTSPLIT returns {"Hello";"World";"Hello!"}, instead of {"Hello";"World";"Hello"}. Mar 1 at 12:52 • Right, that means another 4 bytes unfortunately. Both 1st parameters of TEXTSPLIT() should then have a trailing space. Getting closer to 120 bytes =). This is what you meant with keeping a formula-sharp mind! – JvdV Mar 1 at 12:54 • Down to 106 again using LET() and a single variable. =LET(x,UPPER(A1)&" ",ROWS(UNIQUE(TEXTSPLIT(x,,TEXTSPLIT(x,CHAR(VSTACK(ROW(48:57),ROW(65:90),95)),,1),1)))). It's merely another option in the same application. – JvdV Mar 1 at 13:27 Japt, 14 bytes Yet another one of those occasions I regret suggesting the removal of _ from the \w RegEx class in Japt! f"[%w_]+" üv l  Try it f"[%w_]+" üv l :Implicit input of string f :Match "[%w_]+" : RegEx /[a-z0-9_]/gi ü :Group & sort by v : Lowercase l :Length  • Would it be possible to revert this change and create another alias for [0-9a-zA-Z]? (Which, I think, is the POSIX character class [:alnum:].) Mar 1 at 17:54 • @Arnauld, I think we did have plans to add a new \u class to match [0-9a-zA-Z_] before ETH disappeared. Mar 1 at 21:09 Vyxal, 7 bytes ɽøWǍUL‹  Try it Online! Explanation: ɽ - Lowercases Input øW - Groups string by words into a list Ǎ - Removes all non-alphabetical items, leaving empty list spaces U - Removes all non-unique list items L - Gets length of list ‹ - Decrements by 1, to account for extra list item for first symbol  • Welcome to Code Golf! Mar 3 at 19:04 • I was wondering how a competitive answer with øW would look - I think my best answer was 8 bytes. Great first answer! And welcome to Vyxal too! Mar 3 at 22:02 Factor, 47 46 bytes [ >lower R/ \W/ re-split harvest cardinality ]  Try it online! -1 byte thangs to GammaFunction • >lower convert input to lowercase • R/ \W/ re-split split on non-word characters • harvest remove empty strings • cardinality length without duplicates Splitting and harvesting is shorter than simply getting a list of matches because the word for that (all-matching-slices) is super long. Not sure if there is a way to prevent the empty strings in pure regex, might be shorter. • You can use / \W/ instead. (Thanks @Neil for the comment about the same thing on my answer) Mar 3 at 19:41 • @GammaFunction Thanks! Mar 3 at 20:20 Lua, 103 bytes function x(i)r=0m={}for c in i:lower():gmatch("%w+")do if not m[c]then m[c]=1r=r+1 end end return r end  Try it online! Arturo, 35 bytes $=>[match lower&{/\w+}|unique|size]


Try it

J, 23 bytes

'\w+'#@~.@rxall tolower


Uniquify and count word matches in lowercase input.

Attempt This Online!

Vyxal, 7 bytes

†ẎɽUL


Try it Online!

†Ẏ    # find all matches of \w+
ɽ   # to lowercase
U  # uniquify
L # length


Vyxall, 6 bytes

†ẎɽU


Try it Online!

APL (Dyalog Extended), 27 bytes

f←{⍴∪(⎕A,⎕D,'_')(∊⍨⊆⊢)1⎕C⍵}


Try it online!

f←{⍴∪(⎕A,⎕D,'_')(∊⍨⊆⊢)1⎕C⍵}
1⎕C      to uppercase
(⎕A,⎕D,'_')                 [A-Z0-9_]
(⎕A,⎕D,'_')(∊⍨⊆⊢)1⎕C       Partition (⊆) using Membership (∊)
∪                             remove duplicates
⍴                              count words

• Welcome to Code Golf, and nice first answer! Mar 1 at 7:17

05AB1E, 12 bytes

žQžjмS¡õKlÙg


Assumes the input will only contain printable ASCII characters.

Explanation:

žQ            # Push the constant string with all printable ASCII characters
žj          # Push the constant string with "a-zA-Z0-9_"
м         # Remove all those characters
S        # Convert the string to a list of characters
¡       # Split the (implicit) input-string by each of those characters
õK     # Remove any empty strings from the list
l    # Convert each word to lowercase
Ù   # Uniquify the list of lowercase words
g  # Pop and push the length
# (which is output implicitly as result)


Retina 0.8.2, 15 bytes

TLl
D\w+
\w+


Try it online! Link includes test cases. Explanation:

TLl


Convert to lower case.

D\w+


Deduplicate words.

\w+


Count the number of remaining words.

13 bytes in Retina 1:

D$\w+$l
\w+


Try it online! Link includes test cases. Explanation:

D$\w+$l


Deduplicate words by lowercased value.

\w+


Count the number of remaining words.

Ruby-n, 32 bytes

p$_.upcase.scan(/\w+/).uniq.size  Attempt This Online! Java (JDK), 83 bytes s->java.util.Arrays.stream((" "+s).toLowerCase().split("\\W")).distinct().count()-1  Try it online! Saved 17 bytes thanks to Neil. • +1 from me. Funny how a non-stream approach is exactly 100 bytes as well: try it online. :) Mar 1 at 8:51 • You don't need the +, but I think you can copy @GammaFunction's approach to save 17 bytes: s->java.util.Arrays.stream((" "+s).toLowerCase().split("\\W")).distinct().count()-1 – Neil Mar 1 at 9:45 PHP 8.x, 87 bytes This is a quite long piece of code... fn($z)=>count(array_flip(array_filter(array_map('strtolower',preg_split('/\W+/',$z)))))  It's almost self-explanatory. It's so void of PHP-only tricks that it is so trivial to re-implement it into JavaScript! // PHP-like code let fn = ($z)=>count(array_flip(array_filter(array_map('strtolower',preg_split('\\W+',$z))))); // Test boilerplate that has enough functionality function count(value) { return value.length; } function strtolower(str) { return str.toLowerCase(); } function preg_split(regex, subject) { return subject.split(new RegExp(regex)); } function array_filter(array) { return array.filter(function(value){ return value !== ''; }); } function array_flip(array) { return Array.from(new Set(array)); } function array_map(fn, array) { return array.map(typeof fn === 'string' ? window[fn] : fn); } // Event handler - does the basic output thing text.oninput = function(){ output.innerText = fn(this.value); }; <input type="text" id="text"/> <p>Output: <span id="output">--</span></p> Differences There aren't a lot of differences between the PHP and JavaScript versions: • The fn($z)=>[...] has to be written without the fn bit.
• The regular expression has to be escaped and can't have the slashes.
This means it changes from /\W+/ to \\W+.
• The function array_flip only returns a set of all unique values.
The PHP function returns the array with the keys set from the values.
That is, an array like ['a', 'fox', 'a', 'car'] will be returned as ['a' => 2, 'fox' => 1, 'car' => 3] while JavaScript returns ['a', 'fox', 'car'].
The end result is the same: an array that has the same number of unique elements.

These differences won't affect the accuracy of the results.
But, it's worth to deal with them to give you an improved testing environment.

Mathematica (Wolfram Language), 31 bytes

Length@*WordCounts@*ToLowerCase


The Mathematica built-in WordCounts treats a hyphenated word as a single word. So this program is not going to work correctly on the fourth example, "hello-world2". I discuss this in the comment below.

• If I am interpreting the problem correctly, the string "mother-in-law" should be treated as three words, but "mother_in_law" should be one word. Isn't it more natural to do the opposite? You would never encounter the string "mother_in_law" in published text. Mar 3 at 20:37
• I edited my code to be shorter thanks to a suggestion from @ZaMoC Mar 3 at 21:11

C (GCC), 156 + 48 + 22 = 226 bytes

-3 bytes thanks to ceilingcat

Use compiler flags -DW(a)=for(--s;a!isalnum(*++s)|*s==95;)*s|=32; and -DF(a)=for(a=0;a<i;++a).

char*v[99],*e;i,j,k;f(char*s){e=s+strlen(s);W()for(i=0;s<e;){v[i++]=s;W(!)W((*s=0)|)}F(j)F(k)strcmp(v[j],v[k])|j==k||(*v[j]=0);k=0;F(j)*v[j]&&++k;return k;}


Try It Online!

Explanation:

char*v[99],*e;
i,j,k;
f(char*s)
{
// Set e to the end of s
e=s+strlen(s);
// Set s to the first character that is alphanumerical or an underscore.
W()
// While s hasn't moved past the end of the string ...
for(i=0;s<e;) {
// Store s in v[i], then increment i
v[i++]=s;
// Set s to the first character that isn't alphanumerical or an
// underscore.
W(!)
// Set all characters between the current word and the next to 0, and
// set s to the first character in the next word.
W((*s=0)|)
}
// Iterate through the words with j.
F(j)
// Iterate through the words with k.
F(k)
// If words at index j and k are equal but j and k aren't the same
// index, set the first character in word j to 0, marking the word
// as a duplicate.
strcmp(v[j],v[k])|j==k||(*v[j]=0);
k=0;
F(j)
// For each word that hasn't been marked as a duplicate, increment k.
*v[j]&&++k;
// Return the number of words not marked as a duplicate.
return k;
}

• Suggest *v[j]*=strcmp(v[j],v[k])||j==k; instead of strcmp(v[j],v[k])|j==k||(*v[j]=0); Mar 4 at 17:19

C (gcc), 275254250 249 bytes

• -26 bytes thanks to ceilingcat

To split the words, each uppercased word is stored in a list recursively. Duplicates are nulled out, preventing them from being scanned.

g(s,t,i)char*s,**t;{char*a[2]={0,t},**v,*u;for(;*s&&!isalnum(*s)&&*s-95;s++);u=*a=strdup(s);if(i=*u){for(;*u=*s&&isalnum(*s)|*s==95;*u++=~32&*s++);i=g(s,a);}else for(v=t;v;v=v[1])if(*v)for(i++,t=v;t=t[1];)*t&&!strcmp(*v,*t)?*t=0:0;s=i;}f(s){g(s,0);}


Try it online!

Ungolfed (with a structure instead of an array):

struct list { char *data; struct list *prev; };

int g(char *s, struct list *t) {
int i;
char *u;
struct list a={0,t}, *v;

for(;*s&&!(isalnum(*s)&&*s-'_');s++);
u=a.data=strdup(s); // skip spaces and duplicate the string locally

if(i=*u){ // collect the word and uppercase it
for(;*u=*s&&isalnum(*s)|*s=='_';*u++=~32&*s++);
i=g(s,&a); // recursively generate list
}else // end of string: process the words
for(v=t;v;v=v->prev) // from the end, work backwards
if(v->data) // if not a duplicate
for(i++,t=v;t=t->prev;) // scan for duplicates
t->data&&!strcmp(v->data,t->data)?t->data=0:0; // null out duplicates

s=i; // return the count
}

int f(char *s) { g(s,0); } // initialize the end of list and (implicitly) return the count


Stax, 8 bytes

│ÿîIΔ»╝H


Run and debug it

This is a packed stax program. When unpacked, it is the following:

v"\w+"|Fu%


Run and debug it

Explanation

v          # lowercase the input string
|F   # get all regex pattern matches of regex
"\w+"     # \w+
u  # uniquify
% # length


Go, 181 176 bytes

import(."regexp";."strings")
func f(s string)int{m:=make(map[string]int)
for _,w:=range MustCompile("\\w+").FindAllString(ToLower(s),-1){if _,o:=m[w];!o{m[w]=1}}
return len(m)}


Attempt This Online!

Gets all words matching regex \w+, and adds them to a map (acting as a set in this case). Then it returns the number of items in the map (set).

• -5 bytes by @The Thonnu

Charcoal, 31 bytes

≔⦃⦄θＦΦ⪪↧Ｓ⁻⪪γ¹⊞ＯＥ³⁶⍘ιφ_ι§≔θιιＩＬθ


Attempt This Online! Link is to verbose version of code. Explanation: Inspired by @KevinCruijssen's 05AB1E answer, so assumes the input only contains printable ASCII.

≔⦃⦄θ


ＦΦ⪪↧Ｓ⁻⪪γ¹⊞ＯＥ³⁶⍘ιφ_ι


Split the lowercased input on all printable ASCII except the characters used to encode base 36 and _, and...

§≔θιι


... set each word as a key in the dictionary.

ＩＬθ


Output the final length of the dictionary.

Raku, 21 bytes

+*.lc.comb(/\w+/).Set


Try it online!

This is an anonymous function. The argument (*) is converted to lowercase (.lc), then the substrings matching one or more word characters are extracted (.comb(/\w+/)) and converted to a set (.Set), which discards the duplicates. Finally, that set is expressed as a number (+), yielding its size.