Remove duplicates from a String

Question

Inspired by this unassuming StackOverflow question.

The idea is simple; given a String and an array of Strings, remove any instances of words in the array (ignoring case) from the input String other than the first, along with any additional whitespace this may leave. The words must match entire words in the input String, and not parts of words.

e.g. "A cat called matt sat on a mat and wore a hat A cat called matt sat on a mat and wore a hat", ["cat", "mat"] should output "A cat called matt sat on a mat and wore a hat A called matt sat on a and wore a hat"

Input

• Input can be taken as either a String, and an array of Strings or an array of Strings where the input String is the first element. These parameters can be in either order.
• The input String may not be taken as a list of space-delimited Strings.
• The input String will have no leading, trailing or consecutive spaces.
• All input will only contain characters [A-Za-z0-9] with the exception of the input String also including spaces.
• The input array may be empty or contain words not in the input String.

Output

• The output can either be the return value from a function, or printed to STDOUT
• The output must be in the same case as the original String

Test cases

the blue frog lived in a blue house, [blue] -> the blue frog lived in a house
he liked to read but was filled with dread wherever he would tread while he read, [read] -> he liked to read but was filled with dread wherever he would tread while he
this sentence has no matches, [ten, cheese] -> this sentence has no matches
this one will also stay intact, [] -> this one will also stay intact
All the faith he had had had had no effect on the outcome of his life, [had] -> All the faith he had no effect on the outcome of his life
5 times 5 is 25, [5, 6] -> 5 times is 25
Case for different case, [case] -> Case for different
the letters in the array are in a different case, [In] -> the letters in the array are a different case
This is a test Will this be correct Both will be removed, [this,will] -> This is a test Will be correct Both be removed

As this is code golf, lowest byte count wins!

Answer 1

R, 84 bytes

function(s,w,S=el(strsplit(s," ")),t=tolower)cat(S[!duplicated(x<-t(S))|!x%in%t(w)])

Try it online!

Less than 100 bytes on a string challenge that's not also kolmogorov-complexity?

Explanation:

After we break up the string into words, we need to exclude those that are

1. duplicates and
2. in w

or alternatively, turning that on its head, keeping those that are

1. the first occurrence of a word OR
2. not in w.

duplicated neatly returns logical indices of those that are not the first occurrence, so !duplicated() returns indices of those which are first occurrences, and x%in%w returns logical indices for x of those which are in w. Neat.

Answer 2

Java 8, 117 110 bytes

a->s->{for(String x:a)for(x="(?i)(.*"+x+".* )"+x+"( |$)(.*)";s.matches(x);s=s.replaceAll(x,"$1$3"));return s;}

Explanation:

Try it online.

a->s->{                // Method with String-array and String parameters and String return
  for(String x:a)      //  Loop over the input-array
    for(x="(?i)(.*"+x+".* )"+x+"( |$)(.*)";
                       //   Regex to match
        s.matches(x);  //   Inner loop as long as the input matches this regex
      s=s.replaceAll(x,"$1$3")); 
                       //    Replace the regex-match with the 1st and 3rd capture groups
  return s;}           //  Return the modified input-String

Additional explanation for the regex:

(?i)(.*"+x+".* )"+x+"( |$)(.*)   // Main regex to match:
(?i)                             //  Enable case insensitivity
    (                            //  Open capture group 1
     .*                          //   Zero or more characters
       "+x+"                     //   The input-String
            .*                   //   Zero or more characters, followed by a space
               )                 //  End of capture group 1
                "+x+"            //  The input-String again
                     (           //  Open capture group 2
                       |$        //   Either a space or the end of the String
                         )       //  End of capture group 2
                          (      //  Open capture group 3
                           .*    //   Zero or more characters
                             )   //  End of capture group 3

$1$3                             // Replace the entire match with:
$1                               //  The match of capture group 1
  $3                             //  concatted with the match of capture group 3

Answer 3

MATL, 19 18 bytes

"Ybtk@kmFyfX<(~)Zc

Inputs are: a cell array of strings, then a string.

Try it online! Or verify all test cases.

How it works

"        % Take 1st input (implicit): cell array of strings. For each
  Yb     %   Take 2nd input (implicit) in the first iteration: string; or
         %   use the string from previous iteration. Split on spaces. Gives
         %   a cell array of strings
  tk     %   Duplicate. Make lowercase
  @k     %   Push current string from the array taken as 1st input. Make
         %   lowercase
  m      %   Membership: gives true-false array containing true for strings
         %   in the first input argument that equal the string in the second
         %   input argument
  F      %   Push false
  y      %   Duplicate from below: pushes the true-false array again
  f      %   Find: integer indices of true entries (may be empty)
  X<     %   Minimum (may be empty)
  (      %   Assignment indexing: write false in the true-false array at that
         %   position. So this replaces the first true (if any) by false
  ~      %   Logical negate: false becomes true, true becomes false
  )      %   Reference indexing: in the array of (sub)strings that was
         %   obtained from the second input, keep only those indicated by the
         %   (negated) true-false array
  Zc     %   Join strings in the resulting array, with a space between them
         % End (implicit). Display (implicit)

Answer 4

Perl 5, 49 bytes

@B=<>;$_=join$",grep!(/^$_$/xi~~@B&&$v{+lc}++),@F

Try it online!

Saved 9 (!!) bytes thanks to @TonHospel!

Answer 5

Pyth, 27 bytes

jdeMf!}r0eT@mr0dQmr0dPT._cz

Try it online

Explanation

jdeMf!}r0eT@mr0dQmr0dPT._cz
                          z  Take the string input.
                       ._c   Get all the prefixes...
    f    eT@                 ... which end with something...
     !}         Q    PT      ... which is not in the input and the prefix...
       r0   mr0d mr0d        ... case insensitive.
jdeM                         Join the ends of each valid prefix.

I'm sure the 10 bytes for case insensitive check can be reduced, but I don't see how.

Answer 6

Stax, 21 bytes^CP437

åìøΓ²¬$M¥øHΘQä~╥ôtΔ♫╟

25 bytes when unpacked,

vjcm[]Ii<;e{vm_]IU>*Ciyj@

The result is an array. The convenient output for Stax is one element per line.

Run and debug online!

Explanation

vj                           Convert 1st input to lowercase and split at spaces,
  c                          Duplicate at the main stack
   m                         Map array with the rest of the program 
                                 Implicitly output
    []I                      Get the first index of the current array element in the array
       i<                    Test 1: The first index is smaller than the iteration index
                                 i.e. not the first appearance
         ;                   2nd input
          {vm                Lowercase all elements
             _]I             Index of the current element in the 2nd input (-1 if not found)
                U>           Test 2: The index is non-negative
                                 i.e. current element is a member of the 2nd input
                  *C         If test 1 and test 2, drop the current element
                                 and go on mapping the next
                    iyj@     Fetch the corresponding element in the original input and return it as the mapped result
                                 This preserves the original case

Answer 7

Perl 6, 49 bytes

->$_,+w{~.words.grep:{.lc∉w».lc||!(%){.lc}++}}

Test it

Expanded:

->              # pointy block lambda
  $_,           # first param ｢$_｣ (string)
  +w            # slurpy second param ｢w｣ (words)
{

  ~             # stringify the following (joins with spaces)

  .words        # split into words (implicit method call on ｢$_｣)

  .grep:        # take only the words we want

   {
     .lc        # lowercase the word being tested
     ∉          # is it not an element of
     w».lc      # the list of words, lowercased

     ||         # if it was one of the words we need to do a secondary check

     !          # Boolean invert the following
                # (returns true the first time the word was found)

     (
       %        # anonymous state Hash variable
     ){ .lc }++ # look up with the lowercase of the current word, and increment
   }
}

Answer 8

Perl 5, 50 48 bytes

Includes +1 for -p

Give the target string followed by each filter word on separate lines on STDIN:

perl -pe '$"="|";s%\b(@{[<>]})\s%$&x!$v{lc$1}++%iegx;chop';echo
This is a test Will this be correct Both will be removed
this
will
^D
^D

The chop is only needed to fix the trailing space in case the last word gets removed

Just the code:

$"="|";s%\b(@{[<>]})\s%$&x!$v{lc$1}++%iegx;chop

Try it online!

Answer 9

JavaScript (ES6), 98 bytes

s=>a=>s.split` `.filter(q=x=>(q[x=x.toLowerCase()]=eval(`/\\b${x}\\b/i`).test(a)<<q[x])<2).join` `

Answer 10

K4, 41 bytes

Solution:

{" "/:x_/y@>y:,/1_'&:'(_y)~/:\:_x:" "\:x}

Examples:

q)k){" "/:x_/y@>y:,/1_'&:'(_y)~/:\:_x:" "\:x}["A cat called matt sat on a mat and wore a hat A cat called matt sat on a mat and wore a hat";("cat";"mat")]
"A cat called matt sat on a mat and wore a hat A called matt sat on a and wore a hat"

q)k){" "/:x_/y@>y:,/1_'&:'(_y)~/:\:_x:" "\:x}["Case for different case";enlist "case"]
"Case for different"

q)k){" "/:x_/y@>y:,/1_'&:'(_y)~/:\:_x:" "\:x}["the letters in the array are in a different case";enlist "In"]
"the letters in the array are a different case"

q)k){" "/:x_/y@>y:,/1_'&:'(_y)~/:\:_x:" "\:x}["5 times 5 is 25";(1#"5";1#"6")]
"5 times is 25"

Explanation:

Split on whitespace, lowercase both inputs, look for matches, remove all but the first occurrence, join the string back together.

{" "/:x_/y@>y:,/1_'&:'(_y)~/:\:_x:" "\:x} / the solution
{                                       } / lambda with implicit x & y args
                                  " "\:x  / split (\:) on whitespace " "
                                x:        / save result as x
                               _          / lowercase x
                          ~/:\:           / match (~) each right (/:), each left (\:)
                      (_y)                / lowercase y
                   &:'                    / where (&:) each ('), ie indices of matches
                1_'                       / drop first of each result
              ,/                          / flatten
            y:                            / save result as y
         y@>                              / descending indices (>) apply (@) to y
      x_/                                 / drop (_) from x
 " "/:                                    / join (/:) on whitespace " "

Answer 11

JavaScript (Node.js), 75 bytes

f=(s,a)=>a.map(x=>s=s.replace(eval(`/\\b${x}\\b */ig`),s=>i++?"":s,i=0))&&s

Try it online!

Answer 12

JavaScript ES6, 78 Bytes

f=(s,a,t={})=>s.split` `.filter(w=>a.find(e=>w==e)?(t[w]?0:t[w]=1):1).join` `

How it works:

f=(s,a,t={})=> // Function declaration; t is an empty object by default
s.split` ` // Split the string into an array of words
.filter(w=> // Declare a function that, if it returns false, will delete the word
  a.find(e=>w==e) // Returns undeclared (false) if the word isn't in the list
  ?(t[w]?0 // If it is in the list and t[w] exists, return 0 (false)
    :t[w]=1) // Else make t[w] exist and return 1 (true)
  :1) // If the word isn't in the array, return true (keep the word for sure)
.join` ` // Rejoin the string

Answer 13

PowerShell v3 or later, 104 bytes

Param($s,$w)$w|?{$_-and$s-match($r="\b$_(?: |$)")}|%{$h,$t=$s-split$r;$s="$h$($Matches.0)$(-join$t)"};$s

At the cost of one byte, it can run in PS 2.0 by replacing $Matches.0 with $Matches[0].

Long version:

Param($s, $w)
$w | Where-Object {$_ -and $s -match ($r = "\b$_(?: |$)")} |    # Process each word in the word list, but only if it matches the RegEx (which will be saved in $r).
    ForEach-Object {                                            # \b - word boundary, followed by the word $_, and either a space or the end of the string ($)
        $h, $t = $s -split $r                                   # Split the string on all occurrences of the word; the first substring will end up in $h(ead), the rest in $t(ail) (might be an array)
        $s = "$h$($Matches.0)$(-join $t)"                       # Create a string from the head, the first match (can't use the word, because of the case), and the joined tail array
    }
$s                                                              # Return the result

Usage
Save as Whatever.ps1 and call with the string and the words as arguments. If more than one word needs to be passed, the words need to be wrapped in @():

.\Whatever.ps1 -s "A cat called matt sat on a mat and wore a hat A cat called matt sat on a mat and wore a hat" -w @("cat", "mat")

Alternative without file (can be pasted directly into a PS console):
Save the script as ScriptBlock (inside curly braces) in a variable, then call its Invoke() method, or use it with Invoke-Command:

$f={Param($s,$w)$w|?{$_-and$s-match($r="\b$_(?: |$)")}|%{$h,$t=$s-split$r;$s="$h$($Matches.0)$(-join$t)"};$s}
$f.Invoke("A cat called matt sat on a mat and wore a hat A cat called matt sat on a mat and wore a hat", @("cat", "mat"))
Invoke-Command -ScriptBlock $f -ArgumentList "A cat called matt sat on a mat and wore a hat A cat called matt sat on a mat and wore a hat", @("cat", "mat")

Answer 14

Javascript, 150 bytes

s=(x, y)=>{let z=new Array(y.length).fill(0);let w=[];for(f of x)(y.includes(f))?(!z[y.indexOf(f)])&&(z[y.indexOf(f)]=1,w.push(f)):w.push(f);return w}

Answer 15

Clean, 153 142 138 134 bytes

import StdEnv,StdLib,Text
@ =toUpperCase
$s w#s=split" "s
=join" "[u\\u<-s&j<-[0..]|and[i<>j\\e<-w,i<-drop 1(elemIndices(@e)(map@s))]]

Try it online!

Defines the function $ :: String [String] -> String, pretty much literally doing what the challenge describes. It finds and removes every occurrence after the first, for each target word.

Answer 16

Retina, 46 37 bytes

+i`(^|,)((.+),.*\3.* )\3( |$)
$2
.*,

-14 bytes thanks to @Neil, and +5 bytes for a bug-fix.

Input in the format word1,word2,word3,sentence, because I'm not sure how to have multi-line input (where the inputs are used differently)..

Explanation:

Try it online.

+i`(^|,)((.+),.*\3.* )\3( |$)   Main regex to match:
+i`                              Enable case insensitivity
   (^|,)                          Either the start of the string, or a comma
        (                         Open capture group 2
         (                         Open capture group 3
          .+                        1 or more characters
            )                      Close capture group 3
             ,                     A comma
              .*                   0 or more characters
                \3                 The match of capture group 3
                  .*               0 or more characters, followed by a space
                     )            Close capture group 2
                      \3          The match of capture group 2 again
                        ( |$)     Followed by either a space, or it's the end of the string
$2                              And replace everything with:
                                 The match of capture group 2

.*,                             Then get everything before the last comma (the list)
                                 and remove it (including the comma itself)

Answer 17

Red, 98 bytes

func[s w][foreach v w[parse s[thru[any" "v ahead" "]any[to remove[" "v ahead[" "| end]]| skip]]]s]

Try it online!

f: func [s w][ 
    foreach v w [                   ; for each string in the array
        parse s [                   ; parse the input string as follows:
            thru [                  ; keep everything thru: 
                any " "             ; 0 or more spaces followed by
                v                   ; the current string from the array followed by
                ahead " "           ; look ahead for a space
            ]
            any [ to remove [       ; 0 or more: keep to here; then remove: 
                " "                 ; a space followed by 
                v                   ; the current string from the array
                ahead [" " | end]]  ; look ahead for a space or the end of the string
            | skip                  ; or advance the input by one 
            ]
        ]
    ]
    s                               ; return the processed string 
]

Answer 18

Husk, 13 bytes

wüöVËm_Ṗ3+⁰ew

Takes a list of strings and a single string as arguments, in this order. Assumes that the list is duplicate-free. Try it online!

Explanation

wüöVËm_Ṗ3+⁰ew  Inputs: list of strings L (explicit, accessed with ⁰), string S (implicit).
               For example, L = ["CASE","for"], s = "Case for a different case".
            w  Split S on spaces: ["Case","for","a","different","case"]
 ü             Remove duplicates wrt an equality predicate.
               This means that a function is called on each pair of strings,
               and if it returns a truthy value, the second one is removed.
  öVËm_Ṗ3+⁰e    The predicate. Arguments are two strings, say A = "Case", B = "case".
           e    Put A and B into a list: ["Case","case"]
         +⁰     Concatenate with L: ["CASE","for","Case","case"]
       Ṗ3       All 3-element subsets: [["CASE","for","Case"],["CASE","for","case"],
                                        ["CASE","Case","case"],["for","Case","case"]]
  öV            Does any of them satisfy this:
    Ë            All strings are equal
     m_          after converting each character to lowercase.
                In this case, ["CASE","Case","case"] satisfies the condition.
               Result: ["Case","for","a","different"]
w              Join with spaces, print implicitly.

Answer 19

Min, 125 bytes

=a () =b a 1 get =c a 0 get " " split
(:d (b d in?) ((c d in?) (d b append #b) unless) (d b append #b) if) foreach
b " " join

Input is quot on stack with input String as first element, and a quot of the duplicate strings as second element, i.e.

("this sentence has no matches" ("ten" "cheese"))

Answer 20

Python 3, 168 bytes

def f(s,W):
 s=s.split(" ");c={w:0for w in W}
 for w in W: 
  for i,v in enumerate(s):
   if v.lower()==w.lower():
    c[w]+=1
    if c[w]>1:s.pop(i)
 return" ".join(s)

Try it online!

Answer 21

AWK, 120 bytes

NR%2{for(;r++<NF;)R[tolower($r)]=1}NR%2==0{for(;i++<NF;$i=$(i+s))while(R[x=tolower($(i+s))])U[x]++?++s:i++;NF-=s}NR%2==0

Try it online!

The "remove whitespace" part made this a bit more challenging than I first thought. Setting a field to "", removes a field, but leaves an extra separator.

The TIO link has 28 extra bytes to allow multiple entries.

Input is given over 2 lines. The first line is the list of words and the second is the "sentence". Note that "word" and "word," are not considered identical do to the attached punctuation. Having punctuation requirements would likely make this an even more fun problem.

Answer 22

Ruby, 63 61 60 59 bytes

->s,w{w.any?{|i|s.sub! /\b(#{i}\b.*) #{i}\b/i,'\1'}?redo:s}

Try it online!

A shorter version that's case sensitive and fails ~every 10¹⁵ times because of randomness (37 bytes)

->s,w{s.uniq{|i|w.member?(i)?i:rand}}

Answer 23

Python 2, 140 bytes

from re import*
p='\s?%s'
S,A=input()
for a in A:S=sub(p%a,lambda s:s.end()==search(p%a,S,flags=I).end()and s.group()or'',S,flags=I)
print S

Try it online!

Explanation:

re.sub(..) can take as argument a function instead of replacement string. So here we have some fancy lambda. Function is called for each occurence of pattern and one object is passed to this function - matchobject. This object has information about founded occurence. I'm interested in index of this occurence, that can be retrieved by start() or end() function. Latter is shorter so it is used.

To exclude replacement of first occurence of word, I used another regex search function to get exactly firts one and then compare indexes, using same end()

Flag re.I is short version of re.IGNORECASES