J, 18 bytes
'''&()[-.'&([-.-.)
How it works
''&()[-. the character set, '' stands for '
& is the left operand
([ ) the set> itself, ignore <the input
( -.) the set> except any characters in <the input
(the wrong ones remain)
( -. ) the set itself> except <the wrong characters