R, 119 117 bytes, score=24 22

Thanks to Dominic van Essen for spotting a bug.

cat('Hello, Wo',intToUtf8(nchar("#$%&*-./012479:;<>?@ABDEFGIJKLMNOPQRSVXYZ[]^_`bgjkmquvwyz{|}~")+53),"\x6Cd!",sep="")

Try it online!

Uses the hex code trick to encode the last l as \x6C. This saves 2 bytes: 1 by avoiding the reuse of the l (none of the other characters are used anywhere else), and 1 by avoiding having to escape the \ (as \\) in the string.

R, 119 117 bytes, score=24 22

Thanks to Dominic van Essen for spotting a bug.

cat('Hello, Wo',intToUtf8(nchar("#$%&*-./012479:;<>?@ABDEFGIJKLMNOPQRSVXYZ[]^_`bgjkmquvwyz{|}~")+53),"\x6Cd!",sep="")

Try it online!

Uses the hex code trick to encode the last l as \x6C. This saves 2 bytes: 1 by avoiding the reuse of the l (none of the other characters are used anywhere else), and 1 by avoiding having to escape the \ (as \\) in the string.

The long string in the middle is 61 characters long. They are all necessary to get the character r (ASCII code 114), which is produced by the intToUtf8(nchar("...")+53) part.

R, 119 bytes, score=24

Thanks to Dominic van Essen for spotting a bug.

cat('H',intToUtf8(nchar("\\#$%&*-./0124579:;<>?@ABCDEFGIJKLMNOPQRSVXYZ[]^_`bgjkmquvwxyz{|}~")+36),'llo, World!',sep="")

Try it online!

R, 119 117 bytes, score=24 22

Thanks to Dominic van Essen for spotting a bug.

cat('Hello, Wo',intToUtf8(nchar("#$%&*-./012479:;<>?@ABDEFGIJKLMNOPQRSVXYZ[]^_`bgjkmquvwyz{|}~")+53),"\x6Cd!",sep="")

Try it online!

Uses the hex code trick to encode the last l as \x6C. This saves 2 bytes: 1 by avoiding the reuse of the l (none of the other characters are used anywhere else), and 1 by avoiding having to escape the \ (as \\) in the string.

R, 117 bytes, score=22

cat(intToUtf8(nchar("\\#$%&*-./01234679:;<>?@ABCDEFGHIJKLMNOPQRSVXYZ[]^_`bgjkmquvwxyz{|}~")+5),'ello, World!',sep="")

Try it online!

R, 119 bytes, score=24

Thanks to Dominic van Essen for spotting a bug.

cat('H',intToUtf8(nchar("\\#$%&*-./0124579:;<>?@ABCDEFGIJKLMNOPQRSVXYZ[]^_`bgjkmquvwxyz{|}~")+36),'llo, World!',sep="")

Try it online!