Brain-Flak, 178 bytes

{(([((((()()()()){}){}){}()){}]{}){{}<>(({}){}){}(<>)}{}<({}(<>))(<>)((()()()())({})()){{(({})){({}[()])<>}{}}<>({}<>)<>{}}{}><>{}{})<>}<>([]){{}({}(((({}){}){}){}){}<>)<>([])}<>

Try it online!

Brain-Flak, 178 bytes

{(([((((()()()()){}){}){}()){}]{}){{}<>(({}){}){}(<>)}{}<({}(<>))(<>)((()()()())({})()){{(({})){({}[()])<>}{}}<>({}<>)<>{}}{}><>{}{})<>}<>([]){{}({}(((({}){}){}){}){}<>)<>([])}<>

Try it online!

Explanation

# Step 1: convert to hex.
# For each pair of letters in the input:
{

  (

    # Compare first letter to B
    ([((((()()()()){}){}){}()){}]{})

    # If not B, pop previous output, multiply by 4, and put on third stack.
    # 4 is added automatically from pushing/popping the difference
    # between the letters B and D.
    {{}<>(({}){}){}(<>)}{}

    <

      # Push second letter in pair to other stack
      ({}(<>))

      # Push 4 and 9
      (<>)((()()()())({})())

      # Compute 3-((8-(n mod 9)) mod 4)
      # (same as (n-1) mod 9 mod 4)
      {{(({})){({}[()])<>}{}}<>({}<>)<>{}}{}

    >

    # Add result to third stack (either 0 or 4*previous+4)
    <>{}{}

  # Push onto second stack
  )

<>}

# Step 2: Pair up hex digits.
# While digits remain on right stack:
<>([]){{}

  # Push top of stack + 16*second on stack to left stack
  ({}(((({}){}){}){}){}<>)<>

([])}

# Switch to left stack for output.
<>