2 corrected & improved upon explanatory comments
source | link
var f =
d => d.sort(
    (
     a, b,
     g = (n, [x, y, z] = n) =>               // destructure rolls, e.g. "321" => [3["3",2"2",1]"1"]
        n **=                                // differentiate scores by raising to somethe exponent corresponding to its category in the table above
                                             // all conditionals check for a falsy resultvalue
            n - 421
                ? y * z - 1                  // ends with "11"
                    ? x - y | y - z          // three-of-a-kind
                        ? ~y % x | ~z % y    // straights, with operators chosen to correctly convert strings to numbers
                            ? 1
                            : 2
                        : 3
                    : x - 1                  // three aces
                        ? 3.1
                        : 5
                : 5
    ) =>
    g(b) - g(a)
)
var f =
d => d.sort(
    (
     a, b,
     g = (n, [x, y, z] = n) =>               // destructure rolls, e.g. "321" => [3,2,1]
        n **=                                // differentiate scores by raising to some exponent
                                             // all conditionals check for a falsy result
            n - 421
                ? y * z - 1                  // ends with "11"
                    ? x - y | y - z          // three-of-a-kind
                        ? ~y % x | ~z % y    // straights, with operators chosen to correctly convert strings to numbers
                            ? 1
                            : 2
                        : 3
                    : x - 1                  // three aces
                        ? 3.1
                        : 5
                : 5
    ) =>
    g(b) - g(a)
)
var f =
d => d.sort(
    (
     a, b,
     g = (n, [x, y, z] = n) =>               // destructure rolls, e.g. "321" => ["3","2","1"]
        n **=                                // differentiate scores by raising to the exponent corresponding to its category in the table above
                                             // all conditionals check for a falsy value
            n - 421
                ? y * z - 1                  // ends with "11"
                    ? x - y | y - z          // three-of-a-kind
                        ? ~y % x | ~z % y    // straights, with operators chosen to correctly convert strings to numbers
                            ? 1
                            : 2
                        : 3
                    : x - 1                  // three aces
                        ? 3.1
                        : 5
                : 5
    ) =>
    g(b) - g(a)
)
1
source | link

JavaScript (ES7), 96 bytes

d=>d.sort((a,b,g=(n,[x,y,z]=n)=>n**=n-421?y*z-1?x-y|y-z?~y%x|~z%y?1:2:3:x-1?3.1:5:5)=>g(b)-g(a))

Sorts rolls by adhering closely to the rules of scoring. Expects an array of strings with individual rolls in descending order of value, e.g. ["654"]

Try it online!

Explanation

Categories of rolls are raised to the following exponents:

421                 5
Three Aces          5
Two Aces            3.1
Three-of-a-Kind     3
Straights           2
Other               1

Ungolfed

Mentally unwrapping the conditional checks gives me a headache, and I'm certain it can somehow be further golfed....

var f =
d => d.sort(
    (
     a, b,
     g = (n, [x, y, z] = n) =>               // destructure rolls, e.g. "321" => [3,2,1]
        n **=                                // differentiate scores by raising to some exponent
                                             // all conditionals check for a falsy result
            n - 421
                ? y * z - 1                  // ends with "11"
                    ? x - y | y - z          // three-of-a-kind
                        ? ~y % x | ~z % y    // straights, with operators chosen to correctly convert strings to numbers
                            ? 1
                            : 2
                        : 3
                    : x - 1                  // three aces
                        ? 3.1
                        : 5
                : 5
    ) =>
    g(b) - g(a)
)