Japt, 14 bytes
d@sY ð_§XÃxÈ-Y
Outputs false
for BST, true
for no BST.
Explanation:
d@ Run on each item X, return true if any aren't 0:
sY Ignore the numbers before this index
ð_§XÃ Get the indexes of numbers less than or equal to X
If it is a BST, then this list will be sequential numbers starting from 0
-Y Subtract the position within the index list from each index
eg. [0,1,2] -> [0,0,0] , [0,1,4] -> [0,0,2]
xÈ Sum the resulting array