How?

ỊẠ€T - Link 1: indices of items which only contain "insignificant" values: list of lists
Ị    - insignificant (vectorises) -- 1 if (-1<=value<=1) else 0 
  €  - for €ach:
 Ạ   -   all?
   T - truthy indices

ŒJ_€`Ç€ḟ"J$ịFg"FÇịF - Main Link: list of lists of positive integers, M
ŒJ                  - multi-dimensional indices
    `               - use as right argument as well as left...
   €                -   for €ach:
  _                 -     subtract (vectorises)
      €             - for €ach:
     Ç              -   call last Link (1) as a monad
          $         - last two links as a monad:
         J          -   range of length -> [1,2,3,...,n(elements)]
        "           -   zip with:
       ḟ            -     filter discard (remove the index of the item itself)
            F       - flatten M
           ị        - index into (vectorises) -- getting a list of lists of neighbours
               F    - flatten M
              "     - zip with:
             g      -   greatest common divisor
                Ç   - call last Link (1) as a monad
                  F - flatten M
                 ị  - index into

Jelly, 24 bytes

^{Hmm, seems long.}

ỊẠ€T
ŒJ_€`Ç€ḟ"J$ịFg"FÇịF

Try it online!

Jelly, 24 bytes

^{Hmm, seems long.}

ỊẠ€T
ŒJ_€`Ç€ḟ"J$ịFg"FÇịF

A monadic Link accepting a list of lists of positive integers which returns a list of each of the values which are in hostile neighbourhoods (version 1 with no de-duplication).

Try it online! Or see a test-suite.