Thanks.
Just a guess, but to me this looks like an error caused by precision loss. The MaskGenerator that osgEarth uses to cut holes uses OSG's osgUtil::DelaunayTriangulator internally; that class operates on single-precision floating point coordinates. So it's quite possible that the operation is introducing errors by truncating the input values.
Assuming this is the issue, I'm sorry but we don't have an easy fix at the moment! I recommend logging a GitHub issue (include your sample earth file -- no need for the imagery) so we can track it.
Glenn Waldron / Pelican Mapping