I don’t have any code readily available that I could post, but using the masks property was mostly straight forward:
Let S be the superlayer. It holds an (image) layer L1 such that S.bounds = L1.frame. The transition replaces L1 with another layer L2, L2.frame = L1.frame.
# create a PNG with exactly the same size L1. The PNG has to be completely opaque. (I simply made it a white rectangle with alpha = 100%.)
# load the PNG as a UIImage P.
# create a new layer M, M.frame = S.bounds, and set M.contents = (id)[P CGImage].
# set S.mask = M.
Apparently, Core Animation treats any pixel that falls outside M’s bounds as fully transparant.
Unfortunately, I did not find documentation if L becomes M’s superlayer, i.e. if is sufficient to set L.mask = nil if you want to re-use the mask or if you also have to call [M removeFromSuperlayer] or if the latter is actually the intended way to severethe connection between a layer and its mask. In the end, I did both.
For unrelated reasons I had to introduce an additional layer H inbetween S and L1. (So now L1.superlayer = H and H.superlayer = S.) Then I could forgo setting S.masks: Simply setting S.masksToBounds = YES turned out to be sufficient as soon as the transition did no longer act on S’s direct sublayers anymore but on S’s “grandchildren”.