Extend viewdefs to change a value for a button

I am trying to disable a button in the Save button dropdown in a module's record view.

I would like to do that for just that single button without overriding the entire file, so I thought I would create and extension file for the record view and unset that array element.

Obviously if there are other extensions adding more buttons in this dropdown, I don't want to unset the wrong one, so I tried all kinds of foreach loops with if statements, array_walk_recursive(), array_map() and more foreach loops referencing the original array.

The only actual way I managed to do it is still this:

unset($viewdefs['Calls']['base']['view']['record']['buttons'][1]['buttons'][1]);

And if another button gets added at the top of the dropdown, this should theoretically remove the wrong button!

I cannot find anything in the documentation or examples of how this simple thing should be done besides copying the entire file. I believe that would make this simple change affect any other modules also trying to affect the same view.