Please tell me if there is a better way to do this, as this is both a bit convolluted, complicated, and subject to change at any time. It also requires you to use two excluded classes, and an mx_internal variable. That being said, this is how I approached it.
import mx.core.mx_internal;
import mx.managers.PopUpManagerImpl;
import mx.core.Singleton
use namespace mx_internal;
PopUpManagerImpl(Singleton.getInstance("mx.managers::IPopUpManager")).mx_internal::modalWindowClass = SomeClass;
Please note that you have to import and use mx_internal. You also have to import PopUpManagerImpl, as this is where the variable is defined. And you must also import Singleton, which is how the flex framework accesses singleton implementations within the framework. Once you've done this, you can access the 'modalWindowClass' mx_internal property on PopUpManagerImpl, and set it to your own modal class. When this is done, anytime you add a pop up, the modal blocker will use this class.
Any questions, or SUGGESTIONS on how to do this better, please comment below