I agree. Having a foot in multiple locations - Pack, Troop and District - if your heart isn't totally "all-in" it makes it difficult to serve. And, when you no longer have a boy in the Pack, easier to walk away. That being said, I'm enjoying the Pack Leadership maybe MORE since my son has crossed over to Boy Scouts.
Has anyone said something like, we realize it's hard to walk away but we're ready to take this on and we know how to find you if we get into a bind? Maybe they're staying on because they're afraid of what they built is going to fail. You could be the most competent person ever, but if they've never worked with you before, they may not know that...
If that doesn't work, perhaps working with your Commissioner to help you sort out a resolution.