I am going to preface this with a disclaimer, I have never been a SM or ASM, but I have been SPL, ASPL, and gone through similar to this circumstance, where the SM was focusing on his son's patrol and their needs rather than the troops, it is part of the reason I never never made eagle.
My suggestion is that you confirm weither or not the SM is on the way out, and if the other ASMs are planning on staying on after their boys are done, if the SM is on the way out, see if he will pass some of the duties off onto you or another ASM if they are staying on, as it will both acclimate his replacement to the job and smooth the transition process when he leaves. If his tenure is not ending, see about getting more authority in leading the younger scouts. During my last year as a scout, an ASM did such with the younger boys, taking myself along as an experienced, NYLT qualified scout(I still remember the Announcement Song), to help him, the younger scouts, and their parents on camping trips.