Brian, the ultimate measure of maturity is your willingness to do what is right when you know your actions will be opposed or ridiculed. Persecution is the furnace in which great character is forged, and this is a chance for you to choose the harder right instead of the easier wrong.
Before you is an opportunity to serve those who served their country against real enemies, people who would take their very lives if they could. They sacrificed their comfort and personal safety, their time, and often their lives, to stand against foes who wanted to destroy the things most dear to them. These were men and women who did the right thing because it was right - not because it was easy. To do them the small honor of lending dignity to their ceremony, at a small sacrifice of your time and comfort, would be a large step in building the character of the man you want to become.
I understand what it is to be looked down on by others. When I was in high school lo these many years ago (I graduated in 2002 so I exaggerate a bit), I was bullied and made fun of for all kinds of things - I was small, awkward-looking, and shy. So I had a choice to make - would I let the actions of others determine my right to create a positive experience for myself, or would I take command of myself and choose to have positive experiences despite their attempts to bring me down? And I chose the latter. I ignored their attempts to insult or offend me, I took pleasure in the activities I loved that they mocked, and I was kind to people that were mean to me. I forgave whatever they said or did to me, and continued on with my life. By the time I was a Senior, nobody made fun of me anymore, people started treating me better, and some people even told me as we prepared to graduate that they felt bad about how they had treated me, and wished me success as I went off to college. I made the experience a positive one, even if the circumstances that couched them were not.
Can you make Scouting "cool" in their eyes? No. But can you follow your passion to gain their respect? Yes. A thousand times yes. It will take time. It will be hard. But your ability to find the joy in your choices is stronger than their ability to put them down. You are doing something to serve people who gave up far more, went through much worse, and deserve something much better. Be the man you aspire to be now, and it will be much easier to maintain that dignity of character later. I pray your efforts will be strengthened as you make the choice you feel is right. Because, as you well know, you have more than enough wisdom to know what the right choice is, and more than enough courage to make it happen.