I promise you aren't the only one. It is impossible for me to gain weight unless I drink 2-3 protein shakes, 5-6 meals, and a scoop of peanut butter every night. I'm 5'11, 163. Have been for two years. In HS I was way bulked up at 5'10, 185 and I actually felt way better then. I just don't have time to work out with school and now 60+ hour work weeks.
And like you, I really know how to drop weight. I wrestled my freshman and sophomore years of HS.