Yup. I noticed your truck is just a couple years old, so it may just be the dramaticly different surfaces.
Under the trim is Really clean paint. The rest has been exposed to the elements.

Try claybarring and waxing on a section of your door. You'll be surprised by the difference claying makes, if you have never done it before.