Highway Bonnie

Had my first long ride on a highway, total of 250km and took me forever.
Half the trip was wind free(other than the one created by movement) and traveled at a speed 100-120km/h, while I know it's not fast I still don't feel comfortable enough to go faster..
The other half had strong winds from sides(30-40km/h) and made the experience quite bad, I was gripping the handlebar hard, which I know I shouldn't but couldn't help it as the wind kept moving me around from side to side and I slowed down to an embarrassing 80km/h-90km/h for most of the rest of the trip. Any advice go make highways, especially with wind, more pleasant?

 

I haven't gone to the highway with thr panniers so it cant be that. I also don't think there is anything wrong with the bike even though this is my first big bike, I think its me, for some reason I don't feel safe riding at higher speeds, I'm hoping it will come with time. Just wondering if it's only me or if others have experienced the same when started riding.