Was My Horse Spooked or Being Difficult?

Ok so I know you weren't there witness it so I'll try to describe it the best I can. So I have an extremely green horse. He's been riding for about a week now. He has a tendency to try to turn around when I'm riding him but I don't let him. So today I was riding him and when I was getting to the end of the pasture, he tried to turn around but I didn't let him. On the other side of the fence is five lines of pine trees and you can't even see through them. He was sort of looking in that direction and he had his head high but he wasn't flaring his nostrils. He tried to turn again but I turned him and then he backed up and tried to turn. He did this five or six times before I had him move. this time I turned him around and he jolted forward and he did look like he was scared and he never did this before. I calmed him down and the rest of the way back was fine. The thing that makes me think he was being difficult was the fact that he is sort of buddy sour. The other horses where way on the other side of the pasture but another thing that makes me think he was scared is that I can lead him all the way down the dirt road we live on and they will be out of sight. He would whinny back if they did and stop a couple times but he never tried to pull away. I also rode him in different parts of the pasture with no problem but just the fact of thinking of him just being difficult brings me down because I trained him all on my own with no help because no one would get a trainer. I've worked so hard and finally got him to be ride-able.