The gyro reversing and replacing the servo helped. However, it still doesn't behave quite right. I think there is still some changes to be done on the radio setup and perhaps gyro settings.
I tried several flights like this and each was the same. Time to figure out what was going on... I'm also going to do the "hot glue" modification to the gyros to see if that improves the flight performance.