Could be one of two things....clutch adjustment( because its not fully engaging the clutch while shifting, and once you get into 3rd you can actually shirt without the clutch if needed, or a bend shift fork.....which can be caused by knocking it into gear or missing gears while shifting

Your shifter is connected to a fork inside your case. you must take your shifter off and take the side cover off. there is a bolt that you must take out so the clutch shaft comes out. you will see a fork in there. it is bent and that is why it shifts hard. some times it wont shift when you want it to. go to victory and say that you want that fork, gaskets, and seals to do the job. it will be $250-$300 to do. if you dont fix it, it will get worse. i fixed mine and it works perfect. same problems you had.