I want to print my landscape photos. I have to add white frame around them (something about 500 px to width and height). I have tried many cases (thicknesses). Is there any rule how to apply frame to my photo, for example thickness should be no more 1/10 from width or something other rule from design point of view?
No. This is entirely fashion. As with ties, sometimes skinny is in and sometimes wide. Choose something that looks good to you with your image.
If there's a photography "scene" in your area, or in places you frequent online, you might look at what other people are doing, and then either try to fit in with the prevailing aesthetic or else choose how you want to be different.
You may also want to choose to not leave borders on your actual prints but instead frame and mount them with a mat board border.