To solve this you must use Pythagorean theorem:

a and b are the legs (the sides that form a perpendicular/right angle) 
c is the hypotenuse (the side opposite the right angle) 
In this case... 
a = 5
b = 8 
c = x
^^^Plug these numbers into the theorem 
 
 
simplify
25 + 64 = 

89 = 

To remove the square from x take the square root of both sides to get you...
√89 = x 
(choice A)
Hope this helped!
~Just a girl in love with Shawn Mendes