His challenges annoys/irritates me.

• 他的挑战令我烦恼／激怒我。