Basically.......this is the problem?

You are allowing your emotions to take control of your intelligence and falling into his lustful thoughts toward you and you must stop text and all communication with him and furthermore, let him know you're marriage to his brother and what he is doing is wrong and if he doesn't stop immediately will tell your husband.