I think this is something that you two have to discuss. This is not something that can be left to gather dust. This should be confronted as soon as possible. You should make your feelings clear towards him, how you feel about this. Perhaps you can work something out of this.