Not at all. It is understandable as to why you would, you have your reasons for being here.

I am often away from home so I'm not entirely attached.
[Though really he would have re-created his castle by now, it's just that it wouldn't be the same without his allies accompanying him.]