To not speak with the person knocking at the window at night means security. When we do not speak, the person might think that no one is in the house and may be forced to leave. If the agenda is that important, he/she will be coming back the following day to speak of it.