of course from what u meant in the previous statement , i understand that u are assuming that both parties are sort of already in an semi hostile type of interaction and keeping quiet would give definately negative impression.
i think what oscar wilde meant here would be that as a third person who is not involved and or maybe as a part of a 2 way conversation that is still in amiable situation , when some unpleasant thought flashes in your mind , sometimes it is best to just keep it to yourself instead of voicing it out , even if its is the truth , of course u have to voice out if keep quiet is going to be harmful to anyone , if it harmless , just way better to let it pass.
and yes "you have nothing of value to add" is a great way of interpreting it but the original phrase sounds more "wisdomly" and if u think of it in simplier terms without going deeply into psychology , it does explain abit.
oh its pass midnight now so i can put another quote
"Because bear don't eat shit" -Woody Stevens
Guess the movie =D