What is attachment? It is the body, consciousness. So, misery occurs due to attachment to the body. But, one can be happy physically as well as spiritually if one's senses are under one's control. In fact, sadness is not natural to man. Therefore, methods have to be found out to remove sadness, which is artificial.
Misery can be removed only by prayer and by following the spiritual path. If idle horses are overfed, they will be still lazier. Similarly, if you act according to the whims and fancies of your senses, your senses will get strengthened day by day and ultimately you will cease to be a human being.