For those who had a habit of nodding off, the risk of stroke was two to four times higher than for those who never fell asleep in the day, a study found.
Speaking at the International Stroke Conference, the team advised doctors to check out older people who found they were dropping off in front of the TV.
The study asked 2,000 people how often they dozed off in different situations.
These included while watching TV, sitting and talking to someone, sitting quietly after a lunch without alcohol and stopping briefly in traffic while driving.
The risk of stroke over the next two years was 2.6 times greater for people who reported "some dozing" compared to those with no dozing.
Among those who reported "significant dozing" the risk was 4.5 times higher.
The researchers also found the risk of heart attack or death from vascular disease was 1.6% higher for moderate dozers and 2.6% higher for significant dozers.
Study leader, Dr Bernadette Boden-Albala, assistant professor of neurology at Columbia University, New York, said: "Those are significant numbers. We were surprised that the impact was that high for such a short period of time."
Previous research has shown that people who suffer from sleep apnoea - short periods when breathing stops during sleep - have an increased stroke risk.
It could be that daytime sleepiness is a sign of sleeping poorly at night because of sleep apnoea.
"Given what's known now, it's worth assessing patients for sleep problems," Dr Boden-Albala said.
"If patients are moderately or significantly dozing, physicians need to think about sending them for further evaluation."
She added other studies had shown people were not getting enough sleep, making them consistently tired.
"But the real question is: 'What are we doing to our bodies?'. Sleepiness obviously puts us at risk of stroke."
Dr Heinrich Audebert, consultant stroke physician at Guy's and St Thomas' Hospital in London said the findings seemed reasonable.
"Sleep apnoea is a risk factor for stroke and in Mediterranean countries the siesta is associated with a little bit of an increased daytime risk of stroke."
He explained that patients with sleep apnoea had increased blood pressure levels during the night.
One other potential cause for the findings could be previous undiagnosed minor strokes causing damage to the brain and leading to more sleepiness during the day, he said.
"What we really encourage is that all patients who have breaks in sleeping in the night should have sleep apnoea screening."