The study of 1,300 children by Michigan and Montreal universities found negative effects on older children rose with every hour of toddler TV.
Performance at school was worse, while consumption of junk foods was higher.
UK experts said parents could allow young children to watch "some" high quality TV.
The study, part of the Quebec Longitudinal Study of Child Development Main Exposure, asked parents how much TV their children watched at 29 months (two years and five months) and 53 months (four years and five months).
On average, the two-year-olds watched just under nine hours of TV per week, while for four-year-olds the average was just under 15 hours.
But 11% of the two-year-olds and 23% of four-year-olds watched more than the recommended maximum of two hours of TV a day.
When the children were revisited at the age of 10, teachers were asked to assess the children's academic performance, behavior and health, while body mass index (BMI) was measured at 10 years old.
Higher levels of TV viewing at two was linked to a lower level of engagement in the classroom and poor achievement in maths.
Researchers also found a decrease in general physical activity but an increase in the consumption of soft drinks and in BMI (body mass index).
Dr Linda Pagani, of the University of Montreal, who led the research which was published in the journal Archives of Pediatrics & Adolescent Medicine, said: "Early childhood is a critical period for brain development and formation of behavior.
"High levels of TV consumption during this period can lead to future unhealthy habits.
"Common sense would suggest that television exposure replaces time that could be spent engaging in other developmentally enriching activities and tasks that foster cognitive, behavioural and motor development."
And she added:"Although we expected the impact of early TV viewing to disappear after seven and a half years of childhood, the fact that negative outcomes remained is quite daunting.
"Our findings make a compelling public health argument against excessive TV viewing in early childhood."
The UK's National Literacy Trust campaigns to raise awareness of how to police a toddler's viewing.
It said that until research demonstrated that children under two might benefit from TV, parents should, "limit exposure and encourage other one-to-one language-enhancing activities that centre on talk at mealtime, bath time, shared reading and imaginative play".
But it added: "Encourage exposure to some high-quality, age-appropriate educational television for children aged two to five."
British Psychological Society member Dr Aric Sigman has carried out his own research, which highlighted concerns over young children watching too much TV.
He said: "My recommendation to the government five years ago, and even as recently as three years ago, that they merely issue general guidelines on the amount of TV that children watch and the age at which they start was considered radical and controversial.
"Yet a growing body of evidence is now causing governments and health authorities elsewhere to do just that, and we need to as well.
"This is yet another study reinforcing the need for our society to finally accept that quite aside from good or bad parenting, children's daily screen time is a major independent health issue."