Tokyo, Japan (CNN) -- The Olympics can be a cruel game with winners and losers. And in Tokyo, there is Kohei Jinno, 80 and his wife Yasuko, 79.
The elderly couple are experiencing a cruel sense of déjà vu, certainly not in keeping with the Olympic spirit.
"Why do we need another Olympic here?" Yasuko Jinno asked, referring to the recent decision to award the Summer Games to the city in 2020. "There are still many people there up in Tohoku (the northern Japan area hit by the 2011 earthquake). The money should be spent there, not here."
Their lack of enthusiasm stems from bitter memories of their first eviction.
Is hosting the Olympics a bad idea?
Tokyo celebrates winning 2020 Olympics
Tokyo wins 2020 Summer Olympic bid
Kohei Jinno's family home was razed to make way for the National Olympic Stadium ahead of the '64 Games. They were one of the last to leave the area after fighting in court. He also lost his tobacco shop to the construction project and had to start anew.
The family eventually agreed to move after the city government found a new apartment for them.
And nearly 50 years later, the aging couple is being asked to do it all over again.
Plans to expand the existing National Olympic Stadium means accommodating 80,000 spectators under an all-weather roof, which is the standard for hosting the opening ceremony. And that also means the couple's current apartment, which is owned by the Tokyo government, is in an area at the heart of that project.
Jinno's small tobacco shop at the ground level of the Kasumigaoka Apartments, is also to be demolished for the expanded stadium.
"There was only one explanation session held by the government so far, and that's it," said Yasuko Jinno.
A Tokyo housing official said that the government has been talking to the residents of Kasumigaoka Apartment since last summer when the project for the new stadium was revealed and that it has shown options to the residents to move.
The couple will be close to 90 by the time the Olympic Games returns to their neighborhood.
Kohei Jinno says they have no energy left to fight another drawn-out court case. He cannot open a new tobacco shop, because such businesses are heavily regulated.
"I don't want to see another Olympics," he said. "The money should be spent on the people who are still suffering after the quake and tsunami."
Soon, they'll have to leave their apartment, the open balcony where they've gotten to know their neighbors for the last five decades. Kohei Jinno will no longer take his daily stroll to his tobacco and public phone shop.
Earlier this month, the International Olympic Committee pipped Tokyo over Madrid and Istanbul for the right to host the world's biggest sporting event in seven years. The city raised significant capital and was considered the safe choice, despite the ongoing concerns over high radiation levels at the stricken Fukushima nuclear plant.
Tokyo's government had amassed a $4.9 billion Olympic fund to prepare for the Games, according to aroundtherings.com, which covers the business and politics of the Olympics. The Tokyo bid has been touted as a way of breathing new life into the world's third-largest economy, which has been dogged by deflation, stagnation and the Fukushima leaks in recent years.
When Tokyo won the Olympic bid, crowds at the capital welcomed the news, cheering and holding signs reading "We're waiting for the world to come to Tokyo in 2020."
Instead, Yasuko Jinno felt shock. "I did not think Tokyo would win and the Olympics would come here again."
CNN's Yoko Wakatsuki reported from Japan and Madison Park wrote from Hong Kong.
Loading weather data ...