The crowded, ungoverned settlement housed more than 33,000 people and was not the most comfortable place to live. Water from the taps was too dirty to drink, so residents had to fetch it from other sources.
But if anything, this tough childhood experience stirred the young boy's imagination. Hoi Chiu - whose real name is Choy Kam-chiu - would invent games to entertain himself. "The Walled City was like a maze," says the 38-year-old founder and artistic director of the All Theatre Art Association. "Every time I walked home, I would choose a different path just for fun.
"We had no toys. My father would sometimes bring home old comic books and there would be missing pages in between, so my brothers and I would invent a story for the missing part.
"Maybe it's because of my childhood experience that my mind is trained to be very flexible."
Hoi Chiu's open-mindedness led to a mixed career. With some background in stage production, he has been a puppeteer, an actor, an artistic director, a set and costume designer, an illustrator, and lately an acclaimed sand-painting performer.
His mesmerising sand-painting and storytelling skills have featured in many multimedia performances. They include Fool La La, by his company, set up with his Canadian wife, Maggie Blue O'Hara; Brothers of War for the government's 2012 International Arts Carnival, and a music video for pop star Eason Chan. His current sand-painting exhibition Oasis! Revives!, running at Hollywood Plaza this Easter, supports environmental protection.
Hoi Chiu stumbled upon sand painting online and was hooked. "I like the way the art form allows lots of room and space for me to interact with it. I can immediately see the different things I can draw," he says.
Within a week of his internet discovery, Hoi Chiu went to a local beach to get some sand and began experimenting with drawings. After his first attempt to make a rose and a chicken, he went on to create many beautiful scenes and stories.
These included a mother and child taking care of each other through time, two wartime soldiers battling with the meaning of peace and friendship and, in the current exhibition, a whale trying to survive in a desert.
This work conveys the perils of global warming and the urgent need for us to do something.
Hoi Chiu's sand-painting performances have attracted considerable attention, taking him to the mainland, Singapore, South Korea, Taiwan and England.
"When I paint in sand, I feel like I am performing; I'm like a dancer dancing on a stage in my own world. It's liberating," he says.
The artist's experience taught him that if he sets out to do something, he can achieve it. And that it is OK to choose a different path in the process, as long as the goal is achieved. That is the mindset teenagers should adopt, he says.
"When I was young, I knew I wanted to draw. But I didn't follow the traditional path like getting a university degree to become an artist," he says. "But I kept my passion and tried everything I could, and I have achieved my goal.
"What you need is a fire in you and the courage to follow your passion. It helps to stay curious about everything around you."
Oasis! Revives! runs until April 7 at Hollywood Plaza, Diamond Hill. Watch Hoi Chiu's free live performance at 3pm on April 6. Spend HK$500 in the plaza and you will get a chance to attend one of his 20 workshops.