Wow, those are strong feelings for a kid, especially if they stayed with you with such force. I have a memory of going with my school to outings at the Art Gallery of Ontario in Toronto where I grew up, and don't have those kinds of emotions about it. I do remember seeing a bunch of things there that I thought really didn't belong in an art gallery. There was a temporary exhibit that had things like a bolt of silk and a fan, and when the fan was on it sent waves across the bolt. The whole thing took up the better part of a large room. Also a lucite thing with two liquids in it, one dyed and the other colourless, and when you touched it, it rocked and you could send a blue wave rolling. There was also a metal table with iron shavings on it, and we could use magnets under the table to move the shavings around. More of a playground thing than an art gallery exhibit imho.
Fortunately there was a permanent collection of relatively good paintings, and a huge room of Henry Moore (whose work reminds me of giant dinosaur bones but is otherwise ho-hum). There is a hall there that is full of stuff that looks like it was put together in a kindergarten arts and crafts class. I'm sure the aquisitions department sunk major money into them too.
On the other hand, one of my favourite places to visit was the McMichael Collection in Kleinburg, just north of Toronto, full of the work of The Group of Seven and Innu and First Nations work. I love that place.