You know it just depends on where you live. 300 miles south of me, cactus are frequently seen popping up in people's yards, but there are very few cactus that would grow in my yard.
I have never grown lillies, but I have grown gladiolas and irises which are similar. My only problem with them is the blooms are too short lived. They are so pretty.