Could steam help loosen the glues hold and make it easier to wipe off the residue? I'm thinking hold under a boiling kettle spout or pan - obviously be careful!
If it was non porous surfaces, orange terpenes / limonene / glue remover (with the former two) would do the job. Never tried it on fabric, so may remove the colour.
Again for a non porous surface, I've had good results by using the label glue side down to remove glue residue by pressing down and removing quickly and repeating many times, like you are dabbing it. Back in the day worked well for removing CD case label residue. Have a look on Wikipedia if you don't know what a CD is.
Baby oil (mineral oil) may work. Does wonders for your street cred as well.