Best if you can remove as much of the old silicn as you can, but leave the silicon in between the glass joins. if the silicon in there has come loose you are best to remove the whole piece of glass and reglue it all. You can get away with not doing this but it is the best option.
You need to get the right type of silicon, there are two, acidic cure is the one you are looking for, often they have 'for aquarium use' written on them. Also make sure you get one that doesn't have mold inhibiter in it. The correct silicon has a very strong smell, and must be used in a well ventlated area.