Cut a piece of plastic window screening into a big enough piece to amply cover your pot's drainage hole or holes. This will keep your potting soil in and let the water out. If you don't have window screening, use newspaper, paper towel, or a coffee filter to cover the hole. If your pot is big enough, you can also use a product called, Better Than Rocks,
which will cover the drainage hole and can help aerate your succulent plants’ roots.