Portable speakers have larger cavities than phones, so water sits in places that take longer to clear.
Small drivers, different rules
Earbuds and portable speakers use much smaller drivers than a phone, so they clear faster but are also easier to overdrive.
Keep the volume sensible
Play the cleaning tone at a moderate level and hold the earbud away from your ear. The goal is diaphragm movement, not loudness.
Try the free speaker cleaning tool and measure the result before and after.