You are talking about three pretty different processes there. The rock tumbler takes only weeks because, as you said, it has the right abrasive. Also, the motion is different than would be experienced by a rock or sea glass.

Sea glass comes, unsurprisingly, from the sea. It's brought in by the surf from the ocean, where it is polished by particles suspended in the water.

The stones don't come from the ocean. They come from mountains, where active erosion is happening. They tumble, very very slowly, down streams and rivers, where they are polished not so much by the suspended particles in the water, but more by colliding with other small stones. They do this in steps, caused by floods, mudslides, or other disturbances. Over millions of years, river beds can cement into a rock grading from a conglomerate of rounded rocks towards the bottom and a breccia of angular rocks near the top. While mk's estimate of millions of years would result in these new sedimentary layers, decades is way off too. Depending on the origin, thousands to tens of thousands of years would be about right.