After the Great Hyperspace War, Naga Sadow hung out on Yavin 4 and had his followers build big temples. A few hundred years later, Freedon Nadd showed up and trained with him before killing him. A few hundred years later, Exar Kun showed up and did Sith stuff there too. After that, the Jedi realized they should probably keep an eye on Yavin 4, since it seemed to be one of the leading sources of Sith Lords. The Rebels set up a base there and used it as a launching point for their assault on the Death Star. Luke Skywalker set up his Jedi Praxeum there, which he probably would not have done if he had known its history.
Lots of famous people spent time on Yavin 4. Not the Emperor though. He never went there.