Understanding the difference between these two tenses is rather difficult for ESL students. Look at these two sentences:
Both of them are incorrect. The Past Continuous has been mixed up with the Past Perfect Continous here.
The first sentence requires the Past Perfect Continuous because the action of studying began one time earlier.
The other sentence requires the Past Continuous because studying is just an interrupted action in progress.