We introduce a new task of entailment-relation-aware paraphrase generation which aims at generating a paraphrase conforming to a given entailment relation (eg equivalent, forward entailing, or reverse entailing) with respect to the given input. We propose a reinforcement learning based weaklysupervised paraphrasing system, ERAP, that can be trained using existing paraphrase and natural language inference (NLI) corpora without an explicit task-specific corpus. A combination of automated and human evaluations show that ERAP generates paraphrases conforming to the specified entailment relation and are of good quality compared to baseline and uncontrolled paraphrasing systems. Using ERAP for augmenting training data for downstream textual entailment tasks improves performance over an uncontrolled paraphrasing system, and introduces fewer training artifacts, indicating the benefit of explicit control during paraphrasing.