A source tells PEOPLE many still fear Sean “Diddy” Combs, who’s currently in federal custody amid sex trafficking charges and multiple civil suits

Sean “Diddy” Combs held a lot of power in the entertainment industry — and apparently still does, even from behind bars.

The music mogul, 54, is currently in federal custody in Brooklyn, New York, after getting denied bail twice as he faces federal sex trafficking charges and multiple civil suits, but a source tells PEOPLE many individuals continue to fear his potential actions.

“He’s so powerful. Everyone was afraid to cross him,” says the insider. “Even though he’s behind bars now, he’s still so powerful. We all knew it even back then.”

The source continues, “Everyone always chalked it up to being a creative genius when it came to his obsession with controlling things.”

Last month, Combs was indicted by federal authorities. He faces charges of sex trafficking, racketeering and transportation to engage in prostitution.

The “Last Night” rapper has pleaded not guilty to all the charges, though he could face up to life in prison if convicted.

“He’s incredibly smart,” adds the source of Combs. “He knows a lot of people, and he’s helped a lot of people — so a lot of people owe him. He knows that.”

The insider says the Bad Boy Records founder “is capable of a lot — much more than people think, even now.”

Reps for Combs did not immediately respond to PEOPLE’s request for comment.

In addition to the criminal charges, Combs is named a defendant in multiple lawsuits and could face up to 120 more after high-profile attorney Tony Buzbee claimed that many more potential accusers were planning to file complaints.

The indictment against the Grammy winner centers on alleged “freak offs,” described by authorities as elaborate sex performance with male sex workers and women who were forced or coerced into participating. Attorneys for Combs have denied the accusations.

This week, Diddy’s mom Janice Small Combs released a supportive statement. “My son is not the monster they have painted him to be, and he deserves the chance to tell his side,” she said. “I can only pray that I am alive to see him speak his truth and be vindicated.”

If you or someone you know has been sexually assaulted, please contact the National Sexual Assault Hotline at 1-800-656-HOPE (4673) or go to�rainn.org.