Joe Greene had an outstanding college football career at North Texas State University (now the University of North Texas) (1966–68), leading the team to a 23–5–1 record during his three seasons. In his 29 games at defensive tackle, North Texas State held the opposition to 2,507 yards gained on 1,276 rushes, a per carry average of less than two yards per attempt. His collegiate coach, Rod Rust, said of Greene, “There are two factors behind Joe’s success. First, he has the ability to make the big defensive play and turn the tempo of a game around. Second, he has the speed to be an excellent pursuit player.” A pro scout said, “He’s tough and mean and comes to hit people. He has good killer instincts. He’s mobile and hostile.”

He got his nickname when the Pittsburgh fan base mistakenly assumed that the North Texas team nickname of “Mean Green” was Joe Greene’s nickname; however, it was actually Coach Rust’s wife who wanted to give a nickname to the team’s outstanding defense who laid down the description which stuck in two instances. Since green is the school’s main color, she gave the defense the name “Mean Green”.

In 1984, he was enshrined in the College Football Hall of Fame. In 2006, Greene was voted to the East-West Shrine Game Hall of Fame.