There was no Garment of Many colors as some call it for Joseph. It was a garment of certain or sacred marks.
“The idea of a garment of many colors is an invention. If you look in your Bible every time it mentions many colors the word colors (even in the commentary) is in italics [the word in italics is actually many] because it is put in there by modern editors. It’s found in no ancient source. It’s not a garment of many colors at all. A garment of certain marks is the term that’s used here. We’ll see what it is in a second. ‘This garment had belonged to Abraham, and it already had a long history.’ It’s history was lengthy because it went back to the Garden of Eden, you see. That’s the garment; it’s the only one. Just as we treat the story of Cain and Abel, we trivialize this. We say, ‘Joseph was the youngest kid, so his father favored him and gave him a pretty garment of many colors.’ There is no mention in any ancient source of a garment of many colors. That’s an invention of modern editors trying to explain it. But here it was the garment he gave him. It was the garment of the priesthood. No wonder they were jealous of him, they being the elder brothers and he the younger in the patriarchal line coming down from Abraham. This garment had belonged to Abraham and had come down to Joseph instead of to the other brethren.” - Hugh Nibley, Teachings of the Book of Mormon, vol. 3, 51-52.