Perlmutter was born on 30 May 1930 in Mako, Hungary, to parents Rozalia and Ferencz, who ran a business that produced onions and vegetables. He was the sixth of nine children. He had four sisters named Blima, Faigel, Malka and Rosa, and four brothers named David, Mordechai, Moisha and Izak (later known as Ivor).
They were a Hasidic family.