According to the OED, a "yegg" is a burglar or safe-breaker, and its etymology is given as "Said to be the surname of a certain American burglar and safe-breaker."
The earliest citations given in the OED are
1903 N.Y. Evening Post 23 June (Cent. D. Supp.), The prompt breaking up of the organized gangs of professional beggars and yeggs.
1905 N.Y. Times 2 Jan. (Cent. Dict. Suppl.), Detective Sergeants..captured on the Bowery three men who, they say, are among the most successful ‘yeggmen’, or safe~crackers, in the business.
A related word is "yeggman":
1906 A. Stringer Wire Tappers 100 ‘Now, nitro-glycerine I object to, it's so abominably crude.’.. ‘And so odiously criminal!’ she interpolated. ‘Precisely. We're not exactly yeggmen yet.’
However, using a newspaper database, I found several earlier citations:
Minneapolis Star-Tribune, November 25 1899, p. 12: ...for Mr. Alness did not know until the detectives told him that "John Yegg & Co." is a bit of thieves' slang, "yegg" standing for one who is ready to beg or steal, stealing preferred as more honorable...
Bloomington, IL Pantagraph, December 28 1899, p. 3: The gang ... numbers at least twenty-five members, who call themselves Yeggmen.
New York Sun, January 22 1900, p. 2: Four crooks, whom the Pinkerton agents describe as hobo safe burglars belonging to a new fraternity known as "Yeggs," were arrested in Newark last Thursday night...
The uncommon words "yegg", "yeggs", "yeggman", "yeggmen" reached the zenith of their popularity around 1920-1935 and are only rarely used today. Here is a google ngram search: