Human image synthesis

• Late 2017^[16] and early 2018 saw the surfacing of the deepfakes controversy where porn videos were doctored using deep machine learning so that the face of the actress was replaced by the software's opinion of what another persons face would look like in the same pose and lighting.
• In 2018
  Vicon's motion capture system. The demonstration ran in near real time at 60 frames per second in the Unreal Engine 4.^[17]
• In 2018 at the World Internet Conference in Wuzhen the Xinhua News Agency presented two digital look-alikes made to the resemblance of its real news anchors Qiu Hao (Chinese language)^[18] and Zhang Zhao (English language). The digital look-alikes were made in conjunction with Sogou.^[19] Neither the speech synthesis used nor the gesturing of the digital look-alike anchors were good enough to deceive the watcher to mistake them for real humans imaged with a TV camera.
• In September 2018 Google added "involuntary synthetic pornographic imagery" to its ban list, allowing anyone to request the search engine block results that falsely depict them as "nude or in a sexually explicit situation."^[20]

• In February 2019 Nvidia open sources StyleGAN, a novel generative adversarial network.^[21] Right after this Phillip Wang made the website ThisPersonDoesNotExist.com with StyleGAN to demonstrate that unlimited amounts of often photo-realistic looking facial portraits of no-one can be made automatically using a GAN.^[22] Nvidia's StyleGAN was presented in a not yet peer reviewed paper in late 2018.^[22]

• At the June 2019
  MIT CSAIL
  presented a system titled "Speech2Face: Learning the Face Behind a Voice" that synthesizes likely faces based on just a recording of a voice. It was trained with massive amounts of video of people speaking.
• Since 1 July 2019
  Senate of Virginia by Senator Adam Ebbin
  .

• Since 1 September 2019
  elections a 30-day protection period to the elections during which making and distributing digital look-alikes or synthetic fakes of the candidates is an offense. The law text defines the subject of the law as "a video, created with the intent to deceive, that appears to depict a real person performing an action that did not occur in reality"^[25]
• In September 2019
  public broadcasting company, aired a result of experimental journalism, a deepfake of the President in office Sauli Niinistö
  in its main news broadcast for the purpose of highlighting the advancing disinformation technology and problems that arise from it.
• 1 January 2020[26] California the state law AB-602 came into effect banning the manufacturing and distribution of synthetic pornography without the consent of the people depicted. AB-602 provides victims of synthetic pornography with injunctive relief and poses legal threats of statutory and punitive damages on criminals making or distributing synthetic pornography without consent. The bill AB-602 was signed into law by California Governor Gavin Newsom on 3 October 2019 and was authored by California State Assembly member Marc Berman.^[27]
• 1 January 2020, Chinese law requiring that synthetically faked footage should bear a clear notice about its fakeness came into effect. Failure to comply could be considered a crime the Cyberspace Administration of China stated on its website. China announced this new law in November 2019.^[28] The Chinese government seems to be reserving the right to prosecute both users and online video platforms failing to abide by the rules.^[29]12 November [deepfake]

Key breakthrough to photorealism: reflectance capture

In 1999 Paul Debevec et al. of USC did the first known reflectance capture over the human face with their extremely simple light stage. They presented their method and results in SIGGRAPH 2000.^[4]

The scientific breakthrough required finding the subsurface light component (the simulation models are glowing from within slightly) which can be found using knowledge that light that is reflected from the oil-to-air layer retains its polarization and the subsurface light loses its polarization. So equipped only with a movable light source, movable video camera, 2 polarizers and a computer program doing extremely simple math and the last piece required to reach photorealism was acquired.^[4]

For a believable result both light

• The 3D
  RGB XYZ scanner such as Arius3d or Cyberware (textures from photos, not pure RGB XYZ scanner), stereophotogrammetrically from synchronized photos or even from enough repeated non-simultaneous photos. Digital sculpting
  can be used to make up models of the body parts for which data cannot be acquired e.g. parts of the body covered by clothing.
• For believable results also the reflectance field must be captured or an approximation must be picked from the libraries to form a 7D reflectance model of the target.

Synthesis

The whole process of making digital look-alikes i.e. characters so lifelike and realistic that they can be passed off as pictures of humans is a very complex task as it requires photorealistically

Synthesis with an actor and suitable algorithms is applied using powerful computers. The actor's part in the synthesis is to take care of mimicking human expressions in still picture synthesizing and also human movement in motion picture synthesizing. Algorithms are needed to simulate laws of physics and physiology and to map the models and their appearance, movements and interaction accordingly.

Often both physics/physiology based (i.e. skeletal animation) and image-based modeling and rendering are employed in the synthesis part. Hybrid models employing both approaches have shown best results in realism and ease-of-use. Morph target animation reduces the workload by giving higher level control, where different facial expressions are defined as deformations of the model, which facial allows expressions to be tuned intuitively. Morph target animation can then morph the model between different defined facial expressions or body poses without much need for human intervention.

Using displacement mapping plays an important part in getting a realistic result with fine detail of skin such as pores and wrinkles as small as 100 µm.

Machine learning approach

In the late 2010s,

Similarly, since 2018, deepfake technology has allowed GANs to swap faces between actors; combined with the ability to fake voices, GANs can thus generate fake videos that seem convincing.^[33]

Applications

Main applications fall within the domains of

At the 2018 Conference on Neural Information Processing Systems (NeurIPS) researchers from Google presented the work 'Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis', which transfers learning from speaker verification to achieve text-to-speech synthesis, that can be made to sound almost like anybody from a speech sample of only 5 seconds (listen).^[39]

Sourcing images for AI training raises a question of privacy as people who are used for training didn't consent.^[40]

• Motion-capture acting
• Internet manipulation
• Media synthesis
• Propaganda techniques
• 3D data acquisition and object reconstruction
• 3D reconstruction from multiple images
• 3D pose estimation in general and articulated body pose estimation especially to do with capturing human likeness.
• 4D reconstruction
• Finger tracking
• Gesture recognition
• StyleGAN

References