DALL-E

Given an existing image, DALL·E 2 can produce "variations" of the image as individual outputs based on the original, as well as edit the image to modify or expand upon it. DALL·E 2's "inpainting" and "outpainting" use context from an image to fill in missing areas using a

For example, this can be used to insert a new subject into an image, or expand an image beyond its original borders.^[36] According to OpenAI, "Outpainting takes into account the image’s existing visual elements — including shadows, reflections, and textures — to maintain the context of the original image."^[37]

Technical limitations

DALL·E 2's language understanding has limits. It is sometimes unable to distinguish "A yellow book and a red vase" from "A red book and a yellow vase" or "A panda making latte art" from "Latte art of a panda".^[38] It generates images of "an astronaut riding a horse" when presented with the prompt "a horse riding an astronaut".^[39] It also fails to generate the correct images in a variety of circumstances. Requesting more than three objects, negation, numbers, and connected sentences may result in mistakes, and object features may appear on the wrong object.^[30] Additional limitations include handling text - which, even with legible lettering, almost invariably results in dream-like gibberish - and its limited capacity to address scientific information, such as astronomy or medical imagery.^[40]

Ethical concerns

DALL·E 2's reliance on public datasets influences its results and leads to algorithmic bias in some cases, such as generating higher numbers of men than women for requests that do not mention gender.^[41] DALL·E 2's training data was filtered to remove violent and sexual imagery, but this was found to increase bias in some cases such as reducing the frequency of women being generated.^[42] OpenAI hypothesize that this may be because women were more likely to be sexualized in training data which caused the filter to influence results.^[42] In September 2022, OpenAI confirmed to The Verge that DALL·E invisibly inserts phrases into user prompts to address bias in results; for instance, "black man" and "Asian woman" are inserted into prompts that do not specify gender or race.^[43]

A concern about DALL·E 2 and similar image generation models is that they could be used to propagate deepfakes and other forms of misinformation.^[44][45] As an attempt to mitigate this, the software rejects prompts involving public figures and uploads containing human faces.^[46] Prompts containing potentially objectionable content are blocked, and uploaded images are analyzed to detect offensive material.^[47] A disadvantage of prompt-based filtering is that it is easy to bypass using alternative phrases that result in a similar output. For example, the word "blood" is filtered, but "ketchup" and "red liquid" are not.^[48][47]

Another concern about DALL·E 2 and similar models is that they could cause technological unemployment for artists, photographers, and graphic designers due to their accuracy and popularity.^[49][50] DALL·E 3 is designed to block users from generating art in the style of currently-living artists.^[12]

Reception

Most coverage of DALL·E focuses on a small subset of "surreal"^[23] or "quirky"^[31] outputs. DALL-E's output for "an illustration of a baby daikon radish in a tutu walking a dog" was mentioned in pieces from Input,^[51] NBC,^[52] Nature,^[53] and other publications.^[5][54][55] Its output for "an armchair in the shape of an avocado" was also widely covered.^[23][32]

ExtremeTech stated "you can ask DALL·E for a picture of a phone or vacuum cleaner from a specified period of time, and it understands how those objects have changed".^[28] Engadget also noted its unusual capacity for "understanding how telephones and other objects change over time".^[29]

According to MIT Technology Review, one of OpenAI's objectives was to "give language models a better grasp of the everyday concepts that humans use to make sense of things".^[23]

Wall Street investors have had a positive reception of DALL·E 2, with some firms thinking it could represent a turning point for a future multi-trillion dollar industry. By mid-2019, OpenAI had already received over $1 billion in funding from Microsoft and Khosla Ventures,^[56][57][58] and in January 2023, following the launch of DALL·E 2 and ChatGPT, received an additional $10 billion in funding from Microsoft.^[59]

Japan's

After integrating DALL·E 3 into Bing Chat and ChatGPT, Microsoft and OpenAI faced criticism for excessive content filtering, with critics saying DALL·E had been "lobotomized."[63] The flagging of images generated by prompts such as "man breaks server rack with sledgehammer" was cited as evidence. Over the first days of its launch, filtering was reportedly increased to the point where images generated by some of Bing's own suggested prompts were being blocked.^[63][64] TechRadar argued that leaning too heavily on the side of caution could limit DALL·E's value as a creative tool.^[64]

Open-source implementations

Since OpenAI has not released source code for any of the three models, there have been several attempts to create open-source models offering similar capabilities.^[65][66] Released in 2022 on Hugging Face's Spaces platform, Craiyon (formerly DALL·E Mini until a name change was requested by OpenAI in June 2022) is an AI model based on the original DALL·E that was trained on unfiltered data from the Internet. It attracted substantial media attention in mid-2022, after its release due to its capacity for producing humorous imagery.^[67][68][69]

See also

• Artificial intelligence art
• DeepDream
• Imagen (Google Brain)
• Midjourney
• Stable Diffusion
• Prompt engineering

References

External links

Wikimedia Commons has media related to DALL-E.

• DALL-E 3 System Card
• DALL-E 3 paper by OpenAI
• DALL-E 2 website
• Craiyon website