
Today we release a KNN index for LAION-5B that allows for fast queries of the dataset with the open clip ViT-H-14 CLIP model. This means that users can search through billions of samples quickly and easily, making it a powerful tool for various applications such as image and text retrieval, data filtering and more. With this update also comes a brand new SLURM based inference backend for high-compute environments.

With this users can now:
  - Peer into the superset of data used to train the latest stable-diffusion-v2 models.
  - Easily filter through the dataset to create fine-tuning datasets and averaged embeddings for aesthetic gradients.
  - Quickly compute indices for new datasets with the SLURM backend.
  - Download the index & deploy locally

## The front-end

Our new H/14 index is now available for use on our clip-front demo at https://rom1504.github.io/clip-retrieval or https://knn.laion.ai. 
This new index allows for fast querying using both images and text, making it a valuable tool for a variety of use cases.
To start using the new index right away, simply visit the website and start experimenting with the available query options. 
The demo also allows you to easily download the resulting query as an [img2dataset](https://github.com/rom1504/img2dataset) compatible json file. 
This means that you can quickly create datasets for any use case, making it a valuable resource for creatives, data scientists and researchers alike.

## Using the KNN as an API

The KNN index can be accessed via the API, which allows you to perform nearest-neighbor searches in an easy and intuitive way. 
However, if you would prefer to use the provided knn index programmatically, you can! 
We have a notebook that you can use as a guide on how to do so. You can find the notebook [here](https://colab.research.google.com/github/rom1504/clip-retrieval/blob/master/notebook/clip-retrieval-getting-started.ipynb), it will walk you through the steps necessary to use the provided KNN index programmatically. 
This can be useful if you want to integrate the KNN index into your own application or if you want to automate the process of nearest-neighbor searches. 
Note that if you are looking to integrate the index into your own product, you should deploy it locally.

## Computing your own index

Creating your own index is a great way to interact with and visualize your data. 
With a custom CLIP embedding index you can quickly search for similar images, check what images your prompts summon, or check how unique a generated image may be to the training data.

The clip-retrieval repo offers the ability for users to compute their own indices for their own datasets. 
In an effort to support the creation of our new H-14 index, we added support for SLURM as a backend inference engine. 
This update adds a third option for computing indices meaning whether you are using SLURM, PySpark, or running it on your local machine, the process of creating a CLIP KNN index has never been easier. 
For specific usage please see the project’s [README](https://github.com/rom1504/clip-retrieval#clip-inference) for the inference API and the newest arguments available for creating your own index.

## Deploying Locally

Users who would like to do a lot of queries, or integrate the index into their own product, should download the index and metadata and deploy it locally to their own server. 
In order to do so we have uploaded the pre-computed indices to huggingface which can be found [here](https://huggingface.co/datasets/laion/laion5b-h14-index). 
For full documentation on the exact steps necessary to begin hosting the index yourself please visit the [clip-retrieval docs](https://github.com/rom1504/clip-retrieval/blob/main/docs/laion5B_h14_back.md).


Clip-Retrieval Update: H-14 Index & SLURM Inference


We have trained a new [ViT-G/14 CLIP](https://huggingface.co/laion/CLIP-ViT-bigG-14-laion2B-39B-b160k) model with [OpenCLIP](https://github.com/mlfoundations/open_clip) which achieves **80.1%** zero-shot accuracy on ImageNet and **74.9%** zero-shot image retrieval (Recall@5) on MS COCO. As of January 2023, this is the best open source CLIP model.

We believe this is interesting because:
* CLIP models are useful for zero-shot classification, retrieval, and for guidance/conditioning in generative models (OpenCLIP is used in Stable Diffusion V2 and currently the third most downloaded model on HuggingFace is a CLIP model). The approach underlying CLIP—self supervised learning on a large, heterogeneous dataset—has been shown to produce models which are more [robust](https://openai.com/blog/clip/) and [fair](https://ai.facebook.com/blog/seer-10b-better-fairer-computer-vision-through-self-supervised-learning-training-on-diverse-datasets/).
* Our new ViT-G model achieves the highest zero-shot ImageNet accuracy among models that use only naturally occurring image-text pairs as training data, and without explicit labels, pseudo-labels, or any pretrained image or text encoders.
* Our training run utilized multiple new techniques, including [FLIP](https://arxiv.org/abs/2212.00794) to accelerate training and [model soups](https://arxiv.org/abs/2203.05482) to surpass 80% accuracy.

## Main Results
The following results are with image resolution 224x224 except for CoCa which uses 576x576.

| Model name       | Batch size |               Samples seen              | Text Params | Image params | ImageNet top1 | Mscoco image retrieval at 5 | Flickr30k image retrieval at 5 |
|------------------|:----------:|:---------------------------------------:|:-----------:|:------------:|:-------------:|:---------------------------:|:------------------------------:|
| OpenAI CLIP L/14 | 32k        | 13B                                     | 123.65M     | 303.97M      | 75.4%         | 61.0%                         | 87.0%                            |
| OpenCLIP H/14    | 79k        | 32B (16 epochs of laion2B)              | 354.0M      | 632.08M      | 78.0%         | 73.4%                       | 94%                            |
| OpenCLIP G/14    | 160k       | 32B +unmasked fine-tune (details below) | 694.7M      | 1844.9M      | 80.1%*        | 74.9%                       | 94.9%                          |
| CoCa            | 66k        | 33B                                     | 1100M       | 1000M        | 86.3%**       | 74.2                        | 95.7                           |

\* When using [CuPL](https://arxiv.org/abs/2209.03320) prompts instead of the standard prompts from OpenAI, the zero-shot accuracy is 80.3%. When evaluating at 280x280 and changing resize to squash, Ross Wightman found the model achieves 80.4%.

** In addition to natural language supervision, [CoCa](https://arxiv.org/abs/2205.01917) uses synthetic captions constructed with the labels from the JFT-3B dataset. In addition to natural language supervision, CoCa uses synthetic captions constructed with the labels from the JFT-3B dataset. 973 of the 1,000 ImageNet classes have a corresponding class in JFT (e.g., see here sec C.7.2).

Also see the figure below (figure code by Ross) and our analysis of scaling trends for OpenCLIP model [here](https://arxiv.org/abs/2212.07143).

![](/images/blog/scaling_vit_giant.png)

## Released Checkpoints

We release the checkpoint through [OpenCLIP](https://github.com/mlfoundations/open_clip) and in the [HuggingFace hub](https://huggingface.co/laion/CLIP-ViT-bigG-14-laion2B-39B-b160k).

## Notes on scaling up

To scale up model size while reducing compute we used [Fast Language-Image Pre-training (FLIP)](https://arxiv.org/abs/2212.00794) with patch dropout 0.5. Similar to [Masked Autoencoders (MAE)](https://arxiv.org/abs/2111.06377), FLIP drops out patches during training. FLIP also requires a short “unmasked tuning” phase, which we discuss in training notes below. In addition to reducing Giga multiply–accumulate operations (GMACs) for each forward/backward pass, FLIP allowed us to use a larger per-GPU batch size. Without FLIP, gradient accumulation was necessary to maintain a large batch size. Keeping batch size and number of GPUs consistent (at 160k and 512, respectively) but switching to unmasked fine-tuning resulted in a drop from 46.9 to 20.4 samples per second per GPU. For reference, OpenCLIP H/14 with global batch size 79k across 824 GPUs without patch dropout trained at 42 samples/s/GPU.

To scale up the batch size to 160k, we used [gradient checkpointing](https://arxiv.org/abs/1604.06174v2) and 80GM VRAM A100s. For the unmasked tuning portion, we also used gradient accumulation (see our implementation for the contrastive objective [here](https://github.com/mlfoundations/open_clip/pull/267)). Finally, we used a 2x higher learning rate of 2e-3 compared to our experiments with batch size 80k. The combination of scaling up model, batch size, and learning rate resulted in training instability during the warmup phase. Accordingly, we increased warm-up to 13k steps, trained with layer scale, and used AdamW beta2 0.95. All runs used AMP bfloat16, after previously switching from float16 in prior experiments with L/14 and H/14.

## Training notes

### Phase 1: Patch dropout

For phase 1 we trained ViT-G with [patch dropout](https://arxiv.org/abs/2212.00794) 0.5 on LAION-2B for 32B samples seen. We used batch size 160k, learning rate 2e-3, and a cosine decay schedule. After this phase the model reached 79.07 zero-shot top1 accuracy on ImageNet.

Training was mainly done on 512 to 760 A100s depending on availability. When changing the number of GPUs, local batch size was also modified so that the global batch size remained at 160k. When using 512 GPUs we set local batch size to 313 and observed roughly 24k samples per second or 46.9 samples/s/GPU. When using 760 GPUs we set local batch size 211 and observed roughly 33k samples per second or 43.4 samples/s/GPU.

### Phase 2: Unmasked tuning + Model soups

For phase 2 we followed [FLIP](https://arxiv.org/abs/2212.00794) in conducting a short unmasked tuning phase. We fell short of 80% in our first unmasked fine-tuning phase, reaching only 79.43%. So we tried twice more with different settings (described below) to obtain 79.45% and 79.2%, respectively. Next, we followed [model soups](https://arxiv.org/abs/2203.05482) and averaged the weights of three checkpoints produced by these runs to achieve our final accuracy of 80.1%. [LIMoE](https://arxiv.org/abs/2206.02770) and [PaLI](https://ai.googleblog.com/2022/09/pali-scaling-language-image-learning-in.html) also used model soups for better pre-training.

For our first unmasked fine-tuning run we did not modify the learning rate schedule, but instead doubled the base LR and extended the number of iterations so that the run would proceed for an additional 2B samples seen. LR started at 3.8e-5. For the second run we used LR 5.5e-5 with a full cosine schedule (warmup for roughly 200M samples and a total of 4B samples). The third run had identical hyperparameters to the first but used the LAION-A subset of LAION-2B. LAION-A is a 900M subset of LAION-2B filtered with aesthetic V2 4.5+ and pHash deduplicated. Instead of waiting for the third run to complete we use the checkpoint after approximately 700M samples which, when “[souped](https://arxiv.org/abs/2203.05482)” with the final checkpoints from the two proceeding runs, already allowed us to surpass our goal of 80% accuracy. This indiviual checkpoint achieved 79.2%.

Unmasked fine-tuning was done on 512 A100 GPUs at a speed of roughly 10,450 samples/s or 20.4 samples/s/GPU.

The following plot shows the loss curve for phase 1.

![](/images/blog/loss_vit_giant.png)

## More results

Zero-shot accuracies at resolution 224x224 computed with [CLIP Benchmark](https://github.com/LAION-AI/CLIP_benchmark).

| Dataset         | OpenCLIP H/14 | OpenCLIP G/14 |
|-----------------|---------------|---------------|
| ImageNet        | 78.0         | 80.1         |
| ImageNet-V2     | 70.8         | 73.6         |
| ImageNet-R      | 89.3         | 92.1         |
| ImageNet-Sketch | 66.6         | 68.9         |
| ObjectNet       | 69.7         | 73.0         |
| ImageNet-A      | 59.2         | 69.3         |
| CIFAR-10        | 97.4         | 98.2         |
| CIFAR-100       | 84.7         | 87.5         |
| MNIST           | 72.9         | 71.6         |
| SVHN            | 56.1         | 62.5         |
| Caltech-101     | 85.0         | 86.4         |
| SUN397          | 75.2         | 74.5         |
| FGVC Aircraft   | 42.8         | 49.7         |
| Country211      | 30.0         | 33.8         |
| Cars            | 93.5         | 94.6         |

Here is a summary figure comparing G/14 and H/14 made with evals by Romain Beaumont.

![](/images/blog/summary_vit_giant.png)

## What’s Next?

In the future, we may fine-tune the model to enable multilingual capabilities, or fine-tune at higher resolution. Also, [FSDP is coming to OpenCLIP](https://github.com/mlfoundations/open_clip/pull/358) which will allow even larger models, as is [CoCa](https://github.com/mlfoundations/open_clip/pull/308) which will allow new openclip models to also be captioners. More contributions to OpenCLIP are always welcome!

## Contributions and acknowledgements

Thanks to:
* [Romain Beaumont](https://github.com/rom1504), [Ross Wightman](https://github.com/rwightman), [Mehdi Cherti](https://github.com/mehdidc), [Gabriel Ilharco](https://gabrielilharco.com/), and [Jenia Jitsev](https://github.com/JeniaJitsev) for providing extensive ideas, advice, engineering support, evaluating the model, and maintaining the openclip repository used for model training.
* [Christoph Schuhmann](https://github.com/christophschuhmann) for encouragement and support
* [Richard Vencu](https://github.com/rvencu) for cluster support
* [Phil Wang](https://github.com/lucidrains) and [Haoqi Fan](https://haoqifan.github.io/) for the implementation and discussion regarding patch dropout
* [Sho Yaida](https://www.shoyaida.com/), [Jong Wook Kim](https://jongwook.kim/), [Ari Morcos](http://www.arimorcos.com/) and [Saining Xie](https://www.sainingxie.com/) for helpful remarks regarding hyperparameters
* [Sarah Pratt](https://sarahpratt.github.io/) for implementing CuPL
* [Ludwig Schmidt](https://github.com/ludwigschmidt) and [Ali Farhadi](https://homes.cs.washington.edu/~ali/) for helpful discussions, and to the [RAIVN](https://raivn.cs.washington.edu/) and [EFML](https://github.com/mlfoundations/) labs at the University of Washington

And of course thanks to [Emad](https://twitter.com/EMostaque) and [Stability AI](https://stability.ai/) for providing the compute resources used for these experiments.


Reaching 80% zero-shot accuracy with OpenCLIP: ViT-G/14 trained on LAION-2B


Author: [Konstantinos Thoukydidis](https://dbzer0.com), [hlky](https://twitter.com/hlky__)

We are happy to announce that LAION will be assisted by the Stable Horde to provide aesthetic ratings for existing datasets and a completely new dataset of Stable Diffusion generations, which will also be rated by their community.

We wrote in the past about [LAION-Aesthetics](https://laion.ai/blog/laion-aesthetics/) and how we filtered LAION-5b using an [aesthetic predictor](https://github.com/christophschuhmann/improved-aesthetic-predictor). The predictor, a simple neural net that uses CLIP ViT-L/14 embeddings as input. hlky has retrained the aesthetic predictor using ViT-H-14, and the results are promising.

[hlky’s ViT-H aesthetic predictor](https://github.com/hlky/aesthetic-predictor) was trained on the same datasets as the original - AVA: A Large-Scale Database for Aesthetic Visual Analysis, Simulacra Aesthetic Captions, and LAION-logos. These datasets are limited, totalling around 400k image-rating pairs. This is where the Stable Horde comes in.

[The Stable Horde](https://stablehorde.net/) is a crowdsourced, distributed, [free and open sourced service](https://github.com/db0/AI-Horde) for generating Stable Diffusion images available to be integrated into everything. They have volunteered to direct their community to rate the existing LAION datasets for aesthetic rating of 1-10. As the Stable Horde can be used for free, to incentivize this, they are providing priority (AKA kudos) to anyone who is providing such ratings for their stable diffusion generations.

The integration has been made possible through an API hosted by Sygil.dev which can collect such ratings and be integrated directly into all Stable Horde clients. Already the first integrations have been released on clients like [ArtBot](https://tinybots.net/artbot/rate).

![](/public/images/blog/artbot-image-ratings.png)

Furthermore the Stable Horde has started gathering thousands of Stable Diffusion images and their parameters generated by their community on a voluntary basis, which will then be used to build a completely new AI dataset. These images will not only be incentivized to be rated aesthetically as well, but also to be rated pairwise. 

Stable Horde supports many models in addition to base Stable Diffusion models. At the time of writing this, nearly 80 finetunes are available. There are currently no datasets of generated images from Stable Diffusion finetunes, this will be a world first!

The way this works is that whenever an image set (meaning a number of images out of the same prompt) is generated, a follow-up call can be done to submit ratings for all, or some of them. If the set includes more than one image, optionally the best of the set can be selected, which will serve as a pairwise rating. If no “best” image has been selected but they have been aesthetically rated, the best image will be automatically selected among those ratings.

Through this collaboration, LAION will be provided with a dataset of millions of images rated in such a manner, along with their prompts, which can be made freely accessible to train further models with improved quality.


## What's next?

The Stable Horde plans to onboard more ratings into their dataset, such as tagging images for artifacts, watermarks, nudity and the like. The provided set will continue expanding so long as people volunteer their time and processing power. 

## Support this endeavour

If you have any questions or need support about the Stable Horde or the rating system, they have [a very active discord server you can join](https://discord.gg/3DxrhksKzn). If you want to support this initiative, you can help by either rating images yourself, or by [onboarding your own GPU as a horde worker](https://github.com/db0/AI-Horde-Worker) which will help the community generate more images and thus increase the size of the open dataset they provide.

Collaboration between LAION and the Stable Horde


Author: [Christoph Schuhmann](https://github.com/christophschuhmann), [Andreas Köpf](https://github.com/andreaskoepf) , [Theo Coombes](https://github.com/TheoCoombes), [Richard Vencu](https://github.com/rvencu/), [Benjamin Trom](https://github.com/limiteinductive) , [Romain Beaumont](https://github.com/rom1504) 

**We present LAION-COCO, the world’s largest dataset of 600M generated high-quality captions for publicly available web-images**

Laion5B has five billion natural captions. They provide a lot of information, but could synthetic captions complement them ?

To answer this question, we use a combination of existing, publicly available models to produce high quality captions for images in the style of [MS COCO](https://paperswithcode.com/dataset/coco).

We captioned 600M images from the english subset of Laion-5B with an ensemble of [BLIP](https://github.com/salesforce/BLIP) L/14 and 2 CLIP versions (L/14 and RN50x64).  

With this post we release them openly today.

This will make it possible to investigate the value of generated captions to train models. We’re curious on how these synthetic captions could impact models trained on them!


## Download it

The 600M samples are provided in parquet files. Columns include the original caption, the url, the top caption and a list of alternative captions with lower CLIP-similarity scores.

[https://huggingface.co/datasets/laion/laion-coco](https://huggingface.co/datasets/laion/laion-coco) 


## Samples



<img src="/images/blog/ring.png" style="height:250px; width: auto;"/>


**Original:** LGSY 925 Sterling Silver Double Heart Rings Infinity Love Thin Rings Wedding Engagement Promise Engraved Love Rings for Women for Dainty Gift

**Generated:** An open ring with two hearts on it.



<img src="/images/blog/boot.png" style="height:250px; width: auto;"/>
         

**Original:** Female Thick with Pointy Head High Heel Chelsea Ankle Boots

**Generated:** Red leather ankle boots with gold buckles.



<img src="/images/blog/laion_coco_beach.png" style="height:250px; width: auto;"/>


**Original:** a group of people on horses on a beach

**Generated:** Several people riding horses down the beach on a cloudy day.




<img src="/images/blog/laion_coco_tags.png" style="height:250px; width: auto;"/>


**Original:** a wall with a bunch of graffiti on it

**Generated:** The parking meter is near a graffiti covered building.

                                                                                                                                        



<img src="/images/blog/sheeple.png" style="height:250px; width: auto;"/>


**Original:** sheeple family

**Generated:** A cartoon drawing of sheep watching TV with their babies.


## More samples of images with their generated captions can be found here: 

(no cherry picking)

[http://captions.christoph-schuhmann.de/eval_laion/eval.html](http://captions.christoph-schuhmann.de/eval_laion/eval.html) 


## Method

The method we used to generate these captions was to

1. We use Blip L/14 to generate 40 captions
2. Rank them using openai Clip Open AI L/14 ; selected the best 5 captions
3. Rank using Open AI RN50x64 Clip model to select the best one
4. Use a small, fine-tuned T0 model to roughly repair grammar and punctuation of the texts

The hyperparameters were chosen through a [grid search](https://wandb.ai/andreaskoepf/blip_coco_val_sample_sweep_bayes_02/sweeps/1bsha6b0) ([settings](https://github.com/andreaskoepf/CLIP-Image-Captioning/blob/blip_test/blip_coco_val_sample_sweep_bayes_02.yaml)) by Andreas Köpf to best match the style ( ROUGE scores ) of MS COCO texts.

[laion_idle_cap](https://github.com/andreaskoepf/laion_idle_cap/tree/main/docker) is the script that was used for this processing.


## Evaluation

We evaluated these generated captions by asking human evaluators to guess whether a caption is coming from a human or an AI model. We also asked them to rate the quality on a scale from 0(bad) to 5 (good). 

In a first round we presented the evaluators each 200 samples, that contained 100 AI generated and 100 human written MS COCO captions.

### Observations

 

<img src="/images/blog/eval_laion_coco.png" style="width:400px" />


GT: Y-Axis

Annotation: X-Axis

Mean rating & standard deviation of samples, that were written by a human:

Mean: 3.98

Stdev: 0.99

Mean rating & standard deviation of samples, that were written by an AI 

Mean: 3.89

Stdev: 1.12

Mean rating & standard deviation of samples, where the annotator believed they were written by a human:

Mean: 4.44

Stdev: 0.61

Mean rating & standard deviation of samples, where the annotator believed they were generated by an AI 

Mean: 3.50

Stdev: 1.15

### Interpretation

It is very interesting that the mean scores of the samples generated by humans and generated by the model are very similar. We also notice that the standard deviation of the generated captions is a little bit higher.

We hypothesize that most in most cases the quality of the generated captions is perceived as as good as the quality of the human written captions.

But sometimes the captioning model obviously fails and the quality of the results is pretty low because the model doesn't relevant understand concepts about what is going on in the picture, because it's knowledge is not grounded in a sufficiently sophisticated world model.

### Failure cases



<img src="/images/blog/laion_coco_umbrella.png" style="height:250px; width: auto;"/>


_“Two people posing for the camera in their wedding attire, one with an umbrella over his head and another with long red hair.”_



<img src="/images/blog/laion_coco_man.png" style="height:250px; width: auto;"/>


_“An older man having a heart attack, with his hand on the chest.”_

When we remove all samples from the evaluations that have ratings of either 0 or 1, we Observe that the mean ratings and standard deviations move closer together. 

### Scores without ratings of 0 and 1

Mean rating & standard deviation of samples, that were written by a human:

Mean: 4.07

Stdev: 0.81

Mean rating & standard deviation of samples, that were written by an AI 

Mean: 4.02

Stdev: 0.94

The mean ratings of the generated captions are still a little bit lower and the standard deviation is still a little bit higher, but the trend is pretty clear. By removing samples with rating 2, the gap between the qualities would probably decrease even further. 

Presentation only generated captions:

In a next step, we presented the human evaluators 400 captions that were only generated by the model (no human written captions in between):

Mean rating of all samples 

3.81

Standard deviation of all samples 

0.94

% rated as human

47.5

% rated as AI

52.5

We observe that the human evaluators thought in 47.5% of all cases, that the captions were written by a human. This makes us confident that our captains are on average pretty good. When we told the evaluators later that all captions were generated by the model they told us that it was very hard for them to judge whether a caption was written by a model or a human, and that it only was easy for them in obvious failure cases.

### Conclusions

We conclude that Our ensemble of BLIP and CLIP is already pretty good and capable of generating captions with a quality that is on average pretty close to the human written captions of MS Coco. 

It would be very interesting for future work to let people rate our generated captions at larger scale and then filter out the samples with low rating values. These results could be used to train models to rate the quality of captions and to predict whether a caption looks like a generated or a human written caption.

And even without further automated filtering, an ensemble of our captions and human evaluators would be a pretty good workflow to curate high quality captions at much lower costs than if we would ask humans to write them from scratch.


## Credit assignments



* [Christoph Schuhmann](https://github.com/christophschuhmann) lead the project, implemented a first version of the code, ran most of the generations & conducted the human evaluations
* [Andreas Köpf](https://github.com/andreaskoepf) conducted the hyperparameter search & wrote the code to execute BLIP + CLIP filtering at scale
* [Theo Coombes](https://github.com/TheoCoombes) managed the server that coordinated which GPU worker got which part of LAION to work on
* [Romain Beaumont](https://github.com/rom1504) packaged the .json into parquet files, sent to HF and wrote the first draft of this post
* [Richard Vencu](https://github.com/rvencu/) provided the infra structure to use the idle compute for this project
* [Benjamin Trom](https://github.com/limiteinductive) wrote code that help us to convert the .json files to parquet

We thank [stability.ai](https://stability.ai/) for providing the compute used to generate the captions in the dataset.


Laion coco: 600M synthetic captions from Laion2B-en


Author: [Marianna Nezhurina](https://github.com/marianna13) [Romain Beaumont](https://github.com/rom1504/) [Richard Vencu](https://github.com/rvencu) [Christoph Schuhmann](https://github.com/christophschuhmann)  

Laion5B dataset was automatically collected from a section of the human web (common crawl). Can models generate different and interesting data compared to what humans write?

That’s a question we are interested in investigating. To let the community study it, we translated 3B samples of Laion5B from many languages into English.

We released 3 billions captions for the multilingual part of Laion5B. This makes it possible to use the whole Laion5B dataset to train English models. This also enables training models using these aligned pairs such as [Multilingual-CLIP](https://github.com/FreddeFrallan/Multilingual-CLIP).

We’re curious what you will do using it!


## Downloading it

The dataset is available in huggingface as parquet files containing the caption, translated caption and urls.

[laion1B-nolang-joined-translated-to-en](https://huggingface.co/datasets/laion/laion1B-nolang-joined-translated-to-en) 

[laion2B-multi-joined-translated-to-en](https://huggingface.co/datasets/laion/laion2B-multi-joined-translated-to-en)  


## Processing

Every caption of the original dataset was translated with Facebook’s [M2M100 1.2B model](https://huggingface.co/facebook/m2m100_1.2B) using the following [script](https://github.com/marianna13/translate_dataset/blob/main/translate_data.py). All other fields remain the same as in the original [LAION2B Multi Joined](https://huggingface.co/datasets/laion/laion2B-multi-joined). To make translation possible the original dataset was split into parts with 50k samples in each and every such small part was translated in parallel on GPU nodes and saved in a separate parquet file. The speed of translation depends on the number of nodes and GPUs. The processing was done with 20 nodes with 8 GPUs in each and the speed of translation (including preprocessing and data loading) is 34 samples/per GPU/per second. Then, all translated parquets were [merged together using Spark](https://github.com/marianna13/translate_dataset/blob/main/join_additional.py) and saved as 128 parquet files. The resulting dataset was [joined with the aesthetics scores](https://github.com/marianna13/translate_dataset/blob/main/join_aesthetics.py). 


## Dataset columns



* TEXT (the original text of caption)
* LANGUAGE (language of the original TEXT)
* ENG TEXT (translation in English of the original TEXT)
* URL (URL of the image)
* WIDTH (width of the image)
* HEIGHT (height of the image)
* Hash (hash of the URL and TEXT)
* Pwatermark (probability of being a watermarked image, computed using our [watermark detector](https://github.com/LAION-AI/LAION-5B-WatermarkDetection))
* Punsafe (probability of being an unsafe image, computed using our [clip based detector](https://github.com/LAION-AI/CLIP-based-NSFW-Detector))
* Similarity (cosine between text and image ViT-B/32 embeddings, clip for en, mclip for multi and nolang)
* Prediction (aesthetics score)


## Samples from the translated dataset:


![laion-2B-translated-samples](/images/blog/laion-translated-samples.png "laion-2B-translated-samples")



## Laion2B-multi-translated


### Dataset stats

_Note. Dataset stats were computed using [this](https://github.com/marianna13/translate_dataset/blob/main/get_dataset_stats.py) script._

Number of uniques 2266M (2266193302)                                            

Number with WIDTH >= 0 and WIDTH &lt;= 128 160M (160260569)                        

Number with WIDTH >= 128 and WIDTH &lt;= 256 734M (734166164)                      

Number with WIDTH >= 256 and WIDTH &lt;= 512 849M (849569769)                      

Number with WIDTH >= 512 and WIDTH &lt;= 1024 457M (457572747)                     

Number with WIDTH >= 1024 86M (86750813)                                        

Number with HEIGHT >= 0 and HEIGHT &lt;= 128 103M (103514467)                      

Number with HEIGHT >= 128 and HEIGHT &lt;= 256 614M (614490681)                    

Number with HEIGHT >= 256 and HEIGHT &lt;= 512 753M (753540968)                    

Number with HEIGHT >= 512 and HEIGHT &lt;= 1024 686M (686553437)                   

Number with HEIGHT >= 1024 153M (153139456)                                     

Number with lenengtext >= 0 and lenengtext &lt;= 25 506M (506238532)               

Number with lenengtext >= 25 and lenengtext &lt;= 50 849M (849160165)              

Number with lenengtext >= 50 and lenengtext &lt;= 100 840M (840635023)             

Number with lenengtext >= 100 and lenengtext &lt;= 150 136M (136709119)            

Number with lenengtext >= 150 5M (5148507)


### Similarities between text and images

10000 images and captions were sampled from the dataset, [CLIP embeddings were computed](https://github.com/marianna13/translate_dataset/blob/main/get_clip_embs_similarities.py) (for original texts embeddings were computed using Multilingual CLIP). Then dot products between image and text embeddings were computed (for both original and translated dataset) to get similarities between texts and images. Here’s the distribution of average similarities for two datasets:

Similarity for original dataset:

10% quantile -  0.2552971839904785

20% quantile -  0.2633610963821411

30% quantile -  0.2694466710090637

40% quantile -  0.2750270366668701

50% quantile -  0.28088638186454773

60% quantile -  0.28750720620155334

70% quantile -  0.2950591444969177

80% quantile -  0.3049575388431549

90% quantile -  0.32077282667160034

Similarity for translated dataset:

10% quantile  -  0.23388671875

20% quantile  -  0.25390625

30% quantile  -  0.265869140625

40% quantile  -  0.2763671875

50% quantile  -  0.2861328125

60% quantile  -  0.29638671875

70% quantile  -  0.306884765625

80% quantile  -  0.31982421875

90% quantile  -  0.338134765625



![laion-2B-translated](/images/blog/laion-2B-translated.png "laion-2B-translated")



## Laion1B-nolang-translated

[LAION1B Nolang Joined](https://huggingface.co/datasets/laion/laion1B-nolang-joined) dataset was also translated with a similar [script](https://github.com/marianna13/translate_dataset/tree/main) and in the same way as Multi. This dataset doesn’t have the language column so the model also had to determine language. All other columns in the Nolang dataset are the same as in Multi. And just like translated [LAION2B Multi Joined](https://huggingface.co/datasets/laion/laion2B-multi-joined), LAION1B Nolang was also joined with corresponding [aesthetics scores](https://github.com/marianna13/translate_dataset/blob/main/join_aesthetics.py).


### Nolang dataset Stats

Number of uniques 1260M (1260048307)                                            

Number with WIDTH >= 0 and WIDTH &lt;= 128 90M (90701133)                          

Number with WIDTH >= 128 and WIDTH &lt;= 256 409M (409575445)                      

Number with WIDTH >= 256 and WIDTH &lt;= 512 475M (475885337)                      

Number with WIDTH >= 512 and WIDTH &lt;= 1024 239M (239035772)                     

Number with WIDTH >= 1024 59M (59942110)                                        

Number with HEIGHT >= 0 and HEIGHT &lt;= 128 59M (59814914)                        

Number with HEIGHT >= 128 and HEIGHT &lt;= 256 370M (370913206)                    

Number with HEIGHT >= 256 and HEIGHT &lt;= 512 451M (451897702)                    

Number with HEIGHT >= 512 and HEIGHT &lt;= 1024 316M (316723245)                   

Number with HEIGHT >= 1024 87M (87671543)                                       

Number with lenengtext >= 0 and lenengtext &lt;= 25 312M (312548202)               

Number with lenengtext >= 25 and lenengtext &lt;= 50 555M (555971621)              

Number with lenengtext >= 50 and lenengtext &lt;= 100 413M (413430230)             

Number with lenengtext >= 100 and lenengtext &lt;= 150 20M (20446701)              

Number with lenengtext >= 150 0M (138974)  


### Similarities between text and images for Nolang

10000 images and captions were randomly sampled from the dataset, [CLIP embeddings were computed](https://github.com/marianna13/translate_dataset/blob/main/get_clip_embs_similarities.py) (for original texts embeddings were computed using Multilingual CLIP). Then dot products between image and text embeddings were computed (for both original and translated dataset) to get similarities between texts and images. Here’s the distribution of average similarities for two datasets:

Similarity for original dataset:

10% quantile  -  0.258196085691452

20% quantile  -  0.266357421875

30% quantile  -  0.2728866934776306

40% quantile  -  0.27902457118034363

50% quantile  -  0.28590404987335205

60% quantile  -  0.29329144954681396

70% quantile  -  0.3023602366447449

80% quantile  -  0.31363412737846375

90% quantile  -  0.3313804566860199

Similarity for translated dataset:

10% quantile  -  0.2406005859375

20% quantile  -  0.2607421875

30% quantile  -  0.27490234375

40% quantile  -  0.2861328125

50% quantile  -  0.296142578125

60% quantile  -  0.306396484375

70% quantile  -  0.317626953125

80% quantile  -  0.33203125

90% quantile  -  0.353271484375


![laion-1B-translated](/images/blog/laion-1B-translated.png "laion-1B-translated")


## Credit



* [Marianna Nezhurina](https://github.com/marianna13) translated the samples, packaged them, computed stats and wrote most of this post
* [Romain Beaumont](https://github.com/rom1504/) helped out on packaging and scaling
* [Richard Vencu](https://github.com/rvencu) set up all the infra that made using idle compute possible
* [Christoph Schuhmann](https://github.com/christophschuhmann) suggested the project and guided the work to completion

We thank [https://stability.ai/](https://stability.ai/) for providing the compute for this massive translation. This was a great use of pre-emptible jobs to fill any idle compute available!

Laion translated: 3B captions translated to English from laion5B


We trained three large CLIP models with [OpenCLIP](https://github.com/mlfoundations/open_clip): ViT-L/14, ViT-H/14 and ViT-g/14 (ViT-g/14 was trained only for about a third the epochs compared to the rest). The H/14 model achieves **78.0%** zero shot top-1 accuracy on ImageNet and **73.4%** on zero-shot image retrieval at Recall@5 on MS COCO. As of September 2022, this is the best open source CLIP model.

CLIP makes it possible to compute representations of images and texts to measure how **similar** they are. It can be used for 



* Zero shot classification: compare an image with the text of the class to know which class is most similar (e.g., ImageNet classification)
* Retrieval: compare an image or a text to billions of text or images to find the most similar (e.g. as in [clip-retrieval](https://rom1504.github.io/clip-retrieval/) )
* Generation
    * CLIP guidance: decide a text you want to generate, then use an image generator model, and use the CLIP distance between what’s generated and the text to generate a better image (e.g., VQGAN + CLIP)
    * CLIP conditioning: use a clip text embedding as input of a generator to make it generate this text directly (e.g., stable diffusion)

CLIP models are trained in a self supervised fashion on hundreds of millions or billions of (image, text) pairs.

With LAION, we produced the LAION-5B dataset that contains 5.8 billions of closely related image and text pairs.

The CLIP model ViT B/32, released by OpenAI, was initially used to filter this dataset out of common crawl.

Producing the best open source CLIP model out of this data set completes the open source replication of the [excellent](https://openai.com/blog/clip/) CLIP paper that OpenAI released one year ago.


## Results

We replicated the results from openai CLIP in models of different sizes, then trained bigger models. The full evaluation suite on 39 datasets ([vtab+](https://github.com/LAION-AI/CLIP_benchmark)) are available in this [results notebook](https://github.com/LAION-AI/CLIP_benchmark/blob/main/benchmark/results.ipynb) and show consistent improvements over all datasets.

The larger models we release today are L/14, H/14 and g/14.

L/14 was trained on JUWELS Booster supercomputer by [Ross wightman](https://github.com/rwightman). H/14 and g/14 were trained on stability cluster by [Romain Beaumont](https://github.com/rom1504) . While L/14 and H/14 were trained using 34B samples from LAION-2b, g/14 used a substantially smaller sample scale for training, seeing only 12B samples (see tables for more details).


#### 32B samples seen


| **Model name**                                                                   | **Batch size**                   | **Samples seen**           | **Text Params** | **Image params** | **Imagenet top1** | **Mscoco image retrieval at 5** | **Flickr30k image retrieval at 5** |
|----------------------------------------------------------------------------------|----------------------------------|----------------------------|-----------------|------------------|-------------------|---------------------------------|------------------------------------|
| [B/32](https://wandb.ai/rom1504/eval_openclip/reports/B-32-2B--VmlldzoyNDkwNDMy) | 79k                              | 34B (16 epochs of laion2B) | 63.43M          | 87.85M           | 66.6%             | 65.4%                           | 88.4%                              |
| L/14                                                                             | 79k for 14B samples, 86K for 18B | 32B                        | 123.65M         | 303.97M          | 75.3%             | 71.1%                           | 92.9%                              |
| [H/14](https://wandb.ai/rom1504/eval_openclip/reports/H-14--VmlldzoyNDAxODQ3)    | 79k                              | 32B (16 epochs of laion2B) | 354.03M         | 632.08M          | 78.0%             | 73.4%                           | 94%                                |



#### 12B samples seen


| **Model name**                                                                     | **Batch size**                             | **Samples seen**                        | **Text Params** | **Image params** | **Imagenet top1** | **Mscoco image retrieval at 5** | **Flickr30k image retrieval at 5** |
|------------------------------------------------------------------------------------|--------------------------------------------|-----------------------------------------|-----------------|------------------|-------------------|---------------------------------|------------------------------------|
| B/32                                                                               | 32k                                        | 12B (32 epochs of laion400m)            | 63.43M          | 87.85M           | 62.9%             | 60.8%                           | 85.5%                              |
| B/16                                                                               | 32k                                        | 12B (32 epochs of laion400m)            | 91.16M          | 86.19M           | 69%               | 63.6%                           | 85.5%                              |
| L/14                                                                               | 32k                                        | 12B (32 epochs of laion400m)            | 123.65M         | 303.97M          | 72%               | 68.1%                           | 90.8%                              |
| [g/14](https://wandb.ai/rom1504/eval_openclip/reports/slow-g-14--VmlldzoyNTMwMjg5) | 32k for 8B samples then 64k for 4B samples | 12B (similar to 32 epochs on laion400m) | 354.03M         | 1012.65M         | 76.6%             | 72.4%                           | 93.5%                              |


In addition to having overall better results, we hope the larger text encoder will help improve text understanding. The good performance on the retrieval metrics seems to be a good indicator of this property.

Note the difference in samples seen between the H/14 and the g/14 model. This explains the difference in performance. We picked this lower number to try and fix the stability issue at a lower cost. Eventually they were fixed (by using bfloat16). The performance of this model falls in the scaling curve of 12B sample seen (similar to 32 epochs of laion400m), and a g/14 trained on 32B samples of laion2B would most likely follow the same trends as the other models and get better performance as H/14.

![alt_text](/images/blog/compare3.png "image_tooltip")



## Released checkpoints

We release the checkpoints for the models, they are available through [openclip](https://github.com/mlfoundations/open_clip) and in HuggingFace hub at [B/32](https://huggingface.co/laion/CLIP-ViT-B-32-laion2B-s34B-b79K) [L/14](https://huggingface.co/laion/CLIP-ViT-L-14-laion2B-s32B-b82K) [H/14](https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K) and [g/14](https://huggingface.co/laion/CLIP-ViT-g-14-laion2B-s12B-b42K)


## Related works

Related work results:


| **Model name** | **Samples seen**       | **Imagenet top1** | **Mscoco image retrieval at 5** | **Flickr30k image retrieval at 5** |
|----------------|------------------------|-------------------|---------------------------------|------------------------------------|
| Openai B/32    | 12B (32 epochs of WIT) | 62%               |                                 |                                    |
| Openai B/16    | 12B (32 epochs of WIT) | 69%               |                                 |                                    |
| Openai L/14    | 12B (32 epochs of WIT) | 75.4%             | 61%                             | 87%                                |
| ALIGN          | 20B                    | 76.4%             | 69.8%                           | 93.3%                              |
| BASIC          | 32B                    | 85.7%             |                                 |                                    |
| CoCa           | 32B                    | 86.3%             | 74.2%                           | 95.7%                              |


[BASIC](https://arxiv.org/abs/2111.10050) and  [ALIGN](https://arxiv.org/abs/2102.05918) got excellent imagenet results. They used either different image encoder architecture (EfficientNet, CoAtNet), a larger network scale (BASIC-L with 2.4B params) or pre trained their network with supervised learning on a large dataset (BASIC CoAtNet vision encoder).

[COCA](https://arxiv.org/abs/2205.01917) additionally used captioning loss during training with a multi-modal text decoder which predicted text tokens autoregressively and got 86.3% top1, employing a larger model scale (2.1B params)


## Scaling up notes

During these training runs, we encountered several interesting issues:



* Using many GPUs means many of them can have hardware issues and can freeze, crash or even just be slow. This is a particularly annoying problem to handle as if one GPU has an issue, the synchronized nature of distributed training means that all GPUs get stuck. I created [https://github.com/rom1504/gpu-tester](https://github.com/rom1504/gpu-tester) to figure out what are the bad GPUs and exclude them
* Stability issues! When scaling up the model size, the batch size and the dataset size, at around half the training the loss starts increasing until it reaches a plateau. We tried many possible things (find the list [there](https://docs.google.com/document/d/1EFbMLRWSSV0LUf9Du1pWzWqgeiIRPwEWX2s1C6mAk5c/edit)) and eventually concluded on a surprisingly simple solution: **using amp bfloat16 instead of amp float16 made the training fully stable**

And also made some discoveries:



* It seems using a very large batch size (up to 159k) can help reach even higher performance. This is most likely due to the fact that contrastive learning provides information to the loss as a logit matrix, hence having N times more samples in a batch means N square logits. We did not verify this systematically but BASIC paper provides more experiments and a theoretical justification for this result.
* It’s possible to get a reasonably performing g/14 CLIP by doing a much shorter cosine decay => getting a 68% g/14 in 10k gpu hours.
* Grad checkpointing allows to do 10x on the batch size


### Training stability issues

Stability of training was the main problem we solved in this iteration of the scaling up of OpenCLIP. At around half the training (for L/14, H/14 and g/14), the loss started going up until it plateaued very high (11) and didn’t go down anymore.

We tried many possible fixes (decreasing lr, gradient shrinking, gradient clipping, cosine attention, post layer norm, …) with little to no effect when trying to resume from before the crash. 

Eventually only 2 things worked:



* Finishing the  lr decay very fast : in 8 epochs (compared to the planned 256 epochs). That managed to get most of the performance out of clip H. 
* Switching from float16 to bfloat16 solved the problem while being faster for clip g. We then applied the same fix for clip H and finished its training properly.

[See all the training notes](https://docs.google.com/document/d/1EFbMLRWSSV0LUf9Du1pWzWqgeiIRPwEWX2s1C6mAk5c/edit) with all the details on all the possible ideas that didn’t work.


### Training speeds

To better understand the cost and length of training of clip, we provide these training speed numbers. All numbers assume a100 with 40GB of VRAM. We used gradient checkpointing.

| Model                                                                           | Batch size per gpu | Precision | Number of gpus | Sample per second per gpu |
|---------------------------------------------------------------------------------|--------------------|-----------|----------------|---------------------------|
| [B/32](https://wandb.ai/rom1504/open-clip/runs/rnxrp6k7?workspace=user-rom1504) | 96                 | float16   | 824            | 228                       |
| [H/14](https://wandb.ai/rom1504/open-clip/runs/2zphcgkn?workspace=user-rom1504) | 96                 | float16   | 824            | 30                        |
| [g/14](https://wandb.ai/rom1504/open-clip/runs/21cpomx2?workspace=user-rom1504) | 40                 | float16   | 800            | 20                        |
| [H/14](https://wandb.ai/rom1504/open-clip/runs/3l7ppqh3?workspace=user-rom1504) | 96                 | bfloat16  | 824            | 42                        |
| [g/14](https://wandb.ai/rom1504/open-clip/runs/1pby5fkb?workspace=user-rom1504) | 80                 | bfloat16  | 800            | 31                        |


The speed usually increases with batch size per gpu until a plateau is reached. The speed also increases with the number of gpu. After a certain number of gpus, the curve becomes slower than linear.

Bfloat16 which we used in the second part of training provides both better stability and faster sample/s for clip models.


## What’s next

The models will be used for many applications, including clip guiding and conditioning. Even better results could be reached on models like stable diffusion by using a better clip model!

Now that the scaling properties of clip are proven in an open source reproduction, a lot of doors open. Here are some ideas of next steps:



* Changing the text encoder to work in the multilingual setting (to get a model like [Multilingual-CLIP](https://github.com/FreddeFrallan/Multilingual-CLIP) but trained contrastively, with hopefully even better results!) and scale it up
* Can we get clip models while using less gpu hours ? extracting the knowledge from smaller clips into a bigger one may help bootstrap the learning process (see [encoder-distill](https://github.com/iejMac/encoder-distill) from [iejMac](https://github.com/iejMac) getting some preliminary results on this)  
* The clip idea can be expanded to other modalities, see [CLAP](https://github.com/LAION-AI/CLAP) for text-audio alignment

If you have ideas or want to help out, feel free to reach out in laion server.


## Contributions

Thanks to



* [Romain Beaumont](https://github.com/rom1504) for running the experiments on H/14 and g/14
* [Ross Wightman](https://github.com/rwightman) for conducting all the openclip experiments at JUWELS Booster (Juelich Supercomputing Center) up to L/14 and providing valuable feedback during these H and g clip trainings
* [Phil Wang](https://github.com/lucidrains) for providing ideas and code (cosine attention, post layer norm, ..) during the stability issues
* [Boris Dayma](https://github.com/borisdayma) and [Mitchell Wortsman](https://mitchellnw.github.io/) for both proposing to try float32 that showed precision was an issue and eventually lead to trying bfloat16
* [Blinkdl](https://github.com/Blinkdl) for proposing interesting ideas regarding tuning the learning rate
* [Christoph Schuhmann](https://github.com/christophschuhmann) for daring proposing to train such large clips, following up on all these experiments, and finding very early that training were frozen, saving some valuable time
* [Jenia Jitsev](https://github.com/JeniaJitsev) for providing ideas and feedback during the training issues, supervision and coordination of the compute grants at JUWELS Booster
* [Ludwig Schmidt](https://github.com/ludwigschmidt) for reviewing this post and giving many ideas about LAION datasets and CLIP 
* [Mehdi Cherti](https://github.com/mehdidc) for helping to debug the evaluation scripts and getting comparable results for MS-COCO

And of course [Emad](https://twitter.com/EMostaque) (Stability AI) for providing the many GPUs used during these experiments! (g/14 and H/14!)

For the L/14 training, we gratefully acknowledge the Gauss Centre for Supercomputing e.V. (www.gauss-centre.eu) for funding this part of work by providing computing time through the John von Neumann Institute for Computing (NIC) on the GCS Supercomputer JUWELS Booster at Jülich Supercomputing Centre (JSC), Germany.


Large scale openCLIP: L/14, H/14 and g/14 trained on LAION-2B


We present LAION-Aesthetics, several collections of subsets from LAION 5B with high visual quality.

![](https://raw.githubusercontent.com/LAION-AI/laion.ai/Chris/blog/LAION-Aesthetics.jpg)

To create LAION-Aesthetics we trained several lightweight models that predict the rating people gave when they were asked _“How much do you like this image on a scale from 1 to 10?”_.

## LAION-Aesthetics V1

We started with training a linear model on 5000 image-rating pairs from the [SAC](https://github.com/JD-P/simulacra-aesthetic-captions) dataset (which only contained 5000 samples at that time).

Simulacra Aesthetic Captions is a dataset of over 238000 synthetic images generated with AI models such as CompVis latent GLIDE and Stable Diffusion from over forty thousand user submitted prompts.

As inputs this model uses not the images themselves, but their CLIP Image embeddings produced with the Open AI CLIP VIT L 14 model. We call this model LAION-Aesthetics_Predictor V1.

Its results were so encouraging, that we decided to produce 8M and 120M sample subsets of the LAION 5B images with the highest predicted scores, of those that have english texts.

We call the dataset consisting of these 2 subsets [LAION-Aesthetics V1](https://github.com/LAION-AI/laion-datasets/blob/main/laion-aesthetic.md).

![](https://github.com/LAION-AI/laion.ai/blob/Chris/blog/LAION-Aesthetics%20V1.jpg?raw=true)

The model used for creating this subset can be found [here.](https://github.com/LAION-AI/aesthetic-predictor)

The LAION-Aesthetics V1 dataset & further details about it can be found [here.](https://github.com/LAION-AI/laion-datasets/blob/main/laion-aesthetic.md)

## LAION-Aesthetics V2

After these very encouraging results, we continued to experiment and gathered the following data to train more improved MLP (multi-layer perceptron) models:

- More samples from the SAC dataset, which had grown in the meanwhile
  to 176000 image - rating pairs
- LAION-Logos, a dataset of 15.000 logo image-text pairs with aesthetic
  ratings from 1 to 10. We collected this dataset to improve the models
  abilities to evaluate images with more or less aesthetic texts in
  them.
- [The Aesthetic Visual Analysis (AVA) dataset](https://github.com/imfing/ava_downloader), which is a large-Scale database for aesthetic visual analysis that contains 250000 photos from dpchallenge.com with several aesthetic ratings from 1 to 10 for most images.
- After training several MLPs with different numbers of layers and parameters and different activation functions, we found that a simple linear model on the top of CLIP ViT/14 produced in our subjective view the visually most appealing results when used to rank images of LAION-5B. (Even though other MLPs with e.g. Relu functions produced slightly lower MSE and MAE loss values.) We call the resulting model trained on SAC, LAION-Logos and AVA [LAION-Aesthetics_Predictor V2.](https://github.com/christophschuhmann/improved-aesthetic-predictor)
- Visualizations of sorting all 2.37B images from LAION 5B that have English captions into 40 buckets with the LAION-Aesthetics_Predictor V2 can be found [here.](http://captions.christoph-schuhmann.de/aesthetic_viz_laion_sac+logos+ava1-l14-linearMSE-en-2.37B.html)

Using LAION-Aesthetics_Predictor V2, we created the following subsets of the LAION 5B samples with English captions:

- 1,2B image-text pairs with predicted aesthetics scores of 4.5 or higher: [browse](http://captions.christoph-schuhmann.de/2B-en-4.5.html) [huggingface](https://huggingface.co/datasets/ChristophSchuhmann/improved_aesthetics_4.5plus)
- 939M image-text pairs with predicted aesthetics scores of 4.75 or higher: [browse](http://captions.christoph-schuhmann.de/2B-en-4.75.html) [huggingface](https://huggingface.co/datasets/ChristophSchuhmann/improved_aesthetics_4.75plus)
- 600M image-text pairs with predicted aesthetics scores of 5 or higher: [browse](http://captions.christoph-schuhmann.de/2B-en-5.html) [huggingface](https://huggingface.co/datasets/ChristophSchuhmann/improved_aesthetics_5plus)
- 12M image-text pairs with predicted aesthetics scores of 6 or higher: [browse](http://captions.christoph-schuhmann.de/2B-en-6.html) [huggingface](https://huggingface.co/datasets/ChristophSchuhmann/improved_aesthetics_6plus)
- 3M image-text pairs with predicted aesthetics scores of 6.25 or higher: [browse](http://captions.christoph-schuhmann.de/2B-en-6.25.html) [huggingface](https://huggingface.co/datasets/ChristophSchuhmann/improved_aesthetics_6.25plus)
- 625K image-text pairs with predicted aesthetics scores of 6.5 or higher: [browse](http://captions.christoph-schuhmann.de/2B-en-6.5.html) [huggingface](https://huggingface.co/datasets/ChristophSchuhmann/improved_aesthetics_6.5plus)

These subsets overlap. 5 fully includes 6 which includes 6.25 and so on. We call the collection of these subsetsLAION-Aesthetics V2.

We provided the dataset to the [CompViz](https://github.com/CompVis) team led by Robin Rombach and Patrick Esser. They used the 5+ subset to train [Stable Diffusion V1](https://github.com/CompVis/stable-diffusion/tree/ce05de28194041e030ccfc70c635fe3707cdfc30#stable-diffusion-v1) model.

## What's next?

At the moment we are translating all 2,15B samples from LAION 5B of the multilingual subset to English using the 1,2B parameter [M2M-100](https://github.com/facebookresearch/fairseq/tree/main/examples/m2m_100) model .

This will allow us to roughly double the size of V2.

Additionally, we are already working on new multimodal large-scale dataset, this time at webpage-level, similar to the interleaved image-text dataset Deepmind used for [Flamingo](https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model), but also with audio & video files ... and much, much bigger. :)

Stay tuned & keep checking our blog for more datasets in the near future.

## Connect

If you have any questions or comments or the wish to support our efforts, don’t hesitate to [join our Discord community and contact us.](https://discord.gg/vnjVezbeSJ)

_Christoph Schuhmann ( spirit-from-germany#1488 ) and Romain Beaumont ( rom1504#5008 )_


BLOG