
Author: [Christoph Schuhmann](https://github.com/christophschuhmann), [Andreas Köpf](https://github.com/andreaskoepf) , [Theo Coombes](https://github.com/TheoCoombes), [Richard Vencu](https://github.com/rvencu/), [Benjamin Trom](https://github.com/limiteinductive) , [Romain Beaumont](https://github.com/rom1504) 

**We present LAION-COCO, the world’s largest dataset of 600M generated high-quality captions for publicly available web-images**

Laion5B has five billion natural captions. They provide a lot of information, but could synthetic captions complement them ?

To answer this question, we use a combination of existing, publicly available models to produce high quality captions for images in the style of [MS COCO](https://paperswithcode.com/dataset/coco).

We captioned 600M images from the english subset of Laion-5B with an ensemble of [BLIP](https://github.com/salesforce/BLIP) L/14 and 2 CLIP versions (L/14 and RN50x64).  

With this post we release them openly today.

This will make it possible to investigate the value of generated captions to train models. We’re curious on how these synthetic captions could impact models trained on them!


## Download it

The 600M samples are provided in parquet files. Columns include the original caption, the url, the top caption and a list of alternative captions with lower CLIP-similarity scores.

[https://huggingface.co/datasets/laion/laion-coco](https://huggingface.co/datasets/laion/laion-coco) 


## Samples



<img src="/images/blog/ring.png" style="height:250px; width: auto;"/>


**Original:** LGSY 925 Sterling Silver Double Heart Rings Infinity Love Thin Rings Wedding Engagement Promise Engraved Love Rings for Women for Dainty Gift

**Generated:** An open ring with two hearts on it.



<img src="/images/blog/boot.png" style="height:250px; width: auto;"/>
         

**Original:** Female Thick with Pointy Head High Heel Chelsea Ankle Boots

**Generated:** Red leather ankle boots with gold buckles.



<img src="/images/blog/laion_coco_beach.png" style="height:250px; width: auto;"/>


**Original:** a group of people on horses on a beach

**Generated:** Several people riding horses down the beach on a cloudy day.




<img src="/images/blog/laion_coco_tags.png" style="height:250px; width: auto;"/>


**Original:** a wall with a bunch of graffiti on it

**Generated:** The parking meter is near a graffiti covered building.

                                                                                                                                        



<img src="/images/blog/sheeple.png" style="height:250px; width: auto;"/>


**Original:** sheeple family

**Generated:** A cartoon drawing of sheep watching TV with their babies.


## More samples of images with their generated captions can be found here: 

(no cherry picking)

[http://captions.christoph-schuhmann.de/eval_laion/eval.html](http://captions.christoph-schuhmann.de/eval_laion/eval.html) 


## Method

The method we used to generate these captions was to

1. We use Blip L/14 to generate 40 captions
2. Rank them using openai Clip Open AI L/14 ; selected the best 5 captions
3. Rank using Open AI RN50x64 Clip model to select the best one
4. Use a small, fine-tuned T0 model to roughly repair grammar and punctuation of the texts

The hyperparameters were chosen through a [grid search](https://wandb.ai/andreaskoepf/blip_coco_val_sample_sweep_bayes_02/sweeps/1bsha6b0) ([settings](https://github.com/andreaskoepf/CLIP-Image-Captioning/blob/blip_test/blip_coco_val_sample_sweep_bayes_02.yaml)) by Andreas Köpf to best match the style ( ROUGE scores ) of MS COCO texts.

[laion_idle_cap](https://github.com/andreaskoepf/laion_idle_cap/tree/main/docker) is the script that was used for this processing.


## Evaluation

We evaluated these generated captions by asking human evaluators to guess whether a caption is coming from a human or an AI model. We also asked them to rate the quality on a scale from 0(bad) to 5 (good). 

In a first round we presented the evaluators each 200 samples, that contained 100 AI generated and 100 human written MS COCO captions.

### Observations

 

<img src="/images/blog/eval_laion_coco.png" style="width:400px" />


GT: Y-Axis

Annotation: X-Axis

Mean rating & standard deviation of samples, that were written by a human:

Mean: 3.98

Stdev: 0.99

Mean rating & standard deviation of samples, that were written by an AI 

Mean: 3.89

Stdev: 1.12

Mean rating & standard deviation of samples, where the annotator believed they were written by a human:

Mean: 4.44

Stdev: 0.61

Mean rating & standard deviation of samples, where the annotator believed they were generated by an AI 

Mean: 3.50

Stdev: 1.15

### Interpretation

It is very interesting that the mean scores of the samples generated by humans and generated by the model are very similar. We also notice that the standard deviation of the generated captions is a little bit higher.

We hypothesize that most in most cases the quality of the generated captions is perceived as as good as the quality of the human written captions.

But sometimes the captioning model obviously fails and the quality of the results is pretty low because the model doesn't relevant understand concepts about what is going on in the picture, because it's knowledge is not grounded in a sufficiently sophisticated world model.

### Failure cases



<img src="/images/blog/laion_coco_umbrella.png" style="height:250px; width: auto;"/>


_“Two people posing for the camera in their wedding attire, one with an umbrella over his head and another with long red hair.”_



<img src="/images/blog/laion_coco_man.png" style="height:250px; width: auto;"/>


_“An older man having a heart attack, with his hand on the chest.”_

When we remove all samples from the evaluations that have ratings of either 0 or 1, we Observe that the mean ratings and standard deviations move closer together. 

### Scores without ratings of 0 and 1

Mean rating & standard deviation of samples, that were written by a human:

Mean: 4.07

Stdev: 0.81

Mean rating & standard deviation of samples, that were written by an AI 

Mean: 4.02

Stdev: 0.94

The mean ratings of the generated captions are still a little bit lower and the standard deviation is still a little bit higher, but the trend is pretty clear. By removing samples with rating 2, the gap between the qualities would probably decrease even further. 

Presentation only generated captions:

In a next step, we presented the human evaluators 400 captions that were only generated by the model (no human written captions in between):

Mean rating of all samples 

3.81

Standard deviation of all samples 

0.94

% rated as human

47.5

% rated as AI

52.5

We observe that the human evaluators thought in 47.5% of all cases, that the captions were written by a human. This makes us confident that our captains are on average pretty good. When we told the evaluators later that all captions were generated by the model they told us that it was very hard for them to judge whether a caption was written by a model or a human, and that it only was easy for them in obvious failure cases.

### Conclusions

We conclude that Our ensemble of BLIP and CLIP is already pretty good and capable of generating captions with a quality that is on average pretty close to the human written captions of MS Coco. 

It would be very interesting for future work to let people rate our generated captions at larger scale and then filter out the samples with low rating values. These results could be used to train models to rate the quality of captions and to predict whether a caption looks like a generated or a human written caption.

And even without further automated filtering, an ensemble of our captions and human evaluators would be a pretty good workflow to curate high quality captions at much lower costs than if we would ask humans to write them from scratch.


## Credit assignments



* [Christoph Schuhmann](https://github.com/christophschuhmann) lead the project, implemented a first version of the code, ran most of the generations & conducted the human evaluations
* [Andreas Köpf](https://github.com/andreaskoepf) conducted the hyperparameter search & wrote the code to execute BLIP + CLIP filtering at scale
* [Theo Coombes](https://github.com/TheoCoombes) managed the server that coordinated which GPU worker got which part of LAION to work on
* [Romain Beaumont](https://github.com/rom1504) packaged the .json into parquet files, sent to HF and wrote the first draft of this post
* [Richard Vencu](https://github.com/rvencu/) provided the infra structure to use the idle compute for this project
* [Benjamin Trom](https://github.com/limiteinductive) wrote code that help us to convert the .json files to parquet

We thank [stability.ai](https://stability.ai/) for providing the compute used to generate the captions in the dataset.


Laion coco: 600M synthetic captions from Laion2B-en


Author: [Marianna Nezhurina](https://github.com/marianna13) [Romain Beaumont](https://github.com/rom1504/) [Richard Vencu](https://github.com/rvencu) [Christoph Schuhmann](https://github.com/christophschuhmann)  

Laion5B dataset was automatically collected from a section of the human web (common crawl). Can models generate different and interesting data compared to what humans write?

That’s a question we are interested in investigating. To let the community study it, we translated 3B samples of Laion5B from many languages into English.

We released 3 billions captions for the multilingual part of Laion5B. This makes it possible to use the whole Laion5B dataset to train English models. This also enables training models using these aligned pairs such as [Multilingual-CLIP](https://github.com/FreddeFrallan/Multilingual-CLIP).

We’re curious what you will do using it!


## Downloading it

The dataset is available in huggingface as parquet files containing the caption, translated caption and urls.

[laion1B-nolang-joined-translated-to-en](https://huggingface.co/datasets/laion/laion1B-nolang-joined-translated-to-en) 

[laion2B-multi-joined-translated-to-en](https://huggingface.co/datasets/laion/laion2B-multi-joined-translated-to-en)  


## Processing

Every caption of the original dataset was translated with Facebook’s [M2M100 1.2B model](https://huggingface.co/facebook/m2m100_1.2B) using the following [script](https://github.com/marianna13/translate_dataset/blob/main/translate_data.py). All other fields remain the same as in the original [LAION2B Multi Joined](https://huggingface.co/datasets/laion/laion2B-multi-joined). To make translation possible the original dataset was split into parts with 50k samples in each and every such small part was translated in parallel on GPU nodes and saved in a separate parquet file. The speed of translation depends on the number of nodes and GPUs. The processing was done with 20 nodes with 8 GPUs in each and the speed of translation (including preprocessing and data loading) is 34 samples/per GPU/per second. Then, all translated parquets were [merged together using Spark](https://github.com/marianna13/translate_dataset/blob/main/join_additional.py) and saved as 128 parquet files. The resulting dataset was [joined with the aesthetics scores](https://github.com/marianna13/translate_dataset/blob/main/join_aesthetics.py). 


## Dataset columns



* TEXT (the original text of caption)
* LANGUAGE (language of the original TEXT)
* ENG TEXT (translation in English of the original TEXT)
* URL (URL of the image)
* WIDTH (width of the image)
* HEIGHT (height of the image)
* Hash (hash of the URL and TEXT)
* Pwatermark (probability of being a watermarked image, computed using our [watermark detector](https://github.com/LAION-AI/LAION-5B-WatermarkDetection))
* Punsafe (probability of being an unsafe image, computed using our [clip based detector](https://github.com/LAION-AI/CLIP-based-NSFW-Detector))
* Similarity (cosine between text and image ViT-B/32 embeddings, clip for en, mclip for multi and nolang)
* Prediction (aesthetics score)


## Samples from the translated dataset:


![laion-2B-translated-samples](/images/blog/laion-translated-samples.png "laion-2B-translated-samples")



## Laion2B-multi-translated


### Dataset stats

_Note. Dataset stats were computed using [this](https://github.com/marianna13/translate_dataset/blob/main/get_dataset_stats.py) script._

Number of uniques 2266M (2266193302)                                            

Number with WIDTH >= 0 and WIDTH &lt;= 128 160M (160260569)                        

Number with WIDTH >= 128 and WIDTH &lt;= 256 734M (734166164)                      

Number with WIDTH >= 256 and WIDTH &lt;= 512 849M (849569769)                      

Number with WIDTH >= 512 and WIDTH &lt;= 1024 457M (457572747)                     

Number with WIDTH >= 1024 86M (86750813)                                        

Number with HEIGHT >= 0 and HEIGHT &lt;= 128 103M (103514467)                      

Number with HEIGHT >= 128 and HEIGHT &lt;= 256 614M (614490681)                    

Number with HEIGHT >= 256 and HEIGHT &lt;= 512 753M (753540968)                    

Number with HEIGHT >= 512 and HEIGHT &lt;= 1024 686M (686553437)                   

Number with HEIGHT >= 1024 153M (153139456)                                     

Number with lenengtext >= 0 and lenengtext &lt;= 25 506M (506238532)               

Number with lenengtext >= 25 and lenengtext &lt;= 50 849M (849160165)              

Number with lenengtext >= 50 and lenengtext &lt;= 100 840M (840635023)             

Number with lenengtext >= 100 and lenengtext &lt;= 150 136M (136709119)            

Number with lenengtext >= 150 5M (5148507)


### Similarities between text and images

10000 images and captions were sampled from the dataset, [CLIP embeddings were computed](https://github.com/marianna13/translate_dataset/blob/main/get_clip_embs_similarities.py) (for original texts embeddings were computed using Multilingual CLIP). Then dot products between image and text embeddings were computed (for both original and translated dataset) to get similarities between texts and images. Here’s the distribution of average similarities for two datasets:

Similarity for original dataset:

10% quantile -  0.2552971839904785

20% quantile -  0.2633610963821411

30% quantile -  0.2694466710090637

40% quantile -  0.2750270366668701

50% quantile -  0.28088638186454773

60% quantile -  0.28750720620155334

70% quantile -  0.2950591444969177

80% quantile -  0.3049575388431549

90% quantile -  0.32077282667160034

Similarity for translated dataset:

10% quantile  -  0.23388671875

20% quantile  -  0.25390625

30% quantile  -  0.265869140625

40% quantile  -  0.2763671875

50% quantile  -  0.2861328125

60% quantile  -  0.29638671875

70% quantile  -  0.306884765625

80% quantile  -  0.31982421875

90% quantile  -  0.338134765625



![laion-2B-translated](/images/blog/laion-2B-translated.png "laion-2B-translated")



## Laion1B-nolang-translated

[LAION1B Nolang Joined](https://huggingface.co/datasets/laion/laion1B-nolang-joined) dataset was also translated with a similar [script](https://github.com/marianna13/translate_dataset/tree/main) and in the same way as Multi. This dataset doesn’t have the language column so the model also had to determine language. All other columns in the Nolang dataset are the same as in Multi. And just like translated [LAION2B Multi Joined](https://huggingface.co/datasets/laion/laion2B-multi-joined), LAION1B Nolang was also joined with corresponding [aesthetics scores](https://github.com/marianna13/translate_dataset/blob/main/join_aesthetics.py).


### Nolang dataset Stats

Number of uniques 1260M (1260048307)                                            

Number with WIDTH >= 0 and WIDTH &lt;= 128 90M (90701133)                          

Number with WIDTH >= 128 and WIDTH &lt;= 256 409M (409575445)                      

Number with WIDTH >= 256 and WIDTH &lt;= 512 475M (475885337)                      

Number with WIDTH >= 512 and WIDTH &lt;= 1024 239M (239035772)                     

Number with WIDTH >= 1024 59M (59942110)                                        

Number with HEIGHT >= 0 and HEIGHT &lt;= 128 59M (59814914)                        

Number with HEIGHT >= 128 and HEIGHT &lt;= 256 370M (370913206)                    

Number with HEIGHT >= 256 and HEIGHT &lt;= 512 451M (451897702)                    

Number with HEIGHT >= 512 and HEIGHT &lt;= 1024 316M (316723245)                   

Number with HEIGHT >= 1024 87M (87671543)                                       

Number with lenengtext >= 0 and lenengtext &lt;= 25 312M (312548202)               

Number with lenengtext >= 25 and lenengtext &lt;= 50 555M (555971621)              

Number with lenengtext >= 50 and lenengtext &lt;= 100 413M (413430230)             

Number with lenengtext >= 100 and lenengtext &lt;= 150 20M (20446701)              

Number with lenengtext >= 150 0M (138974)  


### Similarities between text and images for Nolang

10000 images and captions were randomly sampled from the dataset, [CLIP embeddings were computed](https://github.com/marianna13/translate_dataset/blob/main/get_clip_embs_similarities.py) (for original texts embeddings were computed using Multilingual CLIP). Then dot products between image and text embeddings were computed (for both original and translated dataset) to get similarities between texts and images. Here’s the distribution of average similarities for two datasets:

Similarity for original dataset:

10% quantile  -  0.258196085691452

20% quantile  -  0.266357421875

30% quantile  -  0.2728866934776306

40% quantile  -  0.27902457118034363

50% quantile  -  0.28590404987335205

60% quantile  -  0.29329144954681396

70% quantile  -  0.3023602366447449

80% quantile  -  0.31363412737846375

90% quantile  -  0.3313804566860199

Similarity for translated dataset:

10% quantile  -  0.2406005859375

20% quantile  -  0.2607421875

30% quantile  -  0.27490234375

40% quantile  -  0.2861328125

50% quantile  -  0.296142578125

60% quantile  -  0.306396484375

70% quantile  -  0.317626953125

80% quantile  -  0.33203125

90% quantile  -  0.353271484375


![laion-1B-translated](/images/blog/laion-1B-translated.png "laion-1B-translated")


## Credit



* [Marianna Nezhurina](https://github.com/marianna13) translated the samples, packaged them, computed stats and wrote most of this post
* [Romain Beaumont](https://github.com/rom1504/) helped out on packaging and scaling
* [Richard Vencu](https://github.com/rvencu) set up all the infra that made using idle compute possible
* [Christoph Schuhmann](https://github.com/christophschuhmann) suggested the project and guided the work to completion

We thank [https://stability.ai/](https://stability.ai/) for providing the compute for this massive translation. This was a great use of pre-emptible jobs to fill any idle compute available!

Laion translated: 3B captions translated to English from laion5B


We trained three large CLIP models with [OpenCLIP](https://github.com/mlfoundations/open_clip): ViT-L/14, ViT-H/14 and ViT-g/14 (ViT-g/14 was trained only for about a third the epochs compared to the rest). The H/14 model achieves **78.0%** zero shot top-1 accuracy on ImageNet and **73.4%** on zero-shot image retrieval at Recall@5 on MS COCO. As of September 2022, this is the best open source CLIP model.

CLIP makes it possible to compute representations of images and texts to measure how **similar** they are. It can be used for 



* Zero shot classification: compare an image with the text of the class to know which class is most similar (e.g., ImageNet classification)
* Retrieval: compare an image or a text to billions of text or images to find the most similar (e.g. as in [clip-retrieval](https://rom1504.github.io/clip-retrieval/) )
* Generation
    * CLIP guidance: decide a text you want to generate, then use an image generator model, and use the CLIP distance between what’s generated and the text to generate a better image (e.g., VQGAN + CLIP)
    * CLIP conditioning: use a clip text embedding as input of a generator to make it generate this text directly (e.g., stable diffusion)

CLIP models are trained in a self supervised fashion on hundreds of millions or billions of (image, text) pairs.

With LAION, we produced the LAION-5B dataset that contains 5.8 billions of closely related image and text pairs.

The CLIP model ViT B/32, released by OpenAI, was initially used to filter this dataset out of common crawl.

Producing the best open source CLIP model out of this data set completes the open source replication of the [excellent](https://openai.com/blog/clip/) CLIP paper that OpenAI released one year ago.


## Results

We replicated the results from openai CLIP in models of different sizes, then trained bigger models. The full evaluation suite on 39 datasets ([vtab+](https://github.com/LAION-AI/CLIP_benchmark)) are available in this [results notebook](https://github.com/LAION-AI/CLIP_benchmark/blob/main/benchmark/results.ipynb) and show consistent improvements over all datasets.

The larger models we release today are L/14, H/14 and g/14.

L/14 was trained on JUWELS Booster supercomputer by [Ross wightman](https://github.com/rwightman). H/14 and g/14 were trained on stability cluster by [Romain Beaumont](https://github.com/rom1504) . While L/14 and H/14 were trained using 34B samples from LAION-2b, g/14 used a substantially smaller sample scale for training, seeing only 12B samples (see tables for more details).


#### 32B samples seen


| **Model name**                                                                   | **Batch size**                   | **Samples seen**           | **Text Params** | **Image params** | **Imagenet top1** | **Mscoco image retrieval at 5** | **Flickr30k image retrieval at 5** |
|----------------------------------------------------------------------------------|----------------------------------|----------------------------|-----------------|------------------|-------------------|---------------------------------|------------------------------------|
| [B/32](https://wandb.ai/rom1504/eval_openclip/reports/B-32-2B--VmlldzoyNDkwNDMy) | 79k                              | 34B (16 epochs of laion2B) | 63.43M          | 87.85M           | 66.6%             | 65.4%                           | 88.4%                              |
| L/14                                                                             | 79k for 14B samples, 86K for 18B | 32B                        | 123.65M         | 303.97M          | 75.3%             | 71.1%                           | 92.9%                              |
| [H/14](https://wandb.ai/rom1504/eval_openclip/reports/H-14--VmlldzoyNDAxODQ3)    | 79k                              | 32B (16 epochs of laion2B) | 354.03M         | 632.08M          | 78.0%             | 73.4%                           | 94%                                |



#### 12B samples seen


| **Model name**                                                                     | **Batch size**                             | **Samples seen**                        | **Text Params** | **Image params** | **Imagenet top1** | **Mscoco image retrieval at 5** | **Flickr30k image retrieval at 5** |
|------------------------------------------------------------------------------------|--------------------------------------------|-----------------------------------------|-----------------|------------------|-------------------|---------------------------------|------------------------------------|
| B/32                                                                               | 32k                                        | 12B (32 epochs of laion400m)            | 63.43M          | 87.85M           | 62.9%             | 60.8%                           | 85.5%                              |
| B/16                                                                               | 32k                                        | 12B (32 epochs of laion400m)            | 91.16M          | 86.19M           | 69%               | 63.6%                           | 85.5%                              |
| L/14                                                                               | 32k                                        | 12B (32 epochs of laion400m)            | 123.65M         | 303.97M          | 72%               | 68.1%                           | 90.8%                              |
| [g/14](https://wandb.ai/rom1504/eval_openclip/reports/slow-g-14--VmlldzoyNTMwMjg5) | 32k for 8B samples then 64k for 4B samples | 12B (similar to 32 epochs on laion400m) | 354.03M         | 1012.65M         | 76.6%             | 72.4%                           | 93.5%                              |


In addition to having overall better results, we hope the larger text encoder will help improve text understanding. The good performance on the retrieval metrics seems to be a good indicator of this property.

Note the difference in samples seen between the H/14 and the g/14 model. This explains the difference in performance. We picked this lower number to try and fix the stability issue at a lower cost. Eventually they were fixed (by using bfloat16). The performance of this model falls in the scaling curve of 12B sample seen (similar to 32 epochs of laion400m), and a g/14 trained on 32B samples of laion2B would most likely follow the same trends as the other models and get better performance as H/14.

![alt_text](/images/blog/compare3.png "image_tooltip")



## Released checkpoints

We release the checkpoints for the models, they are available through [openclip](https://github.com/mlfoundations/open_clip) and in HuggingFace hub at [B/32](https://huggingface.co/laion/CLIP-ViT-B-32-laion2B-s34B-b79K) [L/14](https://huggingface.co/laion/CLIP-ViT-L-14-laion2B-s32B-b82K) [H/14](https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K) and [g/14](https://huggingface.co/laion/CLIP-ViT-g-14-laion2B-s12B-b42K)


## Related works

Related work results:


| **Model name** | **Samples seen**       | **Imagenet top1** | **Mscoco image retrieval at 5** | **Flickr30k image retrieval at 5** |
|----------------|------------------------|-------------------|---------------------------------|------------------------------------|
| Openai B/32    | 12B (32 epochs of WIT) | 62%               |                                 |                                    |
| Openai B/16    | 12B (32 epochs of WIT) | 69%               |                                 |                                    |
| Openai L/14    | 12B (32 epochs of WIT) | 75.4%             | 61%                             | 87%                                |
| ALIGN          | 20B                    | 76.4%             | 69.8%                           | 93.3%                              |
| BASIC          | 32B                    | 85.7%             |                                 |                                    |
| CoCa           | 32B                    | 86.3%             | 74.2%                           | 95.7%                              |


[BASIC](https://arxiv.org/abs/2111.10050) and  [ALIGN](https://arxiv.org/abs/2102.05918) got excellent imagenet results. They used either different image encoder architecture (EfficientNet, CoAtNet), a larger network scale (BASIC-L with 2.4B params) or pre trained their network with supervised learning on a large dataset (BASIC CoAtNet vision encoder).

[COCA](https://arxiv.org/abs/2205.01917) additionally used captioning loss during training with a multi-modal text decoder which predicted text tokens autoregressively and got 86.3% top1, employing a larger model scale (2.1B params)


## Scaling up notes

During these training runs, we encountered several interesting issues:



* Using many GPUs means many of them can have hardware issues and can freeze, crash or even just be slow. This is a particularly annoying problem to handle as if one GPU has an issue, the synchronized nature of distributed training means that all GPUs get stuck. I created [https://github.com/rom1504/gpu-tester](https://github.com/rom1504/gpu-tester) to figure out what are the bad GPUs and exclude them
* Stability issues! When scaling up the model size, the batch size and the dataset size, at around half the training the loss starts increasing until it reaches a plateau. We tried many possible things (find the list [there](https://docs.google.com/document/d/1EFbMLRWSSV0LUf9Du1pWzWqgeiIRPwEWX2s1C6mAk5c/edit)) and eventually concluded on a surprisingly simple solution: **using amp bfloat16 instead of amp float16 made the training fully stable**

And also made some discoveries:



* It seems using a very large batch size (up to 159k) can help reach even higher performance. This is most likely due to the fact that contrastive learning provides information to the loss as a logit matrix, hence having N times more samples in a batch means N square logits. We did not verify this systematically but BASIC paper provides more experiments and a theoretical justification for this result.
* It’s possible to get a reasonably performing g/14 CLIP by doing a much shorter cosine decay => getting a 68% g/14 in 10k gpu hours.
* Grad checkpointing allows to do 10x on the batch size


### Training stability issues

Stability of training was the main problem we solved in this iteration of the scaling up of OpenCLIP. At around half the training (for L/14, H/14 and g/14), the loss started going up until it plateaued very high (11) and didn’t go down anymore.

We tried many possible fixes (decreasing lr, gradient shrinking, gradient clipping, cosine attention, post layer norm, …) with little to no effect when trying to resume from before the crash. 

Eventually only 2 things worked:



* Finishing the  lr decay very fast : in 8 epochs (compared to the planned 256 epochs). That managed to get most of the performance out of clip H. 
* Switching from float16 to bfloat16 solved the problem while being faster for clip g. We then applied the same fix for clip H and finished its training properly.

[See all the training notes](https://docs.google.com/document/d/1EFbMLRWSSV0LUf9Du1pWzWqgeiIRPwEWX2s1C6mAk5c/edit) with all the details on all the possible ideas that didn’t work.


### Training speeds

To better understand the cost and length of training of clip, we provide these training speed numbers. All numbers assume a100 with 40GB of VRAM. We used gradient checkpointing.

| Model                                                                           | Batch size per gpu | Precision | Number of gpus | Sample per second per gpu |
|---------------------------------------------------------------------------------|--------------------|-----------|----------------|---------------------------|
| [B/32](https://wandb.ai/rom1504/open-clip/runs/rnxrp6k7?workspace=user-rom1504) | 96                 | float16   | 824            | 228                       |
| [H/14](https://wandb.ai/rom1504/open-clip/runs/2zphcgkn?workspace=user-rom1504) | 96                 | float16   | 824            | 30                        |
| [g/14](https://wandb.ai/rom1504/open-clip/runs/21cpomx2?workspace=user-rom1504) | 40                 | float16   | 800            | 20                        |
| [H/14](https://wandb.ai/rom1504/open-clip/runs/3l7ppqh3?workspace=user-rom1504) | 96                 | bfloat16  | 824            | 42                        |
| [g/14](https://wandb.ai/rom1504/open-clip/runs/1pby5fkb?workspace=user-rom1504) | 80                 | bfloat16  | 800            | 31                        |


The speed usually increases with batch size per gpu until a plateau is reached. The speed also increases with the number of gpu. After a certain number of gpus, the curve becomes slower than linear.

Bfloat16 which we used in the second part of training provides both better stability and faster sample/s for clip models.


## What’s next

The models will be used for many applications, including clip guiding and conditioning. Even better results could be reached on models like stable diffusion by using a better clip model!

Now that the scaling properties of clip are proven in an open source reproduction, a lot of doors open. Here are some ideas of next steps:



* Changing the text encoder to work in the multilingual setting (to get a model like [Multilingual-CLIP](https://github.com/FreddeFrallan/Multilingual-CLIP) but trained contrastively, with hopefully even better results!) and scale it up
* Can we get clip models while using less gpu hours ? extracting the knowledge from smaller clips into a bigger one may help bootstrap the learning process (see [encoder-distill](https://github.com/iejMac/encoder-distill) from [iejMac](https://github.com/iejMac) getting some preliminary results on this)  
* The clip idea can be expanded to other modalities, see [CLAP](https://github.com/LAION-AI/CLAP) for text-audio alignment

If you have ideas or want to help out, feel free to reach out in laion server.


## Contributions

Thanks to



* [Romain Beaumont](https://github.com/rom1504) for running the experiments on H/14 and g/14
* [Ross Wightman](https://github.com/rwightman) for conducting all the openclip experiments at JUWELS Booster (Juelich Supercomputing Center) up to L/14 and providing valuable feedback during these H and g clip trainings
* [Phil Wang](https://github.com/lucidrains) for providing ideas and code (cosine attention, post layer norm, ..) during the stability issues
* [Boris Dayma](https://github.com/borisdayma) and [Mitchell Wortsman](https://mitchellnw.github.io/) for both proposing to try float32 that showed precision was an issue and eventually lead to trying bfloat16
* [Blinkdl](https://github.com/Blinkdl) for proposing interesting ideas regarding tuning the learning rate
* [Christoph Schuhmann](https://github.com/christophschuhmann) for daring proposing to train such large clips, following up on all these experiments, and finding very early that training were frozen, saving some valuable time
* [Jenia Jitsev](https://github.com/JeniaJitsev) for providing ideas and feedback during the training issues, supervision and coordination of the compute grants at JUWELS Booster
* [Ludwig Schmidt](https://github.com/ludwigschmidt) for reviewing this post and giving many ideas about LAION datasets and CLIP 
* [Mehdi Cherti](https://github.com/mehdidc) for helping to debug the evaluation scripts and getting comparable results for MS-COCO

And of course [Emad](https://twitter.com/EMostaque) (Stability AI) for providing the many GPUs used during these experiments! (g/14 and H/14!)

For the L/14 training, we gratefully acknowledge the Gauss Centre for Supercomputing e.V. (www.gauss-centre.eu) for funding this part of work by providing computing time through the John von Neumann Institute for Computing (NIC) on the GCS Supercomputer JUWELS Booster at Jülich Supercomputing Centre (JSC), Germany.


Large scale openCLIP: L/14, H/14 and g/14 trained on LAION-2B


We present LAION-Aesthetics, several collections of subsets from LAION 5B with high visual quality.

![](https://raw.githubusercontent.com/LAION-AI/laion.ai/Chris/blog/LAION-Aesthetics.jpg)

To create LAION-Aesthetics we trained several lightweight models that predict the rating people gave when they were asked _“How much do you like this image on a scale from 1 to 10?”_.

## LAION-Aesthetics V1

We started with training a linear model on 5000 image-rating pairs from the [SAC](https://github.com/JD-P/simulacra-aesthetic-captions) dataset (which only contained 5000 samples at that time).

Simulacra Aesthetic Captions is a dataset of over 238000 synthetic images generated with AI models such as CompVis latent GLIDE and Stable Diffusion from over forty thousand user submitted prompts.

As inputs this model uses not the images themselves, but their CLIP Image embeddings produced with the Open AI CLIP VIT L 14 model. We call this model LAION-Aesthetics_Predictor V1.

Its results were so encouraging, that we decided to produce 8M and 120M sample subsets of the LAION 5B images with the highest predicted scores, of those that have english texts.

We call the dataset consisting of these 2 subsets [LAION-Aesthetics V1](https://github.com/LAION-AI/laion-datasets/blob/main/laion-aesthetic.md).

![](https://github.com/LAION-AI/laion.ai/blob/Chris/blog/LAION-Aesthetics%20V1.jpg?raw=true)

The model used for creating this subset can be found [here.](https://github.com/LAION-AI/aesthetic-predictor)

The LAION-Aesthetics V1 dataset & further details about it can be found [here.](https://github.com/LAION-AI/laion-datasets/blob/main/laion-aesthetic.md)

## LAION-Aesthetics V2

After these very encouraging results, we continued to experiment and gathered the following data to train more improved MLP (multi-layer perceptron) models:

- More samples from the SAC dataset, which had grown in the meanwhile
  to 176000 image - rating pairs
- LAION-Logos, a dataset of 15.000 logo image-text pairs with aesthetic
  ratings from 1 to 10. We collected this dataset to improve the models
  abilities to evaluate images with more or less aesthetic texts in
  them.
- [The Aesthetic Visual Analysis (AVA) dataset](https://github.com/imfing/ava_downloader), which is a large-Scale database for aesthetic visual analysis that contains 250000 photos from dpchallenge.com with several aesthetic ratings from 1 to 10 for most images.
- After training several MLPs with different numbers of layers and parameters and different activation functions, we found that a simple linear model on the top of CLIP ViT/14 produced in our subjective view the visually most appealing results when used to rank images of LAION-5B. (Even though other MLPs with e.g. Relu functions produced slightly lower MSE and MAE loss values.) We call the resulting model trained on SAC, LAION-Logos and AVA [LAION-Aesthetics_Predictor V2.](https://github.com/christophschuhmann/improved-aesthetic-predictor)
- Visualizations of sorting all 2.37B images from LAION 5B that have English captions into 40 buckets with the LAION-Aesthetics_Predictor V2 can be found [here.](http://captions.christoph-schuhmann.de/aesthetic_viz_laion_sac+logos+ava1-l14-linearMSE-en-2.37B.html)

Using LAION-Aesthetics_Predictor V2, we created the following subsets of the LAION 5B samples with English captions:

- 1,2B image-text pairs with predicted aesthetics scores of 4.5 or higher: [browse](http://captions.christoph-schuhmann.de/2B-en-4.5.html) [huggingface](https://huggingface.co/datasets/ChristophSchuhmann/improved_aesthetics_4.5plus)
- 939M image-text pairs with predicted aesthetics scores of 4.75 or higher: [browse](http://captions.christoph-schuhmann.de/2B-en-4.75.html) [huggingface](https://huggingface.co/datasets/ChristophSchuhmann/improved_aesthetics_4.75plus)
- 600M image-text pairs with predicted aesthetics scores of 5 or higher: [browse](http://captions.christoph-schuhmann.de/2B-en-5.html) [huggingface](https://huggingface.co/datasets/ChristophSchuhmann/improved_aesthetics_5plus)
- 12M image-text pairs with predicted aesthetics scores of 6 or higher: [browse](http://captions.christoph-schuhmann.de/2B-en-6.html) [huggingface](https://huggingface.co/datasets/ChristophSchuhmann/improved_aesthetics_6plus)
- 3M image-text pairs with predicted aesthetics scores of 6.25 or higher: [browse](http://captions.christoph-schuhmann.de/2B-en-6.25.html) [huggingface](https://huggingface.co/datasets/ChristophSchuhmann/improved_aesthetics_6.25plus)
- 625K image-text pairs with predicted aesthetics scores of 6.5 or higher: [browse](http://captions.christoph-schuhmann.de/2B-en-6.5.html) [huggingface](https://huggingface.co/datasets/ChristophSchuhmann/improved_aesthetics_6.5plus)

These subsets overlap. 5 fully includes 6 which includes 6.25 and so on. We call the collection of these subsetsLAION-Aesthetics V2.

We provided the dataset to the [CompViz](https://github.com/CompVis) team led by Robin Rombach and Patrick Esser. They used the 5+ subset to train [Stable Diffusion V1](https://github.com/CompVis/stable-diffusion/tree/ce05de28194041e030ccfc70c635fe3707cdfc30#stable-diffusion-v1) model.

## What's next?

At the moment we are translating all 2,15B samples from LAION 5B of the multilingual subset to English using the 1,2B parameter [M2M-100](https://github.com/facebookresearch/fairseq/tree/main/examples/m2m_100) model .

This will allow us to roughly double the size of V2.

Additionally, we are already working on new multimodal large-scale dataset, this time at webpage-level, similar to the interleaved image-text dataset Deepmind used for [Flamingo](https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model), but also with audio & video files ... and much, much bigger. :)

Stay tuned & keep checking our blog for more datasets in the near future.

## Connect

If you have any questions or comments or the wish to support our efforts, don’t hesitate to [join our Discord community and contact us.](https://discord.gg/vnjVezbeSJ)

_Christoph Schuhmann ( spirit-from-germany#1488 ) and Romain Beaumont ( rom1504#5008 )_


BLOG