add gpt example, minor changes

Author: AkiRusProd

README.md

@@ -136,12 +136,13 @@ Some [examples](examples/) were trained on the [MNIST](https://pjreddie.com/proj
 3. *[Conway`s Game of Life](examples/conway.py)*  
 4. *[Denoising Diffusion Probabilistic Model](examples/ddpm.py)*
 5. *[Generative Adversarial Network](examples/gan.py)*     
-6. *[Recurrent Digits Classifier](examples/recurrent_digits_classifier.py)*    
-7. *[Recurrent Sequences Classifier](examples/recurrent_sequences_classifier.py)*    
-8. *[Seq2Seq Transformer](examples/seq2seq.py)*
-9. *[Variational Autoencoder](examples/vae.py)*    
-10. *[Vector Quantized Variational Autoencoder](examples/vqvae.py)* 
-11. *[Word2Vec](examples/word2vec.py)*
+6. *[Generative Pre-trained Transformer](examples/gpt.py)*
+7. *[Recurrent Digits Classifier](examples/recurrent_digits_classifier.py)*    
+8. *[Recurrent Sequences Classifier](examples/recurrent_sequences_classifier.py)*    
+9. *[Seq2Seq Transformer](examples/seq2seq.py)*
+10. *[Variational Autoencoder](examples/vae.py)*    
+11. *[Vector Quantized Variational Autoencoder](examples/vqvae.py)* 
+12. *[Word2Vec](examples/word2vec.py)*
 
 
 
@@ -334,7 +335,7 @@ Code:
 <details>
 <summary>Seq2Seq Transformer</summary>
 
-#### Examples of translated sentences of validation set:  
+#### Examples of translated sentences (EN -> DE) of validation set:  
 
 >Example №1  
 *Input sentence: These four people are standing outdoors, with 3 dogs.  
@@ -654,6 +655,28 @@ Training process Example | Interpolation between images Example
 <img src="generated images/gan_training_process.gif"> |  <img src="generated images/gan_vectors_interpolation.gif">
 </details>
 
+<details>
+<summary>Generative Pre-trained Transformer</summary>
+
+#### Examples of a model trained to generate prompts for Stable Diffusion:  
+
+>Example №1  
+*a detailed image of a dark haired cyborg - car 3 d model, a glowing aura, symmetrical, intricate, elegant, highly detailed, digital painting, artstation, concept art, smooth, sharp focus, illustration, art by krenz cushart and artem demura* 
+
+>Example №2  
+*an female warrior, full length, red hair, dark eyes, symmetrical face, highly detailed, digital art, sharp focus, trending on art station, anime art style*  
+
+>Example №3  
+*portrait of a young ruggedly handsome but joyful pirate, male, masculine, upper body, red hair, long hair, d & d, fantasy, sharp features, piercing gaze, sharp features, digital painting, artstation, concept art, matte, sharp*
+
+>Example №4  
+*an anthropomorphic fox wizard, fine art, award winning, intricate, elegant, sharp focus, cinematic lighting, highly detailed, digital painting, 8 k concept art, art by guweiz and z. w. gu, masterpiece, trending on artstation*
+
+>Example №5  
+*a beautiful portrait painting of a cyberpunk city by simon stalenhag and pascal blanche and alphonse mucha, in style of colorful comic. symmetry, hyper detailed. octanev render. trending on artstation*
+
+</details>
+
 <details>
 <summary>Conway`s Game of Life Neural Network Simulation</summary>
 
@@ -842,5 +865,5 @@ Native implementation Example | Neural network Example
 
 ### TODO:
 - [x] Add Seq2Seq Transformer example
-- [ ] Add GPT example
+- [x] Add GPT example
 - [ ] Add lr schedulers

data_loader.py

@@ -104,61 +104,6 @@ def load_utkface(path="datasets/utkface/", image_size=(3, 32, 32)):
 
 
 
-def load_multi30k(path="datasets/multi30k/"):
-    #References: https://pytorch.org/text/stable/_modules/torchtext/datasets/multi30k.html
-    urls = {
-        "train": r"https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/training.tar.gz",
-        "valid": r"https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/validation.tar.gz",
-        "test": r"https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/mmt16_task1_test.tar.gz",
-    }
-
-    filenames = ["mmt16_task1_test.tar.gz", "training.tar.gz", "validation.tar.gz"]
-
-    path = Path(path)
-    if not path.exists():
-        path.mkdir(parents=True)
-
-        download_multi30k_data(urls.values(), path, filenames)
-
-        for filename in filenames:
-            tar = tarfile.open(Path(path) / filename)
-            tar.extractall(path)
-            tar.close()
-
-            print(f'Extracted {filename}')
-
-
-    ret = []
-    filenames = ["train", "val", "test"]
-
-    for filename in filenames:
-
-        examples = []
-
-        en_path = os.path.join(path, filename + '.en')
-        de_path = os.path.join(path, filename + '.de')
-
-        en_file = [l.strip() for l in open(en_path, 'r', encoding='utf-8')]
-        de_file = [l.strip() for l in open(de_path, 'r', encoding='utf-8')]
-
-        assert len(en_file) == len(de_file)
-
-        for i in range(len(en_file)):
-            if en_file[i] == '' or de_file[i] == '':
-                continue
-            en_seq, de_seq = en_file[i], de_file[i]
-
-            examples.append({'en': en_seq, 'de': de_seq})
-    
-        ret.append(examples)
-
-    train_dataset, valid_dataset, test_dataset = ret
-    return train_dataset, valid_dataset, test_dataset
-
-
-
-
-