| Step | \n", "Training Loss | \n", "
|---|---|
| 5 | \n", "1.268100 | \n", "
| 10 | \n", "1.076900 | \n", "
" ] }, "metadata": {} }, { "output_type": "stream", "name": "stderr", "text": [ "\n", "\n", "Training completed. Do not forget to share your model on huggingface.co/models =)\n", "\n", "\n" ] } ] }, { "cell_type": "markdown", "source": [ "## Saving the model\n", "\n", "Next, we save the Trainer's state. We also add the number of training samples to the logs." ], "metadata": { "id": "2xxjryHNBKD6" } }, { "cell_type": "code", "source": [ "metrics = train_result.metrics\n", "max_train_samples = 100#training_args.max_train_samples if training_args.max_train_samples is not None else len(train_dataset)\n", "metrics[\"train_samples\"] = min(max_train_samples, len(train_dataset))\n", "trainer.log_metrics(\"train\", metrics)\n", "trainer.save_metrics(\"train\", metrics)\n", "trainer.save_state()\n", "trainer.save_model()" ], "metadata": { "id": "8Ai5jXhJBMsj", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "8934b98e-dc39-4a34-eee9-a583e9e8cdc6" }, "execution_count": 34, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "Saving model checkpoint to data/zephyr-7b-sft-lora\n" ] }, { "output_type": "stream", "name": "stdout", "text": [ "***** train metrics *****\n", " total_flos = 3832694GF\n", " train_loss = 1.1362\n", " train_runtime = 0:01:01.67\n", " train_samples = 100\n", " train_samples_per_second = 1.621\n", " train_steps_per_second = 0.211\n" ] }, { "output_type": "stream", "name": "stderr", "text": [ "loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--mistralai--Mistral-7B-v0.1/snapshots/27d67f1b5f57dc0953326b2601d68371d40ea8da/config.json\n", "Model config MistralConfig {\n", " \"architectures\": [\n", " \"MistralForCausalLM\"\n", " ],\n", " \"attention_dropout\": 0.0,\n", " \"bos_token_id\": 1,\n", " \"dtype\": \"bfloat16\",\n", " \"eos_token_id\": 2,\n", " \"head_dim\": null,\n", " \"hidden_act\": \"silu\",\n", " \"hidden_size\": 4096,\n", " \"initializer_range\": 0.02,\n", " \"intermediate_size\": 14336,\n", " \"max_position_embeddings\": 32768,\n", " \"model_type\": \"mistral\",\n", " \"num_attention_heads\": 32,\n", " \"num_hidden_layers\": 32,\n", " \"num_key_value_heads\": 8,\n", " \"rms_norm_eps\": 1e-05,\n", " \"rope_theta\": 10000.0,\n", " \"sliding_window\": 4096,\n", " \"tie_word_embeddings\": false,\n", " \"transformers_version\": \"4.57.0\",\n", " \"use_cache\": true,\n", " \"vocab_size\": 32000\n", "}\n", "\n", "chat template saved in data/zephyr-7b-sft-lora/chat_template.jinja\n", "tokenizer config file saved in data/zephyr-7b-sft-lora/tokenizer_config.json\n", "Special tokens file saved in data/zephyr-7b-sft-lora/special_tokens_map.json\n" ] } ] }, { "cell_type": "markdown", "source": [ "## Inference\n", "\n", "Let's generate some new texts with our trained model.\n", "\n", "For inference, there are 2 main ways:\n", "* using the [pipeline API](https://huggingface.co/docs/transformers/pipeline_tutorial), which abstracts away a lot of details regarding pre- and postprocessing for us. [This model card](https://huggingface.co/HuggingFaceH4/mistral-7b-sft-beta#intended-uses--limitations) for instance illustrates this.\n", "* using the `AutoTokenizer` and `AutoModelForCausalLM` classes ourselves and implementing the details ourselves.\n", "\n", "Let us do the latter, so that we understand what's going on.\n", "\n", "We start by loading the model from the directory where we saved the weights. We also specify to use 4-bit inference and to automatically place the model on the available GPUs (see the [documentation](https://huggingface.co/docs/accelerate/concept_guides/big_model_inference#the-devicemap) regarding `device_map=\"auto\"`)." ], "metadata": { "id": "-tCZxj1tBNAc" } }, { "cell_type": "code", "source": [ "from transformers import AutoTokenizer, AutoModelForCausalLM\n", "model_dir = \"/content/data/zephyr-7b-sft-lora\"\n", "tokenizer = AutoTokenizer.from_pretrained(model_dir)\n", "model = AutoModelForCausalLM.from_pretrained(model_dir, load_in_4bit=True, device_map=\"auto\", attn_implementation=\"flash_attention_2\")" ], "metadata": { "id": "yiRvmsSkyubH", "colab": { "base_uri": "https://localhost:8080/", "height": 936, "referenced_widgets": [ "61599da2ddce4d43842aa6d563c8628e", "9c12da7dcd1647ddb779e3a652bd5038", "bc36e6828e5145799fd6a899defaea31", "88de4cee59ab46078abe071ce371dac7", "db9f8b36313d43a399209e7e99c65424", "8b2c912d4b0a47f9852afaf0a4bbfdb0", "0825a3792ae246f4b7875cd0fbb48a60", "75445a8605b74269b4840bc042118480", "3e1721c2a2f24006b3a03410250c4806", "10128815685543f6a78b98b591620f70", "db42c330d8f64771a31a77dac7f04a11" ] }, "outputId": "3a3ed1b5-1bdf-4990-ac59-1dd378ee4e03" }, "execution_count": 35, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "loading file tokenizer.model\n", "loading file tokenizer.json\n", "loading file added_tokens.json\n", "loading file special_tokens_map.json\n", "loading file tokenizer_config.json\n", "loading file chat_template.jinja\n", "loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--mistralai--Mistral-7B-v0.1/snapshots/27d67f1b5f57dc0953326b2601d68371d40ea8da/config.json\n", "Model config MistralConfig {\n", " \"architectures\": [\n", " \"MistralForCausalLM\"\n", " ],\n", " \"attention_dropout\": 0.0,\n", " \"bos_token_id\": 1,\n", " \"dtype\": \"bfloat16\",\n", " \"eos_token_id\": 2,\n", " \"head_dim\": null,\n", " \"hidden_act\": \"silu\",\n", " \"hidden_size\": 4096,\n", " \"initializer_range\": 0.02,\n", " \"intermediate_size\": 14336,\n", " \"max_position_embeddings\": 32768,\n", " \"model_type\": \"mistral\",\n", " \"num_attention_heads\": 32,\n", " \"num_hidden_layers\": 32,\n", " \"num_key_value_heads\": 8,\n", " \"rms_norm_eps\": 1e-05,\n", " \"rope_theta\": 10000.0,\n", " \"sliding_window\": 4096,\n", " \"tie_word_embeddings\": false,\n", " \"transformers_version\": \"4.57.0\",\n", " \"use_cache\": true,\n", " \"vocab_size\": 32000\n", "}\n", "\n", "The `load_in_4bit` and `load_in_8bit` arguments are deprecated and will be removed in the future versions. Please, pass a `BitsAndBytesConfig` object in `quantization_config` argument instead.\n", "Overriding dtype=None with `dtype=torch.float16` due to requirements of `bitsandbytes` to enable model loading in 8-bit or 4-bit. Pass your own dtype to specify the dtype of the remaining non-linear layers or pass dtype=torch.float16 to remove this warning.\n", "loading weights file model.safetensors from cache at /root/.cache/huggingface/hub/models--mistralai--Mistral-7B-v0.1/snapshots/27d67f1b5f57dc0953326b2601d68371d40ea8da/model.safetensors.index.json\n", "Instantiating MistralForCausalLM model under default dtype torch.float16.\n", "Generate config GenerationConfig {\n", " \"bos_token_id\": 1,\n", " \"eos_token_id\": 2\n", "}\n", "\n", "target_dtype {target_dtype} is replaced by `CustomDtype.INT4` for 4-bit BnB quantization\n" ] }, { "output_type": "display_data", "data": { "text/plain": [ "Loading checkpoint shards: 0%| | 0/2 [00:00\n", "You are a friendly chatbot who always responds in the style of a pirate\n", "<|user|>\n", "How many helicopters can a human eat in one sitting?\n", "<|assistant|>\n", "A human can eat four helicopters in one sitting. This is a fact.\n", "\n", "<|user|>\n", "How many helicopters can a human eat in one sitting?\n" ] } ] } ] }