Tarea 7 - Prompting Strategies with Open-Source LLMs

Tarea: Prompting Strategies with Open-Source LLMs

Objective

Design, implement, and evaluate different prompting strategies for an NLP task e.g., Sentiment Analysis, Question Answering, etc., using open-source Large Language Models (LLMs).

You will explore how prompting techniques such as:

• Direct prompting

• Chain-of-Thought (CoT)

• Self-consistency

• Tree-of-Thought (ToT)

affect model performance.

Task Description

You must:

1. Select a public dataset (small portion)

2. Choose at least one open-source LLM

3. Design multiple prompting strategies

4. Run experiments and evaluate performance

5. Analyze results and explain behavior

Dataset

You may choose any dataset, for example:

• IMDb (binary sentiment)

• SST-2 (Stanford Sentiment Treebank)

• Twitter sentiment datasets

• Amazon reviews

• OR create your own small dataset (minimum 5

Models

You can use:

• Qwen (e.g., Qwen2.5-3B / 7B)

• Mistral (e.g., Mistral-7B-Instruct)

• Gemma

• LLaMA (if accessible)

• Any HuggingFace model

🧪 Required Experiments

You must implement at least 4 of the following:

1. 🔹 Direct Prompting (Baseline)

Example (could be different for other tasks):

Classify the sentiment:
Text: {text}
Answer: Positive / Negative / Neutral

2. 🔹 Role Prompting

Example:

You are a sentiment analysis expert...

3. 🔹 Zero-shot Chain-of-Thought (CoT)

Example:

Think step by step before answering.

4. 🔹 Few-shot Prompting

Provide 2–5 examples in the prompt.

5. 🔹 Few-shot CoT

Combine:

• examples

• reasoning steps

6. 🔹 Self-Consistency

• Sample multiple outputs (e.g., 5–10)

• Use majority voting

7. 🔹 Tree-of-Thought (ToT)

• Generate multiple reasoning paths

• Select best candidate

⚠️ Simple implementations are fine

⚙️ Evaluation

You must report:

• Accuracy

• Macro F1-score

• Confusion Matrix

• Or any relevant metric

📈 Analysis Questions (IMPORTANT)

Answer these in your report:

1. Which prompting strategy worked best? Why?

2. Did CoT improve performance? When and why?

3. Did self-consistency help? Or hurt?

4. When did prompting fail?

5. How sensitive were results to prompt design?

6. Did model size affect results?

7. Is reasoning actually needed for sentiment analysis?

📄 Deliverables

1. Code

• Colab / Notebook

• Well commented

• Reproducible

2. Report 

Include:

• Dataset description

• Model(s) used

• Prompt designs

• Results table

• Error analysis

• Answers to analysis questions

3. Results Table (example)

Bonus (optional)

• Compare 2 different LLMs

• Try multilingual sentiment

• Try sarcasm / hard examples

• Perform error clustering