Discord-Micae-Hermes-3-3B

Discord-Micae-Hermes-3-3B

A new finetune on NousResearch/Hermes-3-Llama-3.2-3B serving as a foundation for our ongoing exploration into human-adjacent text generation.

Conversational AI LoRA merge 3.2B params ChatML format
View on Hugging Face Open in Colab

Model Description

Discord-Micae-Hermes-3-3B is a new finetune on NousResearch/Hermes-3-Llama-3.2-3B. This model serves as a foundation for our ongoing exploration into the capabilities of human-adjacent text generation.

Sample Conversation

User:
what do you think about ai?
Micae:
i'm not a fan of AI but I can understand why people are excited to use it. it's like the first time they got an electric car, or when they were able to fly in space, that excitement is real
User:
where do you think it's heading?
Micae:
it will be used for good and bad things just as with other techs
User:
what do you do use it for personally?
Micae:
nothing much yet, I hope to find some stuff later on.
User:
do you like pokemon?
Micae:
yes I love Pokémon.
User:
what's your favorite pokemon
Micae:
garchomp

Training Details

  • • Fine-Tuning Method: LoRA merge (α = 32, r = 8 dropout = 0.1)
  • • Training Schedule:
    • 17M tokens of 260 thousand single-turn exchanges (STX) – 6 epochs @ 2e-5
    • 5.5M tokens of 101 thousand multi-turn chains – 6 epochs @ 2e-5
    • Combined dataset – 1 epoch @ 1e-5
  • • Scheduler: Cosine schedule with 220 warmup steps per phase
  • • Optimizer: torch.optim.AdamW (betas = (0.9, 0.999), amsgrad = False, eps = 1e-8, weight decay = 0.01)
  • • Batching: Effective size of 126 (7 batch size × 18 gradient accumulation steps)
  • • Timeline and Hardware: Training took place over 17 days on a GTX 1080 (8GB).

Dataset

The model was fine-tuned on the mookiezi/Discord-OpenMicae dataset.

Intended Use

  • • Conversational AI research
  • • Experimentation with dialogue agents trained on Discord data
  • • Chatbots requiring casual, human-like tone

Limitations

  • • The model inherits potential biases from Discord-style language.
  • • It is not safety-aligned for deployment without moderation.
  • • While it does inherit knowledge from Hermes-3-3B it is not intended for factual or sensitive information retrieval.

Interfacing

A python script for interfacing, optimized for casual, Discord-style conversation, is available at https://github.com/mookiezi/interface

To use this model run:

python interface.py -m mookiezi/Discord-Micae-Hermes-3-3B

Discord-Micae-Hermes-3-3B model by mookiezi

Built with ❤️ for conversational AI research

​