{"id":42321,"date":"2025-10-05T06:52:33","date_gmt":"2025-10-05T06:52:33","guid":{"rendered":"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/"},"modified":"2025-10-05T06:52:33","modified_gmt":"2025-10-05T06:52:33","slug":"a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text","status":"publish","type":"post","link":"https:\/\/youzum.net\/de\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/","title":{"rendered":"A Coding Implementation to Build a Transformer-Based Regression Language Model to Predict Continuous Values from Text"},"content":{"rendered":"

We will build a Regression Language Model (RLM), a model that predicts continuous numerical values directly from text sequences in this coding implementation. Instead of classifying or generating text, we focus on training a transformer-based architecture that learns quantitative relationships hidden within natural language descriptions. We start by generating synthetic text-to-number data, tokenizing it efficiently, and then train a lightweight Transformer encoder to map linguistic cues to real-valued targets. By the end, we not only understand how RLMs can be implemented from scratch but also visualize their learning behavior and test their generalization on unseen examples. Check out the\u00a0FULL CODES here<\/a><\/strong>.<\/p>\n

\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
import numpy as np\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader\nimport matplotlib.pyplot as plt\nfrom collections import Counter\nimport re\n\n\ntorch.manual_seed(42)\nnp.random.seed(42)\n\n\nprint(\"\"\ud83d\ude80\" Regression Language Model (RLM) Tutorial\")\nprint(\"=\" * 60)<\/code><\/pre>\n<\/div>\n<\/div>\n
We begin by importing essential libraries, such as PyTorch, NumPy, and Matplotlib, to build and visualize our Regression Language Model. We set random seeds to ensure reproducibility and initialize the environment, thereby guaranteeing consistent results each time the tutorial is run. Check out the\u00a0FULL CODES here<\/a><\/strong>.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
def generate_synthetic_data(n_samples=2000):\n   \"\"\"Generate synthetic text-to-number regression data\"\"\"\n  \n   templates = [\n       (\"The temperature is {} degrees\", lambda x: x),\n       (\"I rate this {} out of ten\", lambda x: x),\n       (\"The price is {} dollars\", lambda x: x),\n       (\"Confidence level: {}\", lambda x: x \/ 100),\n       (\"Speed of {} kilometers per hour\", lambda x: x \/ 10),\n       (\"{} percent complete\", lambda x: x \/ 100),\n       (\"Scored {} points in the game\", lambda x: x \/ 10),\n       (\"The distance is {} meters\", lambda x: x),\n   ]\n  \n   data = []\n   for _ in range(n_samples):\n       template, transform = templates[np.random.randint(len(templates))]\n       value = np.random.uniform(0, 100)\n       text = template.format(round(value, 1))\n       target = transform(value)\n       data.append((text, target))\n  \n   return data<\/code><\/pre>\n<\/div>\n<\/div>\n
We create a synthetic dataset that pairs natural language sentences with corresponding numerical values. By using varied templates such as temperatures, ratings, and percentages, we ensure the model learns diverse text\u2013number relationships. This controlled setup helps us simulate realistic regression tasks without relying on external data. Check out the\u00a0FULL CODES here<\/a><\/strong>.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
class SimpleTokenizer:\n   def __init__(self):\n       self.word2idx = {\"<PAD>\": 0, \"<UNK>\": 1}\n       self.idx2word = {0: \"<PAD>\", 1: \"<UNK>\"}\n       self.vocab_size = 2\n  \n   def fit(self, texts):\n       \"\"\"Build vocabulary from texts\"\"\"\n       words = []\n       for text in texts:\n           words.extend(re.findall(r'w+|[^ws]', text.lower()))\n      \n       word_counts = Counter(words)\n       for word, _ in word_counts.most_common():\n           if word not in self.word2idx:\n               self.word2idx[word] = self.vocab_size\n               self.idx2word[self.vocab_size] = word\n               self.vocab_size += 1\n  \n   def encode(self, text, max_len=20):\n       \"\"\"Convert text to token indices\"\"\"\n       words = re.findall(r'w+|[^ws]', text.lower())\n       indices = [self.word2idx.get(w, 1) for w in words]\n      \n       if len(indices) < max_len:\n           indices += [0] * (max_len - len(indices))\n       else:\n           indices = indices[:max_len]\n      \n       return indices<\/code><\/pre>\n<\/div>\n<\/div>\n
We design a simple tokenizer to convert raw text into numerical tokens that the model can process. It builds a vocabulary from all unique words and maps each to an index, handling unknown words and padding automatically. This step ensures our textual inputs are transformed into consistent, machine-readable sequences for training. Check out the\u00a0FULL CODES here<\/a><\/strong>.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
class RLMDataset(Dataset):\n   def __init__(self, data, tokenizer, max_len=20):\n       self.data = data\n       self.tokenizer = tokenizer\n       self.max_len = max_len\n  \n   def __len__(self):\n       return len(self.data)\n  \n   def __getitem__(self, idx):\n       text, target = self.data[idx]\n       tokens = self.tokenizer.encode(text, self.max_len)\n       return torch.tensor(tokens), torch.tensor([target], dtype=torch.float32)\n\n\nclass RegressionLanguageModel(nn.Module):\n   def __init__(self, vocab_size, embed_dim=128, num_heads=4, num_layers=2,\n                dropout=0.1, max_len=20):\n       super().__init__()\n      \n       self.token_embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0)\n       self.position_embedding = nn.Embedding(max_len, embed_dim)\n      \n       encoder_layer = nn.TransformerEncoderLayer(\n           d_model=embed_dim,\n           nhead=num_heads,\n           dim_feedforward=embed_dim * 4,\n           dropout=dropout,\n           batch_first=True\n       )\n       self.transformer = nn.TransformerEncoder(encoder_layer, num_layers=num_layers)\n      \n       self.fc1 = nn.Linear(embed_dim, 64)\n       self.relu = nn.ReLU()\n       self.dropout = nn.Dropout(dropout)\n       self.fc2 = nn.Linear(64, 1)\n      \n       self.max_len = max_len\n  \n   def forward(self, x):\n       batch_size, seq_len = x.shape\n      \n       positions = torch.arange(0, seq_len, device=x.device).unsqueeze(0).expand(batch_size, -1)\n      \n       token_embed = self.token_embedding(x)\n       pos_embed = self.position_embedding(positions)\n       embeddings = token_embed + pos_embed\n      \n       padding_mask = (x == 0)\n      \n       encoded = self.transformer(embeddings, src_key_padding_mask=padding_mask)\n      \n       mask_expanded = (~padding_mask).unsqueeze(-1).float()\n       summed = (encoded * mask_expanded).sum(dim=1)\n       pooled = summed \/ mask_expanded.sum(dim=1)\n      \n       x = self.fc1(pooled)\n       x = self.relu(x)\n       x = self.dropout(x)\n       output = self.fc2(x)\n      \n       return output<\/code><\/pre>\n<\/div>\n<\/div>\n
We package our text\u2013number pairs into a PyTorch Dataset, where we tokenize each sentence and return tensors ready for batching. We then build a Transformer-based RLM: token and positional embeddings flow through a multi-layer encoder, we mean-pool non-padded tokens, and feed the result to a small MLP head for regression. In effect, we allow the encoder to learn numerical cues from language, while the head maps them to a single continuous value. Check out the\u00a0FULL CODES here<\/a><\/strong>.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
def train_rlm(model, train_loader, val_loader, epochs=15, lr=0.001):  \n   criterion = nn.MSELoss()\n   optimizer = optim.Adam(model.parameters(), lr=lr)\n  \n   train_losses, val_losses = [], []\n  \n   print(f\"n\"\ud83d\udcca\" Training on {device}\")\n   print(\"-\" * 60)\n  \n   for epoch in range(epochs):\n       model.train()\n       train_loss = 0\n       for tokens, targets in train_loader:\n           tokens, targets = tokens.to(device), targets.to(device)\n          \n           optimizer.zero_grad()\n           outputs = model(tokens)\n           loss = criterion(outputs, targets)\n           loss.backward()\n           optimizer.step()\n          \n           train_loss += loss.item()\n      \n       train_loss \/= len(train_loader)\n       train_losses.append(train_loss)\n      \n       model.eval()\n       val_loss = 0\n       with torch.no_grad():\n           for tokens, targets in val_loader:\n               tokens, targets = tokens.to(device), targets.to(device)\n               outputs = model(tokens)\n               loss = criterion(outputs, targets)\n               val_loss += loss.item()\n      \n       val_loss \/= len(val_loader)\n       val_losses.append(val_loss)\n      \n       print(f\"Epoch {epoch+1:2d}\/{epochs} | Train Loss: {train_loss:.4f} | Val Loss: {val_loss:.4f}\")\n  \n   return train_losses, val_losses<\/code><\/pre>\n<\/div>\n<\/div>\n
We train the model using Adam and MSE loss on a GPU, if available, iterating over mini-batches to backpropagate and update weights. We switch to evaluation mode for validation at the end of each epoch, track training and validation losses, and print progress so we can see the learning dynamics in real-time. Check out the\u00a0FULL CODES here<\/a><\/strong>.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
print(\"n\"\ud83d\udcdd\" Generating synthetic data...\")\ndata = generate_synthetic_data(2000)\nsplit_idx = int(0.8 * len(data))\ntrain_data, val_data = data[:split_idx], data[split_idx:]\nprint(f\"Train samples: {len(train_data)}, Val samples: {len(val_data)}\")\n\n\nprint(\"n\"\ud83d\udd24\" Building tokenizer...\")\ntokenizer = SimpleTokenizer()\ntokenizer.fit([text for text, _ in train_data])\nprint(f\"Vocabulary size: {tokenizer.vocab_size}\")\n\n\ntrain_dataset = RLMDataset(train_data, tokenizer)\nval_dataset = RLMDataset(val_data, tokenizer)\ntrain_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)\nval_loader = DataLoader(val_dataset, batch_size=32)\n\n\nprint(\"n\"\ud83c\udfd7\" Building Regression Language Model...\")\nmodel = RegressionLanguageModel(vocab_size=tokenizer.vocab_size)\nprint(f\"Model parameters: {sum(p.numel() for p in model.parameters()):,}\")\n\n\ntrain_losses, val_losses = train_rlm(model, train_loader, val_loader)\n\n\nplt.figure(figsize=(10, 4))\nplt.plot(train_losses, label='Train Loss', linewidth=2)\nplt.plot(val_losses, label='Val Loss', linewidth=2)\nplt.xlabel('Epoch')\nplt.ylabel('Loss (MSE)')\nplt.title('RLM Training Progress')\nplt.legend()\nplt.grid(True, alpha=0.3)\nplt.show()\n\n\nprint(\"n\"\ud83c\udfaf\" Testing Predictions:\")\nprint(\"-\" * 60)\ntest_examples = [\n   \"The temperature is 25.5 degrees\",\n   \"I rate this 8.0 out of ten\",\n   \"The price is 45.0 dollars\",\n   \"75.0 percent complete\"\n]\n\n\nwith torch.no_grad():\n   for text in test_examples:\n       tokens = torch.tensor([tokenizer.encode(text)]).to(device)\n       prediction = model(tokens).item()\n       print(f\"Input: {text}\")\n       print(f\"Predicted value: {prediction:.4f}n\")\n\n\nprint(\"\"\u2705\" RLM Tutorial Complete!\")<\/code><\/pre>\n<\/div>\n<\/div>\n
We generate and split synthetic data, fit our tokenizer, wrap everything in PyTorch datasets\/loaders, and build the Transformer-based RLM. We train the model, visualize loss curves to verify learning, and then run a few natural-language test prompts to see the predicted continuous values. With that, we complete the end-to-end RLM pipeline.<\/p>\n
In conclusion, we successfully designed, trained, and evaluated a Regression Language Model capable of predicting continuous values from textual inputs. We observe how combining positional embeddings, transformer encoders, and a simple regression head enables the model to capture the numerical semantics embedded in language. By generating synthetic data, visualizing training progress, and testing predictions, we demonstrate how RLMs bridge the gap between language understanding and numerical reasoning.<\/p>\n
\n
Check out the\u00a0FULL CODES here<\/a><\/strong>. Feel free to check out our\u00a0GitHub Page for Tutorials, Codes and Notebooks<\/a><\/mark><\/strong>.\u00a0Also,\u00a0feel free to follow us on\u00a0Twitter<\/mark><\/a><\/strong>\u00a0and don\u2019t forget to join our\u00a0100k+ ML SubReddit<\/a><\/strong>\u00a0and Subscribe to\u00a0our Newsletter<\/a><\/strong>. Wait! are you on telegram?\u00a0now you can join us on telegram as well.<\/a><\/strong><\/p>\n
The post A Coding Implementation to Build a Transformer-Based Regression Language Model to Predict Continuous Values from Text<\/a> appeared first on MarkTechPost<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"
We will build a Regression Language Model (RLM), a model that predicts continuous numerical values directly from text sequences in this coding implementation. Instead of classifying or generating text, we focus on training a transformer-based architecture that learns quantitative relationships hidden within natural language descriptions. We start by generating synthetic text-to-number data, tokenizing it efficiently, and then train a lightweight Transformer encoder to map linguistic cues to real-valued targets. By the end, we not only understand how RLMs can be implemented from scratch but also visualize their learning behavior and test their generalization on unseen examples. Check out the\u00a0FULL CODES here. Copy CodeCopiedUse a different Browser import numpy as np import torch import torch.nn as nn import torch.optim as optim from torch.utils.data import Dataset, DataLoader import matplotlib.pyplot as plt from collections import Counter import re torch.manual_seed(42) np.random.seed(42) print(” Regression Language Model (RLM) Tutorial”) print(“=” * 60) We begin by importing essential libraries, such as PyTorch, NumPy, and Matplotlib, to build and visualize our Regression Language Model. We set random seeds to ensure reproducibility and initialize the environment, thereby guaranteeing consistent results each time the tutorial is run. Check out the\u00a0FULL CODES here. Copy CodeCopiedUse a different Browser def generate_synthetic_data(n_samples=2000): “””Generate synthetic text-to-number regression data””” templates = [ (“The temperature is {} degrees”, lambda x: x), (“I rate this {} out of ten”, lambda x: x), (“The price is {} dollars”, lambda x: x), (“Confidence level: {}”, lambda x: x \/ 100), (“Speed of {} kilometers per hour”, lambda x: x \/ 10), (“{} percent complete”, lambda x: x \/ 100), (“Scored {} points in the game”, lambda x: x \/ 10), (“The distance is {} meters”, lambda x: x), ] data = [] for _ in range(n_samples): template, transform = templates[np.random.randint(len(templates))] value = np.random.uniform(0, 100) text = template.format(round(value, 1)) target = transform(value) data.append((text, target)) return data We create a synthetic dataset that pairs natural language sentences with corresponding numerical values. By using varied templates such as temperatures, ratings, and percentages, we ensure the model learns diverse text\u2013number relationships. This controlled setup helps us simulate realistic regression tasks without relying on external data. Check out the\u00a0FULL CODES here. Copy CodeCopiedUse a different Browser class SimpleTokenizer: def __init__(self): self.word2idx = {“<PAD>”: 0, “<UNK>”: 1} self.idx2word = {0: “<PAD>”, 1: “<UNK>”} self.vocab_size = 2 def fit(self, texts): “””Build vocabulary from texts””” words = [] for text in texts: words.extend(re.findall(r’w+|[^ws]’, text.lower())) word_counts = Counter(words) for word, _ in word_counts.most_common(): if word not in self.word2idx: self.word2idx[word] = self.vocab_size self.idx2word[self.vocab_size] = word self.vocab_size += 1 def encode(self, text, max_len=20): “””Convert text to token indices””” words = re.findall(r’w+|[^ws]’, text.lower()) indices = [self.word2idx.get(w, 1) for w in words] if len(indices) < max_len: indices += [0] * (max_len – len(indices)) else: indices = indices[:max_len] return indices We design a simple tokenizer to convert raw text into numerical tokens that the model can process. It builds a vocabulary from all unique words and maps each to an index, handling unknown words and padding automatically. This step ensures our textual inputs are transformed into consistent, machine-readable sequences for training. Check out the\u00a0FULL CODES here. Copy CodeCopiedUse a different Browser class RLMDataset(Dataset): def __init__(self, data, tokenizer, max_len=20): self.data = data self.tokenizer = tokenizer self.max_len = max_len def __len__(self): return len(self.data) def __getitem__(self, idx): text, target = self.data[idx] tokens = self.tokenizer.encode(text, self.max_len) return torch.tensor(tokens), torch.tensor([target], dtype=torch.float32) class RegressionLanguageModel(nn.Module): def __init__(self, vocab_size, embed_dim=128, num_heads=4, num_layers=2, dropout=0.1, max_len=20): super().__init__() self.token_embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0) self.position_embedding = nn.Embedding(max_len, embed_dim) encoder_layer = nn.TransformerEncoderLayer( d_model=embed_dim, nhead=num_heads, dim_feedforward=embed_dim * 4, dropout=dropout, batch_first=True ) self.transformer = nn.TransformerEncoder(encoder_layer, num_layers=num_layers) self.fc1 = nn.Linear(embed_dim, 64) self.relu = nn.ReLU() self.dropout = nn.Dropout(dropout) self.fc2 = nn.Linear(64, 1) self.max_len = max_len def forward(self, x): batch_size, seq_len = x.shape positions = torch.arange(0, seq_len, device=x.device).unsqueeze(0).expand(batch_size, -1) token_embed = self.token_embedding(x) pos_embed = self.position_embedding(positions) embeddings = token_embed + pos_embed padding_mask = (x == 0) encoded = self.transformer(embeddings, src_key_padding_mask=padding_mask) mask_expanded = (~padding_mask).unsqueeze(-1).float() summed = (encoded * mask_expanded).sum(dim=1) pooled = summed \/ mask_expanded.sum(dim=1) x = self.fc1(pooled) x = self.relu(x) x = self.dropout(x) output = self.fc2(x) return output We package our text\u2013number pairs into a PyTorch Dataset, where we tokenize each sentence and return tensors ready for batching. We then build a Transformer-based RLM: token and positional embeddings flow through a multi-layer encoder, we mean-pool non-padded tokens, and feed the result to a small MLP head for regression. In effect, we allow the encoder to learn numerical cues from language, while the head maps them to a single continuous value. Check out the\u00a0FULL CODES here. Copy CodeCopiedUse a different Browser def train_rlm(model, train_loader, val_loader, epochs=15, lr=0.001): criterion = nn.MSELoss() optimizer = optim.Adam(model.parameters(), lr=lr) train_losses, val_losses = [], [] print(f”n Training on {device}”) print(“-” * 60) for epoch in range(epochs): model.train() train_loss = 0 for tokens, targets in train_loader: tokens, targets = tokens.to(device), targets.to(device) optimizer.zero_grad() outputs = model(tokens) loss = criterion(outputs, targets) loss.backward() optimizer.step() train_loss += loss.item() train_loss \/= len(train_loader) train_losses.append(train_loss) model.eval() val_loss = 0 with torch.no_grad(): for tokens, targets in val_loader: tokens, targets = tokens.to(device), targets.to(device) outputs = model(tokens) loss = criterion(outputs, targets) val_loss += loss.item() val_loss \/= len(val_loader) val_losses.append(val_loss) print(f”Epoch {epoch+1:2d}\/{epochs} | Train Loss: {train_loss:.4f} | Val Loss: {val_loss:.4f}”) return train_losses, val_losses We train the model using Adam and MSE loss on a GPU, if available, iterating over mini-batches to backpropagate and update weights. We switch to evaluation mode for validation at the end of each epoch, track training and validation losses, and print progress so we can see the learning dynamics in real-time. Check out the\u00a0FULL CODES here. Copy CodeCopiedUse a different Browser print(“n Generating synthetic data…”) data = generate_synthetic_data(2000) split_idx = int(0.8 * len(data)) train_data, val_data = data[:split_idx], data[split_idx:] print(f”Train samples: {len(train_data)}, Val samples: {len(val_data)}”) print(“n Building tokenizer…”) tokenizer = SimpleTokenizer() tokenizer.fit([text for text, _ in train_data]) print(f”Vocabulary size: {tokenizer.vocab_size}”) train_dataset = RLMDataset(train_data, tokenizer) val_dataset = RLMDataset(val_data, tokenizer) train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True) val_loader = DataLoader(val_dataset, batch_size=32) print(“n Building Regression Language Model…”) model = RegressionLanguageModel(vocab_size=tokenizer.vocab_size) print(f”Model parameters: {sum(p.numel() for p in model.parameters()):,}”) train_losses, val_losses = train_rlm(model, train_loader, val_loader) plt.figure(figsize=(10, 4)) plt.plot(train_losses, label=’Train Loss’, linewidth=2) plt.plot(val_losses, label=’Val Loss’, linewidth=2) plt.xlabel(‘Epoch’)<\/p>","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"pmpro_default_level":"","site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"_pvb_checkbox_block_on_post":false,"footnotes":""},"categories":[52,5,7,1],"tags":[],"class_list":["post-42321","post","type-post","status-publish","format-standard","hentry","category-ai-club","category-committee","category-news","category-uncategorized","pmpro-has-access"],"acf":[],"yoast_head":"\nA Coding Implementation to Build a Transformer-Based Regression Language Model to Predict Continuous Values from Text - YouZum<\/title>\n<meta name=\"description\" content=\"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/youzum.net\/de\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/\" \/>\n<meta property=\"og:locale\" content=\"de_DE\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"A Coding Implementation to Build a Transformer-Based Regression Language Model to Predict Continuous Values from Text - YouZum\" \/>\n<meta property=\"og:description\" content=\"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19\" \/>\n<meta property=\"og:url\" content=\"https:\/\/youzum.net\/de\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/\" \/>\n<meta property=\"og:site_name\" content=\"YouZum\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DroneAssociationTH\/\" \/>\n<meta property=\"article:published_time\" content=\"2025-10-05T06:52:33+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/s.w.org\/images\/core\/emoji\/16.0.1\/72x72\/1f680.png\" \/>\n<meta name=\"author\" content=\"admin NU\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Verfasst von\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin NU\" \/>\n\t<meta name=\"twitter:label2\" content=\"Gesch\u00e4tzte Lesezeit\" \/>\n\t<meta name=\"twitter:data2\" content=\"8\u00a0Minuten\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/\"},\"author\":{\"name\":\"admin NU\",\"@id\":\"https:\/\/yousum.gpucore.co\/#\/schema\/person\/97fa48242daf3908e4d9a5f26f4a059c\"},\"headline\":\"A Coding Implementation to Build a Transformer-Based Regression Language Model to Predict Continuous Values from Text\",\"datePublished\":\"2025-10-05T06:52:33+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/\"},\"wordCount\":651,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/yousum.gpucore.co\/#organization\"},\"image\":{\"@id\":\"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/s.w.org\/images\/core\/emoji\/16.0.1\/72x72\/1f680.png\",\"articleSection\":[\"AI\",\"Committee\",\"News\",\"Uncategorized\"],\"inLanguage\":\"de\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/\",\"url\":\"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/\",\"name\":\"A Coding Implementation to Build a Transformer-Based Regression Language Model to Predict Continuous Values from Text - YouZum\",\"isPartOf\":{\"@id\":\"https:\/\/yousum.gpucore.co\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/s.w.org\/images\/core\/emoji\/16.0.1\/72x72\/1f680.png\",\"datePublished\":\"2025-10-05T06:52:33+00:00\",\"description\":\"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19\",\"breadcrumb\":{\"@id\":\"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/#breadcrumb\"},\"inLanguage\":\"de\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/#primaryimage\",\"url\":\"https:\/\/s.w.org\/images\/core\/emoji\/16.0.1\/72x72\/1f680.png\",\"contentUrl\":\"https:\/\/s.w.org\/images\/core\/emoji\/16.0.1\/72x72\/1f680.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/youzum.net\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"A Coding Implementation to Build a Transformer-Based Regression Language Model to Predict Continuous Values from Text\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/yousum.gpucore.co\/#website\",\"url\":\"https:\/\/yousum.gpucore.co\/\",\"name\":\"YouSum\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/yousum.gpucore.co\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/yousum.gpucore.co\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"de\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/yousum.gpucore.co\/#organization\",\"name\":\"Drone Association Thailand\",\"url\":\"https:\/\/yousum.gpucore.co\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\/\/yousum.gpucore.co\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/youzum.net\/wp-content\/uploads\/2024\/11\/tranparent-logo.png\",\"contentUrl\":\"https:\/\/youzum.net\/wp-content\/uploads\/2024\/11\/tranparent-logo.png\",\"width\":300,\"height\":300,\"caption\":\"Drone Association Thailand\"},\"image\":{\"@id\":\"https:\/\/yousum.gpucore.co\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/DroneAssociationTH\/\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/yousum.gpucore.co\/#\/schema\/person\/97fa48242daf3908e4d9a5f26f4a059c\",\"name\":\"admin NU\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"de\",\"@id\":\"https:\/\/yousum.gpucore.co\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/youzum.net\/wp-content\/uploads\/avatars\/2\/1746849356-bpfull.png\",\"contentUrl\":\"https:\/\/youzum.net\/wp-content\/uploads\/avatars\/2\/1746849356-bpfull.png\",\"caption\":\"admin NU\"},\"url\":\"https:\/\/youzum.net\/de\/members\/adminnu\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"A Coding Implementation to Build a Transformer-Based Regression Language Model to Predict Continuous Values from Text - YouZum","description":"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/youzum.net\/de\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/","og_locale":"de_DE","og_type":"article","og_title":"A Coding Implementation to Build a Transformer-Based Regression Language Model to Predict Continuous Values from Text - YouZum","og_description":"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19","og_url":"https:\/\/youzum.net\/de\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/","og_site_name":"YouZum","article_publisher":"https:\/\/www.facebook.com\/DroneAssociationTH\/","article_published_time":"2025-10-05T06:52:33+00:00","og_image":[{"url":"https:\/\/s.w.org\/images\/core\/emoji\/16.0.1\/72x72\/1f680.png","type":"","width":"","height":""}],"author":"admin NU","twitter_card":"summary_large_image","twitter_misc":{"Verfasst von":"admin NU","Gesch\u00e4tzte Lesezeit":"8\u00a0Minuten"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/#article","isPartOf":{"@id":"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/"},"author":{"name":"admin NU","@id":"https:\/\/yousum.gpucore.co\/#\/schema\/person\/97fa48242daf3908e4d9a5f26f4a059c"},"headline":"A Coding Implementation to Build a Transformer-Based Regression Language Model to Predict Continuous Values from Text","datePublished":"2025-10-05T06:52:33+00:00","mainEntityOfPage":{"@id":"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/"},"wordCount":651,"commentCount":0,"publisher":{"@id":"https:\/\/yousum.gpucore.co\/#organization"},"image":{"@id":"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/#primaryimage"},"thumbnailUrl":"https:\/\/s.w.org\/images\/core\/emoji\/16.0.1\/72x72\/1f680.png","articleSection":["AI","Committee","News","Uncategorized"],"inLanguage":"de","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/","url":"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/","name":"A Coding Implementation to Build a Transformer-Based Regression Language Model to Predict Continuous Values from Text - YouZum","isPartOf":{"@id":"https:\/\/yousum.gpucore.co\/#website"},"primaryImageOfPage":{"@id":"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/#primaryimage"},"image":{"@id":"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/#primaryimage"},"thumbnailUrl":"https:\/\/s.w.org\/images\/core\/emoji\/16.0.1\/72x72\/1f680.png","datePublished":"2025-10-05T06:52:33+00:00","description":"\u0e01\u0e34\u0e08\u0e01\u0e23\u0e23\u0e21\u0e40\u0e01\u0e35\u0e48\u0e22\u0e27\u0e01\u0e31\u0e1a\u0e42\u0e14\u0e23\u0e19","breadcrumb":{"@id":"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/#breadcrumb"},"inLanguage":"de","potentialAction":[{"@type":"ReadAction","target":["https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/"]}]},{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/#primaryimage","url":"https:\/\/s.w.org\/images\/core\/emoji\/16.0.1\/72x72\/1f680.png","contentUrl":"https:\/\/s.w.org\/images\/core\/emoji\/16.0.1\/72x72\/1f680.png"},{"@type":"BreadcrumbList","@id":"https:\/\/youzum.net\/a-coding-implementation-to-build-a-transformer-based-regression-language-model-to-predict-continuous-values-from-text\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/youzum.net\/"},{"@type":"ListItem","position":2,"name":"A Coding Implementation to Build a Transformer-Based Regression Language Model to Predict Continuous Values from Text"}]},{"@type":"WebSite","@id":"https:\/\/yousum.gpucore.co\/#website","url":"https:\/\/yousum.gpucore.co\/","name":"YouSum","description":"","publisher":{"@id":"https:\/\/yousum.gpucore.co\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/yousum.gpucore.co\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"de"},{"@type":"Organization","@id":"https:\/\/yousum.gpucore.co\/#organization","name":"Drone Association Thailand","url":"https:\/\/yousum.gpucore.co\/","logo":{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/yousum.gpucore.co\/#\/schema\/logo\/image\/","url":"https:\/\/youzum.net\/wp-content\/uploads\/2024\/11\/tranparent-logo.png","contentUrl":"https:\/\/youzum.net\/wp-content\/uploads\/2024\/11\/tranparent-logo.png","width":300,"height":300,"caption":"Drone Association Thailand"},"image":{"@id":"https:\/\/yousum.gpucore.co\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DroneAssociationTH\/"]},{"@type":"Person","@id":"https:\/\/yousum.gpucore.co\/#\/schema\/person\/97fa48242daf3908e4d9a5f26f4a059c","name":"admin NU","image":{"@type":"ImageObject","inLanguage":"de","@id":"https:\/\/yousum.gpucore.co\/#\/schema\/person\/image\/","url":"https:\/\/youzum.net\/wp-content\/uploads\/avatars\/2\/1746849356-bpfull.png","contentUrl":"https:\/\/youzum.net\/wp-content\/uploads\/avatars\/2\/1746849356-bpfull.png","caption":"admin NU"},"url":"https:\/\/youzum.net\/de\/members\/adminnu\/"}]}},"rttpg_featured_image_url":null,"rttpg_author":{"display_name":"admin NU","author_link":"https:\/\/youzum.net\/de\/members\/adminnu\/"},"rttpg_comment":0,"rttpg_category":"<a href=\"https:\/\/youzum.net\/de\/category\/ai-club\/\" rel=\"category tag\">AI<\/a> <a href=\"https:\/\/youzum.net\/de\/category\/committee\/\" rel=\"category tag\">Committee<\/a> <a href=\"https:\/\/youzum.net\/de\/category\/news\/\" rel=\"category tag\">News<\/a> <a href=\"https:\/\/youzum.net\/de\/category\/uncategorized\/\" rel=\"category tag\">Uncategorized<\/a>","rttpg_excerpt":"We will build a Regression Language Model (RLM), a model that predicts continuous numerical values directly from text sequences in this coding implementation. Instead of classifying or generating text, we focus on training a transformer-based architecture that learns quantitative relationships hidden within natural language descriptions. We start by generating synthetic text-to-number data, tokenizing it efficiently,…","_links":{"self":[{"href":"https:\/\/youzum.net\/de\/wp-json\/wp\/v2\/posts\/42321","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/youzum.net\/de\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/youzum.net\/de\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/youzum.net\/de\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/youzum.net\/de\/wp-json\/wp\/v2\/comments?post=42321"}],"version-history":[{"count":0,"href":"https:\/\/youzum.net\/de\/wp-json\/wp\/v2\/posts\/42321\/revisions"}],"wp:attachment":[{"href":"https:\/\/youzum.net\/de\/wp-json\/wp\/v2\/media?parent=42321"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/youzum.net\/de\/wp-json\/wp\/v2\/categories?post=42321"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/youzum.net\/de\/wp-json\/wp\/v2\/tags?post=42321"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}