AI

arXiv:2601.22795v1 Announce Type: new Abstract: Transformer-based large language models (LLMs) are comprised of billions of parameters arranged in deep and wide computational graphs. Several studies on LLM efficiency optimization argue that it is possible to prune a significant portion of the parameters, while only marginally impacting performance. This suggests that the computation is not uniformly distributed across the parameters. We introduce here a technique to systematically quantify computation density in LLMs. In particular, we design a density estimator drawing on mechanistic interpretability. We experimentally test our estimator and find that: (1) contrary to what has been often assumed, LLM processing generally involves dense computation; (2) computation density is dynamic, in the sense that models shift between sparse and dense processing regimes depending on the input; (3) per-input density is significantly correlated across LLMs, suggesting that the same inputs trigger either low or high density. Investigating the factors influencing density, we observe that predicting rarer tokens requires higher density, and increasing context length often decreases the density. We believe that our computation density estimator will contribute to a better understanding of the processing at work in LLMs, challenging their symbolic interpretation.

arXiv:2505.17595v3 Announce Type: replace-cross Abstract: Large language models (LLMs) achieve impressive performance across domains but face significant challenges when deployed on consumer-grade GPUs or personal devices such as laptops, due to high memory consumption and inference costs. Post-training quantization (PTQ) of LLMs offers a promising solution that reduces their memory footprint and decoding latency. In practice, PTQ with uniform quantization representation is favored due to its efficiency and ease of deployment, as uniform quantization is widely supported by mainstream hardware and software libraries. Recent studies on low-bit uniform quantization have led to noticeable improvements in post-quantization model performance; however, they mainly focus on quantization methodologies, while the initialization of quantization parameters remains underexplored and still relies on the conventional Min-Max formula. In this work, we identify the limitations of the Min-Max formula, move beyond its constraints, and propose NeUQI, a method that efficiently determines near-optimal initialization for uniform quantization. Our NeUQI simplifies the joint optimization of the scale and zero-point by deriving the zero-point for a given scale, thereby reducing the problem to a scale-only optimization. Benefiting from the improved quantization parameters, our NeUQI consistently outperforms existing methods in the experiments with the LLaMA and Qwen families on various settings and tasks. Furthermore, when combined with a lightweight distillation strategy, NeUQI even achieves superior performance to PV-tuning, a considerably more resource-intensive method.

MIT Technology Review’s What’s Next series looks across industries, trends, and technologies to give you a first look at the future. You can read the rest of them here. Demand for electric vehicles and the batteries that power them has never been hotter. In 2025, EVs made up over a quarter of new vehicle sales globally, up from less than 5% in 2020. Some regions are seeing even higher uptake: In China, more than 50% of new vehicle sales last year were battery electric or plug-in hybrids. In Europe, more purely electric vehicles hit the roads in December than gas-powered ones. (The US is the notable exception here, dragging down the global average with a small sales decline from 2024.) As EVs become increasingly common on the roads, the battery world is growing too. Looking ahead, we could soon see wider adoption of new chemistries, including some that deliver lower costs or higher performance. Meanwhile, the geopolitics of batteries are shifting, and so is the policy landscape. Here’s what’s coming next for EV batteries in 2026 and beyond. A big opportunity for sodium-ion batteries Lithium-ion batteries are the default chemistry used in EVs, personal devices, and even stationary storage systems on the grid today. But in a tough environment in some markets like the US, there’s a growing interest in cheaper alternatives. Automakers right now largely care just about batteries’ cost, regardless of performance improvements, says Kara Rodby, a technical principal at Volta Energy Technologies, a venture capital firm that focuses on energy storage technology. Sodium-ion cells have long been held up as a potentially less expensive alternative to lithium. The batteries are limited in their energy density, so they deliver a shorter range than lithium-ion. But sodium is also more abundant, so they could be cheaper. Sodium’s growth has been cursed, however, by the very success of lithium-based batteries, says Shirley Meng, a professor of molecular engineering at the University of Chicago. A lithium-ion battery cell cost $568 per kilowatt-hour in 2013, but that cost had fallen to just $74 per kilowatt-hour by 2025—quite the moving target for cheaper alternatives to chase. Sodium-ion batteries currently cost about $59 per kilowatt-hour on average. That’s less expensive than the average lithium-ion battery. But if you consider only lithium iron phosphate (LFP) cells, a lower-end type of lithium-ion battery that averages $52 per kilowatt-hour, sodium is still more expensive today.  We could soon see an opening for sodium-batteries, though. Lithium prices have been ticking up in recent months, a shift that could soon slow or reverse the steady downward march of prices for lithium-based batteries.  Sodium-ion batteries are already being used commercially, largely for stationary storage on the grid. But we’re starting to see sodium-ion cells incorporated into vehicles, too. The Chinese companies Yadea, JMEV, and HiNa Battery have all started producing sodium-ion batteries in limited numbers for EVs, including small, short-range cars and electric scooters that don’t require a battery with high energy density. CATL, a Chinese battery company that’s the world’s largest, says it recently began producing sodium-ion cells. The company plans to launch its first EV using the chemistry by the middle of this year.  Today, both production and demand for sodium-ion batteries are heavily centered in China. That’s likely to continue, especially after a cutback in tax credits and other financial support for the battery and EV industries in the US. One of the biggest sodium-battery companies in the US, Natron, ceased operations last year after running into funding issues. We could also see progress in sodium-ion research: Companies and researchers are developing new materials for components including the electrolyte and electrodes, so the cells could get more comparable to lower-end lithium-ion cells in terms of energy density, Meng says.  Major tests for solid-state batteries As we enter the second half of this decade, many eyes in the battery world are on big promises and claims about solid-state batteries. These batteries could pack more energy into a smaller package by removing the liquid electrolyte, the material that ions move through when a battery is charging and discharging. With a higher energy density, they could unlock longer-range EVs. Companies have been promising solid-state batteries for years. Toyota, for example, once planned to have them in vehicles by 2020. That timeline has been delayed several times, though the company says it’s now on track to launch the new cells in cars in 2027 or 2028. Historically, battery makers have struggled to produce solid-state batteries at the scale needed to deliver a commercially relevant supply for EVs. There’s been progress in manufacturing techniques, though, and companies could soon actually make good on their promises, Meng says.  Factorial Energy, a US-based company making solid-state batteries, provided cells for a Mercedes test vehicle that drove over 745 miles on a single charge in a real-world test in September. The company says it plans to bring its tech to market as soon as 2027. Quantumscape, another major solid-state player in the US, is testing its cells with automotive partners and plans to have its batteries in commercial production later this decade.   Before we see true solid-state batteries, we could see hybrid technologies, often referred to as semi-solid-state batteries. These commonly use materials like gel electrolytes, reducing the liquid inside cells without removing it entirely. Many Chinese companies are looking to build semi-solid-state batteries before transitioning to entirely solid-state ones, says Evelina Stoikou, head of battery technologies and supply chains at BloombergNEF, an energy consultancy. A global patchwork The picture for the near future of the EV industry looks drastically different depending on where you’re standing. Last year, China overtook Japan as the country with the most global auto sales. And more than one in three EVs made in 2025 had a CATL battery in it. Simply put, China is dominating the global battery industry, and that doesn’t seem likely to change anytime soon. China’s influence outside its domestic market is growing especially quickly. CATL is expected to begin production this year at its second

In this tutorial, we walk through an end-to-end, advanced workflow for knowledge graph embeddings using PyKEEN, actively exploring how modern embedding models are trained, evaluated, optimized, and interpreted in practice. We start by understanding the structure of a real knowledge graph dataset, then systematically train and compare multiple embedding models, tune their hyperparameters, and analyze their performance using robust ranking metrics. Also, we focus not just on running pipelines but on building intuition for link prediction, negative sampling, and embedding geometry, ensuring we understand why each step matters and how it affects downstream reasoning over graphs. Check out the FULL CODES here. Copy CodeCopiedUse a different Browser !pip install -q pykeen torch torchvision import warnings warnings.filterwarnings(‘ignore’) import torch import numpy as np import pandas as pd import matplotlib.pyplot as plt import seaborn as sns from typing import Dict, List, Tuple from pykeen.pipeline import pipeline from pykeen.datasets import Nations, FB15k237, get_dataset from pykeen.models import TransE, ComplEx, RotatE, DistMult from pykeen.training import SLCWATrainingLoop, LCWATrainingLoop from pykeen.evaluation import RankBasedEvaluator from pykeen.triples import TriplesFactory from pykeen.hpo import hpo_pipeline from pykeen.sampling import BasicNegativeSampler from pykeen.losses import MarginRankingLoss, BCEWithLogitsLoss from pykeen.trackers import ConsoleResultTracker print(“PyKEEN setup complete!”) print(f”PyTorch version: {torch.__version__}”) print(f”CUDA available: {torch.cuda.is_available()}”) We set up the complete experimental environment by installing PyKEEN and its deep learning dependencies, and by importing all required libraries for modeling, evaluation, visualization, and optimization. We ensure a clean, reproducible workflow by suppressing warnings and verifying the PyTorch and CUDA configurations for efficient computation. Check out the FULL CODES here. Copy CodeCopiedUse a different Browser print(“n” + “=”*80) print(“SECTION 2: Dataset Exploration”) print(“=”*80 + “n”) dataset = Nations() print(f”Dataset: {dataset}”) print(f”Number of entities: {dataset.num_entities}”) print(f”Number of relations: {dataset.num_relations}”) print(f”Training triples: {dataset.training.num_triples}”) print(f”Testing triples: {dataset.testing.num_triples}”) print(f”Validation triples: {dataset.validation.num_triples}”) print(“nSample triples (head, relation, tail):”) for i in range(5): h, r, t = dataset.training.mapped_triples[i] head = dataset.training.entity_id_to_label[h.item()] rel = dataset.training.relation_id_to_label[r.item()] tail = dataset.training.entity_id_to_label[t.item()] print(f” {head} –[{rel}]–> {tail}”) def analyze_dataset(triples_factory: TriplesFactory) -> pd.DataFrame: “””Compute basic statistics about the knowledge graph.””” stats = { ‘Metric’: [], ‘Value’: [] } stats[‘Metric’].extend([‘Entities’, ‘Relations’, ‘Triples’]) stats[‘Value’].extend([ triples_factory.num_entities, triples_factory.num_relations, triples_factory.num_triples ]) unique, counts = torch.unique(triples_factory.mapped_triples[:, 1], return_counts=True) stats[‘Metric’].extend([‘Avg triples per relation’, ‘Max triples for a relation’]) stats[‘Value’].extend([counts.float().mean().item(), counts.max().item()]) return pd.DataFrame(stats) stats_df = analyze_dataset(dataset.training) print(“nDataset Statistics:”) print(stats_df.to_string(index=False)) We load and explore the Nation’s knowledge graph to understand its scale, structure, and relational complexity before training any models. We inspect sample triples to build intuition about how entities and relations are represented internally using indexed mappings. We then compute core statistics such as relation frequency and triple distribution, allowing us to reason about graph sparsity and modeling difficulty upfront. Check out the FULL CODES here. Copy CodeCopiedUse a different Browser print(“n” + “=”*80) print(“SECTION 3: Training Multiple Models”) print(“=”*80 + “n”) models_config = { ‘TransE’: { ‘model’: ‘TransE’, ‘model_kwargs’: {’embedding_dim’: 50}, ‘loss’: ‘MarginRankingLoss’, ‘loss_kwargs’: {‘margin’: 1.0} }, ‘ComplEx’: { ‘model’: ‘ComplEx’, ‘model_kwargs’: {’embedding_dim’: 50}, ‘loss’: ‘BCEWithLogitsLoss’, }, ‘RotatE’: { ‘model’: ‘RotatE’, ‘model_kwargs’: {’embedding_dim’: 50}, ‘loss’: ‘MarginRankingLoss’, ‘loss_kwargs’: {‘margin’: 3.0} } } training_config = { ‘training_loop’: ‘sLCWA’, ‘negative_sampler’: ‘basic’, ‘negative_sampler_kwargs’: {‘num_negs_per_pos’: 5}, ‘training_kwargs’: { ‘num_epochs’: 100, ‘batch_size’: 128, }, ‘optimizer’: ‘Adam’, ‘optimizer_kwargs’: {‘lr’: 0.001} } results = {} for model_name, config in models_config.items(): print(f”nTraining {model_name}…”) result = pipeline( dataset=dataset, model=config[‘model’], model_kwargs=config.get(‘model_kwargs’, {}), loss=config.get(‘loss’), loss_kwargs=config.get(‘loss_kwargs’, {}), **training_config, random_seed=42, device=’cuda’ if torch.cuda.is_available() else ‘cpu’ ) results[model_name] = result print(f”n{model_name} Results:”) print(f” MRR: {result.metric_results.get_metric(‘mean_reciprocal_rank’):.4f}”) print(f” Hits@1: {result.metric_results.get_metric(‘hits_at_1’):.4f}”) print(f” Hits@3: {result.metric_results.get_metric(‘hits_at_3’):.4f}”) print(f” Hits@10: {result.metric_results.get_metric(‘hits_at_10’):.4f}”) We define a consistent training configuration and systematically train multiple knowledge graph embedding models to enable fair comparison. We use the same dataset, negative sampling strategy, optimizer, and training loop while allowing each model to leverage its own inductive bias and loss formulation. We then evaluate and record standard ranking metrics, such as MRR and Hits@K, to quantitatively assess each embedding approach’s performance on link prediction. Check out the FULL CODES here. Copy CodeCopiedUse a different Browser print(“n” + “=”*80) print(“SECTION 4: Model Comparison”) print(“=”*80 + “n”) metrics_to_compare = [‘mean_reciprocal_rank’, ‘hits_at_1’, ‘hits_at_3’, ‘hits_at_10’] comparison_data = {metric: [] for metric in metrics_to_compare} model_names = [] for model_name, result in results.items(): model_names.append(model_name) for metric in metrics_to_compare: comparison_data[metric].append( result.metric_results.get_metric(metric) ) comparison_df = pd.DataFrame(comparison_data, index=model_names) print(“Model Comparison:”) print(comparison_df.to_string()) fig, axes = plt.subplots(2, 2, figsize=(15, 10)) fig.suptitle(‘Model Performance Comparison’, fontsize=16) for idx, metric in enumerate(metrics_to_compare): ax = axes[idx // 2, idx % 2] comparison_df[metric].plot(kind=’bar’, ax=ax, color=’steelblue’) ax.set_title(metric.replace(‘_’, ‘ ‘).title()) ax.set_ylabel(‘Score’) ax.set_xlabel(‘Model’) ax.grid(axis=’y’, alpha=0.3) ax.set_xticklabels(ax.get_xticklabels(), rotation=45) plt.tight_layout() plt.show() We aggregate evaluation metrics from all trained models into a unified comparison table for direct performance analysis. We visualize key ranking metrics using bar charts, allowing us to quickly identify strengths and weaknesses across different embedding approaches. Check out the FULL CODES here. Copy CodeCopiedUse a different Browser print(“n” + “=”*80) print(“SECTION 5: Hyperparameter Optimization”) print(“=”*80 + “n”) hpo_result = hpo_pipeline( dataset=dataset, model=’TransE’, n_trials=10, training_loop=’sLCWA’, training_kwargs={‘num_epochs’: 50}, device=’cuda’ if torch.cuda.is_available() else ‘cpu’, ) print(“nBest Configuration Found:”) print(f” Embedding Dim: {hpo_result.study.best_params.get(‘model.embedding_dim’, ‘N/A’)}”) print(f” Learning Rate: {hpo_result.study.best_params.get(‘optimizer.lr’, ‘N/A’)}”) print(f” Best MRR: {hpo_result.study.best_value:.4f}”) print(“n” + “=”*80) print(“SECTION 6: Link Prediction”) print(“=”*80 + “n”) best_model_name = comparison_df[‘mean_reciprocal_rank’].idxmax() best_result = results[best_model_name] model = best_result.model print(f”Using {best_model_name} for predictions”) def predict_tails(model, dataset, head_label: str, relation_label: str, top_k: int = 5): “””Predict most likely tail entities for a given head and relation.””” head_id = dataset.entity_to_id[head_label] relation_id = dataset.relation_to_id[relation_label] num_entities = dataset.num_entities heads = torch.tensor([head_id] * num_entities).unsqueeze(1) relations = torch.tensor([relation_id] * num_entities).unsqueeze(1) tails = torch.arange(num_entities).unsqueeze(1) batch = torch.cat([heads, relations, tails], dim=1) with torch.no_grad(): scores = model.predict_hrt(batch) top_scores, top_indices = torch.topk(scores.squeeze(), k=top_k) predictions = [] for score, idx in zip(top_scores, top_indices): tail_label = dataset.entity_id_to_label[idx.item()] predictions.append((tail_label, score.item())) return predictions if dataset.training.num_entities > 10: sample_head = list(dataset.entity_to_id.keys())[0] sample_relation = list(dataset.relation_to_id.keys())[0] print(f”nTop predictions for: {sample_head} –[{sample_relation}]–> ?”) predictions = predict_tails( best_result.model, dataset.training, sample_head, sample_relation, top_k=5 ) for rank, (entity, score) in enumerate(predictions, 1): print(f” {rank}. {entity} (score: {score:.4f})”) We apply automated hyperparameter optimization to systematically search for a stronger TransE configuration that improves ranking performance without manual tuning. We then select the best-performing model based on MRR and use it to perform practical link prediction by scoring all possible tail entities for a given head–relation pair. Check out the FULL CODES here. Copy CodeCopiedUse a different Browser print(“n” +

Civitai—an online marketplace for buying and selling AI-generated content, backed by the venture capital firm Andreessen Horowitz—is letting users buy custom instruction files for generating celebrity deepfakes. Some of these files were specifically designed to make pornographic images banned by the site, a new analysis has found. The study, from researchers at Stanford and Indiana University, looked at people’s requests for content on the site, called “bounties.” The researchers found that between mid-2023 and the end of 2024, most bounties asked for animated content—but a significant portion were for deepfakes of real people, and 90% of these deepfake requests targeted women. (Their findings have not yet been peer reviewed.) The debate around deepfakes, as illustrated by the recent backlash to explicit images on the X-owned chatbot Grok, has revolved around what platforms should do to block such content. Civitai’s situation is a little more complicated. Its marketplace includes actual images, videos, and models, but it also lets individuals buy and sell instruction files called LoRAs that can coach mainstream AI models like Stable Diffusion into generating content they were not trained to produce. Users can then combine these files with other tools to make deepfakes that are graphic or sexual. The researchers found that 86% of deepfake requests on Civitai were for LoRAs. In these bounties, users requested “high quality” models to generate images of public figures like the influencer Charli D’Amelio or the singer Gracie Abrams, often linking to their social media profiles so their images could be grabbed from the web. Some requests specified a desire for models that generated the individual’s entire body, accurately captured their tattoos, or allowed hair color to be changed. Some requests targeted several women in specific niches, like artists who record ASMR videos. One request was for a deepfake of a woman said to be the user’s wife. Anyone on the site could offer up AI models they worked on for the task, and the best submissions received payment—anywhere from $0.50 to $5. And nearly 92% of the deepfake bounties were awarded. Neither Civitai nor Andreessen Horowitz responded to requests for comment. It’s possible that people buy these LoRAs to make deepfakes that aren’t sexually explicit (though they’d still violate Civitai’s terms of use, and they’d still be ethically fraught). But Civitai also offers educational resources on how to use external tools to further customize the outputs of image generators—for example, by changing someone’s pose. The site also hosts user-written articles with details on how to instruct models to generate pornography. The researchers found that the amount of porn on the platform has gone up, and that the majority of requests each week are now for NSFW content. “Not only does Civitai provide the infrastructure that facilitates these issues; they also explicitly teach their users how to utilize them,” says Matthew DeVerna, a postdoctoral researcher at Stanford’s Cyber Policy Center and one of the study’s leaders.  The company used to ban only sexually explicit deepfakes of real people, but in May 2025 it announced it would ban all deepfake content. Nonetheless, countless requests for deepfakes submitted before this ban now remain live on the site, and many of the winning submissions fulfilling those requests remain available for purchase, MIT Technology Review confirmed. “I believe the approach that they’re trying to take is to sort of do as little as possible, such that they can foster as much—I guess they would call it—creativity on the platform,” DeVerna says. Users buy LoRAs with the site’s online currency, called Buzz, which is purchased with real money. In May 2025, Civita’s credit card processor cut off the company because of its ongoing problem with nonconsensual content. To pay for explicit content, users must now use gift cards or cryptocurrency to buy Buzz; the company offers a different scrip for non-explicit content.  Civitai automatically tags bounties requesting deepfakes and lists a way for the person featured in the content to manually request its takedown. This system means that Civitai has a reasonably successful way of knowing which bounties are for deepfakes, but it’s still leaving moderation to the general public rather than carrying it out proactively.  A company’s legal liability for what its users do isn’t totally clear. Generally, tech companies have broad legal protections against such liability for their content under Section 230 of the Communications Decency Act, but those protections aren’t limitless. For example, “you cannot knowingly facilitate illegal transactions on your website,” says Ryan Calo, a professor specializing in technology and AI at the University of Washington’s law school. (Calo wasn’t involved in this new study.) Civitai joined OpenAI, Anthropic, and other AI companies in 2024 in adopting design principles to guard against the creation and spread of AI-generated child sexual abuse material . This move followed a 2023 report from the Stanford Internet Observatory, which found that the vast majority of AI models named in child sexual abuse communities were Stable Diffusion–based models “predominantly obtained via Civitai.” But adult deepfakes have not gotten the same level of attention from content platforms or the venture capital firms that fund them. “They are not afraid enough of it. They are overly tolerant of it,” Calo says. “Neither law enforcement nor civil courts adequately protect against it. It is night and day.” Civitai received a $5 million investment from Andreessen Horowitz (a16z) in November 2023. In a video shared by a16z, Civitai cofounder and CEO Justin Maier described his goal of building the main place where people find and share AI models for their own individual purposes. “We’ve aimed to make this space that’s been very, I guess, niche and engineering-heavy more and more approachable to more and more people,” he said.  Civitai is not the only company with a deepfake problem in a16z’s investment portfolio; in February, MIT Technology Review first reported that another company, Botify AI, was hosting AI companions resembling real actors that stated their age as under 18, engaged in sexually charged conversations, offered “hot photos,” and in some instances described