AI

The growing adoption of open-source large language models such as Llama has introduced new integration challenges for teams previously relying on proprietary systems like OpenAI’s GPT or Anthropic’s Claude. While performance benchmarks for Llama are increasingly competitive, discrepancies in prompt formatting and system message handling often result in degraded output quality when existing prompts are reused without modification. To address this issue, Meta has introduced Llama Prompt Ops, a Python-based toolkit designed to streamline the migration and adaptation of prompts originally constructed for closed models. Now available on GitHub, the toolkit programmatically adjusts and evaluates prompts to align with Llama’s architecture and conversational behavior, minimizing the need for manual experimentation. Prompt engineering remains a central bottleneck in deploying LLMs effectively. Prompts tailored to the internal mechanics of GPT or Claude frequently do not transfer well to Llama, due to differences in how these models interpret system messages, handle user roles, and process context tokens. The result is often unpredictable degradation in task performance. Llama Prompt Ops addresses this mismatch with a utility that automates the transformation process. It operates on the assumption that prompt format and structure can be systematically restructured to match the operational semantics of Llama models, enabling more consistent behavior without retraining or extensive manual tuning. Core Capabilities The toolkit introduces a structured pipeline for prompt adaptation and evaluation, comprising the following components: Automated Prompt Conversion:Llama Prompt Ops parses prompts designed for GPT, Claude, and Gemini, and reconstructs them using model-aware heuristics to better suit Llama’s conversational format. This includes reformatting system instructions, token prefixes, and message roles. Template-Based Fine-Tuning:By providing a small set of labeled query-response pairs (minimum ~50 examples), users can generate task-specific prompt templates. These are optimized through lightweight heuristics and alignment strategies to preserve intent and maximize compatibility with Llama. Quantitative Evaluation Framework:The tool generates side-by-side comparisons of original and optimized prompts, using task-level metrics to assess performance differences. This empirical approach replaces trial-and-error methods with measurable feedback. Together, these functions reduce the cost of prompt migration and provide a consistent methodology for evaluating prompt quality across LLM platforms. Workflow and Implementation Llama Prompt Ops is structured for ease of use with minimal dependencies. The optimization workflow is initiated using three inputs: A YAML configuration file specifying the model and evaluation parameters A JSON file containing prompt examples and expected completions A system prompt, typically designed for a closed model The system applies transformation rules and evaluates outcomes using a defined metric suite. The entire optimization cycle can be completed within approximately five minutes, enabling iterative refinement without the overhead of external APIs or model retraining. Importantly, the toolkit supports reproducibility and customization, allowing users to inspect, modify, or extend transformation templates to fit specific application domains or compliance constraints. Implications and Applications For organizations transitioning from proprietary to open models, Llama Prompt Ops offers a practical mechanism to maintain application behavior consistency without reengineering prompts from scratch. It also supports development of cross-model prompting frameworks by standardizing prompt behavior across different architectures. By automating a previously manual process and providing empirical feedback on prompt revisions, the toolkit contributes to a more structured approach to prompt engineering—a domain that remains under-explored relative to model training and fine-tuning. Conclusion Llama Prompt Ops represents a targeted effort by Meta to reduce friction in the prompt migration process and improve alignment between prompt formats and Llama’s operational semantics. Its utility lies in its simplicity, reproducibility, and focus on measurable outcomes, making it a relevant addition for teams deploying or evaluating Llama in real-world settings. Check out the GitHub Page. All credit for this research goes to the researchers of this project. Also, feel free to follow us on Twitter and don’t forget to join our 95k+ ML SubReddit and Subscribe to our Newsletter. The post Meta Releases Llama Prompt Ops: A Python Package that Automatically Optimizes Prompts for Llama Models appeared first on MarkTechPost.

arXiv:2502.14830v2 Announce Type: replace Abstract: While large language models demonstrate remarkable capabilities at task-specific applications through fine-tuning, extending these benefits across diverse languages is essential for broad accessibility. However, effective cross-lingual transfer is hindered by LLM performance gaps across languages and the scarcity of fine-tuning data in many languages. Through analysis of LLM internal representations from over 1,000+ language pairs, we discover that middle layers exhibit the strongest potential for cross-lingual alignment. Building on this finding, we propose a middle-layer alignment objective integrated into task-specific training. Our experiments on slot filling, machine translation, and structured text generation show consistent improvements in cross-lingual transfer, especially to lower-resource languages. The method is robust to the choice of alignment languages and generalizes to languages unseen during alignment. Furthermore, we show that separately trained alignment modules can be merged with existing task-specific modules, improving cross-lingual capabilities without full re-training. Our code is publicly available (https://github.com/dannigt/mid-align).

arXiv:2505.24229v1 Announce Type: new Abstract: Inverse Text Normalization (ITN) is crucial for converting spoken Automatic Speech Recognition (ASR) outputs into well-formatted written text, enhancing both readability and usability. Despite its importance, the integration of streaming ITN within streaming ASR remains largely unexplored due to challenges in accuracy, efficiency, and adaptability, particularly in low-resource and limited-context scenarios. In this paper, we introduce a streaming pretrained language model for ITN, leveraging pretrained linguistic representations for improved robustness. To address streaming constraints, we propose Dynamic Context-Aware during training and inference, enabling adaptive chunk size adjustments and the integration of right-context information. Experimental results demonstrate that our method achieves accuracy comparable to non-streaming ITN and surpasses existing streaming ITN models on a Vietnamese dataset, all while maintaining low latency, ensuring seamless integration into ASR systems.

arXiv:2502.01349v2 Announce Type: replace Abstract: The advent of Large Language Models (LLMs) has revolutionized product recommenders, yet their susceptibility to adversarial manipulation poses critical challenges, particularly in real-world commercial applications. Our approach is the first one to tap into human psychological principles, seamlessly modifying product descriptions, making such manipulations hard to detect. In this work, we investigate cognitive biases as black-box adversarial strategies, drawing parallels between their effects on LLMs and human purchasing behavior. Through extensive evaluation across models of varying scale, we find that certain biases, such as social proof, consistently boost product recommendation rate and ranking, while others, like scarcity and exclusivity, surprisingly reduce visibility. Our results demonstrate that cognitive biases are deeply embedded in state-of-the-art LLMs, leading to highly unpredictable behavior in product recommendations and posing significant challenges for effective mitigation.

arXiv:2505.24826v1 Announce Type: new Abstract: As large language models (LLMs) are increasingly used in legal applications, current evaluation benchmarks tend to focus mainly on factual accuracy while largely neglecting important linguistic quality aspects such as clarity, coherence, and terminology. To address this gap, we propose three steps: First, we develop a regression model to evaluate the quality of legal texts based on clarity, coherence, and terminology. Second, we create a specialized set of legal questions. Third, we analyze 49 LLMs using this evaluation framework. Our analysis identifies three key findings: First, model quality levels off at 14 billion parameters, with only a marginal improvement of $2.7%$ noted at 72 billion parameters. Second, engineering choices such as quantization and context length have a negligible impact, as indicated by statistical significance thresholds above 0.016. Third, reasoning models consistently outperform base architectures. A significant outcome of our research is the release of a ranking list and Pareto analysis, which highlight the Qwen3 series as the optimal choice for cost-performance tradeoffs. This work not only establishes standardized evaluation protocols for legal LLMs but also uncovers fundamental limitations in current training data refinement approaches. Code and models are available at: https://github.com/lyxx3rd/LegalEval-Q.

arXiv:2505.24040v1 Announce Type: new Abstract: Large Language Models (LLMs) have demonstrated remarkable performance on various medical question-answering (QA) benchmarks, including standardized medical exams. However, correct answers alone do not ensure correct logic, and models may reach accurate conclusions through flawed processes. In this study, we introduce the MedPAIR (Medical Dataset Comparing Physicians and AI Relevance Estimation and Question Answering) dataset to evaluate how physician trainees and LLMs prioritize relevant information when answering QA questions. We obtain annotations on 1,300 QA pairs from 36 physician trainees, labeling each sentence within the question components for relevance. We compare these relevance estimates to those for LLMs, and further evaluate the impact of these “relevant” subsets on downstream task performance for both physician trainees and LLMs. We find that LLMs are frequently not aligned with the content relevance estimates of physician trainees. After filtering out physician trainee-labeled irrelevant sentences, accuracy improves for both the trainees and the LLMs. All LLM and physician trainee-labeled data are available at: http://medpair.csail.mit.edu/.