YouZum

Nachrichten

AI, Committee, Nachrichten, Uncategorized

Agriculture is ready for AI, but its data isn’t

Artificial intelligence is transforming what is possible in agriculture, but industry leaders should be wary of investing in AI without first laying the groundwork.  The use cases are promising, especially for an industry navigating volatile fertilizer costs, unpredictable weather, and margins that leave little room for error. Research shows AI-enabled predictive models can improve crop yield by 26%, reduce water use by 41%, and cut chemical usage by 33%.  However, what AI vendors usually won’t tell you is that these solutions are only effective if you have a clean, solid data foundation. However, at Reltio, we have experience in this area, including leading technology strategy at a major agricultural distributor and building a data platform used by enterprises worldwide–we’ve seen it first hand. What AI vendors won’t tell you  Vendor conversations in agriculture tend to follow a familiar pattern. The pitch leads with grand promises around using AI to monitor crop health in real time, optimize irrigation, and squeeze more yield from every acre.  The promise is compelling, but what rarely comes up is the question of whether the data foundation underneath those promises is accurate and complete. If not, there is a real and significant risk that AI will generate misleading outputs that seem authoritative but inspire action that is, at best, counterproductive.  For instance, a yield prediction model fed inconsistent historical data will generate imprecise forecasts. Similarly, a precision irrigation system drawing on fragmented sensor data will make watering decisions that waste resources instead of saving them.  In each case, the AI is failing because the data it was trained on was not sufficient to produce trustworthy outputs. In agriculture, every AI hallucination is a liability, and the likelihood of error is high. Why agriculture is a uniquely challenging test case The data landscape across a modern agricultural operation or a large distributor serving thousands of growers is extraordinarily complex. Modern farming environments make extensive use of IoT devices and machinery. Irrigation systems are automated, tractors navigate fields autonomously, and drones capture field imagery at scale.  However, machine data is disparate by nature. Add in external sources, including weather feeds, U.S. Department of Agriculture data, and third-party market information, and the question of how you bring all of it together into something coherent becomes a significant undertaking.  Agricultural AI also needs to understand more than just customer attributes; it needs to understand the land: GPS coordinates, farm boundaries, field blocks, and soil variation across a single property. Where do you apply fertilizer, and at what rate, and in which specific area of the farm? Not all parts of a field are the same, and an AI system that treats them as if they are will produce recommendations that are at best imprecise and at worst damaging. There is also a compliance dimension due to the chemicals and the responsibility involved. Operational AI in agriculture needs significantly more checks and governance than it might in a lower-stakes environment. When a flawed recommendation gets acted upon in the field, the consequences can be severe.  What data readiness means in practice  Data readiness is the difference between AI delivering on its promise vs. a “garbage in, garbage out” scenario. Fundamentally, being ready for AI means having a data model that accurately reflects how the business operates.  For a company like Wilbur-Ellis, a 104-year-old, family-owned agricultural distributor, that means understanding who your customers are, which fields they farm, which inputs they need, which suppliers those inputs come from, what they paid last season, and how all of that connects to margin. That information needs to be current, consistent, and accessible across the organization, rather than locked in separate systems that were never designed to talk to each other. Similarly, for farming operations themselves, data readiness means having a reliable, connected picture of what is happening across every field: soil health records, input application histories, yield data from previous seasons, equipment performance, and real-time sensor readings from irrigation systems. Governance matters just as much as structure. Prices change, relationships evolve, and suppliers come and go. An AI system drawing on data that was accurate six months ago but has not been maintained will make recommendations based on a version of the business that no longer exists.  Building the foundation that makes AI trustworthy The good news is that the path to data readiness is feasible. It starts with a strong data model: a single, governed source of truth that connects customers, suppliers, products, pricing, orders, and margins in a way that reflects how the organization operates.  From there, it requires data pipelines fast enough to deliver insights when decisions need to be made, governance frameworks that keep that data trustworthy over time, and security controls that ensure sensitive commercial information is accessible to the right people under the right conditions. This is precisely the challenge that Reltio, an SAP company, was built to solve. Reltio enables companies to unify their fragmented data so AI agents and systems can operate from a complete picture of the business. Reltio builds a trusted system of context, known as the context intelligence layer, that brings all entities, relationships, rules together under one roof and makes business data easy to access and interpret. For Wilbur-Ellis, building that trustworthy data foundation has meant being able to ask more complex questions and trust the answers, which is the precondition for any AI system to be genuinely useful. How agriculture can drive real value from AI The question worth asking before the next AI conversation is not whether the use case is promising. It almost certainly is. The question is whether the underlying data foundation is strong enough to make the output trustworthy.  Agriculture has always required its leaders to make high-stakes decisions under uncertainty, and AI offers the genuine prospect of making those decisions faster and better informed. That prospect is only achievable for organizations that have done the foundational work first, and the businesses that will get the most from AI are the ones investing in that foundation

Agriculture is ready for AI, but its data isn’t Beitrag lesen »

AI, Committee, Nachrichten, Uncategorized

The Download: AI “coworkers” and stratospheric internet

This is today’s edition of The Download, our weekday newsletter that provides a daily dose of what’s going on in the world of technology. AI agents are not your “coworkers” Imagine coming in to work to learn that a new underling will report to you. The worker is not a person but an AI tool—one that your company nonetheless calls Alex, an “employee” with a title and defined responsibilities. How well do you think you would work with Alex? If you’re anything like the managers studied by Boston University professor Emma Wiles, treating that AI as a “coworker” would lead you to do a worse job. They caught 18% fewer errors when the work was attributed to an agentic “AI employee” rather than a chatbot. This is an alarming glimpse of the future Silicon Valley is hurling us toward. Microsoft, OpenAI, Anthropic, and Google have all released tools for managing teams of AI agents, many of which are advertised as digital colleagues. Find out why that’s a losing proposition for workers. —James O’Donnell This story is from The Algorithm, our weekly AI newsletter. Sign up to receive it in your inbox every Monday. This flying solar-powered platform could deliver better internet from the air As soon as August, a giant silver bullet will cut its way through the dry air of the southwestern US and cross the Pacific to reach the coast of Japan. Once there, the roughly 200-foot-long craft, built by the New Mexico–based company Sceye, will park some 18 kilometers above the ocean’s surface in the stratosphere, then use a custom-built antenna to supplement a 5G network, in a test that includes beaming data straight to devices. Sceye (pronounced “sky”) is one of several firms building these high-altitude platform stations, or HAPS. Find out why they plan to connect us from the stratosphere. —Rachel Courtland This story is from the latest edition of our magazine, which is all about engineering. Subscribe now to get a copy, plus all our other issues and a range of subscriber-only content. Longevity’s next frontier: “reprogramming” your body Billions of dollars are flooding into efforts to reverse aging as scientists explore ways to return cells to a younger state. But how far off are these experimental treatments? Will they really work? At a virtual Roundtables event today, MIT Technology Review will examine the science behind the hype. Science editor Mary Beth Griggs and senior biotechnology reporter Jessica Hamzelou will explore longevity’s latest frontier in a subscriber-only discussion. Register here to join the session at 11:30 AM ET / 8:30 AM PT / 16:30 GMT. The must-reads I’ve combed the internet to find you today’s most fun/important/scary/fascinating stories about technology. 1 The US House has passed new youth online safety legislation+ It would set baseline federal standards for kids’ online safety. (Politico $)+ States would be allowed to adopt more aggressive protections. (Reuters $)+ But critics say it lets tech companies avoid accountability. (Axios)+ And tech groups warn it threatens privacy and free expression. (NBC)+ The Senate is expected to push for tougher rules. (The Hill) 2 Ford is rehiring human engineers after AI failed to match quality checksIt said the AI lacked the training and expertise of technicians. (Bloomberg $) + The new hires will train younger staff and reprogram AI tools. (BBC)+ Many firms that replaced workers with AI are now rehiring humans. (Forbes)+ The AI jobs hysteria needs a reality check. (MIT Technology Review) 3 Senator Mark Warren is set to introduce a bill to regulate AI agentsIt would set rules for agent permissions and verification. (The Information $)+ Voters of both parties want tighter AI regulation. (NBC News)+ But politicians are bitterly divided on the rules. (MIT Technology Review) 4 Rocket Lab is buying Iridium for $8 billion to take on SpaceXIt wants to integrate the satellite network with its launch services. (The Verge)+ Which could create a fleet that can compete with SpaceX. (WSJ $) 5 Hackers have exposed secrets about Apple’s upcoming iPhone 18The data was stolen from Tata Electronics, Apple’s Indian supplier. (Reuters $)+ The breach also exposed Tesla secrets. (TechCrunch) 6 Chatbots are replacing therapists despite lacking scientific evidenceExperts question their safety and therapeutic quality. (WSJ $)+ Chatbots may make us lose control of our brains. (MIT Technology Review) 7 Newborn DNA sequencing is edging closer to routine healthcareTrials are expanding despite privacy and ethical concerns. (Economist $)+ The push for perfect babies is an ethical mess. (MIT Technology Review) 8 Astronomers are using AI to find new galaxiesNew tools are reviving decades of space telescope data. (FT $) 9 Remote-controlled cockroach swarms can now breathe underwaterThe cyborg insects could one day explore Mars. (New Scientist $) 10 Drone shows are creating new forms of worship Churches are depicting biblical stories with thousands of UAVs. (Wired $)) Quote of the day “This is taking us back to the 1950s, and that is not progress.”  —Edwin Lyman, director of nuclear power safety at the Union of Concerned Scientists, tells NPR that slashing regulations undoes decades of safety lessons from the industry. One More Thing Design thinking was supposed to fix the world. Where did it go wrong? When Kyle Cornforth walked into IDEO’s San Francisco offices for a meeting about reimagining school lunches, she was impressed. “It was Post-its everywhere, prototypes everywhere,” she recalls. “What I really liked was that they offered a framework for collaboration and creation.” Cornforth was new to IDEO’s way of working: a six-step methodology for innovation called design thinking. But when she looked at the ideas themselves, she had questions: “I was like, ‘You didn’t talk to anyone who works in a school, did you?’ They were not contextualized in the problem at all.” Design thinking broadened the idea of “design,” elevating designers to take on big, knotty problems through a structured process. But critics argue it has produced unrealistic ideas and, by centering designers, reinforced existing inequities. Read the full story on the rise and fall of design thinking. —Rebecca Ackermann We can still have nice things

The Download: AI “coworkers” and stratospheric internet Beitrag lesen »

AI, Committee, Nachrichten, Uncategorized

Roundtables: Longevity’s Next Frontier: “Reprogramming” Your Body

Listen to the session or watch below Billions of dollars are flooding into efforts to reverse aging as scientists explore ways to return cells to a younger state. But how far off are these experimental treatments? Will they really work? Watch a conversation exploring longevity’s new focus. Speakers: Mary Beth Griggs, science editor and Jessica Hamzelou, senior biotechnology reporter Recorded on June 30, 2026 Related Stories: Why “reprogramming” is the buzziest approach to reversing aging right now How scientists want to make you young again Sam Altman invested $180 million into a company trying to delay death

Roundtables: Longevity’s Next Frontier: “Reprogramming” Your Body Beitrag lesen »

AI, Committee, Nachrichten, Uncategorized

DG^VoiC: Speaker Clustering for Fraud Investigation under Real Call-Centre Conditions

arXiv:2606.28048v1 Announce Type: cross Abstract: Insurance fraud remains costly and operationally difficult, particularly in call-centre workflows where many customer interactions begin at FNOL. While recent fraud detection methods mainly rely on structured data, text, or images, repeated speaker identity across calls remains underused as an investigative signal. This paper presents DG^VoiC, a voice clustering framework for customer verification and cross-profile speaker linking on anonymised real call-centre audio. The approach combines sensitive information-aligned anonymisation, speech-focused preprocessing, sliding-window speaker embedding extraction, and cosine similarity based clustering to identify repeated speakers under real telephony conditions. The method was evaluated on 121 recordings, with a curated reference subset of 56 samples in 22 human-agreed speaker clusters. used for validation. The best configuration achieved 96% AMI, 95% ARI, 98% completeness, 100% homogeneity, and 99% V-measure. These results show that speaker clustering can provide a strong additional signal for fraud investigation by helping analysts verify speaker consistency and surface repeated voices across customers.

DG^VoiC: Speaker Clustering for Fraud Investigation under Real Call-Centre Conditions Beitrag lesen »

AI, Committee, Nachrichten, Uncategorized

The Download: metric weaknesses and AI elephant warnings

This is today’s edition of The Download, our weekday newsletter that provides a daily dose of what’s going on in the world of technology. The inevitable weakness of metrics There are plenty of useful things a metric can reveal. There are even more that it can obscure or corrupt. Like a lot of people bitten by the self-quantifying bug, I started gathering personal data to pursue a nebulous collection of goals and desires. I wanted to feel better physically and emotionally, get outside more, and bring order to the messiness and uncertainty of my daily existence. But external metrics and data can never capture what’s truly important. Worse, they inevitably redefine your core sense of what’s important, whether you’re aware of the trap or not. Dive into the dangers of quantifying our lives with metrics. —Bryan Gardiner This story is from the next edition of our magazine, which is all about engineering. Subscribe now to get a copy when it lands! Elephant alert! AI warning systems aim to avoid deadly clashes India is home to about 60% of the world’s wild Asian elephants, and around 80% of their habitat lies outside protected areas. That brings them into close contact with people, and clashes can turn lethal: there have been some 3,000 human casualties in the last five years and over 1,000 elephant deaths since 2014. In response, state forest departments, NGOs, and locals are designing, testing, and deploying a range of AI systems that cut response and warning times to minutes—or even seconds. They range from wildlife eyes in Maharashtra to infrared drones in Chhattisgarh. Find out how they work in our interactive map. —Kanika Gupta The must-reads I’ve combed the internet to find you today’s most fun/important/scary/fascinating stories about technology. 1 The US has allowed Anthropic to release Mythos 5 to “trusted” orgsAbout 100 US companies and federal agencies now have access. (Semafor)+ The White House said appropriate safeguards were now in place. (WSJ $)+ The US had restricted both models over national security concerns. (BBC)+ Which raised new questions about AI safety. (MIT Technology Review)  2 A Chinese AI model has matched Mythos in finding security bugsSecurity researchers say Zhipu AI is poised to reset the AI race. (WSJ $)+ It’s sparked alarm that US restrictions are boosting China’s progress. (NYT $)+ Although it still can’t match Anthropic or OpenAI on general tasks. (Verge)+ In the AI race, China is eyeing a come-from-behind victory. (WP $) 3 Apple is seeking approval to buy chips from a blacklisted Chinese firmIt’s lobbying the White House for clearance to buy from ChangXin. (FT $)+ ChangXin is on a Pentagon list of firms with Chinese military ties. (WP $)+ Chipmakers are profiting off AI at the expense of everyone else. (WSJ $)+ The US is banning imports of more Chinese technology. (Reuters $)+ But Chinese tech companies feel optimistic. (MIT Technology Review) 4. South Korea plans to train its entire military as “drone warriors”It wants to train all 500,000 personnel. (Reuters $)+ And produce 110,000 drones by 2029. (Ars Technica) 5 Google has limited Meta’s use of its Gemini AI modelsMeta wanted more compute than Google could provide. (FT $)+ The cap has disrupted and delayed some Meta AI projects. (Bloomberg $) 6 Zuckerberg wants Meta to work with Polymarket and KalshiMeta wants its own prediction market, but without real-money bets. (NYT $)+ The partnerships could hedge risks and accelerate development. (Reuters $) 7 Extreme heat is putting already hot data centers under pressureSevere weather is now the leading cause of loss for data centers. (CNBC)+ Heat waves also mess with your brain. (MIT Technology Review) 8 Android phones alerted millions moments before Venezuela’s earthquakesThey gave users between seconds and up to two minutes’ notice. (NYT $) 9 Scientists think Uranus and Neptune may not be the icy giants we imaginedThey may have a magma ocean brewing on the inside. (Gizmodo) 10 Too much sleep may be as harmful as too littleA new study suggests 6.4–7.8 hours is the sweet spot. (Economist $) Quote of the day “This kind of powerful weapon that can alter the landscape of cyberwarfare can’t remain solely in American hands.”  —360 Security CEO Zhou Hongyi tells a cybersecurity conference in Beijing why Chinese AI firms need to match the capabilities of their rivals in the US, The Wall Street Journal reports. One More Thing GETTY Why Generation Z falls for online misinformation Research shows that young people are more likely to believe and pass on misinformation if they feel a sense of common identity with the person who shared it in the first place.  Offline, teenagers are likely to draw on the context that their communities provide. Social media, however, promotes credibility based on identity rather than community. And when trust is built on identity, authority shifts to influencers. As young people participate in more political discussions online, those who have successfully cultivated identity-based credibility could become de facto community leaders, attracting like-minded people and steering the conversation. While that has the potential to empower marginalized groups, it also exacerbates the threat of misinformation. Find out what we can all learn about how young people evaluate truth online. —Jennifer Neda John We can still have nice things A place for comfort, fun, and distraction to brighten up your day. (Got any ideas? Drop me a line.) + The Euclid space telescope has captured the most detailed image yet of the Milky Way.+ Here’s a lovely, lilting medieval bardcore cover of Daft Punk’s electronic classic Veridis Quo.+ A toilet plunger becomes an unlikely engineering breakthrough in this quest to build a better blowgun.

The Download: metric weaknesses and AI elephant warnings Beitrag lesen »

AI, Committee, Nachrichten, Uncategorized

Agent confidence on the technical frontier

Enterprise investment in AI is booming. Gartner is calling 2026 an “inflection year” for organizations to align their AI projects with strategic business objectives. As the pressure to prove ROI mounts, executives and technology leaders are looking to agentic AI to drive the measurable financial outcomes their businesses seek. A prime opportunity for AI agents exists in the tech function, where IT infrastructure costs are projected to grow two to three times by 2030, even as budgets remain unchanged, according to McKinsey. And in the last 18 months, tech teams—the engineers, developers, architects, and other practitioners who are building, deploying, and continually improving their organizations’ infrastructure and applications—are clearly putting agents to work. DOWNLOAD THE REPORT The ultimate promise of agents is not only to automate tasks but to manage and coordinate entire workflows, pursuing business goals in a way that allows humans and agents to work together. Given the risks involved in automated decision-making, teams cannot delegate the work that agents do without confidence that they are fully capable of performing the task and that it will do so in a safe, reliable, and secure manner. Among technology experts, our research shows that teams are exceedingly confident about using agentic AI across a significant amount of AI, data, and cloud tasks. Where agent readiness drops is largely due to a lack of business context being supplied to agentic systems. The more complex the task, the more reasoning capability an agent requires and the greater its need for business context. Such context-generation capabilities for agents are still at an early stage of development, especially in situations where enterprise data is difficult to wrangle and connect into the agent lifecycle at the speed and quality in which developers and executives need it. Human oversight is a key factor of success in deploying agentic AI. Knowing that tech teams are in a pivotal position to lead this transformation, the experts we interviewed expect agent confidence to accelerate as experience with agents deepens and business environments mature. “As we design agents to operate within the same operational boundaries, identity systems, and governance models that teams already use, they start to behave more like the systems organizations already trust,” says Jeremy Winter, corporate vice president and chief product officer at Microsoft Azure Platform. This report, based on a survey of 300 global technology experts, ranks 101 tasks across AI, data, and cloud workflows based on respondents’ confidence in agents acting on their behalf. It also examines how technology teams view the opportunities and challenges related to agentic AI, along with the potential for the technology to enhance their careers. Key findings from the report include: Confidence in agents is surging for measurable tasks and growing in areas of complex judgment. Technology experts overwhelmingly believe agents help with everyday work including streamlining processes, improving performance, and reducing repetitive tasks. Confidence is highest for processes like generating reports and boilerplate code, and there is clear opportunity where tasks involve multistep workflows and advanced reasoning to make decisions. Data workflows are the breakthrough domain. Tech teams trust agents most where structure can provide a reliable foundation for decisions. This includes areas such as data quality monitoring, visualization anomaly detection, real-time data stream monitoring, and data profiling. This is where domain experts closest to the point of data generation can provide context to allow agents to act and deliver trusted outcomes. Download the full report. Read the Microsoft Cloud blog by Amanda Silver, corporate vice president of Microsoft 365 Core and Work IQ, which underscores the importance of keeping humans in the loop and how systems thinking advances careers. And for a deeper dive into data workflows as a breakthrough use case for agents, check out the Fabric blog to hear from Kim Manis, corporate vice president of Product for Microsoft Fabric. This content was produced by Insights, the custom content arm of MIT Technology Review. It was not written by MIT Technology Review’s editorial staff. It was researched, designed, and written by human writers, editors, analysts, and illustrators. This includes the writing of surveys and collection of data for surveys. AI tools that may have been used were limited to secondary production processes that passed thorough human review.

Agent confidence on the technical frontier Beitrag lesen »

AI, Committee, Nachrichten, Uncategorized

AI agents are not your “coworkers”

This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here. Imagine coming in to work to learn that a new underling will report to you. The worker is not a person but an AI tool—one that your company nonetheless calls Alex, an “employee” with a title and defined responsibilities. How well do you think you would work with Alex? If you’re anything like the managers recently studied by Emma Wiles, a Boston University business professor, treating Alex as a “coworker” and not a software tool would lead you to do a worse job. Wiles found that people caught 18% fewer errors when the work was said to have come from an agentic “AI employee” rather than a chatbot. It turns out that what’s in a name matters. A lot.  This is an alarming glimpse of the future Silicon Valley is hurling us toward. Last year Nvidia’s CEO, Jensen Huang, talked about workplaces of “digital humans.” Since April, Microsoft, OpenAI, Anthropic, and Google have all released new tools oriented toward managing teams of AI agents, many of which are explicitly advertised as digital colleagues with the flexibility and cognitive power of actual humans. And nearly a third of the 1,261 managers who participated in Wiles’s study said their companies already frame AI agents as employees (23% even list them on org charts). The technical progress of agentic AI is not all hot air, of course. Agents, which can effectively be thought of as AI tools programmed to work in a loop until they achieve a goal, have become measurably better at more complicated tasks. But it’s a huge leap to refer to these tools as coworkers or employees, and doing so will set unrealistic expectations for what AI can do while leaving the human employees supposedly responsible for them worse off. That’s partially because, Wiles’s research suggests, it inverts our sense of who’s in charge. When an AI tool was framed as an employee, participants in the study saw themselves as less responsible for its output. They were also 44% more likely to escalate its questionable work to a manager for further review rather than trusting their own corrections (thus negating the time-saving purpose of using the AI agent in the first place).  That matters far beyond office culture: As AI agents are embedded into health care, warfare, education, and government, there’s a growing risk they’ll become a convenient place to dump blame for failures that are instead the product of bad human decisions, incentives, and oversight (recall how the bomb strike on a girls’ school in Iran was popularly blamed on Claude, when all signs point to a cascade of human errors). “AI agents right now are being marketed as things that can replace humans, and I think that’s just a losing proposition,” says Daron Acemoglu, an economist at MIT who won the Nobel Prize in 2024 and studies AI’s impact on the economy. “They should instead be optimized so that they can improve human capabilities, which is not what they have [been] at the moment.” What could that look like? Consider a new effort at Stanford, where researchers presented 1,500 workers in 104 jobs with information about what tasks AI could potentially do in their work and then asked what would actually be most helpful and productive. Workers did want automation in certain areas: Law clerks thought AI could help ensure that adequate progress was being made across cases, for example. But often the tasks that tech experts deemed most suitable for AI—like verifying customer credit ratings for sales reps—were what the actual workers said they definitely did not want or need an agent to do.  Which brings us back to Alex. Calling Alex an employee is easy—and convenient, especially when something goes wrong—but it’s a branding exercise. It doesn’t make the tool more fit for the job, and as Wiles’s research shows, it makes the humans around it worse at theirs. And recall that they are the ones with the agency that AI is trying to replicate. They deserve better than Alex. 

AI agents are not your “coworkers” Beitrag lesen »

AI, Committee, Nachrichten, Uncategorized

OCRmyPDF Tutorial: Convert Scanned Documents into Searchable PDF/A Files with Sidecar Text Extraction and Batch Processing

In this tutorial, we build an advanced, self-contained OCRmyPDF workflow. We start by installing the required system and Python dependencies, then create a synthetic image-only PDF for scanning so we can test OCR without relying on external files. From there, we use OCRmyPDF’s real public API to convert scanned documents into searchable PDFs, generate PDF/A outputs, extract sidecar text, validate the results, compare file sizes, tune Tesseract settings, clean noisy scans, handle already-OCRed files, process images with DPI hints, run OCR in memory, and batch-process multiple PDFs. Through this workflow, we understand how OCRmyPDF can serve as a practical document digitization pipeline for archival, search, extraction, and automated processing tasks. Installing OCRmyPDF System Dependencies Copy CodeCopiedUse a different Browser import io import os import re import sys import time import shutil import logging import textwrap import subprocess from pathlib import Path INSTALL_JBIG2 = True def sh(cmd: str, check: bool = True) -> int: “””Run a shell command, echo it, and show the tail of its output.””” print(f” $ {cmd}”) r = subprocess.run(cmd, shell=True, text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) if r.stdout and r.stdout.strip(): for ln in r.stdout.strip().splitlines()[-12:]: print(” ” + ln) if check and r.returncode != 0: raise RuntimeError(f”Command failed ({r.returncode}): {cmd}”) return r.returncode def install_dependencies() -> None: “””Install OCRmyPDF’s system + Python dependencies for Colab/Ubuntu.””” apt_pkgs = ( “tesseract-ocr tesseract-ocr-eng tesseract-ocr-osd ” “tesseract-ocr-deu tesseract-ocr-fra ” “ghostscript unpaper pngquant poppler-utils qpdf” ) sh(“apt-get update -qq”, check=False) sh(f”DEBIAN_FRONTEND=noninteractive apt-get install -y -qq {apt_pkgs}”) sh(f'”{sys.executable}” -m pip install -q –upgrade ocrmypdf img2pdf “pillow<12″‘) if INSTALL_JBIG2 and shutil.which(“jbig2”) is None: try: build_pkgs = (“autoconf automake libtool pkg-config ” “libleptonica-dev zlib1g-dev build-essential git”) sh(f”DEBIAN_FRONTEND=noninteractive apt-get install -y -qq {build_pkgs}”) sh(“rm -rf /tmp/jbig2enc && ” “git clone -q https://github.com/agl/jbig2enc.git /tmp/jbig2enc”) sh(“cd /tmp/jbig2enc && ./autogen.sh >/dev/null 2>&1 && ” “./configure >/dev/null 2>&1 && make -j2 >/dev/null 2>&1 && ” “make install >/dev/null 2>&1 && ldconfig”) print(” jbig2enc:”, “installed” if shutil.which(“jbig2”) else “built, but binary not on PATH”) except Exception as e: print(” jbig2enc build skipped (optional):”, e) def ensure_installed() -> None: have_tools = bool(shutil.which(“tesseract”) and shutil.which(“gs”)) try: import ocrmypdf import img2pdf from PIL import Image have_py = True except Exception: have_py = False if have_tools and have_py: print(“Dependencies already present — skipping installation.”) else: print(“Installing dependencies (first run can take a few minutes)…”) install_dependencies() ensure_installed() We set up the complete OCRmyPDF environment for Google Colab by importing the required standard libraries and defining the installation workflow. We install system tools such as Tesseract, Ghostscript, unpaper, pngquant, poppler, and qpdf, along with Python packages like OCRmyPDF, img2pdf, and Pillow. We also optionally build jbig2enc so that advanced PDF optimization can produce smaller outputs for scanned documents. Loading OCRmyPDF and Building Synthetic Scans Copy CodeCopiedUse a different Browser def _purge(*prefixes): for name in [m for m in list(sys.modules) if any(m == p or m.startswith(p + “.”) for p in prefixes)]: del sys.modules[name] def _load_ocrmypdf(): _purge(“PIL”, “ocrmypdf”) import ocrmypdf return ocrmypdf try: ocrmypdf = _load_ocrmypdf() except ImportError as e: if “_Ink” in str(e) or “PIL” in str(e): print(“Repairing an incompatible Pillow (reinstalling pillow<12)…”) sh(f'”{sys.executable}” -m pip install -q –force-reinstall “pillow<12″‘) try: ocrmypdf = _load_ocrmypdf() print(“Pillow repaired — continuing without a restart.”) except Exception: raise RuntimeError( “Pillow is still incompatible in this session. Use the Colab menu: ” “Runtime > Restart session, then run this cell again.” ) else: raise from ocrmypdf.exceptions import ( ExitCode, PriorOcrFoundError, EncryptedPdfError, MissingDependencyError, TaggedPDFError, DigitalSignatureError, DpiError, InputFileError, UnsupportedImageFormatError, ) from ocrmypdf.helpers import check_pdf from ocrmypdf.pdfa import file_claims_pdfa import img2pdf from PIL import Image, ImageDraw, ImageFont, ImageFilter logging.basicConfig(level=logging.WARNING, format=”%(levelname)s: %(message)s”) logging.getLogger(“ocrmypdf”).setLevel(logging.WARNING) logging.getLogger(“pdfminer”).setLevel(logging.ERROR) logging.getLogger(“PIL”).setLevel(logging.WARNING) SAMPLE_TEXT_PAGES = [ “Optical Character Recognition, commonly abbreviated as OCR, is the ” “process of converting images of typed or printed text into machine ” “encoded text. This page was generated as a synthetic scan so that the ” “OCRmyPDF pipeline has something realistic to recognize and search.”, “On 14 March 2026 the archive contained 1,482 pages across 37 folders. ” “Roughly 92 percent of those pages were scanned at 200 to 300 dots per ” “inch. The remaining 8 percent were skewed and required deskewing before ” “any reliable recognition was possible.”, “After OCRmyPDF finishes, the output is a searchable PDF/A file. You can ” “select text, copy it, and run full text search across thousands of ” “documents. The original image resolution is preserved while a hidden ” “text layer is placed accurately underneath the page image.”, ] def _find_font(): for cand in ( “/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf”, “/usr/share/fonts/truetype/liberation/LiberationSans-Regular.ttf”, ): if os.path.exists(cand): return cand return None _FONT_PATH = _find_font() FONT = ImageFont.truetype(_FONT_PATH, 40) if _FONT_PATH else ImageFont.load_default() def _add_speckle(img, n=6000, dark=60): “””Sprinkle light dark specks to imitate scanner noise (motivates –clean).””” import random px = img.load() w, h = img.size for _ in range(n): px[random.randint(0, w – 1), random.randint(0, h – 1)] = random.randint(0, dark) return img def render_page(text, skew=False): “””Render one A4 page (1654×2339 px ≈ 200 DPI) of dark text on white.””” W, H = 1654, 2339 img = Image.new(“L”, (W, H), 255) draw = ImageDraw.Draw(img) draw.multiline_text((150, 180), textwrap.fill(text, width=58), fill=25, font=FONT, spacing=18) if skew: img = img.rotate(6, resample=Image.BICUBIC, expand=False, fillcolor=255) img = img.filter(ImageFilter.GaussianBlur(0.6)) img = _add_speckle(img) return img def build_scanned_pdf(pdf_path: Path, pages_text, skew_index=1): “””Render pages to PNGs and wrap them losslessly into an image-only PDF.””” pngs = [] for i, text in enumerate(pages_text): img = render_page(text, skew=(i == skew_index)) p = pdf_path.parent / f”_pg_{pdf_path.stem}_{i}.png” img.save(p, format=”PNG”, dpi=(200, 200)) pngs.append(str(p)) with open(pdf_path, “wb”) as f: f.write(img2pdf.convert(pngs)) for p in pngs: os.remove(p) return pdf_path def do_ocr(input_file, output_file, **kw): “””Wrapper around ocrmypdf.ocr() that disables the progress bar and times it.””” kw.setdefault(“progress_bar”, False) t0 = time.perf_counter() rc = ocrmypdf.ocr(input_file, output_file, **kw) return rc, time.perf_counter() – t0 def tokens(s: str): return re.findall(r”[a-z0-9]+”, s.lower()) def kb(path) -> str: return f”{Path(path).stat().st_size / 1024:,.1f} KB” def banner(title: str): line = “─” * 74 print(f”n{line}n {title}n{line}”) We safely load OCRmyPDF and repair Pillow compatibility issues if they appear in the Colab runtime. We import OCRmyPDF exceptions, PDF validation helpers, img2pdf, and Pillow utilities used throughout the tutorial. We also define the sample document text and helper functions for rendering synthetic scanned pages,

OCRmyPDF Tutorial: Convert Scanned Documents into Searchable PDF/A Files with Sidecar Text Extraction and Batch Processing Beitrag lesen »

We use cookies to improve your experience and performance on our website. You can learn more at Datenschutzrichtlinie and manage your privacy settings by clicking Settings.

Privacy Preferences

You can choose your cookie settings by turning on/off each type of cookie as you wish, except for essential cookies.

Allow All
Manage Consent Preferences
  • Always Active

Save
de_DE