TaskMatrix.AI: Making big models do small jobs with APIs
2024-03-11
(Press-News.org)
A research team at Microsoft has designed an efficiency tool called TaskMatrix.AI that can be used to accomplish a wide variety of specific AI tasks. TaskMatrix.AI connects general-purpose foundation models like GPT-4, the model behind ChatGPT, with specialized models suitable for certain tasks — much like a human project manager. This research was published Feb. 16 in Intelligent Computing, a Science Partner Journal.
Foundation models and specialized models usually have different mechanisms thus are not easily compatible. Rather than modifying and integrating existing models, TaskMatrix.AI bridges the gaps between them through application programming interfaces, or APIs, which enable software components to communicate.
The research team envisioned an AI ecosystem applicable in office automation, robotics, Internet of Things and other domains. Accordingly, their TaskMatrix.AI can perform various digital and physical tasks, give interpretable responses and learn continuously.
TaskMatrix.AI has four key components: a conversational foundation model that understands user inputs across various modalities (such as text and images) and generates executable action code as input for APIs; an API platform that holds a vast repository of APIs and their documentation; an API selector that chooses the most suitable APIs for the foundation model and an action executor that executes the code given by the model. As the ecosystem evolves, API developers can improve the documentation based on user feedback.
The team demonstrated the use of TaskMatrix.AI for processing images and automatically making PowerPoint slides.
During the image processing task, a human interacted with TaskMatrix.AI by typing natural language instructions for complex visual tasks such as image generation, editing and description. TaskMatrix.AI demonstrated its ability to understand human intentions through text-based inputs and provided satisfactory output.
For example, with a tiny input image of a pink flower with a green background and a single instruction to "extend it to 2048 × 4096", TaskMatrix.AI generated a convincing image of vibrant colorful flowers against lush green leaves through question-answering, captioning and object replacement APIs.
The PowerPoint automation task required TaskMatrix.AI to create a set of slides, each introducing a different tech company. ChatGPT served as the foundation model for understanding complex user instructions, such as inserting text, resizing and relocating images and changing the theme for the PowerPoint slides. For example, TaskMatrix.AI successfully inserted and resized five company logos, which it obtained from the Internet, by calling several relevant APIs.
Despite the preliminary validation of TaskMatrix.AI, the team pointed out some challenges ahead, such as finding and adjusting a powerful foundation model, building and maintaining an ideal API platform, and addressing user-level concerns like data security, privacy and customization needs.
END
ELSE PRESS RELEASES FROM THIS DATE:
2024-03-11
A piece of rotting fruit is likely covered in hundreds if not thousands of microscopic roundworms, including C. elegans—a popular experimental model system for studying neurogenetics. With a lifespan of only a few weeks, C. elegans must reproduce quickly and often. The species is made up of hermaphrodites and males. The hermaphrodites have female bodies, can self-fertilize, and can mate with males. Recent research from of the Portman Lab at the Del Monte Institute for Neuroscience at the University of Rochester, found the males do not mate indiscriminately—they are selective about things like age, mating history, and nutrition.
“We ...
2024-03-11
Anemonefish form mutualistic relationships with the sea anemones they live in and these associations are not random: some species such as the yellow-tail anemonefish (Amphiprion clarkii) are generalists and can live in almost any sea anemone, others like the tomato clownfish (Amphiprion frenatus) are specialists, living in only one sea anemone species, the bubble-tip sea anemone (Entacmaea quadricolor). Reasons for these preferences are unclear because we know very little about the genetic diversity of giant sea anemones.
Researchers at the Marine Eco-Evo-Devo Unit and Marine Genomics Unit at the Okinawa Institute of Science ...
2024-03-11
About The Study: In this study of 15,000 infants born extremely preterm, redirection of care discussions (withdrawal, withholding, or limiting escalation of treatment) occurred less often for Black and Hispanic infants than for white and non-Hispanic infants. It is important to explore the possible reasons underlying these differences.
Authors: Jane E. Brumbaugh, M.D., of the Mayo Clinic in Rochester, Minnesota, is the corresponding author.
To access the embargoed study: Visit our For The Media website at this link https://media.jamanetwork.com/
(doi:10.1001/jamapediatrics.2024.0125)
Editor’s ...
2024-03-11
About The Study: The findings of this study of 50 discharge summaries suggest that large language models can be used to translate discharge summaries into patient-friendly language and formats that are significantly more readable and understandable than discharge summaries as they appear in electronic health records. However, implementation will require improvements in accuracy, completeness, and safety. Given the safety concerns, initial implementation will require physician review.
Authors: Jonah ...
2024-03-11
Key Takeaways
Suicide by firearm rates increased 18% in nine years in states that began allowing firearm owners to openly carry a firearm without a permit, a new study finds.
U.S. rates of firearm-related suicide rose from 21,175 in 2013 to 26,328 in 2021, an increase that may be related to more permissive open carry laws.
CHICAGO: In states that relaxed firearm laws to allow openly carrying a loaded firearm in public without a permit, significantly more people died by firearms and ...
2024-03-11
The particle accelerators that enable high-energy physics and serve many fields of science, such as materials, medical, and fusion research, are driven by superconducting magnets that are, to put it simply, quite finicky.
Superconductors are a special class of materials which, when cooled below a certain temperature, carry large electrical currents without resistance. If you arrange the material in coils, the current passing through will produce strong magnetic fields, effectively storing the potential energy of the moving electrons in the form of magnetic field. But if they get ...
2024-03-11
In many cities worldwide, the notion of walking as sustainable urban mobility is becoming increasingly popular. Improving the walkability of cities has many benefits, including improved health, reduced traffic, and consequently lower air pollution. To improve walkability, it is important to conduct a thorough analysis of what factors make cities more walkable.
One essential aspect in analyzing walkability is understanding traveler behavior. Before and during their journey, various factors can influence their path choices. For example, travelers can consider the most efficient route before starting but could encounter unexpected events necessitating route changes. ...
2024-03-11
A Newcastle research team has been exploring the role of vitamin A in the pathogenesis of psychiatric disorders.
Professor Murray Cairns of the University of Newcastle and Hunter Medical Research Institute said the connectivity between neurons in the brain is thought to be altered in people with schizophrenia and other psychiatric conditions.
“Our previous research suggests this could be to do with vitamin A levels or retinol levels, which are known to play an important role in the differentiation, maturation and synaptic function of brain cells.
“Our new study by William Reay and colleagues combined the summary statistics from thousands of individual ...
2024-03-11
A team of researchers from Bohai University in China have designed and synthesized a bifunctional catalyst that can solve the environmental pollution caused by mustard gas and phenolic compounds. They synthesized this bifunctional catalyst, a new three-dimensional polyoxovanadate-based metal-organic framework, under hydrothermal conditions.
Their work is published in the journal Polyoxometalates on March 4, 2024.
The team’s bifunctional catalyst shows satisfying catalytic performances for the selective oxidation of 2-chloroethyl ethyl sulfide (CEES) to corresponding ...
2024-03-11
University of Virginia School of Medicine scientists have found important answers about strokes, heart attacks and cardiovascular diseases by probing the biological glue our bodies create to protect us from those deadly dangers.
The researchers, led by Mete Civelek, PhD, wanted to better understand factors that influence our risk for cardiovascular diseases such as atherosclerosis, the hardening of the arteries. Atherosclerosis is characterized by the buildup of fatty plaques ...
LAST 30 PRESS RELEASES:
[Press-News.org] TaskMatrix.AI: Making big models do small jobs with APIs
