Ótima lista de pesquisadores e associados à cientometria e altmetrics
"No one can read everything. We rely on filters to make sense of the scholarly literature, but the narrow, traditional filters are being swamped. However, the growth of new, online scholarly tools allows us to make new filters; these altmetrics reflect the broad, rapid impact of scholarship in this burgeoning ecosystem. We call for more tools and research based on altmetrics."
This quote is taken from the introduction to the altmetrics manifesto. And the reason it's a manifesto, rather than a mission or vision statement, is arguably because changing the way scholarly impact is measured is going to need something of a revolution – and no revolution is complete without a manifesto.
So why is a revolution needed? Because long before the tools even existed to do anything about it, many in the research community have bemoaned the stranglehold the impact factor of a research paper has held over research funding, careers and reputations. As bloggers Victor Manning and William Gunn wrote: "Influence is only one dimension of importance". Other bugbears include the slowness of peer review and the fact that impact is not linked to an article, but rather to a journal, as this blog from the Scholarly Kitchen points out.
Now though, the tools exists to consider what other factors may be used to determine importance, and they are being refined daily. Alternative metrics (or altmetrics as they are known) have brought together the tech geeks and the research nerds who are eager to define their own measures of excellence. Though many of the communities they form or join, such as Academia.edu, Mendeley or Total-Impact aren't new, wider changes in the research environment (namely, growing support for open access and policy shifts mandating impact measurement) have given altmetrics a new urgency.
But altmetrics are not universally popular. One commenter on the site writes: "[Impact factors] can be (and are) manipulated to a certain degree ... but the alt, total, ultimate, mega etc. metrics are far worse because the link to research quality is less direct, and in terms of some of the indicators, like twitter activity, it is non-existent. Moreover, these metrics will be much easier to manipulate".
So what does the future of impact assessment hold? How will these new metrics develop and how are they likely to be adopted by the sector? Perhaps most importantly, will altmetrics address the abuses of impact factors or simply create abuses of their own, particularly when importance is determined through social media influence.
Join our live chat, Friday, 21 September at 12 BST, to explore these questions and any others you may have.
The discussion takes place in the comment threads beneath the blog.
This content is brought to you by Guardian Professional. To get more articles like this direct to your inbox, become a member of the Higher Education Network.
Panel
Marie E McVeigh, director, JCR and bibliographic policy, Thomson Reuters
Marie is the director of journal citation reports (JCR) and bibliographic policy in the intellectual property and science business of Thomson Reuters. She researched and published on citation analysis, bibliometrics, open access, and other areas where the publishing and information industries intersect. @TR_ScienceWatch
Heather Piwowar, cofounder, total-impact
Heather Piwowar is a cofounder of total-impact, an online tool that lets scholars and organisations tell the full story of their research impact. Heather is also a postdoc at Duke University and the University of British Columbia; she studies how scientists share and reuse research data. @researchremix
William Gunn, head of academic outreach, Mendeley
Mendeley is a leading research management tool for collaboration and discovery. Frustrated with the inefficiencies of the modern research process, William left academia and established the biology program at Genalyte, a diagnostics startup. From there, he moved to Mendeley to pursue his mission of bringing modern network efficiencies to academic research. @mrgunn
Rachel co-director of AVATAR at the University of Greenwich. Rachel is also a 2010 senior TED fellow, visiting research assistant at the University of Southern Denmark and a sustainability innovator who investigates a new approach to building materials called 'living architecture. @livingarchitect
Thad is an electronic publishing analyst and author based in San Francisco and Vancouver, BC. His site, The Future of Publishing, provides in-depth and comprehensive analysis on the industry. Thad has written extensively about publishing and is on the editorial board of the journal Learned Publishing and the Canadian literary journal, Geist. @ThadMcIlroy
Alessandra is an academic and the co-founder and managing director of Open Book Publishers, a social enterprise devoted to the publication of OA academic books and textbooks. Open Book was created by a group of academics at Cambridge in 2008 - since then we have published over 20 books - all free to read online in their entirety. @openbookpublish
Mike Taylor, research specialist, Elsevier Labs
Mike Taylor is a research specialist with Elsevier Labs, an R&D unit within Elsevier. At the moment he specialises in authorship / contributorship, altmetrics and has been involved with Orcid.org since the start. He's also an improviser with Oxford Comedy Deathmatch but he's not this Mike Taylor. @herrison
Judit is a professor at the department of information science of Bar-Ilan University in Israel. One of her main research interests is the broad area of informetrics (bibliometrics, scientometrics, webometrics). More recently she focuses on altmetrics. She is a member of the editorial board of several information science journals.
Nick manages digital strategy for ODI, the UK's leading think tank on international development and humanitarian issues. At ODI, which tends to publish more and more reports and less and less in journals, he built an Altmetrics-like M&E system to track influence and impact. @nicknet
Ciaran O'Neill, journal development manager, BioMed Central
Ciaran works for the open access publisher BioMed Central, where he is involved in the company's journal and article level metrics initiatives. BioMed Central has been heavily involved in the open access movement, and now publishes well over 200 STM journals. As well as article metrics, Ciaran works with journals experimenting with innovative models of peer review. @cjmoneill
Tom is a co-founder of Ubiquity Press, a researcher-led publisher seeking to incentivise and reward the sharing of academic data and software. He is also a postgraduate student at University College Hospital and the Mullard Space Science Laboratory. @ubiquitypress
Informed Strategies LLC works on content innovation, discovery and analytics. Judy is a past president of the Society for Scholarly Publishing and a contributor to the Scholarly Kitchen where she draws on a background in consulting with publishers and libraries to develop user oriented products and services. @JudyLuther
Stuart was, until recently, professor of information and organisation at the University of Sheffield. He now works part-time in the department of economics at Aalto University in Helsinki. He is also general editor of Prometheus, a journal that takes a critical stance in its attitude to the study of innovation.
Teresa Penfield, DESCRIBE Project Manager, University of Exeter
Teresa's current work is focusing on university research impact and what this means to researchers in different disciplines. She is working with academics and professional experts to develop an understanding of how impact can be evaluated and defined. @UofE_impact
The more scholars move their work online from where it was once ephemeral and hidden, the more they are integrating social media to their communication, the closer we are to telling what is the value that they themselves add to their content - and to blending these isolated factors to create a certain taste, a flavor.
Thanks for collecting these posts. I wrote one a couple of months ago over at Jabberwocky Ecology. It mostly focuses on what we should do if we actually want a decent journal level metric, though at the end I concur with several of the posts in the list that it's really time to just move on to paper level metrics anyway.
ReplyDelete
Inspired by four recent blog posts and their comments (Comments at journal websites: just turn them off, Open Access and The Dramatic Growth of PLoS ONE, No Comment?, If you email it, they will comment), I created a graphic to show what users do with PLoS ONE papers.
Scholarly publishing is, and always has been, an adaptable beast. What started as the hand-scribed musings of ancient philosophers evolved into the printed manuscripts of wealthy gentlemen-scientists observing nature or conducting experiments using their own pounds sterling.
Grading journals on how well they share information with readers will help deliver accountability to an industry that often lacks it. By Adam Marcus and Ivan Oransky | August 1, 2012 Scientists are universally familiar with the Impact Factor, even if they're often frustrated with how it can be manipulated and misused.
London startup Mendeley is already beloved by researchers around the planet for helping them manage their work. Now it's unveiled a new product that it hopes can help universities get a better handle on what's happening right now. Goodbye slow, stuffy academia.
“Mind The Gap” inscription on the London Underground, Bank station. (Photo credit: Wikipedia)
With the speed of communication today, researchers, authors, and grant funders are impatient to get an indicator of its value. Waiting 1-3 years for publication and citation seems interminable. Conflating an article’s impact with its journals’ impact creates uncertainty, as well.
Altmetrics attempts to close that gap by providing more timely measures that are also more pertinent to the researcher and their article. Use metrics from downloads and blogs, and attention metrics such as tweets and bookmarks, can provide immediate indicators of interest. Although metrics associated with these activities are in the developmental stage, there is growing investment in the broader landscape to produce more current metrics that serve the researcher, their communities and funding agencies.
In January, the Chronicle of Higher Education highlighted the work of Jason Priem, a PhD candidate at the School of Information and Library Science at University of North Carolina-Chapel Hill, who coined the term “altmetrics.” In his post, “Altmetrics: a Manifesto,” Jason noted the limitations and slowness of peer review and citations. He suggests that the speed with which altmetrics data are available could potentially lead to real-time recommendation and collaborative filtering systems. Jason and Heather Piwowar, who works at the Dryad Digital Repository, created Total-impact as a prototype in their spare time last year. Two months ago, they received a grant from the Sloan Foundation, and next month Heather will be working full time to more fully develop it and provide context on the data set.
While it may be easy to dismiss the idea that social media metrics can be meaningful for scholars, PLoS has been developing a suite of measures over the last three years referred to as article-level metrics (ALM) that provide a view of the performance and reach of an article. Their approach is to present totals of multiple data points including:
- Usage data (HTML views and PDF downloads)
- Citations (PubMed Central, Scopus, Crossref, Web of Science)
- Social networks (CiteULike, Connotea, Facebook, Mendeley)
- Blogs and media coverage (Nature, Research blogging, Trackbacks)
- Discussion activity on PLoS (reader’s comments, notes and ratings)
Martin Fenner, an MD and cancer researcher in Germany, is working full-time as technical lead on PLoS ALM as of this summer. He brings experience as the creator of ScienceCard (author-level metrics) and is involved with the Open Researcher and Contributor ID (ORCID). As Cameron Neylon said in his 2009 article with Shirley Wu in PLoS Biology, “The great thing about metrics . . . is that there are so many to choose from.”
So which measures matter? Earlier this year, Phil Davis questioned Gunther Eysenbach’s assertion that tweets can predict citations in his article. Mendeley data, however, appear more relevant, and several research papers presented this year show a strong correlation with citation data. In fact, patterns of use indicate that some papers are widely shared but seldom cited while others are frequently cited but appear to have limited readership. In a recent presentation, William Gunn of Mendeley noted looking ahead that:
Useful as these new metrics are, they tell only part of the story. It’s a useful bit of info to know the volume of citations or bookmarks or tweets about a paper or research field, but the real value lies in knowing what meaning the author intended to express when he linked paper A to paper B. Through the layer of social metadata collected around research objects at Mendeley we can start to address this challenge and add some quality to the quantitative metrics currently available.
A growing community is forming around the topic and the conversation in June at the Altmetrics12 program focused on exploring the use of emerging tools and sharing findings of research results. As part of the Association for Computing Machinery (ACM) Web Science Conference, this daylong workshop attracted 60 very active participants in this budding community. Keynotes by Johan Bollen (Indiana University) and Gregg Gordon (SSRN) were accompanied by discussions of research and demonstration of 11 different tools.
One of those tools, Altmetric.com, created by Euan Adie, won Elsevier’s Apps for Science competition last year and now is part of the family of research tools at Digital Science supported by Macmillan Publishers. The Altmetric Explorer tracks conversations around the scientific articles in tweets, blog posts, and news that are analyzed and represented by a score in a “donut” made of colors that reflect the mix of sources.
An important component of the altmetric community was represented by founders of two leading academic social networks: Mendeley (which also competes with citation managers) and Academia.edu (whose competition includes ResearchGate). Since these tools enable researchers to collaborate by posting and sharing their work, the data from these systems could potentially offer fertile ground for data to support the growth of altmetrics.
The most recent entrant in this arena is Plum Analytics, founded by Andrea Michalek and Mike Buschman, who were team leaders in the successful development and launch of ProQuest’s Summon. Andrea is building a “researcher reputation graph” that mines the web, social networks, and university-hosted data to map relationships between a researcher, his institution, his work, and those who engage with it. An interview with Andrea in semanticweb.com described how she is dealing with the issues of identifying a single researcher and a single article:
The Researcher Graph is seeded with departmental ontologies, document objects IDs for published articles, ISBNs for books, and other information universities typically already have data about. For the Document ID, it is creating a set of aliases and rules to find different URIs by which a work is referenced, since one paper can be living legitimately at 50 different places around the web; so, when someone tweets a link for one of these resources it will know it is the same one that might go by a different alias from another publisher.
The resulting data set could provide a current complement to the institutional citation analysis services offered by Thomson’s Research in View and Elsevier’s SciVal.
The dimensions of altmetrics extend well beyond an effort to capture social media activity and use it as an indicator of subsequent citation ranking. New services, such as Mendeley whose goal is ‘to manage your research’ and Academia.edu whose goal is ‘to accelerate research’ are tools for researchers where use data is a by-product that contributes to the scholars’ ‘footprint’. Altmetrics seeks to quantify the response to research and ultimately its influence across a global community.
Competition to secure grants and promotions are familiar drivers in the demand for metrics that represent value. While it is still early days for altmetrics, this nascent movement is gathering steam and will be the topic of many future conversations as we engage in evaluating the qualitative aspects of a new set of metrics that will — like medicine — be complementary rather than alternative.
Like this:
4 bloggers like this.
Journal ranking schemes may seem useful, but Björn Brembs discusses how the Thompson Reuters Impact Factor appears to be a reliable predictor of the number of retractions, rather than citations a given paper will receive. Should academics think twice about the benefits of publishing in a 'high impact' journal?
Ferramenta para medida de métricas alternativas. Grátis para indivíduos poderem medir artigos e disponível para publicadores.
"Altmetric Explorer is an exciting, powerful and useful tool"
Altmetric Explorer is an exciting, powerful and useful tool for BioMed Central. It gives us a targeted view of how much impact each individual paper is having, and where in the world of social media and the web these comments are being made.
Rebecca Fairbairn, BioMed CentralAltmetric is currently the only product which provides us with article level metrics on social media activity. This allows us to monitor the public discussions of content in social media and to evaluate these, which confirms our interest in discussions that go beyond the publishing process.
Martijn Roelandse, Springer"Has been really popular with our users"
Altmetric Explorer has the potential to change the way we understand the impact that research has within academia and more importantly the wider impact that it has in society.
Ian Mulvany, eLifeThe Altmetric API made integration with Utopia Documents really easy. Having up-to-date information about a paper's impact has been really popular with our users, and really helps keep articles plugged into the ongoing conversation.
Steve Pettifer, Utopia Docs"Keeps articles plugged into the ongoing conversation"
We want to know what people think - and are saying - about the articles Nature Chemistry is publishing. Altmetric Explorer is a very quick and easy way to identify which papers seem to be generating buzz and links out to the actual coverage and so lets us know who is engaging with our content.
Stuart Cantrill, Nature Publishing Group
Site que agrega diferentes métricas de artigos e autores (incluindo apresentações no Slide-Share)
Total Impact
Tentativa mal-sucedida de entender o fator de impacto
The integrity of data, and transparency about their acquisition, are vital to science. The impact factor data that are gathered and sold by Thomson Scientific (formerly the Institute of Scientific Information, or ISI) have a strong influence on the scientific community, affecting decisions on where to publish, whom to promote or hire (1), the success of grant applications (2), and even salary bonuses (3).
Apresentação de Kristen Fisher Ratan (PLoS) sobre métricas alternativas
Ferramenta de Article-Level Metrics (ALM) desenvolvida pela PLoS
Want NIH applications to consider research achievements more broadly in the Biosketch? Respond to the NIH RFI,... Latest Developments in PLoS Article-Level Metrics... altmetrics12: a workshop on the research and practice of tracking scholarly impact. submit your abstract by May... Martin Fenner interviews Martin Fenner... New data sources added to article-level metrics...
No one can read everything. We rely on filters to make sense of the scholarly literature, but the narrow, traditional filters are being swamped. However, the growth of new, online scholarly tools allows us to make new filters; these altmetrics reflect the broad, rapid impact of scholarship in this burgeoning ecosystem. We call for more tools and research based on altmetrics.
As the volume of academic literature explodes, scholars rely on filters to select the most relevant and significant sources from the rest. Unfortunately, scholarship’s three main filters for importance are failing:
- Peer-review has served scholarship well, but is beginning to show its age. It is slow, encourages conventionality, and fails to hold reviewers accountable. Moreover, given that most papers are eventually published somewhere, peer-review fails to limit the volume of research.
- Citation counting measures are useful, but not sufficient. Metrics like the h-index are even slower than peer-review: a work’s first citation can take years. Citation measures are narrow; influential work may remain uncited. These metrics are narrow; they neglect impact outside the academy, and also ignore the context and reasons for citation.
- The JIF, which measures journals’ average citations per article, is often incorrectly used to assess the impact of individual articles. It’s troubling that the exact details of the JIF are a trade secret, and that significant gaming is relatively easy.
Tomorrow’s filters: altmetrics
In growing numbers, scholars are moving their everyday work to the web. Online reference managers Zotero and Mendeley each claim to store over 40 million articles (making them substantially larger than PubMed); as many as a third of scholars are on Twitter, and a growing number tend scholarly blogs.
These new forms reflect and transmit scholarly impact: that dog-eared (but uncited) article that used to live on a shelf now lives in Mendeley, CiteULike, or Zotero–where we can see and count it. That hallway conversation about a recent finding has moved to blogs and social networks–now, we can listen in. The local genomics dataset has moved to an online repository–now, we can track it. This diverse group of activities forms a composite trace of impact far richer than any available before. We call the elements of this trace altmetrics.
Altmetrics expand our view of what impact looks like, but also of what’s making the impact. This matters because expressions of scholarship are becoming more diverse. Articles are increasingly joined by:
- The sharing of “raw science” like datasets, code, and experimental designs
- Semantic publishing or “nanopublication,” where the citeable unit is an argument or passage rather than entire article.
- Widespread self-publishing via blogging, microblogging, and comments or annotations on existing work.
Because altmetrics are themselves diverse, they’re great for measuring impact in this diverse scholarly ecosystem. In fact, altmetrics will be essential to sift these new forms, since they’re outside the scope of traditional filters. This diversity can also help in measuring the aggregate impact of the research enterprise itself.
Altmetrics are fast, using public APIs to gather data in days or weeks. They’re open–not just the data, but the scripts and algorithms that collect and interpret it. Altmetrics look beyond counting and emphasize semantic content like usernames, timestamps, and tags. Altmetrics aren’t citations, nor are they webometrics; although these latter approaches are related to altmetrics, they are relatively slow, unstructured, and closed.
How can altmetrics improve existing filters?
With altmetrics, we can crowdsource peer-review. Instead of waiting months for two opinions, an article’s impact might be assessed by thousands of conversations and bookmarks in a week. In the short term, this is likely to supplement traditional peer-review, perhaps augmenting rapid review in journals like PLoS ONE, BMC Research Notes, or BMJ Open. In the future, greater participation and better systems for identifying expert contributors may allow peer review to be performed entirely from altmetrics.
Unlike the JIF, altmetrics reflect the impact of the article itself, not its venue. Unlike citation metrics, altmetrics will track impact outside the academy, impact of influential but uncited work, and impact from sources that aren’t peer-reviewed. Some have suggested altmetrics would be too easy to game; we argue the opposite. The JIF is appallingly open to manipulation; mature altmetrics systems could be more robust, leveraging the diversity of of altmetrics and statistical power of big data to algorithmically detect and correct for fraudulent activity. This approach already works for online advertisers, social news sites, Wikipedia, and search engines.
The speed of altmetrics presents the opportunity to create real-time recommendation and collaborative filtering systems: instead of subscribing to dozens of tables-of-contents, a researcher could get a feed of this week’s most significant work in her field. This becomes especially powerful when combined with quick “alt-publications” like blogs or preprint servers, shrinking the communication cycle from years to weeks or days. Faster, broader impact metrics could also play a role in funding and promotion decisions.
Road map for altmetrics
Speculation regarding altmetrics (Taraborelli, 2008; Neylon and Wu, 2009; Priem and Hemminger, 2010) is beginning to yield to empirical investigation and working tools. Priem and Costello (2010) and Groth and Gurney (2010) find citation on Twitter and blogs respectively. ReaderMeter computes impact indicators from readership in reference management systems. Datacite promotes metrics for datasets. Future work must continue along these lines.
Researchers must ask if altmetrics really reflect impact, or just empty buzz. Work should correlate between altmetrics and existing measures, predict citations from altmetrics, and compare altmetrics with expert evaluation. Application designers should continue to build systems to display altmetrics, develop methods to detect and repair gaming, and create metrics for use and reuse of data. Ultimately, our tools should use the rich semantic data from altmetrics to ask “how and why?” as well as “how many?”
Altmetrics are in their early stages; many questions are unanswered. But given the crisis facing existing filters and the rapid evolution of scholarly communication, the speed, richness, and breadth of altmetrics make them worth investing in.
v 1.0 – October 26, 2010
v 1.01 – September 28, 2011: removed dash in alt-metrics
Don't worry, we won't post anything.
3
15
