Thursday, February 5, 2009

Academic Publications 2.0

Maybe I'm in the wrong line of work, but I hate writing academic papers.

  • Current conventions in computer science mandate that each time I write a paper, I should state my result four times: once in the abstract, once in the introduction, once in the body, and once in the conclusion.

  • The introduction section is, to a large extent, a waste of everybody's time. Nearly every paper about a given topic has the same introduction: "Topic X is important because A, B, C."

  • Here's a simple formula that seems to work for (and is overwhelmingly used in) computer science papers:

    Abstract: We solve problem Y.
    Introduction: Problem X is so important, that if we solve it, the world will be a much better place. Y is an approximation (or subproblem) to problem X. We solve Y.
    Body: Here's a convoluted explanation of how to solve Y.
    Conclusion: We did Y. Doing X would be awesome.

  • Once a paper is "published," it is set on stone and cannot really be changed, even if you find a much better way to convey the results or if you find that the data is better explained by a different hypothesis. The reason for this restriction is that, 30 years ago, papers were published using physical paper. Such a restriction makes no sense today.

  • As an academic community, it sometimes feels that the final goal of doing research is publishing papers. The goal of doing research should be, well, doing research. I understand that communicating the results of our work is important, but surely there is a better method than one that was invented before computers were around.

  • Given the number of people working in computer science and the fact that publishing papers is considered the goal of our work, there is an insane number of papers written every year, the vast majority of which contribute very little (or not at all) to our collective knowledge. This is basically spam. In fact, for many papers (including some of my own), the actual idea of the paper could be stated in one paragraph, but somehow people manage to write 10 pages of it.

Can a combination of a wiki, karma, and a voting method like reddit or digg substitute the current system of academic publication?

55 comments:

  1. I think there's a vast amount of potential being wasted holding on to the paper medium of the past.

    Simple systems like GPeerReview could take care of providing review and allow for tenure credit (e.g. a signed hash value of a good review of your paper on your CV by someone that's respected). I think the peer review would still be helpful to prevent a "pop culture" mentality that digg might cause.

    I wouldn't stop at writing, I would love it if people with new ideas would learn Presentation Zen skills and convey their results in a YouTube like presentation. Pecha Kucha style would be preferred.

    Perhaps a TED for Computer Science?

    If you go with the traditional writing route, how about some school have a academic golf challenge of who can get the best idea down in the shortest, but easy to read, fashion? Feel free to use color and design skills.

    What's stopping you from leading this?

    Fore!

    ReplyDelete
  2. Voting method like Digg sounds awesome. Seems like the best way to get your research reviewed without having to go through this massive paper waste/nonsense.

    ReplyDelete
  3. For many people research is publishing papers only. The world start and ends with it. Something should be done in this regard, but what are the alternatives????

    ReplyDelete
  4. "Research = Papers" is mostly a cultural problem. One thing that is important to understand (especially for PhD students) is that the number of published papers is not a measure that is widely used when evaluating you. When hiring an assistant professor, for example, I'd say the impact of your top three projects is looked at a lot more than your total number of papers. Widespread understanding of this may help people stop thinking that "research = papers."

    ReplyDelete
  5. Just curious: how do you assess impact?

    ReplyDelete
  6. There is no perfect way to evaluate impact, but some questions that are typically asked are: Have we heard of the work? Has it gotten awards? Does it affect a lot of people? Has it gotten press? How many other papers cite it? How impactful are the other papers that cite it (think PageRank)?

    ReplyDelete
  7. That sounds like a reasonable heuristic. It seems that "all" that is needed to make your 2.0 system work is the right core group agreeing to use it. It'll bootstrap your PageRank philosophy. If the new approach already gets the first half of what you mentioned, it shouldn't be impossible.

    For me, additional paths include hearing about something in Communications of the ACM or a Google Tech Talk is important; this is how I learned about your work.

    Don't settle for 1.0. Keep pushing for this; CS needs this.

    Thanks for the response!

    ReplyDelete
  8. I think ideally you'd want to have two streams of review:
    - reviewing of content (theories, models and research), such as "professor Y's theory of Z",
    - reviewing/rating/etc. of a given presentation of that model.

    The linking of content and presentation is just a structural artifact of the paper system, and should be severed. I think it's also the only way to incorporate something like Moser's points on multimedia presentations, while avoiding the vapid popularity issues of rating systems.

    ReplyDelete
  9. thanks for this post! some 'researchers' in computer science have more than 15-20 papers a year, i.e. two weeks for research, one week of writing, next paper,... i am really tired about this kind of doing research and i am therefore not sure whether i should not leave to industry on the long term. and finally you need a given amount of papers for your tenure track....

    in fact i have another point to make: you write 'How many other papers cite it?' as a measure of impact. that is another difficult problem. the more papers we all write, the less one of these will be cited very often. moreover this is a measure for senior researchers not for phd students and post-docs. moreover such measure depends very much on your community within computer science. there are some areas with only few people working on the topic (but the topic can still be relevant). moreover we already see citing-networks of people citing each other work again and again to improve their h-index (and google scholar (i hate it) only lists like 50% of them and updates 2-3 times a year... not very helpful on post-doc level again. anyway thanks for your post!

    ReplyDelete
  10. I too can't understand, why the research community is behind in switching to the new medium.
    [:O]
    We guys are supposed to be changing the the society with our new and cool ideas, not getting stuck with ancient techniques.

    ReplyDelete
  11. Hmmm. Luis, I think your statement about looking at impact is not true. If it were, I think we would see more professors without PhDs, or professors who came back to teaching after being in industry for a while (the only prof i know who did this is Hank Levy). Several people have made humongous impacts in industry.

    ReplyDelete
  12. Granted, there are many caveats here. First, universities look mostly for "academic" impact. I think this is not necessarily good, but that's the world we live in.

    My statement was meant to more to say that number of papers is less important than the quality of the papers -- which is *certainly* true when we hire in CS here at CMU.

    ReplyDelete
  13. As you point out, top schools and research labs already consider quality / impact over quantity. Academic impact may be a bit incestuous, but I don't see that centuries' (millenia?) old tradition changing any time soon.

    Besides, I thought grants counted at least as much as publications.

    ReplyDelete
  14. Although research !- papers, the act of focusing on a paper often helps focus on a specific problem. This is particularly useful for new students I think, who have a hard time focusing on something and spending enough time on it.

    What I tell students is that a paper is a tangible marker, a goal post if you wish, that helps them focus on making progress with research. Often, the marker will move as they conduct further research, but that's fine: the important point is the forward gradient.

    ReplyDelete
  15. A wiki edit can also be a tangible marker :) I still dislike papers and the fascination academics have with them.

    ReplyDelete
  16. We are not shifting away from traditional publication models because the existing alternative publication models are not providing significantly greater benefits.

    For example, the existing conference and journal systems provide the benefits of recognition and impact. Any alternative model will have to match the strengths of the existing models in addition to addressing their weaknesses in order to be adopted by the academic community.

    With regards to recognition and impact, tools that measure the impact of an online publication could be one way to encourage the use of an alternative. I'd be delighted to see an environment that has the benefits of the existing models and provides additional unique strengths, but it could take some effort.

    ReplyDelete
  17. I think we're not shifting cuz we're stuck on our ways.

    ReplyDelete
  18. I think we have to separate out a number of different factors.

    I don't think papers by themselves are a bad thing. What you seem to object to are (1) the need to write 10 pages when 4 would suffice; and (2) the proliferation of "bad" papers.

    (1) is easy to fix: when reviewing papers, don't penalize them for being short! (Though I find I more often wish a paper were longer than wish it were shorter; maybe papers are just, in general, badly written. This should also be fixed.)

    As for (2), I would be all in favor of cutting down on the number of conferences. That would help, but still wouldn't solve the problem. As I see it, there are two contributing factors to this problem: (2a) the need to publish as much as everyone else (for tenure, especially), and (2b) the need to publish for each grant you are involved in. (2a) could be fixed if people generally published less; reducing the number of conferences would force this to some extent, and as a community we could decide to move in this direction. As for (2b), I don't see any way to fix this in the system we currently have (where professors need to hold several grants in order to get a full-year salary and support graduate students). Perhaps, in the long term, this system can change.

    ReplyDelete
  19. Hi Jonathan :)

    I would also add to my list of objections: (3) The static nature of papers and (4) the fact that papers seem to have become our main goal in life.

    (3) Is quite easy to fix by simply making the "authoritative" version of each paper the one that's posted on your site, or making papers be wikis that only the author can edit.

    (4) is probably my biggest complaint of academia and fixing it would require a major culture shift. For some areas such as theory/math, I think it's less of a problem since the end results are mostly theorems and theorems are approximately equivalent to papers (although not identical, so there is some room to improve things). But for more applied areas, I think papers should really be an afterthought. Currently papers are the end goal of our research so a good strategy for academics is to do enough work just to produce a paper (i.e. test your system with 10 users, build only a mock-up, solve a problem that is of no consequence but enough for a paper, etc.)

    I find it hard to motivate my students to work on a project past what's necessary for a paper, and I think that's a big problem.

    ReplyDelete
  20. I believe that a paper is never finished, and there are always things to improve it. By this reason, I consider it good that one has no chance to modify it, and can move on to other things.
    We could also apply the same idea to films: the director would keep changing slightly the movie.

    ReplyDelete
  21. I'm all in favor of making papers "evolve" in time. I've seen several papers about almost the same subject being published with just a few add-ons, that showed the research evolution.

    Then you have the position paper, the mid-research paper and then the final results paper. You have these several, typical, milestones (which, I recognize, are good to plan your work) that you go through, but that are hard to follow for someone trying to get into the state-of-the-art of your research domain.

    You think: "Ok, these guys are working on this..but...what paper do I cite? Only the last one?"

    That is why the paradigm should shift, at least, to a more flexible, evolving media. The focus would be to define a "place" or "holder" of the continuous work of a certain topic or problem domain.

    We are doing "Deadline-Driven Development", and, although it might have benefits, it just seems old-fashioned and lacking purpose...

    ReplyDelete
  22. Nice article highlighting some truths many people avoid to admit.

    Here are some useful related links:

    Using PageRank to assess scientific importance:
    http://arstechnica.com/science/news/2009/01/using-pagerank-to-assess-scientific-importance.ars

    ranking and mapping scientific journals:
    http://www.eigenfactor.org/methods.htm

    For even more results:
    http://friendfeed.com/search?q=pagerank+scientific

    ReplyDelete
  23. Responding to Luis's point 4) - To play devil's advocate, I could argue that it isn't the part of the academic to develop an idea past the proof of concept stage. Because of the serious amounts of time, teamwork, and other capital required to develop an idea beyond the "5 research subject" stage, this development is best left to start-ups and industry. The proper role of an academic, in this conception, is to perform a breadth-first search over the space of ideas to find something that merits a deeper investment, by taking on "blue sky" directions that can't justify industrial or start-up investment.

    I don't actually believe this, but I do hear it from time to time.

    ReplyDelete
  24. Of course, nobody will argue that pushing papers just to push papers is acceptable, but writing a lot can help you think better. See this blog post of mine:

    http://www.daniel-lemire.com/blog/archives/2008/07/11/do-you-think-because-you-write-or-write-because-you-think/

    ReplyDelete
  25. I disagree that, in the past, results were set on stone because they were published on paper. Authors published several editions of their books, thoroughly revised most of the time. And, for articles, they revisited, for the best of them, the content several years after the initial publication, by making additions and corrections. Anyway publications are just a photographic picture of one's thought at a point in time.

    ReplyDelete
  26. I enjoyed this post (the incipit is fantastic), and the discussion in the comments. I believe that:(i) we (the research community) *must* be aware that the Web, Web 2.0, etc. do provide new tools, and (ii) these tools might change the way we disseminate science (or maybe even the way we do science), and we are the right persons to understand if, when, and how.

    I also add a link to:

    http://etuttounmagnamagna.blogspot.com/2009/03/whats-wrong-with-scholarly-publication.html

    S.

    ReplyDelete
  27. Great Post! I just found this through a cross posting to the valve...

    Its funny how these things vary by discipline. In history the measure of research is the monograph: you are only somebody if you've published a book! (Preferably lots of them). Some schools are catching on and switching to articles but its still measured in book equivalents(4 articles = a monograph).

    I am in the middle of (procrastinating) grading my students essays. I wish there was another way for them to demonstrate their understanding of history: like maybe putting on a Mystery Play or writing a haiku about the Industrial Revolution (with footnotes).

    Thanks again for the great post!

    ReplyDelete
  28. i agree with the post owner, academic writing takes lots of time and research.

    http://academic-essay-writing.blogspot.com

    ReplyDelete
  29. it's really quite interesting post, i enjoyed the comments also.

    ReplyDelete
  30. I am into academic paper writing and I love my work so much. In fact, I am also a part time term paper help tutor to students who have a hard time writing. This is a nice post. Keep posting

    ReplyDelete
  31. Luis, you and your readers may be interested in this paper I wrote that describes something similar to what you're suggesting, and it references this blog article!

    ReplyDelete
  32. This is a good post which poses an excellent question. The advent of Web 2.0 has certainly given the paper based publications a run for their money. Although these may be publishers which may be considered as authorities on various subjects, the fact remains that the general consensus created by the users on Web 2.0 properties may drive recognition online. Therefore a two-pronged approach where Digg, Wiki, etc. along with conventional paper based publications will create the biggest buzz for any paper.

    ReplyDelete
  33. Great article keep on posting this type of fantastic articles, thanks you guys

    ReplyDelete
  34. To draw a parallel from law, the law is created from lots of separate bills that are short pieces of information like papers, most of which just amend existing laws. Thus the law is contained in a set of living documents that are refined through amendments. Each amendment is a well-formed document in its own right, letting an observer see the reasoning for any modification. Some projects now exist to tell you the state of those living laws over time e.g. http://portsea.austlii.edu.au/pit/xml/sa/act/

    ReplyDelete
  35. From what the competition sounds like, my friend has a real opportunity to build something valued by the industry. I hope he goes for it.

    ReplyDelete
  36. Therefore a two-pronged approach where Digg, Wiki, etc. along with conventional paper based publications will create the biggest buzz for any paper.
    Assignment | Coursework | Dissertation

    ReplyDelete
  37. Thus the law is contained in a set of living documents that are refined through amendments. Each amendment is a well-formed document in its own right.
    Essay | Thesis

    ReplyDelete
  38. Many institutions limit access to their online information. Making this information available will be an asset to all.

    ReplyDelete
  39. Many institutions limit access to their online information. Making this information available will be an asset to all.

    ReplyDelete
  40. Whenever i see the post like your's i feel that there are still helpful people who share information for the help of others, it must be helpful for other's. thanx and good job.

    Masters Dissertation Writing

    ReplyDelete
  41. I should state my result four times: once in the abstract, once in the introduction, cara meninggikan badanonce in the body, and once in the conclusion.

    ReplyDelete
  42. A dream is your creative vision for your life in the future. You must break out of your current comfort zone and become comfortable with the unfamiliar and the unknown.
    payday loans alberta, payday loans edmonton, payday loans ottawa

    ReplyDelete
  43. The reason for this restriction is that, 30 years ago, papers were published using physical paper. Such a restriction makes no sense today.tinggi badan

    ReplyDelete
  44. nice post thnks for the info. i will be doing my P.hD after two years from now. useful info thnk you all..

    ReplyDelete
  45. After all, it is the process of writing and choosing the right books to use to support your claim that will create impact on your essay.

    ReplyDelete
  46. Valuable information which you have discussed in your article.I enjoyed very well while reading.Thanks for providing this for us. Really Great post, I look forward to reading more.
    Custom Essays Writing Services UK
    Writing Services for Custom Essays UK, USA, and Canada

    ReplyDelete
  47. I found this a bit too late. :( I really needed it few days ago. But now it will be in my list.

    ReplyDelete
  48. I like the activity in the commenting beginning. It is like second article.

    ReplyDelete