Information asymmetry, complexity, and structured products

Sorry for the long inactivity, but I am totally caught up in end-of-year wrapping up. I have to admit that I find it quite dissatisfying when you realize the year is almost over and there are still so many things on your list that should have been done… So the last months of the year somehow always end up in total chaos…. anyways…

I came across a very interesting, recent paper (via Mike’s blog post – read also his excellent post) by Arora, Barak, Brunnermeier, and Ge with the title “Computational Complexity and Information Asymmetry in Financial Products” – the authors also provide an informal discussion of the relevance for derivative pricing in practice. As I worked with structured products myself for some time, this paper raised my interest and if you are interested in the trade-off between, say, full rationality and bounded-rationality when it comes to pricing, you should definitely give it a look as well.

The paper deals with the effect of information asymmetry between the structuring entity (which is often also the seller) and the buyer of a structure. The considered derivatives in the paper are CDO like structures and, running the risk of over-simplification, the main points are as follows:

1. Having a set of assets that should be structured into derivatives, here CDOs, a malicious structurer can hide a significant amount of junk assets when assigning the assets to the derivatives. More precisely, the structurer can ensure that the junk assets are overrepresented in a certain subset of the derivatives to be structured which significantly deteriorates their value.
2. A buyer with full-rationality (which can here perform exponential time computations) can actually detect this tampering by testing all possible assignment subsets and verifying that there is actually an/no over-representation.
3. On the other hand, a buyer with limited computational resources, say which is only capable of performing polynomial time computations (the standard assumption when considering efficient algorithms that behave well in the size of the input) cannot detect that the assignments of the assets to the derivatives has been tampered.
4. Under some additional assumptions, the tampering is even ex post undetectable.
5. The authors propose different payoff function that are more resistant to tampering in the sense that heavy, detectable tampering is needed to skew the payoff profile significantly.

Now the authors devise a model similar to Akerlof’s lemons. Stated in a simplified way, the buyer, knowing that he cannot detect the tampering, will assume that a tampering has been performed and is only willing to pay the adjusted price factoring in the potential tampering of the structure – adverse selection. The honest structurer is not willing to sell his derivatives for the reduced price and leaves the market. This effect, based on the information asymmetry between buyer and seller (which was exemplified in Akerlof’s paper using the market of used cars) in the classical setting would lead to a complete collapse of the market as it would repeat ad infinitum until nobody would be left willing to trade. Countermeasures stopping this vicious circle are warranties in the case of the cars. The variant considered here for the structured products will likely converge to the point where the maximum amount of tampering has been performed and buyers and sellers expectations or levels of information are aligned.

What particularly fascinated me is the type of problem encoded to establish intractability. Contrary to the classical NP-hard problems known in optimization that mostly ask for some kind of an optimal combinatorial solution, the authors use the densest subgraph problem/assumption which asserts that deciding between two random distributions (here the fair one and the tampered one) cannot be done in polynomial time (provided that the tampering is not too obvious). In particular:

Densest subgraph problem. Let be a bipartite graph with out-degree for all vertices in . The densest subgraph problem for is to distinguish between the two distributions:

1. which is obtained by choosing for every vertex in an amount of neighbors in  randomly. (what would be the fair assignment)
2. which is obtained by first choosing and with and , and then choosing neighbors for every vertex outside of , and random neighbors for every vertex in . Then we choose random additional neighbors in for every vertex in . (which means that we choose some assets and some derivatives a priori and we prefer to add edges between those sets — slightly simplified. On the rest we do random assignments)

Then the densest subgraph assumption states that whenever, as functions of are chosen sufficiently moderate, then we cannot distinguish between those two distributions, i.e., we cannot detect the tampering with a polynomial time algorithm:

Densest subgraph assumption. Let be such that , then there is no and poly-time algorithm that distinguishes between and with advantage .

Note that the vertices correspond to the structures and the to the underlyings/assets. Although asymptotically intractable, what would be interesting to know is what one can do in practice for reasonable instance sizes, i.e, up to which degree one would be actually able to detect tampering. As Mike already said:

In particular, if a group put out 1000 groupings of financial instruments, and I needed to solve the densest subgraph problem on the resulting instance, I would work very hard at getting an integer program, constraint program, dynamic program, or other program to actually solve the instance (particularly if someone is willing to pay me millions to do so).  If the group then responded with 10,000 groupings, I would then simply declare that they are tampering and invoke whatever level of adverse selection correction you like (including just refusing to have anything to do with them).  Intractable does not mean unsolvable, and not every size instance needs more computing than “the fastest computers on earth put together”.

Another point might be that there are potentially billions of ways of tampering structured products. Especially when the payoff profiles are highly non-linear (e.g., FX-ratchet swaps with compounding coupons) deliberate over-/underestimation of parameters might completely change the valuation of the structures. The proposed framework highlights that there might be ways of tampering that we cannot detect in the worst case, even ex-post (under additional assumptions). But before we can actually detect tampering we have to be aware of this kind of tampering and we have a real problem if tampering is undetectable ex post – how to prove it? This is in some sense related to the stated open question 3: Is there an axiomatic way of showing that there are no tamper-proof derivatives – slightly weakened: with respect to ex-post undetectability.

I could also very well imagine that when giving a closer look to traded structures (especially the nasty OTC ones), that there will be more pricing problems that are essentially intractable. It is almost like one of the main hurdles so far to establish intractability was the more stochastical character of prizing problems while hardness is often stated in terms of some kind of combinatorial problem. An approach like the one proposed in the article might overcome this issue by establishing hardness via distinguishing two distributions.

Sketch your perfect picture

Check this out:

We present a system that composes a realistic picture from a simple
freehand sketch annotated with text labels. The composed picture
is generated by seamlessly stitching several photographs in agreement
with the sketch and text labels; these are found by searching
the Internet. Although online image search generates many inappropriate
results, our system is able to automatically select suitable
photographs to generate a high quality composition, using a filtering
scheme to exclude undesirable images. We also provide a novel
image blending algorithm to allow seamless image composition.
Each blending result is given a numeric score, allowing us to find
an optimal combination of discovered images. Experimental results
show the method is very successful; we also evaluate our system using
the results from two user studies.

Here is the video:

And the link to the paper — they actually use dynamic programming for parts of their algorithm.

Arms race in quantitative trading or not?

Rick Bookstaber recently argued that the arms race in high frequency trading, a form of quantitative trading where effectively time = money , results in a net drain of social welfare:

A second reason is that high frequency trading is embroiled in an arms race. And arms races are negative sum games. The arms in this case are not tanks and jets, but computer chips and throughput. But like any arms race, the result is a cycle of spending which leaves everyone in the same relative position, only poorer. Put another way, like any arms race, what is happening with high frequency trading is a net drain on social welfare.

It is all about milliseconds and being a tiny little bit faster:

In terms of chips, I gave a talk at an Intel conference a few years ago, when they were launching their newest chip, dubbed the Tigerton. The various financial firms who had to be as fast as everyone else then shelled out an aggregate of hundreds of millions of dollar to upgrade, so that they could now execute trades in thirty milliseconds rather than forty milliseconds – or whatever, I really can’t remember, except that it is too fast for anyone to care were it not that other people were also doing it. And now there is a new chip, code named Nehalem. So another hundred million dollars all around, and latency will be dropped a few milliseconds more.

In terms of throughput and latency, the standard tricks are to get your servers as close to the data source as possible, use really big lines, and break data into little bite-sized packets. I was speaking at Reuters last week, and they mentioned to me that they were breaking their news flows into optimized sixty byte packets for their arms race-oriented clients, because that was the fastest way through network. (Anything smaller gets queued by some network algorithms, so sixty bytes seems to be the magic number).

Although high-frequency trading is basically about being fast and thus time is the critical resource, in quantitative trading, in general, it is all about computational resources and having the best/smartest ideas and strategies. The best strategy is worthless if you lack the computational resources to crunch the numbers and, vice versa, if you do have the computational power but no smart strategies this does not get you anywhere either.

Jasmina Hasanhodzic, Andrew W. Lo, Emanuele Viola argue in their latest paper “A Computational View of Market Efficiency” that efficiency in markets has to be considered with respect to the level of computational sophistication, i.e., as market can (appear to) be efficient for those participants which use only a low level of computational resources, whereas it can be inefficient for those participants that invest a higher amount of computational resources.

In this paper we suggest that a reinterpretation of market efficiency in computational terms might be the key to reconciling this theory with the possibility of making profits based on past prices alone. We believe that it does not make sense to talk about market efficiency without taking into account that market participants have bounded resources. In other words, instead of saying that a market is “efficient” we should say, borrowing from theoretical computer science, that a market is efficient with respect to resources S, e.g., time, memory, etc., if no strategy using resources S can generate a substantial profit. Similarly, we cannot say that investors act optimally given all the available information, but rather they act optimally within their resources. This allows for markets to be efficient for some investors, but not for others; for example, a computationally powerful hedge fund may extract profits from a market which looks very efficient from the point of view of a day-trader who has less resources at his disposal—arguably the status quo.

More precisely, it is even argued that the high-complexity traders gain from the low-complexity traders (of course, within the studied, simplified market model – but nonetheless!!):

The next claim shows a pattern where a high-memory strategy can make a bigger profit after a low-memory strategy has acted and modified the market pattern. This profit is bigger than the profit that is obtainable by a high-memory strategy without the low-memory strategy acting beforehand, and even bigger than the profit obtainable after another high- memory strategy acts beforehand. Thus it is precisely the presence of low-memory strategies that creates opportunities for high-memory strategies which were not present initially. This example provides explanation for the real-life status quo which sees a growing quantitative sophistication among asset managers.

Informally, the proof of the claim exhibits a market with a certain “symmetry.” For high-memory strategies, the best choice is to maintain the symmetry by profiting in multiple points. But a low-memory strategy will be unable to do. Its optimal choice will be to “break the symmetry,” creating new profit opportunities for high-memory strategies.

So although in pure high-frequency trading, the relevance of smart strategies might be smaller and thus it is more (almost only?) about speed, in general quantitative trading it seems like (again in the considered model) that the combination of strategy and high computational resources might generate a (longer-term) edge. This edge cannot necessarily be compensated with increased computational resources only, as you still need to have access to the strategy. The considered model considers memory as a the main computational/limiting resource. One might argue that it reflects the sophistication of the strategy along with the real computational resources implicitly, as limited memory might not be able to hold a complex strategy. On the other hand a lot of memory is pointless without a strategy using it. So both might be considered to be intrinsically linked.

An easy example illustrating this point is maybe the following. Consider the sequence “MDMD” and suppose that you can only store, say these 4 letters. A 4-letter-strategy might predict something like “MD” for the next two letters. If those letters though represent the initial of the weekdays, the next 3 letters will be “FSS”. It is impossible though to predict this sequence solely using information about the past on the last 4 letters. The situation changes if we can store up to 7 letters “FSSMDMD”. Then a prediction is possible.

One point of the paper is now that the high-complexity traders might fuel their profits by the shortsightedness of the low-complexity traders. And thus an arms race might be a consequence (to exploit this asymmetry on the one hand and to protect against exploitation on the other). To some extent this is exactly what we are seeing already when traders with “sophisticated” models, that for example are capable of accounting for volatility skew, arbitrage out less sophisticated traders. On the other hand, it does not help to use a sophisticated model (i.e., more computational resources) if one doesn’t know how to use it, e.g., a Libor market model without an appropriate calibration (non-trivial) is worthless.

Interesting things to read…

I just came back from the ISMP in Chicago and as I didn’t write a single line about it (I feel guilty :S), I at least wanted to share a few interesting links with you.

1. Terry Tao’s talk on mathematics and the internet [pdf]
   (Source: Michael Nielsen’s Blog)
2. The impact factor’s Matthew effect: a natural experiment in bibliometrics
   “Using an original method for controlling the intrinsic value of papers–identical duplicate papers published in different journals with different impact factors–this paper shows that the journal in which papers are published have a strong influence on their citation rates, as duplicate papers published in high impact journals obtain, on average, twice as much citations as their identical counterparts published in journals with lower impact factors. The intrinsic value of a paper is thus not the only reason a given paper gets cited or not; there is a specific Matthew effect attached to journals and this gives to paper published there an added value over and above their intrinsic quality. “
   (Source: Michael Nielsen’s Blog)

3. US Top All-Time Donors 1989-2008
   Slightly off-topic but I actually have to admit that I was surprised
   (Source: Michael Nielsen’s Blog)
4. The Status of the P Versus NP Problem (L. Fortnow)
   A nice summary about P Versus NP, why it is so hard and about potential consequences if the question would be settled.
5. The Arms Race in High Frequency Trading (R. Bookstaber)
   “A second reason is that high frequency trading is embroiled in an arms race. And arms races are negative sum games. The arms in this case are not tanks and jets, but computer chips and throughput. But like any arms race, the result is a cycle of spending which leaves everyone in the same relative position, only poorer. Put another way, like any arms race, what is happening with high frequency trading is a net drain on social welfare.”
6. Stockmeyer’s Approximate Counting Method (R.J. Lipton)
   Great blog post on approximate counting.
7. Multitasking Muddles Brains, Even When the Computer Is Off (Wired)
   On the perils of preemptive multi-tasking and just quickly checking emails yet again…
8. Goldman and High Frequency Trading (R. Bookstaber)
   related to 5. above.
9. Not with a Bang but a Whimper – The Risk from High Frequency and Algorithmic Trading (R. Bookstaber)
   related to 5. above.

Rejecta Mathematica – your paper just got an extra life

A few weeks ago The Economist ran an article (thx for the pointer) on Rejecta Mathematica, an open access journal that offers an outlet for papers that do not necessarily fit well into any other journal or have been rejected for various other reasons. From The Economist article:

PAUL LAUTERBUR, the father of magnetic-resonance imaging, had his seminal paper rejected when he first submitted it to Nature. Peter Higgs, eponymous predictor of physics’s missing boson, faced similar trouble with Physics Letters. But Lauterbur went on to win a Nobel prize for his work, and Dr Higgs is an odds-on favourite to get one soon. A good, rejected paper, then, is by no means an oxymoron.

And that observation is the basis of Rejecta Mathematica, an open-source academic journal that recently went online. As its name suggests, the new journal publishes only papers that, like Lauterbur’s and Dr Higgs’s, have been previously submitted to, and rejected by, others. With Annals of Mathematics, one of the best, denying entry to more than 300 last year alone, Rejecta could be busy.

The inaugural issue from July 2009 is available here. Don’t miss out on the letter from the editors in that issue which provides insight into the why, who, how, and when. From the letter:

First, there is ample evidence that in the traditional review process, significant (even Nobel prize-winning) research is occasionally overlooked and groundbreaking work is some-times actively shunned [2–4]. Perhaps this is most dramatically illustrated in the fact that at least “36 future Nobel Laureates encountered resistance on [the] part of scientific journal editors or referees to manuscripts that dealt with discoveries that on [a] later date would assure them the Nobel Prize” [5]. While it would be presumptuous for us to assume that we can spot significant work that others may have missed, we can provide a venue to introduce rejected work to the community and  increase the chances that its value will be appreciated sooner rather than later.

Second, there is also evidence that a research community can derive value from a centralized repository of rejected papers, even when (and perhaps especially when) the results are either incorrect or not significant enough to warrant consideration for a ma jor international prize. Rejecta Mathematica can benefit authors looking for feedback on their work, wanting to warn the community against false starts (i.e., the classic “null results” that never see the light of day, only to be repeated by others) [6, 7], or wanting to illuminate the occasional vagaries of the peer review process to enhance accountability and scientific integrity [8]. Our journal can also benefit readers who want access to “minor results” that may be useful but not publishable in isolation. Indeed, Rejecta Mathematica has existed in folklore for many years as a fictitious place to send papers that were never to see the light of day, and the concept of a formal repository for rejected papers hoping to be discovered and revived (called Rejuvenatable Mathematics) has also been proposed [9].

Mario AI Competition 2009

Yesterday I learned about the Mario AI Competition 2009 (thx for the pointer). The goal of the competition is to provide an agent that automatically navigates through a version of the Mario game:

This competition is about learning, or otherwise developing, the best controller (agent) for a version of Super Mario Bros.

The controller’s job is to win as many levels (of increasing difficulty) as possible. Each time step (24 per second in simualated time) the controller has to decide what action to take (left, right, jump etc) in response to the environment around Mario.

We are basing the competition on a heavily modified version of the Infinite Mario Bros game by Markus Persson. That game is an all-Java tribute to the Nintendo’s seminal Super Mario Bros game, with the added benefit of endless random level generation. We believe that playing this game well is a challenge worthy of the best players, the best programmers and the best learning algorithms alike.

Sounds like a perfect application for some optimization and in fact Robin Baumgarten programmed an agent partly based on the A* algorithm. Check out the video below to see how fast the agent is actually moving through the levels:

A comment on the youtube video: “So I undestand correctly: da computer will play video games for us, so we have more free time? Way cool.”

GLPK 4.39 released / Gusek update available

A new version of the GNU Linear Programming Kit (GLPK) has been released yesterday. From the release notes:

GLPK 4.39 — Release Information
********************************

Release date: Jul 26, 2009

GLPK (GNU Linear Programming Kit) is intended for solving large-scale
linear programming (LP), mixed integer linear programming (MIP), and
other related problems. It is a set of routines written in ANSI C and
organized as a callable library.

In this release:

The following new API routines were added:

glp_warm_up           “warm up” LP basis
glp_set_vertex_name   assign (change) vertex name
glp_create_v_index    create vertex name index
glp_find_vertex       find vertex by its name
glp_delete_v_index    delete vertex name index
glp_read_asnprob      read assignment problem data in DIMACS
format
glp_write_asnprob     write assignment problem data in DIMACS
format
glp_check_asnprob     check correctness of assignment problem
data
glp_asnprob_lp        convert assignment problem to LP
glp_asnprob_okalg     solve assignment problem with the
out-of-kilter algorithm
glp_asnprob_hall      find bipartite matching of maxumum
cardinality with Hall’s algorithm

Also were added some API routines to read plain data files.

The API routines glp_read_lp and glp_write_lp to read/write
files in CPLEX LP format were re-implemented. Now glp_write_lp
correctly writes double-bounded (ranged) rows by introducing
slack variables rather than by duplicating the rows.

Also a new version of Gusek including GLPK 4.39 has been released.

The Netflix Prize – shootout on the finish line

From a New York times article:

After nearly three years and entries from more than 50,000 contestants, a multinational team says that it has met the requirements to win the million-dollar Netflix Prize: It developed powerful algorithms that improve the movie recommendations made by Netflix’s existing software by more than 10 percent.

The online movie rental service uses its Cinematch software to analyze each customer’s film-viewing habits and recommends other movies that customer might enjoy. Because accurate recommendations increase Netflix’s appeal to its customers, the movie rental company started a contest in October 2006, offering $1 million to the first contestant that could improve the predictions by at least 10 percent.

On June 26th, 2009 the team “BellKor’s Pragmatic Chaos” were the first to submit a solution that improves more than 10% over the cinematch algorithm used by Netflix to match customers and movies. This triggered a final 30 days period in which other teams have the chance to beat that submission – the final winners will be the one with the best improvement. It looked very much like the BellKor’s Pragmatic Chaos would be the winners until yesterday… but then, suddenly a new team “The Ensemble” popped up out of nothingness (more or less – it is actually a collaborative efforts of several other teams) and made a submission on July 25th, 18:32:29 which outperforms the one of  BellKor’s Pragmatic Chaos by a tiny fraction. Snapshot of the leaderboard:

Given that the 30 days period was triggered on June 26th and depending on day counting convention this looks very much like a shootout on the finish line. Maybe, who knows, there is another team lurking in the dark making a last minute submission? Stay tuned!

Update 26.07.2009: We have new submissions and the match continues:

Update 26.07.2009: Game over

Contest Closed

Thank you for your interest in the Netflix Prize.

We are delighted to report that, after almost three years and more than 43,000 entries from over 5,100 teams in over 185 countries, the Netflix Prize Contest stopped accepting entries on 2009-07-26 18:42:37 UTC. The closing of the contest is in accordance with the Rules — thirty (30) days after a submitted prediction set achieved the Grand Prize qualifying RMSE on the quiz subset.

Team registration, team updates and the dataset download are also closed. The Contest Forum and Leaderboard remain open.

Qualified entries will be evaluated as described in the Rules. We look forward to awarding the Grand Prize, which we expect to announce in a few weeks. However if a Grand Prize cannot be awarded because no submission can be verified by the judges, the Contest will reopen. We will make an announcement on the Forum after the Contest judges reach a decision.

Once the Grand Prize is awarded, the ratings for the qualifying set will be released and the combined training data and qualifying sets will become available upon request at the Machine Learning Archive at UC Irvine.

Thank you again for your interest in the Netflix Prize. Keep checking this site for updates in the coming weeks.

Update 26.07.2009: There are several rumors spreading that the final winner is not yet determined as the score posted online is the one for a data set that is used for reporting the performance (only), whereas netflix uses a different one internally to do the real performance judgment. From the FAQ:

Why this whole quiz/test subset structure? Why not reveal a submission’s RMSE on the test subset?

We wanted a way of informing you and your competitive colleagues about your progress toward a prize while making it difficult for you to simply train and optimize against “the answer oracle”. We also wanted a way for the judges to determine how robust your algorithm is. So we have you supply nearly 3 million predictions, then tell you and the world how you did on one half (the “quiz” subset) while we judge you on how you did on the other half (the “test” subset), without telling you that score or which prediction you make applies to which subset.

So it is possible that the story looks different on the “test” subset, especially given that both teams were so close together.

If you are interested in the math behind it, then have a look here! At the end of that article you will find additional links.

Are twitter and friends increasing volatility in the market?

Recently browsing the internet I found google insights, somewhat the bigger brother of google trends. There you can compare not only the trends of certain words but you can also split the results into various time / location buckets and compare them. For example you can compare the searches run in the US for “carribean” to the ones for “recession” from Jan 2007 until today resulting in something like this (blue -> carribean / red -> recession):

One can see that queries for “carribean” already started to drop in Jan 2008 and dropped significantly further in Sep 2008 while the ones for recession started to significantly rise in Aug/Sep 2008. In hindsight it is easy to see patterns – just search long enough – and it is not clear if they constitute any correlation.

Further, while interesting for a lot of applications, historical information is not well suited for making predictions. But there are also other services such as twitter and facebook out there where users pour in tons of data in real time. Especially the latter can be easily searched in real time for trends and phrases as well. New information is quickly propagated through the network and made available to millions of people combing for specific phrases such as, e.g., “oil” or “oil price”. The following trend search is from Twist, a twitter trend service. For any point in time a click reveals the post written – everything updated in real time:

Now, having information on price changes and “market research” available even faster and more immediate than ever before (and not only for those with Bloomberg or Reuters access) one might suspect that the volatility in the market increases as people might act more impulsively and emotionally (as often claimed e.g., in behavorial finance) especially if prices go down. Having a delay in the information processing chains smoothens the trading behavior, effectively reducing volatility. If these delays are reduced to instanteneous information availability (short term) volatility increases.

Another, maybe even more critical problem could be that using twitter and other mass-publication-of-micro-information services the pump-and-dump strategies for microcaps, which are illegal (under most jurisdictions), can be performed even more effectively than ever before. As spam filters got more and more effective pump-and-dump via spamming got harder and harder. But with micro-blogging the whole story changes. By definition there are no spammers as you follow somebody and you do not get unsolicitated emails/spam. Due to this there might be some special legal issues here that deserve extra attention: When somebody is writing a tweet to spread wrong information concealed as “personal opinion” and millions eavesdrop, can the person be held responsibility if wrong information leads, e.g., to a fire sale? The story goes even a bit further: other people might re-tweet or copy the story multiplying the number of readers and adding credibility to it as more and more versions of it are out there (a turbo-charged version of the Matthew effect and its generalizations) – who is going to reconstruct the time line when money is at stake and a decision has to be taken now?

Probably soon hedge funds will pop up trading these noises by mining millions of tweets for signals trying to extract some cash from the market.

Mendeley revisited

As promised roughly half a year ago, here is my opinion on Mendeley. I was reminded of writing my review two days ago after I downloaded the latest version of Mendeley. After the update I fired up Mendeley and I was shocked (at first) – ALL MY DATA WAS GONE. Although I started out to test Mendeley only, I got so used to it that I couldn’t believe that my data was gone. The desktop client just asked me to log in but all the categories, groups etc were gone. More or less out of desperation I logged in (again) and the software immediately synchronized with the web service which has a backup of all the references (I chose to just save the references and not the pdfs). After less than a minute all the references were transferred back to my machine and even the links to the files on my hard drive worked. I was quite impressed with this backup/synchronization feature so that I decided that it was time to keep my promise and write a few lines about it.

So first of all, what is Mendeley? From the FAQ:

Mendeley is a combination of a desktop application and a website which helps you manage, share and discover both content and contacts in research.

Our software, Mendeley Desktop, offers you:

• Automatic extraction of document details (authors, title, journal etc.) from academic papers into a library database, which saves you a lot of manual typing! As more people use Mendeley, the quality of the data extraction improves.
• Super-efficient management of your papers: “Live” full-text search across all your papers – the results start to appear as you type! Mendeley Desktop also lets you filter your library by authors, journals or keywords. You can also use document groups, notes and tags to organize your knowledge, and export the document details in different citation styles.
• Sharing and synchronisation of your library (or parts of it) with selected colleagues. This is perfect for jointly managing all the papers in your lab!
• More great features: A plug-in for citing your articles in Microsoft Word, OCR (image-to-text conversion, so you can full-text search all your scanned PDFs) and lots more new features being worked upon.

Our website, Mendeley Web, complements Mendeley Desktop by offering you these features:

• An online back up of your library: Store your documents in your account and access them from anywhere through your browser.
• Detailed statistics of all things interesting: You can upload your own publications to your research profiles, then track the evolution of your readership. How often are your papers downloaded? How often are they read? From which academic disciplines and geographic regions do your readers come from? Additionally, there are detailed statistics for each academic discipline and research topic. Who are the up-and-coming authors in your discipline? Is the interest in a research topic growing or declining? What are the most widely read papers on a specific subject?
• A research network that allows you to keep track of your colleagues’ publications, conference participations, awards etc., and helps you discover people with research interests similar to yours.
• A recommendation engine for papers that might interest you, but are not yet in your library! Based on what you know already, what should you read next? Coming soon

To be honest, in the beginning when I first started to use Mendeley in Jan 2009 it did not quite convince me. The interface of the desktop software was slow, the meta-data extraction was quite crappy, etc. For short, I just didn’t like the user experience. I knew Papers from before but it didn’t convince me either and so I stuck with the one that was free – Mendeley. Then, over time more and more updates for Mendeley were released. The user interface got better and also the meta-data extraction improved significantly – nobody wants to enter all the meta-data by hand… The current version of Mendeley is 0.9.xx. It is still beta but the latest version is really a great piece of software. Some of the features:

1. Grouping and categorizing of your papers – you can slice and dice it in almost any way.
2. Shared groups – you are collaborating with somebody and you want to share your papers. Mendeley can do that for you. One guy adds a new paper, the other guy gets it automatically the next time (s)he uses Mendeley.
3. Full text search – search within all your papers. fast.
4. Integrated pdf viewer that allows annotation, notes, etc.
5. DOI / arxiv support – Add the corresponding references and Mendeley updates meta-data automatically. No need to enter anything.
6. Statistics functions (via the web service) – you can get detailed statistics about the papers in your library. Most-read journals, more-read authors, etc.
7. Every entry can have one or more pdfs (or other formats) attached to it – One click opens the paper in the built-in pdf viewer.
8. It is free – according to the faq, the current service level will remain free. There might be some premium features coming up at some point that will be charged for extra.
9. Backup/synchronization with the online service – all our references and notes are synchronized with the web service so that you can use your library on different machines and you always have a backup.
10. You can log-in to your Mendeley account from all over the world and access your references on the web (part of the synchronization feature).
11. Meta-data extraction – The extraction works quite well, conditional on the pdfs being somewhat reasonably tagged.

So if you are still looking for a great software for organizing your papers with a great overall concept then you should definitely give it a try!!