Edit filter log

'{{Lowercase|title=tf–idf}} The '''tf–idf''' weight (term frequency–inverse document frequency) is a weight often used in [[information retrieval]] and [[text mining]]. This weight is a statistical measure used to evaluate how important a word is to a [[document]] in a collection or [[Text corpus|corpus]]. The importance increases [[Proportionality (mathematics)|proportionally]] to the number of times a word appears in the document but is offset by the frequency of the word in the corpus. Variations of the tf–idf weighting scheme are often used by [[search engine]]s as a central tool in scoring and ranking a document's [[Relevance (information retrieval)|relevance]] given a user [[query]]. One of the simplest [[ranking function]]s is computed by summing the tf-idf for each query term; many more sophisticated ranking functions are variants of this simple model. ==Motivation== Suppose we have a set of English text documents and wish to determine which document is most relevant to the query "the brown cow." A simple way to start out is by eliminating documents that do not contain all three words "the," "brown," and "cow," but this still leaves many documents. To further distinguish them, we might count the number of times each term occurs in each document and sum them all together; the number of times a term occurs in a document is called its ''term frequency''. However, because the term "the" is so common, this will tend to incorrectly emphasize documents which happen to use the word "the" more, without giving enough weight to the more meaningful terms "brown" and "cow". Also the term "the" is not a good keyword to distinguish relevant and non-relevant documents and terms like "brown" and "cow" that occur rarely are good keywords to distinguish relevant documents from the non-relevant documents. Hence an ''inverse document frequency'' factor is incorporated which diminishes the weight of terms that occur very frequently in the collection and increases the weight of terms that occur rarely. ==Mathematical details== The ''term count'' in the given document is simply the number of times a given [[term (language)|term]] appears in that document. This count is usually normalized to prevent a bias towards longer documents (which may have a higher term count regardless of the actual importance of that term in the document) to give a measure of the importance of the term <math> t_{i} </math> within the particular document <math>d_{j}</math>. Thus we have the ''term frequency'', defined as follows. :<math> \mathrm{tf_{i,j}} = \frac{n_{i,j}}{\sum_k n_{k,j}}</math> where <math>n_{i,j}</math> is the number of occurrences of the considered term (<math> t_{i} </math>) in document <math>d_{j}</math>, and the denominator is the sum of number of occurrences of all terms in document <math>d_{j}</math>. The ''inverse document frequency'' is a measure of the general importance of the term (obtained by dividing the total number of [[documents]] by the number of documents containing the term, and then taking the [[logarithm]] of that [[quotient]]). :<math> \mathrm{idf_{i}} = \log \frac{|D|}{|\{d: t_{i} \in d\}|}</math> with * <math> |D| </math>: total number of documents in the corpus * <math> |\{d : t_{i} \in d\}| </math> : number of documents where the term <math> t_{i} </math> appears (that is <math> n_{i,j} \neq 0</math>). If the term is not in the corpus, this will lead to a division-by-zero. It is therefore common to use <math>1 + |\{d : t_{i} \in d\}|</math> Then :<math> \mathrm{(tf\mbox{-}idf)_{i,j}} = \mathrm{tf_{i,j}} \times \mathrm{idf_{i}} </math> A high weight in tf–idf is reached by a high term [[frequency (statistics)|frequency]] (in the given document) and a low document frequency of the term in the whole collection of documents; the weights hence tend to filter out common terms. The tf-idf value for a term will always be greater than or equal to zero. Various (mathematical) forms of the tf-idf term weight can be derived from a probabilistic retrieval model that mimicks human relevance decision making. ==Example== Consider a document containing 100 words wherein the word ''cow'' appears 3 times. Following the previously defined formulas, the term frequency (TF) for ''cow'' is then (3 / 100) = 0.03. Now, assume we have 10 million documents and ''cow'' appears in one thousand of these. Then, the inverse document frequency is calculated as log(10 000 000 / 1 000) = 4. The TF-IDF score is the product of these quantities: 0.03 × 4 = 0.12. ==Applications in vector space model== The tf-idf weighting scheme is often used in the [[vector space model]] together with [[cosine similarity]] to determine the [[similarity]] between two documents. ==See also== * [[Okapi BM25]] * [[Noun phrase]] * [[Word count]] * [[Kullback-Leibler divergence]] * [[Mutual Information]] * [[Latent semantic analysis]] * [[Latent semantic indexing]] * [[Latent Dirichlet allocation]] ==References== * {{Cite journal | author = [[Karen Spärck Jones|Spärck Jones, Karen]] | year = 1972 | title = A statistical interpretation of term specificity and its application in retrieval | journal = [[Journal of Documentation]] | volume = 28 | issue = 1 | pages = 11–21 | url = http://www.soi.city.ac.uk/~ser/idfpapers/ksj_orig.pdf | doi = 10.1108/eb026526 }} * {{Cite book | author = [[Gerard Salton|Salton, G.]] and M. J. McGill | year = 1983 | title = Introduction to modern information retrieval | publisher = [[McGraw-Hill]] | isbn = 0070544840 }} * {{Cite journal | author = Salton, Gerard, Edward A. Fox & Harry Wu | year = 1983 | month = November | title = Extended Boolean information retrieval | journal = [[Communications of the ACM]] | volume = 26 | issue = 11 | pages = 1022–1036 | url = http://portal.acm.org/citation.cfm?id=358466 | doi = 10.1145/182.358466 }} * {{Cite journal | author = Salton, Gerard and Buckley, C. | year = 1988 | title = Term-weighting approaches in automatic text retrieval | journal = [[Information Processing & Management]] | volume = 24 | issue = 5 | pages = 513–523 | doi = 10.1016/0306-4573(88)90021-0 }} * {{Cite journal | author = H.C. Wu, R.W.P. Luk, K.F. Wong, K.L. Kwok | year = 2008 | title = Interpreting TF-IDF term weights as making relevance decisions | journal = [[ACM Transactions on Information Systems]] | volume = 26 | issue = 3 | doi = 10.1145/1361684.1361686 }} ==External links== * [http://nlp.fi.muni.cz/projekty/gensim Gensim] is a Python+[[NumPy]] framework for Vector Space modelling. It contains incremental (memory-efficient) algorithms for Tf–idf, [[Latent Semantic Indexing]] and [[Latent Dirichlet Allocation]]. *[http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.101.9086 Term Weighting Approaches in Automatic Text Retrieval] *[http://bscit.berkeley.edu/cgi-bin/pl_dochome?query_src=&format=html&collection=Wilensky_papers&id=3&show_doc=yes Robust Hyperlinking]: An application of tf–idf for stable document addressability. *[http://code.google.com/p/tfidf/ A library implementing Tf-idf] *[http://infinova.co.uk/2010/01/26/tfidf-weighting/ A demo of using TF-IDF with PHP and Euclidean distance for Classification] *[http://github.com/timtrueman/tf-idf/ A Simple TF-IDF implementation in Python] *[http://gist.github.com/464760 Fast TF-IDF implementation using Python and Redis] *[http://www.codeproject.com/KB/IP/AnatomyOfASearchEngine1.aspx Anatomy of a search engine] {{DEFAULTSORT:Tf–Idf}} [[Category:Information retrieval]] [[Category:Artificial intelligence applications]] [[Category:Statistical natural language processing]] [[Category:Ranking functions]] [[ar:تي اف-اي دي دف]] [[de:Tf-idf-Formel]] [[fa:فراوانی وزنی تی‌اف‌ای‌دی‌اف]] [[fr:TF-IDF]] [[ko:TF-IDF]] [[ja:Tf-idf]] [[pl:TFIDF]] [[ru:TF-IDF]] [[zh:TF-IDF]]'