Text mining

Text mining, text data mining (TDM) or text analytics is the process of deriving high-quality

Text analysis involves

Text analytics describes a set of

The term text analytics also describes that application of text analytics to respond to business problems, whether independently or in conjunction with query and analysis of fielded, numerical data. It is a truism that 80 percent of business-relevant information originates in unstructured form, primarily text.^[8] These techniques and processes discover and present knowledge – facts, business rules, and relationships – that is otherwise locked in textual form, impenetrable to automated processing.

Text analysis processes

Subtasks—components of a larger text-analytics effort—typically include:

• Dimensionality reduction is important technique for pre-processing data. Technique is used to identify the root word for actual words and reduce the size of the text data.^{[citation needed]}
• Information retrieval or identification of a corpus is a preparatory step: collecting or identifying a set of textual materials, on the Web or held in a file system, database, or content corpus manager, for analysis.
• Although some text analytics systems apply exclusively advanced statistical methods, many others apply more extensive
  part of speech tagging, syntactic parsing, and other types of linguistic analysis.^[9]
• Named entity recognition
  is the use of gazetteers or statistical techniques to identify named text features: people, organizations, place names, stock ticker symbols, certain abbreviations, and so on.
• Disambiguation—the use of
  contextual clues—may be required to decide where, for instance, "Ford" can refer to a former U.S. president, a vehicle manufacturer, a movie star, a river crossing, or some other entity.^[10]
• Recognition of Pattern Identified Entities: Features such as telephone numbers, e-mail addresses, quantities (with units) can be discerned via regular expression or other pattern matches.
• Document clustering: identification of sets of similar text documents.^[11]
• Coreference: identification of noun phrases and other terms that refer to the same object.
• Relationship, fact, and event Extraction: identification of associations among entities and other information in texts.
• Sentiment analysis involves discerning subjective (as opposed to factual) material and extracting various forms of attitudinal information: sentiment, opinion, mood, and emotion. Text analytics techniques help analyze sentiment at the entity, concept, or topic level and distinguish opinion holders and objects.^[12]
• Quantitative text analysis is a set of techniques stemming from the social sciences where either a human judge or a computer extracts semantic or grammatical relationships between words in order to find out the meaning or stylistic patterns of, usually, a casual personal text for the purpose of
  psychological profiling etc.^[13]
• Pre-processing usually involves tasks such as tokenization, filtering and stemming.

Text analytics is being used in business, particularly, in marketing, such as in customer relationship management.^[29] Coussement and Van den Poel (2008)^[30][31] apply it to improve predictive analytics models for customer churn (customer attrition).^[30] Text mining is also being applied in stock returns prediction.^[32]

Sentiment analysis

Sentiment analysis may involve analysis of products such as movies, books, or hotel reviews for estimating how favorable a review is for the product.^[33] Such an analysis may need a labeled data set or labeling of the affectivity of words. Resources for affectivity of words and concepts have been made for

Text has been used to detect emotions in the related area of affective computing.[36] Text based approaches to affective computing have been used on multiple corpora such as students evaluations, children stories and news stories.

Scientific literature mining and academic applications

The issue of text mining is of importance to publishers who hold large

Academic institutions have also become involved in the text mining initiative:

• The
  social sciences
  .
• In the United States, the School of Information at University of California, Berkeley is developing a program called BioText to assist biology researchers in text mining and analysis.
• The Text Analysis Portal for Research (TAPoR), currently housed at the University of Alberta, is a scholarly project to catalogue text analysis applications and create a gateway for researchers new to the practice.

Methods for scientific literature mining

Computational methods have been developed to assist with information retrieval from scientific literature. Published approaches include methods for searching,^[40] determining novelty,^[41] and clarifying homonyms^[42] among technical reports.

Digital humanities and computational sociology

The automatic analysis of vast textual corpora has created the possibility for scholars to analyze millions of documents in multiple languages with very limited manual intervention. Key enabling technologies have been parsing, machine translation, topic categorization, and machine learning.

The automatic parsing of textual corpora has enabled the extraction of actors and their relational networks on a vast scale, turning textual data into network data. The resulting networks, which can contain thousands of nodes, are then analyzed by using tools from network theory to identify the key actors, the key communities or parties, and general properties such as robustness or structural stability of the overall network, or centrality of certain nodes.

Text mining computer programs are available from many commercial and open source companies and sources.

Intellectual property law

Situation in Europe

Under

The European Commission facilitated stakeholder discussion on text and data mining in 2013, under the title of Licenses for Europe.^[55] The fact that the focus on the solution to this legal issue was licenses, and not limitations and exceptions to copyright law, led representatives of universities, researchers, libraries, civil society groups and open access publishers to leave the stakeholder dialogue in May 2013.^[56]

Situation in the United States

References

Citations