The semantic pipeline – understanding and connecting data

Never before has there been as much information as there is today. Just ten years ago, the amount of data generated globally was roughly 30 exabytes. Only four years later, that volume had already multiplied by a factor of 10, and by now the amount of data generated globally has hit a new all-time high. Today, experts estimate that the combined volume of data generated, captured, copied, and consumed worldwide will continue to climb.

It only follows that handling this steadily mounting volume of data effectively is becoming increasingly difficult. As such, companies face the very real risk of not being equipped to fully leverage their own treasure trove of data.

Which is where intelligent insight engines like Mindbreeze InSpire enter the picture. They pave the way for companies to analyze their data (both structured and unstructured) efficiently and to capitalize fully on the information buried within it – to the benefit of every single area of the business.

Conventional enterprise search gets an upgrade

Insight engines already combine familiar technologies from the field of enterprise search with methods of artificial intelligence like machine learning, neural networks, and speech recognition.

Using what is known as a semantic pipeline, they are capable of extracting all of the structured and, more importantly, unstructured data from documents and queries. This multi-level semantic processing is used to process documents, metadata, and content, but it also processes incoming queries.

And that makes insight engines more powerful than conventional enterprise search solutions. Instead of simply locating and displaying the results of a search, they proactively deliver users the right answers in the right context.

The various levels of this pipeline can be defined as:

• Natural language processing
  The complexity of human language calls for a high level of intelligent technology. Since it is very difficult to capture dialects, irony, or ambiguity, insight engines utilize innovative speech recognition approaches like natural language processing (NLP), natural language understanding (NLU), and natural language question answering (NLQA).
  
  Thanks to these technologies, users can enter search queries in natural language and the systems can process them immediately. In this process, the system can analyze and understand both the structured metadata as well as its textual content, enabling it to correctly identify the needs of the user. NLP deals with processing human language by machine, while NLU is primarily concerned with determining user intent, and NLQA works to create a natural dialogue.
   
• The AI pipeline
  In this part of the pipeline, artificially intelligent technologies such as machine and deep learning come into play. They allow the insight engine’s knowledge to expand steadily. As a result, its search performance becomes progressively optimized the longer it’s in use.
  
  And this optimization is driven by user behavior. To this end, the insight engine analyzes the way people work, their search and click behavior, and how often or in what context they retrieve certain information. That data can be used as the basis for calculating specific relevance models. As a result, the system can easily derive, classify, and proactively supply the relevance for each search query and each piece of content.
   
• Taxonomies/ontologies/catalogs
  Companies usually have a highly specific language – their corporate language. It includes all the internal technical terms, special abbreviations, acronyms, and so on. It follows that these phrases are included in the company documents, e-mails, notes, calendar entries, and so forth. By integrating in-house catalogs and glossaries into the insight engine, locating specific abbreviations or terms is no problem at all. 
  
  In the same vein, taxonomies and ontologies can be incorporated just as easily at this stage of the semantic pipeline. This way, the insight engine is capable of recognizing and storing concepts and hierarchies in order to then be able to extract them.
   
• Entity recognition
  Rule-based extraction lets users create highly customizable metadata definitions. In turn, this allows the insight engine to identify and extract specific entities, such as project IDs, product codes, and registration numbers.