Subscribe via Email
Enter your Email Address:
Delivered by FeedBurner

Wednesday, May 2, 2012

Federated Search & Big Data gets bigger

The world of independent Federated Search is diminishing; last week IBM announced that they will be acquiring Vivisimo.[1]  There are a number of interesting aspects to this, and the analysts have covered some of them [2],[3], but some particular quotes from IBM itself and the analysts piqued my interest:

“The combination of IBM's big data analytics capabilities with Vivisimo software will further IBM's efforts to automate the flow of data into business analytics applications …” [IBM]
IBM also intends to use Vivisimo's technology to help fuel the learning process for their Watson
applications.” [IDC]
Overall, this is a very smart move for IBM, and it indicates that unstructured information is going to play an increasingly large role in the Big Data story…” [IDC]

All this shows the handling of structured and unstructured information growing in importance.

What does IBM want Vivisimo for? It seems to all stem round Big Data and the analytics that it can produce to enable better corporate decisions.  Of course, there’s also the lovely teaser of a better performing Watson! Both Watson and Analytics massage vast amounts of data and information to draw conclusions, assign values, and create relationships. But, like all such endeavors, the quality of the result depends critically on the quality of the incoming data. GIGO says it all!

Big Data analytics work very well with structured data, where the “meaning” of each number or term is exactly known and can be algorithmically combined with its peers, parents, siblings, and opposites to give a visualization of the state of play at the moment or over time. Gathering such data is a tedious process (hooray for computers!), but is not intrinsically difficult. All that needs to happen is to set up a mapping from each data Source to the master and let it run. The mappings are precise and the process effective, but the volumes are vast and the time-to-repeat rather slow for today’s fast paced world.

However, now add the fact that not everything you want to know is held in those nice regular relational database tables, and the picture looks far less rosy. Product reviews are unstructured, press releases are vague, social comments are fleeting, and technical and legal documents tend to be obtuse. But all these are vital if you want to make a really informed decision. So bring in Federated Search to the rescue.

Federated Search is a real time activity. It is focused on just what data or information is needed now. And it provides quality data. It is directed to just those Sources needed for “this report”, and it analyzes them in terms of known semantics so that the reviews, blogs, etc. mesh with the numerical analytics, and then provide the essential “external view” of the situation. And this is done right now, in real time. For the knowledge based systems (like Watson) the FS Sources provide in-depth data pertinent to the current problem. And if the Sources don’t have it, FS goes and finds it, thus allowing  Watson (as an example)  to add it to its knowledge base, and provide a more informed opinion.

So that is why IBM is adding Federated Search to its armory. What are the issues? In a word (or two): coverage and completeness.

All the Big Data systems use standardized access to the massive databases of the corporation’s transaction and repository systems. Most of these understand SQL or some other standard access language, and the customization is a matter of reading a schema mapping table. That mapping table is the same for every SharePoint or Exchange system (or similar), so once created, it is easily deployed. These types of standardized accesses are often referred to as “Indexing Connectors” because they extract enough data to enable the content to be indexed and searched. (For more on this see a future post on the deep differences between Connectors and Crawlers.)

Now, move to the world of web data and the complexity and difficulty escalates enormously.  The number of formats and access methods multiplies almost to the point of one-to-one for each Source. As an example look at the two press releases for this acquisition: IBM’s is a press release, with an initial dateline, and no tags, Vivisimo’s [4] is a blog post with tags and an author. The same Connector will not make sense of both at the level of detail needed for a decision making analysis.

Add in the velocity of the data in the social media (“velocity”, as you will recall, is one of the 3 “v”s that define Big Data – Volume, Variety, Velocity) and the relatively slow to aggregate times of conventional databases become a problem. Timing is an issue because of volume, but also because applications have to analyze input data from users and other sources, store it in their transactional database, and then the ETL function has to extract from that database and move the data to the analytics database or storage area. These are two stages, both relatively slow, that must be batched together.

So, once moving from structured data to unstructured data, and from the sheltered waters of the corporation to the rough seas of the Web, a very different set of techniques is needed. And that is where Federated Search (FS) comes in.   This is the truly hard, difficult part, and it’s where MuseGlobal shines.   But first, some more information on what FS is, and what it needs to do.

FS is immediate, which involves many synchronization and “freshness” issues, but essentially solves the “velocity” problem by obtaining data as it is needed. That is because FS is a “on demand” service. It is brought into play just-in-time to get the data when needed, not in batch mode to store it away just-in-case. Since it is used when needed it needs to be able to target the Sources of interest right now. That means it is flexible and dynamically configured, not painstakingly set up ahead of time and left alone.

Since it is a focused operation, targeting only the data needed, it must be able to get the maximum out of each Source. This requires two levels of complexity not common in other types of connectors or crawlers. These Sources have specific protocols and search languages and often security requirements. All these must be handled by the FS Connector so that the search is faithfully translated to the language of the Source, and the results are accurately retrieved. Second is getting the retrieved data into a useable form (and format). This involves a “deep extract” involving record formats, field/tag/schema semantics, content semantics, data normalization and cleansing, reference to ontologies, field splitting, field combination, entity extraction on rules and vocabularies, conversion to standard forms, enhancement with data from third Sources, and other manipulations. None of this is off-the-shelf processing where a single connector can be parameterized to work with all Sources. So FS has started at the “single, deep” end of the spectrum (crawlers are the epitome of the “broad, shallow” end) and builds Connectors to the characteristics of each Source.

These Connectors bring focused, quality data, but they come at a price. Vivisimo and MuseGlobal, and the other FS vendors build a very special type of software – something that we know will eventually fail, when the characteristics of the Source change. This needs a special dynamic architecture to accommodate it. It needs very powerful ways to build Connectors which can involve data analysts and programmers, as well as highly sophisticated tools, such as the Muse Connector Builder. It needs a robust and automated way to check for end-of-life situations, such as the Muse Source Checker, and a highly automated build and deploy process – the Muse Source Factory has been delivering automated software updates for 11 years now. Source Connectors *will* stop working, and a big part of a viable FS ecosystem is being able to get them back on line quickly and reliably.   MuseGlobal has put together a data virtualization platform with thousands of Connectors, because we know there’s a one-on-one relationship with each data source if you want to connect to the world out there.   Figuring out the unstructured data problem was one of our main goals at Muse from the very beginning, some 11 years ago.

Of course, building Connectors in the first place is an equal challenge, including the human element of dealing with a multitude of companies publishing information and data. This is something all FS vendors have to handle, and MuseGlobal chose to create a Content Partner Program about 10 years ago where we talk regularly to hundreds of major Sources and content vendors. Breadth of coverage of the Connector library is a major factor in “getting up and running” time, and a major investment for the FS vendors. We believe that Muse has one of the largest libraries with over 6,000 Source specific Connectors, as well as all the standard API and protocol and search languages ones for access where that is appropriate – but still with the “deep extraction” which is the hallmark of Federated Search.

It is not an easy task to get right at a quality and sustainable level, but a few vendors have produced the technology. MuseGlobal is one – and Vivisimo is another.

IBM Analytics and Watson are set for a real quality revolution!

Another analyst 's comments can be found on enterprise search blog at [6].

(*) You will need to be a subscriber to see the report


Bruno Araujo said...

I'm searching for new info about big data. I would post the results at my blog
Best regards,

Richard Majece said...

I want to find something to get inspiration from like cats) Cause I am a writer and I use this writing info to write my essays.

Michael Jones said...

Hello I am michael, SEO Expert in complete my assignment services. There is one information, it is very useful to you or biggeners.Are your looking for someone to write your assignment, here are expert assignment helpers of Programming Assignment Help are well efficient and capable of creating unique assignments.

Jonna Richard said...

This makes students like you aware of websites offering low quality All Assignment Help reviews In this way the precious amount you are about to waste on fraudulent
websites will be saved. You can go through all the websites reviewed at BEST WRITERS REVIEWS keeping in mind that it took us so long to check and analyze the services, advantages and drawbacks of particular sites. We are sure after reading the reviews on our site,
every student can make a wise decision regarding choosing a good academic writing website.

Students Assignment Help said...

Students Assignment Help provides the best statistics dissertation help with their best UK Assignment Writers. Our professional assignment writers deliver students full knowledge and understanding about the subject. Students can avail help from our online assignment help experts anytime.

james cook said...

Congrats on having such well managed site! It has good looks and contains informative content as well.We are an online platform where students check & write reviews for assignments related websites.Here you can check allassignmenthelp reviews

Helpful Sources said...

This is great information and provide me some other idea out of the box. I would recommend for clean master and if you want more about this you can download app cache cleaner for android. I think it was very useful because it helped me clean out the junk in my mobile.

Suhana Williams said...

Handling assignment works is not a very easy thing. It can be tiresome to maintain all the busy scheduled daily tasks and then again work on your paper. Therefore, it is always beneficial to choose assignment experts who are professionals and can deal with your work.

My Essay Help UK said...

I would like to thank you for the efforts you have made in writing this post. Thanks for posting Really Such Things. I should recommend your site to my friends.It is of a great advantage to take coursework help. It eases the burden of students which they have to carry. Taking law essay writing service also gives a lot of benefits to the students. Our essay help uk expert are highly committed to helping academic students to finish their essay.

Jijo said...

Best Assignment Help in UK at Lowest Price, Have you been stressed out due to your Assignments? No worry, We are here to help you with 100% plagiarism free Assignment Help in UK. We offers lowest price on all kind of dissertations, essay, assignments and more from 4000+ Ph.D. Experts with 24/7 support to make sure you will achieve A+ Grades in your college/universities.

Find examples of research paper topics at MyAssignmentHelp. Interesting high school & college research paper topics. Order in simple steps and get good, easy, best ideas and examples for research paper topics.

Different Essay writing service experts say that writing attractive hooks are indeed very important for your essay and research papers. A hook is the very first line of your introduction. The line confirms a reader’s attention on your academic paper.

divine vibe said...

Are you caught up in the celebrity world and fascinated with Hermès Birkin bags in basic black? There is the perfect place to find what you are looking for – for the best in Hermes Birkin bags in black and every other color.

Let’s be frank. There are too many of you that lust after the Hermès mini Birkin bag. The supple texture of the leather and the hand crafted design is to die for. Too bad Hermès prices are well out of your range. Go to instead. Shop for the perfect Hermès mini Birkin bag at a fraction of the cost.

MyAssignmentHelpAu said...

This post is not just informative but impressive also. The post is so convincing that it created an urge to choose Assignment Help services.

Unknown said...

Are you caught up in the celebrity world and fascinated with Hermès Birkin bags in basic black? There is the perfect place to find what you are looking for – for the best in Hermes Birkin bags in black and every other color.

Let’s be frank. There are too many of you that lust after the Hermès mini Birkin bag. The supple texture of the leather and the hand crafted design is to die for. Too bad Hermès prices are well out of your range. Go to instead. Shop for the perfect Hermès mini Birkin bag at a fraction of the cost.

alvin hopkins said...

In finance dissertation a wrong selection of topic, the writer may have thought about scrapping the content and starting all over again from the scratch. Thus, a lot of time will be wasted to find finance dissertation topics.To make it easier for you, we have listed 30 brilliant topics, which can make the whole process of writing finance dissertation much easier.

Raihan Mohammad said...

Best Homework Help is home to more than 3000 online live college homework tutors who are quite impressive in solving various primary homework related issues. In fact, some of these tutors hold a PhD degree, which clearly shows how competent they are at handling various school and college homework problems.

Dave Leena said...

If you are in need for online writing assistance for an intricate thesis topic, then avail our assignment help service in U.S. and save your time to relax and do your studies properly. Our assignment help online service in USA has earned huge popularity among both domestic and international students. There’s no better place in the USA than MyAssignmenthelp. Contact us now to buy assignments online in the USA Leave your tensions to us and enjoy your free time.

Students Assignment Help said...

Students Assignment Help provides the best dissertation writers UAE services for Students. Our expert team is available any time to help in all academic writings like Dissertation Writing, Essay assignment Help, course work, Assignment Help. You can email us at or WhatsApp +44-755-536-9184 to hire our experts.

Make My Assignments said...

Students could often find it tough to write their academic tasks. It could lead to they not getting desired grades in the academics. Opting for our Assignment Help Online could be ideal for such students and they can get a complete assignment solution from us.
Accounting Assignment Help New Zealand

Leadership Assignment Help New Zealand

HRM Assignment Help New Zealand

Harrison Aiden said...

Thanks for sharing such a nice blog of information to us. This is very knowledgeable for me. I am offering Assignment Help all over the world.

Saivy Hopkins said...

The dissertation tutor online service has qualified experts who have profound experience. They know that most of the students face significant trouble with dissertations. In these situations, what they actually need is a little push and assistance. With the diligent help from the dissertation tutors, the challenging hurdle like writing a dissertation can seem a cakewalk for them. They are also very helpful for the students with weak English language skills and poor knowledge about the subject. There are many students too who fear the extreme short deadlines in which they need to complete the dissertations. It can be useful and beneficial in all these scenarios with the dissertation writing tutorial service and dissertation proposal writing tutorial service.

Nikki Heysen said...

Don't think much for the best Do My Assignments Uk services then rush at and make sure all your work is finishing well. We have highly-qualified and certified experts in the team that is working continue 24x7 and even offer its unlimited free revision services.

Assignment Service said...

Looking for python assignment writing help
, seek help from online expertise and get your work on before the provided time.

FirstAssignmentHelp said...

The above post has given reliable and genuine information about Assignment Help Australia. Looking forward to avail their eminent services.

MyAssignmentHelpSG said...

An unmatched and nonpareil post i have ever seen. The content is so appealing that it has created an impulse to avail Assignment Help Singapore services.

Unknown said...

The first British owned real estate company in Kuwait.

Unknown said...

MyHome App
Book verified professionals for your home maintenance needs
MyHome makes it easy to find the right person for your home maintenance needs. Let us know the service you need and when you need it, and we'll match you with the right professional.

wajahat hussain khan said...

This article is indeed worth reading . I think this was just an amazing information regarding an important issue You should also read this out to check how to free fb hacks can be done

wajahat hussain khan said...

This article is indeed worth reading . I think this was just an amazing information regarding an important issue You should also read this out to check how to free fb hacks can be done

Students Assignment Help said...

Students Assignment Help provides the do my math assignment services outstandingly to the college or university students. Our assignment writers are well-qualified and have gained their degree from the top universities around the world. Hire our online assignment writers at:

Amy Willor said...

A high-standard post with all imperative information about Assignment Help UK services. Looking forward to avail the premium services.

RItuparna Das said...

Keep posting in same way really appreciate the efforts you put in writing waiting for next edition
Delhi Escorts Booking
Escorts Kolkata
Goa call girls
sexiest escorts in Delhi

Theresa Delcas said...

My Assignment Services provides a 24-hour online Australian assignment help or academic assistance and consultation to the students. Be it any subject such as Nursing, Economics, Law, Engineering, or Management, we provide the most reliable help with assignment online by our highly-proficient academic writers. This is because there are a multitude of online academic help services and picking the best is always going to be a trial and error method. However, My Assignment Services is a well-established and prominent name in the best myob assignment help provider & high-quality instant assignment help online to students since almost a decade. You can trust our academic ghostwriters completely to get best quality write-ups including case studies, research proposals, dissertations and theses, and more. Australian Assignment Help providing experts understand that price is one of the major factors that university students consider before paying someone to do it for them. This is because university students often have stringent budgets and are already burdened with student debts. This is why we offer regular and seasonal discounts on cdr writing service or other assignments so that you achieve high distinction without burning a hole in your pocket.

Online Assignment Help said...

Are you stuck in your assignment and need help for Assignment? Then do not worry out writing assignment service are here to help you. Myassignment help provides the best assignment help for all students in Australia. Our professional expert writers provide academic assistance services to all students. Students can get help from our online assignment writers 24*7. For more offers visit Myassignment Help now.

Dylan Eales said...

Those who are searching over internet assignment help can contact with us now. We are the best assignment writing service provider in melbourne, Australia. Our Academic assignment writers available 24*7 hours for the students, if you really want to need assignment help online at cheapest price meet assignment helper at sample assignment and get high distinction grades.

Holiganbet Giriş said...

2014 yılında kurulan Holiganbet bahis sitesi, spor bahisleri, canlı bahisler, casino, canlı casino ve tombala oyunları sunmaktadır. Holiganbet'e kayıt olarak 200TL  bonus kazanabilirsiniz. Holiganbet

Holiganbet Giriş said...

Asyabahis bahis sitesine üye olarak 20TL deneme bonusu alabilirsiniz. Betconstruct altyapılı bahis şirketi Curacao lisansına da sahip. Asyabahis

Betticket bahis sitesi %100 oranında 500TL bonus veriyor. Betticket ödemeleri ortalama 30 dakika içerisinde yapmaktadır. Betticket

Grandbetting bahis sitesi ilk üyelerine 400TL hoş geldin bonusu vermektedir. Grandbet giriş adresinden siteye erişin. Grandbetting

Holiganbet bahis şirketi güvenilir hizmetlerini 400TL hoş geldin bonusu ile süslüyor. %100 oranında bonusu alabilirsiniz. Holiganbet

Nathan William said...

The nurses cannot perform the task of serving in the community alone. They need support from their colleagues. Nursing often requires advice and referral from the other practitioners in the industry. It is just like what students do when they struggle with their work. Here, “do my assignment Australia” may not be the ideal call for help, but a combined effort is needed for better service in the community.
The tonality in assignments varies from the sub-categories you are choosing. For assignment writing service, you have to portray your idea in a much formal way than an essay paper. Thesis statements need more professionalism in order to depict the researches you have conducted.

Students Assignment Help said...

Contract specialists of Students Assignment Help to compose coursework assignments at a shabby cost. We have in excess of 3000 qualified and master journalists who are giving help through buy case studies online administrations. Our specialists are accessible online 24x7 for understudies help. To help email us at