MuseGlobal
Subscribe via Email
Enter your Email Address:
Delivered by FeedBurner

Wednesday, May 23, 2012

The Rise of the Connector - Part 2

Connectors are the heart and soul of Federated Search (FS) engines and with the rise in importance of FS in today’s fast paced, Big Data, analyze everything world, they are crucial to smooth and efficient data virtualization and flow. MuseGlobal has been building Connectors, and the architecture to use them (the Muse/ICE platform) and maintain and support them (the Muse Source Factory) for over 12 years. The people who design and build Connectors must be both computer savvy, and also have a deep understanding of data and information and its myriad formulations.

This second in the series of posts looks at the problems arising as data is needed from outside the enterprise, and the complexities of access and extraction that result. Not surprisingly, as a leading FS platform Muse and its ecosystem are in the forefront of providing solutions to data complexity problems in the modern world. (The first post considers the growing importance of being able to access data from inside an organization.)



Part 2         A broader perspective

All of this speaks to the volume and velocity (of change) of the data – two of the trio of defining “v”s of Big Data. The third v is variety and this is now encompassing much more than the internal data silos of the enterprise. Increasingly decisions need to take account of the outside world: competitors, news media, commentators and analysts, customer feedback, social postings and tweets.

Most of these sources are also fleeting. Customer records will last for years, a tweet is gone in 9 days. Even product reviews are only relevant until the next version of the product is released. And there are another couple of additional hurdles to jump to get this valuable “perspective” data.

This data lives outside the enterprise. Some other person or organization has control of it. And that means the old ETL trick of grabbing everything is likely to be severely frowned on – especially if it is tried every night. Commercial considerations mean that, if this data is valuable to you, then it is valuable to others, and the owners will not let you have it all for free. This means the strategy of asking for exactly what is needed is the way to go. It takes less time everywhere, will cost less in processing and transmission, will cost less in data license fees, and will not alienate valuable data sources. So “sipping gently” is the way to go.

Yes, in the paragraph above you saw “fees” mentioned. Once the commercial details have been sorted out, there is still the tricky technical matter of getting access through the paywall to the data you need, and are entitled to. Some services will provide some of the data you want for free, but most will require authenticated access even of there is no charge Those who are selling their data will certainly want to know that you are a legitimate user, and be sure you are getting what you have paid for – and no more.

For both of these considerations Federated Search engines, especially in their harvesting mode allow all the “virtual data” to become yours when you need it. Access control is one of the mainstays of the better FS systems to ensure just this fair use of data. And gentle sipping for just the required data is their whole purpose. Again a tool for the task arises. MuseGlobal runs a Content Partner Program to ensure we deal fairly and accurately with the data we retrieve from the thousands of sources we can connect to, both technically and as a matter of respecting the contractual relationship between the provider and consumer. We are the Switzerland of data access – totally neutral and scrupulously fair, and secure.


Complexity everywhere

So now you are accessing internal and external data for your BI reports. Unfortunately, while you might have a nice clean Master Data Managed situation in your company, it is not the one the external data sources are using (not unless you are Walmart or GM and can impose your will on your suppliers, that is). And this means the analysis will be pretty bad unless you can get internal product codes to match to popular names in posts and tweets. There is a world of semantic hurt lurking here.

You need tools. Fortunately the Federated Search engine you are now employing to gather your virtual data is able to help. Data re-formatting, field level semantics, content level semantics, controlled ontologies, normalized forms, content merging and de-merging, enumeration, duplicate control, all these are tools within the FS system. They are powerful tools and they are very precise, and they come with a health warning: “This Connector is for use with this source only”.

Connectors are built, and maintained, very specifically for a single Target. They know all about that target, from its communications protocol to the abbreviations it uses in the data. Thus they produce the deepest possible data extraction possible. And can deliver that data in a consistent format suited to the Data Model and systems which are going to use it. They are data transformers extraordinaire. This contrasts with crawlers at the other end of the scale where the aim is to get a simple sufficiency of data to handle keyword indexing.

This precision means that they are in need of “tuning” whenever their target changes in some way. Major changes like access protocols are rare, but a website changing the layout of its reviews is common and frequent. Complexity like this is handled by a “tools infrastructure” for the FS engine whereby testing, modification, testing again, and deployment are highly automated actions, reducing the human input to the problem solving, not the rote.

And now another wrinkle: some of the data needed for the analysis is not contained in the records you retrieve, and the only way to determine this is to examine those records and then go and get it. As a simple example think of a tweet which references a blog post. The tweet has the link, but not the content of the post. For a meaningful analysis, you need that original post. Fortunately the better FS systems have a feature called enhancement which allows for just this possibility. It allows the system to build completely virtual records from the content of others. Think more deeply of a hospital patient record. This will have administrative details, but no financial data, no medical history notes, not results of blood tests, no scans, no operation reports, no list of past and current drugs. And even if you gather all this, the list of drugs will not include their interactions, so there could be more digging to do. A properly configured and authenticated FS system will deliver this complete record.

Analysis these days is more than just a list of what people said about your product. It involves demographics and sentiment, and timeliness and location. All these can come from a good analysis engine – if it has the raw data to work from. Enhanced virtual records from a wide spectrum of sources will give a lot, but making the connections may not be that simple. We mentioned above “official” and popular product names and the need to reconcile them. Think for a moment of drug names. Fortunately a good FS system can do a lot of this thinking for you, and your analytics engine. Extraction of entities by mining the unstructured text of reviews and posts and news article and scientific literature allows them to be tagged so that the analysis recognizes the sameness of them. Good FS engines will allow this to a degree. Better ones will also allow that a specialist text miner can be incorporated in the workflow and give each record its special treatment – all invisibly to the BI system asking for the data.


Partnership at last

There is a lot of data out there, and a great deal of it is probably very useful to you and your company. Using the correct analysis engines and Federated Search “feeding” tools enables that data to be brought together in a flexible, efficient, and accurate manner to give the information needed for informed decisions.

Federated Search is still a very powerful and effective way to search for humans, but it has grown up to be one of the most effective tools for systems integration, the breaking down of corporate silos of data, and the incorporation of data from the whole Internet into a unified, useable data set to create real knowledge.

Muse is one of those tools which can supply the complete range from end user fed search portals, to embedded data virtualization, and we intend to keep up with the next turn of data events.

32 comments:

Rick Campbell said...

Nice! And I would also like to draw your attention to data room due diligence and to cloud technology in general.

reginald surict said...

After reading your post I thought you would also enjoy reading this article.

Barbara Lineberger said...

real racing 3 mod apk ppsspp gold apk hungry shark world mod apk

Dylan Eales said...

Those who are searching over internet IT assignment help can contact with us now. We are the best assignment writing service provider in melbourne, Australia. Our Academic assignment writers available 24*7 hours for the students, if you really want to need IT assignment help, Computer science assignment help online at cheapest price and get high distinction grades.

Unknown said...

As an established Australia assignment help organization, we always aim to increase the number of in-house experts we have so that we can offer you the perfect assignment writing help in conventional courses like Management, Law, Engineering, etc. Biostatistics assignment help service can be taken by a student so that the understanding of this field becomes easy. The importance of biostatistics is largely felt in need of estimating the risk factors or the subject at risk from a particular disease or ailment or under circumstances of measuring the response rate to a particular drug. TAFE assignment help is one of the many premium services provided by My Assignment Services. We have a dedicated team of expert tutors who have vast experience in providing TAFE assignment help to students all over the world.

Maria Carter said...

Really mean a lot. Thanks for sharing your valuable post. I like the way of the elaboration of your topic so please keep informative posting. I am waiting for your next post, so please give us a link to your next post here and visit HP Printer Support for HP help. You may also get help for the printer through the HP Printer Offline Fix !! HP Printer Offline Windows 10

Seema Singh said...

canon printer support online
hp printer support toll free number
epson printer support toll free number
canon customer service phone number for printer
epson printer customer care phone number
hp printer customer service number usa

haryjohn8feb said...

Excellent and nice post. It will beneficial for everyone. Thanks for sharing such a wonderful post. Avail No 1 custom essay writing help UK from certified PhD writers. It is extremely helpful for me. You can email us at info@ukdissertationhelp.co.uk or Phone Number - 020 8144 9988

Aster said...

I can only express a word of thanks. Because with the content on this blog I can add knowledge I, thank has been sharing this information. Do not forget to visit our website to share information hotele

edubirdie said...

This is a great article. It gave me a lot of useful information. thank you very much. We are also a service provider that deal in the assignment writing help sector. So, the students struggling to write their college assignments can opt for our online dissertation formatting service and can get a quality assignment written from us.

monica said...

big data is the field that treats ways to analyze, systematically to provide exactly the same information. as the name shows the big data in which the data sets are too large or complex to be dealt with. if we talk about the current usage the team big data tends to refer to the use of predictive analytics, user behavior analytics.
i have received an order to do my homework for me cheap that's a reason i have a knowledge about this topic.

HP Printer Troubleshooting said...

There are many companies in the market which is manufacturing printers but HP is one of the best of them. Millions of people around the world using HP printer to complete their printing task. You should contact HP Printer Tech Support Phone Number to overcome printer issues like paper jam, printer offline, print head and other issues.

charlie Joy said...

When you try to start up HP laptop, you fail to start. It shows that your device has some kinds of technical errors. To repair your HP laptop technically, you need to follow some technical steps suggested by technical experts. We are an independent third party tech support service provider, providing unlimited technical support or help for HP laptop users in very nominal charges. Our technical support professionals are highly trained and experienced to resolve startups issue fully from the origin. Our HP laptop support team is available 24/7 to help you for any type of technical issues.

Email Tech Support said...

There is no doubt that AOL Desktop Gold is world-wide popular software that is designed to lessen the workload of employees. But, due to some technical glitches, AOL Gold may start to behave in an improper way. It may show sudden freezing, start-up issues, unexpected crashing and many more issues which you haven’t even heard of in your entire life. However, you can easily eliminate these nasty errors by taking aid from highly experienced experts. In order to take their guidance, you need to put a call at AOL Desktop Gold support number which is accessible throughout the day and night.

Rayan Warner said...

I always used to be worried about several issue related to office 365, but all thanks to the amazing assistance from the professionals of Microsoft 365 Support . Now, I don’t have to face any kind of issues and if any crops up, then they easily help me out. Get support click here
Office 365 Help
Office 365 Support
Microsoft Office 365 Support
Outlook Support
Microsoft Outlook Support
Outlook Support Number
Outlook Support Phone Number
Outlook Tech Support Phone Number
Outlook Technical Support Phone Number
Outlook Customer Service Phone Number
Outlook Customer Support Phone Number
Outlook Support Number

meldaresearch said...

You need not to worry any more, we are here to offer you the best academic writing
Solutions; order quality best writing service from us and learn how to purchase term paper today!

computersupport said...

Just dial 1-877-916-7666, the tech support number of Assistance for All. We are the most popular tech support agency operating around the country.

jacklinemelda said...

Our company is the best online solution in offering best research paper writing service since we hire professional writers who have years of experience in online essay writer services.

Appslure said...

Really very informative and creative. This sharing concept is a good way to enhance knowledge. We are top Mobile App Development.

Unknown said...

Popular Fashion Blogs in Surat
Fashion Blogger in Surat
Surat Blogger
Indian Fashion Blogger
Fashion Blogger in India

office-comofficeoffice said...

Norton.com/setup download through which different bits of programming can be downloaded, for example, Norton setup Security Standard, Norton Small Business and essentially more that will encourage their specific needs.
http://nor-norton.com/setup/

office-comofficeoffice said...

Log in to your Microsoft account using the login accreditations and after that type a veritable Office.com/setup key.
http://office-comofficeoffice.com/

office-comofficeoffice said...

I'm looking for some good blog sites for Office.com/setup. I was searching for search engines and found your blog site. Keep doing it .To install MS Office, make certain that you have a fine internet connection.
http://officecom-officeoffice.com

John Luther said...

Your opinion are very interesting, attractive article, so i shared it on my facebook. Promise that a lot of people will enjoy it. QuickBooks Error tech Support

Canon Support said...

how to connect canon printer to wifi
Connecting your Canon printer to your Wi-Fi is very easy once you follow these steps:
• With the power button turn on your printer.
• Click setting button
• After that click arrow button>
• Go to device setting
• Click OK
• Click arrow button > you see LAN Settings and then click OK.
• Click arrow button > you go to wireless LAN setup and then click OK.
• The printer will start searching for a Wi-Fi network; for now, the light will be blinking.
• If the searching process takes too long, you can press Stop, and it’ll go to wireless LAN setup, standard setup. Press OK.
• Click arrow button >until you find your Wi-Fi network and then click OK.
• Enter your password for the Wi-Fi and click OK.
• Press OK again once the screen says Connected.
This is the method of connecting your printer to Wi-Fi.

creative web solution said...

Get The Best Repairing Service of office chairs shockers of your sofa or billiards and more other furniturea at best price.For more info visit our site given below :-

Chair Repairing in nawada
Shocker Repairing in dwarka
Sofa Repairing in delhi ncr
Furniture Repairing in janakpuri
Billiards Repairing in nehru place
chair repair in dwarka
sofa repair in south delhi
shocker repair in nawada
furniture repair in nawada

TurboTax Customer Service said...

TurboTax is very user-friendly and people find it easy to navigate and search for the option they want. Furthermore, even if you are experiencing errors while using the software, you can contact the TurboTax Customer Service phone number +844-616-4459 (toll-free) for resolution. All you need to do is, provide the TurboTax Customer Service team with your tax-related information.

creative web solution said...

Get the best furniture manufacturers in delhi ncr for more info visit our site
top Sofa Manufacturers in Delhi
Sofa set manufacturers in Delhi
Best Furniture dealers in Delhi
Furniture repairing in Delhi
Sofa repairing in Delhi
designed furniture manufacturers in delhi
Best bed repairing in Delhi
Best furniture manufacturing in Delhi NCR
headboard manufacturing services
Best headboard manufacturing  manufacturers in Delhi
 Home Furniture repairing in Delhi

Jack Smith said...

Quickbooks Payroll Support Phone Number +1-888-412-7852 is available 24*7 at QuickBooks Payroll Customer Service Number to guide the users.

Smith watson said...

QuickBooks Customer Care - you can call our dedicated QuickBooks experts who are in this market for more than a decade. Our QuickBooks Customer Care Number team is always there to assist you.

Smith watson said...

Our QuickBooks Support Team is always with you. We provide support to errors generating in payrolls, pos, pro, enterprise sales calculations, the company file, single and multiple modes, backup data, etc. Call Quickbooks support Number +1-888-412-7852.

creative web solution said...

We are the best service provider for the ads and posters on auto or wall paints for all informations visit our site

Rural wall painting advertising
Auto rickshaw ad in delhi
Advertising wall painting
Auto Rickshaw advertising
Digital wall painting
Hoardings in Delhi
Wall painting advertising
Outdoor advertising
Outdoor Advertising Companies in Delhi
Hoarding advertising