MuseGlobal
Subscribe via Email
Enter your Email Address:
Delivered by FeedBurner

Monday, May 14, 2012

The Rise of the Connector - Part 1



Connectors are the heart and soul of Federated Search (FS) engines and with the rise in importance of FS in today’s fast paced, Big Data, analyze everything world, they are crucial to smooth and efficient data virtualization and flow. MuseGlobal has been building Connectors, and the architecture to use them (the Muse/ICE platform) and maintain and support them (the Muse Source Factory) for over 12 years. The people who design and build Connectors must have rich technical expertise, and also have a deep understanding of data and information and its myriad formulations.

This series of posts will look at the problems arising as data grew in volume, spread across systems, moved outside the enterprise, and became all important for the business intelligence which informs current corporate decisions. Not surprisingly, as a leading FS platform Muse and its ecosystem are in the forefront of providing solutions to data problems in the modern world.

This first post considers the growing importance of being able to access data from inside an organization. (The second post looks at the problems arising as data is needed from outside the enterprise, and the complexities of access and extraction that result.)

Part 1         Wanted: data from over there, over here


As the world of Big Data grows daily and the importance of unstructured data becomes more evident to information workers and managers everywhere, methods of accessing that data become critical to success.

Typically in an enterprise the majority of their data is held in relational DBMS’s which are attached to the transaction systems that generate and use it. These include HR, Bill of Materials, Asset Management systems and the like.  However for managers to make strategic decisions on even this data is difficult, they need to see it all at once. The analysis managers need is performed by a Business Intelligence (BI) system, and it works on data held in its own (OLAP) database, which is specially structured to give quick answers to pre-formulated questions.

And here is the first problem: transaction systems with lots of data, and an analysis system with an empty database.  The solution: set up and run a batch process for each working database that takes a snapshot of its data and transforms and loads it into the OLAP database. This is ETL (Extract, Transform, Load) and is where most big company systems are at the moment. The transaction systems have no method of exporting the data, and the analysis engine just works from what it has. This three part solution works and it works well, but it has some problems.

Running a snapshot ETL on each working system at “midnight” obviously takes time, and can be nearly a day old before the process starts. This lack of “freshness” of the data didn’t matter too much 5 or even 2 years ago. It took so long to change systems as a result of the analysis that data a day or so old was not on the critical path. But today’s systems can adapt much more rapidly, and business decisions need to be based on hourly or even by-the-minute data. (Of course, if you are in the stock and financial markets then your timescale is down to micro-seconds, and you have specialist systems tailored for that level of response.)  So first we need to improve on our timing.

In order to do that we need to move from a just-in-case operation to a just-in-time one. Rather than collect all the data once a day, we need to be able to gather it exactly when we need it. Of course gathering it overnight as historic data is still important and makes the whole process work more smoothly and quickly as the just-in-time data is now only a few hour’s worth and so can be processed that much quicker to get it into the BI system. Now we have a two-legged approach: batch bulk and focused immediate updates. Sounds good, but the ETL software for the batch work will not handle the real time nature of the j-i-t data requests.

For a start the ETL process grabs everything in the transaction system database – all customers, all products, all markets. But a manager is generally going to ask for a report on a specific customer or product. It would be endlessly wasteful to grab all that “fresh” data for all customers, when only data for one is needed. So the j-i-t process has to be able to query the transaction system, rather than sweep up everything. It is also almost certain that the required report will need data from more than one transaction system, but probably not all of them. ETL is not set up to do this; it needs a system capable of directing queries at designated systems and transforming those results. And, finally, the extracted data may well need to be in a different format. After all now we are loading the data directly into the Business Intelligence analysis engine for this report (for speed), and not importing it to the OLAP database.  This means that the structure and semantics are all different.

Increasingly the tools of choice for these j-i-t operations are Federated Search (FS) systems such as MuseGlobal’s Muse platform. They can search a designated set of sources (transaction systems), run a specific query against them, and then re-format the results and send them directly to the analysis engine. Initial examples of FS systems are user driven, but for this data integration purpose, the more sophisticated FS systems are able to accept command strings and messages in a wide variety of protocols, formats and languages and act on them, thus allowing the FS system to act a s a middleman getting the data the BI engine needs exactly when it needs it. Muse, for example, through its use of “Bridges” can accept command inputs in over a dozen distinctly different protocols, and can query all the major enterprise management suites in a native or standards-based protocol.


Should we move?


The need for speed of analysis and the volume of data involved grows every day it seems. It takes time to extract all that data and to build a big OLAP database just in case we want it.  What’s more, building, and changing the structure to adapt to changing analysis needs takes time – a lot of it.

So modern BI systems have moved to holding their database in memory, rather than on disk, just so everything is that much faster. Modern analysis engines, many based on the Apache project’s Hadoop engine, can handle a lot of data in a big computer, and do it rapidly. Both Oracle (Exalytics) and SAP (Hana) have introduced these combined in-memory database plus analytics engine, and others are coming. (See here for an InformationWeek take on the war of words surrounding them.) These engines can be rapidly configured (often in real time, through a dashboard) to give a new analysis report – as long as they have the data!

Moving all that data from the transaction system takes time, so the current mode is to leave it there and rely on real-time acquisition of what is needed. This is of course much less disruptive, fresher, and much more focused on the analysis at hand. This is not to say that historical data is not important; it is, and it is used by these engines, but the emphasis is more and more on that last bar on the graph.

So, again we need a delivery engine to get our data for us from all the corporate data silos, get it when it is needed, and then deliver it to the maw of the BI analytics engine. Once again the systems integration, dynamic configuration and deep extraction technologies of a Federated Search engine come to the rescue. Muse supports the real time capabilities, parallel processing architecture, session management, and protocol flexibility to deliver large quantities of data when asked for, or on a continuing “feed” basis.

49 comments:

Richard Majece said...

To have connection between my all devices is one of the main rules for me. Cause I lost my phone few weeks ago and it was possible for me to track my phone number using https://cellspyapps.org/track-phone-number/ because of it

Michael Jones said...

Buying an assignment help services are the intelligent decision by the student. As we all know in academics submission of the assignment before the deadline is essential for obtaining good grades and the student who failed to do this they end up with a low score.

Law assignment help
Programming Assignment help

MyAssignmentHelpAu said...

This post is not just informative but impressive also. The post is so convincing that it created an urge to choose Assignment Help services. You can email us at Info@Myassignmenthelpau.Com or Phone Number: +61-2-8005-8227

Students Assignment Help said...

Students Assignment Help offers the best science assignment help services to the students. We have 3000+ expert writers in the industry and they provide the excellent quality assignment writing services to the students from.

Dave Leena said...

If you are in need for online writing assistance for an intricate thesis topic, then avail our assignment writing service in U.S. and save your time to relax and do your studies properly. Our assignment help online service in USA has earned huge popularity among both domestic and international students. There’s no better place in the USA than MyAssignmenthelp. Contact us now to buy assignments online in the USA Leave your tensions to us and enjoy your free time.

Julie T. Kirwan said...

green farm 3 mod apk bully anniversary edition mod apk mx player pro apk

Pankaj Singh said...

I think this an amazing blog I have seen, thank you so much for sharing this. Visit for
Hong Kong Honeymoon Packages

Dylan Eales said...

Sample Assignment is the leading brand in Australia for Australian assignment help that offers complete University assignment help services like contract law assignment, it assignment help, financial assignment help, project management assignment, electrical engineering help and more. All students who looking for assignment help  Melbourne, Adelaide, Perth etc

https://amantoothtown.blogspot.com/ said...

Nice post..
best orthodontist in delhi

Veav by nimisha priya said...

Very nice post.
Designer dress in Mumbai

Y and H Cargo India said...

Great..!!thanks for sharing with us.
Shipping Company in India

Lillian said...

Due to the pressure of assignments, it has been found that the grades of the students are affected vehemently. Most of them often find it difficult to accomplish and miss the deadline. That is why; the need for different homework help services has come up. It is also found that students often urge for physics help online services. As the subject physics involves different technical stuff, students often fail to manage the academic paper works without a proper guidance. t is found that many of the students often get panicked with their finance homework. With the pressure of the accomplishing the overburdened syllabus, it often becomes troublesome to you to complete the academic papers. This is the very reason you need a finance homework help service.

Theresa Delcas said...

My Assignment Services constantly aim to expand our base of assignment help and call in international experts who are ex-professors from reputed business schools, management schools, engineering universities from across the globe. this provides you with an opportunity to get a global and world perspective in your answers and lets you connect with a writer who understands you. My Assignment Services do not make empty promises and have always achieved what we promised our clients by online Do my assignment. If you need any academic consultation for any assignment type, our team of online assignment helper is available ready to cater to your needs any given moment. We are working round the clock and 365 days of the year. My Assignment Services is your first and last destination for every query you have related to ‘help my assignment’ because we guarantee you the lowest possible price of assignment help sydney with the unmatched quality of assignments in Australia.

MyAssignmentHelpAu said...

Hi this is maxwillor and i am SEO Expert in a reputed company myassignmenthelpau. you are providing very informative and knowledgeable article so thank you for sharing with us. such a nice article. visit now:- assignment help

johnnickson said...

Our GotoAssignmentHelp Best Assignment Help Website for Quality Assignment Writing Help. Take advantage of a pocket-friendly academic assignment help service. Visit us now and book an order.
australia assignment help
online assignment help usa
online assignment help uk
assignment writing service canada
online assignments for students nz
help to do assignment in malaysia
assignment help online singapore
online assignment help uae
cdr writing service

MyAssignmentHelpSG said...

The above information you shared give me a new look at this topic. I am working as a full-time academic writer in myassignmentHelpsg providing my assignment Help Singapore services to college students.

Nikki Heysen said...

Feel amazing and enhance your academic marks by simply getting Online Assignment Help services from the homework writers of StudentsAssignmentHelp.com. Our Canadian assignment writers have great knowledge in all your academic disciplines and topics.

Nathan William said...

Every single student is searching for a proper essay generator who can understand the need of the work and do the essay completely flawless. However, it often happens that students fail to figure out who can be one of the best instant essay typer for their work.
Academic essay writing has been a parameter to judge the merits of the students for a long time. The present time is none the less. In fact, as time is passing by, students are having more and more requirements to write an essay that will be graded. Hence seeking Do my essay help is on the rise and it will be.

James Wayne said...

Great post and nice thought. Please share some related topic. I am very Eager for your next post and visit for HP Printer Offline Windows 10 if you are getting issues with the printer or visit Microsoft Outlook support web page for Outlook information.
Thanks

Online Essay Help said...

GotoEssayHelp service to forget about college stress and struggle. They help students to improve the essays they have written, sharpening the focus of your assignment. Get online essay writing help from our experts at affordable prices.

Our Other Services:

write my essay for me
essay writing services
cheap essay writing
do my essay online
online essay writing services

Mike Rock said...

thanks for sharing it PicTapGo on PC

Unknown said...

Get the best essay writing service by profound writers of MyAssignmenthelpAu at pocket friendly price. We are available 24*7 to assist you in every possible way. For further details on academic assessment email us at cs@Myassignmenthelpau.Com or Phone Number: +61-2-8005-8227.

ejob notifications said...

"Hello, i think that i saw you visited my site thus i came to _return the favor_.I'm attempting to find things to enhance my web site!I suppose its ok to use some of your ideas!! mywegmansconnect | mywegmansconnect login
"

Microsoft Outlook Office said...

Great job! I am really inspired from archive of post and big fan of outlook Support emailing service. You can establish the fine connection with our expert through dialing outlook support phone number. Our site is open 24 hours in a day. Visit here:- http://www.microsoftoutlookoffice.com/
Read more:- office 365 support
office 365 help
Microsoft office 365 support
office 365 support
outlook Support Number
outlook Tech Support
outlook technical Support
Microsoft outlook Support
outlook customer Service
outlook customer Support
outlook help
Microsoft outlook help
outlook help Number
outlook help Phone Number
outlook Tech Support Number
outlook Tech Support Phone Number
outlook Technical Support Number
outlook Technical Support Phone Number
outlook customer Support number
outlook customer Support phone number
outlook customer Service number
outlook customer Service Phone number

HP Printer Troubleshooting said...

HP printer offering a wide range of features and implementing latest technology in its printers to provide good quality printouts. But in case, you face any technical issues with HP printer then contact HP Printer Phone Number to get help from experts. Technician will guide you to resolve your printer’s error.

omeo jones said...

Why should you choose an instant essay typer at MyAssignmentHelp.com? This because the company runs from over a decade, have a team of PhD essay experts, on-time delivery, 24 hour availability, worldwide access etc.
It is not just about how goals have been met; rather the essay should reflect on the personal and academic development of the student by using exemplary instances from classes attended and how these classes, in the perusal of goals, have contributed to the growth of the candidate. We offer variety of assignments including auditing assignment.
It becomes strategically imperative to reevaluate one’s own personal development in an ordered way. Know more about custom assignment writing service.

haryjohn8feb said...

Excellent and nice post. It will beneficial for everyone. Thanks for sharing such a wonderful post. Avail No 1 dissertation editing services UK from certified PhD writers. It is extremely helpful for me. You can email us at info@ukdissertationhelp.co.uk or Phone Number - 020 8144 9988

onlinecontentwritinghelp said...

Truly, this blog post is really one of the very best on the internet. I am an antique ’blog post’ reader and I always try to find a new blog post. essay editing

etharparker said...

Allessaywriter can provide you all the answers regarding who can "write my essay". In our service, we have some of the best essay typer who are professional in this field. We ensure a high quality work within the stipulated time. Not only that, but our prices are so reasonable that you do not have to peep into your pocket before hiring us.

john alexa said...

Thank you so much for great post,i really love it, Are programming really scare you? Or Do you feel programming boring for you? Are you not enable to solve try-catch programming problem? not to be confuse and come to us for Programming Assignment Help where we help the student to ease down the programming errors by the help of our online Programming Help service.

Ethan Lee said...

Myassignmenthelp.com is the best place for anyone looking for Clinical Reasoning Cycle and Nursing Assignment Help, you should chose them if you want good marks in your academics, although there are lot of academic writers in the market but no one can give you Nursing Assignment Help Services like them.

TrustedEssayReviews said...

MyAssignmenthelp review reveal the fact that their experts are quite proficient at providing assignment help on any given topic, for any level. They also suggest that their tasks are delivered on time, and the quality is quite unmatched. It is pretty evident why most of the students have rated it with five stars. One of the most highlighting facts about this portal is that their services are way cheaper than its rivals, which is the reason why MyAssignmenthelp has more clients than any other assignment providers on the internet.

Robert Smith said...

Nice blog. I enjoyed reading your post.I have bookmarked it and I am looking forward to reading new articles. great job... write my essay online

Assignment Help said...


We are expertise in providing Online assignment help
in assignments, thesis and dissertation or be it essays. We have specialists in our organization to write best assignment paper for you on various subjects.

ameelia brown said...

I really appreciate the kind of topics you post here. Thanks for sharing us great information that is actually helpful. Good day! Get instant accounting help, also get 24*7 support form myassignmenthelp.com

HP Printer Troubleshooting said...

Along with the HP printer you need reliable support service to overcome technical glitches and setup hurdles. There might be a possibility of technical error with the printer like paper jam, printer offline, ink cartridge issue and much more. Contact HP Support Printer to easily avail reliable help from experts.

kristen said...

Students Assignment Help provides the best cheapest essay writing service to the students. Our expert writer can write any assignment as they are well-versed with every assignment topic.

studentsassignmenthelp said...

Students don't have to think much for the java homework help services from the professional experts of StudentsAssignmentHelp.com. Our top team is always working with amazing knowledge and they can complete all your task accurately and in the asked format.

Martinlutharz said...

You know, one phone can save your newly bought Apple device from numerous issues. Just give a ring at Apple support number to get establish with me. Here, I will be available all the time offering proper solutions to the needy customers like you. You can also make connection with me and clear out your entire hurdles related to Apple devices or software easily.

Ethan Lee said...

Are you looking for a reliable essay writing service provider that will deliver well-knit essays on any given subject matter at the best industry price? Well, look nowhere else, and choose to hire the best cheap essay writing service provider in your country.
MyAssignmenthelp.com happens to be one of the oldest essay writing companies in the business. Throughout these years, our experts have served a great variety of requests on various proposal essays. If you have a problem with your due proposal essay paper, we can help you solve that just the way you want, irrespective of the proposal essay topics.

rose said...

It is one of the finest websites I have stumbled upon. It is not only well developed but has good content as well. It could prove to be an inspiration for many other entrepreneurs. assignment help australia

Jennifer Brown said...
This comment has been removed by the author.
AmyraLyall said...

GotoAssignmentHelp Services is one among the best Australian
Assignment help provider which will help you out with all your academic related problems. They have a lot of professional experts who have expertise knowledge in giving you proper guidance and providing you with cheap assignment help services. You are free to contact their experts by visiting their website and get your pending presentations completed on the given time. They are giving this services to all the students of different Subjects like nursing assignment help, management help services, humanities help etc . Many students see that Singapore assignment help experts are proficient in writing your assignments but no one wonders the reason behind it.

onlinecontentwritinghelp said...

It’s good to know about your blog and its post. You have done pretty impressive work on this blog and it posts, Keep up doing well. English homework help

Assignment Help said...


You can get that Programming help that you really need online with on-time delivery offered at an economical rate in case you are looking for this service.

Unknown said...

From edubirdie.com review to ukessays reviews and Myassignmenthelp.com Review to Grademiners Review, visiting EssayCritics.com will help you with a comprehensive solution in terms of assignment writing feedbacks and authentic reviews. If an assignment writing firm turns out to be fake or in case the experts offer poor services, then only the feedbacks that are genuinely posted by the students criticizing the flawed areas will be entertained instead of welcoming almost any review without confirming the fact whether the feedback is genuine and coming from real students. At EssayCritics.com, our experts aim to help you find the best online assignment writing firm, so that none of your assignment priorities remains unanswered and unattended in the long run.

Jack Jonas said...

Are you seeking for online study help so that you can score great marks in your exams on having a proper preparation? Well, Get online study help from the expert's writers of My Assignment help. Get connected with us today and enjoy our academic writing service.

Eason Miller said...

Many customers reports that they are facing Canon b200 Error with their canon printer. Most of the printer users are unaware of error codes and do not what does it means. Get immediate help from certified experts to resolve canon printer error b200.

Robert Smith said...

Wonderful illustrated information. I thank you about that. No doubt it will be very useful for my future projects. Myassignmenthelp