Michael's profileMichael's BI SpacePhotosBlogListsMore Tools Help

Blog


    November 02

    Pentaho Solutions: Business Intelligence and Data Warehousing with Pentaho and MySQL

    (ISBN: 978-0-470-48432-6, Paperback: 648 pages)
    Congrats to Roland and Jos for completing and releasing the much awaited tome for Pentaho Solutions !
    It's had some good reviews and as it is with tech specific subjects expects some level of understanding re DWH/BI (DWH Toolkit vol. 2)

    Pentaho's Beekeeper blurb also does a good job of tackling the open source model from their perspective.
    October 21

    SAP/BOE XI3.1 - Scheduling Scaling

    It's not always a case of just throwing more CPU and hardware when intending to scale up for increased processing activities - you need to understand the components that wil be impacted prior to scaling up either horizontally or vertically.

    Scheduling specific:

    Primary throttle point:      

    • Web Intelligence Job Server – scheduling request manager1 CPU can supposedly support 5 jobs – PROD currently has 2 job servers.

    Secondary throttle point:  

    • Web Intelligence Report Server – creates the report instances – will expand to use as many CPU’s as it requires and are available – user address memory cap of 2 GB per report server – Prod currently has 2 report servers.
      Note: Physical Address Extension (PAE) switch changes the addressing mod to enable the OS to access more than 4 GB RAM i.e. the Report Server 2 GB cap limit still applies, but you can effectively run multiple heavy processes.

    In summary:

    2 more CPU’s could double the current scheduling capacity but I would suggest adding more RAM and looking into PAE.
    We need:

    • To understand and adjust some parameters for XI 3’s built-in memory management.
      I.e. it is possible to recapture memory based on the “Enable Memory Analysis” options – I think the defaults are still in place.
    • Additional info on the unmanaged disk PDF generation overheads.
    • Average anticipated report run times / and sizes in case we have to adjust the -RequestTimeout parameter and the cache settings on the Report server.
      I.e.it’s not enough to just know anticipated disk space requirements as one also needs to scale parameters for anticipated load increases.
    • Review Tomcat heap settings and/or Tomcat 6 upgrade.
    • Review the CMS database for optimal use.   
    Generic:

    Web Application Server (Tomcat):

    • 400 sessions
    • 100 simultaneous requests per processor – non report requests i.e. folder clicks etc
    • 50 simultaneous report viewing requests

    CMS Database:

    • 600 concurrent active users
    Additional:
    • If Prod is a quad / 8 GB – and you're running on a 2 CPU license, it has yet to be determined if & how the CPU license cap throttles back on a server that is scoped beyond the cap.
    • Always collate detailed statistics on the existing installation prior to expanding – suggest this as priority 1.
      E.g. Use Perfmon or a similar monitoring tool over some consecutive batch and heavy user windows for OS / memory / processor / application stats.
    • A review of ALL XI component configuration parameters - apart from the built in memory management mentioned previously, there are also Web Intelligence Report Server cache clean / dirty memory settings still use the defaults.
    • The vendor has no recommendations for the Connection Server – possible throttle point – need to determine some thresholds on this component asap.
    • Tomcat 6.0.2 supposedly has improved Memory usage optimizations / Advanced IO capabilities - the default install is 5.5.
      i.e. always look if it is possible to gain from any new releases e.g. XI 3.1 SP2 running on Tomcat 6 - benchmark this first!
    • Crystal / Olap Voyager
      A total rethink will be required if intending to adopt / use these components along with scaling up for increased batch activity!
    • Etc.


    October 11

    FOSS/FLOSS/Open Source/Freeware

    Most do not understand the delineations between the various offerings and it's even more disheartening to hear this from the mouths of account managers or CIO's - much to the detriment of some serious contenders for their BI space.
    James Dixon's Blog takes a novel approach to rectifying this.

    On the subject - Micro Strategy now offers the Report module with 100 free seats and Pentaho has some more stuff from Lucidera!!
    September 15

    Muay Thai BI

    Humanity prides itself on the Freedom of Choice - but with BI one cannot be emotive or overly logical about the selection of a technological direction.
    New entrants probably have it easier as there is no legacy or emotive baggage re a preferred vendor and tool set, but that does open Pandora's Cube with respect to the variety and interoperability of today's many offerings.
    Ramping up from an existing set of solutions oft see's brand loyalty get blindsided by the shiny new toys on offer from competitors all a-twitter with the very latest buzzwords promising rainbows of information amongst the 'BI Clouds'.
    But where is brand loyalty with all the mergers - or does vendor consolidation and RDBMS tool set tie ins dictate more than they used to in the sense that the one stop SAP, Microsoft, IBM and Oracle shops make more sense these days
    Sure - there are many benefits to be had from specific applications like Qlikview or from the open source application or enterprise offerings like Mondrian and Pentaho - not to mention BI Application Servers from the likes of IBM/Cognos etc, so the field is pretty complex. 
    And then there are escalating costs....... ! Always a strong motivator for corporations busy reviewing what they have and what they need and what they might like against What They Will Pay!

    BI for some is mission critical and you're sure to find them amongst the TDWI events and actively monitoring their investments on the Gartner Quadrants.
    Others will be more open to changing technologies when there current outdated installations warrant a major upgrade or migration.

    And then we're back to choice - you can buy a pre-created tool comparison from Ciny Howson or search online for a list of many offering the same e.g, BI Toolbox.
    If you don't want to buy, then create your own comparison matrix: Example

    The fact is that each site has specific requirements with a number of overlapping factors that influence the best fit of technological choices catering for current and future needs.
    Local support and resources, compliance with the organisations overall IT strategy, scalability and scope for future requirements are but a few factors to take into consideration that have not much to do with the delivery promises of the vendor and his magic bag of tricks.

    It ALL boils down to which product(s) best suite your organization in the most multi-faceted way!

    And exactly how does Muay Thai fit into all this?
    Well, lets just say that if you want an effective no frills systems that works for a stand up fight then Muay Thai is the way to go.
    To expand the fight down to the ground as in UFC you will need some serious grappling skills - Jiu Jitsu.
    I've trained in the standard styles which were great for learning the basics and discipline, but to be quite frank, I've rather switched to a Muay Thai regime for the sheer beauty of it's effectiveness.
    There is a place for all styles - normally matched by your inner resolve and body type as to why you want this training.
    The Chinese and Indian styles are for slim practitioners that rely on circular movements to offset their lack of bulk and power.
    The Japanese and koreans are brutally leg and body strong so have opted for linear aggressive attack lines.
    The Brazilians like flair, are very mobile and embrace beauty and balance with their Capoeira routines.
    Not all training is available anywhere, and there have been many offshoots and break away schools - Bruce Lee was the initiator of taking the old and adapting it to something new and effective, whilst the Gracie family in Brazil took wrestling and judo to task with their ground and pound Jui-Jitsu.
    Steven Segal promoted Aikido - interesting it's mechanism of using an opponents force against himself, whilst Jean Claude Van Damme trained in Shotokan and even did Ballet training for stretching and flexibility.
    The russians keep appearing - hard men with Spetsnaz training.

    But pound for pound and minimalistic effective approach, my money would be on Muay Thai ... maybe it's the music they play at the matches lol but the same could be said for many BI tool sets and all the marketing and hype doing the rounds.



    September 08

    Forecasting: Tree of Knowledge or Sour Apples?

    The most obvious question one can ask about all things forecasting is: Where were you at the eco crunch?
    I had just finished reading The Black Swan by http://www.fooledbyrandomness.com/ when the economy slid into the ocean of debt and was quite gobsmacked at the timing of it all.
    Like anything, Analytics and Forecasting are useless unless seen and understood in the greater context and by the RIGHT people.
    There were many voices trying to scream a warning about the financial state that turned nasty but they were drowned out by man's greed and the fact that people simply don't like bad news - or more to the point, will not make a judgement on the side of caution.
    We're positive gambler's by nature, always looking for the big score or to increase what we have so it's those that wave the red flag that get nailed to the ridiculous cross either in front of the building as a warning, or way out of sight to stop a possible infection.
    The latest tactic is to just ignore these prophets of doom and wait for the daily deluge of facts and white noise to wash it all away.

    Be that as it may, with the right approach and champion, one can slowly educate a dept, corporation, industry or even a planet, as long as the basics are adhered to:
    Forecasting Methodology Tree
    9 Common Forecasting Errors

    August 24

    In Memory Business Intelligence

    In memory processing has been part of the delivery vision for quite some time, but it finally looks to be a reality thanks to the latest processing capabilities.
    The very mention of enterprise servers with graphics cards for GPU utilization has me chuckle at the temptation for alter ego administrator / gamers lol
    Qlikview has it - MS Gemini is promising it and Palo is about to deliver it .... so it's a space worth watching ...

    In Memory Write Up

    Jedox / Palo Vision

    HugoWorld comment
     

    August 12

    Optimising Tomcat startup

    http://wiki.apache.org/tomcat/HowTo/FasterStartUp lists a number of mechanisms with which to help Tomcat startup faster and most of us do not have much leeway apart from removing all unnecessary modules and possibly cleaning up the server.xml file.
    The format of Tomcat's server.xml in a non xml editor (notepad etc) is pretty terrible but I discovered a few benefits of installing the Admin module on Tomcat:
    • Simplifies SSL connector creation - gui with enterable parameter fields
    • Reformats the server.xml file so that it is very readable in notepad.
    • Removes all unnecessary XML defs in server.xml and this was a MAJOR win re the time Tomcat takes to start up!!  
    August 10

    BI Clouds = BICL?

    Acronyms are a good indicator of market acceptance so will BI Clouds be reduced to BICL in the near future?
    All the usual suspects are actively pursuing this offering and the technology is ready for it - but is the customer?
    Data Security and reliance on vendor intervention for configuration and upgrades have always been the biggest challenges but when it comes to resources required to vertically integrate analytical software with their own propriety cloud, Goldberg predicts that vendors will partner with cloud providers or rent cloud capacity from rivals.

    Can it be that the BI service industry is willing to extend the olive branch to enable this remains to be seen, but nevertheless, Clouds are here and we're sure to see some interesting application of this technology in the near future.

    Excellent overview of BICL:
    http://blogs.zdnet.com/Gardner/?p=2986

    Interesting answers to some key challenges:
    http://blog.in2clouds.com/2009/06/bi-and-cloud.html

    New startup?: http://businessintelligence.me/en/
    August 06

    OSCON

    Open Source is making steady inroads into all sectors so I extend my interest beyond the BI and DWH sector.
    Every additional industry adopted component and application helps increase corporate acceptance so an event like OSCON is key to enabling this emerging industry.

    http://www.h-online.com/open/SourceForge-Community-Choice-Awards-2009-winners-announced--/news/113848

    June 22

    Divide & Conquer - BI

    Most organisations will adopt a divide and conquer approach to all projects.
    There are a number of more technical terms used to describe this: iterative, agile & scrum, RAD etc.
    These are effective in their focus and delivery, but face some shortcomings for the organisation as a whole as most do not have a 'game plan' or roadmap in place that is realistic and adaptive enough to adapt along with new discoveries, changes in tactics, or shifting requirements and goal posts - not to mention departmental shifts and political impacts.
    A new or even maturing BI environment must be adaptive enough to change in midstream re it's direction, and yet not lose sight of a greater plan that has to be amended in line with any possible influence on it (and that includes any departmental changes and project impacts from source / feeder systems).
     
    Some interesing posts to this end:
     
     
     
    April 23

    SAP/BO XI 3.1 & VBS

    I had some fun times trying to get some scripts going on XI 3.1 --
     
    For the BIAR comand line scripts: http://meetthetaylors.com/files/Start_BOE.cmd.txt
    To wrap these in a VBS shell:
     
    Dim WshShell
    Set WshShell = WScript.CreateObject("WScript.Shell")
    WScript.Echo WshShell.CurrentDirectory
    WshShell.CurrentDirectory = "C:\Program Files\Business Objects\common\4.0\java\lib\"
    WScript.Echo WshShell.CurrentDirectory
    WshShell.Run "java -jar biarengine.jar biar.properties"
     
    For a really cool VBS editor packed with samples for just about anything: http://www.vbsedit.com/scripts/os/process/procthd/scr_992.asp
     
    April 15

    KPI's

    The critical component that links the business unit to the technical units re BI is the KPI. Proper identification and prioritisation of these metrics is key to ensuring that the focus stays where it should - on deivering usable information as opposed to gathering data and technology.
    To this end there are many kpi's and metrics that are industry specific and standard to most organisations - as seen by most packaged applications these days - but there are also sites dedicated to sharing experiences and KPI's
     
     
     
    April 08

    BI Ethics and Morality

    Sitting in the tech trenches we oft forget the c/overt influences of information on people and processes. We are creatures of habit, and the vastness of the web is constantly undermined by humans falling back into routine and visiting favourite sites.
    Once in a while I throw a combination of unlikely terms together and do some random searching just to break the more obvious patterns of acquiring knowledge. "Business Intelligence and Empirical" soon led to

    Business Intelligence meets Moral Intelligence Call for Papers for Vol. 10 (01/2009)

    The paper can be accessed at the International Review of Information Ehics: http://www.i-r-i-e.net/current_issue.htm

     

    April 06

    Mashups, Dashboards & Surfboards

    Designing a dashboard is like chosing a surfboard - you want optimal performance, with the latest flow design and it must look good and be really effective.
    Sometimes you don't really know what you want until you see it - so trolling online stores, visiting numerous retail shops and checking out the boards in use at your local break are par for the course until you narrow down the various choices into a suitable price range (and it must look radical! lol).
     
    For dashboards the same rule applies - except most BI dpt's get tasked to emulate an excel spreadsheet that's packed with various tabs and charts etc as a starting point.
     
    Look no further - this has to be the largest site of exmples - EVER http://dashboardspy.com/ - the top toolbar lists them in ranges of 1 - 250 etc until a 1000! 
    April 05

    Bypassing the Kimball-Inmon Arg - CDWA vs DDP

    It's oft said that an organization's DWH strategy either adopts Bill Inmon's Ralph Kimball's philosophy:

    Bill Inmon's paradigm: Data warehouse is one part of the overall business intelligence system. An enterprise has one data warehouse, and data marts source their information from the data warehouse. In the data warehouse, information is stored in 3rd normal form.

    Ralph Kimball's paradigm: Data warehouse is the conglomerate of all data marts within the enterprise. Information is always stored in the dimensional model.

    There is no right or wrong between these two ideas, as they represent different data warehousing philosophies. In reality, the data warehouse in most enterprises are closer to Ralph Kimball's idea. This is because most data warehouses started out as a departmental effort, and hence they originated as a data mart. Only when more data marts are built later do they evolve into a data warehouse.

    But - it is never a perfect world and the above are typically referred to as Classic Data Warehouse Architectures (CDWA) - and here is a very interesing discussion of the Data Delivery Platform (DDP) in trying to introduce versatility and reality with information delivery:

    http://www.b-eye-network.com/view/9960

     

    April 02

    Oh OLAP!

    I have not had time to check out the Mondrian inclusion with the latest Pentaho suite, but there are a few additional enhacements in this area from an open source perspective:
    http://freeanalysis.wordpress.com/ is an interesting concept, but I don't know of many sites that allow a myriad of technologies that would require having a generic meta data transfer mechanism. The side bar links highlight most of the more well known alternatives for OLAP technologies.
     
    This space http://www.sqlsolutions.com/products/OLAPHeartbeat/index.html is always worth keeping an eye on as it offers some effective addon's to things SQL - so it's nice to see the OLAP offerings coming into play.
     
    The founder of the Mondiran OLAP engine is busy running some presentations at the 2009 Pentaho Partner Summit this week - and he has a blog http://julianhyde.blogspot.com/2009/03/pentaho-partner-summit-2009.html
     
    April 01

    Information Portal - Simplified

    As developers and solution providers we oft forget to provide a more holistic approach to providing information in a consolidated environment.
    Faced with a growing web farm of development, test and production installations - some shifting across virtual servers, I tried a number of open source collaboration installs to try and provide an information hub with updated links and documents for Business Objects XI 3.1.
    After numerous attempts in trying to add a simple portal to the same Tomcat instance running on Windows 2003 for XI, I eventually settled for http://www.emforge.org/
    with a link to a mindmap of all the sites links.
    I included the Eclipse module for Online Business Objects Documents and am using the News tab for release announcements. 
    March 31

    Recession & BI - Open Source initiator or not?

    The current state of the global economy should prove to be a catalyst for more aceptance and adoption of Open Source technologies - but experience tells me that cost reductions typically drives a shift to short term thinking with respect to all projects and new endeavors.
    Most simply batten down the hatches and pause or cancel al projects that were slated to add to the existing solution spectrums.
    Conversely we know and hear that if there ever was a time that BI and Analytical systems and processes were required, it should be now, but most regard this as simply closing the door on the barn after the bull has left and short term thinking becomes the de facto driver for new projects - including solution expansions.
    Those that are already on a 3 or 5 year strategy might see cost reductions affecting either architecture expansions or additional software purchases - but I doubt that this will cause a shift to re-evaluating tool purchases for leverage of open source or cheaper alternatives.
     
    Having said that - the current situation should see many new comers to BI and Analytics look elsewhere for solutions - and hopefully this will be the impetus that open source could do with re gaining a further foothold into acknowledgement and enterprise usage.
     
    On an enterprise level it might be worth keeping a b eye on http://www.infobright.com/ and http://www.kickfire.com/ - the latter being an interesting option when considering the leaps MYSql is making in scaling itself up.
     
     
     
     
     
    March 30

    Real Time OLAP & Visuals

    OLAP is always under or over rated due to the prebuild and load factors - interesting to see a real time Olap endeavor http://www.aleri.com/cep/aleri-streaming-platform/aleri-live-olap
     
    The XI 3.0 Scoraecard is available at http://www.biscorecard.com/ - sure you have to purchase it, but well worth the value if faced with the migration decision from previous releases versus a shift to another Enteprise tool set.
    I receive many questions re the decision making process of adopting Business Objects, and cannot believe that most miss the non technical rakings of local support and vendor coverage.
     
    On the subject of visualizing - here's an old link that has some interesting coverage of social and network visualization tools:
    January 27

    Xtreme Data visualization - Selling your idea / project

    Conceptually - everyone has an idea as to how to present data - or how to sell a concept - but fall short in understanding the dynamics involved .....
     
    Most vendors are adding more flash to their toolsets - so it's up to reality driven project managers to know what and how to pursue these additional offerings - and when!
     
    All the best in the Year of the Ox .....