About Drikus

I think of myself sometimes as a wandering and wondering idiot, walking the forest of reality. I follow little paths and discover newness all round like an explorer. It doesn't necessarily make it a pleasure and I thought at my age I would have figured life for its greater part, but yet the forest is dense and the paths many.

Visual Studio 2013 Database Project: Item Templates with VSIX

Visual Studio 2013 Database Project: Database Item Templates with VSIX

Overview

Strictly speaking VSIX is not a database project or related in kind, but by virtue of Item Templates.  With VSIX projects and the TemplateBuilder (NuGet) package, templates can be built from the contents of an ItemsTemplates folder.  The latter (if not created by TemplateBuilder) should be manually created as ItemTemplates folder as an item of the VSIX project.

NOTE: SQL Server (database) templates are not supported in Visual Studio 2013 VSIX and requires a manual effort to install on a developer’s IDE.  As indicated by the error item below, if the database files added to a VSIX project are changed to have a vstemplate file extension the compiler (on build) should issue this error saying that only CSharp and VBasic templates. Similarly if a Reference is added in the Asset tab of the manifest, the error The targer “TemplateProjectOuputGroup” does not exist in the project” is presented.

VS Database Template Support

VS Database Template Support

VSIX Template

The VSIX project template is available from the Visual Studio 2013 SDK and found under the menu: [Visual C#] >> Extensibility >> VSIX Project.

Template Builder

The TemplateBuilder package is installed from the Package Manager console or from Visual Studio: Tools >> NuGet Package Manager >> Package Manger Console.  Once installed, template are built to the folder: <Drive:>:\Users\<User>\Documents\Visual Studio 2013\Templates\ItemTemplates and <Drive:>\Users\<User>\Documents\Visual Studio 2013\Templates\ProjectTemplates.

NuGet Package Manager

Download and install the NuGet Package Manager for Visual Studio 2013.  It has a VSIX extension but installs by double-click and run.

SideWaffle

SideWaffle is a template pack that adds a selection of project and item templates to Visual Studio, including enabling templates to use for creating new ones. A VSIX project generates an experimental invocation of Visual Studio in which to test these templates so that the real instance is not affected.  To configure templates require a work-flow that is enabled by a separate application (that you create) to populate template and custom parameters when creating a new item.  This article, for now, mentions it but is not using it to create templates.

Create a New VSIX Project for Template Deployment

In order to author customised solutions, projects, code-snippets, and templates, a VSIX project is required.  Depending on the user-environment set-up for Visual Studio, file paths may vary slightly.

Create a new project:

  1. Follow the path: File >> New Project >> [Custom Path] >> [Visual C#] >> extensibility >> VSIX Project
  2. Provide a name for the project and check the solution details
  3. Fill out the VS VSIX Manifest with the appropriate details (reference)
  4. Install the TemplateBuilder package.
VS VSIX Manifest

VS VSIX Manifest

Installing TemplateBuilder

This installation requires the Package Manager Console (NuGet):

  • Follow the path: Tools >> NuGet Package Manager >> Package Manager Console.
  • Type Install-Package TemplateBuilder -Pre at the PM> prompt and hit [Enter].  NOTE: Internet access to https://www.nuget.org/api/v2/ is required.
VS Package Manager Console

VS Package Manager Console

The TemplateManager installation responds favourably on completion of the install, if it succeeds.  NOTE: The NuGet package is downloaded into the file location: <Drive>:\Users\<user name>\AppData\Local\NuGet\Cache and in the event of not having the default Internet location (https://www.nuget.org/api/v2/) available, Visual Studio looks here for whatever the package is being installed.

VS NuGet Installation of TemplateBuilder

VS NuGet Installation of TemplateBuilder

The installation process deposits an example template into: <Drive:>\Users\<User>\Documents\Visual Studio 2013\Templates\ProjectTemplates.  NOTE: there may not actually be an example template in this folder.

Create a Template Solution and Project

The Visual Studio Solution should contain the VSIX project as well as other projects, i.e. those used in which to create the templates.  For example, to create database templates a database project is added to the solution and the corresponding templates with parameters created in there.  This article uses the template created in a previous post (Visual Studio 2013 Database Project: Create a Stored Procedure Item Template) to illustrate how the VSIX project functions in this respect.

There are two methods:

  1. Exporting and importing a template (zip file)
  2. Referencing the template source project from the VSIX project where they are in the same solution

The latter (point 2) does not work for database templates

Exporting the Template

The template (or stored procedure created in the templates project) is created by exporting the necessary file (the actual stored procedure and any referenced files in the case of a project or c# type template) to the folder: <Drive:>\Users\<User>\Documents\Visual Studio 2013\My Exported Templates.  Follow the next steps to affect this:

  • Create, for example a stored procedure template (reference the above link on how to do that) an File >> Export Template >> Item Template, select the correct project, i.e. the database project containing the stored procedure created.
  • Select the correct stored procedure and complete the wizard.  The output template is locatable in the folder: <Drive:>\Users\<User>\Documents\Visual Studio 2013\My Exported Templates in a file called: <Template Name>.zip.

NOTE: Edit the MyTemplate.vstemplate file (contained inside the zip file) in Visual Studio and add an element TemplateGroupID (depicted in the code below) between the templateData tags to help with the filtering of template in the New Item dialog (reference).

VS Template Projects and Item Template

VS Template Projects and Item Template

1
 <TemplateGroupID>Database</TemplateGroupID>

The Item Template (Stored Procedure) is added to the VSIX project.  Use the .vsixmanifest file and add an Itemtemplate item to the Assets tab. Add a:

  • New item,
  • Pick ItemTemplate and
  • Point the File Location to the folder where the copied template zip file has been placed.

NOTE: The path could be: :\Users\\Documents\Visual Studio 2013\Templates\ItemTemplates\\Database\Programmability, but it is up to choice.  However, Visual Studio, on starting up, reads from this location on a user-instance, i.e. local machine.  The project is appended with a new folder, i.e. ItemTemplates and the zip file is appended.

On the  the property of the newly added template (zip) file, set the property Copy to Output Directory is set tot Copy Always.

NOTE: To inspect the XML code of the project file: right-click >> Remove and then right-click >> edit) reveals this setting in its XML structure, as depicted below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
    "ItemTemplates\Samples\StoredProcedure.zip">
      Always
      true
    
    "packages.config" />
    "Properties\template-builder.props" />
    "source.extension.vsixmanifest">
      Designer
    
  

NOTE: Turn off the Build and Deploy options in the Configuration Manager for the VSIX project.

Using a Reference to the Template Project in the same Solution

A reference to the template project (the one where the stored procedure is authored) is added to the Reference of the VSIX project by using the source.extension.vsixmanifest file’s Assets tab, as before.  But, in this case, the reference is to a project in the same solution.  The fields [Type],

[Source], and [Project] are completed, with project being the name of the said template project.  This adds a reference to the VSIX project.

Set the Properties of the reference (see how), especially the VSIX Sub Path that determines where the template is installed on the menu-tree of the New Item menu function.

NOTE: Visual Studio does not provide a project template for creating database objects, so it seems the export method is the only way in which to get the zip file into the VSIX for deployment.  This aspect may be resolvable by creating the zipped files in a database project and let the project zip them like the wizard and place in the appropriate folder.  Not yet sure how to do this.

Build the Project and Install the Templates

The VSIX project is built and it copies the template zip file to the output folder: \Users\\Documents\Visual Studio 2013\Projects\\\bin\Debug as a .vsix (installable) file along with a TemplateBuilder.dll and the folder structure (Visual Studio) that contains the actual template file.

NOTE: For developer projects, this installation file should be made available on a network shared drive and installed as updates are made, or from a Private Gallery reference) that notifies users via RSS feeds of new extensions.  Alternatively the same file can be uploaded to the Visual Studio Gallery for community consumption and addition.

file in the VSIX project to set the [Assets] by creating a new reference to an ItemTemplate of a project (the one containing the stored procedure template) that also adds it to the References of the VSIX project. Here the Properties of the added assembly can be altered to achieve a variety of things. See the screen image (below) for a visual explanation of what is described.

VS VSIX Template Database Item Folder Location Reference

VS VSIX Template Database Item Folder Location Reference

Executing the <Name>.vsix (installable) file invokes all the things that were set in the manifest to install the template(s).  These are available in Visual Studio and can be inspected via: Tools >> Extensions & Updates >> Templates
, illustrated below.

VS Extensions and Updates Templates folder

VS Extensions and Updates Templates folder

Using the newly installed Templates

After installing the VSIX template package and verifying the extension is in Visual Studio, the template can be used from which to create Projects or Items.  During the installation the templates are deployed to a random folder: %LocalAppData%\Microsoft\VisualStudio\12.0\Extensions and by virtue of %LocalAppData% it only applies to the current user.  Alternatively, setting the manifest to AllUsers=true the extension is installed under Common7\IDE\Extensions (where VS is installed) and appears to all users on the computer (reference).

Advertisements

Visual Studio 2013 Database Project: Working the Project LocalDbs

Visual Studio 2013 Database Project: Working the Project LocalDbs

Overview

Visual Studio (VS) has evolved much to what it is capable of today for database development and deployment.  With the addition of a much better table create facility Visual Studio is the development tool of choice for the database developer.  SQL Server Management Studio still has its place as a light free tool, but for serious database development with Team Foundation Integration, Visual Studio is the tool of choice.

Using Local Databases

Visual Studio creates local databases when a database project is created, viz.

  1. MSSQLLocalDb — SQL 2014 local db
  2. ProjectsV12 — created by SQL Server Data Tools (SSDT)
  3. V11.0 — SQL 2013 local db
VS SQL Server Object Explorer

VS SQL Server Object Explorer

These local database instances are located on a file location on the local drive of the computer that runs Visual Studio at: C:\Users\hendriekusr\AppData\Local\Microsoft\Microsoft SQL Server Local DB\Instances.

The server name consequently is: (localdb)\ProjectV12.  Details about these installed databases can be obtained by running the SQlLocalBD.exe (located in C:\Program Files\Microsoft SQL Server\110\Tools\Binn\) against the name of the database, e.g. info ProjectsV12.

VS Local Database Information

VS Local Database Information

Use the Instance pipe name (from above execution) to connect to a local database from SQL Server Management Studio (SSMS), as depicted below.

SSMS Local Db Connection Dialog

SSMS Local Db Connection Dialog

A connection (to view database objects) is possible from VS and SSMS with the latter using the pipe connection string.  The SSMS screen (below) illustrates how the connection is made to the localDB as well as a Stored Procedure created from a SSMS template that was added to the database.  The next article [Create a Stored Procedure Item Template] illustrates how this same SSMS Stored Procedure template is created for Visual Studio.

SSMS Connection to LocalDB with Stored Procedure from Template

SSMS Connection to LocalDB with Stored Procedure from Template

Previous Article (2) [Create Database Project Folders]. . . Next Article (4) [Create a Stored Procedure Item Template]

Visual Studio 2013 Database Project: Create a Stored Procedure Item Template

Visual Studio 2013 Database Project-Create a Stored Procedure Item Template

Overview

Using Visual Studio for database development requires that database objects like stored procedures be created, amongst other items such as tables, view, triggers, etc.  It is advantageous if creating an item that the proposed script is a template that already conforms to the company’s standards, by providing a boiler-plate script that speeds up the developer’s options.  Even better would be if these templates are available to all developers as they check items in-and-out of Team Foundation Server.  Bear in mind that a particular organisation might have a single development office or location, but others may be geographically dispersed.  It would be expedient if all of them are singing from the same hymnal.

This article explores the basics of Visual Studio Templates for database development.  Using this phrase as a search term, an Internet search engine divvies up a few articles in support.  One of which is called An Overview of Templates for Database and Server Objects (Microsoft).  It suggests that existing templates can be customised to affect all templates used by the team.  Customisable project item/ stubs accelerate development by providing the developer with a head-start. Templates are comprised of project and item templates.  Every time a user opens a menu item in Visual Studio to create something, a template is invoked, e.g. New Project, or Add New Item.

Templates exist in the user community (How to Add Items Using Online Item Templates) or a VSIX project with SideWaffle (Code4Fun), but these aren’t discussed here.  To Illustrate the concept, a template for a Stored Procedure is created and added as an Item Template to the local developer machine.  This is absorbed into Visual Studio and provides the developer with a new item in the Add New Item option.

Creating the Template

A Stored Procedure (via Add New Item ) in Visual Studio (VS) and for that matter in Microsoft SQL Server Management Studio (SSMS) is pretty uneventful.  Therefore, should a developer create quite a few items, the work required to customise these to the company standard absorbs unnecessary time and effort.  A template should make this much easier.

Either use VS or SSMS to create a stored procedure that conforms to all general company standards.  This one to be used in Visual Studio, so the usage between SSMS and VS is different from a parameterisation perspective.

In SSMS the code-block (below) illustrates how parameter are coded and then on invocation provided by Ctrl+Shift+M.  In Visual Studio this is done through an XML configuration and only what the respective dialog requests on New Item is user-provided, e.g. the project or object name.

1
2
3
4
5
6
CREATE PROCEDURE	[<Schema_Name, sysname, Schema_Name>].[<Procedure_Name, SYSNAME, ProcedureName>] 
					<@Param1, SYSNAME, @p1> <Datatype_For_Param1, , INTEGER> = <Default_Value_For_Param1, , 0>
				,	<@Param2, SYSNAME, @p2> <Datatype_For_Param2, , INTEGER> = <Default_Value_For_Param2, , 0>
				,	<@Param3, SYSNAME, @p3> <Datatype_For_Param2, , INTEGER OUTPUT>
WITH EXECUTE AS [CALLER	| SELF | 'user_name'] -- Default: CALLER				
AS

The flowing code-block represents the Visual Studio equivalent of the obove code for the same purpose.  The $fileinputname$ is the file name provided by the developer when creating a new stored procedure.  All the other parameters are configured in the Item Template’s XML file.  Therefore, the XML configuration should prescribe the purpose of the stored procedure, I.e. suit exactly what it is being created for. 

1
2
3
4
5
6
CREATE PROCEDURE	[$SchemaName$].[$fileinputname$] 
					$ProcParam1$	$ProcParam1DataType$	=	$ProcParam1Value$
				,	$ProcParam2$	$ProcParam2DataType$	=	$ProcParam2Value$
				,	$ProcParam3$	$ProcParam3DataType$		$ProcParam3Type$
WITH EXECUTE AS [CALLER	| SELF | 'user_name'] -- Default: CALLER <-- make a selection in the executing principle context.				
AS

A Visual Studio Item Template for a Custom Stored Procedure

The provided template fast-tracks a developer and provides a consistent standard for every stored procedure created base on this template.  This particular standard is comprised of a few sections, viz.

  • Create with execution principal context
  • Flower box containing the abstract and owner details
  • Initial proc-wide directives, e.g. NOCOUNT, EXACABORT, and TRANSACTION ISOLATION
  • Error-handling variables (for RAISERROR)
  • Local variable declaration & initialisation
  • Main query execution block wrapped in TRY. . . CATCH and transaction
  • CATCH block with detailed error handling with XACT_STATE
  • Return value for application event handling
  • Directive switch to system defaults
  • Metadata section for object details (metadata) when it’s built in the database
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
CREATE PROCEDURE	[$SchemaName$].[$fileinputname$] 
					$ProcParam1$	$ProcParam1DataType$	=	$ProcParam1Value$
				,	$ProcParam2$	$ProcParam2DataType$	=	$ProcParam2Value$
				,	$ProcParam3$	$ProcParam3DataType$		$ProcParam3Type$
WITH EXECUTE AS [CALLER	| SELF | 'user_name'] -- Default: CALLER <-- make a selection in the executing principle context.				
AS
/* ==================================================================================================================================
	ABSTRACT:	This procedure executes for $projectname$ and. . . 
	VERSION:	$ItemVersion$ (MAJOR.MINOR.PATCH)

	OWNER:		$ItemOwner$
	TEMPLATE:	$TemplateVersion$ (MAJOR.MINOR.PATCH)

	<NOTE: to developer, complete the metadata at the end of this procedure script.>

											<< See bottom of item for details >>
=================================================================================================================================== */
BEGIN
	SET NOCOUNT ON;
	--SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED
	--				READ COMMITTED
	--				REPEATABLE READ
	--				SNAPSHOT
	--				SERIALIZABLE;
	
	-- --------------------------------------------------------------------------------------------------
	--	Renders entire txn uncommittable if constraint violation occurs. 
	--	ms-help://MS.SQLCC.v10/MS.SQLSVR.v10.en/s10de_6tsql/html/cbcaa433-58f2-4dc3-a077-27273bef65b5.htm
	-- --------------------------------------------------------------------------------------------------
	SET XACT_ABORT ON;
	
	-- ------------------------------------------------------------------------------------
	--	VALIDATE INCOMING PARAMETERS
	-- ------------------------------------------------------------------------------------
	-- None.
		
	-- -------------------------------------------------------------------------------------
	--	DECLARE LOCAL VARIABLES
	-- -------------------------------------------------------------------------------------
	-- Error Handling Local Variables
	-- Delete this for SQL 2014 unless required for custom error logging
	DECLARE			@vErrorNumber		INTEGER	=	0
				,	@vErrorMessage		NVARCHAR(2048) = N''
				,	@vErrorSeverity		INTEGER = 0
				,	@vErrorState		TINYINT = 0
				,	@vErrorLine			INTEGER = 0
				,	@vErrorProcedure	NVARCHAR(256) = NULL;
	
	-- Processing Local Variables
	DECLARE			@LocalVariable1Name	INTEGER
				, 	@LocalVariable2Name	DATETIME;
			
	-- -------------------------------------------------------------------------------------
	--	INITIALISE LOCAL VARIABLES
	-- -------------------------------------------------------------------------------------
	
	-- Initialise processing variables
	SET			@LocalVariable1Name	=	0;
	SET			@LocalVariable2Name	=	'2011-01-01';

	/* =========================================================================================
		MAIN CODE EXECUTION
	============================================================================================ */
	BEGIN TRY
		BEGIN TRANSACTION;
		-- { sql_statement | statement_block }
		COMMIT TRANSACTION;
		RETURN(@vErrorNumber);
	END TRY
	BEGIN CATCH
		/* ------------------------------------------------------------------------------------------
			ERROR HANDLING
			Severity 0-10 (information) are not caught by TRY. . . CATCH
			Severity 11-16 (user-correctable)
			Severity 17 (Insufficient Resources)
			Severity 18 (non-fatal error)
			Severity 19 (SQL resource error)
			Severity 20-25 (fatal system error) execution termination with SQL close
			https://msdn.microsoft.com/en-us/library/ms164086.aspx

			Any sys.messages valid messages can be utilised but must be passed to RAISERROR or THROW.
		-------------------------------------------------------------------------------------------- */

		-- Grab error indicators.
		-- Delete this for SQL 2014 unless required for custom error logging
		SELECT			@vErrorNumber		=	ERROR_NUMBER() 
					,	@vErrorLine			=	ERROR_LINE()
					,	@vErrorSeverity		=	ERROR_SEVERITY()
					,	@vErrorState		=	ERROR_STATE()
					,	@vErrorProcedure	=	ERROR_PROCEDURE()
					,	@vErrorMessage		=	ERROR_MESSAGE()+ CHAR(10)
												+ N'Executing code item name or type ('+ISNULL(@vErrorProcedure,N'AdHoc SQL')
												+ N'), error number ('+ CONVERT(NVARCHAR(20), @vErrorNumber)
												+ N'), severity ('+CONVERT(NVARCHAR(20), @vErrorSeverity)
												+ N'), code error line ('+CONVERT(NVARCHAR(20),@vErrorLine)										
												+ N').'+CHAR(10) +CHAR(13);
		-- -------------------------------------------------------------------------
		--	END TRANSACTION CORRECTLY.		
		--	XACT_STATE =  1 ~ Transaction is committable.
		--	XACT_STATE = -1 ~ Transaction is uncommittable - roll back 
		--	XACT_STATE = 0 ~ No transaction - commit or rollback  results in error.
		--	ms-help://MS.SQLCC.v10/MS.SQLSVR.v10.en/s10de_6tsql/html/248df62a-7334-4bca-8262-235a28f4b07f.htm
		--	https://msdn.microsoft.com/en-us/library/ms175976.aspx
		--  Any custom error loggin should be implemented here
		-- -------------------------------------------------------------------------
		IF (XACT_STATE() = 1) 
		BEGIN
			COMMIT TRANSACTION;
			PRINT N'Transaction('+ISNULL(ERROR_PROCEDURE(),N'AdHoc SQL')+N'): COMMITTED';
			RETURN(@vErrorNumber);
		END ELSE
		BEGIN
			IF (XACT_STATE() != 0) ROLLBACK TRANSACTION;
			SET @vErrorMessage	= @vErrorMessage + N'Transaction: ROLLED BACK';

			-- SQL 2000+
			RAISERROR	(	@vErrorMessage		-- { msg_id | msg_str | @local_variable }
						,	@vErrorSeverity
						,	@vErrorState		-- [ ,argument [ ,...n ] ] )  [ WITH option [ ,...n ] ]
						)
			-- SQL Server 2014+
			--THROW; 
			RETURN(@vErrorNumber);
		END
	END CATCH;

	-- Cleanup/ Housekeeping
	SET TRANSACTION ISOLATION LEVEL READ COMMITTED; -- Database Default
	SET XACT_ABORT OFF;
	SET NOCOUNT OFF;
END

/* ==================================================================================================================================
	TEMPLATE AUTHOR: $TemplateAuthor$, COMPANY: $OwnerCompany$

	OBJECT VERSION HISTORY
	AUTHOR:	<Name.Last Name>, ORGANISATION: <Company Name>, AUTHORED DATE <YYYY-MM-DD>

	VERSION		DATE		NAME SURNAME		CHANGE DETAILS
	---------	-----------	-----------------	---------------------------------------------------------------------------------------
	$ItemVersion$		YYYY-MM-DD	Name.Surname		Change details are.....

												<< See bottom of item for details >>
=================================================================================================================================== */
GO
/* ---------------------------------------------------------------------------------------------
	OBJECT METADATA
	NOTE:	This section has to be edited and run for all attributes after each edit to keep the
			database metadata up to date.
	 Add extended properties to the view (in database objects)
		1. Author name
		2. View naming specifics
		3. View version
		4. Editor name
		5. Edit date
		6. Template version
 ----------------------------------------------------------------------------------------------- */

-- Author details
EXECUTE sys.sp_addextendedproperty	@name = N'Author Name'
				,	@value = N'$ItemAuthor$' --<-- Change this only!
				,	@level0type = N'SCHEMA'
				,	@level0name = N'$SchemaName$'
				,	@level1type = N'PROCEDURE'
				,	@level1name = N'$fileinputname$'	
				,	@level2type = NULL
				,	@level2name = NULL;
GO				

-- Object description
EXECUTE sys.sp_addextendedproperty	@name = N'Functional Description'
				,	@value = N'This procedure executes for $projectname$ and. . .' --<-- Change this only!
				,	@level0type = N'SCHEMA'
				,	@level0name = N'$SchemaName$'
				,	@level1type = N'PROCEDURE'
				,	@level1name = N'$fileinputname$'	
				,	@level2type = NULL
				,	@level2name = NULL;
GO

-- This view version
EXECUTE sys.sp_addextendedproperty	@name = N'Version'
				,	@value = N'$ItemVersion$' --<-- Change this only!
				,	@level0type = N'SCHEMA'
				,	@level0name = N'$SchemaName$'
				,	@level1type = N'PROCEDURE'
				,	@level1name = N'$fileinputname$'	
				,	@level2type = NULL
				,	@level2name = NULL;
GO

-- Editor details
EXECUTE sys.sp_addextendedproperty		@name = N'Editor Name'
				,	@value = N'$EditorName$' --<-- Change this only
				,	@level0type = N'SCHEMA'
				,	@level0name = N'$SchemaName$'
				,	@level1type = N'PROCEDURE'
				,	@level1name = N'$fileinputname$'
				,	@level2type = NULL
				,	@level2name = NULL;
GO					
-- Edit date
EXECUTE sys.sp_addextendedproperty		@name = N'Edit Date'
				,	@value = N'$EditDate$' --<-- Change this only!
				,	@level0type = N'SCHEMA'
				,	@level0name = N'$SchemaName$'
				,	@level1type = N'PROCEDURE'
				,	@level1name = N'$fileinputname$'
				,	@level2type = NULL
				,	@level2name = NULL;	
GO
-- Template version
EXECUTE sys.sp_addextendedproperty	@name = N'Template version'
				,	@value = N'$TemplateVersion$' --<-- Change this only!
				,	@level0type = N'SCHEMA'
				,	@level0name = N'$SchemaName$'
				,	@level1type = N'PROCEDURE'
				,	@level1name = N'$fileinputname$'
				,	@level2type = NULL
				,	@level2name = NULL;
GO
-- End --------------------------------------------------------------------------------------

Configuring the Stored Procedure Template in Visual Studio

The template can be created in any SQL editor, viz. SSMS or VS.  Once it is created, open Visual Studio and add a Stored Procedure to an existing Project, using the default VS functionality.

VS Add new Item to Project

VS Add new Item to Project

Follow these steps:

  1. Select Stored Procedure and create it by giving it a name that would suit the template Stored Procedure.
  2. Take the newly created template SQL code for Stored Procedure (listed in the code-block above) and paste it over the text of the newly created item.
  3. Save it and then open: File >> Export Template
  4. Select Item Template radio-button from the [Export Template Wizard] and the appropriate project into which the template Stored Procedure has been saved
  5. Click the [Next] button
  6. Select the name of the Stored Procedure just created and saved in the project
  7. There should be no Item References to select so click the [Next] button
  8. Provide a Template Name, description, Icon, Preview Image, and deselect the Automatically import the template into Visual Studio (reference), keep the Display an explorer window on the output files folder selected
  9. Click the [Finish] button, and the output files folder is presented

A zip-file is created that contains three files:

  1. TemplateIcon.ic
  2. MyTemplate.vstemplate
  3. <Stored Procedure>.sql (the template SQL)

The MyTemplate.vstempate file is the one used to configure how the stored procedure is created, i.e. provide the name, description, input parameters, and other functions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
<VSTemplate Version="3.0.0" xmlns="http://schemas.microsoft.com/developer/vstemplate/2005" Type="Item">
  <TemplateData>
    <DefaultName>ACreateStoredProcedure.sql</DefaultName>
    <Name>ACreateStoredProcedure</Name>
    <Description>Default CreateStoredProcedure for Acme Corporation</Description>
    <ProjectType>SQLDB</ProjectType>
    <SortOrder>10</SortOrder>
    <Icon>__TemplateIcon.ico</Icon>
  </TemplateData>
  <TemplateContent>
    <References />
    <ProjectItem ReplaceParameters="true" TargetFileName="$fileinputname$.sql" >Z Create Stored Procedure.sql</ProjectItem>
    <CustomParameters>
	<CustomParameter Name="$projectname$" Value="API"/>
        <CustomParameter Name="$SchemaName$" Value="dbo"/>
	<CustomParameter Name="$ProcParam1$" Value="@InputParameter1"/>
	<CustomParameter Name="$ProcParam1DataType$" Value="INTEGER"/>
	<CustomParameter Name="$ProcParam1Value$" Value="0"/>
	<CustomParameter Name="$ProcParam2$" Value="@InputParameter2"/>
	<CustomParameter Name="$ProcParam2DataType$" Value="INTEGER"/>
	<CustomParameter Name="$ProcParam2Value$" Value="0"/>
	<CustomParameter Name="$ProcParam3$" Value="@OutputParameter"/>
	<CustomParameter Name="$ProcParam3DataType$" Value="INTEGER"/>
	<CustomParameter Name="$ProcParam3Type$" Value="OUTPUT"/>
	<CustomParameter Name="$ItemVersion$" Value="0.0.1"/>
	<CustomParameter Name="$TemplateVersion$" Value="0.0.1"/>
	<CustomParameter Name="$ItemOwner$" Value="c 2015 Acme (Pty) Ltd."/>
	<CustomParameter Name="$OwnerCompany$" Value="Acme (Pty) Ltd."/>
	<CustomParameter Name="$TemplateAuthor$" Value="Hendriekus J.Roux"/>
	<CustomParameter Name="$ItemAuthor$" Value="Hendriekus J.Roux"/>
	<CustomParameter Name="$EditorName$" Value="Hendriekus J.Roux"/>
	<CustomParameter Name="$EditDate$" Value="2015-03-09"/>
    </CustomParameters>
  </TemplateContent>
</VSTemplate>

The listing of the MyTemplate.vstemplate (above) illustrates two parameter functions, viz. system parameter and custom parameters. These are placed in the template SQL code as place-holders that are filled-out either by providing a name in response to an action in Visual Studio like New >> Project or New >> Item where a name is provided by the developer.  Or, the custom parameters are filled-out in correspondence to the values provided in the MyTemplate.vstemplate file.  The latter should be considered in affinity to the functional intension of the Item, I.e. Stored Procedure in this case.

 Present the Template Available in Visual Studio

Visual Studio present a dialog when a new Stored Procedure is created.

VS Create New Item Dialog Option List

VS Create New Item Dialog Option List

The template should be presented here in a section appropriate to the company and project.  To create a new section the template is added to a folder location on the developer’s machine.  The Visual Studio custom templates are located here: <drive>:\Users\<user name>\Documents\Visual Studio 2013\Templates\ItemTemplates.

VS Custom Templates Folder

VS Custom Templates Folder

By adding templates either into these locations or by creating a new folder, say Acme Items, any new custom templates could be placed there and become available from the New Item dialog.

In this example create a folder here called: ItemTemplates\Acme Corporation\Programmability into which the zip-file (<drive>:\Users\<user>\Documents\Visual Studio 2013\My Exported Templates) is copied.

To use the new template, restart Visual Studio and add a new Stored Procedure to the template using the new template.

VS Custom Template New Item Dialog

VS Custom Template New Item Dialog

Unfortunately, at this point, none of the parameters (except the procedure name) is substituted in the actual SQL code.  These, i.e. the custom parametes must yet be added to the MyTemplate.vstemplate file.

Adding Custom Parameters to a Template

Follow these steps to add these parameters and their values:

  1. Extract the template’s zip file from its location into the [My Exported Templates] location and open the  file in Notepad
  2. Take the parameters between the <CustomParameters> tag from the code (above) and add it to the current template’s file in the appropriate location
  3. Save the file and drag the saved version from the explorer window over the zip file in the template location  and choose the overwrite option to up date the template
  4. Restart Visual Studio and create a new Stored Procedure from the template to see all parameters being populated.

Previous Article (3) [Create a Database Project with Folder Structure]. . . Next Article (4) [Coming Soon]

Visual Studio 2013 and Database Projects: Create a Database Project with Folder Structure

Visual Studio 2013 and Database Projects: Create a Database Project with Folder Structure

Overview

Data base projects in Visual Studio is a bit of an after thought on application development.  It started with Gert Drapers (the data dude) and became SSDT (SQL Server Data Tools) that had to be installed (as an add-on) to Visual Studio.  This had some varying function over the years, but was aimed at reducing the difficulty of database development in Visual Studio and keep database changes in synch across a solution (matching database and application versions) and multiple server environments through the application life-cycle.

Creating a Database Project

This is quite easy using Visual Studio 2013.  On the first landing page (can be turned off) a Start section has the option from which to create a new project, or the traditional FILE menu is just as simple.

Visual Studio for Database Development First Screen

Visual Studio for Database Development First Screen

VS Create New Database Project Dialog

VS Create New Database Project Dialog

Be mindful that if a project is created and Team Foundation Server (TFS) is being used to take care of how the solution and projects contained by the solution are demarcated.  It is not in the scope of this article to explain how to do this, but suffice to say that where solutions are stored separately from projects, to first create an empty solution and check it in.  Then the projects that are added back to the solution so that all are checked into TFS in their correct locations.

Further take not of the Visual Studio boxes, viz. Solution Explorer, SQL Server Object Explorer, and Properties for example.  These should be orientated (docked) in positions that make sense, I.e. for easy reference.  Below is an example of an instance that contains a Solution with three projects.

VS Database Project Screen

VS Database Project Screen

The two most important boxes are the Solution Explorer and the SQL Server Object Explorer.  The former depicts the Solutions and the database projects.  Where no database actually exists (this is where you create it) SQL Server Objects can be created/ added to the project, or (where a database already exists) imported by a few means.  The first is using a DACPAC (SQL Server Data-tier application extract file), or connecting directly to a SQL Server database, or create one using a script file containing all the object definitions.

Choosing a Folder Structure

Importing a database allows for the folder structure in the project to be changed.  The project folder structure should allow for ease of maintenance and readability.  The folder structure does not influence deployment.

VS Database Project Import Dialog Folder Structure

VS Database Project Import Dialog Folder Structure

An import of an existing database (where no structure is chosen, or defined dumps all files in the root of the project, which is moderately messy, so it must be carefully considered.  A folder is not created where an imported database does not have objects, e.g. Stored Procedures.  If subsequent objects are required, the folder will first have to be created.

Organising the database advances readability, consequently also maintenance, because developers are able to find things quicker.  Any database objects without any discernable grouping are dumped in the root of the project on import.

NOTE:Things that impact the display of objects and what access rights are differentiated in practice, bears on the configuration of the naming scheme.  Making use of database schemas (to group objects and assign login permissions) is an effective strategy for data architecture over using prefixes on objects like sp or pr for a Stored Procedure (source).  The folder structure in Visual Studio and SQL Server Management Studio already informs the object type and with an added schema denotes an organisational name space for such objects.  SQL Server uses a four-part naming scheme: Server.Database.DatabaseSchema.DatabaseObject.  Objects in a schema are owned by the security principle that owns the schema and a schema can contain objects owned by different users.  Any predefined schema (dbo, sys, etc) can be dropped from the model database if not required consequently preventing them from being created in new databases.  A schema owner can be changed without breaking the code, but if an object owner, say Tommy.Object is removed, an ownership change must first be performed before Tommy’s account can be deleted.  However, any fully qualified references will break in the code, e.g.  Server.Database.[tommy[.DatabaseObject. The default dbo schema is owned by the dbo user. Users assigned to the dbo schema do not inherit dbo user account permissions, but the database objects contained in the schema (source).  Using dbo as the owner of all object can greatly simplify managing objects (source).

Previous article (1) [Installation]. . . Next Article (3) [Working the Project]

Visual Studio 2013 and Database Projects: Installation

Visual Studio 2013 and Database Projects–Installation

This installation is for Visual Studio Premium.  Once the image is mounted, the folder contains the installation file for the install, i.e. vs_premium.

VS Premium Folder

VS Premium Folder

Execute the vs_premium file and agree to the License Terms and Privacy Policy, and the location of the installation. Two options are quite important:

  1. Light Switch
  2. Microsoft SQL Server Data Tools

Light Switch is a technology to shorten the development time on business applications and data services for data-centric business applications.  SQL Server Data Tools are the add-on previously (VS 2012) downloaded and installed separately, but now part of the install.  It provides database schema modelling and support for querying live databases.

The other default option is Blend, a visual user-interface design tool using XAML (web) and WYSIWYG (desktop), before part of the Microsoft Expression Studio toolset.  This is not a database specific tool.

Once installed (it takes a while) the Microsoft Account Login dialog is presented for which you are required to have credentials.  Either use an existing account or create one (reference) and then associate an MSDN subscription (licenses that your company pays for) to this identity. Note: a licence key may also be used if you don’t have an MSDN subscription to activate the software (see how).

Microsoft Account Login

Microsoft Account Login

Once logged in access your account details to find the MSDN section where the Subscriber ID and company email are used to link the MSDN profile to the Microsoft User Identity.

Once the subscription is activated the software is licensed.

MSDN Subscription Activation Dialog

MSDN Subscription Activation Dialog

Visual Studio starts with an environment set-up that can be changed at a later stage also.  Here the Database environment (as that is the purpose of this instance) is selected and the colour scheme of your choice.

VS Environment Set-up Dialog

VS Environment Set-up Dialog

This is the installation process and you have arrived at the first screen.

Visual Studio for Database Development First Screen

Visual Studio for Database Development First Screen

Next article in series. . . 2 [coming soon]

Navigate. . . Next Article (2)

Enterprise Integration

Preamble

Enterprise Integration sets itself to connecting and combining people, processes, systems and technologies in order to converge people, processes, resources and information optimally at the required time of use.  The goal is to reduce waste and establish a continuous improvement effort, in order to render the business as efficient as it possible.  A business environment that uses information systems, the ever increasing urbanisation of systems require a regimen through which business goals can be ensured.  This warrants that the business processes and quality requirements be achieved by its information systems.  An organisation requires that information systems be integrated, the urbanisation of systems be agile, so that the required information be availed in time, as business intelligence, i.e. information that can be used from which to make timely business decisions.  A cycle of continuous improvement must govern the urbanisation in an effort to reduce waste, so that business efforts are optimal.

Information System Urbanisation

Figure 1 Information System Urbanisation

To reach an economic and balanced business system (for the organisation), capable of providing a solution to its requirements without wasting resources, requires that the enterprise architecture be engineered for efficiency. That means selecting the right tools, methodologies and solutions delivery life cycle processes, by which to engineer and govern application, data and information architectures, i.e. a total or complete business operating model through which to achieve its function. The main feature of the enterprise architecture is the information system urbanisation classification that groups and thus separates concerns into a taxonomy by which to identify and group the segregated classes of an architecture, as the landscape is urbanised. This classification, in association with the driving tools, methodologies and business goals, propels the management of the urbanisation in a sustained cycle of continuous improvement.

Problem Statement and Background

Envisioning an enterprise integration approach bears on an understanding of enterprise engineering, which relates to systems engineering that is the application of knowledge and methods of systems engineering to the design of business, as a purposefully designed system.

Therefore, the ultimate reason for conducting enterprise integration is to enable business as an optimal a structure as feasibly as possible in the fulfillment of its goals.  This is done by connecting and combining people, processes, systems and technologies to converge people, processes, resources and information optimally at the required time.  In short, this is to run the business, in aid of technology and human resources, as cost effectively as possible.  This answers to the Total Quality Management optimisation philosophy of W. Edwards Deming, by the reduction of waste, after which (and as a result of) quality increases, cost is reduced, and processes are optimal, resulting in the maximisation of revenue and the resultant profit.

In order to reduce waste the excessive, unused, or surplus resources and effort have to be identified and eliminated from the business operations.  This is done by looking at the production process with the sole objective of reducing waste, i.e. to run it as lean as possible with the greatest possible profit.  It makes the best possible use of raw materials and resources,to produce a product that is sold at the best price.

In an information technology (IT) environment these same principals apply just as much as the engineering concept applies to mechanics.  Therefore, measures used in production engineering may be transposed for use in IT to achieve similar even the same outcome.

Enterprise Engineering, within a financial business, utilising Information Technology solutions to enable the business, has to implement the IT equivalent of the mechanical engineering principles, on the same basis, to optimise its enterprise architecture.

The basis of Enterprise Engineering, transposed from the mechanical engineering principles, speaks of “The creative application of scientific principles to design or develop enterprises, or parts/ aspects thereof; or to construct or operate the same with full cognisance of their design; or to forecast their behaviour under specific operating conditions; all as respects and intended function, economics of operation, and the safety to life and property.” (Enterprise Engineering, 2013).

The function of Enterprise Engineering is to create an Enterprise Architecture within which business goals are realised based on the principle of waste reduction.  Therefore, the Enterprise Architecture is defined as “A coherent set of descriptions, covering a regulations-oriented, design-oriented, and patterns-oriented perspective on an enterprise, which provides indicators and controls that enable the informed governance of the enterprise’s evolution and success.” (Op’t Land, 2009).

This article’s focus is the application integration domain (Application and Data) within the enterprise architecture to make the enterprise engineering evident for these disciplines.

TOGAF and Enterprise Architecture

The TOGAF (TOGAF, 2013) is a framework of tools for developing an enterprise architecture, part of The Open Group for development of open, vendor-neutral IT standards and certifications.  Enterprise Engineering is required to optimise the enterprise, one that is often comprised of legacy and fragmented processes, into an integrated environment that is facilitated by the guidance of the TOGAF framework.  This is to strike a balance between the Information Technology efficiency and business innovation.  These optimisations are categorised into four main categories:

  1. More efficient business operation
  2. More efficient Information Technology operation
  3. Better return on existing investment, reduced risk for future investment
  4. Faster, simpler, and cheaper procurement

TOGAF provides instant guide-lines and artefacts that are industry proven so that an enterprise is not shackled by having to re-invent itself under the burden of having no reference architectures.  This is both costly and time consuming and would, where such reference architectures exist, displace the organisation into a position of innovation-laggard because time would be robbed from innovation by having to re-invent the wheel.

The main goal to achieve through an efficient Enterprise Architecture is that Business is empowered with the required systems and process support to meet their strategic objectives.  This means:

  1. Identifying and refining the requirements that the stakeholders from business have
  2. Developing views of the architecture that show how the concerns and requirements are going to be addressed
  3. Showing the trade-offs that are going to be made in reconciling the potentially conflicting concerns of different stakeholders

TOGAF embraces the ISO/IEC 42010:2007 terminology and uses the Architecture Development Method (ADM) to develop architectures in response to business goals and opportunities in a controlled manner.

Engineering the Enterprise Holistically

Engineering the enterprise into a business goal-achieving, lean operation with optimal profit, requires the optimisation of the business, its systems, processes, and policies.  This is an optimisation spanning across the enterprise, aimed to establish an integrated environment that is responsive to change (agile) and in support of the delivery of the business strategy.  The latter having been identified by the business as Drivers and in the context of the Archimate notation as Motivations for what the business has set itself to achieve.  Achieving strategic goals is the aim of the business in the execution of tis daily operations.  The operations require systems and processes that are used by its employees to conduct the daily business.  But, these activities need to deliver the corporate strategy, i.e. the strategic goals.

In response to these (strategic) business goals or objectives, i.e. what the company wants to achieve, Enterprise Engineering must render the enterprise architecture so as to relate every aspect of its composure to be in support and in aid of the delivery of the business strategy.  The business strategy must correlate to persons, processes, systems, policies, information, and the resulting instrumentation controls to measure how the aforementioned are progressing on the road to realising these.

There are real manifestations of tangible benefits if an organisation is able to clearly set its strategic goals of which the following three find themselves at the top of the list (Dummies, 2015):

  1. Increase operating margins by establishing a direct link for employees between the corporate goals and their ability to contribute to it:
    1. Set SMART goals by which to achieve engagement
    2. Provide up/ down visibility and across reporting levels
    3. Shared responsibility through cascading goals
    4. Communicate expectations clearly through every stage of goal completion
  2. Quicker execution of company strategy
    1. Tighter alignment and cohesion result in quicker execution
    2. Creates visibility on where to focus resources
    3. Exposes redundant business activities so that duplication of effort is avoided
    4. Goal alignment strengthens the company leadership so they avoid redundancy and focus themselves and their subordinates appropriately
  3. Reduce employee turnover
    1. An engage employee is one who stays an employee
    2. Clear goal alignment achieves an engaged employee

Company wide engineering starts with goal-setting.  SMART (Specific, Measurable, Attainable, Relevant, and Timely) goals are intelligent by design and must translate the company’s strategic goals (the five-year plan) down the cascading and empowering hierarchy of the company resources.  In the daily business, technology is employed to achieve these goals and the employee executes their business on systems that make use of business processes and subscribe to company policies.  These efforts must be steered so that the objectives are met and is practically done by monitoring the business activities.  These activities produce data, both from the computer systems as well as the execution and adherence to business processes.  Controls must be instrumented to obtain these metrics (data) and provide this to systems with the ability to convert the data into information that can be used for timely business decisions.  The enterprise architecture must be in support  of this.  Organisational goals won’t be achievable (or partially achievable) if they are not SMART goals, and the enterprise architecture has the ability to collect data and work that into information informing on the progression in meeting them.

In addition, human resources executing business processes must have the relevant processes and policies to adhere to, the systems to use in fulfillment, and the knowledge of what to do and that this knowledge is in support and an enabler for attaining the business objectives.  The entire context you operate in greatly impacts your results (Torben, 2011) and includes the company culture.  The company culture has an overarching influence on the ultimate efficiency of the organisation and neglecting it at the expense of systems and processes will not allow the latter to contribute to the ultimate success of the organisation.  A solid and objective-achieving, high-performance culture is required with the cooperation or enablement through processes and systems to achieve the company objectives optimally.  There are ten general ingredients for creating a high-performance culture:

  • Clearly define the characteristics of winning–what does success look like in the company–a look across departmental disciplines
  • Define the required preferred culture–similar to a vision statement, what should the preferred culture look like, i.e. what are the guiding principles/ values and what do we want to change, and then establish those preferred behaviours
  • Set stretch targets–ones that energise, i.e. are able to be met
  • Describe the future and connect the employees to it.  Set targets and link them to the employees that they are in contact and motivated as contributing to the bigger purpose
  • Develop an ownership mentality–provide freedom to decide and act within the boundaries of employment in an effort to meet the company goals and to think like an owner
  • Improve performance through transparency–increase employee ownership by communicating not only financial but also other performance metrics, like how closely is the affinity of the company’s current performance and the realisation of the strategic objectives?
  • Increase employee engagement and performance
  • Story telling–leaders must use stories to motivate employees and drive company change
  • Internal communication–should be top of the agenda, so that the heard message is being believed, and people know the meaning, and they have internalised it.
  • Celebrate real successes–milestones, and concrete achievements.  Motivation comes from the acknowledgment of an employee’s contributions, in addition to their remuneration and other perks, a time to celebrate together with their peers.
  • Culture eats strategy for breakfast–Plan, monitor, and manage it that it remains aligned to the goals.  Culture eats strategy for breakfast.  The slogan: “you can’t complain here. . .if you see something wrong, you must fix it. Anything can be changed. We aren’t victim to anyone. We own the culture. (Eventbrite)”  “Good culture creates an internal coherence in actions taken by a very diverse group of employees.” (Aulet, 2014).  “I came to see in my time at IBM that culture isn’t just one aspect of the game – it is the game. In the end an organization is nothing more than the collective capacity of its people to create value.” (Lou Gerstner, CEO IBM).

On the last point of how to achieve a high-performance work-force, consider that people are loyal to culture, not to strategy and a resilient culture is more important that new strategies and a corrupted culture will cause a company to collapse when the pressure is on and people will desert the company in such times. The corporate culture should create a competitive differentiation for the company, i.e your culture is your brand and any bad culture could spark a contrarian business strategy.  No publicity can be bought like that of a good culture but a brittle culture can doom even a great organisation.  A complacent culture can blind and organisation to external threats and when strategy and culture collide, culture wins, i.e. cultural miscues are more damaging than strategic ones.  The question that should be asked: “If and when you have a social media crisis, will your people come to your defense or join in the attack?”  Strategies can be copied, but no one can copy your culture and a great culture is the ultimate recruiting tool.  A great culture provides greater discipline than disciplinary action does and how does your culture enforce attitude and behavioural expectations? (Tye, 2013).

Organisational Goal Enablers

Some of the key universal organisational goals are (Ryerson, 2015):

  • Market share
  • Innovation–new products tend to attract sales
  • Productivity–reduce waste
  • Physical and financial resources–renovate the business
  • Profitability
  • Management performance and development–use only the BEST management practices
  • Employee’s performance and attitude goals–invigorate a high-performance culture
  • Social responsibility–bolster the outward repsonsibility of the company’s image

This article constrains itself to three business objectives as universal enablers of organisational goals, viz. Integration, Intelligence, and Agility.

Universal Enablers of Organisational Goals

Figure 3 Universal Enablers of Organisational Goals

These words (Figure 3) represent concepts, that when realised, produces an architecture optimised to enable business goals. The identification of what to engineer and how to get to the ideal state of the enterprise architecture is directed by applying the TOGAF framework to develop it.

The notion of converging people, processes, resources and information optimally, at the required time, so that the result produces an enterprise meeting business objectives by satisfying the business goals of being Agile, integrated, and intelligence capable, affects certain areas in the engineering process.  These areas are depicted in Figure 4 Enterprise Engineering Aspects, and requires a reciprocal entity in TOGAF, as entry point for utilising that framework.

Enterprise Engineering Aspects

Figure 4 Enterprise Engineering Aspects

The deficiency of the model, used to depict the Enterprise Engineering Aspects (Figure 4), is sufficiently distilled into the TOGAF ADM concepts of Information Systems and Business Architecture.  The Business Process and Human Resources Integration are enveloped into the TOGAF principle of Business Architecture and is bolstered by any other business objectives, required to meet the agreed architecture vision.  The Enterprise Application and Data Integration are brought in under Information Integration by being enveloped by the TOGAF principle of Information System Architecture.  Technology Architecture (not being part of the Enterprise Engineering Aspects) is manifested in TOGAF to cover the physical infrastructure, hosting applications, network, etc.

The Architecture Development Method (ADM) is an executable framework.  It uses architecture assets, from a repository, to consider these for use when designing an architecture.  These assets includes reference architectures, models, and patterns accepted for use, within the enterprise, both industry practices and/ or enterprise artefacts.  The ADM is executed repeatedly, over time, and the repository gradually populated with more artefacts.  Any customisation of the ADM, to suit it to enterprise specific needs, is considered an “enterprise-specific” ADM.  Such adaptations may be necessary to adapt differences in phases, or integration with other frameworks.

The focus of this paper is the TOGAF element: Phase C. Information Systems Architecture.

Information Systems Architecture

The objective of the TOGAF element Phase C, i.e. Information Systems Architecture is to develop target architectures covering the Data and Application Systems domains.

The inputs to Information Systems Architecture are:

  • Application principles
  • Data principals
  • Requests for architecture work
  • Statement of architecture work
  • Architecture vision
  • Enterprise continuum
  • Baseline business architecture
  • Target business architecture
  • Baseline applications architecture
  • Target applications architecture
  • Relevant technical requirements applicable to Phase C
  • Gap analysis results from Business Architecture
  • Reusable building blocks from continuum, if available

Each Information Systems Architecture sub-category, viz. Data and Applications Architecture provide guidance architectures for consideration during the execution of the ADM.  These may refer to generic models or patterns or make use of artefact in the Enterprise Continuum.  However, the architecture must support all of the essential information processing needs of the organisation.  The main inputs being application and data principles, the baseline and target architectures with the defined gap analysis.

Data Architecture

Data architecture aims to identify all data sources in support of business to be comprehensible, consistent and complete, and stable.  It is not concerned with the design and layout of storage systems, but rather sets the standards for all systems as a vision of the target state interactions between systems.  Data integration denotes the exchanges of data between two or more data systems.  Data architecture describes (in order to reach the target state) how data is processed, stored, and utilised in an information system.

In the data architecture model, either a generic or business continuum reference can be made, as a data resource, related to the specific business’ vertical industry sector.  The gap analysis, in response the architecture vision of the business goals for realisation, addresses stakeholder concerns about data access and use deficiencies in the current architecture.  The main goal of the data architect is to produce a path to the target (ideal) data architecture.

Application Architecture

“Software architecture encompasses the set of significant decisions about the organisation of a software system including the selection of the structural elements and their interfaces by which the system is composed; behaviour as specified in collaboration among those elements; composition of these structural and behavioural elements into larger subsystems; and an architectural style that guides this organisation.  Software architecture also involves functionality, usability, resilience, performance, reuse, comprehensibility, economy, and technology constraints, trade-offs, and aesthetic concerns.” (Philippe Kruchten, 2009).

Application Architecture should (MSDN, 2009):

  • Expose system’s structure, but hide implementation details
  • Realise all use-cases and scenarios
  • Endeavour addressing stakeholder requirements
  • Deal with functional and quality requirements

Along with data architecture, the concepts of a reference architecture, potentially comprising an vertical industry sector generic model, or industry standard reference patterns and objects, and an enterprise continuum, is used as reference (guidance) for developing the target architecture, as guided by the gap analysis.

Enterprise Integration

The Application Architecture’s objective is to define the kinds of applications (not design them) that are relevant to the enterprise, and what those applications need to do in order to manage data and to represent information to the human and computer actors in the enterprise.  Applications are defined as groups of capabilities that manage data objects (in the data architecture) and support the business functions in the business architecture, without making mention to a specific technology.  This is because the application is considered stable, while technology frequently changes over time.

Architecture Change Paradigm

Figure 5 Architecture Change Paradigm

Understanding that the realisation of business objectives undergirds the architecture vision for the existing landscape, altering it at the hand of a gap analysis into the ideal state or target architecture, is the basis for enterprise engineering. The architectural vision is realised by building upon industry proven data and application principles. These may have been assimilated into the business continuum repository, from where they are referenced and applied when building solutions.

The architecture vision is thus realised, at the hand of the TOGAF ADM, specifically in the focus of this article, on application and data architectures by realising the business objectives through the architecture vision. The gap analysis, between the baseline and target architectures, is the scope of change and the change of the architecture uses the data and application principals as modelling references.

The Business Objectives being the points of motion for the Architecture Vision, driving change to the architecture, having been condensed to three words, viz. Agility, Integration, and Intelligence, also suiting the phrase: “to converge people, processes, resources and information optimally at the required time” are clarified in context with the greater beneficial attributes to be realised, in Table 1 Business Objectives.

“The performance of the enterprise depends on the efficiency of its information system. . . in order to ensure its survival and its sustainability, it must be agile permanently; . . the enterprise must be driven according to a governance framework, relying on an appropriate set of best practices and/ or standards.” (Rabah Imache, 2012).

Architecture Vision Embodiment of Business Objectives

Figure 6 Architecture Vision Embodiment of Business Objectives

Table 1 Business Objectives
Agility Economy of Scale Information
Versatility Reuse Business Decisions
Adaptability Interconnectivity Data Transformation
Modularity Waste/ Cost Reduction Competitive Intelligence
Service Orientation Consolidation Timely Delivery
Flexible Holistic Exploitation
Lean Cooperative Data Integration
Changing Conditions Lean Encapsulation/ Abstraction
 Fast Change Encapsulation/ Abstraction Ease of integration

The answer to the Business Objectives attainment to the Total Quality Management principle of cost reduction through the reduction of waste is reached through adherence to Agility, Integration, and Intelligence, as expressed by the attributes of each concept listed in Table 1. Any envisaged architecture has to be engineered in accordance with these objectives. The objective is to converge any existing heterogeneous, autonomous and distributed system into an imperatively flexible and cooperative system that works together to achieve business objectives. Agility may be viewed as (Moe, Ågerfalk, & Smite, 2010):
• Technology agility
• Process agility
• People agility
• Structure agility

Information System Urbanisation

System complexity makes enterprise integration challenging, and it can be compares to human cities or urban systems, being urbanised.  Consequently the urbanisation of information systems is a trigger for enhancing agility. Information System Urbanisation is seen as the engine of evolution in a business, with the key to business success being the effectual exploitation of information.  Urbanisation addresses the need of evolution of Information Systems by providing a strategic context for the evolution of the system in response to the constantly changing needs of the business environment. (Trabelsi & Abid, 2013).

Urbanisation and Alignment

Figure 7 Urbanisation and Alignment

Continuous Improvement is another Total Quality Management concept, coined by Dr. W. Edwards Deming, being punted as the basis for achieving Information System agility regulation and preservation.  Improvement must be permanent.  Continuous improvement is an endless project that aims to take into account failures and strategy orientation changes as early as possible with agile practices.

Information System Governance allows for (Rabah Imache, 2012):

  • Better decision making to increase efficiency
  • Clarification of the roles of different actors, for rights and duty clarification
  • Better understanding of key processes in relation to Information Systems

An Enterprise Integration approach should therefore endeavour to leverage agility through the continuous improvement of Information System Urbanisation, promulgated from an ever-changing environment, and the natural aging of systems.

Urbanisation Plan

An Information System’s complexity is vested in both the structural and systemic aspects; structural because of the software applications, and systemic as a result of the interaction between its parts.  Understanding all the states of software applications, lead to maintenance and evolution problems.  Inflexibility in systems are brought about by the “chaotic” manner in which they were built by individual organisational units, leading to overly complex, expensive, and non-integrated systems. (Guetat, 2013).

nformation System Urbanisation Plan

Figure 8 Information System Urbanisation Plan

The Information System Urbanisation Plan (Guetat, 2013) (Figure 8) applies structure for the classification of information system applications, functions, or services in a coherent manner. Information System Urbanisation has a few primary constraints (Guetat, 2013):
• Strong coherence and weak coupling
• Separation of concerns
• Standard communication protocols
• Data hiding

Inbound/ Outbound flows management area

This area handles all the business’ external environment processes, dedicated to the management of information flows with external parties.

Party Relationship Area

This area links the business and its customers through the particular communication channel.

Integration and Shared Information Area

The Integration Area allows for the information flows and services between the back and front office applications, while the Shared Information Area contains information shared by all the applications of the organisation’s information systems, e.g. customer and product repositories.

Back-office

The back-office maintains areas for Business Intelligence, for decision making, and Support processes, and each Business Area, which is a functional aspect of the business such as Equity Trading.

Architectural layering of an Urbanisation Plan

An urbanised application therefore should be comprised of a set of parts that have public and private resources, communicating through a standard communication protocol.  The architecture is separated into functional layers, e.g. Interface, Navigation, Orchestration and Choreography, Services, Data Access, and the Technical Services layer.   The contractual boundaries between layers allow a standardised data-set, though modifiable parameters, stored in the layer’s repository, to pass the boundary.  These layers honour the Information System Urbanisation Plan (Guetat, 2013) (Figure 8) but specialises the architecture into a separation of concerns.

Figure 9 Urbanised Architecture Model Layers (Guetat, 2013)
Layer Role Support Functions Layer Repository Examples Architecture Rule Examples
Interface Mange end-user interaction for each technical communication channel
Mange graphical perspective of human interface
Components: screens, editions, formatting elements
Presentation management
Screen content display
Syntax control of data
Surface control of input data
Online help
Display colour
Message lables
Screens associated with language
Only Interface layer modules can interact with end-users
Navigation Description of the progress of the screens kinematics
Management of data specific to the interaction between the software system and its human end-users
Management of context related to the informational flows exchanged with the Orchestrations and Choreography layer
Components: technical communication channels kinematics, e.g. Internet, Mainframe, etc.
Identification of screens for performing tasks
Calls to the Orchestration and Choreography layer modules for control actions in the organisational process supported by software system
Routing of displayed information to local printers
List of screens associated with task Only Navigation layer modules can call Orchestration layer modules
Orchestration and Choreography Identification of the organisational process activities supported by the software system
Management of sequence of tasks, supported by the software system
Description of end-user roles
Control of informational and service exchanges with other systems
Management of a context related to task running, in order to allow interruptions, without data publication.
Start and complete a sequence of tasks
Expose services to other software systems
Call of services exposed by other software systems
Sequence of services invocation
Service orchestration
Service choreography
List of tasks making up a use-case
List of services used by a use-case
Description of the sequence of tasks making up a use case
Business data are managed in a process context, during process execution, which is managed exclusively by a module of the Orchestration and Choreography layer. No module of another layer can access, even read this information.
Only models of the Orchestration and Choreography layer can exchange informational flows and services with other software systems
Only the Orchestration and Choreography layer can expose services for other software systems.
Services Hosting the rules, applicable to the software system informational entities.
Implementation of the functions processing the software system informational entities
Description of the state of the software system informational entities
Carrying out simple and complex controls
Data processing
Description of services supported by the software system
Recording of status changes of the software system informational entities
Computing rules
Information related to data processing interest rates, legal, etc.
The Services layer guarantees the inter-business process consistency through checking the software system information entities.
Data Access Providing access to operational persistent data belonging to the software system, ensuring a real independence between processing and physical data models and performing data integrity controls. Data selection, update, delete, create, edit, join, and integrity control List of tables to be used to store information related to the an informational entity Only modules of the Data Access layer provide data read/ write services of operational persisted data belonging to the software system

Read and write services exposed by the Data Access layer are defined according to the logical data model of the software system.

Urbanisation Information Systems and Enterprise Integration

The goal of an urbanisation project is to produce a map of the information system’s structure that can be used to improve its performance and its development, it also helps to give the company the resources to develop its information systems in full knowledge of the facts. (Urbanization of the IS (information system), 2013).  “Urbanization is to organize the gradual and continuous transformation of an information system to simplify it, to optimize its added value and to make it more responsive and flexible towards strategic business changes, while relying on technological opportunities of the market.” (Amel Boussis, 2013).

There are a few methods of integration, viz Vertical, Star, Horizontal, and Common Data Format.  The integration of object orientated systems (object types) is divided in structure and behaviour based integration.  Structure based integration puts emphasis on the integration of static schemas and heterogeneous data structures, i.e. object types and their attributes.  Behaviour based integration is the integration of the methods implemented by object types and the order of their execution. (Georg Grossmann, 2009).

The process of urbanisation is based on three key areas that feed each other (Amel Boussis, 2013):

  • Modelling Strategy
  • Mapping of existing systems
  • Determination of target systems

The process of urbanisation of the IS includes other (Amel Boussis, 2013):

  • Set the target IS, aligned to business strategy
  • Determine the path to follow to achieve this target IS

Vertical Integration

Vertical Integration is where local systems link to higher level systems within similar functionality. (Kevin Grant, 2010).  This integration type encompasses all the work-flow activities in the production value chain of the process that the application has to complete, in order to render a product. This can be equated to a production process spanning all tasks from the order to the sale.  In software terms, from an online order, purchase, payment, billing, clearing, goods dispatch, stock control, accounting, etc. (Rolf T. Wigand, 2003).  The objective of vertical integration is to gain efficiency in supply chains. (David L. Olson, 2010).

Horizontal Integration

Horizontal Integration are systems integrated across different functions, as a one-stop shop. (Kevin Grant, 2010).

Spaghetti/ Star Integration Methods

Spaghetti or Star Integration

Figure 10 Spaghetti or Star Integration

This type of integration has an exponentially rising cost, as more system are added to the architecture, due to the complexity of the integration.

Common Data Format

This integration method avoids the multiplicity of data adapters between products, but proposes a common, application-independent (common) data format.  The common format is achieved through an arbitrary data transformation service that translates source data to a common format, for use in applications.

Application Integration

“Application Integration is the secure and orchestrated sharing of processes and/ or data between applications within the enterprise.” (Hojaji, 2012).  It refers to middle-ware as a collection of technologies and services to integrate systems and applications across the enterprise to the end of simplifying and automating business processes without making radical changes to existing structures.  Gartner expresses this notion as: the “unrestricted sharing of data and business processes among any connected application or data sources in the enterprise” (Kress, 2009).

A point-to-point (star/ spaghetti) integration pattern results where integration is applied without following a structured Enterprise Application Integration approach.  Applications are added to the architecture, in the urbanisation of the information system, on an impromptu basis, resulting in complex inter dependencies.  An application does not only share data, but enables business processes.  Therefore, business rules may be extracted, along with data integration, as sub-components of application integration, in order to modularise the architecture, on the back-end.  The different application user-interfaces (front-end) are unified into a single and consistent interface by which application functions are accessed, providing users with a single-faceted user-experience to a kaleidoscope of business functions.  This is known as the Federated pattern, in contrast to a Mediation pattern that acts as an integration bus, sending updates to the relevant and dependent applications of the architecture.

These integration approaches make use of two basic topologies, viz. hub and spoke and the bus. In the hub-and-spoke topology, connection points are fewer than spaghetti integration, with the integration point common (the hub) between the applications along the spokes.  Such a hub requires special resource and communication considerations because all payloads pass through it.  However, the hub allows for complex business logic to be centralised, as opposed to being executed by all the satellite applications.  The service bus allows the communication between mutually interacting applications in a services orientated architecture of heterogeneous and complex situations.  The main purpose of a bus is routing, congestion mitigation, control over services, data transformations, mappings, security and exception handling.  An enterprise service bus finds itself as an abstraction to a message system, comprised of modular components.  This reduces point-to-point connections, but all clients direct their requests through the bus, instead of a direct route to the other application.  Most of a service bus’ services are directed at dealing with directing and marshaling the routing of messages, as well as common data transformation.

Data Integration

By virtue of the word “data” this integration type concerns itself with providing a unified view of business data.  This concept is further extrapolated by enterprise information integration, to include all data and information.  The goal is to represent heterogeneous data, to the user of a system, as a single, homogeneous data source, known as the “uniform information representation”. (Lans, 2012).  Refer to “Data Virtualisation: The Fit” (Drikus Roux, 2015).

Works Cited

Amel Boussis, F. N. (2013). Urbanization of Information Systems with a Service Oriented Architecture according to the PRAXEME Approach. Algiers: National School of Computer Science.

Analyst, M. (2010, 02 15). Modern Analyst. Retrieved 02 15, 2010, from Modern Analyst: http://www.modernanalyst.com/

Aulet, B. (2014, April 12). Culture Eats Strategy For Breakfast. Retrieved from TechCrunch: http://techcrunch.com/2014/04/12/culture-eats-strategy-for-breakfast/

David L. Olson, S. K. (2010). Enterprise Information Systems. Danvers: Stallion Press.

Dummies, B. E. (2015, February 25). The Incredible Power of Company-Wide Goal Alignment & Organizational Business Goals. Retrieved from Success Factors: https://www.successfactors.com/en_us/lp/articles/corporate-goal-alignment.html

Enterprise Engineering. (2013, 05 08). Retrieved from Enterprise Engineering Team: http://www.ee-team.eu/focus/enterprise-engineering

Georg Grossmann, M. S. (2009). Exploiting Semantics of Inter-Process Dependencies to Instantiate Predefined Integration Patterns. Adelaide: University of South Australia.

Guetat, S. B. (2013). A multi-layered software architecture model for building software solution in an urbanised information system. Le Mans University, ARGUMANS Lab. Le mans: Le Mans University. Retrieved May 09, 2013, from http://www.sciencesphere.org/ijispm

Hojaji, F. (2012). Five Aspects of Application Integration Requirements. Amirkabir University of Technology, Computer Engineering & IT Department. Tehran: Amirkabir University of Technology. Retrieved May 13, 2013, from http://scientific-journals.org/journalofsystemsandsoftware/archive/vol2no3/vol2no3_2.pdf

Kevin Grant, R. H. (2010). Strategic Information Systems Management. Andover, Hampshire, United Kingdom: Thomas Rennie. Retrieved May 09, 2013, from http://books.google.co.za/books?id=-XAc6gUfj8oC&printsec=frontcover#v=onepage&q&f=false

Kress, M. (2009). Intelligent Business Process Optimization for the Service Industry. Karlsruhe, Germany: Scientific Publishing. Retrieved May 13, 2013, from http://books.google.co.za/books?id=kowlIvgbEiIC&printsec=frontcover#v=onepage&q&f=false

Lans, R. L. (2012). Data Virtualization for Business Intelligence Systems. Amsterdam: Morgan Kaufman.

Moe, N. B., Ågerfalk, P. J., & Smite, D. (2010). Agility Across Time and SPace. Berlin, Germany: Springer-Verlag. Retrieved May 09, 2013, from http://books.google.co.za/books?id=sTfWY6B7OHkC&printsec=frontcover#v=onepage&q&f=false

MSDN. (2009). Chapter 1: What is Software Architecture? Retrieved from MSDN.

Op’t Land, M. P. (2009). Enterprise Architecture. Springer.

Philippe Kruchten, G. B. (2009). Chapter 1: What is Software Architecture? Retrieved from MSDN: http://msdn.microsoft.com/en-us/library/ee658098.aspx

Rabah Imache, S. I.-N. (2012). An Enterprise Information System Agility Assessment Model. University of Boumerdes, Faculty of Sciences, Computer Science Department. Boumerdes: University of Boumerdes. doi:10.2298/CSIS101110041I

Rolf T. Wigand, P. M. (2003). Introduction to Business Information Systems. New York: Springer. Retrieved May 09 , 2013, from http://books.google.co.za/books?id=u-1rLZ_WqTwC&printsec=frontcover#v=onepage&q&f=false

Ryerson. (2015, February 25). Organizational Goals. Retrieved from Ryerson: http://www.ryerson.ca/~meinhard/841notes/goals.html

TOGAF. (2013, May 08). Retrieved from The Open Group: http://pubs.opengroup.org/architecture/togaf9-doc/arch/

Torben, R. (2011, April 10). TOP 10+ KEY ELEMENTS IN CREATING A HIGH PERFORMANCE CULTURE. Retrieved from Meliorate: http://www.torbenrick.eu/blog/performance-management/10-key-elements-in-creating-a-high-performance-culture/

Trabelsi, L., & Abid, I. H. (2013). Urbanization of Information Systems as a Trigger for enhancing Agility. Business School IAE de Nice. Nice: European Journal of Business and Management. Retrieved May 09, 2013, from http://www.iiste.org

Tye, J. (2013, September 07). 12 Reasons Culture Eats Strategy for Lunch. Retrieved from Slide Share: http://www.slideshare.net/joetye/12-reasons-culture-eats-strategy-for-lunch-25988674

Urbanization of the IS (information system). (2013, May). Retrieved from Kioskea.net: http://en.kioskea.net/contents/657-urbanization-of-the-is-information-system

Notes on Making use of eProcurement

eProcurement Use Cases Itemised

  • Online retailing—unless it is fully integrated into the supply chain—is simply a mildly effective extension of the sales process
  • The real value of the Internet comes from business-to-business buyer-vendor transactions that include electronic procurement and full integration of the electronic supply chain from customer to supplier.
  • What is the company’s e-business strategy?
  • Business benefits come not only from the functionality of the software system, but often more importantly, from changes in the way employees do their work
  • E-procurement is the central function between electronic buying and selling between businesses, i.e.
    • Business-to-business e-procurement (B2B).
    • Business-to-business e-procurement (B2B) has the biggest potential for cost savings and business improvement than online retailing or enterprise resource and planning (ERP) systems.
  • Organisations are finding that the more they can integrate e-procurement processes and systems directly into their supply chain, the greater the effect in terms of cost savings and process improvement.
  • Research indicates that 40% of the total cost of purchasing materials comes from transaction costs associated with processing and managing the order.
  • It was only with the shift from client-server to Web-based technologies that a true consolidation of data—a single, extended enterprise-system approach—became possible.
  • Part of the problem is simply that for most companies the procurement process is still seen as tactical rather than strategic—as a cost rather than a benefit—to the company.
  • A recent study by Net Profit in Europe found that although there was a very high level of interest in e-procurement among European companies—58% were considering an Internet-based purchasing system and 75% of respondents thought that the Internet would be “very” or “extremely” important for procurement in three years’ time—only 14% of companies had actually installed systems.
  • More often, companies complain that although they have access to the Internet, there remains such a complete separation between the Internet and any of their company’s legacy systems, that even when they select products from online catalogues, they are forced to work through the same old manual, paper-based requisition and purchase order process.
  • Many company leaders are still justifiably concerned about the daunting prospect of integrating multiple systems—demand planning, ERP materials management, payment systems, and so on—particularly in the context of ever changing major software platforms and e-procurement offerings.
  • The most often-cited reason for not moving immediately into e-procurement has been concerns over security and trust
  • It is worth noting that, for one thing, the return on investment for e-procurement tends to be much greater than any of the business improvement revolutions—business process re-engineering, strategic sourcing, activity-based costing, ERP—that have preoccupied organizations for the past decade.
  • The Center for Advanced Purchasing Studies estimates that the average company tends to spend 40% of its revenue on non-production purchasing, depending, of course, on the industry.
  • A recent study by Group Trade concluded that although leveraging your buying power through online e-markets might reduce the price paid by 10% to 20%, the real value to the company—up to 70% of total savings—comes from savings realized through reducing transaction costs.
  • Without accurate, real-time decision support tools that can help a company to understand the underlying costs of the process—transaction costs, vendor delivery or quality issues—firms (and individual departments) far too often focus exclusively on price, many times at the expense of broader “value creation” opportunities.
  • These savings are best measured in terms of transaction costs, which include things such as lost discounts and “maverick” buying, hours spent in supplier relationship management, time and labor hours spent in the paper-based order generation, requisitioning and approval process, quality assurance, returns, capital cost of warehouses’ excess inventory, carrying costs that come as a result of disconnected replenishment channels, shop-floor downtime from items needed but missing, and so on.
  • Because of the nature of an all-inclusive e-procurement system, one of the most important benefits to be realized with automation of the procurement process is that, for the first time, an organization will be able to accurately track costs for labor, error reconciliation, lost orders, maverick buying, and delays in the process.
  • Attempts by the CFO to reduce inefficiency through cost-based accounting and departmental-based incentive programs often create conflicting, silo-based priorities, disrupting the horizontal supply chain process flow, forcing managers back into positions where their department budget becomes more important than overall efficiency or cost of production to the firm.
  • With Enterprise Resource Planning (ERP), Advanced Planning and Scheduling (APS), and independent supply chain software vendors moving quickly into alliances that will help extend a buyer’s supply chain back into their supplier’s systems, by all accounts, e-procurement will be a major leap toward achieving the—until now, primarily theoretical—“pull” type of relationship that manufacturers would like to have with their suppliers, with vendors able to electronically and automatically “see” through the entire supply chain and take responsibility for inventory and JIT delivery of production materials.
  • The inter-vision ability of e-procurement between supplier products, prices, availability, delivery, and payment of supplier is readily available and payments can be made electronically.
  • Approvals for standard supplies are automatic, and exceptions are immediately routed through a list of approval deputies, ensuring that if no response is received in a matter of minutes, the approval is routed quickly to another pre-named deputy. There is a complete audit trail of the request, price, approval, and payment information, and transaction information is captured and recorded for vendor performance analysis. There is an instant update for other interested parties—accounts payable and receiving—and most transactions will take only a few minutes. All purchase order and delivery details are available online for both the supplier and the buyer to see.
  • In its most perfect state, the simultaneous availability of data to all the parties in the extended supply chain means that—with the elimination of uncertainty, people, and paper—transaction time will be reduced to very little more than the actual time it takes to physically transport the materials.
  • Enterprise Resource and Planning (ERP) systems have improved internal company processes and e-Procurement should improve external B2B processes by first being a strategy and then a technology. The automation of the procurement process should have a compelling value proposition for corporate officers.
  • The e-procurement initiative provides an invaluable opportunity for organizations to restructure themselves and their partner relationships in ways that are fundamental and were never before possible.
  • To create a fully collaborative supply chain, a firm needs to tightly integrate a suite of products that extends from sales force automation through to forecasting tools, the ERP backbone, the online e-procurement system, and ultimately, into the suppliers’ systems.
  • Because customers both initiate and drive their own purchases through the entire supply chain, and because instant or customized sales require customer service agents to be able to provide real-time availability and delivery information to customers, CRM systems are now being seen as critical to the full integration of the e-procurement process.
  • Many industry observers continue to point to Microsoft as an up-and-coming key member of the e-procurement community of vendors, not only because of its MS Market online procurement suite, but also because, for many customers, compatibility with the Microsoft business software platforms is essential in order to provide a procurement functionality directly to the employee desktop.
  • True business efficiency only comes from seamless inter-connectivity with back-end systems, and therefore the first phase of an e-procurement project may be—particularly for mission-critical direct inventory and supply chain—to create the internal links with the ERP systems.
  • There is expertise needed for e-procurement software implementation, but also for strategy, procurement, leading practices, business process redesign, project management, legacy systems integration, and change management.  Too often, companies believe that they can use their own people to re-engineer processes and implement organizational change but although their input is valuable, it is essential to get new ideas, and these almost invariably come from bench-marking or lessons learned brought in by a third-party specialist.

Fundamentals of eProcurement

Procurement is comprised of direct (resulting in a finished product) and indirect (not resulting in a tangible product) materials

  • Direct
    • Manufactured product
  • Indirect
    • Office products and travel services (ORM—Operating Resource Management)
    • Replacement parts (MRO—Maintenance, Repair, and Operations)

All companies purchase large amounts of indirect goods, i.e. 40-60% of the company revenue.  Products can be further classified (2) as standardised (homogenous products between different suppliers) and differentiated.  The latter being unique enough not to attract price competition with the former under fierce prices scrutiny.

Product Suitibility for eProcurement

Figure 1–Product Suitability for eProcurement

There are four influencing factors that contribute to a product’s susceptibility to e-procurement (2):

  1. Function within the business—indirect goods
  2. Product characteristics—standardised products from a catalogue
  3. Product value—items of lower value require less negotiation
  4. Number of suppliers—a larger supplier base achieves faster critical mass than a smaller one

Two key-cost areas for indirect procurement are:

  1. Inefficiency and labour-intensity of the process
    1. Non-standardised policies with disparities between departments (decentralised procurement model)
    2. Cumbersome approval processes causing delays
  2. Cost
    1. Maverick spending
    2. Discount neglect
    3. Non-standard vendor management and contracting

Procurement typically iterates the following steps (also see Figure 1):

  • Select goods – shop/ catalogue/ internet
  • Complete the Requisition Process of the company
  • Wait for approval
  • Create Purchase Order
  • Receive goods
  • Invoice
  • Pay
The Traditional Procurement Process

Figure 2–The Traditional Procurement Process

The Strategic Nature of eProcurement

Procurement is not a back-office tactical operation but is a critical part of the business and consequently of the company’s overall e-business strategy.  E-Business is the use of the Internet for computing and communications to execute both front and back-end business processes.  E-business is a key supply chain integration enabler and e-procurement is a part of it.  E-business impacts the supply chain integration in four key dimensions (2):

  1. Information integration—real time transparency across supply chain
  2. Synchronised planning—collaboration in planning and replenishing across supply chain
  3. Workflow coordination—automate business processes
  4. New business models—supply and sell-side models not previously available in the paper world

Savings have usually been made through discount negotiations or bulk buying; but even these efforts are often offset by lack of choice and excess carrying costs.  Although the potential effects of e-procurement are significant, it is important to remember that driving inefficiency out of the supply chain is part of an ongoing technical and structural revolution, and that as unstoppable as it is, it is not all that revolutionary. In fact, it is probably worthwhile to point out that this “evolutionary” shift has been taking place for the past several decades, and is manifested in several important ways.

First, e-procurement systems continue the trend toward automation of the process and the replacement of human labour through information technology (IT). E-procurement systems automate the requisitioning, approval, shipping, receiving process, and payment systems, and provides for automated routing and tracking capabilities, essentially eliminating the need for human intervention other than on an exceptions-only basis. Second, e-procurement continues to enhance the breakdown of traditional vertical silos and to shift management’s focus toward horizontal processes and the empowerment of individual employees. In the past, the purchasing process was seen as a set of separate activities and functions, controlled centrally or departmentally, often focused narrowly on silo-based incentives rather than on total cost.

For indirect and MRO materials, e-procurement systems allow for a far greater level of individual empowerment as pre-approved and fully audit-able purchasing is handed over to individual employees.  As an integrated set of processes and systems that create a seamless, electronically initiated and monitored exchange of information, goods, services, and payment between buying and selling organizations of all types, e-procurement is integral to a process that can, and should, stretch from the planning and forecasting function of the buying company all the way through to the delivery and payment functions of the selling organization, regardless of the product or service, and whether direct or indirect.

Without a comprehensive system for collecting key corporate information, all other systems are limited in their scale to piecemeal and departmental functions.  However ERP systems (historically) largely could not provide usable information to business by which to seamlessly extend their supply chain through the purchasing of materials and resources through a supplier network.  The Internet, B2B e-commerce, and especially e-procurement systems come and step into this breach.

E-procurement is not simply purchasing done electronically but is a more encompassing where key procurement information—in the form of item masters, purchase orders, product catalogues, inventory lists, and even payments—can be digitised and distribute simultaneously and instantly to multiple parties electronically, regardless of location.

The Focus of E-Procurement

Figure 3–The Focus of E-Procurement

eProcurement Business Case

Costs and Savings

Two costs make up procurement cost:

  1. Price of the goods
  2. Cost of the acquisition process

Any procurement optimisation should focus on:

  • Process efficiency
  • Compliance
  • Leverage

Savings in procurement are:

  1. Savings that come from automating the process, eliminating paperwork and human intervention, and reducing transaction costs and cycle time.
  2. Structural than procedural, and comes from shifting the selection and ordering process back to the employees’ desktop, eliminating the multiple purchasing middlemen now involved in everyday indirect goods procurement, and giving the individual employee the choice—and responsibility—for purchasing goods.

Process Automation

Automating procurement encompass the following:

  1. The entire procurement process—budgeting, requisitioning, ordering, approval, purchase order development, payment, and delivery—should be completed electronically and, as much as possible, simultaneously, so that there is a minimum of manual intervention or delay.
  2. The entire process becomes “re-balanced” so that the hundreds of thousands of ordinary and uncontroversial purchases take place with the minimum of supervision or human intervention, and only “exceptions” are flagged through exception reports for the attention of procurement specialists or management.

The biggest external time-gobbler and cost is approval routing.  Any e-procurement system should provide an automated approval workflow, customised around the particular parameters of the business, e.g. selecting a next approver (business continuity) should a first neglect to work on the approval on the approving manager’s computer screen.  Another is Order Management that has to be automated, for ordering, shipping, and reordering, and receiving and invoice approval functions.  The reconciliation between requisitions and purchase orders (tracked in real-time) must be done automatically to remove tedious manual rationalisation with a paper trail.  The need to provide end-to-end management of resources, once purchased, and to monitor and anticipate the need for replacement has recently driven many software platforms to include asset and cost management modules.

Purchasing often debates a centralised and de-centralised purchasing model, but the obvious effect of providing a universally accessible purchasing system for those responsible for initialising purchasing transactions is shifting the responsibility from the purchasing department into the users in the front-line who purchase via e-procurement on their individual computing devices to free centralised purchasing specialists for more strategic work.  This self-service requires a great deal of change management in re-thinking rules and process to change behaviour; but is ultimately the key to eliminating redundant processes and extra-policy buying.

One of the most important advantages that come from an e-procurement infrastructure is that robust new reporting and decision support tools now help procurement specialists to scrutinize their buying patterns, providing more dependable information on performance, compliance, and the effectiveness of comparative buying practices or supplier selection.

Strategic Sourcing

Strategic souring, through the use of business analytics, costs, performance measurements, and comparisons are made accurate and in real time, with an unambiguous audit trail, to realise the benefits of strategic sourcing, i.e. which ones are most effective.  This information enables that procurement specialists are freed-up to make better decisions of the supplier financial or delivery performance to negotiate contracts and change procurement practices.

Procurement Specialist

A key change introduction of e-procurement is the amalgamation of the traditional centralised and decentralised purchasing models.  Any procurement role that deals with a paper trail should be terminated.  New, e-procurement roles should be:

  • Electronic catalogue and content manager
  • Internal purchasing policy development
  • Compliance
  • Super user decision support
  • Vendor management and service-level negotiations

A big requirement internally is to have a person with good client management skills to ensure good communication and purchasing policy effectiveness throughout the organisation.  A strong decision support (super user) should master sophisticated decision support tools to advise on the buying criteria.  Monitoring compliance from e-procurement systems is making procurement specialists in different departments view purchasing as a corporate rather than a unit function.

The e-Procurement Software Landscape

E-procurement came from the pre-cursor called Electronic Data Interchange (EDI) that is a dedicated data line between trading organisations sending and receiving procurement data.  Modern e-procurement started with electronic product catalogues but was plagued with a lack of standards.

Procurement involve at least two parties, the buyer and seller and other role players, viz. initiator, the user (of the product), influencers, deciders and a buyer.  Supplier side role-players depend on the channel strategy chosen, e.g. direct or indirect product channels, with the latter utilising intermediaries to sell their products.  The Internet might dis-intermediate these sales channels by providing direct links but they may also be replaced by info-mediaries, i.e. systems providing information rather than dictating purchases (2).

South Africa has unique problems with e-business and e-procurement.  It has a limited and monopolised supply base but requires an online community with large numbers of participants to be successful.  It also has a small supply base per commodity group, controlled by a few large participants that reduces the effectiveness of an online solution such as e-procurement. This technology also being an expensive options that only large organisations can afford, hence an implementation is usually not integrated with supplier systems and the advantages of a totally integrated solution are seldom realised (2).

To create a fully collaborative supply chain, a firm needs to tightly integrate a suite of products that extends from sales force automation through to forecasting tools, the ERP backbone, the online e-procurement system, and ultimately, into the suppliers’ systems (1).

An e-procurement initiative is more than just a system; it is an entirely new way of working and requires dramatic changes to business processes and to ways of thinking and behaving.  There is greater audit visibility on spending and changes to staffing roles.  Many e-procurement initiatives fail because long-term company-wide change management is broader in its focus than project success and involves many of the long-term issues that result in a successful business transformation (1).  Consequently, executive involvement and endorsement, the involvement of all key stakeholders, internal and from suppliers and partners are required to ensure success.  The following ground rules are to be observed for an e-procurement project:

  1. Ensure executive sponsorship and participation
  2. Make the e-procurement project integral to the e-business strategy of the company
  3. Build a legitimate business case
  4. Set the guiding principles before initiating the project
  5. Develop a strong change management program
  6. Re-engineer the business processes before selecting software[1]

[1] “Automating a mess yields an automated mess”

References

  1. Neef, Dale. e-Procurement From Strategy to Implementaion. New Jersey : Prentice Hall, 2001. ISBN 0-13-091411-8.
  2. A FRAMEWORK FOR THE IMPLEMENTATION OF E-PROCUREMENT. Jooste, M V and van Schoor, C de W. 1-22, Pretoria : SA Journal of Industrial Engineering 2003, 2003, Vol. 14(2).

Data Virtualisation: The Fit

1      What and how relevant is it?

1.1    Introduction

Managing an organisation’s data and/ or information, e.g. as the structured concept of Enterprise Information Management (EIM), as an organised program to design, catalogue, and safeguard all information assets to maximize their value, usefulness, and accessibility, and security (1) requires a holistic approach to balance technology, the organisation, governance, metrics, and process dimensions.  Essentially it is the destructing of the silo-based isolation of systems and data by a master data management (MDM) program to effect and reduce the obstructions and complexity of application integration and cross-functional operations and make applications and data more interoperable.  Isolated and specialised applications should morph and be absorbed as composite-based architectures, i.e. a number of discrete, loosely coupled, semi-independent components that can be integrated into an application framework to form a coherent solution, i.e. a composite-based architecture.  To help this architecture, data (being governed by the Enterprise Information Management aka Enterprise Data Integration aka Enterprise Information Integration) must be viewed as a core Information Technology (IT) element and governed for the purpose of promoting the achievement of a composite application architecture.  Because EIM supports faster integration and interoperability between applications, it makes it possible to deliver dynamic business services as required in architectures like a Service Orientated Architecture (SOA).  To adequately enable this the relationships between infrastructure components, processes, applications, and repositories integration must be clearly understood.

To achieve effective Enterprise Information (data) Management (EIM) the technology architecture must enable (1):

  1. Data integration (interoperability) across business entities like customer, product, employee, etc.
  2. Provide seamless data continuously across transactional, operational, and analytical environments in support of real-time business agility
  3. By master/ metadata management ensure clean and integral data consistently
  4. Manage unified and relevant content and content aggregations
  5. Enrich, profile, clean, transform, and access data

These tenets are especially important for Data Lakes and Hadoop solutions for Big Data.

Business service orchestration require that application components (while using data) and the business processes be executed to realise business and system functions.  The data layer, supported by the master and metadata, standards, business rules, classifications, and the data services and integration technologies, undergirds the data application portfolio and the higher business process platform that executes business functions.  The EIM infrastructure is comprised of:

  • Data Services for the access provisioning of data across the organisation
  • Master/ metadata management and semantic reconciliation for the optimisation, abstraction, and semantic reconciliation of all disparate metadata in support of reuse, consistency, integrity, and share ability of the enterprise information (data) asset. This especially prevalent within a SOA architectures in that developing, using, supporting, and auditing become problematic.  This simply because the underlying data sources and structures, meanings, quality, connecting methods, etc. are unknown
  • Close-loop information flows for master data and convergent content are difficult in ensuring the seamlessness and continuousness of data between and across boundaries. To save data into Business Intelligence warehouses improves analysis, but it does not solve anything for transaction processing.  The closed-loop ensures a bi-directional data flow, i.e. from transactions down to that analytical stores and intelligence in return to the applications.

Note: Also see the Data Delivery Platform as defined by Rick van der Lans in the Representing Data for Various Purposes section of this paper.  The above mentioned precepts are all exemplified by this concept.

Technology has matured over the years and support the enterprise data management  function much more adequately today.  Data virtualisation is such an enabler along side the plethora of other tecnologies for managing heterogeneous, and Big data integration to the enterprise, not only on the analytical level, but also for transaction processing.

Data virtualisation is a concept and an approach to data management that allows the consumption of data from a data source without having to know the technical details nor the physical location of the data (1).  Data Federation is akin to data virtualisation, often used in the same sense (2) as “aka” the same thing, however, not quite, and using these terms interchangeably leads to confusion on the subject (3) and makes a strategy determination problematic.  Philip Russom, research manager with TDWI Research says that “Data virtualization must abstract the underlying complexity and provide a business-friendly view of trusted data on-demand.” (4) Consequently Data Federation is a sub-component of Data virtualisation.  Its purpose is somewhat benign to that of the bigger Data Virtualisation visage, i.e. a pooling of data integration resources.  Data Federation only federates data from different (heterogeneous) data sources in real-time.  A conundrum appears, comprised of definitions pertaining to data virtualisation, federation, enterprise information integration and even mash-ups.  Sybase, under Data provisioning and Integration, refer to the provisioning of data, to users and applications across the organisation, as a data services layer or data grid (6).  They also indicate this layer to have the ability of supplying heterogeneous data as: “some applications expect relational data; others need data in XML form”.  The data grid provides a single layer for access (read/ write), a unified data catalogue (access to heterogeneous data), unified access control across networks, locations, departments and companies.

Rick F. van der Lans of R20 consulting and the author of Data Virtualization for Business Intelligence Architectures (5) concurs that there is a fair degree of confusion in the market, mainly because of the vendors creating it in the definition of their products.

The Data Delivery Platform (Rick van der Lans) imbibes a few data integration types, viz. “Data Virtualisation, Enterprise Service Bus, and Object Relational Mapping (ORM)” (Reference).  The ORM also acts as a data virtualisation enabler in its capacity as data provider, converter, and object model presenter. (Reference)

In order to gain a fair idea of what data virtualisation is and how it should be used within a company’s enterprise information integration (EII) and/ or Enterprise Information Management (EIM) the definitions of each should be firmed up and agreed on.  This paper follows the definition and guidance by author Rick F. van der Lans primarily and bolsters it with opinions from the industry.

1.1    General Considerations on the Adoption of an Innovation

Data Virtualisation and other Data Integration technologies are being innovated progressively and makes it difficult for companies to decide on using such technologies now, or rather to wait for maturity, if it ever comes in a satisfactorily way.  This is the fulcrum of innovation and the eternal question: when to jump on the band wagon, i.e. before the band, or after the band?

Implementing an innovative technology requires a fair degree of cultural infusion in the psyche of the company.  This is the Diffusion of Innovations concept popularised by Everett Rogers (7).  It is a theory that seeks to explain how new ideas and technology spread through cultures (21).

Five Segments of Adopters

Figure 1: Five Segments of Adopters

The adoption rate typically manifests in the well-known categories of adopters, viz.: innovators (2.5%), early adopters (13.5%), early majority (34%), late majority (34%), and laggards (16%), i.e. Figure 1. Each group of adopters has its own personality in relation to attitude to a particular innovation.

Time needs to pass in order for any innovation to be adopted, unless the adopters are closely monitoring innovators and have support for their ideas or they are partners of design[1] (9).  A couple of factors, viz. communication channels and social systems play a critical role in determining what category of adoption the adopter falls in and how much benefit they would obtain from the adoption.  The social system has internal and external influences in the strength of the social relationships and the distance from opinion leaders as influences on the potential adopter.  As for the laggards, they are the ones holding out to the bitter end, only seeing high risk, staying up all night to look for arguments against it, having all facets of their paradigm confronted and affronted.

Innovations are gauged according to a few factors and judged holistically:

  • Relative advantage to current tools and processes
  • Compatibility with pre-existing systems
  • Complexity and difficulty to learn
  • Test and proof of concept ease and ability
  • Versatility of the innovation, i.e. if it can be used for other things too
  • Observable effects of the innovation on the enterprise

Data Virtualisation is an innovation with a small core and large periphery that makes it easier to adopt.  However, Enterprise Data/ Information Integration is the larger concept that bears on the simplicity of data virtualisation and how it fits into the scheme.  The single most potent duo of qualities that increases the likelihood of an adopter to adopt an innovation is the combination “ability and motivation”.  People who have the organisational power and agency to cause change and those who more frequently participate in actions and ideas with a large degree of variety are the most likely to adopt an innovation.  “An organisation is an aggregate of its individuals and its own system with a set of procedures and norms.”  The adoption of an innovation is diffused from its individuals according to:

  • Tension for change, i.e. motivation and ability
  • Innovation-system fit, i.e. compatibility
  • Assessment of implications , i.e. observability

The process of adoption is through:

  • Awareness or knowledge or agenda-setting
  • Interest or persuasion or matching
  • Evaluation or decision or redefine/ restructure
  • Trial or implementation or clarifying
  • Adoption or confirmation or create the routine

Any decision made for innovation is made as a type of decision, viz.

  • Optional innovation-decision, i.e. made by an individual who is distinguishable from others
  • Collective innovation-decision, i.e. made collectively by all participants
  • Authority innovation-decision, i.e. made for the entire social system by individuals in positions of influence of power

The type of person (innovator/ implementer) required in the process of innovating and implementing the innovation spans homophiles and hetrophiles.  Cooperation is best achieved by wholly homophilic people working together by no innovation.  The best combination is homophiles in every characteristic except that of knowledge and innovation.  Inter-personal ties between heterophiles are weaker, and harder to create and maintain.  Therefore, it is very important to have peer-to-peer conversations and networks (9).  Conversations spread adoption and conversations promote the management of risk and uncertainty.  To be an early adopter, you should be “on the lookout for advantages and tend to see the risks as low because they are financially more secure, more personally confident, and better informed about the particular product or behaviour.”

[1] In line with the concept: “Diffusion of Innovations” (Everett Rogers) and how an organisation can be an early adopter of innovation (or innovate at all), these employ Enterprise Architects who should be innovators and creative thinkers whose sole purpose it is to partner with the business contingent to enable innovative and innovation adoption as early as possible.

1.1    Cost of Data

Frequently asked, seldom receives a meaningful answer.  Data is a business asset thrust into the hands of the Information Technology department (11).  The cost can be measured by the cost of losing data or a data breach occurrence.  As data grows the number of losses/ breaches grow reciprocally (12), express by the cost defined in the Ponemon Institute CODB report of 2013/14 (13).  The Information Week Wall Street and Technology Dr. Howard Rubin (14) says that 92% of the cost of business—the financial services business—is “data”.  He argues that data should be expressed as a ratio of Terabytes per one million of the monetary value of revenue/ operating expense or Terabytes of data per employee and that banking and financial services are the more data intensive sectors from a revenue, operating expense, and employee perspective (0.82 TB per $1M revenue) and only outpaced by the media sector (0.76 TB per 1M revenue) because of video and audio content (14).  Data intensity is also the primary driver for technology, operating, and terabytes-per-employee costs across sectors.  More than 92% of business cost (financial services) is data, but only 3% of the net revenue is derived from market data.  The rest of the cost is absorbed by data acquisition, transmission and distribution, processing, and the storage and retrieval of data, over IT and operations, i.e. the people and the facilities.

The financial services sector spends 9.2% of the global technology spend ($4.5 trillion in 2011) defrayed over acquisition, distribution, processing, storage, retrieval, and delivery (83% of this technology spend), i.e. $333 billion in 2011 (Figure 2).

Technology Expenses in the Financial Sector

Figure 2: Technology Expenses in the Financial Sector

The cost of data, or at least where the money is being spent, is in the management of data. The core IT and Business Process Outsourcing marketplace should see a shift from the focus of colocation, managed services, outsourcing and transaction processing to that of data management.

Data valuation is challenged by the rapid growth of data and the measures of management required.  Data management decisions must be made with a long-term view of business gains as opposed to the lure of short-term resource cutting cost savings (6).  Data Virtualisation technologies can be used as a key technology in an optimised data management strategy.  More value must be drawn out of existing data through greater business agility and less time to market, all of which these technologies promise.  Ineffective management of data creates high cost, by bearing detrimentally on areas like business intelligence, application development, and business project completions.  Actions like creating test databases and allocating storage are costly and disruptive, even copying data to these test databases.  Reducing these procedures optimises cost and time.  Data, being a business driver and enabler, drive cost that overspills to the IT sphere and is vested in business productivity.  This is impacted by not knowing what the data is, where to get it, and how to interpret it into intelligence or consequential information that enables a timely business decision ability.  The movement of data requires infrastructure and staff efforts, and the cost compounds if the business community has difficulty in obtaining and understanding the data.  To harmonise data and make it interoperable, especially for heterogeneous data source types calls for sophisticated mechanisms.  Consider a contingent of semi-structured and free-text (unstructured) data that exists alongside relational stores of transactional and vast vaults of historic and even data warehouse data.  A total integration is implied in a distributed database and integrated into a logical database, managed under a global control authority in a consistent way, by collating the federated data sources into one logical presentation.  The cost of data, as the enterprise data stores grow larger over time and the integration of applications increase, grows accordingly, unless data-handling reductions, through some efficient data integration technology is able to reduce the handling cost.

Poor data interoperability can result in 66% of the total cost of rework, errors, and downtime (16).  Database integration is the key to data interoperability (17).  Database integration, on a distributed architecture, managed by heterogeneous database management systems, files, or spreadsheets, is the resolve of interoperability issues.  The associated costs, as elaborated above, as well as the propensity for incompleteness, inaccuracy, and data inconsistencies in data acquisition and processing, lack of coordination resulting in duplication of efforts and of resources, and eventually, the conflicts in the allocation of responsibilities for data maintenance, make the cost of data exceptionally high.

Storing data is expensive, especially if the same data is stored more than once and Gartner (18) suggests in Action 6 to contain storage growth through storage virtualisation, automated tiering, and storage resource management tools.  So, also is processing data, or handling it more than once, to suffice various needs, very costly.  Infrastructure data cost spans disk storage, switching and network time, these often over international boundaries.  The cost compounds where staff are required to oversee the huge storage and maintenance of systems.  Business users requiring data, from which to derive insights, lean on IT staff in non-automated environments and often compromise the data because of lacking security or inadvertent data loss through communication channels like email and off-site storage on flash-drives.  Clean data are required for efficient business operations, both on systems and for analytics, presented in an automated and simplistic (abstracted and encapsulated) manner.  An increased manual incurance compromises delivery times, data errors, productivity loss and delayed schedules.  It follows-on that business productivity suffers because of an inefficient dependence on an overly manual data provisioning and governance.  Data is the cornerstone of business enablers like application development and deployment, business intelligence, business projects, and strategic decision making.  Any inefficiency in the provisioning and quality of the data directly translates into a detrimental impact on the business, in the form of erroneous data, late data, and incomplete data, or data that is not understandable in relation to a business understanding.

Enterprise Resource and Planning (ERP) systems are often grouped in the systems of record space of the classification stack[1] but in the pursuit of differentiation some aspects of these need to be positioned within the systems of differentiation and systems of innovation strata of the stack, especially from a data perspective, i.e. to manage the data across the ERP architecture, lifecycle and business processes.  This solidifies the core ERP components and raises capabilities that help a business outpace competitors.  A more effective utilisation of ERP allows reporting and business intelligence tools to unlock information from the data and expedite that to the relevant business owners.  The key-issue is to unlock more business value out of system (ERP) investments and provide a greater return in that investment, while providing a differentiation of systems and market representation of the business.  This speaks to the data generated by these systems and timely using that data for business decision making.  The Gartner PACE-layered approach should be used into which to map all enterprise (ERP and non-ERP) systems (7) so that their individual pace of change and degree of differentiation is segregated for the purpose of applying a specific and cost-effective governance structure.  However, as the data analytics requirement is on the increase, it is more necessary to have the right and clean data available in a consistent manner, at the appropriate time as and when the user requires it and the ability to obtain this data in a timely manner.  This requirement spans transactional and historical data that may be sourced from heterogeneous and geographically dispersed systems.

One cost reduction can be found in Infrastructure and Operations (cited as Action 3 in the Gartner list: 10 Key Actions to Reduce IT Infrastructure and Operations Cost. (19)) through standardisation (to industry patterns and practices), integration, and virtualisation (both infrastructure and data).  Gartner states that the consolidation of infrastructure and data processing into larger data centres to be the trend into the future.  Generally servers run at a very low utilisation rate (<15%) and should be virtualised not only for servers but also for the utilisation of server demand, or the elastic infrastructure.  This ties with Action 9 to reduce the cost of asset management.  The cost of data is reduced by a few factors.

However this sides, making better use of resources (both for data and infrastructure) reduces the cost.  Reducing the costs in evaluation planning, data collection, and analysis include (20):

  • Focus evaluations to eliminate non-essential data collection
  • Reduce the frequency of collecting data
  • Use available data
  • Reduce the sample size
  • Use cheaper data collection options

[1] The classification stack is comprised of Systems of Record (non-differentiated utility systems), Systems of Differentiation (doing things a bit smarter), and Systems of Innovation (doing things uniquely).  This is the Gartner PACE-layer classification of systems.

1.1    What is Data Integration? (3)

Integration is the combination of parts to form an integral whole.  Data integration is the process of combining data from a heterogeneous set of data stores to create one unified view of all that data.

The definition of data integration does not enforce how the integration takes place.  For example, consider the traditional business intelligence footwork or ETL, evaluate, transform and load, and compare that to live/ real-time and virtual transformations where data does not end up in another physical database.  Both are valid data integration techniques, but don’t mandate the method.

Enterprise Information Integration (EII) is not data integration as data and information are different entities.  The latter is a derived elevated view of data upon which some interpretation has been performed, i.e. intelligence derived, hence the concept of data intelligence or business intelligence.  Data worked into a meaningful entity called information, i.e. something from which some kind of business interpretation/ intelligence can be derived, can be used for business decision making.  Data is therefore integrated (not information) that leads to information of business intelligence, two separate but successive concepts.  The industry uses EII as a synonym for data integration, but it should rather be Enterprise Data Integration (EDI).  Data virtualisation therefore may not require data integration and is dependent on the number of data sources accessed.  However, federation always requires data integration and is a style of integration.

1.1    What is Data Virtualisation? (3)

Virtualisation is a technique to extend the physical boundaries of information.  Things like memory paging of information, virtual processors, networks, and even operating systems are all logical constructs of physical entities with enlarged boundaries that create the illusion of having more than what really exists.  Like the time-share of a holiday flat, more than one person can own the property and share time around its occupation, creating the illusion that they are all sole owners of the property, but due to constraints, can only visit it some times.  This is an abstraction of the concern, i.e. the details of it is hidden from the users.

In computing systems it implies that the location, technical interface, implementation, platform, size, and available quantity of the physical resources are abstracted from the users (6).  Only the portions exposed through the logical layer (the virtualised effigy of the resource) are known.  Consequently, the virtualisation of data is the extension or presentation of data in a logical architecture and access interface to it by hiding (abstracting) the technical aspects of the physically stored data (location, structure, API, language, and storage technology) from the consumers.  This is to see those details in logical abstraction pertaining to their use-case, as a virtual or derived version of these facilities from the actual physical implementation of it.

Data Virtualisation therefore provides the abstraction layer for accessing data in a consistent, albeit virtualised or logical manner.  The user can be an application.  The virtualisation or abstraction encapsulates (hides) the technical details/ aspects of the physical data in storage from the user.  The implementation of data virtualisation is manifold:

  • Federated data, i.e. collecting heterogeneous data from sources and present these as a singular view, in a dingle data base, i.e. a virtual database.
  • An Enterprise Service Bus (ESB) comprised of a layer of services that allow access to data, via SOAP or REST through the ESB as the abstraction layer.
  • Access to data in the Cloud. In this case the Cloud abstracts and encapsulates the access and physical data properties.
  • In-memory database with the abstraction and encapsulation presented to the user by this virtual data store

Red Hat defines data virtualisation as a data supply and integration solution, delivering the needed data—in the required form—at the right time to any application or user (7).  Similarly Denodo (8) describes it as the support of data integration, for data in any format, technology. Location, e.g. cloud or web, in a scalable architecture to advance performance in order to obtain data in the right location when needed.  Cisco data virtualization is agile data integration software to abstract and view data from any physical location, i.e. a data integration as a single data source from a federated view (9).  Informatica states data virtualization integrates data from disparate sources, without physically moving data, and provides a single access point to manage and view this data (10).

The concept of the Data Lake (data liberalisation (26)) has also taken hold in with for example the Business Data Lake development by Capgemini and Pivotal (2013) that “distills raw data into high quality information” (26) through hundreds of high-speed connectors that parse, integrate, and cleanse data for making raw data information and what other people call a “search engine in a box. (28)”  It comes with a visual development environment, reusable business rules, and flexible deployment models that accelerate the implementation of production –ready data pipelines; executes data governance and end-to-end data lineage to ensure that the data can be trusted.  Master data management create authoritative, 360 degree views, of relationships between critical entities of business data.  However, Gartner says beware the Data Lake Fallacy (27) as it has created a lot of confusion in the information management space.  They say that data lakes are marketed as enterprise wide data management platforms for analysing disparate sources of data in their native form.  This sounds similar to the Data Hub concept (28), but data lakes lack sematic consistency and governed metadata because data is simply dumped into the lake.  To obtain value from the data, is the onus of the business end user and although technology could aid with this there should be some semblance of information governance applied to prevent the lake from becoming a cesspool.  Accepting data without any oversight or governance and the lack of representative and accurate metadata means that analysis would have to start from scratch every time with very little security and access control by the nature of the unknown data.  Gartner recommends “that organizations focus on semantic consistency and performance in upstream applications and data stores instead of information consolidation in a data lake.”  To get to data, some kind of logic needs to be applied.  In the case of structured relational data, the data is known, consequently the query is predictable and the access to the data governable.  But, where data are heterogeneous in kind and/ or source and/or location, or all of the mentioned, the consolidation and governance of the data becomes exponentially complex.  Finding context is probably the biggest concern.

1.1    What is Data Federation? (3)

Federation refers to a collection of individual and autonomous operating entities that may be controlled by a federal host.  Data federation is the combining of autonomous data stores/ entities into one/ single data store, i.e. a virtual database.  Consequently data federation is a type of data virtualisation where data are stored in a heterogeneous set of autonomous data stores and made available in one integrated data store using on-demand integration.  Data federation is not required if data is virtualised into a single data store, but data federation always results in data virtualisation.  Data federation enables the collation of heterogeneous data stores (in data and implementation) by integrating data (through transformations) into a single virtual store allowing applications, and tools to access this data.  The operation of these heterogeneous stores is not impeded upon by the federated model, as they remain autonomous in nature, i.e. independent.  In order to render a single virtual and integrated representation of the federated data it requires transformation, cleansing, and potentially enrichment of the data.  This takes place on-demand, i.e. when the data is summoned.

1.1    Data Virtualisation, Use Case

Generally speaking, one could assume that data virtualisation should be the sine qua non for all enterprise data access, but is it?

The basic requirement of a business is to obtain trustworthy data when required from which to derive intelligence.  Consequently an analytical capability should be supported by a data virtualisation technology.  Business Intelligence (the interpretation of enterprise information, derived from enterprise data) are multi-faceted and polymorphic, i.e. taking on many forms.  The basis of BI is to obtain data from the appropriate sources and then modify the canonical[1] business view to accommodate the type of governance, metadata, and access to the data.  The objective is to establish reusable and common, business data services.  This to be achieved through a thorough business analysis to adequately design these services and prevent a restriction in flexibility by having increasingly more complex entities being added to an architecture that struggles to support these.  For example, services exposing arbitrary variations of the same business information should be avoided, also many instances performing data mapping in the ESB and business processes.  Only good planning, and/ or thorough subsequent house-keeping can aid in the elimination of occurrence of these.  A canonical data model provides a common format for the data/ information content of messages produced by individual services and leads to a horizontal alignment of message (data) formats across services and data views (11).  The data virtualisation platform provides the canonical data model and is extended to the ESB for SOA messages.  All of the work that a business performs comes to rest in its data.  The data is the surest source for getting an understanding about what the application architecture should be and also to know how the business is performing. The canonical data model is a critical foundation for effective data governance and provides the solid structure from which to extrapolate the application architecture and to know what information users need.

IBM states that a canonical data model is specified only at two different levels of granularity or two phases of definition within the same model (11):

  1. Conceptual Model and
  2. Logical Model

[1] Canon = rule/ standard, archetypal, typical. Unique, distinguished exemplar (11) as a design pattern that communicates between different formats as a form of enterprise application integration as a standardisation for integration, e.g. a standard for message structures and content, i.e. common data names, definitions, and values within a generalised framework (12).  The canonical data model defines the structure of an organisation’s data, not in the physical data stores, nor in a single database, but is a reference for all data entities and their relationships across all of the data stores and related legacy applications encompassed by the SOA architecture (13).

1.1.1    Conceptual Data Model

The conceptual data model is a canonical model at the highest level of abstraction with business entities as the fundamental building blocks. Each attribute of an attribute collection, are aligned to a business glossary for terminology and semantic definitions, and includes entities and their most important relationships but is not granular enough for service specification.  A conceptual data model provides a few key-concepts to help an organisation define its data landscape as an expression of its business understanding:

  • High-level data groups to be used in identifying business services, process modelling, business analysis, etc.
  • Baseline data for services and lines of business
  • Data governance roles and responsibilities, policy development, ownership, responsibility with ownership, etc.
  • Basis for identifying data sharing (asset reuse) between services
  • Risk reduction in omitting any data required and how to subsequently share that and align enterprise wide data consumption

Business unit data sharing starting point for analysis, future impact analysis, and accurately estimate further projects.

1.1.1    Logical Data Model

Derived from the conceptual model it further elaborated the attributes, in extension of the enterprise data model, as inputs to developing data structures and representations across architectural layers in physical data, message, and component models.  The logical model also assists in creating business process models, as service consumers, where the service analysis exposes services to be consumed for data.

A logical model has a primary data key to distinguish entities and create a unique and stable reference that does not change over time.  All attributes of an entity are manifested in the logical model and must be sufficient to support the data structure being specified.  These attributes must be business names consistent with the business glossary and be of a consistent nomenclature.  Where attributes have been given names consistent with a business that has evolved over the years, having left the nomenclature of these entities in an arcane or archaic state, a logical transformation within the enterprise data integration (EDI) platform should re-align these attributes with the business glossary.  Physical transformations of the source/ legacy data entities are not required, but performed in-use and on-demand.  Referential integrity (both on the physical as well as the virtual stores) ensure the integrity of the logical data models and in the same vein normalisation techniques, as super-to-sub-type relationships may be consistently modelled.  The logical attributes (of entities) have physical data types and is vertically aligned in the database schema and horizontally aligned in the message model.  Rules governing domains of specific values, ranges of values, or logical restrictions are specified for attributes, e.g. [order data > delivery date] and these are enforced by data services and/ or the physical/ virtual database.

1.1.1    Disparate or Ageing Physical Data Stores

As previously stated, any ageing physical data model (and its entity attributes) diverge from the business glossary over time, leaving the nomenclature of these entities greatly at odds and injects confusion and problematic data models into the business landscape and the utilising applications that are used by the business people and their customer/ client base.  Several views of a customer may come about, e.g. being called a client, customer, partner, etc.  Business mergers and acquisitions may also drive these divergences.

Such circumstances call for a rationalisation within the data and application (more so the message SOA) landscapes and demands for a logical data model alignment with the universal business glossary and semantic basis, to align with what application users see when they access business application functionality.  Doing this is encapsulating the legacy nomenclature and abstracting it from the business applications and the interlinking SOA messages and logical data sources.  It has an organisational impact and is not trivial to achieve, but contemporary enterprise data integration technologies provide elegant platforms to encapsulate and abstract these disparate entities and present them in a way that aligns with the actual business on the ground.  It also makes one of the most compelling use-cases for Data Virtualisation that principally improves the structure and flexibility of the data architecture of the enterprise.  This through collating data from a federation of sources (often heterogeneous in nature) to increase data availability and intrinsic consistency in alignment with the business glossary.  The virtual platform provides the abstraction required to re-model data in accordance with the changed enterprise characteristics and consumer audience.

However, some considerations to entertain, and the single most important one, with the ability to nimbly adapt the data and application architectures with either an ageing business or one combined with another, through a virtualised data platform, proffers the singular disadvantage is performance.  Any layer between the source and the consumer has a performance impediment, if the technology does not provide adequate mitigation, such as a load balance distribution of processing.

Despite this, the biggest boon of such an enterprise data integration technology is that it provides a singular point of access and maintenance that allows for the divergence with the business glossary to be absorbed and levels a conducive platform for data governance.  However, it does not completely abstract the user-base from changes to the underlying physical stores and by means of the immediate delivery to the user-base, manifests immediately in the computing environment of a business.  The primary gain of data virtualisation is agility, time to market, and simplicity, all in terms of how quickly and easily it is to integrate heterogeneous sources of data, but also the abstraction from the physical infrastructure (12).

The data virtualisation layer should be the ubiquitous layer within the infrastructure that all data consumers use to access enterprise data (13) because it focuses on the data consumer.  However, for virtualising data or loosening the tight coupling to data stores (point-to-point integration) it is a preferred technology but for data extractions such as ETL it might not be, and only when it is absolutely required.

ETL is traditionally a batch process that handles the bulk transfer or structured, mainly internal data, from operational and transactional stored to data warehouses.  Data virtualisation technologies can augment the present ETL processes by providing the ability to embrace a more agile integration of a multitude of different data formats and sources.  ETL provides persistence while virtualisation provides the agility (17).  Business Intelligence is almost useless if captured and analysed too late (18).  But, building and enterprise data warehouse is a very expensive initiative that takes long to implement (19).  Data virtualisation can be employed to extend the data warehouse.  The ETL aspect should be considered in the frame of data integration and not just because it can be done, performed regardless by the data virtualisation technology.

1.1.1    Representing Data for Various Purposes

Access to data, for developing and testing per se, is the biggest constraint limiting development and testing (14) and it makes sense to enable all applications for data virtualisation, primarily for time-to-environment and cost reduction.  The provision of server and data virtualisation (even for files and directories) is required for development and testing environment provisioning on the fly.  In addition, the smaller portion is production, therefore it makes a good case to look to virtualisation in the former architectures from a cost-to-agility perspective.

The broader sense of representing data for various purposes is to have the relevant data available.  The data does not have to be available in any known structure or source, but can also be brought together by data integration technologies such as Hadoop (usually associated with Big Data) and the concept of the Data Lake.  These technologies has an impact on a traditional data architecture and will have to revolutionise the enterprise when adopted.  The adoption rate is probably in the Early Majority (Figure 1, page 3) for Big Data (Hadoop) and Data Lake styled technologies (at te present time), even maybe for data virtualisation technologies in combination with these in the provisioning of a Data Delivery Platform[1] (38).

Knowing the context of data queried, particularly for unstructured data, poses a problem if the data can’t be adequately described, or when the data was sourced is unknown.  To derive value from data the data must be consistent, the context known, and be describable.  Simply having data thrown into a Data Lake is not sufficient,

Data Lake and Appliction Cloud

Figure 3: Data Lake and Appliction Cloud

but requires some interpretation.  Even where Hadoop becomes the default destination, governance and metadata become more important.  In such a case Hadoop becomes an application platform around the data lake (Figure 3) as an elastic distributed data computing platform, for operational and analytical functions.  The data lake adding security and governance layers and data availability increases in direct proportion to the application deployment time decreasing (32).

[1] A Data Delivery Platform is a business intelligence architecture that delivers data and metadata to data consumers in support of decision making, reporting, and data retrieval, whereby data and metadata stored are decoupled from the data consumers through a metadata driven layer to increase flexibility, an whereby data and metadata are presented in subject-orientated, time-variant, and reproducible style. (Rick van der Lans) (38)

The sole purpose of data integration is to connect enterprise data.

The Data Lake and its data provisioning technology (whatever that is) are together a data delivery platform.  According to author Rick van der Lans, the requirements for a data delivery platform are (40):

  • Deliverable data
  • Data stores and access
  • Querying and data consumption
  • Metadata
  • Acquisition and integration of data
  • Performance, scalability, and availability
  • Data quality
  • Security
  • Transactions

The Data Delivery Platform therefore applies all the characteristics in caveat to make the concept of a Data Lake and its enabling technology feasible as an enterprise data/information integration method.

2      References

  1. Newman, David;. EIM Reference Architecture: An Essential. Chicago : Gartner, 2005. G00130553.
  2. Rouse, Margaret. data virtualization definition. TechTarget. [Online] TechTarget, 2015. [Cited: 15 February 2015.] http://searchdatamanagement.techtarget.com/definition/data-virtualization.
  3. Data Federation. Business Intelligence. [Online] Business Intelligence, 2015. [Cited: 16 February 2015.] http://businessintelligence.com/dictionary/data-federation/.
  4. van der Lans, Rick. Clearly Defining Data Virtualization, Data Federation, and Data Integration. BeyeNetwork. [Online] BeyeNetwork, 16 December 2010. [Cited: 16 February 2015.] http://www.b-eye-network.com/view/14815.
  5. Virtualization, Federation, EII and other non-synonyms. B-eye-Network. [Online] B-eye-Network, 02 Decemebr 2010. [Cited: 16 Fabruary 2015.] http://www.b-eye-network.com/blogs/devlin/archives/2010/12/virtualization_federation_eii.php.
  6. SyBase. Data Provisioning and Integration. http://infocenter.sybase.com/. [Online] SyBase, 2015. [Cited: 24 February 2015.] http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.stf.df.doc-SWS-2.0.0/projects/datafed/avaki/en/source/c_data_prov_main.html.
  7. van der Lans, Rick L. Data Virtualization for Business Intelligence Systems: Revolutionizing Data Integration for Data Warehouses (The Morgan Kaufmann Series on Business Intelligence). Amazon. [Online] Amazon, 08 August 2012. [Cited: 16 February 2015.] http://www.amazon.com/Data-Virtualization-Business-Intelligence-Systems/dp/0123944252/ref=la_B001HPXTW8_1_2?s=books&ie=UTF8&qid=1424069146&sr=1-2.
  8. WikiPedia. Everett Rogers. WikiPedia. [Online] WikiPedia, 19 November 2014. [Cited: 24 February 2015.] http://en.wikipedia.org/wiki/Everett_Rogers.
  9. —. Diffusion of innovations. WikiPedia. [Online] WikiPedia, 14 February 2015. [Cited: 24 February 2015.] http://en.wikipedia.org/wiki/Diffusion_of_innovations.
  10. Robinson, Les. Enabling Change. Chicago : Amazon, 2009.
  11. Simmonds, Stephen. Data Provisioning: a new paradigm. datatechnology.wordpress.com. [Online] datatechnology.wordpress.com, 07 April 2009. [Cited: 24 February 2015.] https://datatechnology.wordpress.com/2009/04/07/hello-world/.
  12. Jacobs, Jay. Analyzing Ponemon Cost of Data Breach. Data Driven Security. [Online] Data Driven Security, 11 December 2014. [Cited: 24 February 2015.] http://datadrivensecurity.info/blog/posts/2014/Dec/ponemon/.
  13. Ponemon Institute. 2014 Cost of Data Breach Study: Global Analysis. IBM. [Online] Ponemon Institute, 2014. [Cited: 24 February 2015.] http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=SA&subtype=WH&htmlfid=SEL03027USEN#loaded.
  14. Rubin, Howard. Technology Economics: The Cost of Data. Information Week Wallstreet & Technology. [Online] Information Week Wallstreet & Technology, 22 August 2011. [Cited: 24 February 2015.] http://www.wallstreetandtech.com/data-management/technology-economics-the-cost-of-data/d/d-id/1265161?.
  15. Marden, Matthew; Perry, Randy; Olofson, Carl W;. Taking a Holistic View of the Total Cost of Data Helps CIOs. Framingham : April, 2014.
  16. Infographic – The Cost of inadequate Interoperability. McLaren Sotware. [Online] McLaren Sotware, 2015. [Cited: 24 February 2015.] http://www.mclarensoftware.com/resources/publications/infographics/infographic-the-cost-of-inadequate-interoperability-.aspx.
  17. Parent, Christine; Spaccapietra, Stefano;. DATABASE INTEGRATION: The Key to Data Interoperability. Cambridge : The MIT Press, 2000.
  18. Gartner. 10 Key Actions to Reduce IT Infrastructure and Operations Costs. Gartner. [Online] Gartner, 11 September 2011. [Cited: 24 February 2015.] http://www.gartner.com/newsroom/id/1807615.
  19. Roux, Hendriekus. JSE EA ERP Systems of Record and ERP Systems. EA, JSE. Sandton : JSE, 2015.
  20. Evaluation, Better. Strategies to reduce costs. Better Evaluation. [Online] Better Evaluation, 2015. [Cited: 24 February 2015.] http://betterevaluation.org/evaluation-options/strat_for_reducing_eval_costs.
  21. Swoyer, Stephen. Why Data Virtualization Trumps Data Federation Alone. TDWI. [Online] TDWI, 01 December 2010. [Cited: 16 February 2015.] http://tdwi.org/articles/2010/12/01/Data-Virtualization-Trumps-Data-Federation.aspx.
  22. RedHat. Turn fragmented data into actionable information at business speed. RedHat. [Online] RedHat, 2015. [Cited: 16 February 2015.] http://www.redhat.com/en/technologies/jboss-middleware/data-virtualization.
  23. Denodo. Denodo Platform Overview. Denodo. [Online] Denodo, 2015. [Cited: 16 February 2015.] http://www.denodo.com/denodo-platform/overview.
  24. Cisco. Data Virtualization. Cisco. [Online] Cisco, 2015. [Cited: 16 February 2015.] http://www.cisco.com/web/services/enterprise-it-services/data-virtualization/index.html.
  25. Informatica. Deliver New Data and Reports in Days vs. Months with Data Virtualization. Informatica. [Online] Informatica, 2015. [Cited: 16 February 2015.] http://www.informatica.com/us/data-virtualization/#fbid=iob80lQNSdA.
  26. Oliver, Andrew C.;. Gartner gets the ‘data lake’ concept all wrong. Infoworld. [Online] Infoworld, 31 July 2014. [Cited: 24 February 2015.] http://www.infoworld.com/article/2608618/application-development/gartner-gets-the–data-lake–concept-all-wrong.html.
  27. Informatica. Informatica becomes part of Capgemini and Pivotal’s Business Data Lake ecosystem. Informatica. [Online] Informatica, 24 February 2015. [Cited: 24 February 2015.] http://www.informatica.com/us/company/news-and-events-calendar/press-releases/informatica-capgemini-pivotal-business-data-lake.aspx#fbid=iob80lQNSdA.
  28. INFORMATICA DIVES INTO ‘MARKET DISRUPTING’ BUSINESS DATA LAKE. Computer Business Review. [Online] Computer Business Review, 24 February 2015. [Cited: 24 February 2015.] http://www.cbronline.com/news/tech/software/analytics/informatica-dives-into-market-disrupting-business-data-lake-4518816.
  29. Gartner. Gartner Says Beware of the Data Lake Fallacy. Gartner. [Online] Gartner, 28 July 2014. [Cited: 24 February 2015.] http://www.gartner.com/newsroom/id/2809117.
  30. Datamere. Hadoop as a Data Hub: A Sears Case Study. Slideshare. [Online] Slideshare, 05 July 2013. [Cited: 24 February 2015.] http://www.slideshare.net/Datameer/hadoop-as-a-data-hub-featuring-sears-23956693.
  31. The information perspective of SOA design, Part 4: The value of applying the canonical modeling pattern in SOA. DeveloperWorks. [Online] DeveloperWorks, 2015. [Cited: 16 February 2015.] http://www.ibm.com/developerworks/data/library/techarticle/dm-0803sauter/.
  32. Harper, Jelani. Data Virtualization’s Value: Myth or Reality? DataVersity. [Online] DataVersity, 01 August 2013. [Cited: 16 February 2015.] http://www.dataversity.net/data-virtualizations-value-myth-or-reality/.
  33. Eve, Robert;. When Should We Use Data Virtualization? Sys-con.tv. [Online] Sys-con.tv, 17 May 2011. [Cited: 16 Fabruary 2015.] http://tv.sys-con.com/node/1833397.
  34. Moxton, Paul; Chandrasekaran, Suresh;. Use Data Virtualization to Breathe New Life into ETL Processes. Big data. [Online] Denodo, 31 March 2014. [Cited: 19 February 2015.] http://sandhill.com/article/use-data-virtualization-to-breathe-new-life-into-etl-processes/.
  35. James, Ezekiel. Data Virtualization and Real-time Data. DataVirsity. [Online] DataVirsity, 21 May 2013. [Cited: 19 February 2015.] http://www.dataversity.net/data-virtualization-and-real-time-data/.
  36. A Perspective on the Benefits of Data Virtualization Technology. Bologa, Ana-Ramona and Bologa, Razvan. 4/2011, Bucharest : Academy of Economic Studies, Bucharest, Romania, 2011, Vol. 15.
  37. WILL THE REAL DATA VIRTUALIZATION PLEASE STAND UP? EvDBT. [Online] EvDBT, 16 November 2014. [Cited: 17 February 2015.] http://evdbt.com/will-the-real-data-virtualization-please-stand-up/.
  38. van der Lans, Rick. Channel: Data Delivery Platform – Rick van der Lans. BeyeNetowrk. [Online] BeyeNetowrk, 2015. [Cited: 24 February 2015.] http://www.b-eye-network.com/channels/5087/.
  39. Bumbill, Edd. The Data Lake Dream. Forbes. [Online] Forbes, 14 January 2014. [Cited: 24 February 2015.] http://www.forbes.com/sites/edddumbill/2014/01/14/the-data-lake-dream/2/.
  40. Roux, Hendriekus J. The Data Delivery Platform. Data Chatter Box. [Online] Data Chatter Box, 24 October 2012. [Cited: 24 February 2015.] https://datachatter.wordpress.com/2012/10/24/the-data-delivery-platform/.
  41. WikiPedia. Canonical. WikiPedia. [Online] WikiPedia, 14 February 2015. [Cited: 16 February 2015.] http://en.wikipedia.org/wiki/Canonical.
  42. —. Canonical model. WikiPedia. [Online] WikiPedia, 07 January 2015. [Cited: 16 February 2015.] http://en.wikipedia.org/wiki/Canonical_model.
  43. Cisco. Unified Data Center Fabric: Reduce Costs and Improve Flexibility. Cisco. [Online] Cisco, 2015. [Cited: 24 February 2015.] http://www.cisco.com/c/en/us/products/collateral/switches/nexus-5020-switch/white_paper_c11-462181.html.

Systems of Record and ERP

  1. INTRODUCTION

An enterprise in the twenty-first century is profoundly disposed to the utilisation of information technology systems to execute its business.  This execution is classified into three layers, viz (Pettey, Christy; Goasduff, Laurence, 2012):

  1. Systems of record—support the core of business transactions and manage critical master data with a slow rate of change, common functions between organisations and often subject to regulatory changes.
  2. Systems of differentiation—applications utilising unique company processes or industry specific capabilities, with medium-cycle change requiring frequent reconfiguration to accommodate changing business practices or customer requirements.
  3. Systems of innovation—new applications build adhoc to address new business requirements or opportunities, with a short life-cycle using departmental or outside resources and consumer-grade technologies.

These layers correspond to the notion of business leaders having common ideas, different ideas, and new ideas, as a means of classifying the supporting systems.  Systems of record (SSoR) are applications in support of the core business, while systems of differentiation create a competitive advantage in terms of process efficiency and effectiveness, and systems of innovation support strategic processes that enable and support business transformation.

Yvonne Genovese of Gartner says: “There is a gap developing between the business users of enterprise applications and the IT professionals charged with providing these applications.”  This mainly because Information Technology resources are working hard to standardise application suites for the least amount of integration issues, maximum security, and reduced expenditure while business users want modern, easy, and flexible applications that are quickly deployed to solve specific problems in response to a market opportunity.

In recommendation, Gartner suggests that in order to overcome this predicament, innovation must be accelerated by adopting a pace-layered application strategy, also referred to as shearing layers (WikiPedia, 2013).  Each stratum’s pace is defined by a specific application classification and enterprise differentiation.  Gartner’s Pace-Layered Application Strategy is a new methodology for categorising applications and developing a differentiated management and governance process that reflects how they are used and their rate of change, viz. Systems of record, Systems of differentiation, and Systems of innovation.

It is important to note that the strategy for selecting, deploying, and managing applications between these strata are different, because applications (based on how they are used by the organisation) are fundamentally dissimilar.  The pace layers can be used to build a business application strategy that delivers faster response and better return on investment without sacrificing integration, integrity, and/ or governance.

The enterprise must establish a new strategy for business applications that responds to the use of technology to establish sustainable differentiation and drive innovative new processes, while providing a secure and cost-effective environment to support core business processes.

This paper examines the positioning of systems in support of core business processes, i.e. Systems of Record, as classified by the Gartner pace-layered application strategy and how these should be thought of.

Many enterprises are presently disposed to utilising a variety of systems to perform its support services vested in the Systems of Record layer, e.g. Finance, Human Resources, Information and Communication Technology, Procurement, etc. some hosted in the company’s data centre while others are cloud services.  Each individual business process and functions should be matched to an application and classified according to the pace-layer application classification to allow for an elegant reference ability.

For example, financial accounting, order entry, and collaborative demand planning are often part of a single Enterprise Resource and Planning (ERP[1]) package, but are separate application modules that belong in three different layers in the Pace-Layered Application Strategy.  This approach should also be used to classify individually packaged or custom-developed applications.  This allows the organisation to apply the appropriate governance, funding, and data models, based on the characteristics of each application to the different applications to manage them optimally.

Inefficiencies in the current enterprise systems, mostly because of their heterogeneous nature, i.e. different vendor applications with disparate integration standards, result in elevated cost of ownership (through multi-vendor management), support resources (diverse architecture), and none or sub-standard integration that diminishes the availability of useful information, representative of the enterprise, as analytic information that can be used for decision support.

Ineffective integration isolates functions and renders silos of data that are not uniformly accessible for enterprise reporting and communications and requires extensive efforts to converge and present data as one enterprise information source.  Consequently business intelligence is not easily transformable for corporate performance support and management, i.e. in decision making processes.

Business performance (part of financial management and control) has three main activities:

  • Goal selection
  • Consolidation of measurement information relevant to the organisation’s progress against its goals
  • Interventions made by management in light of the information in order to improve performance against goals

Performance management, through organisational activities, requires the collation and reporting of volumes of data, using business intelligence tools that explicitly suggests software being a definitive component of the approach.  Decision making is based on the information from these corporate systems and rely on the accuracy and time-relevance of the information.

Although corporate solutions have been automated progressively, the collation of data remain a challenge due to the lack of infrastructure for data exchange or due to incompatibilities between support systems, or simply the lack of proper data integration between various systems.

Corporate performance management (CPM) builds on a foundation of business intelligence, according to key performance indicators such as revenue, return on investment, overhead, and operational costs and is the crucial capability by which to monitor and manage an organisation’s performance.  According to Gartner: “CPM is an umbrella term that describes the methodologies, metrics, processes and systems used to monitor and manage the business performance of an enterprise.  Applications that enable CPM translate strategically focused information to operational plans and send aggregated results.  CPM must be supported by a suite of analytic applications that provide the functionality to support the business processes, methodologies and metrics.” (Gartner, 2015)  A corporate performance system should include forecasting, budgeting and planning, graphical score cards and dashboards to display and deliver corporate information (Rose, Margaret;, 2011).  An enterprise wide corporate performance management cannot be had without business intelligence (Dresner, 2014).

If business intelligence is the key to corporate performance management, i.e. giving it context and it is a program that causes a shift in organisational culture, processes, and technology, it reaches all levels of the organisation and requires a revolution in organisational thinking.  Without a proper CPM strategy it would be impossible to monitor and manage the company’s performance and make it extremely difficult to know if the corporate strategy and goals have been attained.

This document proposes that CPM be formalised and complemented by an Enterprise Resource and Planning (ERP) system for system of record applications, classified according the Gartner Pace-layers System Architecture (Gartner, 2015), with a strong and efficiently implemented analytic and reporting ability for these providing business intelligence in real-time via dashboards and other data visualisation tools to the relevant decision makers in the company to empower them to make business decisions in order to attain to strategic goals.  These strategic goals could be come of the floowing:

  • Innovative Products and Services,
  • Rigorous Cost Understanding and Focus,
  • Technology Delivery,
  • Market and Client Intelligence,
  • Regulatory and Market Credibility,
  • Integrated, Customer-focused,
  • Collaborative Culture, and
  • A Diversified, Capable, Respected Workforce.

[1] Enterprise Resource and Planning (ERP): Integrated Application Suite on Best-practice Functionality.

1.1.                        Objectives

The objective of this article is to provide a motivation in support of a company CPM methodology, related business intelligence capability, an application classification structure for application governance, and a system of record utility suite for meeting the commoditised (mainly financial management) business processes requirements by utilising computing software that are deployed in the most optimal, modern, and cost effective manner.

1.1.                        Assumptions, Considerations, and Exclusions

Many organisations recognise that their culture is not innovative, its business processes are deficient, and many are executed manually in instances, and that its technology architecture is not optimally integrated for commoditised system of record applications.  Its work force may not be the adequately diversified, capable, and respected and that technology delivery is not optimal in many cases, most notably in the differentiation and innovative classification of the pace-layered system view of applications.  This status-quo causes arduous business processing, cumbersome technology changes, and problematic and error prone implementations and upgrades of software, errors in reporting and billing, and tedious and strenuous interactions with customers.  This erodes the company’s ability to adapt to changing market conditions and provide products and services that maximise profit and customer service.

Where a business intelligence strategy has been tendered and accepted by the executive committee, allowing for divisional key performance indicators (KPI) to be identified, this has to be employed before an Enterprise Resource and Planning (ERP) system and Corporate Performance Management (CPM) methodology are possible, both of which should bear on the defined KPIs.  A CPM strategy has strong dependence on a stable and analytical system of record suite of applications, e.g. an ERP system, to produce accurate data on which analysis is possible.  Then, to monitor performance indicators and controls that monitor the achievement of the company’s strategy, scorecards, vision, mission, and goals, as well as policies and business processes, to ensure that the correct state of the company is known at all times and that remedial action can be taken in time to align the business operations with achieving its goals.

1.1.                     Scope

The scope of this paper is to present the case for an ERP suite of applications with a strong analytic capability, and a complementary CPM requirement for the commoditised stratum of system of record applications required by the organisation.

A classification strategy, proposed by Gartner, i.e. the pace-layered classification of systems, for all the classification of applications in the company and to subsequently be applied to the selection of ERP components, especially in the postmodern view of ERP and its hosting options.

This paper does not recommend and ERP system, neither does it discuss any implementation pitfalls, but it proposes that an ERP system is the best solution for system of record applications.  It looks at the various problems with ERP and how that is being addressed in the postmodern ERP era.  The move to The Cloud is punted as the future for these system types and this paper explores the different options in overview.

A strong reliance on analysis, integration, and application governance relate ERP systems with business intelligence and corporate governance with financial management as the fulcrum.

1.  Document Specifics

1.1.                        Conventions

This document makes use of references to external sources of information and collates them in a citations addendum with intra-document references.  Any strongly typed words are for emphasis and quotations are in parenthesis with paraphrases in italics, both with cited references.  Lists are bulleted and steps or finite elements are numbered.

1.2.                        Audience and Reading Suggestions

The suggested audience is the Chief Executive Officer (CEO), the Chief financial officer (CFO), the Chief Information Officer (CIO), and Enterprise Business Architects (EBA).  This paper should be read as a strategy consideration to spark off a broader debate from which to obtain the business and technology drivers for a strategy and technology road-map that realise the corporate strategic objectives pertaining to the managing of Systems of Record, i.e. utility applications, specifically Enterprise Resource and Planning (ERP) application with business intelligence and corporate governance.

1.3.                        Definitions

The table (below) contains working Terms/ Acronyms and Definitions for any term used by this document that the reader audience may understand.  These include specific terms, abbreviations, and acronyms.

Term Definition
ERP Enterprise Resource and Planning (ERP): Integrated Application Suite on Best-practice Functionality
CPM Corporate Performance Management
BI Business Intelligence, i.e. a set of tools and techniques for transforming data into meaningful and useful information for business analysis purposes.
BAM Business Application Management
COTS Custom Off The Shelf
KPI A key performance indicator is a measurable value that demonstrates how effectively a company is achieving key business objectives. Organisations use KPIs at multiple levels to evaluate their success at reaching targets. High-level KPIs may focus on the overall performance of the enterprise, while low-level KPIs may focus on processes in departments such as sales, marketing, or a customer services centre (What is a KPI?, 2014).
PMF Performance Management Framework
Systems of Record (SSoR) The general use of the term means authoritative data sources otherwise known as databases, but also understood as ERP-type systems companies rely on to run the business, e.g. financials, manufacturing, customer relationship management (CRM), and human resources (HR). These have to be integrated and intact (from a data perspective) to be consistent.
Systems of Engagement (SoE) These perform corporate functions such as email, collaboration, social networking and learning to engage employees, especially from a human resources perspective. ERP systems like SAP have a facility called “Employee Central” and Oracle has “Fusion” with Microsoft (Richards, 2013) providing a host of applications in their Dynamics ERP suite through which employees are engaged in what is called “talent management” systems (McCann, David;, 2012)
SaaS Software as a Service is a software distribution model in which applications are hosted by a vendor or service provider and made available to customers over a network, i.e. the Internet.
IaaS Infrastructure as a Service is a Cloud service comprised of highly automated offerings of computing resources, complemented by storage and network to customers on-demand (Mitchell, David;, 2012).
PaaS Platform as a Service is an offering between the SaaS layer above and IaaS layer below comprised of a broad collection of application infrastructure (middleware) services, including application platform, integration, business process management and database services.
BPaaS Business Process as a Service is the delivery of business process outsourcing services that are sourced from the cloud and constructed for multitenancy.
Multitenancy Refers to a principle in software architecture where a single instance of the software executes on a server, serving multiples tenants, i.e. a group of users sharing the same view on the software they use.

1.  Executive Summary

A financial management & control service is a key function in the organisation to control business from a financial and operational perspective.  It relies on many aspects, e.g. business processes, controls, performance indicators, available and relevant data, and information technology systems.  Financial management functions are commoditised and should be standard practice across industries globally, with slight regional nuances.  Consequently the company should view such functions as common and utilise standardised applications, or a suite of applications, such as ERP, and augment it with a strong business intelligence capability, in fulfillment of a corporate governance methodology to monitor and manage the company’s performance and alignment to strategic goals and objectives.

Making corporate performance management a successful endeavour will optimise the opportunity for the company to attain to a competitive, cost optimised, and agile to change organisation; but this requires a methodology and systems.

Simply looking at finance in isolation would be short-sighted because with the advent of enterprise resource and planning systems (ERP) more and more business functions have been assimilated into this fabric and the postmodern ERP landscape has enabled a plethora of different suite compositions and deployment options.  Not only that, but internal ERP processes have been automated; and different vendor ERP components provide seamless ERP integration via APIs, making inter-component integration effortless and part of the solution, but loosens the dependency on the mega-vendor paradigm of ERP.

ERP systems remove a great deal of effort and error from systems of record applications and recently with the postmodern ERP concept allow for much more flexible deployment options in the cloud, hybrids, and multi-vendor components.  However, to be effective in the management of corporate performance, an ERP system is not the panacea to all performance measurements and controls, but requires the ERP to have a strong business intelligence capability for the data stored in the ERP system, in real-time, in a relevant context, visually and dynamically presented across the enterprise by which to make agile business decisions.

Organisational strategies and scorecards are part of corporate performance and have to be empowered by the ERP system in combination and as a result of the BI capability, to fulfill corporate performance and allow the performance, overhead, and operational costs of the company to always be known at any instance in time.

Therefore, the elements towards a lean organisation, capable of delivering relevant and contextual decisions, operating model changes (in order to adapt to market conditions) in an economic and timely fashion are:

  • Business intelligence strategy implemented
  • Relevant and functional key performance indicators
  • Relevant and functional business processes and policies
  • Enterprise resource and planning solution that is integrated with a complementary business intelligence solution
  • A defined corporate governance methodology making use of the business intelligence of the company

1.  Gartner Pace-Layered System Architecture

Gartner produced the concept of the Paced-layer view of applications (Figure 1) used for categorising applications (Figure 5 ) and developing a differentiated management and governance process that reflects how these are used and what their rate of change (Figure 6) is.  This strategy addresses the problem that most organisations have, i.e. a heterogeneous portfolio of business applications of different technologies, different purposes, some as simple as spreadsheets, compounded by the fact that the proliferation of this is getting worse and not better.  The efforts to converge technologies and solidify processes and move towards a better release cycle are not appreciated (understood) by the business (Syx, 2011).  These layers are used to build a business application strategy that delivers a faster response and better return on investment without sacrificing integration, integrity, and/ or governance, but actually promote it.  This paper regards the Systems of Record (Figure 4), i.e. those utility type common industry applications (in support of the core business) in separation but in support of the other classifications, viz. Systems of Differentiation and Systems of Innovation.  The pace-layered approach is not static, applications will move up and down over time, but they must be classified at all times to allow proper and appropriate governance.

Gartner Pace-layered view of Applications

Figure 1–Gartner Pace-layered view of Applications

To obtain a pace-layered classification, application portfolios are decomposed into individual applications and the business processes that are in support of each.  The characteristics of these determine the stratum of the pace-layer model to which the application and process belong.  Applications belonging to the Financial Management application suite are part of the systems of record and together with connective tissue[1] (refer to SAP’s View of Gartner’s Pace-layered view of Applications [Figure 2] on page 17) form the foundation, usually a suite of core financial applications, e.g. general ledger (GL), accounts payable (AP), accounts receivable (AR) and fixed assets(FA) of the enterprise application architecture.  It is necessary to ensure a strong foundation for systems of record be in place, with a ten year degree of comfort that these will be able to support current and projected business activities (Rayner, Nigel; van Decker, John E;, 2011).

[1] Connective Tissue such as master data management and process integration

Systems of Record are comprised of a commoditised and stable, integrated, application suite, as a functional core with infrequent changes, a long life-span that is a functionally well-defined and known across organisations, with high data integrity and shared across the enterprise as a utility (Table 2).  Applications may migrate between pace-layers and systems of record may have elements of differentiation as part of the suite/ configuration as they become commoditised and common in function within the industry.

Characteristic Record Differentiation Innovation
Strategic Focus Improve execution Competitive differentiation Business transformation
Lifespan 10-20 years 3-5 years 12 months to 3 years
Pace of Change Infrequent and incremental More frequent, configurability is key Rapid deployment of new capabilities
Business Process Understood and stablel Understood and dynamic Ambiguous and dynamic
Data Integrity High High Moderate
Support Requirements 75% technical, 25% business 50% technical, 50% business 25% technical, 75% business
Sourcing Integrated applications suite Best-of-breed or suite vendor extension Specialist niche vendor, possibly larger vendors
Budget Shared-service/ IT Mix of line of business and IT Line of Business
Investment Capital Capital and/ or expense Capital and/ or expense

Connective technologies and governance (Figure 2) address an overarching (common) base of elements that span the different layers. The connective tissue allows for interaction between applications (Colbert, 2011) in their layers, through master data management, process and data integration, business service repository (SOA), integrated composition technology (Syx, 2011), a common security architecture that is flexible enough to meet the requirements of all the layers, integrated monitoring and management, and external connectivity. This connective tissue should have definite standards. Governance across layers suggests that there is no one solution for all and requires a governance model for each layer. Realistic process and data integrity rules should exist across the layers by categorising the existing portfolio in to individual applications and associate each with a grouping and process (by a panel of users and IT application experts) and then assign the application to the correct layer based on its characteristics, e.g. the rate of change and finally establish connective technologies to facilitate interoperability. Governance is adapted to suit each layer accordingly.

SAP View of PACE

Figure 2–SAP View of PACE

Gartner provides a guide for the classification of applications according to the pace-layer view of applications (Figure 3) classifying a few commoditised functions as system of record applications.

PACE with Application Types

Figure 3–PACE with Application Types

To understand an example of classification consider the financial accounting process (at a high level) for the supplier invoice to payment process, that it is a non-differentiating process well supported by an ERP system.  However, at a more granular level sub-processes and activities, within invoice to payment, offer opportunities for differentiation, even innovation, such as e-invoicing that removes paper from the process altogether.  By using the pace-layered approach, organisations can move away from standard ERP functionality by soliciting specialist vendors that provide e-invoice capabilities on-top of the ERP system (Rayner, Nigel;, 2012).  Similarly the paced-layered approach assists in the classification of ERP functionality in such a manner that governance pertaining to the strata is custom for the stratum.  This helps the management of the ERP life-cycle by being responsive to demands for differentiation and innovation and the different rates of change in the application portfolio as opposed to treating the application portfolio as a singular entity.  Systems of record should be managed on a seven year cycle and planning horizon.

PACE System of Record View

Figure 4–PACE System of Record View

System of Record applications (of which financial applications are vital) have a strong affinity to ERP systems (Figure 3) because of the industry common (non-differentiated) functions provided. Gartner says that the world is finding itself in the “third era of enterprise IT” (Gartner, 2014) or in relation to ERP a post-modern ERP era. The movement saw the industry moving from IT craftsmanship (with sporadic automation and innovation) to IT industrialisation (services and solutions efficiency and effectiveness) to standing on the precipice of moving into Digitalisation with digital business innovation and new types of value. Classic computer application systems provided a classic set of best-of-breed applications, driven by market and user immaturity, resulting in fairly happy users but with integration and analytic problems. The ERP paradigm sought to resolve this by centralising IT, driven by vendors, leaving users increasingly frustrated with integration trumping agility and fit to requirements. Postmodern ERP is set to deconstruct the ERP suite into multi-vendor units handing back the control to the users and to provide new integration and analysis for this fabric with modern connectivity (mobile and the like) with social platform integration (Figure 5).

PACE View of Application Strata and Types

Figure 5–PACE View of Application Strata and Types

No single strategy or governance model can be appropriate for all applications and the problem worsens with cloud and SaaS (software as a service) options are utilised. The greatest benefits from System of Record systems come when they are implemented using out-of-the box functionality to drive greater efficiency from transaction processing. These provide a stable core for recording and managing all financial activity of the organisation. An integrated suite of financial applications typically make up system of record applications for finance. Sub-ledgers should only be deployed if these are classed as systems of differentiation. Such financial systems should be scoped to last for ten years or need to be stabilised that they do and that they support regulatory changes that may come during that time. Vendor upgrades must be anticipated by viewing their road-map for the ERP product and also how it impacts the company’s current release level. Define a clear strategy for finance systems of record as documented and agreed by the organisation and only review if there is a significant and unanticipated change in business conditions or the scope of the organisation’s business activities. Standardise on business processes for systems of record across the organisation and support them with industry standards and best practice. Because the rate of change is slow for systems of record, a long term investment in applications is possible. Where business processes are required to be dynamic to support the business strategy, these are likely systems of differentiation or innovation and not system of record processes and should be properly classified in the pace-layered scheme. Where the ERP system of record components interact with other (extra) system components, the interaction must be properly defined.
The characteristics of the pace-layered view of systems as depicted in Table 2 is also graphically displayed in Figure 6 and indicate a few other dimensions such as Development practice, business engagement, and architecture.
Systems of record are typically the more stable, standard (commoditised), non-volatile, and utility type applications that does not differentiate one company from another.

PACE Governance View of Applications

Figure 6–PACE Governance View of Applications

5. CORPORATE PERFORMANCE MANAGEMENT
Corporate performance management (CPM) should not be viewed as synonymous with financial performance management (FPM) or Financial Management and Control. CPM is an internal organisational revolution and it provides context to business intelligence (BI).
CPM should provide and organisation, its employees, and customers with corporate performance dashboards, intelligent payslips or invoices that reflect elements of the vision, strategy, goals and objectives and how these are being reached. It should provide managers with the ability to make fact-based decisions faster and with a higher degree of confidence. A company may opt to use a CPM system coupled with an ERP system or stick with spreadsheet and manual data extraction processes. Both options can perform the requirement, but to provide the capabilities along with flexibility, advanced analysis, higher accuracy, faster, and with a better decision making, requires specifically integrated components that work together as one.
“CPM is an umbrella term that describes the methodologies, metrics, processes, and systems used to monitor and manage the business performance of an enterprise. Applications that enable CPM translate strategically focuses information to operational plans and send aggregated results. These applications are also integrated into many elements of the planning and control cycle, or they address business application management (BAM) or customer relationship optimisation needs. CPM must be supported by a suite of analysis applications that provide the functionality to support these processes, methodologies, and metrics.” (Gartner, 2014).
Finance is the source of enterprise key performance indicators such as revenue, return on investment, overheads, and operational costs (Rouse, 2014) that are used in determining business/ enterprise performance through corporate performance management. CPM processes typically are (Tagetik, 2014):

  • Budgeting, Planning and Forecasting
  • Cash flow Planning
  • Production cost, Planning Control
  • Statutory and Management Consolidation
  • Cost Allocation and Profitability Analysis
  • Financial Close and Fast Closing
  • Dash-boarding

Corporate Performance Management should be able to link operational activities with the business strategy without requiring complex integration. It should use pervasive and detailed budgeting, planning and forecasting processes, across the enterprise. Communications and collaboration across multiple finance processes should be facilitated. Link activities and task links to the budget objectives and actual performance. Improve control and workflow or activities and processes across the enterprise and unify and consolidate, reporting, budgeting, and forecasting processes. Adapt new business needs quickly and without much effort and manage external compliance, disclosure, and financial reporting from a single source. Reduce cycle times and iterations of financial processes to enable finance to take on more.
5.1. Measuring Performance
Many organisations produce corporate performance information from the finance department using:

  • Combinations of spreadsheets, custom applications (in combinations with other COTS products) to manage the budgeting process, consolidate results or provide rudimentary analysis
  • Spend great amounts of time developing, copying, and formulating data into spreadsheets
  • Scramble to provide information, often switching between a number of different applications, with reports lacking consistency and accuracy
  • Struggle to respond quickly to new regulatory or statutory requirements

The only viable and effective manner in which to measure performance is through the corporate financial and operational planning processes whereby any number of users can consume and contribute information without adding complexity, by identifying areas where increased collaboration can improve business analysis and manage performance more effectively and to plan at the required level of detail, no matter how granular or frequent. The world is moving towards an increasingly more data-driven culture, with an ever expanding participation and a more sophisticated data analysis requirement for providing more useful business insight for increasing the organisation’s abilities.
Financial management applications manage finance processes. These are the primary responsibility of the CFO and the office of finance, but others such as Corporate Performance Management (CPM) suites have a wider application in an organisation. Gartner provides the main application segments as follows:

  • Core financial applications—part of the ERP market including visualisation applications that provide insight into the enterprise financial positions through automation and process support for any financial activity, e.g. general ledger, accounts receivable, and project accounting. These also provide financial reporting data as needed by local and international regulations.
  • Corporate Performance Management (CPM) suites—processes used to manage corporate performance, like strategy formulation and budgeting, planning and forecasting, as well as the methodologies in support of these processes, like balanced score-card, or value-based management, and the metrics used to measure performance against strategic and operational performance goals. It’s also made up of analysis applications, e.g. reporting solutions targeted at the CFO and the finance team, senior executives and corporate-level decision makers.
  • Financial governance applications—focused on improving financial processes and controls, particularly prior to disclosure, i.e. close management and disclosure management.
  • Financial value chain (FVC) applications—internal cash management, supplier-facing and client-facing applications. Specialist solutions boost core ERP functions in this discipline, e.g. accounts payable invoice automation, collection management, and corporate treasury.

Corporate performance management has three main activities (WikiPedia, 2014):

  1. Selection of goals
  2. Consolidation of measurement information relevant to the organisation’s progress against said goals
  3. Interventions made by managers in light of this information with the view of improving future performance against goals

Measuring the corporate’s performance is partially vested in the Finance Management and Control function (Figure 7), comprised of the functions Internal Audit, Management Accounting, Financial Processes, and Financial Applications.

Financial Management & Control Business Functions

Figure 7–Financial Management & Control Business Functions

PWC Corporate Performance Framework

Figure 8–PWC Corporate Performance Framework

5.1.1. Internal Audit
The Internal Audit business function may involve a range of activities, however the essential element is ensuring that controls operate as designed and intended. All finance and organisational activities are subject to internal auditing. It is consequently not essentially a finance function but a financially inclined counterpart. However, the internal audit function requires a significant amount of depth and understanding of the company’s strategy and operations.
Internal Audit comprises the examination, monitoring and analysis of activities related to a company operation, including its business structure, employee behaviour and information systems. An internal audit is designed to review what the company is doing in order to identify potential threats to its health and profitability, and to make suggestions for mitigating the risk associated with those threats in order to minimise costs.
This ties in with the company’s risk management because an internal audit helps the company identify issues before they are substantial problems. An internal auditor has to challenge current practice, champion best proactive and be a catalyst for improvement with the objective of ensuring that the company holistically can achieve its objectives. Internal audit advises management (via audit committee) by providing assurance that the company is able to meets its objectives, its governance, risks and controls and serves as an input back into the strategic planning, market analysis, compliance, change management and the use of information technology.
Internal auditing is responsible for:

  • Evaluating controls and advising managers at all levels
  • Evaluating risks, e.g. not only control based but proactively
  • Analysing operations and confirming information, e.g. aligning operations with strategy in view of performance monitoring from financial controls
  • Review compliance, e.g. laws, regulations, policies, guiding principles

5.1.2. Management Accounting
The Management Accounting business function is a catch net comprised of a wide range of techniques designed to obtain decision orientated information, remedy certain business conundrums, and to use these techniques for making organisational management decisions, based on financial information, to provide expertise in financial reporting and control to assist management in the formulation and implementation of an organisation’s strategy. Management accounting also assists in the formulation of policies and in the planning and control of the operations of the company. Management accounting is therefore much more concerned with forward looking decision making that affect the future of the company. The key difference between management accounting and financial accounting is that the former uses financial information and aids managers in proactive decision making, while the latter reports financial information.
Management accounting is broken into three applications:

  1. Strategic management
  2. Performance management (business decision making and monitoring of controls)
  3. Risk management (report risks to the achievement of objectives)

5.1.3. Financial Processes
The Financial Processes business function is a Financial Management and Control function and these processes include the development, production, and analysis of information used for management and control purposes.
Both financial and non-financial information are used. Financial information used is planning, forecasting, budgeting, target setting, analysis and reporting. Actual data analysis against futuristic analysis provide a basis for action and the next cycle of forward looking planning. The content and scope are depended on organisationally defined aggregations and analysis of financial data and transactions, covering both the past and the future. A link is found between the underlying organisation activities and the financial information in support. Non-financial information relates to information such as customer feedback (mood analysis) and aspects of the business score card. Such information may be combined with financial information to provide guidance to financial management and control.
5.1.4. Financial Applications
The Financial Applications business function (not IT applications) relate to how information is used for management and control, i.e. how information is aggregated and analysed (profit- or cost-based) with the following principle applications:

  • General management and control of operational activities, generally with an implicit focus on profitability or value of money goals.
  • Cash management and treasury has two aspects. Firstly, as a key performance dimension through company valuations based on the net present value of projected future cash flow. Secondly, as an asset class to be managed through treasury activities.
  • Investment appraisals seek to assess the merits of investment options, potentially using both cash approaches, e.g. net present value and internal rate of return and profit-based appraisals, e.g. return on investment.
  • Tax management involves the consideration of a range of factors such as decisions on transfer pricing, legal entity structures, operating locations and financing structures. The focus is prudentially minimising tax liabilities.

5.1.5. Business Performance Framework
5.1.5.1. THE OBJECTIVE OF A FRAMEWORK
The balanced score card is often used as the basis for business performance management activity within organisations. The corporate strategies are diffused into actionable metrics and analysis are used to expose the cause-and-effect relationships that could provide insight into decision-making. It is but one framework employed implementing a business performance management system, but the most widely used. Others are Six Sigma, Activity-Based Costing, Total Quality management, Economic Value-Add, Integrated Strategic Measurement, and Theory of Constraints (WikiPedia, 2014).
The identification and definition of key performance indicators (KPIs) is the first to establish for attaining to a CPM system. “A KPI is a set of quantifiable measures that a company or industry uses to gauge or compare performance in terms of meeting their strategic and operational goals. KPIs vary between companies and industries, depending on their priorities or performance criteria. Also referred to as Key Success Indicators (KSI).” (Investopedia, 2014).
A performance management framework (PMF) endeavours to initiate, implement, and sustain performance management through sequential performance management cycles taking principles into practice for all stakeholders converting experience into a capacity for learning and improving to create accountability and produce better results for the company shareholders and employees.
Simply superimposing a performance management process onto a traditionally managed company may sound good, but practically won’t make any difference. A real improvement for the company would also see the company culture being addressed. A performance management paradigm should be implemented as an ongoing, systematic approach to improving results through evidence (outcome) based decision making, continuous organisational learning, and focus on accountability for performance. It must be integrated into all aspects of the company’s management and policy making processes, transforming the company’s practices so it is focused on achieving improves results for its customers, shareholders, and employees.
A delineating separation between performance measurement and management must be made. Performance management systematically uses measurement and data analysis as well as other tools to facilitate learning and improving and strengthen a focus on outcomes. The company is required to move from measuring and reporting to managing and improving results.
Accurate and timely data must be shared by everyone at the same time, according to agreed times. To accelerate learning (knowing the environment’s performance and have to ability to commend remedial actions) regular meetings should be conducted to reflect on the performance information. A problem-solving model that works for the company must be established as standard approaches to work effectively within the culture and outcomes measured against benchmarks. A basic performance evaluation includes the following phases:

  • Defining the question
  • Establishing a data collection strategy
  • Collecting the data
  • Analysing and reporting conclusions

Evaluation must be a component of performance management because understanding the relationship between the activities the company carries out and the results it achieves is necessary of learning, improvement, and accountability.
Two examples, from industry specialists (Figure 8 & Figure 9), are provided for a quick reference about how a business performance framework should look like.
5.1.5.2. TYPES OF EVALUATION
Evaluation distinguishes approaches that focus on improvement and accountability, and distinguishes between formative and summative evaluations. A formative evaluation provides a feedback loop to individuals on how much a program execution differs between individuals and has the goal of making appropriate adjustments. Summative evaluation focuses on whether or not a program is successful and the intention is about whether to terminate the program or modify it. The data collection between these is different.
There are also distinctions made between evaluations aimed at accountability and those intended for learning and improvement. Accountability evaluations are usually referred to as audits and learning and improvement evaluations provide the how and why information. Knowing what decisions to make requires knowledge on which program is being evaluated.
A measurement (control) is a practice used to develop, collect, store, analyse, and understand performance, including indicators of workload or activity, effectiveness, efficiency, and actual results or improvements. Reporting communicates performance measurement information to audiences including internal staff (employees), management, and executives, along with other organisations and interest groups and rating agencies and the public. Performance measures provide factual information used in decision making for the planning, budgeting, management, and evaluation of company services. Measures can inform decision makers on a wide variety of topics. A control must track a useful measurement.
5.1.5.3. PRINCIPLES OF PERFORMANCE MANAGEMENT
The framework is established on seven principles that help transform and unite the company processes, e.g. planning, budgeting and forecasting, management, and evaluation. This condensed into a single system, well aligned and integrated, for improving results. These principles are:

  1. A results focus permeates strategy, processes, organisational culture, and decisions
  2. Information, measures, goals, priorities, and activities are relevant to priorities and the well-being of the company
  3. Goals, programs, activities, and resources are aligned with priorities and desired results
  4. Decisions and processes are driven by timely, accurate, and meaningful data
  5. Practices are sustainable over time and across organisational changes
  6. Performance management transforms the organisation, its management, and the policy making process

5.1.5.4. IMPLEMENTATION STEPS
It is not a momentary action that will bring about the corporate performance management framework in a standard manner for business conduct. However a few key initial aspects can accomplish great inroads towards the holistic strategy, viz. performance driven planning, changing the budgeting process, and training managers and employees on using data to improve programs and services.
Consequently the following steps are representative in the establishment of a CPM framework:

  • Expose the case for CPM to appropriate decision makers for support, authorisation, and to secure resources
  • Identify key purposes and objectives of initiating performance management
  • Define the performance management process

5.1.5.5. ELEMENTS OF THE PERFORMANCE MANAGEMENT FRAMEWORK
A CPM system should be comprised of a cascading alignment to align objectives, strategies, and measurable controls to the needs of the members, shareholders, and employees, followed by programs and services, and lastly individual outcomes (Rose, Jerry R;, 2014).
The following elements are typically included:

  • Planning process that defines the organisational mission, and priorities that drive performance
  • A process for engaging customers and shareholders to identify needs
  • A budget that allocates resources according to priorities
  • A measurement process that supports the entire performance management system
  • Accountability mechanisms
  • A mechanism for collecting, validating, organising, and storing data
  • A process for analysing and reporting performance data
  • A process for using performance information to drive improvement
PWC Corporate Performance Framework

Figure 8–PWC Corporate Performance Framework

Accenture Enterprise Performance Management Framework

Figure 9–Accenture Enterprise Performance Management Framework

6. ENTERPRISE RESOURCE AND PLANNING
The primary achievement for Enterprise Resource and Planning (ERP) systems is integrated processes. The integration characteristic yields great integration benefits and good integration brings homogeneity in the representation and handling of enterprise data. Business performance management (BPM) relies on business intelligence, without which it is impossible and without BPM and enterprise cannot be competitive and its product and service offerings are average and slow to change with market variations.
“In many respects, companies have been facing a nasty storm when it comes to their standalone (un-integrated) enterprise systems: Many companies are over-softwared right now, and there’s been a substantial backlash and pleas for real-world, usable innovation. With analytics, in particular, surveys have shown that 40% of executives still trusted their gut in decision making and many more are frustrated with CIOs and IT for failing to give the business what it needs and deserves with analytic and decision-making tools. In the Aberdeen report, the analysts preach an integration mind-set when it comes to a more perfect ERP and BI union. A good first step: Form cross-functional teams for both ERP and BI projects. When left entirely to IT, the success of projects is often measured by cost and speed of implementation, write Jutras and Hatch. These are important factors but using them as the exclusive measure of success loses sight of the original business goals of the project.” (Wailgum, Thomas;, 2009).
ERP consequently is a software solution that addresses the enterprise needs, taking a process view of the overall organisation to meet the goals, by tightly integrating all functions under a common software platform. It achieves a transparency of information using a single data source across the organisation while rendering software that is modular, flexible, and easy to add functions and provide a growth path. Coupled with business intelligence, getting the right information to the right decision makers at the right time, the ERP system provides an enterprise wide platform that supports reporting, analysis and decision making for fact based decision making and a single version of the truth including reporting and analytics. It remedies the data everywhere but information nowhere situation, experienced with non-integrated systems. It relates the ERP (with its core CRM system) to the BI capability, where the majority of the data lies in the ERP system and when the ERP solution integrates business intelligence (as opposed to simply having a reporting function) managers can benefit from deeper business insights. From customisable dashboards and graphical charts, business intelligence reports gather, compare, share, and analyse real-time data. BI applications drive business process improvement across an enterprise. ERP systems, since 2012 (Carr, 2014) have become more and more business intelligence proficient by doing strategy management, interactive dashboards, scorecards, comparing current to historic business activities and future plans to deliver contextual information to keep performance on track. Essentially, the goal is to use ERP data to take action, which is only possible if the relevant data gets in the hands of those able to make improved business decisions as information. The most important moment in a piece of data’s lifetime is the moment it is created and the moment it is used. If either or both of these moments happen outside of IT, i.e. some system, the event is not accessible for the purpose of opinion or business intelligence. To harvest value from data and control its quality, IT systems have to control those movements, but IT cannot fix a substandard business process, all it can do is perform data fixes at a very high cost (Redman, 2012).
For an ERP system (both its IT infrastructure and applications and the business processes) to work efficiently and provide that optimal yield of the power of integration a variety of management has to be unified. Different kinds of assets, people, capital, technology, and data demand different kinds of management. The responsibility of data must be moved out of IT and into business and leave IT to run the systems in which the data, according to business processes, are to be captured and used (Redman, 2012). However, an ERP system by itself won’t provide better access to data. That is reliant on a business intelligence strategy. The ERP system may hold the data and provide the software tools and reports, but not the strategy. A BI strategy must be in place prior to acquiring an ERP solution.
An ERP system cannot be used to examine BI needs, but instead the BI strategy must be assessed first and through that improve access to information related to the ERP system.
However, once a BI strategy is in place and support business processes and key performance indicators are known, an ERP system (with its superior integration) covers the data capture and delivery paradigm in an automated fashion without which the BI capability would suffer detrimentally in answering in making data available in real-time for relevant and effective business decisions.
One of the biggest criticisms leveled at ERP systems is that they are inflexible and do not support business change. According to Gartner (Rayner, Nigel;, 2012) in most cases the perceive inflexibility of the ERP suite is attributable more to how the ERP applications have been purchased and deployed than to any inherent flaws in the technology. The Gartner pace-layering model application to ERP systems has a positive impact in building a balanced application portfolio of integrated suite applications sourced from a strategic\ specialist ERP vendor.

PACE for ERP

Figure 10–PACE for ERP

Most companies find themselves within an application environment that had grown over time as a series of incremental projects making the technical architecture more complex without yielding obvious strategic benefits. This paves the notion to “go ERP” without clear goals. The application strategy is often then dictated by the ERP vendor and not by the organisation’s business strategy. ERP systems are primarily Systems of record (Figure 11) from a macro viewpoint and deliver standard processes and create data and process foundation for most other business activities (Rayner, Nigel;, 2012).
The Gartner Pace-layered approach to application strategy suggests that ERP is only a part of the answer to business requirements for systems of differentiation or innovation, but ideal for systems of record. ERP systems are only effective once common processes have become established across an industry or industries. The main aspect of any ERP strategy is to define a business process and functionality scope to delineate a boundary for the ERP system using the pace-layered approach.
ERP applications should be governed by a ten to fifteen year planning horizon, meaning that companies should have a clear long-term capital funding plan, including funding for upgrades that are likely within that period. ERP should be managed as a foundation systems and replacement should only be considered when there is a business strategy change in the original decision factors in support of the ERP system.
Most organisations will have systems of differentiation or innovation on the ERP boundary and any elements of the ERP that fall outside of the systems of record should be managed differently. An ERP system that spans pace layers require a more nuanced approach to life-cycle management.
A primary argument for suite-based solutions (ERP) is that they provide the necessary integration, especially for maintaining the consistency of financial data. The audit-ability of financial record-keeping is a fundamental principle of a well-managed finance operation. Integrated solutions maintain both process and data integrity.

Systems of Record in PACE

Figure 11–Systems of Record in PACE

The ERP landscape is a fast changing one even for Systems of Record. Gartner terms this move into the digitised era of computing as the postmodern era for ERP systems. This mainly bears on technologies moving into the cloud and assuming a more service orientated architecture with business process integration between components through open API standards.
One of the main perks of an ERP system is the financial management capabilities (Tagetik, 2014) and should improve financial management as follows:

  • Compliance—provide tracking and audit trails
  • Accurate financial positioning—dashboards and key performance indicators provide a running overview of the financial standing of the company in real-time allowing informed decisions
  • Reduce accounting overhead—automated system processes for daily accounting tasks and third party software integration

1.1.                   Post-modern ERP

Post-modern ERP came about in the late 2000’s where mega-vendors are no longer dominant and in control and the users of ERP are seeking more control amidst new integration and reporting challenges (Guay, Mike;, 2014).  In our consideration of ERP for System of record applications, we have to acknowledge the positioning of ERP in our era.

In a post-modern ERP view (Figure 12) the mega-suite ERP has been overwhelmed by equivalent cloud specialist vendors who only develop a subset of the former megalithic ERP suite and do so really well.  Cloud specialists build these offerings at speeds that no other conventional vendors are able to match.  Loosely coupled ERP suites are emerging in the cloud, based on developed and acquired products plus Platform as a Service (PaaS) offerings.  Integration complexity between these products are hidden in the cloud by cloud product vendors.  Consequently post-modern ERP systems can be viewed as loosely coupled, federated, increasingly cloud-based, rented facilities.

Ultimately most ERP functions are likely to be consumed through the cloud from Business Process Outsourcing (BPO) providers or as cloud services.  However, because 70% of transformation projects fail, any post-modern ERP option will disappoint if organisations don’t get a lot better at changing.

ERP value can only be measured positively if there is a measurable improvement in business performance.  Value can only be delivered if the company is able to embrace change (culture and politics) and adopt design agility and flexibility and link changes to business value.

For an ERP success the business must have a deep and broad understanding of the strategy and business priorities, interconnections, develop (or acquire) change management expertise, match business and IT agility, and change to focus and efforts to produce value.  It takes time to partner with business stakeholders and over-arching goals should be defined to achieve success together, i.e. working together is the key to success.

Post-modern ERP HOOF Model

Figure 12–Post-modern ERP HOOF Model

ERP systems are no longer a single product but provide a nexus of offerings ranging Information, Cloud, Social and Mobile components. These systems are comprised of public could applications for people, customers, finances, suppliers, and business with other partner and internet service provider applications, complemented by databases, infrastructure, and platforms for analysis, e-commerce, and self-service portals (Gartner, 2014). It is essential of the companyto recognise a plan for the total lifetime cost of ERP solutions rather than focus on short-term tactical costs. Social technology integrations into traditional applications are taking place slower than anticipated and procurement decisions for applications are moving away from IT and into business (Ganly, Kyte, Rayner, & Hardcastle, 2013). ERP implementations benefit organisations more about substantial changes to business processes and changing working practices than the actual benefits of ERP and many organisations had spent ten times more on customisation than on licensing costs. Inflexibility in systems prove to be the biggest bane of ERP in the enterprise and according to Gartner, any system that is not sufficiently flexible to meet changing business needs is a legacy system.
The two most common flavours of cloud ERP being offered are:

  • Hosted with application management services—Internet based, limited elasticity; logging for active users only; dedicated hardware or some level of virtualisation; traditional features-based ERP implementation with extensive configuration; perpetual license or subscription based on a fixed number of users and modules; some infrastructure services; fixed location of data; limited support of migration off the solution; and full control over upgrade cycles and software changes by the end user’s organisation.
  • Multitenant Saas ERP—Internet based access; level of elasticity can be chosen, but typically with some limitations; flexible payments within pre-set boundaries or minimum fee if not used; fully shared application logic; product features prevalent, but subscription of software services; underlying solution fully managed by provider; floating location of data, but mostly between pre-set data centres; limited support of migration off; limited influence on upgrade cycles; and software changes fully controlled by provider or limited possibility to postpone software changes by provider. This is the combination that most SaaS ERP vendors in the market offer.

1.1.1.                Software as a Service (SaaS)

Software as a Service (SaaS) is a software distribution model in which applications are hosted by a vendor or service provider and made available to customers over a network, typically the Internet and is what is commonly referred to as The Cloud.

Many ERP vendors like Netsuite, SAP ByDesign, Intacct, SalesForce.com, Workday, and Sugar CRM are typical SaaS providers in the cloud.

Presently, ERP application leaders are struggling to respond to fast changing business and line-of-business users are increasingly bypassing an IT-led ERP strategy by licensing SaaS applications from specialist vendors to overcome this obstacle.  This however, creates a new generation of silo SaaS applications with fragmented data and processes.  Applications like procure-to-pay, talent management, and travel expense management is disrupting ERP suites.  This has the potential to create silo applications with an integration nightmare similar to the classic best-of-breed environment applications before ERP.  Consequently, ERP and other application leaders are starting to manage a portfolio of on-premise and SaaS applications rather that a single ERP suite, i.e. postmodern ERP.  For such hybrids (on premise and SaaS combination) it is important to balance between integrated suites and specialist SaaS applications and apply best practices to best support a hybrid ERP environment.

The ERP vendor SAP (being the market leader according to Gartner in 2013) says that it has the most comprehensive cloud computing portfolio on the market with 35 million users (Wilson, 2014).  Gartner also redefines highly customised ERP systems as legacy by 2016 (Meyer, 2014) because it is not sufficiently flexible to meet changing business needs.  This results in the notion that over the next ten years most organisations will transition their legacy ERP systems to postmodern ERP systems, i.e. into the cloud by 2018.  In the interim the hybrid model is the most expedient as SaaS providers enhance the ERP cloud functionality, however Gartner says that by 2017, 70% of organisations that had adopted a hybrid model will fail to improve the cost-benefit outcomes unless cloud applications provide differentiating functionality and even predict that a hybrid model may increase costs.

The development of sophisticated cloud-based best-of-breed point solutions with strong integration capabilities, combined with very strong growth in business process outsourcing, will provide ample opportunities for organisations labouring under inflexible and expensive ERP modules to seek alternative means of supporting their business processes.  However, any System of Record Application should require little to no customisation, while differentiating processes and innovative activities will use alternative delivery models that are integrated to the ERP system of record capabilities (Ganly, Kyte, Rayner, & Hardcastle, 2013).  Gartner suggests that companies with a revenue base of less than $1 billion should review their ERP strategy to see if there is an opportunity to leverage cloud ERP to reduce the costs of managing, maintaining, and upgrading ERP while at the same time delivering equivalent or improved functionality.

1.1.2.                Infrastructure as a Service (IaaS)

Infrastructure as a Service provides resources, assets, IT equipment and computing on demand, as a substitute for on-premises infrastructure hardware and IT expertise, software, and manpower, but outsources and hosted in the cloud.

1.1.3.                Platform as a Service (PaaS)

Platform as a Service features a web-based platform (hosted on a secure cloud) to allow for the development of applications of all sorts.  Cloud computing vendors host all the necessary resources and tools for application development.

1.1.4.                Business Process as a Service (BPaaS)

Business Process as a Service and cloud application (ERP) solutions, provide greater scale-ability, agility, and cost savings for business by delivering any type of horizontal or vertical business process via a cloud services model.  Because of the service-orientated nature of services these require integration in accordance with the company’s business processes ensuring that a more consistent process exists across the organisation.  Business processes can consequently be interlinked between the cloud components (applications) or between the cloud and on premise applications in the data centre, for example managing email, shipping a package, or managing customer credit.  BPaaS is designed to be service-orientated with well-defined interfaces, standardised for use by multiple organisations and can therefore be much more automated and standardised to deliver the service in a repeatable fashion.  BPaaS is situated on top of Saas, PaaS, and IaaS and is configured based on the process being designed and require well-defined APIs for integration with other components and related services with the ability to scale over massive architectures for a huge user base by utilising the cloud base for elasticity and scaling.

1.  Business Analysis and Intelligence

Business intelligence (BI) helps businesses to centralise, better access, and better understand key business metrics and predict future performance through business analysis.  BI benefits are faster answers without having to guess, on key business metrics reports on demand, to obtain business insight on behaviours of things controlled (measured) for the ability to identify deficiencies (learn how to remedy and streamline processes), or to enable cross and up-selling opportunities, improve inefficiencies in general and to understand the true costs of doing business.  Through this function time savings, easier and quicker access to information that is correct and relevant for making business decisions must realised.

Access to mission-critical business information is crucial and if impeded, limits an organisation’s ability to succeed.  When accurate business information is not easily available to the enterprise—in real time—it’s tough to make optimised decisions for a more competitive business position.  To solve the latter requires a business intelligence strategy.  Such a strategy is aimed to address the needs of the organisation to improve access to information.  Business intelligence is a key to enhancing business process improvement and cultivating business performance, i.e. better information lead to better decision making.  A BI strategy is much more that ERP reports and dashboards, it is a formal examination of overall business information needs and the alignment of tactical business intelligence resources to address these needs in the most effective ways to drive business process improvement.  A strategic approach to business intelligence helps prioritise business requirements, increase transparency and promote user access to key information.

The core elements of a business intelligence strategy are (Consultants, 2014):

  • Identification of the information needs
  • Including key performance metrics
  • The definition of the business view of information
  • How to look for information
  • What are the dimensions of the data?
  • The identification of the source of information
  • The selection of the technology to be used to support the BI plan
  • The users
  • The BI development plan
  • The BI roll-out plan

In order to derive a BI strategy the following areas have to be considered, among others:

  • Auditing the current state of reporting, metrics, and information access
  • Identifying enterprise-wide information needs in strategic, tactical, and operational areas
  • Setting the future state goals for adding transparency and insight into every function area for increased performance
  • Setting key performance metrics that support overall business process goals including key performance indicators (KPIs), balanced scorecard views, forecasting, and other analysis
  • Constructing an achievable life-cycle plan for rolling out individual initiatives
  • Defining effective business views of information.  What presentation formats support rapid internalization and understanding?  What are the dimensions of the data?
  • Analysing the frequency of data access that is required by all users, including aggregate and detail-level information
  • Identifying the sources of information
  • Auditing current technology tools now used to generate business information and identifying areas for increased efficiency

The Aberdeen Group research, on business intelligence, (Henderson, Santiago;, 2014) reveals the magnitude of the challenges faced when managing business data, analysis, and reporting:

  • 46% of businesses realise that too little data is used for business analysis
  • 77% of managers are not notified when important changes affect their business
  • It takes 11.5 days, on average, for managers to be notified of events happening that affect their business

Business intelligence also requires access by all levels of a business and not just the upper levels.  Much more complex tools have become available in recent times that help speed (at their most basic level) information across all appropriate levels of a company to foster collaboration and inform decision making.  BI as an analysis tool must be made available to users in a dashboard/ intuitive web interface as organised reports in self-service fashion with relevant tools to help users set up and select data that applies to business processes related to their roles.  So, also for BI and decision making, to allow the creation of key performance indicators that are displayed on a dashboard to show relevant information for business processes in a user’s area from which they are empowered to take decisions.  Functions like drill-downs, pulling of other data sources into graphs, maps or gauges, in real-time, should present the relevant analytic information to a user.  Centralised information allows users to collaborate on a single version of the truth with aids like data visualisation, using consistent up-to-date information, of a high quality, even available on mobile platforms.

There are several reasons why BI should be considered for reporting (Richards Foreman, 2011) but mainly because a BI component provides on-screen reports where columns and rows are manipulated, scaled, and outliers removed to generally work out what should be displayed.  Many BI component (application) vendors would provide BI content, e.g. standard reports, configured cubes, hierarchies and calculations that creating reports becomes a drag-and-drop effort.  Access allows the execution of these reports on-demand on current data or to run custom reports for other periods of data.  Because these BI reports are flexible, new columns and elements can be readily added or tweaked and drill-down functions allow successive investigations on current conclusions without effort.  Visualising the data and the ability to change the report aspects like switching between a tabular and graphical view or put year over year on the same graph to track seasonality or adding margins to revenue graphs, change periods, eliminate columns or consolidate data from various systems.  The ERP BI component should let data be used to discover trends, outliers, correlations, and anomalies, create alerts, and share data discoveries (collaboration).  For data to be valuable to a business it needs to be high quality, accessible, usable, and intelligent.  By integrating with the ERP system, the BI solution has access to such data and by employing dashboards and other data visualisation techniques the data becomes usable to the enterprise.  Self-service and mobile access ensure that data is accessible and a good BI engine turns complex data into business intelligence.

1.1.                        ERP, BI and CPM

An ERP system with a BI complement is the most comprehensive answer to corporate performance management (CPM) and is usually part of the overall CPM solution that cuts across all facets of an organisation (Girardi, Joni;, 2014).  CPM systems are built over the platform of ERP and BI and provides a bi-directional integration of strategy, operational and tactical business process metrics where strategy is broken down into operational targets and the execution is monitored and adjusted to achieve the set targets.  CPM makes the drive for performance the responsibility of each person in the company and builds on a number of methodologies and system solutions to achieve it.  However, CPM is not just about technology.  It involves processes, methodologies, metrics, and technology that are used to monitor, measure, and manage a business.  BI and analysis are used to understand and exploit what metrics and key performance indicators (KPI) are relevant to the business.  Not having decision support information that is relevant to the business risks CPM as it probably won’t suit the company’s needs.

The approach to CPM, based on the principle of predictability, visibility, accountability, and confidence emanating in the eyes of stakeholders and regulators is comprised of (Turbo Charging ERP with BI and CPM, 2014):

  • Enterprise risk management that requires predictive risk analysis capability
  • More stringent corporate governance to meet legislation accountability
  • Formation of executable strategy
  • Goals and rewards linked to strategy
  • Becoming an agile and real-time enterprise
  • Management process to improve beyond ERP

ERP is generally not a substitute for BI or CPM solutions (BDO Solutions, 2014) but many ERP vendors and ERP component vendors are providing business intelligence capabilities in their suites.  For example the cloud ERP system NETSUITE (NETSUITE, 2014) provides business intelligence built-in with their ERP offering, providing real-time dashboards, reporting, and analysis across all the integrated processes within the software suite.  Similarly Microsoft Dynamics AX, through various third party vendors with whom they have partnered, like ZAP or TARGIT, are providing bolt-on business intelligence capabilities.

1.  Works Cited

Accenture. (2014). Accenture Enterprise Performance Management Framewor. Retrieved from Accenture: http://www.accenture.com/sitecollectiondocuments/pdf/accenture_aaepm_oil_and_gas_brochure.pdf

Arthur Andersen. (1997). Redesigning the Finance Function. Institute of Management Accountants . Montvale: Institute of Management Accountants. Retrieved from http://www.imanet.org/PDFs/Public/Research/SMA/Redesigning%20the%20Finance%20Function.pdf

Barton, D., & Court, D. (2013, March). Three keys to building a data-driven strategy. Retrieved from McKinsey & Company: http://www.mckinsey.com/insights/business_technology/three_keys_to_building_a_data_driven_strategy

BDO Solutions. (2014, June 11). 7 Reasons Why ERP is Not Enough. Retrieved from BDO: http://www.erpsoftwareblog.com/2014/06/7-reasons-why-erp-is-not-enough/

Carr, J. (2014, March 07). Business Intelligence Trends: ERP and Business Intelligence. Retrieved from Ultra Consultants: http://www.ultraconsultants.com/business-intelligence-trends-erp-and-business-intelligence/

Colbert, B. (2011, October 18). Gartner – Driving Innovation Using Pace Layering. Retrieved from BrettColbert: http://www.brettcolbert.me/2011/10/gartner-driving-innovation-using-pace.html

Consultants, U. (2014). Business Intelligence Strategy. Retrieved from Ultra Consultants: http://www.ultraconsultants.com/services/business-intelligence-software/business-intelligence-strategy/

Dresner, H. (2014, 11 18). Howard Dresner. Retrieved from Dresner Advisory: http://www.dresneradvisory.com/

ExecutiveBoard. (2014, 08 13). Finance Transformation Diagnosic Gameboard. Retrieved from Executive Board: http://www.executiveboard.com/exbd-resources/content/finance-transformation-diagnostic/gameboard.html

FrontAccounting. (2014, August 18). Welcome to FrontAccounting. Retrieved from FrontAccounting: http://frontaccounting.com/

Ganly, D., Kyte, A., Rayner, N., & Hardcastle, C. (2013). Predicts 2014: The Rise of the Postmodern ERP and Enterprise Applications World. Gartner. Chigago: Gartner. Retrieved January 28, 2015, from http://www.gartner.com/document/2633315

Gartner. (2013, June 5). Gartner Says Organizations Must Treat Information as an Asset in its Own Right. Retrieved from Gartner Newsroom: http://www.gartner.com/newsroom/id/2506315

Gartner. (2014, 11 18). CPM (corporate performance management). Retrieved from Grtner: http://www.gartner.com/it-glossary/cpm-corporate-performance-management

Gartner. (2014). Postmodern ERP. Sun City: Gartner.

Gartner. (2015, January 27). CPM (corporate performance management). Retrieved from Gartner IT Glossary: http://www.gartner.com/it-glossary/cpm-corporate-performance-management

Gartner. (2015, January 27). Pace-Layered Application Strategy. Retrieved from Gartner IT Glossary: http://www.gartner.com/it-glossary/pace-layered-application-strategy

Gattenio, C. A. (1997, January 02). How Does Your Finance Department Measure Up? Retrieved from Journal of Accountancy: http://www.journalofaccountancy.com/Issues/1997/Jan/measrup.htm?action=print

Girardi, Joni;. (2014). BI & Analytics vs. Corporate Performance Management (CPM). Retrieved from DataSelf: http://www.dataself.com/bi-analytics-vs-corporate-performance-management-cpm/

Guay, Mike;. (2014). Postmodern ERP: Cultural, Political and Organizational Change Imperatives. Las Vegas: Gartner.

Hackett Group. (2014, March 13). Finance Leaders Asked to Realign Resources in 2014. Retrieved from The Hackett Group: http://www.thehackettgroup.com/about/research-alerts-press-releases/2014/03132014-finance-leaders-asked-realign-resources-2014.jsp

Henderson, Santiago;. (2014, September 30). ERP for Analyzing Data with Business Intelligence. Retrieved from The Answer Company: http://www.theanswerco.com/tac-blog/erp-for-analyzing-data-with-business-intelligence

Hendriks, C. J. (2012). Integrated Financial Management Information Systems: Guidelines for effective implementation by the public sector of South Africa. Potchefstroom: University of the Free State. Retrieved August 04, 2014, from http://www.sajim.co.za/index.php/SAJIM/article/download/529/599

ICAEW;. (2011). The Finance Function A Framework for Analysis. The Institute of Chartered Accountants in England and Wales, Finance & Management Faculty. London: ICAEW. Retrieved August 04, 2014, from https://www.icaew.com/~/media/Files/Technical/Business-and-financial-management/finance-direction/finance-function-a-framework-for-analysis.pdf

Investopedia. (2014, November 18). Key Performance Indicators – KPI. Retrieved from Investopedia: http://www.investopedia.com/terms/k/kpi.asp

KPMG. (2013). Intelligent Finance Organizations: Insight into Empowering Finance and the Enterprise. KPMG. Boston: CFO Publishing LLC. Retrieved August 01, 2014, from http://www.kpmg.com/US/en/IssuesAndInsights/ArticlesPublications/Documents/intelligent-finance-organizations-summary.pdf

Lynds, Heather;. (2014). Creating Account Reconciliation Process Efficiencies while Reducing Risk and AUdit Fees. Chicago: Trintech.

McCann, David;. (2012, August 10). A Cloudy Revolution for HR Tech Arena. CFO, 1. Retrieved January 28, 2015, from http://ww2.cfo.com/people/2012/08/a-cloudy-revolution-for-hr-tech-arena/

Meyer, L. (2014, February 02). Gartner Redefines Highly Customized ERP Systems as Legacy. Retrieved from Campus Technology: http://campustechnology.com/articles/2014/02/04/gartner-redefines-highly-customized-erp-systems-as-legacy.aspx?=CTITTNL

Mitchell, David;. (2012). Cloud Computing. Retrieved from Gartner: http://www.gartner.com/technology/research/cloud-computing/report/iaas-cloud.jsp

NETSUITE. (2014). NetSuite Business Intelligence. Retrieved from NETSUITE: http://www.netsuite.com/portal/products/business-intelligence.shtml

Open Group. (2014, August 04). Foundation Architecture: Technical Reference Model. Retrieved from The Open Group: http://pubs.opengroup.org/architecture/togaf8-doc/arch/chap19.html

Open Group. (2014, August 04). Integrated Information Infrastructure Reference Model. Retrieved from The Open Group: http://pubs.opengroup.org/architecture/togaf9-doc/arch/chap44.html

Oracle. (2012). Reducing the Pain of Account Reconciliations. Chigago: Oracle.

Pettey, Christy; Goasduff, Laurence. (2012, February 14). Gartner Says Adopting a Pace-Layered Application Strategy Can Accelerate Innovation. Retrieved from Gartner Newsroom: http://www.gartner.com/newsroom/id/1923014

PWC. (2008). CPM. Retrieved from PWC: http://www.pwc.com/en_US/us/corporate-performance-management/assets/corporate_performance_management.pdf

Rayner, Nigel;. (2012). Applying Pace Layering to ERP. Gartner. Boston: Gartner. Retrieved January 27, 2015, from http://www.pega.com/sites/default/files/docs/resources/Applying-Pace-Layering-to-ERP-Strategy.pdf

Rayner, Nigel; van Decker, John E;. (2011). Use Gartner’s Pace Layers Model to Better Manage Your. Boston: Gartner. Retrieved January 27, 2015, from http://www.financialexecutives.org/KenticoCMS/getattachment/Communities/Committees/Finance—IT-%28CFIT%29/resources/CFIT-September-2011/Use-Gartner-s-Pace-Layer-Model.pdf.aspx

Redman, T. (2012, October 22). Get Responsibility for Data Out of IT. Retrieved from Harvard Business Review: https://hbr.org/2012/10/get-responsiblity-for-data-out?cm_sp=blog_flyout-_-cs-_-get_responsiblity_for_data_out

Richards Foreman, S. (2011, October 28). 7 Reasons Microsoft Dynamics ERP Users Should Consider BI for Reporting. Retrieved from ERP Software Blog: http://www.erpsoftwareblog.com/2011/10/7-reasons-microsoft-dynamics-erp-users-should-consider-bi-for-reporting/

Richards, J. (2013, October 4). Talent Management in Microsoft Dynamics AX 2012. Retrieved from Merit Solutions: http://meritsolutions.com/meritmatters/archives/619-Talent-Management-in-Microsoft-Dynamics-AX-2012.html

Rose, Jerry R;. (2014, November 18). Baldrige Criteria for Performance Excellence. Retrieved from Nits.Gov: http://www.nist.gov/baldrige/publications/upload/2013_2014_Baldrige_Criteria_Brochure.pdf

Rose, Margaret;. (2011, April 01). corporate performance management (CPM). Retrieved from TechTarget: http://searchdatamanagement.techtarget.com/definition/corporate-performance-management

Rouse, M. (2014). corporate performance management (CPM) definition. Retrieved from TechTarget: http://searchdatamanagement.techtarget.com/definition/corporate-performance-management

Savitz, E. (2012, February 09). The End Of ERP. Retrieved from Forbes: http://www.forbes.com/sites/ciocentral/2012/02/09/the-end-of-erp/

Speck, Arne;. (2013). Die Technologieplattform der Zunkunft. Munich: SAP. Retrieved from http://www.sapevent.ch/landingpagesfr/Manager/uploads/194/Technologieplattform%28Adobe%29.pdf

Sullivan, J. (2013, February 25). How Google became the #3 most valuable firm by using people analytics to reinvent HR. Retrieved from ere.net: http://www.ere.net/2013/02/25/how-google-became-the-3-most-valuable-firm-by-using-people-analytics-to-reinvent-hr/

Syx, S. (2011, October 18). Accelerating Innovation by Adopting a Pace-Layered Application Strategy. Retrieved from Doozer: http://www.doozer.com/developer-exchange/accelerating-innovation-by-adopting-a-pace-layered-application-strategy

Tagetik. (2014). Corporate Performance Management. Retrieved from Tagetik: http://www.tagetik.com/software/pm#.VMi4pUfQqm4

Turbo Charging ERP with BI and CPM. (2014). Retrieved from MSG Management Study Guide: http://managementstudyguide.com/turbo-charging-erp-bi-and-cpm.htm

Virgin Atlantic. (2014). A Treasury Technology Roadmap at Virgin Atlantic. Retrieved from Treasury Management International: http://www.treasury-management.com/article/1/96/859/a-treasury-technology-roadmap-at-virgin-atlantic.html

Wailgum, Thomas;. (2009, August 4). ERP and BI: A Match Made in Heaven, If You’re in Data Hell. Retrieved from CIO.com: http://www.cio.com/article/2425821/enterprise-software/erp-and-bi–a-match-made-in-heaven–if-you-re-in-data-hell.html

What is a KPI? (2014, November 18). Retrieved from Klipfolio: http://www.klipfolio.com/resources/articles/what-is-a-key-performance-indicator

WikiPedia. (2013, December 9). Shearing layers. Retrieved from WikiPedia: http://en.wikipedia.org/wiki/Shearing_layers

WikiPedia. (2014, October 08). Business performance management. Retrieved from WikiPedia: http://en.wikipedia.org/wiki/Business_performance_management

WikiPedia. (2014, June 30). Financial inclusion. Retrieved from WikiPedia: http://en.wikipedia.org/wiki/Financial_inclusion

WikiPedia. (2014, May 19). Heterarchy. Retrieved from WikiPedia: http://en.wikipedia.org/wiki/Heterarchy

Wilson, T. (2014, May 26). Cloud a Key Part of ‘Postmodern ERP’. Retrieved from Enterprise Apps Today: http://www.enterpriseappstoday.com/erp/cloud-a-key-part-of-postmodern-erp.html

Hadoop for a Data Driven Organisation

Using Big Data: Interoperability Issues

To derive meaning quickly, from huge quantities of data, has been the objective of enterprises since the inception of databases.  Data virtualisation makes large and complex data sets understandable and enables domain experts to spot underlying trends and patterns. Companies would want to learn of signs and patterns in their data to warn them, not only of hazards such as fraud or production delays, but provide insight into opportunities such as the treatment of diseases or leveraging different logistics approaches.  Traditionally, business intelligence required static reports, created from an underlying data warehouse, comprised of conspicuously formatted data in specialised data tables. Now, with a proper visualisation tool, on top of big data[1], the business analytics process may be shortcut; stakeholders use a visualisation tool to model their data requirements to see relationships and spot patterns that analytics professionals might be missing. (Linthicum, 2013).

Hadoop[2] has become the preferred platform for processing large quantities of data[3].  These large quantities may also refer to structured data that cannot be analysed in a reasonable length of time.  The better information is, and the more routinely it is put in the hands of those who need it, the higher the intelligence derived from it.  Despite this, Hadoop is a batch-oriented[4] system and cannot (as yet) answer simple queries with a good turnaround time (Hortonworks, 2013).  But, Hadoop handles data that traditional relational databases, data warehouses, and other analytic platforms have been unable to effectively manage—including user-generated data through social media and machine generated data from sensors, appliances, and applications.  That is, all forms of data.  Hadoop does this through a distributed file system, called the Hive Distributed File System – HDFS[5], providing the portability, scalability (over commodity hardware[6]) and reliability and MapReduce processing for parallel distributed systems (Srivastava, 2011).  “Once data exceeds the capabilities of a single computer, we face the problem of distributing its storage and processing.” (Downey, 2011).

Big Data - Horton Works

Big Data – Horton Works

The Open Group suggests The Need for Boundaryless Information Flow (III-RM).  The boundaryless Information Flow is essentially resolving the problem of getting information to the right people at the right time in a secure, reliable manner, in order to support the operations that are core to the extended enterprise.  According to the Boundaryless Organisation it is not implied that there are no boundaries in the flow of information, only that they should be made permeable or penetrable.  However, the information systems in place today do not allow for information to flow in support of the boundaryless organisation, but when data do, it will be a boundaryless information flow.  The Open Group published the Interoperable[7] Enterprise Business Scenario in 2001 that crystallises this need for boundaryless information flow, describing how this need drives information customers’ deployment of their information systems.  A key driver is to “create a worldwide market for interoperable information products supporting access to integrated information, in which all stakeholders’ needs are addressed.” (The Open Group, 2002).

The lack of the right information to the right person at the right time is equably a lack of interoperability, preventing organisations from achieving their business objectives.  Information as an asset, in principle, passes the point of departure “that information technology merely supports or enables the business – increasingly, information is the business.” (The Open Group, 2002).  To address the lack of interoperability between systems that provide access to data is to attend to the boundaryless information flow.  Technology infrastructure is used to facilitate interoperability by either using Common-off-the-shelf (COTS) appliances or a collection of best of breed integration[8] to reduce interoperability issues.

The Open Group (The Open Group, 2002) identifies high-level technical requirements for a boundaryless information flow architecture:

  1. Openness – standards based open interfaces need to exist for critical interfaces to enable interoperability more easily and support integration of components, for areas:
    1. Protocols
    2. Web portals
    3. Collaboration
    4. Database
    5. Storage Connections
    6. Data Integrity – the same data stored in multiple systems must be conformed to mean the same thing throughout
    7. Availability – must be guaranteed and where maintenance requires downtime, a fall-back must be available
    8. Security – support different security levels and protection based on the sensitivity and use of the data, as well as the normal requirements for data integrity, confidentiality, logging and tracking.  Must provide flexible policy management and strong authentication providing users with a single sign-on and user administration.
    9. Accessibility – solutions must provide global accessibility to the information without compromising security
    10. Manageability – solutions must be manageable
    11. Internationalisation and Localisation – accommodate multiple languages and adapt to culture where deployed internationally
Technology Architecture III-RM

Technology Architecture III-RM – The Open Group

Standardisation of the information technology of an organisation, to achieve a boundaryless information flow, must be measurable in terms of business value, improving business operations, effectiveness, and also that of the Information Technology capability in the organisation.

  • Drive revenue growth
  • Lower Information Technology expenditure
  • Increase % of procurements against standards
  • Reduce spend on customisations
  • Improve cycle time for rolling out upgrades

In the Business Intelligence sphere, a few key trends are identified with the emerging use of big data (Linthicum, 2013):

  • The ability to leverage both structured and unstructured data, and visualise that data
  • Ability to imply structure at the time of analysis, to provide flexibility from the underlying structure of the data (be it structured or unstructured) through decoupling
  • Leverage current or near-real-time data for critical information
  • Allowing BI analyst to mash up data that are outside of the organisation (like the cloud) to enhance the analysis process
  • Ability to bind data analytics to business processes and applications to allow issue resolution without human intervention, i.e. known as embedded analytics
Data and Hadoop Architecture

Data and Hadoop Architecture

An organisation should seek to support of the boundaryless information flow idea in any strategy it evaluates for information integration.  Hadoop, being a framework in support of Big Data, i.e. to provision results/ data over large heterogeneous data sources to analytics tools, or data stores, is a technology to be considered in support of the boundaryless information flow.

Business Intelligence and Big Data

Big data analytics has a potential, which is at odds with the traditional Business Intelligence approach.  To force this existing paradigm on big data analytics is missing the full potential of this technology, or to fail altogether (Linthicum, 2013).

In the previous two decades, calculating summaries, sums and averages were sufficient, but large complex data require new techniques.  To recognise customer preferences requires the analysis of purchase history, browsing behaviour, products viewed, comments, suggestions, complaints, to predict future behaviours (Cloudera, 2013).

Hadoop[9], being able to work on Big Data, structured, and/ or unstructured (heterogeneous sources), is ideally positioned for provisioning data for Business Intelligence.  Problems like advertisement targeting, recommendations engine (predicting customer preferences), customer churn (attrition) analyses, risk modelling, point-of-sale transactional analysis, predicting network data failures, threat analysis, and trade surveillance[10], can be solved with the ability of Hadoop (Cloudera, 2013).

The challenge with Big Data is the transformation of it into intelligent data.  Because Hadoop is primarily a batch-orientated processing system, it is very suitable for Data Analytics (Payandeh, 2013).  It is the crow-bar (no pun intended) that wrests open the capacious data-lake to parse, cleanse, apply structure and transform the data and arranges  it for consumption by analytics applications.

HADOOP Specifically

The most popular spearhead of Big Data, right now, appears to be Hadoop (Kernochan, 2011).  Hadoop is open-source software that achieves reliable, scalable, and distributed computing of large data sets across clusters of computers using programming models, designed to scale from a single to thousands of machines (Welcome to Apache™ Hadoop®!, 2013).  Hadoop is a framework that supports data-intensive distributed applications that run in parallel on large clusters of commodity hardware that provide both reliability and data motion to applications.  By using MapReduce[11][12] (reducing the workload into smaller fragments, parallelising the effort over multiple servers) and a distributed file system that stores data on the compute nodes, it provides a huge aggregated bandwidth across the cluster, as well as the automatic handling of failures by the framework.  “It enables applications to work with thousands of computational-independent computers and petabytes of data.” (WikiPedia, 2013).

Hadoop is also more cost-effective than traditional analytics platforms such as data warehouses, and it moves analytics closer to where the data is.  It may commonly be used for optimising internal data management operations, but it is rapidly gaining ground as a strategic big data analytics platform (Big Data Analytics: Unleashing the Power of Hadoop, 2013).

The Apache-Hadoop platform is comprise of:

  1. Common – common utilities that support the Hadoop modules
  2. MapReduce – YARN-based system for parallel processing of large data sets
  3. Distributed File System (HDFS) – distributed file system for high throughput access to application data
  4. YARN – framework for job scheduling and cluster resource management

Other related projects are:

  • Amabar – web-based tool for provisioning, managing, and monitoring Hadoop clusters, with dashboard
  • Avro – data serialisation system
  • Cassandra – scalable multi-master database with no single points of failure
  • Chukwa – data collection system for managing distributed systems
  • HBase – scalable, distributed database that supports structured data storage for large tables
  • Hive – a data warehouse infrastructure that provides data summarisation and adhoc querying
  • Pig – high level data-flow language and execution framework for parallel computation
  • Zookeeper – high-performance coordination service for distributed applications

The Hadoop File System (HDFS) can store huge and many files to scale across the commodity nodes, breaking down enterprise data silos[13], this common data globule is referred to as a “data lake” (Gualtieri, 2013).  These “data lakes” are queried by MapReduce, a distributed data processing query, in a highly parallel fashion.

Dealing with only files on disk, essentially indicates a lack of structure for the content, data maps are applied over the unstructured content to define the core metadata for it.  For example, using Hive to provide a mechanism to project structure onto the data allows the data to be queried by a SQL-like language called HiveQL, or Pig, a MapReduce mechanism.  A BI tool will thus obtain file metadata from the cluster (serving from both structured and unstructured sources) and retrieve the data from the data nodes into the structure using MapReduce to get to the required data, in parallel, and return the data to a BI visualisation tool, e.g. Tableau[14], or into other types of processing, into specific data structures, e.g. a dimensional model for complex analytical processing (Linthicum, 2013).

Big data is usually only processed periodically, e.g. daily or weekly.  During processing, the load is dispersed over multiple processing nodes to speed it up.  But, after the query is executed those nodes are no longer required until the next run.  This is indicative of a perfect expansion-contraction pattern.  To reduce cost and maximise hardware utilisation, it would be a boon to have the fabric dynamically provisioning the nodes in the grid, to jobs that are executing, and so provide an agile infrastructure.  Hadoop should be thought of as a distributed operating system with many computers storing and processing data as if formed on a single machine (Ken Krugler[15]).  Adding to this, the ability to host Hadoop in the Cloud, to achieve an agile infrastructure, sees efficiency maximised and costs reduced for a multiple number of use-cases (Downey, 2011).

Hadoop is essentially used for (Julio, 2009):

  • Searching
  • Log processing
  • Recommendation systems
  • Analytics
  • Video and Image analysis
  • Data Retention
General BI Scenario with Hadoop

General BI Scenario with Hadoop

Visualisation of Data and Hadoop

Hadoop is supported by Data Virtualisation technologies like Informatica, Composite Server, and Denodo.  Visualisation technologies like Tableau[16] can connect to multiple flavours of Hadoop, to bring data in-memory and perform fast adhoc visualisations to find patterns and outliers, in all the data in the Hadoop cluster.  It is said about Tableau that it is a very elegant solution, obviating the need to move huge data into a relational store before analysing it (Tableau, 2013).   Tableau uses ODBC to connect to the Cloudera connector for Tableau that uses HiveQL to connect to Hive and MapReduce over the HDFS and HBase, Hadoop (Popescu, 2012).

A visualisation tool (in this example Tableau) is connected to Hadoop from which to build visualisations just like when connected to a traditional database.  Tableau achieves “on-the-fly ETL” by connecting to Hive with Custom SQL, allowing the batch-orientated[17] nature of Hadoop to handle layers of analytical queries on top of complex Custom SQL with only an incremental increase in time.

Although Hadoop is essentially a batch-orientated system, there are a number of techniques for improving the performance of visualisations and dashboards, built from data stored on a Hadoop cluster.  Custom SQL limits the data-set size to speed up exploring a new data set and build initial visualisations.   Extracting data for Tableau[18], although Tableau is capable of handling large sets of data, it is nowhere near as well positioned as Hadoop to handle really large, or Big Data loads.  An extract can be pre-formatted for Tableau, by using Hadoop to roll up dates, hide unused fields, and aggregate visible dimensions, to create a broader view of the data.  Hive can, for example, partition records on disk and allow tableau to direct a query, aligned to these partitions, to greatly reduce the time spent on getting to the right data (Hortonworks, 2013).

HADOOP Use-case

Why is Hadoop most typically the technology underpinning “Big Data”?  Data comes from a set of data sources, e.g. CRM, ERP, and Custom Applications.  This data has to be queried somehow, either via an elaborate point-to-point integration, a virtual data store, data warehouse or even a Massively Parallel Processing Systems (MPP), e.g. Microsoft’s Parallel Data Warehouse.  The application doing the query is probably some high-end business intelligence tool, e.g. SAS.  As other unstructured machine generated or social origin data are added, the data integration becomes more complex.

Hadoop is not replacing traditional data systems, but is complementary to them (Hortonworks, Apache Hadoop Patterns of Use, 2013).

Enterprise Data Architecture

Enterprise Data Architecture

The traditional Enterprise Data Architecture is indicated inside the red-marked area, with the additionally added Hadoop ability.  The specific use of Hadoop is directed by a few use-cases or patterns.

Patterns

Observing analytic challenges from an infrastructure level, some clear patterns emerge and can fit into the following three patterns (Hortonworks, Apache Hadoop Patterns of Use, 2013):

  1. Hadoop as a Data Refinery
  2. Data Exploration with Hadoop
  3. Application Enrichment

Hadoop as a Data Refinery

Refine data sources into their commonly used analytics applications, e.g. providing a central view of Customer from all the data about them in either the ERP, CRM or bespoke systems.  This pattern is especially powerful where Customer data are incorporated from their web sessions on the enterprise’s web site, to see what they have accessed and what they are interested in.

This pattern uses Hadoop[19] to parse, cleanse, apply structure and transform such data and then push it into an analytics store for use with existing analytics tools.  This pattern uses Hadoop as a type of staging environment between the source and the analytics tools, with Hadoop being the transformation agent.

Data Exploration with Hadoop

Hadoop is used directly on the data to explore it.  It still acts the role of parser, cleaner, structure and transformer, but it provisions the data to visualisation tools, e.g. Tableau (Tableau, 2013).  Financial institutions can use this pattern to discover identity fraud, i.e. perform some kind of surveillance activity.

Application Enrichment

Data, stored in Hadoop, is being used to impact an application’s behaviour, e.g. a web application using a user’s access patterns to curtail its depiction to suit the returning user’s usage characteristics.  Hadoop parses, cleanses, applies structure and transforms the data and passes it directly into the application.  This is an ability used by the large web companies in the world, like Yahoo and Facebook.  The right data is enriched at the right time to the right customer.  Massive amounts of patterns and repeatable behaviour are identified in order to customise the web-application experience of the user, by serving the right content to the right person, at the right time, increasing the conversation rates for purchase.

End Notes


[1] Too big for Excel is not Big Data! (Chris, 2013).  The New York Stock Exchange generates about 1 TB of new trading data per day, Facebook hosts about 10 billion photos (1 Petabyte), Angestry.com stores 2.5 Petabytes of genealogy data, and the Internet Archives stores around 2 Petabytes, growing at a rate of 20 Tb per month (Seminar Report Downloads, 2012).

[2] It was first conceived as a web search engine for yahoo (Big Data Analytics: Unleashing the Power of Hadoop, 2013)

[3] A (subjective) figure of twenty Terra Bytes is cited as a tipping point for considering Hadoop (Chris, 2013) “Rama Ramasamy”.

[4] There are projects that provide real-time capabilities to applications for results from Hadoop, but it is primarily a batch-orientated processing solution (HandsonERP & Mir, 2013).

[5] LZO or Snappy Compression can be applied to HDFS files to reduce the bytes read or written thus improving the efficiency of network bandwidth and disk space (Julio, 2009).

[6] Easy to obtain, available and affordable hardware that is not unique in any function.

[7] The ability of two or more entities or components to exchange information

and to use the information that has been exchanged “to meet a defined mission or objective; in

this scenario this term specifically relates to access to information and the infrastructure that supports it. (The Open Group, 2002)

[8] The process of combining components into an effective overall system. In this scenario the phrase “access to integrated information” is used repeatedly. The term “integrated information” means an overall system in which a single view of information is presented, and is achieved by combining various information sources (The Open Group, 2002).

[9] By 2015, 50% of Enterprise data will be processed by Hadoop (HandsonERP & Mir, 2013).

[10][10] A Hadoop cluster can run alongside a trading system obtaining copies of the trading data, coupled with the reference data of parties to the trade, the system can continually monitor trade activity to build relationships between people and organisations trading with each other, watching for patterns that reflect rogue trading (Cloudera, 2013).

[11] MapReduce is a programming model for processing large data sets with a parallel and distributed algorithm on a cluster, doing filtering and sorting on the distributed servers, running various tasks in parallel, managing all communications and data transfers between the various parts of the system, providing for redundancy and fault tolerance and overall management of the whole process (WikiPedia, 2013).

[12] The progress estimator for MapReduce in Hadoop is simplistic, and often produces inaccurate estimates.

[13] Collapsing data silos creates a pro-BI data base, on which to position BI and Visualisation tools.

[14] To use Hortonworks’ online Hadoop data source for Tableau, connect to the Hortonworks Hadoop Hive Connection of Tableau to 192.168.56.101 (Sandbox VM) to start visualising data from Hadoop (Hortonworks, HOW TO: Connect Tableau to Hortonworks Sandbox, 2013).  Get the driver from here: http://www.tableausoftware.com/drivers.

[15] http://www.linkedin.com/in/kenkrugler

[16] Through a partnership with Cloudera, Tableau software has built a Hadoop connector for doing big data visualisation (Williams, 2012).  Also see footnote  14 on how to use Hortonworks online Hadoop data source for Tableau.

[17] Hadoop is not yet capable of answering simple queries with very quick turnaround. (Hortonworks, 2013).

[18] Hadoop query cancellation is not straight forward.  Queries used by Tableau can only be abandoned, but the query continues on the Hadoop cluster, consuming resources (Hortonworks, 2013).

[19] Large quantities of data are distilled to something more manageable (Hortonworks, Apache Hadoop Patterns of Use, 2013).

References

Big Data Analytics: Unleashing the Power of Hadoop. (2013, August 21). Retrieved from Database Trends and Applications: http://www.dbta.com/Articles/Editorial/Trends-and-Applications/Big-Data-Analytics-Unleashing-the-Power-of-Hadoop-91261.aspx

Chris, S. (2013, September 16). Don’t use Hadoop – your data isn’t that big. Retrieved from Chris Stucchio: http://www.chrisstucchio.com/blog/2013/hadoop_hatred.html

Cloudera. (2013). TEN COMMON . Chicago: Cloudera. Retrieved from http://www.cloudera.com/content/dam/cloudera/Resources/PDF/cloudera_White_Paper_Ten_Hadoopable_Problems_Real_World_Use_Cases.pdf

Downey, J. (2011, July 1). Big Data and Hadoop: A Coud Use Case. Retrieved from Jim Downey: http://jimdowney.net/2011/07/01/big-data-and-hadoop-a-cloud-use-case/

Gualtieri, M. (2013, october 23). 5 Reasons Hadoop Is Kicking Can and Taking Names. Retrieved from Information Management : http://www.information-management.com/blogs/5-reasons-hadoop-is-kicking-can-and-taking-names-10024986-1.html

HandsonERP, & Mir, H. (2013, April 5). Hadoop Tutorial 1 – What is Hadoop? Retrieved from YouTube: http://www.youtube.com/watch?v=xWgdny19yQ4

Hortonworks. (2013). Apache Hadoop Patterns of Use. Chigago: Hortonworks. Retrieved from http://hortonworks.com/wp-content/uploads/downloads/2013/04/Hortonworks.ApacheHadoopPatternsOfUse.v1.0.pdf

Hortonworks. (2013, August 26). Best Practices for Hadoop Data Analysis with Tableau and Hortonworks Data Platform. Retrieved from SlideShare: http://www.slideshare.net/hortonworks/best-practices-forhadoopdataanalysiswithtableauv10-25612601

Hortonworks. (2013, Novemeber 07). HOW TO: Connect Tableau to Hortonworks Sandbox. Retrieved from Hortonworks KB: http://hortonworks.com/kb/how-to-connect-tableau-to-hortonworks-sandbox/

Julio, P. (2009, October 19). What is Hadoop used for>. Retrieved from SlideShare: http://www.slideshare.net/PhilippeJulio/hadoop-architecture

Kernochan, W. (2011, October 20). Big Data, MapReduce, Hadoop, NoSQL: The Relational Technology Behind the Curtain. Retrieved from enterpriseappstoday: http://www.enterpriseappstoday.com/business-intelligence/big-data-mapreduce-hadoop-nosql-2.html

Linthicum, D. (2013). Making Sense of Big Data. Chicago: IBM.

Payandeh, F. (2013, September 8). Hadoop vs. NoSql vs. Sql vs. NewSql By Example. Retrieved from Big Data Studio: http://bigdatastudio.com/2013/09/08/hadoop-vs-nosql-vs-sql-vs-newsql-by-example/

Popescu, A. (2012, February 8). Visualizing Hadoop data with Tableau Software and Cloudera Connector for Tableau. Retrieved from myNoSQL: http://nosql.mypopescu.com/post/17262685876/visualizing-hadoop-data-with-tableau-software-and

Seminar Report Downloads. (2012, February 06). Hadoop. Retrieved from Seminar Report Downloads: http://seminarreportdownloads.wordpress.com/tag/new-york-stock-exchange/

Srivastava, A. (2011, April 26). A Hadoop Use Case . Retrieved from Anjur-sa.blogspot.com: http://ankur-sa.blogspot.com/2011/04/hadoop-use-case.html

Tableau. (2013, October 25). Hadoop. Retrieved from Tableausoftwar: http://www.tableausoftware.com/solutions/hadoop-analysis

The Open Group. (2002). Interoperable Enterprise Business Scenario. Chicago: The Open Group.

Welcome to Apache™ Hadoop®! (2013, October 24). Retrieved from hadoop.apache.org: http://hadoop.apache.org/

WikiPedia. (2013, October 23). Apache Hadoop. Retrieved from WikiPedia: http://en.wikipedia.org/wiki/Apache_Hadoop

WikiPedia. (2013, October 19). MapReduce. Retrieved from WikiPedia: http://en.wikipedia.org/wiki/MapReduce

Williams, A. (2012, November 9). Hadoop. Retrieved from Siliconangle: http://siliconangle.com/blog/2011/11/09/tableau-brings-data-visualization-to-hadoop/