DATA MINING

Data Mining is an analytic process designed to explore data (usually large amounts of data – typically business or market related) in search of consistent patterns and/or systematic relationships between variables, and then to validate the findings by applying the detected patterns to new subsets of data. The ultimate goal of data mining is prediction – and predictive data mining is the most common type of data mining and one that has the most direct business applications.

The process of data mining consists of three stages:

(1) The initial exploration,

(2) Model building or pattern identification with validation/verification, and

(3) Deployment (i.e., the application of the model to new data in order to generate predictions).

(1) Exploration:

This stage usually starts with data preparation which may involve cleaning data, data transformations, selecting subsets of records and – in case of data sets with large numbers of variables (“fields”) – performing some preliminary feature selection operations to bring the number of variables to a manageable range (depending on the statistical methods which are being considered). Then, depending on the nature of the analytic problem, this first stage of the process of data mining may involve anywhere between a simple choice of straightforward predictors for a regression model, to elaborate exploratory analyses using a wide variety of graphical and statistical methods  in order to identify the most relevant variables and determine the complexity and/or the general nature of models that can be taken into account in the next stage.

Model building and validation:

This stage involves considering various models and choosing the best one based on their predictive performance (i.e., explaining the variability in question and producing stable results across samples). This may sound like a simple operation, but in fact, it sometimes involves a very elaborate process. There are a variety of techniques developed to achieve that goal – many of which are based on so-called “competitive evaluation of models,” that is, applying different models to the same data set and then comparing their performance to choose the best. These techniques – which are often considered the core of predictive data mining – include: Bagging (Voting, Averaging), Boosting, Stacking (Stacked Generalizations), and Meta-Learning.

Deployment:

That final stage involves using the model selected as best in the previous stage and applying it to new data in order to generate predictions or estimates of the expected outcome.

The concept of Data Mining is becoming increasingly popular as a business information management tool where it is expected to reveal knowledge structures that can guide decisions in conditions of limited certainty. Recently, there has been increased interest in developing new analytic techniques specifically designed to address the issues relevant to business Data Mining, but Data Mining is still based on the conceptual principles of statistics including the traditional Exploratory Data Analysis (EDA) and modeling and it shares with them both some components of its general approaches and specific techniques.

However, an important general difference in the focus and purpose between Data Mining and the traditional Exploratory Data Analysis (EDA) is that Data Mining is more oriented towards applications than the basic nature of the underlying phenomena. In other words, Data Mining is relatively less concerned with identifying the specific relations between the involved variables. For example, uncovering the nature of the underlying functions or the specific types of interactive, multivariate dependencies between variables are not the main goal of Data Mining. Instead, the focus is on producing a solution that can generate useful predictions. Therefore, Data Mining accepts among others a “black box” approach to data exploration or knowledge discovery and uses not only the traditional Exploratory Data Analysis (EDA) techniques, but also such techniques as Neural Networks which can generate valid predictions but are not capable of identifying the specific nature of the interrelations between the variables on which the predictions are based.

Data Mining in Business

Through the use of automated statistical analysis (or “data mining”) techniques, businesses are discovering new trends and patterns of behavior that previously went unnoticed. Once they’ve uncovered this vital intelligence, it can be used in a predictive manner for a variety of applications.

Gathering Data

The first step toward building a productive data mining program is to gather data. Most businesses already perform these data gathering tasks to some extent — the key here is to locate the data critical to your business, refine it and prepare it for the data mining process. If you’re currently tracking customer data in a modern DBMS, chances are you’re almost done.

Selecting an Algorithm

At this point, take a moment to pat yourself on the back. You have a data warehouse! The next step is to choose one or more data mining algorithms to apply to your problem. If you’re just starting out, it’s probably a good idea to experiment with several techniques to give yourself a feel for how they work. Your choice of algorithm will depend upon the data you’ve gathered, the problem you’re trying to solve and the computing tools you have available to you. Let’s take a brief look at two of the more popular algorithms.

1) Regression

Regression is the oldest and most well-known statistical technique that the data mining community utilizes. Basically, regression takes a numerical dataset and develops a mathematical formula that fits the data. When you’re ready to use the results to predict future behavior, you simply take your new data, plug it into the developed formula and you’ve got a prediction! The major limitation of this technique is that it only works well with continuous quantitative data (like weight, speed or age). If you’re working with categorical data where order is not significant (like color, name or gender) you’re better off choosing another technique.

2) Classification

Working with categorical data or a mixture of continuous numeric and categorical data? Classification analysis might suit your needs well. This technique is capable of processing a wider variety of data than regression and is growing in popularity. You’ll also find output that is much easier to interpret. Instead of the complicated mathematical formula given by the regression technique you’ll receive a decision tree that requires a series of binary decisions. One popular classification algorithm is the k-means clustering algorithm.

In the next Ill discuss about some crucial concepts of data mining.

Expression Trees  – What’s this ?

Will begin the post by trying to explain whats Expression Tree is all about and at the end of the post will try to explain why we need this.

In general, Expression Trees are basic tree structure which helps to represent any algebraic expression(Binary expressions €€œ x < y). Usually with single root node with clusters of sub-nodes and ending with leaf nodes.

.Net 3.5 framework introduced this Expression Trees (Visual Studio 2008), helps translating executable code {or} Represent language-level( c# )code in the form of data, and visualize in a tree-like format.

Expression Trees €€œ A Data Structure

Yes, in simple words Expression Trees are Data structure, which helps to store executable code blocks. An executable code could be any code  block containing expressions of type delegate.

for ex.)   Func<int, int, int> function = (a,b) => a + b;

here (a,b) => a + b is the executable code (an lambda expression) which is of a delegate type Func.

Creating an Expression Tree

Expression is an abstract  class present in System.Linq.Expressions namespace.

a simple expression object initialization will look as below,

Expression<Func<int, int, int>> expression = (a,b) => a + b;

in the above code, we have created an expression instance of delegate type and refers to an executable code block( in this case a lambda expression €œ (a,b) => a + b€ )

Viewing an Expression Tree

Once we have created an expression instance as above, it can be visualized using a ExpressionTreeVisualizer( a free add-on available in MSDN €€œLinq samples). After the installation of this add on, in the debug mode while we perform a mouse hover on the expression instance variable we get an magnifier icon similar to the one shown below,

on clicking the image, we can view the complete executable code that we mapped in the form of a tree.

 Body: Retrieve the body of the expression.

Parameters: Retrieve the parameters of the lambda expression.

NodeType: Gets the Expression Type for some node in the tree. Like (<), (>), those that add values together (+), Lambda etc.

Type: Gets the static type of the expression. In this case, the expression is of type Func<int, int, int>.

What’s the need for Expression Trees ?

Yes, i am able to sense the restless in each one of you, as to why we need such a data structure and what i am trying to demonstrate in this post. I hope the following advantages i am listing will do some justice to the need for Expression trees.

Advantages

1.  Expression Tree €€œDOM:-  it has its own DOM, with which we can traverse any of the tree node and retrieve data that we require.

BinaryExpression body = (BinaryExpression)expression.Body;

ParameterExpression left = (ParameterExpression)body.Left;

2.  Expression Tree €€œ Compilation:- converting data back into code, (i.e) once we have mapped the executable code block to an Expression instance,  then we can just make use of the Compile method of the Expression class to execute the code block. below code we are compiling the same expression that we created above initially in the post.

int result = expression.Compile()(3, 5);

Console.WriteLine(result);

3.  Expression Trees in LINQ:- Language Integrated Query, with the introduction of LINQ, the need for Expression Trees data structure became inevitable. To understand this, we should get back to the basics of LINQ to SQL.

Consider the following LINQ query,

var query = from c in db.Customers

                    where c.City == "Nantes" select new { c.City, c.CompanyName };

though the query is written in c# editor, the linq query is not directly complied into binaries by the c# compiler and sent to the SQL Server process to retrieve the data (because for the sql server to interpret the binaries and respond with result is non-sense).

So, the ideal think to do, is to convert the above linq into its equivalent string query and send across the wire to the SQL server process, so that it can interpret and respond with the desired result.

If so, then how to generate the best, efficient and equivalent string query ? this question lead to the invention of the Expression Tree Data structure, along with the effective built-in algo to convert the data structure into its equivalent string representations (in case of Linq to SQL).

If you drill into the type definition  of the above linq query, you will notice that the query returns a IQueryable  type and whose definition in turn looks as below,

public interface IQueryable : IEnumerable

{

Type ElementType { get; }

Expression Expression { get; }

IQueryProvider Provider { get; }

}

So now its apparent that whatever Linq to SQL query we write, it gets stored in the Expression data member of the IQueryable  type, then the in-built algo of Linq to SQL is intelligent enough to generate the best equivalent sql query string by parsing through the data structure and send it across to the sql server process.

Hence with the above advantage what i have explained (one of the in-built implementation of Expression Trees), i am sure we developers have the freedom to improvise and make use of Expression Trees in best possible ways as demanded by our application.

This is a Windows client which does not require Visual Studio and target the needs of the tester. Here a tester can set up test manuscripts, run tests and report bugs. What was also really interesting is that while running tests, a tester can choose to record his steps and when filing a bug, he can automatically send a film or pictures of the situation. The recording can then also be used when verifying that a bug is fixed: the macro retakes all steps to the failing step, which makes the validation faster.

What also was interesting is the possibility to set up virtual environments for different test scenarios, for example different cultural settings and configurations. Since a test is run in a specific environment, we can keep better track of the environment in which the tests are run and not run. Also, when filing bugs, a snapshot of the environment at the time of the failure is possible.

Lab Management product  is an integrated solution that brings virtualization to the heart of application lifecycle management space.

• Create libraries of virtualized multi-tier test configurations really quickly
• Automatically deploy new builds of your application to these environments
• Seamless integration of our dev and test capabilities with the virtualized environments
• Take the €œno-more no-repro€  theme to the next level by leveraging snapshots

Lab Management  Architecture

The diagram below shows a high level architecture diagram for Lab Management.

On the server side, Lab Management service is one of the many services running inside Team Foundation Server (TFS). This is what makes the Lab Management solution unique for software testers and developers. Now we can map our lab resources, such as, hosts, virtual machines and storage to Team Project Collections and Team Projects; thus aligning lab hardware needs with the business needs for the projects we are working on.

The lab management service in TFS uses System Center Virtual Machine Manager (SCVMM) for management of lab infrastructure and provisioning of virtual machines across multiple virtualization platforms.

On the client side, the €œMicrosoft Test and Lab Manager€ tool is  the tool to manage  virtualized assets. This is a Windows Presentation Foundation (WPF) based rich client that allows you to define test plans, test suites, test cases and run them in physical or virtual environments.

Basic Concepts

Hardware virtualization is a disruptive technology that is changing the face of computing. Therefore, it is important to understand some of the basic concepts around virtualization and how these are used in Lab Management to understand this paradigm shift.

Virtual Machine

A virtual machine (VM) is a computer within a computer, implemented in software. A VM emulates a complete hardware system, from processor to network card, in a self-contained, isolated software environment, enabling the simultaneous operation of otherwise incompatible operating systems on a single physical computer. Each operating system runs in its own isolated software partition.

Virtual Machine look

A virtual machine snapshot is a file-based snapshot of the state, disk data, and configuration of a VM at a specific point in time. A VM snapshot is similar in functionality to laptop hibernation state with the additional flexibility that a VM supports multiple snapshots. You can roll back the VM to any of the previously taken snapshots and continue operating from there. The picture below shows a snapshot tree for a Hyper-V VM.

Host

A host is a physical

computer that hosts one or more virtual machines.

Host Group

A host group is a custom group of virtual machine hosts, which an administrator can create in SCVMM for ease of monitoring and management. Host groups can be used to allocate and determine the resources reserved for various team projects. For example, an administrator could create a host group named €œGlobal Bank Hosts€ for a team that works on €œGlobal Bank€ project and bind it to the corresponding team project in Team Foundation Admin Console.

Library Share

One of the beauties of virtual machines is that you don’t need to tie up a host if you are not actively using a VM. You can store it on a disk and bring it back to life on a host in a few minutes. SCVMM supports the concept of a library share where you can store virtual machines and other resources, such as, ISO images. The library share is nothing but a file share that is accessible to all the hosts. Similar to host groups, you can create multiple library shares for ease of management. For example, you could have a library share for storing pristine or golden OS images. Another library share could be used for storing VMs that have various application software components installed.

Environment

A typical multi-tier application consists of multiple roles, such as, Database Server, Web Server, Client, etc. Each role could be running on one or more computer. You could also have multiple roles running on a single computer. An environment is a set of roles that are required to run a specific application and the lab machines to be used for each role.

Managing environments for multi-tier applications is an error prone task today. Replicating the same environment at same or another site is even a bigger problem.

Lab Management surfaces environments as a first class entity.

Environment brings with it €˜strong’ group notion. That is when you do an operation on an environment, such as, start, stop, take snapshot, etc., that operation is applied on all the virtual machines that are part of the environment.

This document intends to provide the steps that can be followed to host a windows application from a web application.
First step would be to deploy the windows application using ClickOnce deployment.
Follow these steps to perform the ClickOnce deployment
1. In Solution Explorer, select the application project.
2. Right-click the project node and choose Publish.
The Publish Wizard appears.
3. On the Where do you want to publish the application? page, enter a valid URL using the format http://www.microsoft.com/foldername, then click Next.

4. In the Will the application be available offline? page, click the appropriate option:
€¢ If you want to enable the application to be run when the user is disconnected from the network, click Yes, this application will be available online or offline. A shortcut on the Start menu will be created for the application.
€¢ If you want to run the application directly from the publish location, click No, this application is only available online. A shortcut on the Start menu will not be created.
Click Next to continue.
5. Click Finish to publish the application.
Publishing status is displayed in the status notification area of the taskbar.
Once the windows application is published to a particular url then the next step would be to call it from a web application.
We can either use Process.start()method or we can use Respone.Redirect().
Below I have used the Response.Redirect(€œURL€) method and I am calling it in the button click event, so when the button is clicked the windows application will be opened.
private void btn_Click(object sender, EventArgs e) {
Response.Redirect(ConfigurationManager.AppSettings["WindowsApplicationURL"].ToString())
}
In the web.config file we can specify the exact path where we have published the windows application as shown below.
We can also pass querystring variable from the web application and use the same in the windows appliation to get only related data.
To pass the querystring we need to modify the url as shown below
http://localhost/GeneratePpt\ + GeneratePpt.application
i.e it will be the complete path where the windows application is published + the folder path + €œ.€ + application.
We also will have to modify the button click event to pass the query string as below:
private void btn_Click(object sender, EventArgs e) {
Response.Redirect(http://localhost/GeneratePpt\GeneratePpt.application?QueryStingId + querystringvalue);
}

A New Feature of .Net:

As we all know, every release of .Net framework carries with it, a greater list of new features and functionalities which makes coding easier and impressive though. Here in this post i would like to brief about the  feature, popularly known as Extension methods.

Extension Methods:

Earlier when we wanted to add some new features to an existing class, we usually end up creating custom classes and then write custom methods to achieve the desired functionality. Though in many places implementing custom classes is inevitable and unavoidable, there are scenarios where we can avoid them by simple using this new feature called Extension methods.

Short and Simple:

KISS, As one of the famous  mantras of coding -Keep It Short and Simple, extension methods of .Net  does the same. Yes, they are very simple to implement and very clean to use.

For ex..) we are all familiar with the DateTime structure of c# €€œ(Represents an instant in time, typically expressed as a date and time of day). Assume that we have a requirement where, we need to show up date in English(Culture €€œ en-US)at many places in the application. If so, we can create a extension method to the DateTime object and use it as similar to any other method of the  DateTime object.

Implementation:

Rules-

1. Extension methods are to be part of a static class.

2. Extension methods are defined as static methods but are called by using instance method syntax.

3. The first parameter of the method, specifies, on which type the method operates on (in our case its the DateTime type) and the parameter holds this keyword as its successor /modifier.

4. There could be additional parameters provided based on the functionality required for the implementation.

Pseudo code- creating Extension methods

public static class DateFormatter
    {
        public static string ToEnglishDate(this DateTime obj)
        {
            CultureInfo culture = new CultureInfo("en-US");
            return obj.ToString("dd MMMM yyyy", culture);
        }
    }

Pseudo code- calling Extension methods

DateTime time = DateTime.Now;
           Console.WriteLine(time.ToEnglishDate());
           Console.ReadLine();

Note:

In the above pseudo code block if you observe carefully, the extension method ToEnglishDate is being added to the DateTime type and is accessed as natively as we access the other methods and properties of the DateTime structure.

I hope this post added some value in introducing a simple but a very impressive feature of .Net framework, so lets start using them extensively and enjoy €€œ Happy Coding.

Java Architecture for XML Binding (JAXB) allows Java developers to map Java classes to XML representations.JAXB provides two main features: the ability to marshal Java objects into XML and the inverse, i.e. to unmarshal XML back into Java objects. In other words, JAXB allows storing and retrieving data in memory in any XML format, without the need to implement a specific set of XML loading and saving routines for the program’s class structure.

JAXB Design Goals

— Easy to use

€€œ Lower “barrier to entry” to manipulating XML  documents within Java programs

€€œ Don’t have to deal with complexities of SAX and DOM

— Customizable

€€œ Sophisticated applications sometimes require fine control over the structure and content of schema derived classes

€€œ Allows keeping pace with schema evolution Portable

€€œ JAXB components can be replaced without having to make significant changes to the rest of the source code

— Validation on demand

€€œ Validate the tree against the constraints in the source schema

— Clean “round-tripping”

€€œ Converting a Java content tree to XML content and back to Java content again should result in equivalent Java content trees before and after the conversion

JAXB Architecture

The binding compiler, which binds a given XML schema to a set of generated Java classes

The binding runtime framework, which provides unmarshalling, marshalling, and validation functionalities

The JAXB binding compiler (or xbj) lets you generate Java classes from a given XML schema. The JAXB binding compiler transforms an XML schema into a collection of Java classes that match the structure described in the XML schema. These classes are annotated with special JAXB annotations, which provide the runtime framework with the mappings it needs to process the corresponding XML documents.

The binding runtime framework provides an efficient and easy-to-use mechanism for unmarshalling (or reading) and marshalling (or writing) XML documents. It lets you transform an XML document into a hierarchy of Java objects (unmarshalling) or, inversely, transform a Java object hierarchy into XML format (marshalling). The term marshalling traditionally refers to disposing troops in some suitable manner. In networking, it refers to placing data items into a buffer before sending them over a communication channel.

Combined, these two components produce a technology that lets Java developers easily manipulate XML data in the form of Java objects, without having to know the nitty-gritty details of the Simple API for XML Processing (SAX) or the Document Object Model (DOM), or even the subtleties of XML Schema.

JAXBContext context = JAXBContext.newInstance(“<package-name>”) ;
Unmarshaller unmarshaller = context.createUnmarshaller() ;
Library library = (Library)unmarshaller.unmarshal (new FileInputStream(“library.xml”)) ;

————————————————————————————————————————————

library.xml is an example XML document that conforms to the schema file from which the JAXB-generated Java classes and interfaces are created, and Library is the root object in the XML document. Hereafter, we can use java idiom to manipulate the objects in the library.

An application can directly create java objects by using the ObjectFactory created by the xjc process.First, we get an instance of ObjectFactory.This factory instance is then used to create a library object.

We create objects in that library tree, set attributes for such objects& add the objects to the root.Finally, an instance of Marshaller class, is created and used to send the object to xml file.

ObjectFactory factory= new ObjectFactory();
Library library = factory.createLibrary();
Book book1 = factory.createBook();
book1.setTitle(“java today”);
book1.setPrice(“200Rs”);
library.add(book1);
Marshaller marshaller = context.createMarshaller();
marshaller.marshal(library, new FileOutputStream(“library.xml”)) ;
———————————————————————————————————————————-
Thus, the process of converting a java-object tree to xml is known as Marshalling.
To, recapitulate,
Unmarshalling means creating a java object tree for an XML document.
Marshalling means converting a javaobject tree into equivalent XML.

While developing web application comprising of several user controls we stumbled on a need for a possibility of bubbling client side event of one of the user control to invoke client side function in another user control or possibly a set of functions of different user controls.
The general approach of calling all dependant JavaScript functions on the control event will make the user controls tightly coupled. In other words, interdependent such that the event generating user control must know if the event dependant user controls are present or not; hence reducing the reusability of user controls. Further whenever a new user control is introduced with function that should be invoked on some existing user control event then the event source control must be modified every time.
So how can a user control integrated on a page only send messages to the controls that are interested in receiving the messages without knowing the identities of the receivers?
This is a classic context where Publisher/Subscriber pattern is applicable.
Just to brief on the Publisher/Subscriber pattern, the intent of this pattern is to define a loosely coupled many-to-many dependency between objects so that when some object changes state, all its dependents are notified and updated automatically. However, the object changing its state should not be concerned (or know) about presence of the dependants.
Thus enable listening user controls to subscribe to specific events and provide a mechanism that sends notification to all interested subscribers on the occurrence of specific events.
The three variations of the Publish/Subscribe pattern to create such a mechanism that sends messages to all interested subscribers are:
€¢ List-Based Publish/Subscribe
€¢ Broadcast-Based Publish/Subscribe
€¢ Content-Based Publish/Subscribe
I will not to get into the details of each variation, as it deviates from our main topic, but I should mention the option we adopted is List-Based Publish/Subscribe which advises us to identify a subject and to maintain a list of subscribers for that subject. When events occur, the subject notifies each subscriber on the subscription list.
As the topic suggests we will keep our implementation simple; so what we need now is an ability to maintain events and a list of subscriptions corresponding to each event. Speaking in terms of JavaScript what we need is an array to contain event names and another array of array to contain an array of functions (subscriptions) corresponding to each event name.
var _eventNameList = new Array();
var _functionList = new Array();

Note: _functionList is an array of which each element will be an array of subscriptions (functions).
Mapping to each event name in _eventNameList, corresponding to the same index value in the _functionList will be an array of functions that are interested in the specific event.
Next we need a JavaScript function that will return us the list of subscriptions corresponding to a given event name. Further this function will initialize an empty list if the given event name is not already present in the event name list.
function getFunctionList(eventName)
{
for(var i=0; i<_eventNameList.length; i++)
{
if(_eventNameList[i] == eventName)
return _functionList[i];
}
var index = _eventNameList.length;
_eventNameList[index] = eventName;
_functionList[index] = new Array();
return _functionList[index];
}
With the ability to manage the list in place, all we need now are two functions to complete the behavior of subject; one that allows listener JavaScript functions to subscribe to specific event and another that allows subjects to notify all listeners on the occurrence of the event.
function subscribeToEvent(eventName, fn)
{
var fnList = getFunctionList(eventName);
var index = fnList.length;
fnList[index] = fn;
}

function publishEvent(eventName)
{
var fnList = getFunctionList(eventName);
for(var i=0;i {
fnList[i]();
}
}
With this we have completed our publisher-subscriber implementation (in its bare minimal form).
Now, all that the user control interested in a specific event needs to do in order to subscribe for an event is call subscribeToEvent passing the name of the event and the actual JavaScript function that should be invoked.
function control2_Button1OfControl1Clicked()
{
alert(‘control2 invoked when button1 on control1 is clicked’);
}

subscribeToEvent(‘Button1OfControl1Clicked’, control2_Button1OfControl1Clicked);
The last piece is the invocation of publishEvent by the subject on the occurrence of the event.
function control1_Button1Clicked()
{
publishEvent(‘Button1OfControl1Clicked’);
}
And that completes our implementation of barely sufficient publisher subscriber pattern in JavaScript, which allows user controls to subscribe to event that may be raised by any other user control(s) and the notifying user controls remain ignorant of interested (subscribing) user controls.
This basic plan of implementation can be extended for passing event related information to the listener functions by modifying the publishEvent function.
function publishEvent(eventName, eventArgs)
{
var fnList = getFunctionList(eventName);
for(var i=0;i {
fnList[i]( eventArgs);
}
}
The listeners function€¦
function control2_Button1OfControl1Clicked(eventArgs)
{
alert(‘control2 invoked when button1 on control1 is clicked’ + eventArgs);
}

function control1_Button1Clicked(e)
{
if (!e) var e = window.event;
publishEvent(‘Button1OfControl1Clicked’, e);
}
The above described implementation can further be improvised to check before subscription if a listener has already subscribed for an event or/and ability to un-subscribe from an event, etc.

With Office 2007, Microsoft introduced a new document format called OpenXml. Its an xml based format for storing Mirosoft office documents. Technically, an Office 2007 file (pptx, docx etc) is simply a zip file containing a set of xml files and related content. You can actually, rename it to .zip and extract its contents! Although the idea is novel, the xml format itself is a little complex to work with. Microsoft does provide a OpenXML SDK to make life a little easier for programmers.

However, you’ll still need to learn a lot of tricks to get things done in OpenXml. I’ll demonstrate one such common task here. If you’re working with an OpenXml PowerPoint (pptx) file, one of the first things you may want to learn is how to clone a slide. i.e. make a copy of a slide along with all its contents. There are several sites that’ll give you code to Clone a slide. Here’s a good one which I’m using as a reference for my code: http://blogs.msdn.com/brian_jones/archive/2009/08/13/adding-repeating-data-to-powerpoint.aspx

However, I didn’t find any site that has a code which works with charts. When you clone a slide, all its parts should also be cloned for it to work properly. For eg. images are stored in a property called ImagePart. If you want images from source slide to appear in the cloned slide, you should copy the ImagePart too.

Similary charts are stored in a ChartPart. However, charts are more tricky as they refer to an embedded excel sheet that contains chart data. You’ll need to copy this excel sheet too. The following function has the sample code for charts and the embedded excel. It also has code to clone images to help you compare the differences:

public static SlidePart CloneSlidePartWithImagesAndCharts(PresentationPart presentationPart, SlidePart slideTemplate)
{
int i = presentationPart.SlideParts.Count();
//Create a new slide part in the presentation.
SlidePart newSlidePart = presentationPart.AddNewPart<SlidePart>(“newSlide” + i);
i++;
//Add the source slide content into the new slide.
newSlidePart.FeedData(slideTemplate.GetStream(FileMode.Open));
//Make sure the new slide references the proper slide layout.
newSlidePart.AddPart(slideTemplate.SlideLayoutPart, slideTemplate.GetIdOfPart(slideTemplate.SlideLayoutPart));

// copy the image parts
foreach (ImagePart ipart in slideTemplate.ImageParts)
{
ImagePart newipart = newSlidePart.AddImagePart(ipart.ContentType, slideTemplate.GetIdOfPart(ipart));
newipart.FeedData(ipart.GetStream());
}

// copy the chart parts
foreach (ChartPart cpart in slideTemplate.ChartParts)
{
ChartPart newcpart = newSlidePart.AddNewPart<ChartPart>(slideTemplate.GetIdOfPart(cpart));
newcpart.FeedData(cpart.GetStream());
// copy the embedded excel file
EmbeddedPackagePart epart = newcpart.AddEmbeddedPackagePart(cpart.EmbeddedPackagePart.ContentType);
epart.FeedData(cpart.EmbeddedPackagePart.GetStream());
// link the excel to the chart
(((newcpart.ChartSpace)).ExternalData).Id = newcpart.GetIdOfPart(epart);
newcpart.ChartSpace.Save();
}

//Get the list of slide ids.
SlideIdList slideIdList = presentationPart.Presentation.SlideIdList;

//Deternmine where to add the next slide (find max number of slides).
uint maxSlideId = 1;
SlideId prevSlideId = null;
foreach (SlideId slideId in slideIdList.ChildElements)
{
if (slideId.Id > maxSlideId)
{
maxSlideId = slideId.Id;
prevSlideId = slideId;
}
}
maxSlideId++;
//Add the new slide at the end of the deck.
SlideId newSlideId = slideIdList.InsertAfter(new SlideId(), prevSlideId);
//Make sure the id and relid are set appropriately.
newSlideId.Id = maxSlideId;
newSlideId.RelationshipId = presentationPart.GetIdOfPart(newSlidePart);
newSlidePart.Slide.Save();
return newSlidePart;
}

In current economic situations, IT folks worry about one thing €€œ €œreduce cost€. I have been frequently asked €œhow to reduce testing cost€. A no brainer answer would €œdo not do testing €¦ at all€. How many buy this idea €¦ can current breed of IT applications sustain with less or no testing at all? When I use the term testing €€œI am referring to €œnon-programmer€ testing.

Here is my draft list of suggestions …

1. Closely work with developers, do some parallel testing with them as the product/feature is getting developed

2. Identify and eliminate non-testing activities that occur in the name of process, documentation, management, metrics etc.

3. Analyze and profile every application under the portfolio to determine €œstable€ and €œwell tested€ areas of the application. These areas should receive the least or no testing effort.

4. Analyze the test scripts suite and remove redundant, worn out ones. Aim to reduce scripted test repository as small as you can.

5. Review and reduce €œregression testing€ on the basis of €œwell tested/stable areas€ of the application

6. Switch from resource intensive and highly scripted testing approach to highly improvisational exploratory /rapid testing approaches

7. Plan testing in small but frequent cycles (Session based exploratory testing approach) €€œ reduce planning and management overheads

8. Analyze and reduce the usage of costly tool licenses – especially those do not help in testing directly (test management tools)

9. Cut down on lengthy test plans, testing reports, dashboards €€œ switch to simple but frequent test reporting.

10. Simplify defect management process €€œ reduce defect life cycle €€œ resort to informal/quick defect communication.

Some this advice might look like a simple common sense (eliminate waste, focus on tasks that impact end result DIRECTLY). With so much selling happening about €œtesting tools€, €œfactory models€, €œcheap and best testing services€ €€œ any common sense is difficult to come by.

How would IT community react to these suggestions €€œ most likely response would €œThis would not work, how can we reduce testing, those test cases, processes, metrics, management practice?€. These suggestions would be most likely to be rejected on the grounds that testing cost needs to be reduced without €œcompromising quality€. Many IT folks think that quality comes from test scripts, processes, metrics, testing tools, automation etc. I am afraid quality is not such a simple thing.
Again, there are no free lunches here €¦ if you are thinking about reducing cost of testing, there are always risks of impacting quality (roughly goodness or confidence in the product) in one or other way. If you approach the problem (cost vs quality) from a quality side (improve testing – test better, deeper and wider), then the chances of achieving good quality and also €œsome€ cost benefits are more likely. However, if you approach it from €œcost€ side of the equation €€œ you might do achieve that albeit some impact on overall goodness/quality of delivered product.

Note that some suggestions mentioned in this list call for some smart testers who can think on their feet, work with least supervision, least (optimum) documentation and processes and so on. I think, the focus should shift from process, tools, management, documentation to Skill. There can be problems in getting such resources in IT scenario (especially in outsourced/offshored world)

You have choice €¦ which side you would like to approach the problem ..?

System testing in an agile development environment is a challenge, but the benefits to having good testing practice always add value to the overall project.

Few practices that can ensure agile testing success can be:

1. Test Early – The key to agile testing is iteration: Develop -> Test the developed code -> Develop again -> Test the latest developed code, the key here is starting the testing activity early. Here testing should start at the very beginning of the development cycle rather than after the first few sprints.

2. Test Often – A good agile test process emphasizes on frequency of testing. Here the objective should be to find as much as defects early in the testing cycle. The longer the defects wait in the code, the harder and more expensive they will be to fix.

3. Code Refactoring/Regression €€œ Stop every few weeks (at regular intervals) to focus on overall system stability. Fix bugs, refactor old code, and run extensive regression testing suite to make sure that no bugs slipped during the ongoing testing activities. This cycle should be repeated periodically

4. Test from a Customer (Business) Point of View – As with any development process, it’s critical that the testers and developers know the customer’s point of view. That means having good business scenarios that are derived from the customer, and then sharing those scenarios with the development team as well as the testing team.

5. Separate Testing from Development – This is often difficult for smaller teams, but testers should be always independent from development team. Keeping testers separate means they can develop true testing expertise while focusing on finding bugs. This also helps testers in having a perspective that is different from development team that will eventually help in finding bugs that are out of developer’s imagination.

6. Effective Communication €€œ Having good communication between the testers, developers, and product guys is a key essential to a solid agile process. Even though testers should be separate from developers, they should work closely together to get the most from testing activity.

7. Automate what you can – The best agile teams automate as much of their manual testing load as they can. Repeatedly testing the same test case over and over is a waste of time and money. Locating new bugs is far more valuable for any testing team than running same test cases again and again.

As the key to Agile development is speed, the key to testing in an Agile development project is also speed – creating test plans and test cases quickly, producing test results quickly, structuring your tests so that they are easy to update, and a test suite that withstands minor UI changes and/or needs little effort to maintain when it does break at some point of time.