Ease of use, flexibility, integrability, extensibility, performance, etc… are few terms that we make use to describe the standard of any tool or a software. Before adopting any software or api, we try to rate it on these parameters.

Today, I will be giving a brief introduction about an open source tool in Java which qualifies to be associated with majority if not all of these terms.

Business Intelligence and Reporting Tool (BIRT) is a java based open source tool that can make your job of report generation as a developer very very quick and easy. Backed by Actuate, BIRT is an Eclipse based reporting tool that is capable of generating pdf, excel, ppt, ods, odt, etc.. and many other reports formats. It can read data from a variety of sources including flat files, excel spread sheets, databases, xml, etc and also supports addition of custom sources as eclipse plugins.

There are a couple of other tools that can do something similar, their comparison can be found here: http://www.innoventsolutions.com/birt-jasper-pentaho-comparison-matrix.html & http://www.innoventsolutions.com/open-source-reporting-review-birt-jasper-pentaho.html.

Generating a report from a data source (db, files, excel, or a custom source) hardly takes any time. All that has to be done is to design the layout in a WYSIWYG report designer and it is enough to generate multiple formats (xls, pdf, ppt, etc) right away. Another very important advantage of using BIRT is, it has a very well maintained community collaboration website. BIRT Exchange (http://www.birt-exchange.org) is a place where BIRT users collaborate to resolve impediments, share tips and tricks and also contribute to the community by writing many custom utility plugins.

Besides that, Actuate hosts plugin development contests at regular intervals and winners get to walk away with brand new iPads and other consolation prizes. One such contest is underway and it would be  nice to see Provangers contributing to it. It isn’t rocket science and many including myself have won in such contests in the past.

Follow this to find out more about the contest http://www.birt-exchange.org/org/wiki/index.php?title=Plug_In_2_BIRT_Contest_for_Spring_2013_Details_and_Examples.

To know more about BIRT follow these links:

http://www.birt-exchange.org/org/new-to-birt/

http://www.birt-exchange.org/org/connect/

To know further, you can get in touch with me as well..

Happy BIRTing…

Megha Nidhi Dahal

JConsole is a graphical tool for monitoring Java applications. JConsole tool is available with the Java Development Kit since version 5. If you have never heard of JConsole you can refer to Using JConsole.

I’ll be explaining how JConsole can be used to monitor and tune Java applications.

Lets first start with monitoring. Here is the scenario, we have a multi-threaded batch processing job deployed in Production, we have added extensive logging as well. We get a call from production support that the job is stuck for couple of hours and we have to check what is going on. The first thing that we would do is request the log generated. The log file only tells us what happened before the process hung, but not where exactly it is stuck.

This is where JConsole comes in. We can quickly connect to the remote process and see what exactly is going on. The line of code where the process is stuck, the memory being used at that time, the number of threads being spawned and the state of each thread along with the call stack. All of which are vital to help us get to the root of the problem. And here is the best part, we get all this information in real time. We don’t have to sift through logs wondering what went wrong, we  can see what is going wrong then and there.

This comes in handy especially for a multi-threaded process where we may have threads blocking each other to get hold of expensive resources like a database connection. If we don’t have patience to check the state and call stack of each thread, JConsole can even tell us if there is a deadlock at the click of a button. I would say having monitoring enabled for an application or process is vital in maintaining it. Here is how you can setup remote monitoring.

Now that we know how JConsole helps us in monitoring Java applications, lets focus on performance tuning. This is very useful during the development phase.

Lets start with memory optimization. Lets say our application is a xml parser. It parses xmls and dumps the data into a relational database. We have hundreds or even thousands of xmls to parse and we don’t want to do it in a single thread, so we have built a multi-threaded application. What a lot of people neglect is the jvm heap memory required by an application. We usually think about it when we get the dreaded OutOfMemory error. Even then we try to solve it by trial and error, not knowing how much memory the application actually requires. This is where JConsole comes in handy. We can start the application and monitor the memory usage.

The above screenshot shows how it appears. You can see that the memory usage never goes very high and garbage collector is able to clear unused objects from heap.

 Understanding the memory usage is important especially when dealing with large data where one is not sure how much of it to hold in memory and how often serialize it to database. You can start the application and monitor it with JConsole. If you see a graph which is increasing linearly without dropping frequently it means you are not De-referencing objects to help clear memory. And if the memory usage reaches close to the limit set frequent GC cycles will bring down the performance of the application drastically. The graph should never be an increasing one, but like the one shown above where the memory usage is always in the mid range.

I’ll take up process optimization in the the next blog as it is a little big to cover in this one.

Often while writing an application or adding a module to an existing application we wonder if we should go with multithreading. Multithreading can be a powerful tool, but only if used the right way. Lets examine.

There are two main reasons when one should consider multithreading,

1. We have a GUI application and want to improve the user experience.
2. We have a batch job processing application and want to improve the performance.

Lets take the first situation. I have developed a desktop application which splits large files into smaller chunks. I choose a movie file and want to split this into smaller chunks. Depending on the size of the file this operation may take anywhere from few seconds to few minutes. Lets say in the middle of the operation, I realize I chose the wrong file. If my application is single threaded I cant do anything to stop this as the main thread is working away splitting the file. So, if my application uses one thread, I can do only one thing at a time.

Now it makes sense to have this application multithreaded. You can start the file splitting on one thread while you have the main thread listening to user inputs. The thread doing the splitting work can be interrupted in case the user cancels the operation or needs to do something else.

So, writing GUI applications with multithreading makes a lot of sense and almost all modern day applications are multithreaded.

Now lets take the second scenario. We have a batch processing job to do. We have a bunch of xml files which we have to parse and persist in a database. If I do this operation in my main thread, I would have to parse each xml file and put the data in database sequentially. If I want to speed this process up, the logical thing to do would be to process these files and update the database in parallel. But its not that simple, there is a catch.

One CPU can process only one thread at a given time. There is also an overhead involved in thread scheduling as time slicing will be done to give CPU time to different threads. So if 50 threads are started to process 100 files, the job will definitely slowdown on a single CPU machine. At the same time doing the process in one thread does not make sense as the process involves i/o and db operations which are time consuming. A thread waiting on i/o is automatically removed and another thread is given a chance by most operating systems today. So the ideal number here would be around 5 threads. In case the environment has more than one CPU, the number of threads can be increased. This can even be decided at run time. The number of CPUs available for the VM can be got from the Java API Runtime.getRuntime().availableProcessors(). This can be multiplied with a suitable factor to get the number of threads required to process the job.

Multithreading for batch jobs should be considered based on the platform on which the process runs. Even better would be to have this configurable at run time.

It is good to consider the above while developing multithreaded GUI or batch jobs. In case we didn’t get the required performance, we now know where we were going wrong.

Relax and Let me explain what I mean …

I have a sample problem to start with. For explanation purpose I am going to keep it simple, but I am sure the idea will be conveyed well. The problem statement goes like this.

As a real world user, I need to create a banking account to keep my money safe and easily manageable.

On creation of the account, I should get an account number (to keep it simple, we just return the name of the user for whom the account is being created as the account number)

I should have a choice of creating a Savings Account and a Current Account for various reasons like different rates of interest that I get in these types of accounts (but lets skip that for now).

Let me show you one of the solutions for this problem below:

I have the following UserTest class:

You can see, here I have two tests and both pass the criteria and I as a user am able to create both, Savings and Current account. Lets also look at how my User Class look like:

So here, I have a User class with followng methods:

createSavingsAccount and createCurrentAccount

Lets also have a look at our other classes / interfaces like Account, SavingsAccount and CurrentAccount

Now, the point here is that this code works perfectly fine and I get the user accounts of different types which also meet the criteria of my business. WOW!!

Or is it really a WOW!!

Let’s see whats the heck am I trying to say here. What is the problem here?

Well according to me, here, I have just made the compiler happy. So, I just looked form the compiler perspective and created some classes (OO language you see ) and dumped some methods in those classes and called those methods form other classes (my test in this case) and GOT THE JOB DONE!!

But, are we really here to make the compiler happy. I mean, that’s important, without that it won’t let you go anywhere. But is that all we care about? What about the human part of it.

Now have a closer look at what we did in the code. Go to the User class once again and imagine that he is a real world user. Imagine asking a real world user to create a SavingAccount or a CurrentAccount for you. You get a strange look there.. common, I am the user who wants the account to be created for me. How in the world should I know about creating an account?

Exactly, so what are these methods createSavingsAccount and createCurrentAccount doing there. THIS IS WRONG, if not INSANE to put these methods there. Common they don’t belong here.

Where is the Bank? A bank should know how to create a savings or a current account for a user.

So, maybe, when we thought of making just the compiler as our best friend, we forgot to create, what I would like to call a FIRST CLASS OBJECT Bank here.

Remember, Compiler is GOOD Friend but not nice enough to tell you how to make relationships. You have to be Human (of course you are, ain’t you) and start thinking like one to make good relationships (at least in software, its possible )

So, what according to me in this situation should be the better solution, and keeping the Compiler also as a good friend is as follows:

So, you see the difference, not much from he Compiler perspective, or form the number of lines of code perspective, but this code now is just more HUMAN code I would say

We just created a first class object Bank here and also made this bank responsible to create different accounts for me rather than the user himself, which to me is the correct way

That’s why I say, “Writing software is about being “Human” not “Compiler“”

I hope I have made the point clear.

Your comments are welcome.

Problem

Most Java 2 Platform, Enterprise Edition (J2EE) applications have a search and query requirement to search and list certain data. In some cases, such a search and query operation could yield results that can be quite large. It is impractical to return the full result set when the client’s requirements are to traverse the results, rather than process the complete set. Typically, a client uses the results of a query for read-only purposes, such as displaying the result list. Often, the client views only the first few matching records, and then may discard the remaining records and attempt a new query. [From Sun Developer Network]

Solution:

Pagination is the process of dividing information (content) into discrete pages. Pagination can be handled client-side or server-side. Server-side pagination is more common. Below image show a simple pagination:

Consider 20 records are shown per page.  How many records you fetch from database? 20 or complete result set or 20X10=200? Ideally you should restrict the fetch size to 200 for better performance.

Code snippet of EmployeeSearchValueListHandler:

public class EmployeeSearchValueListHandler {
private Integer currentPage;
private List<EmployeeDto> searchResult = new ArrayList<EmployeeDto>();
private List<EmployeeDto> currentPageRecords = new ArrayList<EmployeeDto>();
private Integer firstPageNbr;
private Integer lastPageNbr;
private final Integer MAX_RECORDS_PERPAGE=20;
private final Integer MAX_PAGELINKS=10;
private boolean isNextClicked=false;
private boolean isPreviousClicked=false;
private Map searchParams=new HashMap();
public void reset(){
// set default value to properties
}
// set this value when the user click on the page nbr
public void setSelectedPage(Integer selectedPageNbr){
this.currentPage=selectedPageNbr;
setNextPageRecords();
}
public void setNextClicked(boolean isNextClicked){
this.isNextClicked=isNextClicked;
}
public void setPreviousClicked(boolean isPreviousClicked){
this.isPreviousClicked=isPreviousClicked;
}
private void setNextPageRecords() {
if (isNextClicked) {
// retrieve next set of records from database by passing searchParams and addAll rows to searchResult
firstPageNbr=currentPage+1;
lastPageNbr=firstPageNbr+searchResult.size()/MAX_PAGELINKS;
if(searchResult.size()%MAX_PAGELINKS>0){
lastPageNbr++;
}
}
}
public List<EmployeeDto> getCurrentPageRecords(){
currentPageRecords.clear();
for(int count=1;count<=MAX_RECORDS_PERPAGE && (currentPage*MAX_RECORDS_PERPAGE+count)<searchResult.size();count++){
currentPageRecords.add(searchResult.get(count));
}
return currentPageRecords;
}
}

Save the instance of EmployeeSearchValueListHandler in session scope and reuse the same. Remove this object from session when the user navigates to other page.

If the trigger is calling the parent table select statement that results below oracle error

ORA-04091: table <tablespace>.<table> is mutating, trigger/function may not see it

eg:

create or replace TRIGGER EMPLOYEE_AUTRG AFTER UPDATE
ON EMPLOYEE
REFERENCING NEW AS NEW OLD AS OLD
FOR EACH ROW
  declare
    P_DEP_ID INTEGER;
    …………
    begin
       BEGIN
         FOR CUR_REC IN (
         SELECT  DEP_ID FROM EMPLOYEE WHERE EMPLYEE_NAME LIKE ‘<VALUE>’)
         LOOP
            ………………..
            UPDATE DEPARTMENT SET DEPTNAME = <value>  WHERE DEP_ID = CUR_REC.DEP_ID;
            ………………..
        END IF;
    END IF;
   EXCEPTION when OTHERS THEN
      dbms_output.put_line('no data found ');
   END;
  end if;
END;

Solution 1.

to resolve the above error add PRAGMA AUTONOMOUS_TRANSACTION with in the declare variables provided there must not be any update statement in the trigger

If update statement is available in the trigger which results in different sql exception (ORA-06519: active autonomous transaction detected and rolled back)  for such cases we can use commit after update statement  , here we can use commit in trigger with PRAGMA AUTONOMOUS_TRANSACTION in declare variables

Workable updated trigger:

create or replace TRIGGER EMPLOYEE_AUTRG AFTER UPDATE
ON EMPLOYEE
REFERENCING NEW AS NEW OLD AS OLD
FOR EACH ROW
  declare
    PRAGMA AUTONOMOUS_TRANSACTION;
    P_DEP_ID INTEGER;
    …………
    begin
       BEGIN
         FOR CUR_REC IN (
         SELECT  DEP_ID FROM EMPLOYEE WHERE EMPLYEE_NAME LIKE ‘<VALUE>’)
         LOOP
            ………………..
            UPDATE DEPARTMENT SET DEPTNAME = <value>  WHERE DEP_ID = CUR_REC.DEP_ID;
            commit;
            ………………..
        END IF;
    END IF;
   EXCEPTION when OTHERS THEN
      dbms_output.put_line('no data found ');
   END;
  end if;
END;

Solution 2.

We can use statement level triggers by removing  the row level we can avoid mutating ,trigger/function error

Statement level trigger example:

create or replace TRIGGER EMPLOYEE_AUTRG AFTER UPDATE
ON EMPLOYEE
REFERENCING NEW AS NEW OLD AS OLD
  declare
    PRAGMA AUTONOMOUS_TRANSACTION;
    P_DEP_ID INTEGER;
    …………
    begin
       BEGIN
         FOR CUR_REC IN (
         SELECT  DEP_ID FROM EMPLOYEE WHERE EMPLYEE_NAME LIKE ‘<VALUE>’)
         LOOP
            ………………..
            UPDATE DEPARTMENT SET DEPTNAME = <value>  WHERE DEP_ID = CUR_REC.DEP_ID;
            commit;
            ………………..
        END IF;
    END IF;
   EXCEPTION when OTHERS THEN
      dbms_output.put_line('no data found ');
   END;
  end if;
END;

Solution 3.

We can use compound trigger to avoid mutating,trigger/function error and this is introduced in oracle 11g

Compound trigger example:

create or replace TRIGGER EMPLOYEE_AUTRG AFTER UPDATE
ON EMPLOYEE
REFERENCING NEW AS NEW OLD AS OLD
FOR EACH ROW
COMPOUND TRIGGER
  declare
    P_DEP_ID INTEGER;
    …………
    begin
       BEGIN
         FOR CUR_REC IN (
         SELECT  DEP_ID FROM EMPLOYEE WHERE EMPLYEE_NAME LIKE ‘<VALUE>’)
         LOOP
            ………………..
            UPDATE DEPARTMENT SET DEPTNAME = <value>  WHERE DEP_ID = CUR_REC.DEP_ID;
            ………………..
        END IF;
    END IF;
   EXCEPTION when OTHERS THEN
      dbms_output.put_line('no data found ');
   END;
  end if;
END;

Sonar analyzes your source code according to the following quality axes :

• Respect of coding rules violations

• Density of documented API

• Density of duplicated Code

• Level of code coverage by Unit Tests

• Density of potential bugs

• Complexity distribution

Sonar drives many tools including among others :

• Static analysis : Checkstyle, PMD/CPD, Findbugs, JavaNCSS.
• Dynamic analysis for unit tests : Cobertura, Clover, Surefire.

Sonar Requirements

Sonar runs on any operating system that supports Java and Maven. Those two pieces of software need to be installed first:

• Java Development Kit v1.5 or later
• Maven 2.0.9+, 2.1.+ or 2.2.+

Sonar requires a relational database for storage of measures data. Sonar supports :

• MySQL 5.x+
• Oracle 10g+
• PostgreSQL 8.3+ (8.2 since Sonar 1.12)
• MS SQLServer 2005 since Sonar 2.0

• It is recommended to create an UTF-8 database named ‘sonar’ accessed by specific user ‘sonar’. Here is an example with MySQL:

mysql> CREATE DATABASE sonar CHARACTER SET utf8 COLLATE utf8_general_ci;

mysql> grant all privileges on sonar.\* to ‘sonar’@'localhost’ identified by ‘\[your password\]‘;

mysql> flush privileges;

Sonar Installations

• Download and Unzip the distribution
• Execute on windows

bin\windows-x86-32\StartSonar.bat

• If you do not use the default embedded database, edit conf/sonar.properties to configure the database access. Templates are available for every supported database. Just uncomment them and comment the first four lines dedicated to derby.

* sonar.jdbc.url : the URL of the database

* sonar.jdbc.driver : the class of the driver

* sonar.jdbc.user : the username (default is ‘sonar’)

* sonar.jdbc.password : the password (default is ‘sonar’)

Example for MySQL :

#sonar.jdbc.url: jdbc:derby://localhost:1527/sonar;create=true

#sonar.jdbc.driver: org.apache.derby.jdbc.ClientDriver

#sonar.jdbc.defaultTransactionIsolation: 1

#sonar.jdbc.validationQuery: values(1)

sonar.jdbc.url: jdbc:mysql://localhost:3306/sonar?useUnicode=true&characterEncoding=utf8

sonar.jdbc.driver: com.mysql.jdbc.Driver

sonar.jdbc.validationQuery: select 1

Install and Configure Sonar plugin on Hudson

Click on Manage Hudson – Manage Plugins. Click now on the Available tab, you should see the Hudson Sonar Plugin. Check this plugin and click on the Install button at the far right hand corner.

Restart Hudson and you should be able to see the plugin in the Installed tab as shown below:

Next, configure the Sonar plugin from within Manage Hudson -> Configure System. I am using the default database, and so didn’t make any changes here. However, if you are using a different database server, you need to provide the database URL, username, password and so on as shown below.

more…

java.util.Concurrent was added to the Java platform in version 1.5 . This package contains an Executor Framework, which is a flexible interface-based task execution facility. Creating a work queue:

ExecutorService executor = Executors.newSingleThreadExecutor();

Here is how to submit a runnable for execution:

executor.execute(runnable);

And here is how to tell the executor to terminate gracefully (if you fail to do this, it is likely that your VM will not exit):

executor.shutdown();

You can do many more things with an executor service. For example, you can wait for a particular task to, you can retrieve the results of tasks one by one as they complete (using an ExecutorCompletionService), you can wait for the executor service’s graceful termination to complete (using the awaitTermination method) , you can wait for any or all of a collection of tasks to complete (using the invokeAny or invokeAll methods) and so on.

If you want more than one thread to process requests from the queue, simply call a different static factory that creates a different kind of executor service called a thread pool. You can create a it with a fixed or variable number of threads. The java.util.concurrent.Executors class contains static factories that provide most of the executors you’ll ever need. If, however, you want some thing out of the ordinary, you can use the ThreadPoolExecutor class directly.This ThreadPoolExecutor  class lets you control nearly every aspect of a thread pool’s operation.

Choosing the executor service for a particular application can be tricky. If you’re writing a small program, or a lightly loaded server, using Executors.new-CachedThreadPool is generally a good choice, as it demands no configuration and generally €œdoes the right thing.€ But a cached thread pool is not a good choice for a heavily loaded production server! In a cached thread pool, submitted tasks are not queued but immediately handed off to a thread for execution. If no threads are available, a new one is created. If a server is so heavily loaded that all of its CPUs are fully utilized, and more tasks arrive, more threads will be created, which will only make matters worse. Therefore, in a heavily loaded production server, you are much better off using Executors.newFixedThreadPool, which gives you a pool with a fixed number of threads, or using the ThreadPoolExecutor class directly, for maximum control.

Not only should you refrain from writing your own work queues, but you should generally refrain from working directly with threads. The key abstraction is no longer Thread, which served as both the unit of work and the mechanism for executing it. Now the unit of work and mechanism are separate. The key abstraction is the unit of work, which is called a task. There are two kinds of tasks: Runnable and its close cousin, Callable (which is like Runnable, except that it returns a value). The general mechanism for executing tasks is the executor service. If you think in terms of tasks and let an executor service execute them for you, you gain great flexibility in terms of selecting appropriate execution policies. In essence, the Executor Framework does for execution what the Collections Framework did for aggregation.

ScheduledThreadPoolExecutor is  an Executor Framework  which is easier to use a timer which is more flexible than the java.util.Timer. A timer uses only a single thread for task execution, which can hurt timing accuracy in the presence of long running tasks. If a timer’s sole thread throws an uncaught exception, the timer ceases to operate. A scheduled thread pool executor supports multiple threads and recovers gracefully from tasks that throw unchecked exceptions

We often come across these standard questions while designing a software solution as to how efficient the design is? does it justify the project priorities? does it substantiate the architectural constraints and strategies? and most importantly, does it fullfill the requirement?
The idea behind this post is to review some of the design principles laid down by experts who have practiced and proven them over a period of time.

[Note: the usage of the term 'unit' is to represent a group of code which can be a component, class or function.]

• Law of Demeter: a.k.a. €œPrinciple of Least Knowledge€
  A unit should be focused on what it has to do with knowledge of existance of only a few other closely related units and minimal possible information about the properties of these close friend units. In effect a unit should not communicate with a logically stranger unit. For example:-
  Without applying the principal…

  class A
  {
             B b = new B();
            C c = new C();

                        MethodX()
                       {  
                                b.MethodY();
                                c.MethodZ();
                        }
             }

             class B
            {
                      C c = new C();

                      MethodY()
                     {
                               //do something with c
                     }
           }

           Applying the principal (making class A ignorant about the existance of class C)…

          class A
         {
                        B b = new B();

                        MethodX()
                       {  
                                b.MethodY();
                                b.MethodZ();
                        }
           }

           class B
           {
                      C c = new C();

                      MethodY()
                     {
                               //do something with c
                     }

                     MethodZ()
                     {
                               c.MethodZ();
                     }
          }

•  Subtraction:
  A design is finished when you cannot take anything else away. A design should make common things easy, and rare things possible and anything more should be removed.
  This principal also intersects with the following principle.
• Simplicity before generality:
  A common problem we find in frameworks is that they are designed to be general purpose without reference to actual systems.
  This leads to a dizzying array of options that are often unused, misused or just not useful. However, most developers work on specific systems, and the quest for generality does not always serve them well. The best route to generality is through understanding well-defined specific examples. So, this principle acts as the tie breaker between otherwise equally viable design alternatives. Of course, it is entirely possible that the simpler solution is the more general one.
• Once and once only:
  Avoid duplication of logic and structure where the duplication is not accidental, ie where both pieces of code express the same intent for the same reason.

These are the primary set of rules that lead to a good design approach or atleast the basic considerations to be made while developing or reviewing a design.
With the next post we will dwell deeper into more technical principals like Reflexivity, Orthogonality, etc.

The above JDBC error ORA-01000 might happen due to two factors-
1.Shortage in the database memory allocation(problem in database side).
2.Due to unclosed Statements and Connections in JDBC API (Problem in JDBC)

Solution:
1. If issue is due to first factor the DBA need to increase the memory.
2. If the second factor is the reason then all the statements and connections in JDBC API  need to be closed.
The second scenario can happen in multithreaded java application where all the threads are trying to access same JDBC API .If the connection & statements are not closed ,then pretty soon the memory will be over and the above error will be thrown.Below is a sample scripts to handle this problem(in JDBC)

Sample API:

public synchronized String getBundle_Code(String charge_code) throws Exception {
ResultSet rs = null;
String result = “”;
String query = “”;
try {
query = “select * from test”;
rs = getDBHandler().executeSQL(query);
if (rs.last() && rs.getRow() == 1) {
result = rs.getString(1);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
close(rs);
}
return result;

}

//DBHandler  Class contains all database access related queries

private synchronized DBHandler getDBHandler() {

if (dbh == null) {

config.initialize();

dbh = new DBHandler();

}

return dbh;

}

//This API will close all the connection and statements

private void close(ResultSet rs) throws SQLException {

if (rs != null) {

rs.close();

}

getDBHandler().closeStmt();

getDBHandler().disconnect();

}