Garbage Collector

A Garbage Collector is a Java program which tracked the referenced (live) objects and allowed them to keep in the heap memory whereas the memory of the unreferenced (dead) objects is reclaimed and reused for future object allocation. Here, one point to be noted that there is no explicit deletion and no memory is given back to the operating system. As long as an object is being referenced, the JVM considers it alive. Once an object is no longer referenced (dead) and therefore is not reachable by the application code, the garbage collector removes it and reclaims the unused memory. This process is also called as automatic memory management.

Type of Garbage Collector: 

There are four types of Garbage Collectors:

1. Serial Garbage Collector

2. Parallel Garbage Collector

3. CMS Garbage Collector

4. G1 Garbage Collector

Garbage Collection method involves:

1. Mark: To mark the generation of the object. 

2. Delete/Sweep: To delete the unreachable/dead object.

3. Compact: To compact the memory by moving around the object.

Some important terms:

1. Live Object: The object which is referenced by another object

2. Dead Object: The object which is not referenced by any other object and unreachable

3. Daemon Thread: Garbage collection process is carried out by Daemon Thread

4. System.gc(): It is used to invoke the garbage collector and on invocation, the garbage collector will run to reclaim the unused memory space. The System.gc() is a static method.

Java objects are created in Heap and this heap memory is divided into smaller parts or generations so that the performance of the object allocation can be increased. This object allocation behaviour is used to enhance the performance of the JVM as well. Therefore, the heap is broken up into 3 major parts or generations. 

These heap generations are: 

1. New or Young Generation: New Generation is further divided into three parts known as Eden space, Survivor 1 and Survivor 2 space. When an object first created in heap, memory gets allocated under new generation inside Eden space and after subsequent minor garbage collection if an object survives it gets moved to survivor 1 and then survivor 2. Minor garbage collections can be optimized assuming a high object mortality rate.

2. Old or Tenured Generation: The Old Generation is used to store long surviving objects. Typically, a threshold is set for young generation object and when that age is met, the object gets moved to the old generation. Eventually, the old generation needs to be collected. This event is called a major garbage collection. Major garbage collection reclaimed the memory of the old generation.

3. Permanent Generation: This space is reserved and used by JVM to store Metadata about classes and methods, String pool and Class level details. The permanent generation is populated by the JVM at runtime based on classes in use by the application. In addition, Java SE library classes and methods may be stored here.

How does GC work?

Object Creation and Memory Allocation: When new objects are created by JVM then heap memory is allocated to each object. To understand the GC in brief I took one example. 

Let's consider the limit of Eden is 5 i.e. only 5 objects can accommodate in Eden at a time. I named these objects as A, B, C, D and E. When these 5 objects are created, respective memory is allocated to them in Eden under young generation.  

When Eden limit is over: After some time, some of the objects become alive and some of the objects become dead. You can call the live object as referenced objects because those objects are referenced by some other objects and will be used in near future. The objects who become dead are called as unreferenced objects. 

In our example, you can see object A, B and E are live objects whereas object C and D are dead objects. 

Minor Garbage Collection 1: The minor garbage collection is an intermediate process which runs when Eden memory limit is over. The simple definition of Minor GC is to move the live objects from Eden to Survivor 1 (or Survivor 2) when Eden memory limit is over. There is no impact on old (Tenure) generation due to the execution of minor GC. During minor GC, 2 tasks happen:

a. Moving the live objects from Eden to S1 or S2

b. Remove the dead objects and reclaim the memory for new objects

In the figure you can see that object A, B and E have been moved to S1 and memory is reclaimed by removing the dead object C and D.

Again New Object Creation and Memory Allocation along with unreferenced objects: One more time the first step repeats and new objects are created by JVM and heap memory is allocated to each object. This time new objects F, G, H, I and J are created and Eden memory space is allocated to them. The only difference in the first step and this step is; this time some objects are available in the S1 as well, so now total we have 8 objects. During this activity, some of the objects in the Eden as well as in S1 may dead like object B and I. 

Minor Garbage Collection 2: Since the Eden memory is full so this situation again leads minor GC. Here one point to note that minor GC is a periodic process whenever Eden memory limit is over and every time it does the same tasks which I have mentioned above. Now, the live objects F, G, H and J will move to S2 along with A and E which exist in S1. The dead objects B and I will be removed and Eden memory will be reclaimed. 

Promotion: There is one more process which takes place along with all above-mentioned activities i.e. Object Promotion. After each minor GC, the objects become old and their age keeps on increasing. When aged objects reach a certain age threshold they are promoted from young generation to old generation. This process is called as Promotion. 

In our example you can see object A, G, J and O are promoted to old generation after 'n' number of minor GC executions. The promotion process leads to fill old generation memory space. One important point to note down here some of the objects like String and Array objects are directly created in tenured generation memory space.    

Major GC: Major GC cleans up the old generation. The task of Major GC is as same as the minor GC, but the only difference is minor GC reclaims the memory of the young generation whereas major GC reclaims the memory of the old generation. It is also said that many major GCs are triggered by minor GCs.

Full GC: A Full GC is triggered whenever the heap fills up. In such a case the young generation is swept first followed by the old generation. If the old generation is too full to accept the objects of the young generation, the young generation GC is omitted and the old generation GC (major GC) is used to clean and compact the full heap, either in parallel or serial. Either way, the whole heap is collected with a stop-the-world event.  

Related Term:

Stop the World Event: It is a time when all the application threads are stopped until the garbage collection operation completes. Both minor and major garbage collections are "Stop the World" events. A major GC is much slower than minor GC because it involves all live objects.


Reference:- https://perfmatrix.blogspot.com/

Loadtesting Websphere IBM MQ server using JMETER

For Loadtesting Apache Active MQ , JMS-Point-to-Point sampler in Jmeter will help us. Think that you do not have JNDI settings and no apache activemq for JMS messaging, instead IBM Websphere MQ is in place.  One day your manager is coming and asking you to test IBM MQ using open source testing tool as they do not ready to pay for small scale performance testing.

In that case, here is the steps to use Jmeter to load test IBM Websphere MQ.

Step 1:-

Below are the Java and MQ jars needed.  Download all these jar files and place it under Jmeter_Home/ lib/ext directory. 

1. jmeter-jms-skip-jndi-0.0.1.jar is the jar contains the MQ declaration and coding to connect it. You can down it from the URL  ‘http://www.java2s.com/Code/Jar/j/Downloadjmeterjmsskipjndi001jar.htm‘

2. com.ibm.mq.jar http://www.java2s.com/Code/Jar/c/Downloadcomibmmqjar.htm

3.org.springframework.beans.jar http://www.java2s.com/Code/Jar/o/Downloadorgspringframeworkbeansjar.htm

4.org.springframework.jms.jar https://mvnrepository.com/artifact/org.springframework/spring-jms/3.0.4.RELEASE

5.javax.jms.jar https://mvnrepository.com/artifact/javax.jms/javax.jms-api/2.0

6.spring-core.jar “https://mvnrepository.com/artifact/org.springframework/spring-core/3.1.1.RELEASE”

·Open Jmeter -> File-> New->TestPlan -> Add Thread Group

·Under Thread Group add Samplers->JMS Point-to-Point Sampler and configure the parameter. As given in the below Screenshot.

·Run Jmeter Script and Enjoy IBM MQ load testing.

You should be very care full about the JNDI properties they are case sensitive

1.channel:- BSIS.CLIENT

2.hostName:- bsisst.linux.us.ams1907.com

3.port:- 1420

4.queueManager:- BSISQM

5.transportType:-1

6.initial context factory:- ·com.elega9t.jmeter.jms.InitialContextFactory

7.Queueconnectionfactory:- CONNECTION_FACTORY

8.Request Queue: - QUEUE_ N413.DICE.BSIS.OCR.METRICS.QA3

How to determine Number of Load generator's required.

To determine the required number of load generators:

Record a script using a protocol

Replay a single Vuser in Controller and check the average CPU and the peak memory consumption of the mdrv.exe process by adding a counter for % Processor Time and Private Bytes.

Based on your load generator hardware and the CPU and memory consumption of a single Vuser, calculate the number of Vusers per machine.

For example:

Let us assume that each of the load generators has 8 core processors and 16GB of memory. Let us also assume that a single Vuser consumes 100MB of peak memory and 10% CPU on average for the specific business process.

From a CPU perspective:

If we limit the CPU consumption up to 70% utilization, we can have 7 Vusers per core processor (70% /10%). If our load generator has a total of 8 cores processors, 8 * 7 Vusers per processor equals 56 Vusers per load generator.

From a memory perspective:

The load generator machine has the 16GB memory of which 15GB is available for the Vusers so approximately 150 Vusers per load generator machine (15GB / 100MB). Around down the near whole number as we can't run part of the Vuser.

Therefore, to meet both the CPU and memory capacity limits, we use the lower number of Vusers and we calculate that for this business process, we can run approximately 56 Vusers per load generator.

Write Correlated/Extracted Values to a File Using Jmeter

After getting the data using a regular expression extractor/JSON Extractor,
Add a BeanShell PostProcessor element. Use the code below to write the variables to a file:


import org.apache.jmeter.services.FileServer;

String path=FileServer.getFileServer().getBaseDir();

name1= vars.get("user_Name_value");
name2= vars.get("UserId_value");

f = new FileOutputStream("E://csvfile/result.csv", true); //spec-ify true if you want to overwrite file. Keep blank otherwise.
p = new PrintStream(f);
this.interpreter.setOut(p);
p.println(name1+"," +name2);
f.close();

How to open Jmeter jtl file in Excel example

How to open Jmeter jtl file in Excel example gives you knowledge about open jmeter Test results jtl file in Excel workbook with real time example.Please follow below steps to open Jmeter jtl file in Excel.Using Jmeter you can save Test result file in csv,xls,txt and jtl formats and with the help of Simple Data Writer Listener as provided specific path in local system.

How to open Jmeter jtl file in Excel example

In order to perform above task we require below details,those are

1. Jmeter Test Plan.
2.Some HTTP/SOAP/REST service Samples.
3. Listener - Simple Data Writer.
4.Write results to file Path .
5.Set Up Thread group for Number of virtual users to run.
6.In case you are using apache-jmeter-3.0 new version then you need to copy paste below xsl files from old version and my old apache jmeter version is "apache-jmeter-2.13".
Go to apache-jmeter-2.13/extras folder then you will find below XSL format files,those are

1. jmeter-results-detail-report.xsl
2. jmeter-results-detail-report_21.xsl
3. jmeter-results-report.xsl
4. jmeter-results-report_21.xsl

7.Just copy paste above old version files in new version that is apache-jmeter-3.0/extras,it will looks like below.Let's see below example to generate jmeter results file in jtl format which is save in xml format with .jtl extension and adding some simple path in xml file,then only jtl file will open in Excel workbook.

Configurations in Simple Data Writer:

Before running your Test plan in Jmeter,you need to configure Sample Result Save Configuration in Simple Data Writer window panel,Please follow below steps.

STEP 1 : Add Simple Data writer from Right click on Thread Group - Add - Listener - Simple Data Writer
STEP 2 : Click on Configure button.
STEP 3 : Select require check boxes to save in results file.

Note : In case Save as XML checkbox is not selected in Sample Result Save Configuration dialog box then your jtl result file save in csv format but we require to save it in XML format.

STEP 4 : Select Save as XML check box in Sample Result Save Configuration dialog box.
STEP 5 : Now enter Write results to file/Read from File path with file name as E:\JMETER\TestResults\Test-results.jtl

Note : You should not create jtl extension file manually because Simple Data Writer automatically creates given file name with .jtl extension.

STEP 6 : Now Run your Test Plan
STEP 7 : Now open Specified File name results path and you will see Test-results.jtl file is generated as below.

JTL Test Result File

STEP 8 : Now open .jtl file in Notepad or Wordpad or Notepad++ ,it will open as below.

STEP 9 : Now copy below path in between <?xml version="1.0" encoding="UTF-8"?> and <testResults version="1.2">

Code : <?xml-stylesheet type="text/xsl" href="F:\Java_Applications\apache-jmeter-3.0\extras\jmeter-results-detail-report_21.xsl"?>

STEP 10 : Save your .jtl file and close the same.
STEP 11 : Open empty Excel workbook and drag and drop .jtl test results file into Excel workbook,select "Open the file with the following style sheet applied(Select One)"

STEP 12 : Click on OK button.
STEP 13 : Again click on Yes button.
STEP 14 : Now your Test Results reports will generate in Excel workbook as below.

Test Result:

You will see below Load Test Results.

Reading AWR Reports (Basic Approach)

If you are new to the AWR reports, the first thing you should probably do is run the ADDM report for the specific time period. The ADDM report provides root cause analysis of the parts of the system consuming the most time. It is often quicker to start with the ADDM report to help narrow down your area of focus in the AWR report.

When looking at an AWR report, a good place to start is the "Top 5 Timed Foreground Events" section, near the top of the report. This gives you an indication of the bottlenecks in the system during this sample period.

Once you've identified the top events, drill down to see what SQL and PL/SQL are consuming the majority of those resources. On the "Main Report" section, click the "SQL Statistics" link.

On the "SQL Statistics" section, click the "SQL ordered by ??" link that most closely relates to the wait event you identified in the "Top 5 Timed Foreground Events" section. In this case, the "DB CPU" was the top event, so it would seem sensible to try the "SQL ordered by CPU Time" link first.

You are then presented with the SQL and PL/SQL that are using most of the specified resource during the sample period. You can then attempt to tune these areas to reduce the impact.

AWR(Automatic Workload Repository) & It's Features

The AWR is used to collect performance statistics including:

• Wait events used to identify performance problems.
• Time model statistics indicating the amount of DB time associated with a process from the V$SESS_TIME_MODEL and V$SYS_TIME_MODEL views.
• Active Session History (ASH) statistics from the V$ACTIVE_SESSION_HISTORY view.
• Some system and session statistics from the V$SYSSTAT and V$SESSTAT views.
• Object usage statistics.
• Resource intensive SQL statements.

How to read AWR report?

This section contains detailed guidance for evaluating each section of an AWR report.  

Report Summary: This gives an overall summary of the instance during the snapshot period, and it contains important aggregate summary information.

Cache Sizes (end): This shows the size of each SGA region after AMM has changed them.  This information can be compared to the original init.oraparameters at the end of the AWR report.

Load Profile: This important section shows important rates expressed in units of per second and transactions per second.

Instance Efficiency Percentages: With a target of 100%, these are high-level ratios for activity in the SGA.

Shared Pool Statistics: This is a good summary of changes to the shared pool during the snapshot period.

Top 5 Timed Events: This is the most important section in the AWR report.  It shows the top wait events and can quickly show the overall database bottleneck.

Wait Events Statistics Section: This section shows a breakdown of the main wait events in the database including foreground and background database wait events as well as time model, operating system, service, and wait classes statistics.

Wait Events: This AWR report section provides more detailed wait event information for foreground user processes which includes Top 5 wait events and many other wait events that occurred during the snapshot interval.

Background Wait Events: This section is relevant to the background process wait events.

Time Model Statistics: Time mode statistics report how database-processing time is spent. This section contains detailed timing information on particular components participating in database processing.

Operating System Statistics: The stress on the Oracle server is important, and this section shows the main external resources including I/O, CPU, memory, and network usage.

Service Statistics: The service statistics section gives information about how particular services configured in the database are operating.

SQL Section: This section displays top SQL, ordered by important SQL execution metrics.

          SQL Ordered by Elapsed Time: Includes SQL statements that took significant execution time during processing.

          SQL Ordered by CPU Time: Includes SQL statements that consumed significant CPU time during its processing.

          SQL Ordered by Gets: These SQLs performed a high number of logical reads while retrieving data.

          SQL Ordered by Reads: These SQLs performed a high number of physical disk reads while retrieving data.

          SQL Ordered by Parse Calls: These SQLs experienced a high number of reparsing operations.

          SQL Ordered by Sharable Memory: Includes SQL statements cursors which consumed a large amount of SGA shared pool memory.

          SQL Ordered by Version Count: These SQLs have a large number of versions in shared pool for some reason.

Instance Activity Stats: This section contains statistical information describing how the database operated during the snapshot period.

          Instance Activity Stats (Absolute Values): This section contains statistics that have absolute values not derived from end and start snapshots.

          Instance Activity Stats (Thread Activity): This report section reports a log switch activity statistic.

I/O Section: This section shows the all important I/O activity for the instance and shows I/O activity by tablespace, data file, and includes buffer pool statistics.

          Tablespace IO Stats 

          File IO Stats 

          Buffer Pool Statistics 

Advisory Section: This section show details of the advisories for the buffer, shared pool, PGA and Java pool.

          Buffer Pool Advisory 

          PGA Aggr Summary: PGA Aggr Target Stats; PGA Aggr Target Histogram; and PGA Memory Advisory. 

          Shared Pool Advisory 

          Java Pool Advisory 

Buffer Wait Statistics: This important section shows buffer cache waits statistics.

Enqueue Activity: This important section shows how enqueue operates in the database. Enqueues are special internal structures which provide concurrent access to various database resources.

Undo Segment Summary: This section gives a summary about how undo segments are used by the database.

Undo Segment Stats: This section shows detailed history information about undo segment activity.

Latch Activity: This section shows details about latch statistics. Latches are a lightweight serialization mechanism that is used to single-thread access to internal Oracle structures.

          Latch Sleep Breakdown

          Latch Miss Sources

          Parent Latch Statistics

          Child Latch Statistics

Segment Section: This report section provides details about hot segments using the following criteria:

          Segments by Logical Reads: Includes top segments which experienced high number of logical reads.

          Segments by Physical Reads: Includes top segments which experienced high number of disk physical reads.

          Segments by Buffer Busy Waits: These segments have the largest number of buffer waits caused by their data blocks.

          Segments by Row Lock Waits: Includes segments that had a large number of row locks on their data.

          Segments by ITL Waits: Includes segments that had a large contention for Interested Transaction List (ITL). The contention for ITL can be reduced by increasing INITRANS storage parameter of the table.

Dictionary Cache Stats: This section exposes details about how the data dictionary cache is operating.

Library Cache Activity: Includes library cache statistics describing how shared library objects are managed by Oracle.

SGA Memory Summary: This section provides summary information about various SGA regions.

init.ora Parameters: This section shows the original init.ora  parameters for the instance during the snapshot period.

Database Performance testing using JDBC connection through Jmeter

Follow below steps to configure the Jmeter to connect your database.



STEP 1: Open a new Test Plan and add a Thread Group in it.
STEP 2: Add a 'JDBC Connection Configuration' from 'Configuration Element' menu into the Thread group.

STEP 3: Configure the 'JDBC Connection configuration' with respective database/tables as described below.

*Variable Name: We need to create a name for the JDBC pool. So that we can reference the pool name while executing a SQL query.

*Pool Configuration: Need to provide the number of concurrent connection in 'JDBC POOL'. (we used just 5, which means we can run just 5 parallel users only). Need to specify the pool timeout value too.
* Database Connection Configuration: This is the place we need to provide our database name, host and port.
Database URL:- jdbc:mysql://ip:port/databasename for MySql database
The access credentials need to be added into it.
* JDBC Driver Class:
Jmeter does not come with MySQL JDBC driver. We need to download the latest JDBC MySQL connector jar file driver(ex;- Mysql-connector-java-5.1.23-bin) from Oracle site and hold a copy in Jmeter/lib folder.
Mention the driver name in this column as "com.mysql.jbdc.driver" for MYSQL database,
we can execute the query.
STEP 4 : JDBC Sampler
Add a JDBC sampler from ADD > Sampler > JDBC Request.

STEP 5: Add the SQL

* Give the sampler a valid name to be referenced in results
* Provide a valid Variable pool name, which we created in STEP 3 ( 'JDBC POOL')
* Type your SQL Query in the space provided.

STEP 6: Add a View Results Tree to observe the results.

STEP 7 : Para-metering the SQL
Create a list of parameters to be used in SQL in a 'User Defined Variables' configuration.

Refer the above created parameters in the SQL as required.





STEP 8: Execute the Jmeter

Configure the Thread group with number of Threads. These value give the number of Vusers run concurrently.
Please make sure these value does not exceed the pool configuration connection count 5(STEP 3)

Now you can observe the result of SQL in the 'View Results Tree' node.

Creating Web Services Script in LoadRunner using WebService/Web(http/html) Protocol

Some Important Definitions:
 Web Services:
It is a web application components which can be published & used on web.
Web services do not require the use of browsers or HTML.
In Web Service XML is used to tag the data, SOAP is used to transfer the data, WSDL is used for describing the services available and UDDI is used for listing what services are available.
 XML: EXtensive Markup Language
XML is designed to describe the data & act as a tool for carrying information.
 SOAP: Simple Object Access Protocol
It is a protocol for accessing web services.
Soap is a format for sending messages & used for communication between applications.
 WSDL: Web Services description Language
It is an XML document which is used to describe & locate web services.
It specifies the location of the service and the operations (or methods).

 Data requirements:
•Sample Request & Response XML.
•End Point URL(Host) where request xml to be posted. E.ghttp://webservice.nam.nsroot.net/
•WSDLfile to get the information & soap action for all the existing operations/methods.
•Operation details which needs to be invoked like “GetLoadDetails”.
•SSL Certificate(also known as a Web Server Certificate, Secure Server Certificate, and Digital Certificate) & password if the host is SSL secured.
Two different ways of creating Web Services Script:

We can do web services scripting in two ways:

•By using Web Services Protocol.
•By using Web(HTTP/HTML) Protocol.
Steps for writing script:




•Open the Vugen and select Web Services protocol.
•Click on Import SOAP on top navigation bar and select request xml file.
•Click LOAD and provide End Point URL(Host) & Soap action then press OK.
•Soap request will be created with the step name “SOAP Request”.
Error Handling & Validation:

lr_xml_find and lr_xml_get_valuesfunctions can be used to validate the page. If-Else statement can be used to validate the resposne.
Example:
lr_xml_find("XML={response}",
"Query=//SOAP-ENV:Body/*/LoanAmount",
"Value=2000",
LAST );
lr_xml_get_values("Xml={response}",
"Query=//SOAP-ENV:Header/*/UUID",
"ValueParam=UUID",
LAST);
Error Handling:
if (strcmp(lr_eval_string("{Response}"), "0") != 0 )
{lr_end_transaction (“Submit_Request", LR_FAIL);
LR_EXIT_ITERATION_AND_CONTINUE;
lr_error_message(“Invalid Response"));
return 0;
}
else
{lr_end_transaction ("Submit_Request ", LR_PASS);
}

How to Handle SSL Issue in Scripting and Execution.

How to Handle SSL Issue in Scripting and Execution.

1.Shut connection during attempt to negotiate SSL session.

When this error occurs first verfiy the

1.web_set_sockets_option("SSL_VERSION","TLS");
try different options like TLS1.2,2,3,2&3.

2. Go to RUN-TIMESETTINGS

preferences ---> options---> Authatication and keep 1. Enable integrated authacation as YES.

And Use latest version of Load-generator only.
version impact should be there.

Add Parameter Name in the Transaction Name

It was just a case of using  lr_eval_string in the call to the transaction wrapper.


1. lr_start_transaction(lr_eval_string("Do something {piteration}"));
......................................

lr_end_transaction(lr_eval_string("Do something {pIteration}"),LR_AUTO);



you have to be careful above to make sure that start and end transactions names are the same. To overcome that problem I created a string variable to hold loadrunner transaction name.


char sTranName[20];
........

sprint(sTranName,lr_eval_string("TransactionA_{pIteration}"));

 lr_start_transaction(sTranName);
...........................

lr_end_transaction(sTranName,LR_AUTO);

 

How to handle java script generated Session ID in Vugen using Load runner

A session id (GUID) is generated on client side via a javascript and sent to the server. This cannot be correlated because it doesn’t come from the server.

You will need to figure this out from the page source. In my case, the sessionID is assigned a value of window.SessionId where window.SessionID = NewGuid()

The body of NewGuid() is -

function NewGuid() {
-------

-Creates a new pseudo GUID.
------

var guid = ‘xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx’.replace(/[xy]/g, function (c) {
var r = Math.random() * 16 | 0, v = c == ‘x’ ? r : (r & 0×3 | 0×8);
return v.toString(16);
});
return guid;

}



This generates a session ID like : 3f58555c-d6f4-4b1e-ac2a-b89f5d173944

then Use web_js_run() function.
Save the above javascript code with .js extension in the script folder.
Add the script under Extra files via Add Files option
Go to RunTimeSettings -Preferences-Options-Web Javascript header-Enable running javascript code == change the value to Yes
Write the following function.the the result is stored in param1