Category: Elasticsearch

Bulk Insert Java API of Elasticsearch

The bulk API allows one to index and delete several documents in a single request. Here is a sample usage

accounts.json (Json file which needs to be inserted in elasticsearch)

acccounts (download file from the given link)

Sample data present in file:

“lastname”:”Duke”,”age”:32,”gender”:”M”,”address”:”880 Holmes Lane”,”employer”:”Pyrami”,”email”:””,”city”:”Brogan”,

Java File

package com.elasticsearch.index;

import org.elasticsearch.action.bulk.BulkRequestBuilder;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.client.Client;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.InetSocketTransportAddress;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import org.json.simple.parser.ParseException;
public class IndexData {

public static void main(String[] args) throws ParseException, IOException {

// configuration setting
Settings settings = Settings.settingsBuilder()
.put(“”, “test-cluster”).build();
TransportClient client = TransportClient.builder().settings(settings).build();

String hostname = “<Your-Hostname>”;
int port = 9300;
client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(hostname),port));

// bulk API
BulkRequestBuilder bulkBuilder = client.prepareBulk();

long bulkBuilderLength = 0;
String readLine = “”;
String index = “testindex”;
String type = “testtype”;
String _id = null;

BufferedReader br = new BufferedReader(new InputStreamReader(new DataInputStream(new FileInputStream(“accounts.json”))));
JSONParser parser = new JSONParser();

while((readLine = br.readLine()) != null){
// it will skip the metadata line which is supported by bulk insert format
if (readLine.startsWith(“{\”index”)){
} else {

Object json = parser.parse(readLine);
_id = String.valueOf(((JSONObject)json).get(“account_number”));

//_id is unique field in elasticsearch
JSONObject jsonObject = (JSONObject) json;
bulkBuilder.add(client.prepareIndex(index, type, String.valueOf(_id)).setSource(jsonObject));

try {
if(bulkBuilderLength % 100== 0){
System.out.println(“##### ” + bulkBuilderLength + ” data indexed.”);
BulkResponse bulkRes = bulkBuilder.execute().actionGet();
System.out.println(“##### Bulk Request failure with error: ” +                                              bulkRes.buildFailureMessage());
bulkBuilder = client.prepareBulk();
} catch (Exception e) {

if(bulkBuilder.numberOfActions() > 0){
System.out.println(“##### ” + bulkBuilderLength + ” data indexed.”);
BulkResponse bulkRes = bulkBuilder.execute().actionGet();
System.out.println(“##### Bulk Request failure with error: ” + bulkRes.buildFailureMessage());
bulkBuilder = client.prepareBulk();


Maven dependencies:


Hit the elasticsearch URL to check the number of records inserted :

curl XGET https://<ip-address&gt;:9200/corsearch/unified/_count

Response : 
count: 1000,
total: 3,
successful: 3,
failed: 0



ElasticSearch cluster setup on AWS

Elasticsearch is a distributed search server offering powerful search functionality over schema-free documents in (near) real time. All of this functionality is exposed via a RESTful JSON API. It’s built on top of Apache Lucene and like all great projects it’s open source.

I’ll assume you already have an Amazon AWS account, and can navigate yourself around the AWS console. If not – get started over at

Fire up instances with your favorite AMI. I chose 3 instances of  m3.xlarge running the latest Ubuntu, but whatever works for you.

Steps to setup Elasticsearch cluster :

1. Download a copy of Elasticsearch 1.5.2


2. Unzip the zipped folder


3. Install the cloud AWS plugin so you can take advantage of features like automatic node discovery-

– Go to bin folder of ELASTICSEARCH_HOME and type –

. /plugin install elasticsearch/elasticsearch-cloud-aws/2.5.0

– You can check installed plugin through:

. /plugin –list

4. Configuring Elasticsearch by editing its elasticsearch.yml file – test-cluster "AWS-ES-1"
node.master: true true
index.number_of_shards: 3
index.number_of_replicas: 0
plugin.mandatory: cloud-aws _ec2_
discovery.type: ec2 ["", "", ""] xxxxxxxxxxxxxxx xxxxxxxxxxxxxxx


  • Pink colored configurations can be changed as per requirement.
  • name.node will be different for all the nodes and should be same.
  • Blue colored configurations will remain same for all the nodes.

5. Run elasticsearch


Your node is now running and is accessible (when using default configuration) on port 9200

curl -XGET ‘http://<EC2 instance ip>:9200/_cluster/health?pretty=true’
“cluster_name” : “test-cluster”,
“status” : “green”,
“timed_out” : false,
“number_of_nodes” : 3,
“number_of_data_nodes” : 3,
“active_primary_shards” : 3,
“active_shards” : 3,
“relocating_shards” : 0,
“initializing_shards” : 0,
“unassigned_shards” : 0,
“delayed_unassigned_shards”: 0,
“number_of_pending_tasks” : 0,
“number_of_in_flight_fetch”: 0,
“task_max_waiting_in_queue_millis”: 0,
“active_shards_percent_as_number”: 100

6. Repeat same steps for the rest of the nodes in your cluster.

That’s all !!