Wednesday, August 15, 2018

15. MongoDB : MongoDB Performance

15. MongoDB : MongoDB Performance



Commands to check STATS ###########################################


//execute below command from bin , before that u should have started mongod
//below files exist in bin folder
mongostat;   //check statistics
mongotop; // to track all read and write activity


db.serverStatus(); 
db.stats();    //number of collection, objects etc..
db.items.stats();  //collection level statistics


Journaling Mechanics ###########################################


- If we launch MongoD in Journal mode, then all write 1st goes to Journal before it goes to DB
     -- mongod.exe --dbpath D:\mongoDBDatabase --rest --journal
- Adv : fast and cheap (eg: while writing if server goes down, then when ever server comes up it takes from journal later)



Storage Engines ###########################################


1> memory map1 - read heavy operation , this is based on m-map files.
2> Wired-Tiger - heavy write operation, eg: facebook, twitter etc.
               - complex - updates possible
               - Uses commit chckpoint - after every 60 secs moving data from Journal to memory



Index - UnderIndex/OverIndex - poor design ###########################################


- All indexes are written to RAM, only if RAM is full then it writes to Harddisc which is very slow.
- Usually - in production 10% of Harddisck RAM shoud be there
  Eg: 500 GB HD,  50G GB RAM

Mongoos ###########################################


- Always use Mongoos tool - which handles all connection pool internally

Backup data ###########################################


- 1> copy paste all collections which is inside "D://mongoDBDatabase"   all "wt" files.
     always enable journal if you need backup
- 2> mongodump    //reads data from a mongDB database and creates BSON files. (it takes only documents as backup, it wont take index's, takes what index on what field)
     mongorestore   //


Lock DB during backup ###########################################


- use admin
- db.runCommand({"fsync":1, "lock":1});   //lock during backup

- db.$cmd.sys.unlock.findOne();    //unlock after backup is done.

14. MongoDB : Custom Search

14. MongoDB : Custom Search




Sample Data
{
"_id":0,
"name":"Pavan",
"results": [
{"evaluation":"term1","score":25.46},
{"evaluation":"term2","score":11.78},
{"evaluation":"term3","score":56.67}
]
}




1.      Find count and percentage of employees who failed in term 1, the passing score being 37
2.      Find employees who failed in aggregate (term1 + term2 + term3)
3.      Find the Average score of trainees for term1
4.      Find the Average score of trainees for aggregate (term1 + term2 + term3)    
5.      Find number of employees who failed in all the three (term1 + term2 + term3)
1.      Find the number of employees who failed in any of the three (term1 + term2 + term3)

SSample data:

cDatabase - "employee"

Collection - "training"

{"_id":0,"name":"Pavan","results":[{"evaluation":"term1","score":1.463179736705023},{"evaluation":"term2","score":11.78273309957772},{"evaluation":"term3","score":6.676176060654615}]}

{"_id":1,"name":"Vijay","results":[{"evaluation":"term1","score":60.06045071030959},{"evaluation":"term2","score":52.79790691903873},{"evaluation":"term3","score":71.76133439165544}]}
{"_id":2,"name":"amar","results":[{"evaluation":"term1","score":67.03077096065002},{"evaluation":"term2","score":6.301851677835235},{"evaluation":"term3","score":20.18160621941858}]}
{"_id":3,"name":"Girish","results":[{"evaluation":"term1","score":71.64343899778332},{"evaluation":"term2","score":24.80221293650313},{"evaluation":"term3","score":1.694720653897219}]}
{"_id":4,"name":"Shiv","results":[{"evaluation":"term1","score":78.68385091304332},{"evaluation":"term2","score":90.2963101368042},{"evaluation":"term3","score":34.41620148042529}]}
{"_id":5,"name":"Shobith","results":[{"evaluation":"term1","score":44.87186330181261},{"evaluation":"term2","score":25.72395114668016},{"evaluation":"term3","score":10.53058536508186}]}
{"_id":6,"name":"Bhavana","results":[{"evaluation":"term1","score":37.32285459166097},{"evaluation":"term2","score":28.32634976913737},{"evaluation":"term3","score":16.58341639738951}]}
{"_id":7,"name":"Monica","results":[{"evaluation":"term1","score":90.37826509157176},{"evaluation":"term2","score":42.48780666956811},{"evaluation":"term3","score":67.18328596625217}]}


################################################################################

package com.mongodb.customsearch;
import static java.util.Arrays.asList;

import java.util.ArrayList;
import java.util.Collection;

import org.bson.Document;

import com.mongodb.Block;
import com.mongodb.MongoClient;
import com.mongodb.client.AggregateIterable;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.MongoDatabase;
public class EmployeeTrainingScores {

public static void main(String[] args) {
try{
// Creating mongo client default connection host:localhost port:12027
//MongoClient mongoClient = new MongoClient();
MongoClient mongoClient = new MongoClient("localhost", 27017);
// fetching a database with name students
MongoDatabase db = mongoClient.getDatabase("employee");
// fetching collection scores from database students
MongoCollection collection = db.getCollection("training");
//find one record
findFailedStudInTerm1(collection);
//failed employees in aggregate (term1+term2+term3)
failedInAggregate(collection);
//average score of trainees in term1
averageScoreTerm1(collection);
//Average score of class for aggregate (term1 + term2 + term3)
averageClassScore(collection);
//count of employee failed in all three terms
employeeCountFailInAlLTerms(collection);
//count of employee failed in either of three terms
employeeCountFailAtleastATerm(collection);
mongoClient.close();
}catch(Exception exception){
        System.err.println(exception.getClass().getName() + ":" + exception.getMessage());
    }

}

/**
* function to find a record input : mongodb collection
* 1. Find count and percentage of employees who failed in term 1, the passing score being 37
*/
public static Long findFailedStudInTerm1(MongoCollection<Document> collection) {

Long count = collection.count(new Document("results.evaluation", "term1").append("results.score", new Document("$lt",37)));
Long totalStudents=collection.count();
Long per_Stud=(count*100/totalStudents);
System.out.println("############################# 1. Find count and percentage of employees who failed in term 1, the passing score being 37 #############################");
System.out.println("Number of students failed in exams in Term1: passing marks 37 ======> " + count);
System.out.println("Percentage of students failed in exams  in Term1: passing marks 37 ======> " + per_Stud +" %");
return per_Stud;
}
/**
* function to find a record input : mongodb collection
* 2. Find employees who failed in aggregate (term1 + term2 + term3)
*/
private static void failedInAggregate(MongoCollection collection) {
System.out.println("");
System.out.println("############################# 2. Find employees who failed in aggregate (term1 + term2 + term3) #############################");
//term1 + term2 + term3
//37 + 37 + 37 = 111
Collection<Document> employees = collection.aggregate(asList(new Document("$unwind","$results"),new Document("$group", new Document("_id","$name").append("total", new Document("$sum","$results.score"))),new Document("$match", new Document("total",new Document("$lt",111))))).into(new ArrayList<Document>());
for(Document doc:employees){
System.out.println("employees who failed in aggregate (term1+ term2 + term3) : " + doc.toJson());
}
}
/**
* function to find a record input : mongodb collection
* 3. Find the Average score of trainees for term1
*/
private static void averageScoreTerm1(MongoCollection collection) {
System.out.println("");
System.out.println("############################# 3. Find the Average score of trainees for term1 #############################");
Collection<Document> employees = collection.aggregate(asList( new Document("$unwind","$results"), new Document("$match",new Document("results.evaluation","term1")),new Document("$group",new Document("_id",null).append("Average", new Document("$avg","$results.score"))))).into(new ArrayList<Document>());
for(Document doc:employees){
System.out.println("Average score of trainees for term1 : " + doc.toJson());
}
}
/**
* function to find a record input : mongodb collection
* 4. Find the Average score of trainees for aggregate (term1 + term2 + term3)
*/
private static void averageClassScore(MongoCollection collection) {
System.out.println("");
System.out.println("############################# 4. Find the Average score of trainees for aggregate (term1 + term2 + term3)      #############################");
Collection<Document> employees = collection.aggregate(asList(new Document("$unwind","$results"),new Document("$group", new Document("_id","$name").append("Average", new Document("$avg","$results.score"))))).into(new ArrayList<Document>());
for(Document doc:employees){
System.out.println("Average score of trainees for aggregate (term1+ term2 + term3) : " + doc.toJson());
}
}
/**
* function to find a record input : mongodb collection
* 5. Find number of employees who failed in all the three (term1 + term2 + term3)
*/
private static void employeeCountFailInAlLTerms(MongoCollection collection) {
System.out.println("");
System.out.println("############################# 5. Find number of employees who failed in all the three (term1 + term2 + term3) #############################");
Long count=collection.count(new Document("results.0.score",new Document("$lt",37)).append("results.1.score",new Document("$lt",37)).append("results.2.score",new Document("$lt",37)));
System.out.println("Count of employees failing in all terms : " + count);
}
/**
* function to find a record input : mongodb collection
* 6. Find the number of employees who failed in any of the three (term1 + term2 + term3)
*/
private static void employeeCountFailAtleastATerm(MongoCollection collection) {
System.out.println("");
System.out.println("############################# 6. Find the number of employees who failed in any of the three (term1 + term2 + term3) #############################");
Long count=collection.count(new Document("$or",asList(new Document("results.0.score",new Document("$lt",37)), new Document("results.1.score",new Document("$lt",37)),new Document("results.2.score",new Document("$lt",37)))));
System.out.println("Count of employees failing in either of the terms : " + count);
}





}




################################################################################


13. MongoDB : Java program to Insert / Retrieve the documents

13. MongoDB : Java program to Insert / Retrieve / Update / Delete the documents



*************************************************************
1> MongoClient mongoClient = new MongoClient("localhost", 27017);
2> MongoDatabase db = mongoClient.getDatabase("test");
3> MongoCollection<Document> collection = db.getCollection("java_item");
4> Document document = new Document("title", "MongoDB").append("description", "database").append("likes", 100).append("name","Pavan");

5> collection.insertOne(document);
***************************************************************



///////////////////////////////////////////INSERT ONE DOCUMENT #############################

package com.mongodb.createdocuments;
import org.bson.Document;
import com.mongodb.MongoClient;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.MongoDatabase;
public class InsertDocuments {
    public static void main(String[] args) {

        // To connect to the mongoDB server
        MongoClient mongoClient = new MongoClient("localhost", 27017);
        // Now connect to your database, in mogoDB 4.0, DB has changed to MongoDatabase
        MongoDatabase db = mongoClient.getDatabase("test");
        System.out.println("Connection to the database successful.");
        MongoCollection<Document> collection = db.getCollection("java_item");
        System.out.println("Collection item selected successfully.");
        Document document = new Document("title", "MongoDB").append("description", "database").append("likes", 100).append("name","Pavan");
        collection.insertOne(document);
        System.out.println("Document inserted successfully.");
        mongoClient.close();

    }
}

#############################RETRIEVE DOCUMENTS####################################################

package com.mongodb.retrievedocuments;
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.DBCursor;
import com.mongodb.MongoClient;
//import com.mongodb.client.MongoDatabase;
public class RetrievedDocuments {
   
    public static void main(String[] args) {
        // To connect to the mongoDB server
        MongoClient mongoClient = new MongoClient("localhost", 27017);
        // Now connect to your database
        DB db = mongoClient.getDB("test");
        //MongoDatabase db = mongoClient.getDatabase("test");
        System.out.println("Connection to the database successful.");
        DBCollection collection = db.getCollection("items");
        System.out.println("Collection items selected successfully.");
        DBCursor cursor = collection.find();
        int index = 1;
        while (cursor.hasNext()) {
            System.out.println("Inserted document: " + index);
            System.out.println(cursor.next());
            index++;
        }
        System.out.println("Documents retrieved successfully.");
        mongoClient.close();
    }
}

#############################UPDATE DOCUMENTS####################################################

package com.mongodb.updateddouments;
import com.mongodb.BasicDBObject;
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.DBCursor;
import com.mongodb.DBObject;
import com.mongodb.MongoClient;
public class UpdateDocuments {
    public static void main(String[] args) {
        try{
        // To connect to the mongoDB server
        MongoClient mongoClient = new MongoClient("localhost", 27017);
        // Now connect to your database
        DB db = mongoClient.getDB("test");
        System.out.println("Connection to the database successful.");
        DBCollection collection = db.getCollection("java_item");
        DBCursor cursor = collection.find();
        System.out.println("Collection items selected successfully.");
        while(cursor.hasNext()){
        DBObject updateDocument = cursor.next();
        updateDocument.put("likes", 300);
        BasicDBObject query = new BasicDBObject().append("title", "MongoDB");
        collection.update(query, updateDocument, false, false);
        }
        System.out.println("Document updated successfully.");
        cursor = collection.find();
        int index=1;
        while(cursor.hasNext()){
            System.out.println("Updated Document: " + index);
            System.out.println(cursor.next());
            index++;
        }
        mongoClient.close();
        }catch(Exception exception){
            System.err.println(exception.getClass().getName() + ":" + exception.getMessage());
        }
    }

}

#############################DELETE DOCUMENTS####################################################

package com.mongodb.deletedocument;
import com.mongodb.BasicDBObject;
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.DBCursor;
import com.mongodb.MongoClient;
public class DeleteDocument {
   
    public static void main(String[] args) throws InterruptedException {
        // To connect to the mongoDB server
        MongoClient mongoClient = new MongoClient("localhost", 27017);
        // Now connect to your database
        DB db = mongoClient.getDB("test");
        System.out.println("Connection to the database successful.");
        DBCollection collection = db.getCollection("java_item");
        System.out.println("Collection items selected successfully.");
        BasicDBObject searchQuery = new BasicDBObject().append("title", "MongoDB");
        collection.remove(searchQuery);
        DBCursor cursor = collection.find();
        int index = 1;
        while (cursor.hasNext()) {
            System.out.println("Inserted document: " + index);
            System.out.println(cursor.next());
            index++;
        }
        System.out.println("Documents retrieved successfully.");
        mongoClient.close();
    }
}









12. MongoDB : GridFS - to store (static data, audio, video large files)

12. MongoDB : GridFS - to store (static data, audio, video large files)



**************************************************************************
//save file on db
GridFSInputFile gfsFile = gfsPhoto.createFile(imageFile);
gfsFile.save();

//get all grid files
DBCursor cursor = gfsPhoto.getFileList();

//get particular grid file
GridFSDBFile imageForOutput = gfsPhoto.findOne(newFileName);

//remove grid file from db
gfsPhoto.remove(gfsPhoto.findOne(newFileName));

**************************************************************************
bez we know 16MB is the max in document.
GridFS -> how it stores more than 16MB, internally it split in to chunks and stores



###############################JAVA PROGRAM TO STORE/READ/DELETE GRIDFILE FROM DB###############################################

-- Download JDK 1.8 (Version that you prefer) for Windows/Linux
-- Install Eclipse oxygen
-- Create a project in eclipse
-- Create a simple Java Class in eclipse
-- Use the code below to upload an image file
-- Get these dependencies and add it to the java classes build path

mongodb-driver-core-3.4.2.jar
mongodb-driver-3.4.2.jar
bson-3.4.2.jar

https://mvnrepository.com/artifact/org.mongodb/mongo-java-driver/3.4.2   -> mongodb-driver-core-3.4.2.jar
https://mvnrepository.com/artifact/org.mongodb/mongodb-driver/3.4.2   -> mongodb-driver-3.4.2.jar
https://mvnrepository.com/artifact/org.mongodb/bson/3.4.2    -> bson-3.4.2.jar


##############################################################################

package com.mongodb.gridfs;
import java.io.File;
import java.io.IOException;
import com.mongodb.DB;
import com.mongodb.DBCursor;
import com.mongodb.MongoClient;
import com.mongodb.gridfs.GridFS;
import com.mongodb.gridfs.GridFSDBFile;
import com.mongodb.gridfs.GridFSInputFile;
public class UploadFile {
    public static void main(String[] args) throws IOException {

        MongoClient mongoClient = new MongoClient("localhost", 27017);
        DB db = mongoClient.getDB("sample");
        String newFileName = "gridFS-java-image";
        File imageFile = new File("C:\\Users\\Pictures\\MyFile.png");
        // create a "photo" namespace
        GridFS gfsPhoto = new GridFS(db, "photo");
        // get image from local drive
        GridFSInputFile gfsFile = gfsPhoto.createFile(imageFile);
        // set the new file name for identity
        gfsFile.setFilename(newFileName);
        // save the image file to mongoDB
        gfsFile.save();
        // print the result
        DBCursor cursor = gfsPhoto.getFileList();

while (cursor.hasNext()){
            System.out.println(cursor.next());
        }
        // get the image file by it's filename
        GridFSDBFile imageForOutput = gfsPhoto.findOne(newFileName);
        // save it to a new image file
        imageForOutput.writeTo("C:\\Users\\Pictures\\MyFileFromMongDB.png");
        System.out.println("Done");
        // Remove the file.
        gfsPhoto.remove(gfsPhoto.findOne(newFileName));
        // close the connection
        mongoClient.close();
    }
}

##############################################################################

11. MongoDB : CAP COLLECTION & TTL Index.

11. MongoDB : CAP COLLECTION & TTL Index.


CAP COLLECTION - Create collection with fixed size. ######################################

Why we need this kind of collections ...? 
- Eg: Server Log - we dont want to keep historical data, as soon as new data comes in old data has to delete.

//max size of 5242880 bytes, and 5000 documents
db.createCollection("webserverlog", {capped:true,size:5242880,max:5000});
show collections
db.webserverlog.isCapped()
db.runCommand({"convertToCapped":"items",size:100000});  //to convert to cap collection
db.items.isCapped()





//How to delete the documents
TTL Index (Time to live) ###########################################
-----------------------------
//document will be deleted after (createdDate + 3600 sec), eg: ('May 30, 2015 14:00:00' + 3600 sec)
db.webserverlog.createIndex({"createdDate":1},{expireAfterSeconds:3600});

//document will be deleted after (expireDate + 0 sec), eg: ('June 30, 2015 14:00:00' + 0 sec)
db.webserverlog.createIndex({"expireDate":1},{expireAfterSeconds:0});
db.webserverlog.insert({"expireDate":new Date('June 30, 2015 14:00:00')});



10. MongoDB : Sharding

10. MongoDB : Sharding



9. MongoDB : Replication

9. MongoDB : Replication