Database « Intelligrape Groovy & Grails Blogs

Archive for the ‘ Database ’ Category

Export very large data from a MongoDB collection using script

Posted by on July 4th, 2012

In this post, we will learn how to use javascript as MongoDB shell script to export (fields of) a collection.

 

If you just want to execute MongoDB commands then look at the another post there..

 

OK, so let’s focus back to the objective of this post.
Generally, to export data from a MongoDB collection, we use the following command -

mongoexport -vvvv --host 127.0.0.1 --db my-db --username='usr' --password='pwd' --collection profiles --csv --out /home/data/usernames.csv -f 'username'

Definitely I prefer the above mentioned way. But let’s consider that you have millions or billions of records in a MongoDB collection. And you want to export all these.
Probably, this will just hang. Well, I am not sure what happens on your computer but on my machine it was just stuck for hours and then I terminated the process. And came up with scripting solution – where I created multiple files (chunks) of the exported data.

 

var size = 1000000;
var maxCount = 1;
for (x=0;x<maxCount;x=x+1)
{
   var recToSkip = x*size;
   db.profiles.find().skip(recToSkip).limit(size).forEach( function(record){
     var username = record.username;
     print(record.username);
   })
}

And ofcourse to run the script (if above javascript is in file named – ‘usernames.js’)

  mongo 127.0.0.1/my-db --username='username' --password='pwd' export-usernames.js > output.txt

The above shell command will render all the output (from export-usernames.js) to the output.txt file. And mind it, that this is single file where you get all exported data. But if you want to chunkify this – just write your wrapper script or run multiple times. It’s upto you.

 

I hope this might help someone. Or please feel free to add your comments.

 

Cheers!
Salil Kalia
Salil [at] IntelliGrape [dot] com
Twitter LinkedIn

Removing/replacing special character from database.

Posted by on June 11th, 2012

Recently in one of my project i faced a problem that database was having special character, which is shown as space on the User Interface.

So as to solve this issue i found a very simple solution which consists of following steps:

1. First, you just need to identify which type special character to be removed/replaced like in my case it was shown as space but stored as <?> in the db.

For that you can use the query.

select HEX(email) from person;

and you can also refer to this link for ASCII code of character. In my case it was 160.

2.  Now, next step is simple just run this query.

 update person set email=replace(email,char(160),''); 

Thats it !!!  :)

It helps me a lot!!! Hope that helps you guys!!!

Anuj Aneja

Intelligrape Software Pvt. Ltd.

Tags: ,
Posted in Database

get totalCount of records when using limit in one Query

Posted by on May 17th, 2012

For pagination we generally execute two query ,first for geting resultset by using limit in sql query and second to count the total no. of records . For total no. of records we again execute the same query with count(*).

You would need two queries like these:

SELECT COUNT(*) FROM author WHERE name LIKE 'a%';

SELECT name, email FROM author WHERE name LIKE 'a%' LIMIT 10;

But if you have a complex query that joins several tables and takes a while to execute – well, you probably wouldn’t want to execute it twice and waste server resources.

Since MYSQL 4.0 we can use SQL_CALC_FOUND_ROWS option in  query which will tell MySQL to count total number of rows disregarding LIMIT clause. In main query add SQL_CALC_FOUND_ROWS option just after SELECT and in second query  use FOUND_ROWS() function to get total number of rows without executing the query.

Queries would look like this:

SELECT SQL_CALC_FOUND_ROWS name, email FROM author WHERE name LIKE 'a%' LIMIT 10;

SELECT FOUND_ROWS();

Limitation: Must call second query immediately after the first one(or before next one) because SQL_CALC_FOUND_ROWS does not save number of rows anywhere.In the absence of the SQL_CALC_FOUND_ROWS option in the most recent successful SELECT statement, FOUND_ROWS() returns the number of rows in the result set returned by that statement.
(more…)

Posted in Database

Handling corrupted references through ignoreNotFound database mapping in Grails

Posted by on April 17th, 2012

Hi,
 
Recently i had a use-case where we have the legacy database and it contains the corrupted references of a non existent record in many-to-one relationship, and i have to populate a grid that contain info also from referenced record. As referenced record doesn’t exist, So when we refer to the certain record will result in Hibernate throwing exception :

org.hibernate.ObjectNotFoundException

and not a single record loaded on grid.
 
As it is legacy database i am not sure what should be the correct value for the particular record.
 
After few searches, i come accross a property in grails database mapping to handle such cases, let see how it work :

class Task{
   String name
   ParentTask parent

   static mapping = {
       parent ignoreNotFound: true
   }
}

class ParentTask{
   String name
}

Basically, ignoreNotFound simply mapping hibernate not-found property.
 
Note : A record loaded with ignoreNotFound: true will throw an exception during call of save() because of the missing reference.
 
Either you can first update the invalid property or corrupted reference to the correct reference and then update other property that will solve your problem. It depend how you handle such scenario on your usecase.
 
Thanks,
Tarun Pareek
tarun@intelligrape.com
http://in.linkedin.com/in/tarunpareek
More Blogs by Me
 

Posted in Database, Grails

Clear Time function for MySQL

Posted by on April 12th, 2012

Hi all,

Here is a simple function that can allow you to clear time from the DATETIME field of your database table

DROP FUNCTION IF EXISTS cleartime;
delimiter //
CREATE FUNCTION cleartime(dt datetime) RETURNS DATETIME
  NO SQL
  DETERMINISTIC
BEGIN
  DECLARE t    varchar(15);               -- the time part of the dt
  DECLARE rdt  datetime default dt;    	  -- the datetime after time part is cleared

  SET t = TIME(dt); 
  SET rdt = SUBTIME(dt,t);

  RETURN rdt;
END //
delimiter ;

To import this function to your database just copy and paste above code in MySQL command prompt with your database selected.

The implementation is shown in following example:

select cleartime(datetime_field) from table_name;     -- to view field with its time cleared
                     --OR
update table_name set datetime_field=cleartime(datetime_feild) from table_name;
Posted in Database

mysqldump of particular records without create – drop of tables

Posted by on March 27th, 2012

Hi,
 
Recently i was in a situation where i need to take the dump of particular records in mysql. We are pretty much familiar with the normal mysqldump statement given below :

mysqldump --user=usr --password=pwd database_name table_name > dumpFile.sql;

The above statement will provide us the dump of the respective database and table into the dumpFile.sql. But if you open the dump file you will see something like this :
 

    //dumpFile.sql
DROP TABLE IF EXISTS `table_name`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `table_name` (
  ...
) ENGINE=InnoDB AUTO_INCREMENT=204182 DEFAULT CHARSET=latin1;

--
-- Dumping data for table `table_name`

LOCK TABLES `table_name` WRITE;
/*!40000 ALTER TABLE `table_name` DISABLE KEYS */;
INSERT INTO `table_name` VALUES ('1','data1');
/*!40000 ALTER TABLE `table_name` ENABLE KEYS */;
UNLOCK TABLES;

Because of the above dump, it will always first drop and recreate the table before sourcing your dump into the database.
So our first task to remove the creation and drop of table from the mysqldump, because when we take dump of particular records, What we want is just inserting the records in database without create and drop of tables and no loss of existing data.
 

    //Creating Dump without create and drop of table :
mysqldump --user=usr --password=pwd --no-create-info database_name table_name > dumpFile.sql;

If you compare the first and above statement there is only one change, –no-create-info option :
It don’t write CREATE TABLE statements that re-create each dumped table. So our dumpFile.sql now contains only this :

--
-- Dumping data for table `table_name`

LOCK TABLES `table_name` WRITE;
/*!40000 ALTER TABLE `table_name` DISABLE KEYS */;
INSERT INTO `table_name` VALUES ('1','data1');
/*!40000 ALTER TABLE `table_name` ENABLE KEYS */;
UNLOCK TABLES;

 

    //Creating Dump of particular records using conditional clause and without create and drop of table :
mysqldump --user=usr --password=pwd --no-create-info database_name table_name --where="id in (1,2,3)"> dumpFile.sql;

In above statement with –no-create-info, we removed the create-drop of table and by –where option we can specify our condition on basis of which we want to fetch the particular records. In above statement it will fetch the record in table that has value 1, 2 or 3 in field ‘id’. So our dumpFile.sql now contains only this :

--
-- Dumping data for table `table_name`
--
-- WHERE:  id in (1,2,3)

LOCK TABLES `table_name` WRITE;
/*!40000 ALTER TABLE `table_name` DISABLE KEYS */;
INSERT INTO `table_name` VALUES ('1','data1'),('2','data2'),('3','data3');
/*!40000 ALTER TABLE `table_name` ENABLE KEYS */;
UNLOCK TABLES;

 
I hope it helps. :)
 
Thanks,
Tarun Pareek
tarun@intelligrape.com
http://in.linkedin.com/in/tarunpareek
More Blogs by Me

Posted in Database

Alphanumeric Sorting using Criteria Query (with MySQL database)

Posted by on February 2nd, 2012

I am working on a Grails application with MySQL database. I had a use case in which I had to implement alphanumeric sorting using Criteria Query on Grails. By alphanumeric sorting I mean if there is a class Employee with field empId then on doing :

 
Employee e1 = new Employee(empId:'emp10').save()
Employee e2 = new Employee(empId:'emp2').save()
Employee e3 = new Employee(empId:'emp1').save()
Employee e4 = new Employee(empId:'1emp').save()
Employee e5 = new Employee(empId:'10emp').save()
Employee e6 = new Employee(empId:'2emp').save()

Employee.createCriteria().list([sort:'empId'])
            /* OR */
Employee.createCriteria().list{
    order('empId')
}

should give result like [e4,e6,e5,e3,e2,e1]. But according to sql sorting the result would be [e5,e4,e6,e3,e1,e2]

One thing to note was that criteria query implemented sorting at the database end.
After googling around, I came across a script that had a SQL function which converted the field value to string that can alphanumerically sorted easily at database end.

To migrate the functions in script to database you have to unzip the script and write following command on terminal:

mysql -u username -p database_name < /path/to/unzipped/directory/natsort.install.mysql

Now only thing I had to do was to get a derived field from the Grails property that had to be sorted alphanumerically which was easily possible as the Grails had ability to create transient field using formula (thanks to wonderful Blog by Uday). Thus I had to create a new transient field using SQL function in formula. So I did something like this :

class Employee {
    String empId
    String empIdSortField           //transient field
    static mapping = {
        empIdSortField formula: 'natsort_canon(empId, "natural")'
    }
}

Now if I had to get Employee objects are sorted by empId alphanumerically, what I would do is :

Employee.createCriteria().list {
    order('empIdSortField')
}

Hope this was helpful to you!

Posted in Database, GORM, Grails

Dbconsole in Grails.

Posted by on December 25th, 2011

So, Grails 2.0 was released a few days back and I upgraded my application to it as soon as I came to know of its final release and it rocks.!!

Among the many things which are making a lot of noise on grails 2.0, there seems to be a lack of noise over the GUI Database console which grails has provided. Developers can connect to the database from a GUI right inside there application, see the data in tables and execute simple queries. Probably its not polished yet so that could be a reason why there is not much noise over it but it still is a usable feature.

All you need to do is, run your grails 2.0 app navigate to http://localhost:8080/app-name/dbconsole/ and configure your database connection. Just select what database you are using, Select the driver, a couple of entries for username and password and you are good to go.

The query builder isn’t too great, but that is no reason to miss this useful tool. No need to move to any other application or console to see whats in the database. The database is available right in your application, and its available only in development mode, so no worries of this slipping into production. Hope this saves some time for all of us.

Thanks.
Sachin Anand
mail : sachin[at]intelligrape[dot]com
twitter : @sachin__anand

Posted in Database, Grails

Batch update performance enhancements using SQL withBatch()

Posted by on July 5th, 2011

Hi guys,

Recently as part of a project, I had to populate a SQLite database with large amounts of data pertaining to a number of classes requiring more than 5000 inserts and updates per class. I created a new SQLite database using Groovy’s Sql class. The initial strategy involved creating prepared statements and executing individual insert/update statements for each record that needed to be inserted/updated in the new SQLite database.

However the process was taking longer than expected, and the cumulative time taken for the whole application to sync less than 30 classes was coming to be more than 2 minutes. This time taken was extremely high regarding the context of the application and was a real performance bottleneck. What I did notice was that the insert statements were identical for a particular class, disregarding the values that needed to be inserted. The same was the case with the update statements.

After looking through the Sql GDK, I found a method named Sql.withBatch() that performs batch manipulation of records in a database. See the following code for illustration:

Sql sql = Sql.newInstance("jdbc:sqlite:/home/ron/Desktop/test.db", "org.sqlite.JDBC") 
sql.execute("create table dummyTable(number)")
Long startTime = System.currentTimeMillis()
100.times {
     sql.execute("insert into dummyTable(number) values(${it})")
}
Long endTime = System.currentTimeMillis()

println "Time taken: " + ((endTime - startTime)/1000)

The output of the above code comes out to be 14.313 seconds. That is to execute 100 insert statements with only a single attribute, it would take around 15 seconds. The time taken to insert records is dependent on the number of records being inserted and increases exponentially. Clearly a performance bottleneck in any application involving batch inserts and updates.

Let us consider the same code and the performance with the withBatch() closure.

Sql sql = Sql.newInstance("jdbc:sqlite:/home/ron/Desktop/test.db", "org.sqlite.JDBC")
sql.execute("create table dummyTable(number)")
Long startTime = System.currentTimeMillis()
sql.withBatch {stmt-> 
    100.times {
      stmt.addBatch("insert into dummyTable(number) values(${it})")
    }
    stmt.executeBatch()
}
Long endTime = System.currentTimeMillis()
println "Time taken: " + ((endTime - startTime)/1000)

The time taken with the above code comes out to be 0.103 seconds! A remarkable performance improvement over the conventional method of inserting records using the execute() method.

The only drawback with using the withBatch() closure is that it does not allow prepared statements to be added to the batch. This limits the use of batch statements as we have to manually create insert or update statements.

Cheers
Roni C Thomas
roni[at]intelligrape[dot]com
@ronicthomas

Posted in Database, Grails, Groovy

How to apply liquibase change set conditionally

Posted by on May 18th, 2011

In our recent grails project, we were about to move to production so we decided to manage the database changes with liquibase. So we generated the changelog from development env by executing the command “grails generate-changelog /home/uday/projects/projectName/grails-app/migrations/changelog.xml” which contained all the table creation changesets. We updated the dbCreate=’none’ in all env. When we ran the application in test env, having the database in create-drop mode, it worked, but in other envs, which had database in “update” mode, it started failing. Now we wanted some hook to execute the creation changeset only if the tables did not exist.
We just went through the liquibase site and we got the answer to our problem. The solution was in liquibase PRECONDITIONS. We added the preconditions to the changesets to check whether the table existed or not.

<changeSet author="uday (generated)" id="1305002274717-1">
    <preConditions onFail="MARK_RAN">
      <not>
        <tableExists tableName="account"/>
      </not>
    </preConditions>
    <createTable tableName="account">
      <column autoIncrement="true" name="id" type="BIGINT">
        <constraints nullable="false" primaryKey="true"/>
      </column>
      <column name="version" type="BIGINT">
        <constraints nullable="false"/>
      </column>
      <column name="balance" type="DECIMAL(19,2)">
        <constraints nullable="false"/>
      </column>
      <column name="date_created" type="DATETIME">
        <constraints nullable="false"/>
      </column>
      <column name="last_updated" type="DATETIME">
        <constraints nullable="false"/>
      </column>
      <column name="firstname" type="VARCHAR(255)"/>
    </createTable>
  </changeSet>

In the above example the changeset will execute only if the precondition passes, if the condition fails you can specify what needs to be done. In our case we mark it as run so that liquibase marks it as being executed. You can do other things on fail event like HALT, CONTINUE, WARN,MARK_RAN.
The documentation is quite good, you can find the examples of other changesets like constraints and indexes as well.

Hope it helps
Uday Pratap Singh
uday@intelligrape.com
https://twitter.com/meudaypratap
http://in.linkedin.com/in/meudaypratap

Posted in Database, Grails, Plugin