It is common to import CSV files into MySQL database. You can do this with phpMyAdmin with small CSV files but with large ones, you would probably encounter the memory error and had to switch to MySQL command line “LOAD DATA LOCAL INFILE” to do the job.

It looks like something like this:

LOAD DATA LOCAL INFILE 'your.csv'
INTO TABLE `your_table`
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(field1, field2, field3)

And then you encounter another problem that it only imports the first 1 or 2 rows and then it stops. After some researching and trying, I was sure it’s something to do with the “LINES TERMINATED BY” directive. Depending on the platform that the CSV file is created on, the line delimiter may be

1. \n
2. \r\n
3. \r

And you need to be correct on the line delimiter to properly parse the CSV file.

So the solution is to try them all one by one and see which one of them works. Chances are, one of them would make the whole command successfully import ALL rows.

Another simple approach is to deprive the whole command of the “LINES TERMINATED BY” directive and let MySQL do the call. It’ll probably detect things right but in my case, this doesn’t work but specifying ‘\r’.

Related Posts:

• MySQL: Export Table to CSV Text Files for Excel
• MySQL log files are taking a lot of disk space – How to disable and delete them?
• SQL: Randomly Shuffle Rows or Records – Reorder them in a random order
• MySQL: How to export a database / table to XML?
• MySQL: Update Multiple Rows or Records with One Single Query

I’m not an expert but I’ve had my own errors and trials along the way so here I am. These are all the tips I believe you should use when you are optimizing for MySQL database performance. Some stuff may be very handy, but they may not be good for performance.

Design

1. Normalize first, and de-normalize where appropriate. Table joins are never lags in performance. Use them!
2. Choose the appropriate collation / charset. UTF16 is omnipotent but it requires 2 times the storage. UTF8 delivers more characters but it’s slower than latin1. Unless it’s necessary, use latin1.
3. utf8_general_ci is slightly faster than utf8_unicode_ci, thus better in performance.
4. Use NOT NULL wherever possible.
5. Use indexes. Create indexes by analyzing the SELECTs you are doing on the table.
6. Indexing increases performance in selecting, yet decreases performance in inserting. So don’t index everything but only those necessary. And don’t make duplicate indexes.

Querying

1. Use smallest data types possible.
2. IN (…) is a total fail. Avoid it wherever possible.
3. ORDER BY RAND() is a fail too.
4. When you need to SELECT … to make sure there’s no duplicate unique index values before inserting something, use INSERT … ON DUPLICATE KEY UPDATE … instead.
5. If your app writes a lot, use InnoDB; if it reads a lot, use MyISAM. Here’s a comparison.
6. Never use LIMIT xxxx1, xxxx2 with large tables especially when offset xxxx1 is a pretty large number. Instead, give a more strict WHERE … clause with LIMIT xxxx2.
7. If a feature is deprecated, stop using it.
8. Avoid wildcards % or _ at the beginning of LIKE queries, e.g. LIKE ‘%bloom’ – this should be avoided. For instance, you should store email addresses reversed (REVERSED()) so it’s faster to search by domain.
9. If you have a large set of data rows to be inserted at once, create a combined INSERT query string in PHP first – that is, to bulk insert them in batches rather than one by one. LOAD DATA is also a better option than individual INSERT.
10. Prefer GROUP BY over DISTINCT.
11. Frequently visit Slow Query Log.
12. Avoid calculated values in WHERE … , LIMIT … , ORDER BY … , etc.
13. Use EXPLAIN … to get an idea of how MySQL is doing with your queries thus better optimize it. For example, it would suggest the indexes you should have created.
14. Don’t SELECT * … on everything, SELECT only what you need.
15. Be wary of lots of small queries that may be triggered in loops. Instead, combine them and use one large query wherever possible.

Storage Engines

1. Know the pros and cons of different storage engines and use them accordingly. But try them out in the test environment before making the decision.
2. Archive old data such as logs in ARCHIVE tables or MERGE tables.
3. BLACKHOLE engine is very fast for busy things like logs.

my.ini

1. Increase the default key buffer size: key_buffer = 128M
2. Increase the default number of tables that can be kept open: table_cache = 128
3. Increase the default sort buffer size:
   sort_buffer_size = 32M
   myisam_sort_buffer_size = 32M
4. Disable binary logging that is for data replication by commenting out the line like this:  # log-bin=mysql-bin

What to read from here:

1. Indexing: http://dev.mysql.com/doc/refman/5.0/en/mysql-indexes.html
2. “LIMIT” Optimization: http://dev.mysql.com/doc/refman/5.0/en/limit-optimization.html
3. “ORDER BY” Optimizationhttp://dev.mysql.com/doc/refman/5.0/en/order-by-optimization.html
4. Efficient Paging: http://www.slideshare.net/suratbhati/efficient-pagination-using-mysql-6187107
5. Configuration Tweaks: http://docs.cellblue.nl/2007/03/17/easy-mysql-performance-tweaks/

Related Posts:

• MySQL: Huge Table Not Responding after Adding Index
• MySQL: id BETWEEN start AND end Instead of LIMIT start, step For Better Database Performance
• MySQL: Get the exact size of a database by SQL query
• MySQL: Insert if doesn’t exist otherwise update the existing row
• MySQL: INSERT INTO … SELECT … FROM … Selectively on Table Columns / Fields to Combine Multiple Tables into One

phpMyAdmin doesn’t show the exact size in MB when the size of your database exceeds 1GB. It just shows something like 4.2GB, truncating everything out in the 100MB precision. So is it possible to get the database size in MB in exact numbers using MySQL query?

Yes, it is:

SELECT CONCAT(sum(ROUND(((DATA_LENGTH + INDEX_LENGTH - DATA_FREE) / 1024 / 1024),2))," MB") AS Size FROM INFORMATION_SCHEMA.TABLES where TABLE_SCHEMA like 'YOUR_DATABASE_NAME_HERE%' ;

Just change YOUR_DATABASE_NAME_HERE into the name of your database. And it would prints out something like this:

+------------+
| Size       |
+------------+
| 4906.79 MB |
+------------+
1 row in set (0.05 sec)

You can also get the size of a specific table. Read here: http://www.novell.com/communities/node/8706/check-mysql-database-size-using-sql-query

Related Posts:

• MySQL: Check if a table exists
• MySQL: Select and Show all MySQL Users
• MySQL, PHP: Store form textarea value or data to MySQL database table
• MySQL, PHP: Display MySQL table fields and data
• MySQL: How to export a database / table to XML?

ASCII is the most fundamental character set that has been around since the early days of command line interfaces. There are 128 (0 – 127) most basic ASCII characters such as a-z, A-Z, 0-9, and all the printable punctuations you can type out by a single strike of your keyboard.

As all ASCII characters have an internal decimal value of 0 – 127, which is 0×00 – 0x0F in heximal values, you can find all the non-ASCII characters in my_column by query:

SELECT * FROM my_table WHERE NOT HEX(my_column) REGEXP '^([0-7][0-9A-F])*$';

On the other hand, if you wish to find all records that a certain column (my_column) contains ASCII characters:

SELECT * FROM my_table WHERE HEX(my_column) REGEXP '^([0-7][0-9A-F])*$';

Related Posts:

• MySQL: Deleting Duplicate Rows or Records | Selecting Unique Rows into A New Table
• MySQL: Find Duplicate Entry / Rows / Records
• SQL: INSERT INTO … SELECT … to generate or combine records from existing ones and insert into an existing table
• MySQL: Update Multiple Rows or Records with One Single Query
• SQL: CREATE TABLE … SELECT … to generate new table from existing tables

You can export any MySQL database or table into an XML file by the exporting capabilities of phpMyAdmin, the web interface of MySQL. The problem with this approach, however, is that with large tables, you may have to manually export the table more than once by sections into several sequential XML files.

A better approach is by native MySQL command line that will create and store all entries of the table into one XML file:

mysql -u db_user -p db_name --xml -e "SELECT * FROM table_name" > table_name.xml

Wherein db_user is your MySQL server user, db_name is the name of the database and table_name is the name of the table that you would like exported to XML. The resulted XML will be stored in table_name.xml.

Note that this is different from mysqldump in that it’s basically executing a query by the –e switch and output the results in XML (–xml). The results is then directed to the out file rather than displayed on terminal screen.

Related Posts:

• MySQL: How to backup ALL databases as root with mysqldump at once?
• Backup and recover a MySQL database under command line
• MySQL: id BETWEEN start AND end Instead of LIMIT start, step For Better Database Performance
• MySQL: Export Table to CSV Text Files for Excel
• How to Recover or Reset MySQL root Password after You Forgot and Lost It

MySQL tables can be exported to SQL dump file which is basically a text file of SQL queries that can be used to import the table back into database. To export MySQL tables into other formats such as CSV, phpMyAdmin proves to be very handy if you have changed the execution timeout in seconds to zero (so it never times out) – or it won’t work with large tables. However, there’s another way to do this in native SQL query and it works until the end of a very large table:

SELECT * FROM mytable INTO OUTFILE "c:/mytable.csv"
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"'
LINES TERMINATED BY "\n";

The INTO OUTFILE command in this MySQL query will store all rows selected from the table mytable into the text file c:/mytable.csv in the form:

"1","Anthony","24"
"2","Rachel","27"

Now you can open and use the CSV file in another application such as Excel and manipulate it any way you like.

If you need another format, just change the query accordingly.

Add column / field headers at the beginning of the CSV file

To add column names at the beginning of the CSV file so each field is identified, use this SQL snippet:

SELECT 'id', 'name', 'age' UNION
SELECT * FROM mytable INTO OUTFILE "c:/mytable.csv"
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"'
LINES TERMINATED BY "\n";

That is, adding “SELECT ‘id’, ‘title’, ‘slug’ UNION”. And you will have:

"id","name","age"
"1","Anthony","24"
"2","Rachel","27"

The awkward thing is you have to manually add the field names one by one. For now, I know no option other than this that can add the column names before all the records when you are dumping CSV files from MySQL, except with phpMyAdmin.

Related Posts:

• MySQL: LOAD DATA LOCAL INFILE only imports 1 or 2 rows?
• MySQL: How to export a database / table to XML?
• MySQL, PHP: Display MySQL table fields and data
• SQL: CREATE TABLE … SELECT … to generate new table from existing tables
• MySQL: Replace Substring with Another String – the MySQL String Replace Function

I have WAMP server installed on my local computer for PHP and MySQL development. After using it for a while, the MySQL installation folder has run up some seriously huge log files that are taking enormous amount of disk space. We are talking 10s of GB here. The file names are like ibdata1, mysql-bin.000017, etc.. It appears it’s because I frequently have large databases in and out by the ‘mysql’ and ‘mysqldump’ command, building up large stacks of data importing and exporting logs for data rollback and recovery or something.

Simply deleting the log files may result in problems such as being unable to start MySQL again. Tried that and I had to reinstall it to make it work again.

After some searching, I’ve found this query to solve my problem. Just execute it after you have logged in MySQL via command line – of course you will need the root privileges:

PURGE BINARY LOGS BEFORE '2036-12-12 12:12:12';

Something like that would purge and delete all the huge logs to free the disk. Just make the date time to be as sci-fi as possible so that all log files are purged.

Related Posts:

• MySQL: LOAD DATA LOCAL INFILE only imports 1 or 2 rows?
• 1 tip about ALTER TABLE xxx ADD INDEX yyy
• mysql command line character set option for importing SQL files encoded in UTF8
• PHP: Copy, Rename or Move a File
• MySQL: Export Table to CSV Text Files for Excel

It can be annoying when MySQL imports your UTF8 database (which contains exotic characters other than those in English) in the default character set of latin1_swedish_ci, jeopardizing the text content. It is also annoying when phpMyAdmin does the same and when you forgot to set the collation to utf8_general_ci for the new database which is very probably going to store utf8 characters.

Let’s fix this once and for all.

Just locate and open the MySQL configuration file my.ini and find the section [mysqld]. Add the following directive:

[mysqld]
default-character-set=utf8

Save to my.ini and restart MySQL demon. Now MySQL will use utf8 as the default character set when importing databases or creating new databases. The default collation of phpMyAdmin has also changed to utf8_general_ci.

Related Posts:

• mysql command line character set option for importing SQL files encoded in UTF8
• MySQL: LOAD DATA LOCAL INFILE only imports 1 or 2 rows?
• PHP: How to process a string one byte (character) at a time
• MySQL: Find non-ASCII characters in a table column
• MySQL: How to backup ALL databases as root with mysqldump at once?

When I was creating an installation script, I needed to check if a table exists to make sure the installation had not been performed yet. How did I do that?

I use this simple query to get whether a table exists in the specified database:

SELECT table_name
FROM information_schema.tables
WHERE table_schema = 'db_name'
AND table_name = 'table_name'

Just fill in the ‘db_name’ as well as ‘table_name’. If this query returns 1 row, the table db_name.table_name does exist, otherwise it does not.

Related Posts:

• MySQL: Get the exact size of a database by SQL query
• Using SQL to Find records existing in one table but not in another
• MySQL: INSERT INTO … SELECT … FROM … Selectively on Table Columns / Fields to Combine Multiple Tables into One
• SQL: CREATE TABLE … SELECT … to generate new table from existing tables
• MySQL, PHP: Display MySQL table fields and data

AND and OR are common logic operators across all programming languages and in MySQL, they mean the same thing. A statement (a.k.a, a condition in any WHERE clauses such as id > 100) can be either true or false. A group / combination of statements can be true or false, depending on their own values (true or false) and the logic operators that have joined them.

AND and OR each can join 2 statements.

AND – both statements must be true

For example, in this MySQL query:

SELECT * FROM student WHERE age > 10 AND grade < 4

All students who are graded less than 4 and older than 10 will be selected. The logic operator AND commands that both of the conditional statements (age > 10 and grade < 4) must be true for the whole statement (age > 10 AND grade < 4) to be true. You can add more conditions and ask them all to be true to narrow down the selection:

SELECT * FROM student WHERE age > 10 AND grade < 4 AND sex = 'female'

Which effectively selects all the female students who are older than 10 and have a grade of lower than 4 because the AND operators mean that all 3 statements must be true for the whole, joined statement (age > 10 AND grade < 4 AND sex = ‘female’) to be true.

OR – at least one statement must be true

Consider this example,

SELECT * FROM student WHERE grade > 4 OR grade < 2

This simply selects all the students who have a grade higher than 4 or lower than 2. All the students who satisfy one or more of these statements are selected. Students who satisfy none of these statements are neglected. You can as well use more than one OR in a series:

SELECT * FROM student WHERE grade > 4 OR age < 9 OR surname = 'Bush'

So that any student who has a grade higher than 4 or is less than 9 years old or has a surname of Bush will be selected.

How to use AND and OR together?

Consider the following example,

SELECT * FROM student WHERE grade > 2 AND grade < 5 OR age > 8 AND age < 12

As AND has a higher priority (effectively so in almost all programming languages) than OR, it will be evaluated before any OR statements. Therefore, this query will select all those students whose grade is higher than 2 and lower than 5 or those whose age is older than 8 and younger than 12.

You can add parenthesis but because AND has a higher priority in parsing the logic statement, it’s the same:

SELECT * FROM student WHERE (grade > 2 AND grade < 5) OR (age > 8 AND age < 12)

However, as parenthesis has the highest priority, if you add them in a different way such as this:

SELECT * FROM student WHERE (grade > 2 AND grade < 5 OR age > 8) AND age < 12

It would give completely different results. Firstly, grade > 2 AND grade < 5 must be true or age > 8 must be true, secondly, age < 12 must be true. For example, these students will satisfy this WHERE clause:

1. grade = 4, age = 7
2. age = 9
3. grade = 3, age = 11

These students don’t satisfy this WHERE clause:

1. age = 14
2. grade = 5, age = 7

Related Posts:

• PHP: Count the Number of Occurrences of Each Unique Value in an Array
• CSS: Difference between opacity:0, visibility:hidden and display:none
• jQuery: Get the text and value of the selected option of HTML select element
• A grand new redesign – MagMania wordpress theme
• SQL: CREATE TABLE … SELECT … to generate new table from existing tables

The other day I was very upset about some extraordinary down times of my unmanaged VPS box at Linode. As it’s unmanaged, support staff at Linode are not responsible for the failures. I contacted them and they told me it’s OOM (Out Of Memory), pointing me to the right documentation to figure out how to get the problem sorted out myself. After a few tweaks and observations for a week, so it seems that I have successfully optimized my VPS server to take on more traffic with less resources such as RAM.

The problem almost always lies in where the user is free to feed stuff to your website or program. Sometimes Convert Hub spikes in memory usage and forces my box to use swap that relies on disk I/O to work. This happens when someone uploads an ultra large picture to be processed or converted. While I may restrict the size of the picture that is allowed to be uploaded, I may also do the following settings to optimize the entire LAMP environment so the other websites enjoy it as well.

Apache 2 Low-Memory Optimization

Use this command to identify the MPM you are using:

apache2 -V | grep 'MPM' # for Debian-based systems

httpd -V | grep 'MPM' # for Fedora/CentOS systems

Find and change these settings in your Apache 2 configuration file (usually found at /etc/apache2/apache2.conf):

StartServers 1
MinSpareServers 3
MaxSpareServers 6
ServerLimit 24
MaxClients 24
MaxRequestsPerChild 3000

Switch to Lighttpd or Litespeed if possible.

MySQL Low-Memory Optimization

Same as above, find and change these settings of the MySQL configuration file (may be at /etc/mysql/my.cnf) accordingly:

key_buffer = 16K
max_allowed_packet = 1M
thread_stack = 64K
table_cache = 4
sort_buffer = 64K
net_buffer_length = 2K

PHP Low-Memory Optimization

Find your PHP configuration file (php.ini) and modify the PHP script memory limit to 32M or less (default is 128M):

memory_limit = 32M

Related Posts:

• PHP: Get the File Uploading Limit – Max File Size Allowed to Upload
• Check how much memory your PHP script is using. (PHP script memory usage)
• PHP: Subject String Length Limit of Regular Expression Matching Functions
• MySQL Performance Tips
• Linux: How to ‘find’ and search ONLY text files?

I’d like to share with you some tips about hardening the database part of your application. Here are a few things you can do in protecting the databases from being compromised in security:

1. Create separate users with ONLY necessary privileges (as few as possible) to connect to the database for common daily tasks. Never use the database owner / creator or even MySQL root user in your PHP scripts to perform routine tasks.
2. Protect against SQL injection attacks by escaping ALL incoming input after ensuring data types with a variety of PHP variable type and character type validation functions.
3. The sprintf() function is both useful and secure in constructing SQL queries because of the built-in type checking. Better yet, use PDO.
4. Turn off error messages MySQL or PHP outputs when things go wrong so crackers know nothing about the technical details of your build such as database schema. As a matter of fact, a good rule of thumb in web application security is that the less people know about your application’s internal structure, the better.
5. For advanced SQL developers, extra abstraction layer in SQL in the form of stored procedures can benefit security because you implement yet another depth of defense and hide the schema of the database from the outside world.
6. For mission critical applications, it goes without saying that custom logging of database accesses can help a lot in identifying security risks.
7. If the database contains very sensitive data such as credit card information, you are strongly recommended to encrypt these tables or fields. Just use PHP cryptography extensions such as Mcrypt to encrypt any data that are to be stored and decrypt them when they are being retrieved.

Related Posts:

• MySQL: Get the exact size of a database by SQL query
• Web Application Security Books (PHP, MySQL, Apache), the Best at Amazon
• The Secure Way to Store Passwords with PHP
• MySQL: Select and Show all MySQL Users
• Best MySQL Books to Learn MySQL Database Programming and Development (+ PHP Applications)

When you have a VPS or dedicated server to manage, typically you’d have a bunch of different mysql users granted the privileges of every particular database for the sake of security. While this works well in segregating the privileges and preventing hackers from gaining access to all databases upon the compromise of only one mysql user account, it means much more work when you are backing up all those databases on the server.

The solution is to use the mysqldump command as root. To backup all databases on the mysql server no matter which users they belong to, try the following via SSH:

mysqldump -u root -p --all-databases > all.sql

It’d then prompt you for root password. After successful authentication, the mysql server will start dumping all databases into a text SQL file: all.sql. After it’s done, you can compress (“tar zcf all.tar.gz all.sql”) and put all.sql somewhere safe for backup.

To restore all.sql into the entire MySQL:

mysql -u root -p < all.sql

Can’t be any simpler.

Related Posts:

• How to Recover or Reset MySQL root Password after You Forgot and Lost It
• Backup and recover a MySQL database under command line
• MySQL log files are taking a lot of disk space – How to disable and delete them?
• MySQL: Select and Show all MySQL Users
• How to check if a MySQL database connection is successful in PHP?

To show all MySQL databases, you can simply type SHOW DATABASES; or SHOW TABLES; to show all tables of the current database. However, if you want to list and display all the MySQL users, SHOW USERS; won’t work.

So how to display a list of the MySQL users?

As all user information such as user name and privilege data are stored in the table user of the database mysql that comes automatically created and populated after MySQL installation. You can show all the MySQL users of the current host by the following select query:

SELECT user FROM mysql.user;

Which would output a list of rows of all the active users, in the same format of what a SHOW DATABASES; would do.

Related Posts:

• MySQL, PHP: Display MySQL table fields and data
• MySQL: How to export a database / table to XML?
• MySQL: How to backup ALL databases as root with mysqldump at once?
• MySQL: Get the exact size of a database by SQL query
• MySQL: Export Table to CSV Text Files for Excel

After we have discussed the advantages and disadvantages of MyISAM and InnoDB, you might want to convert them back and forth and see the performance differences yourself. For small website, the difference may be minor but for a large popular application,

• MyISAM tables yield better performance results in selecting data records
• InnoDB tables give better performance results in inserting and updating data records

To change a MyISAM table to an InnoDB table:

ALTER TABLE table_name ENGINE = InnoDB;

To convert an InnoDB table to a MyISAM table:

ALTER TABLE table_name ENGINE = MyISAM;

Related Posts:

• MySQL Engines: InnoDB vs. MyISAM – A Comparison of Pros and Cons
• MySQL Performance Tips
• Using SQL to Find records existing in one table but not in another
• MySQL: id BETWEEN start AND end Instead of LIMIT start, step For Better Database Performance
• MySQL, PHP: Store form textarea value or data to MySQL database table