{"id":12212,"date":"2019-03-23T15:00:00","date_gmt":"2019-03-23T15:00:00","guid":{"rendered":"http:\/\/www.appservgrid.com\/paw92\/?p=12212"},"modified":"2019-03-23T15:00:00","modified_gmt":"2019-03-23T15:00:00","slug":"how-to-convert-files-to-utf-8-encoding-in-linux","status":"publish","type":"post","link":"https:\/\/www.appservgrid.com\/paw92\/index.php\/2019\/03\/23\/how-to-convert-files-to-utf-8-encoding-in-linux\/","title":{"rendered":"How to Convert Files to UTF-8 Encoding in Linux"},"content":{"rendered":"

In this guide, we will describe what character encoding and cover a few examples of converting files from one character encoding to another using a command line tool. Then finally, we will look at how to convert several files from any character set (charset<\/strong>) to\u00a0UTF-8<\/strong>\u00a0encoding in Linux.<\/p>\n

As you may probably have in mind already, a computer does not understand or store letters, numbers or anything else that we as humans can perceive except bits. A bit has only two possible values, that is either a\u00a00<\/code>or\u00a01<\/code>,\u00a0true<\/code>\u00a0or\u00a0false<\/code>,\u00a0yes<\/code>\u00a0or\u00a0no<\/code>. Every other thing such as letters, numbers, images must be represented in bits for a computer to process.<\/p>\n

In simple terms,\u00a0character encoding<\/strong>\u00a0is a way of informing a computer how to interpret raw zeroes and ones into actual characters, where a character is represented by set of numbers. When we type text in a file, the words and sentences we form are cooked-up from different characters, and characters are organized into a\u00a0charset<\/strong>.<\/p>\n

There are various encoding schemes out there such as\u00a0ASCII<\/strong>,\u00a0ANSI<\/strong>,\u00a0Unicode<\/strong>\u00a0among others. Below is an example of\u00a0ASCII<\/strong>\u00a0encoding.<\/p>\n

Character  bits\r\nA               01000001\r\nB               01000010<\/pre>\n
In Linux, the\u00a0iconv<\/strong>\u00a0command line tool is used to convert text from one form of encoding to another.<\/p>\n
You can check the encoding of a file using the\u00a0file<\/strong>\u00a0command, by using the\u00a0-i<\/code>\u00a0or\u00a0--mime<\/code>\u00a0flag which enables printing of mime type string as in the examples below:<\/p>\n
$ file -i Car.java\r\n$ file -i CarDriver.java\r\n<\/pre>\n
\n
\"Check<\/a><\/p>\n
Check File Encoding in Linux<\/p>\n<\/div>\n
The syntax for using\u00a0iconv<\/strong>\u00a0is as follows:<\/p>\n
$ iconv option\r\n$ iconv options -f from-encoding -t to-encoding inputfile(s) -o outputfile \r\n<\/pre>\n
Where\u00a0-f<\/code>\u00a0or\u00a0--from-code<\/code>\u00a0means input encoding and\u00a0-t<\/code>\u00a0or\u00a0--to-encoding<\/code>\u00a0specifies output encoding.<\/p>\n
To list all known coded character sets, run the command below:<\/p>\n
$ iconv -l \r\n<\/pre>\n
\n
\"List<\/a><\/p>\n
List Coded Charsets in Linux<\/p>\n<\/div>\n
Convert Files from UTF-8 to ASCII Encoding<\/h3>\n
Next, we will learn how to convert from one encoding scheme to another. The command below converts from\u00a0ISO-8859-1<\/strong>\u00a0to\u00a0UTF-8<\/strong>\u00a0encoding.<\/p>\n
Consider a file named\u00a0input.file<\/code>\u00a0which contains the characters:<\/p>\n
\ufffd \ufffd \ufffd \ufffd\r\n<\/pre>\n
Let us start by checking the encoding of the characters in the file and then view the file contents. Closely, we can convert all the characters to\u00a0ASCII<\/strong>\u00a0encoding.<\/p>\n
After running the\u00a0iconv<\/strong>\u00a0command, we then check the contents of the output file and the new encoding of the characters as below.<\/p>\n
$ file -i input.file\r\n$ cat input.file \r\n$ iconv -f ISO-8859-1 -t UTF-8\/\/TRANSLIT input.file -o out.file\r\n$ cat out.file \r\n$ file -i out.file \r\n<\/pre>\n
\n
\"Convert<\/a><\/p>\n
Convert UTF-8 to ASCII in Linux<\/p>\n<\/div>\n
Note<\/strong>: In case the string\u00a0\/\/IGNORE<\/code>\u00a0is added to to-encoding, characters that can\u2019t be converted and an error is displayed after conversion.<\/p>\n
Again, supposing the string\u00a0\/\/TRANSLIT<\/code>\u00a0is added to to-encoding as in the example above (ASCII\/\/TRANSLIT<\/strong>), characters being converted are transliterated as needed and if possible. Which implies in the event that a character can\u2019t be represented in the target character set, it can be approximated through one or more similar looking characters.<\/p>\n
Consequently, any character that can\u2019t be transliterated and is not in target character set is replaced with a question mark\u00a0(?)<\/code>\u00a0in the output.<\/p>\n
Convert Multiple Files to UTF-8 Encoding<\/h3>\n
Coming back to our main topic, to convert multiple or all files in a directory to UTF-8 encoding, you can write a small shell script called\u00a0encoding.sh<\/strong>\u00a0as follows:<\/p>\n
#!\/bin\/bash\r\n#enter input encoding here\r\nFROM_ENCODING=\"value_here\"\r\n#output encoding(UTF-8)\r\nTO_ENCODING=\"UTF-8\"\r\n#convert\r\nCONVERT=\" iconv  -f   $FROM_ENCODING  -t   $TO_ENCODING\"\r\n#loop to convert multiple files \r\nfor  file  in  *.txt; do\r\n     $CONVERT   \"$file\"   -o  \"${file%.txt}.utf8.converted\"\r\ndone\r\nexit 0\r\n<\/pre>\n
\n
Save the file, then make the script executable. Run it from the directory where your files (*.txt<\/code>) are located.<\/p>\n
$ chmod  +x  encoding.sh\r\n$ .\/encoding.sh\r\n<\/pre>\n
Important<\/strong>: You can as well use this script for general conversion of multiple files from one given encoding to another, simply play around with the values of the\u00a0FROM_ENCODING<\/code>\u00a0and\u00a0TO_ENCODING<\/code>\u00a0variable, not forgetting the output file name\u00a0\"${file%.txt}.utf8.converted\"<\/code>.<\/p>\n
For more information, look through the\u00a0iconv<\/strong>\u00a0man page.<\/p>\n
$ man iconv\r\n<\/pre>\n
To sum up this guide, understanding encoding and how to convert from one character encoding scheme to another is necessary knowledge for every computer user more so for programmers when it comes to dealing with text.<\/p>\n<\/div>\n
Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
In this guide, we will describe what character encoding and cover a few examples of converting files from one character encoding to another using a command line tool. Then finally, we will look at how to convert several files from any character set (charset) to\u00a0UTF-8\u00a0encoding in Linux. As you may probably have in mind already, … <\/p>\n
Continue reading “How to Convert Files to UTF-8 Encoding in Linux”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-12212","post","type-post","status-publish","format-standard","hentry","category-linux"],"_links":{"self":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/12212","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/comments?post=12212"}],"version-history":[{"count":1,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/12212\/revisions"}],"predecessor-version":[{"id":12213,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/12212\/revisions\/12213"}],"wp:attachment":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/media?parent=12212"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/categories?post=12212"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/tags?post=12212"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}