tedsedawkandvim

dedicated to sed, awk and vim

AWK – Generate Insert Statements

As I have mentioned before a good reason to know sed, awk and vim is that you may run into a situation where you have no access to other power tools to accomplish a job. Below is an awk script which will generate an insert statement for each line of a delimited input file. You may then use the statements to run on any database so that you can import data. Take note of the example command-line and be sure you make your call appropriate to whether you are on Linux of Windows. Originally the below was written for when I was working on Windows and I had to modify it so that it works properly for Linux. If you are running on Windows you will need to use the escape backslash “\” to escape the parentheses (that appear in the generated input statement) and also the quote and then just use the Windows example statement. If you are on Linux the below should be good to go … just make sure you use the Linux example command-line.

# Program Name: awkIns
# Purpose: Generate INSERT statement for delimited input file
# Created By: Ted Colman ( tedsedawkandvim.com )
# Date: 05/19/2017

# To use: 
# In Section 1:
#  modify BEGIN clause as indicated in below comment
#  modify array variable "t" as directed in below comment
# In the example command-line change
# testTblInput.txt and test_Results_output.txt according 
# to your input / output files
# Remember to place this awk script and input / output script
# in the same directory and then run script!
# example command-line (Windows): 
# awk -F\t -f awkIns testTblInput.txt > test_Results_output.txt
# example command-line (Linux): 
# awk -F "\t" -f awkIns testTblInput.txt > test_Results_output.txt 

BEGIN{

################## Section 1

#  Modify INSERT clause between the below dotted lines to correspond 
#  to your table
#  ..................................................................
x="INSERT compare.testTbl (rowID, desc, typeID, desc2) \nVALUES (";
#  ..................................................................

#  Modify below so that fields requiring quotes have a quote
#  Note that fields 2 and 4 are varchar fields and require a quote
t[2]="'";
t[4]="'";
}

################## Section 2
# Note: Nothing to change in this section

# initialize variable y to blank at beginning of each new line
{y=""}

# build VALUES clause
{for (j=1; j<=NF; ++j)
{if (j != NF) 
  y=y" "t[j]$j t[j]","
else y=y" "t[j]$j t[j];} 
}

# print entire INSERT statement 
{print x y " );\n\n"}

Execute sed / awk script in VIM

Recently I had a need to use an AWK script within VIM. It was going to be the easiest way to accomplish the task at hand – just a simple one-liner awk statement. I got pretty frusterated as I had not done this in awhile and spent a few minutes too long looking for how to do this. So for future reference here it is:

{range}!{filter}

For example: From VIM command mode (this will print the last field of each line)

:%!awk '{ print $NF }'

An important note … Remember when using sed / awk and vim to always keep your own cheaat sheet handy. Re-using a command you have done before saves an incredible amount of time and frustration. It’s a big reason I have started this site!

Cheat Sheet Added

I have updated this site with a menu item “Cheat Sheet.”  It’s not very pretty but it has quite a bit of sed awk and vim reg ex commands.  They are very useful as a starting point or template (of sorts) which can be then modified to do the specific text edits you are looking for.

Method to Writing Regular Expressions

1. Knowing what it is you want to match and how it might appear in the text
2. Writing a pattern to describe what you want to match
3. Testing the pattern to see what it matches

The above approach to writing regular expressions is taken from the book “sed & awk (2nd Edition)” by Dale Dougherty and Arnold Robbins.  I have found that it is especially important to visit the basics when it comes to writing regular expressions and with sed, awk and vim in general.  Complex expressions are best crafted by starting at the most granular level possible and then added onto until you have something that performs as expected.  When approaching problems to solve keep this method in mind to get the results you expect.

Format List for SQL WHERE clause

This VIM Ex command is very helpful in database programming when you need to take a very large list of items and put each item in quotes and separate by a comma so that the list can simply be cut and pasted into your SQL WHERE clause.

Here is the regex:

%s/[ \t]*// | %s/[ \t]*$// | %s/.*/'\0',/g
--trim beg-----trim end------put into single quotes with comma at end

And here is why this is very useful

A list which is cut and pasted from another source may be in a form like the below …

  apples
   oranges
     bananas
  pears

*Note: If you copied a list from another source you probably have whitespace before and after your word on each line (as above) and this command will strip that out so that each word will line up in the first column.

…and you need to quickly put this list into a form that can be used in a SQL statement WHERE clause such as:

WHERE fruit in (
‘apples’,
‘oranges’,
‘bananas’,
‘pears’
)

*Note: Your last line will actually have the quotes and comma (i.e. ‘pears’,) but it is a quick and simple manual touch up required to finalize your WHERE clause.