Programming stuff: Bash, Linux, SQL and Java

we have a web application at work, that updates a web page comet style.

it is running on tomcat.

the update is done based on some events from a backend application, and the connections are kept open for a long time, untill the event triggers the update.

we had the 1 thread per connection situation and as the traffic grew, we had to make a change.

tomcat had an event based comet model, and i couldn’t make it work.

so i switched to jetty and its continuation thing.
in jetty, the principle is that an Exception is thrown, and then the connector catches it and frees the thread, retruning it to the pool so it can be reused for other requests.

But, my application was a struts one, and i had the suspend() call inside an Action.
after many tries, all my efforts made the examples i found on the net work perfectly, but my app didn’t.
all it did was to spit the retryrequest runtime exception in the web page.
finally , the trick was that ProcessRequest class in the struts stack was throwing all exceptions that occured in the Actions like this:
throw new ServletException(exception) , and the SelectChannelConnector in the jetty stack couldn’t get its hand on its retryRequest exception.
so i changed the ProcessRequest class, so that RetryRequest exception is skipped, and all was great.
so, in order to make jetty’s continuations work with struts stack you have to be aware of this RetryRequest  runtime exception to be thrown exactly as it is in the struts stack and not wrapped in the ServletException.

Comments(1)

i set gvim as the editor in the preferences of gftp.
the problem was that, when saving and exiting gvim, the file was removed from the temp folder, but was not uploaded back on the server.
this used to work with gedit.
a search on google took me to a post where a guy had the same problem as me, but no answer.
a man vim solved it.
it says there that gvim detaches itself from the shell , so gftp won’t see any exit code when you close gvim.
all i did was putting /usr/bin/gvim –nofork in the editor field.
voila!

Comments(0)

if you want to remove class and method information line when you do logging with java.util.logging , then you have 2 choices:
1. subClass your handler and make your own publish method
or
2. subClass the formatter you pass to the handler and modify the format(LogRecord record) method.

i did the second one.
here it is:
public String format(LogRecord lr){
                           return lr.getMessage()+"\n";
                   }

Comments(0)

in order to process a text file in any sense, and being used with vim, and not with sed
i use the following for deleting googlebot lines from a web log.

vim -e -c ":d/googlebot/g" -c :wq website.YYYYMMDD.log
-e doesn’t put vim in interactive mode, so you can script it and put it in crontab.

or if you need to pipe it:

cat website.YYYYMMDD.log | vim - -e –c ":d/googlebot/g" -c :wq

maybe i will add more vim issues as comments to this post.

Comments(0)

below is a case with only 2 varchar columns and an int
create table example(
id int unsigned auto_increment not null,
column1 varchar(200),
column2 varchar(200),
column3 int unsigned,
primary key(id)
);

mysql -u USERNAME -pPASSWORD DBNAME -e "insert into example (column1,column2,column3) values (char(FLOOR(65 + (RAND() * 20)),FLOOR(65 + (RAND() * 20)),FLOOR(65 + (RAND() * 20)),FLOOR(65 + (RAND() * 20)),FLOOR(65 + (RAND() * 20)),FLOOR(65 + (RAND() * 20))),char(FLOOR(65 + (RAND() * 20)),FLOOR(65 + (RAND() * 20)),FLOOR(65 + (RAND() * 20)),FLOOR(65 + (RAND() * 20)),FLOOR(65 + (RAND() * 20)),FLOOR(65 + (RAND() * 20))),FLOOR(1000 + (RAND() * 1000)))"

it will generate uppercase strings and integers between 1000 and 2000 ,and insert them into the table.
it is important to put unsigned, thus you will increase the range for the primary key. you don’t need any negative values for id anyway.

Comments(0)

….not anymore, if you use IF.

there are cases where you can get rid of OR and get better performance in mysql.
here is a case a table:
book
id
french_name
engl_name

the books have only one name and only one, the other being null
you have a form on your web page and want to do a search for the books that contain a certain substring of characters. the client wants to retrieve both english and french name that contain that substring.

first shot: select * from book where french_name like ‘%CHARS%’ or engl_name like ‘%CHARS%’

this is very expensive.
the better solution:
select * from book where if(french_name is null,engl_name,french_name) like ‘%CHARS%’

on my system is 2 times faster.

Comments(1)

Well, today I spent 2 hours on a query that involved group_concat.
we had a table that looked like this:
Columns:
……….
client_name
phone
mobile
email
……….
and in a complex query we needed to have group_concat(client_name),  group_concat(phone), group_concat(mobile), group_concat(email) …
[note: actually we did : cast(group_concat(client_name) as char),  cast(group_concat(phone) as char), cast(group_concat(mobile) as char), cast(group_concat(email)  as char)… ]
the thing is that phone and mobile were missing for most of the clients, and the value was NULL.
now if you have for phone values:
NULL
123456789
NULL
NULL
then group_concat(phone) is 123456789
and i woud get:
 clients                                             phone
john,george,alex,vio                123456789

and i wouldn’t know which client had that phone.
what i needed was somethin like:
 clients                                             phone
john,george,alex,vio                _,123456789,_,_
and i would know that george had phone number 123456789.

the sql for that is:
select …. cast(group_concat(ifnull(client_name,’_')) as char),  cast(group_concat(ifnull(phone,’_')) as char), cast(group_concat(ifnull(mobile,’_')) as char) … from clients_table …

Comments(0)

some of my clients were having trouble sending large attachements throush my postfix server.
It went like this: when the file attached was bigger than 1 MB, the receiver got the mail around 8 times.
First i thought it is the person who sent it, being tired of waiting [they were on a slow connection to the internet] it clicked the send button several times.
it wasn’t the case.
then i thought , maybe there is a microsoft thing that didn’t comply with the standards. No it wasn’t.
i disabled their anti-virus software, check my postfix configuration, analysed the logs. nothing.
then, i saw a setting in the outlook called "server timeout". Bingo. that did it.
it was set to 1 minute and a half, and i increased it to 8 minutes.
no duplicate sent emails any more.

another client satisfied. Next! :)

Comments(0)

Today, at work, we faced insert and delete problems in our mysql database.
usually we have bulks of inserts/updates/deletes from a backend engine, and selects from the website.
the inserts/updates/deletes bulks are usually around 100-500, but at peak times it goes up to 40 000.
the application that does it is a java application and it does it with PreparedStatement and batch.
When we first did this, we thought it would be enough, and no further optimisation could occur.
To my surprise, there is more to it. instead of 5000 "insert into table values (?,?,?,?,?)", this proves to be 5 times faster, in our case:
"insert into table values(?,?,?,?,?),(?,?,?,?,?),(?,?,?,?,?),….5000 times..,(?,?,?,?,?)".
there is some time wasted in building the string, but it is much better this way, as the load on our servers is on the mysql engine, the rest of the applications are low.
you can find the source code of a java class that tests the difference between the 2 approaches, so you can test it at home.
the times are: ~ 1235 ms for the thousand of inserts in batch and 211 ms for the one big insert with 5000 values() pairs.
the stringbuilding of the sql takes ~ 300 ms which is good for our purposes.

1. note: for deleting, always use delete from table where id IN (?,?,?…..?)

2. for the source code file, i did copy from eclipse and paste in vim on the web-server. the code got autoindenting and from 1 comment it commented all the way down. here is the trick that saves you: :set paste before pasting and :set nopaste after.

Comments(0)