Film-Tech Forum ARCHIVE: Perl CGI and the database decision

This is topic Perl CGI and the database decision in forum Film-Yak at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=8;t=005094

Posted by Dominic Espinosa (Member # 2122) on 05-10-2007, 01:04 PM:

I wonder if I might bounce these ideas of mine off some of you techn-savvy techers [Wink]

In short I'm coding an online catalog in PERL from the ground up for controls sake. That's why I'm not using an existing catalog/cart system.

Anyhow, there are 3 ways to handle the database (that I can think of) and I'm having trouble deciding which to go with.
The first and simplest would be a flat-file with a row for each item in CSV format.
Bearing in mind of course that multiple instances of the same script would be potentially accessing this file at once.
This format also adds a few lines of code for the feature of sub-categories, etc.

The second would be to break down the catalog into directories with a text file for each item. This would allow infinite sub categories (since the same routine is called recursively to load the sub-directories and data files into lists) and probably the most flexibility.
The downside here is that you have multiple instances of the script reading entire directories constantly and I'm worried all the open/read/close calls would tie up system resources more than necessary.

The ideal solution would be an SQL database such as mysql. Most web hosts support this system and it's easy enough to get up and running and the SQL server handles all the read requests and keeps the dbase in memory for the most part. Only one process reading files, less overall system workload.
Again though in the interest of flexibility and creating a self contained system this introduces another set of problems all it's own including table limits, etc.

Any ideas?

Posted by John Hawkinson (Member # 1135) on 05-10-2007, 08:29 PM:

In general, it's best not to roll-your-own. Make use of existing modules and packages to do most of the work for you. Otherwise you'll find yourself making many errors that other people have already made and fixed.

So of your choices, using an SQL database is reasonable. Or something like SQLite.

I have to say, I wonder if you're biting off more than you can chew. When you say, "In short I'm coding an online catalog in PERL from the ground up for controls sake. That's why I'm not using an existing catalog/cart system." that's not entirely clear.

Do you mean you need more control than any existing system can give you? You are probably better off modifying something else than writing it from the ground-up...

--jhawk

Posted by Daryl C. W. O'Shea (Member # 1303) on 05-11-2007, 03:11 AM:

If you're set on using Perl, use DBI; and then use whatever DBD you want or is available on whatever system you end up on. Don't try to manage a flat file or filesystem based system yourself, you won't get it right unless it's all read only. If it's for a real busy site you won't get I/O under control in your own code.

Here's the part Adam's going to love (from a guy who writes thousands of lines of Perl code every week):

Use PHP. Unless you only have access to PHP4, then use Perl. PHP5 has taken care of most of the major security issues that previous versions had. If user entry is going to be at a minimum (more of a click and browse thing) then developing it in PHP will be faster and easier to maintain. I'm guessing that since you typed Perl wrong, it's Perl or perl, not PERL, you're not too attached to it... so switching to PHP shouldn't hurt too much.

Of course, if you're going to be doing a lot of text processing then Perl is the only sane choice. For added fun you could use both.

Posted by Scott Jentsch (Member # 1681) on 05-11-2007, 10:19 AM:

I concur on using a database system over the flat file systems you mentioned. Way back in the mid 90's, we used quasi-databases with hashed files (sorry if I'm mis-speaking this, many cobwebs), but nothing replaces the functionality provided by a database server.

Perl vs. PHP is a preference thing. I have found that people that have cut their teeth on Perl have a very hard time transitioning to PHP, but I took the plunge on PHP and haven't looked back. I dread every time I have to go back into some of the remaining Perl scripts I have to make changes.

PHP and MySQL can handle millions of transactions per day without breaking a sweat, and that's on relatively modest hardware. Well-designed applications are very possible, and unless you know Perl inside and out, I have found PHP to be much easier to maintain due to the many functions and PEAR libraries available.

I roll my own code whenever possible, but then, I'm a bit of a control freak about stuff like that, so there has to be overwhelming advantages to using pre-canned applications. The flip-side is you become the system administrator of the application, and you need to have the time and resources to do the job right. If that describes you, you're all set.

By all means, if your application needs a database, use a database server to do those tasks.

Posted by Daryl C. W. O'Shea (Member # 1303) on 05-11-2007, 12:49 PM:

quote: Scott Jentsch
Perl vs. PHP is a preference thing. I have found that people that have cut their teeth on Perl have a very hard time transitioning to PHP

If you know Perl, especially if you know Perl "inside-out", PHP should be trivial for you. You'll probably find that it's not hard in a "I don't get it" sense, rather it's hard to give up the power of Perl to do things in a faster/easier/simpler way. For simply chucking stuff from a database at a user, though, PHP wins since you're not going to be doing much text processing anyway.

quote: Scott Jentsch
I have found PHP to be much easier to maintain due to the many functions and PEAR libraries available.

You've heard of CPAN, the inspiration for PEAR, right? I've found PEAR extremely lacking in comparison to Perl's CPAN.

Posted by Edwin Sheldon (Member # 4001) on 05-15-2007, 08:38 AM:

If you're just chucking things at a database, why use either one? I don't understand why people cling to perl's horrible syntax when Python or Ruby can do the same thing in a simpler, more legible syntax. There are even pre-built libraries to do this sort of thing. </soapbox>

Posted by Dominic Espinosa (Member # 2122) on 05-15-2007, 11:45 AM:

The reason I'm using "Perl" [Wink]

is because I'm used to it, it's been around a good long while and is relatively cross-platform compatible.
I prefer to roll my own libraries to ensure the whole thing is a.) self contained and b.) doesn't carry as much code-bloat at run-time.

I think I'm going to suck it up and go the database route, I guess it's time to fix my Linux box and set up a test-server on it [Wink]

Thanks for the 411 guys.

Posted by Kyle McEachern (Member # 2250) on 05-18-2007, 01:16 PM:

I'll agree with Daryl:

PHP 5 is the easiest solution, especially with the introduction of PDO with PHP 5, it's the PHP Group's implementation (for all intents and purposes) of Perl DBI, but when plugged into the PHP framework, makes life very easy to use.

Don't worry about cross-platform functionality really in the debate between the two languages, I don't really think either one has a clear advantage over the other. If either one is installed right, they'll work where you install it.

If you have to go with Perl, then DBI is the solution, combined with some form of SQL to go along with it, as managing a database based off of CSV/text file/etc. is just ASKING for trouble...the SQL servers (MySQL is my preference for stuff that's not super high volume, Postgres once you get into thousands of transactions a minute and tables pushing close to a billion rows) have daemons that you have to go through for a reason, cause they know how to handle these things! (If you want to *really* get it right, then use MySQL + InnoDB table structure in order to get ACID Compliance [translation: transaction-based queries so you can do a set of queries and they all run at once without getting messed up by another person using the database])

And yeah CPAN > PEAR in just about every way once you've seen how it works, but I don't think any language can much compare to http://www.php.net/ in terms of documentation of all the language's core functionality and what you can do with it.

The problem with Ruby (especially Ruby on Rails) is that it's not a scalable solution, at ALL. It's cool for the people who are all into it as the new hip thing, but at its current developmental stage (and it's EXTREMELY developmental still from all I've seen), you can't use it at any real high volume, and it doesn't have near the functionality that PHP/Perl do. ActiveRecords doesn't make a language, usability, scalability, documentation, ease of use, and ease of modularity do. Python + Django is more refined than Ruby on Rails, but it's still going to fail miserably on a scaling test vs. (especially Perl) or PHP. Same goes for using lighttpd instead of apache, for that matter. The new cool 'light' frameworks are all cool, but they aren't as developed for serious code use as the others are (twitter.com is an example of a Rails app that doesn't scale at all, they've had major issues with their site since they got popular).

Posted by Adam Martin (Member # 641) on 05-18-2007, 07:17 PM:

Jesus Christ, Kyle, you turned into a dork!

Posted by Daryl C. W. O'Shea (Member # 1303) on 05-19-2007, 01:45 AM:

Probably about the same time you did, oh say, years ago.

SDBM, or DBM if you don't have SDBM, isn't too shabby for a non-SQL backend either. I know of many sites doing billions of queries and updates a day, on million+ row tables, with SDBM. Just be sure to use flock for locking if you don't have to worry about NFS.