README: $Id: README,v 1.5 2000/01/11 22:49:28 danielr Exp $

	Please read this entire file before starting. I haven't had
time to go back an reorganize it to flow better so something important
maybe mentioned in an odd place at this point.

################################################################
# Why this tool exists
################################################################

        Not being happy with the existing REQ to RT converters out
there, I ended up creating yet another one to convert our queue of apx
19750 requests that had been generated over the past 5 years. My
motivation for this was:

Handling of merged requests
        Each request is created individually and then the RT merge
	action is used to combine them.

Date preservation
        Each request is imported with it's original date and followup
        eMail dates are preserved. The action times are not preserved
        though.

User importing
        As much as possible the user performing the action is
	transfered into RT. So, if someone owned a request under REQ,
	they would also own that request under RT. Requires making RT
	users.

Action importing
        As much as possible the actions performed on a request are
        imported as individual transactions in RT. In some cases REQ let
        you do actions on a request that RT doesn't. The program tries to
        do the right thing and translate the REQ action into the
        equivalent set of RT actions.

Fixing buggy requests
        Over time some requests in our REQ queue got munged in various
        ways. As part of importing the tool makes sure all requests are as
        clean as possible before importing them into RT.

################################################################
# How to use this tool?
################################################################

1) Using this tool isn't as simple as using the other REQ to RT
converters, but if you spend the time to go through the follows steps
you will get an RT queue that includes proper merging, individual RT
transactions for each corresponding RT transaction, correct REQ user
to RT user mapping, and learning more than you wanted to about the
quirks that REQ can introduce into REQ files.

2) Run the program in a diagnostic mode so that it generates what
users need to be created or mapped for proper tracking when importing
and generating errors for problem REQs.

3) Optionally, but highly recommended, repeat step 2 until you get a
clean run.

4) Import the queue into RT

################################################################
# Basic configuration
################################################################

Have a quick look through the yar2r.pl file and make the needed
changes for your site. In particular you will need to change:

	# Change this to where you have RT's lib directory
	push( @INC, "/share/adm/packages/rt-1.0.1/lib" );

	# Change this to the location of RT's config.pm
	require "/share/adm/packages/rt-1.0.1/etc/config.pm";

	# Domain to append to username that don't include an @
	my( $DOMAIN ) = "example.edu";

	# What Queue to import REQs into
	my( $QUEUE ) = "systems";

	# Default RT user for bad username (see below)
	my( $RTUSER ) = "root";

	# Set this priorities that might be used. Default will be used
	# if a found priority doesn't match one of the other values.
	my( %PRIORITIES ) = ( "low", 25,
			      "normal", 50,
			      "high", 75,
			      "default", 50 );

	# Where your queues live
	my( @directory ) = ( "/net/request/resolved",
			     "/net/request/active" );

################################################################
# Step 2
################################################################

You should run step 2 with the following settings:

	my( $USER_WARN ) = 1;
	my( $COMMIT ) = 0;
	my( $VERBOSE ) = 1;
	my( $DEBUG ) = 0;

I would also recommend piping the output to a file as it will be
generating error messages that you will need to look at.

################################################################
# Step 2 - Part 1 - Users
################################################################

If you are _NOT_ interested in mapping REQ users to RT users then set:
	my( $USER_WARN ) = 0;
Make sure $RTUSER is a valid RT user and ignore the rest of this section.

If you are interested in keeping track of what user has done what
transactions on a REQ when you import it into RT, you need to make the
script aware of what valid users are and what their usernames are. In
order to do this you will need to run yar2r with the following
settings (done by editing the script):

	my( $USER_WARN ) = 1;

	This will print out error lines like this:

***** ERROR: "danielr" doesn't exist

What this means is that if you want the user danielr to have actions
properly attributed to them (or a corrected username), you will need
to make sure that the final username has modify ability in the import
queue and that yar2r knows how to handle that user.

In some cases you will get a warning about a user that is an alternate
form of a valid username such as:

***** ERROR: "JeffC" doesn't exist

Instead of having to create a new account for this odd user you can
just map them to a valid username. This is all done through the %USERS
hash. In it you provide mapping for both valid user to username
mappings and odd user to valid username mappings. For example a
segment of my %USERS hash:

my( %USERS ) = ( "Dave", "dave",
  	         "Jay", "jay",
		 "JeffC", "jeffc",
		 "danielr", "danielr",
		 "dave", "dave",
		 "jay", "jay",
		 "jeffc", "jeffc" );

This maps to 4 valid RT users (danielr, dave, jay, and jeffc), however
during parsing the other listed odd users where found. When those odd
user entries are found in the future they will now instead be mapped
to the valid RT usernames.
	
At some point there will be user entries that you don't want to map to
a valid RT username or don't really care about. At this point set
$USER_WARN = 0 and any user entry that isn't mapped will automatically
be mapped to the value of $RTUSER, which must also be a valid RT
username.

################################################################
# Step 2 - Part 1 - Corrupted REQ files
################################################################

In order to correctly import a REQ file into RT it needs to be able to
properly parse it. However, I've noticed that occasionally a REQ file
will get corrupted for one reason or another, so before you can import
that REQ you need to fix it. This is a mini HOWTO on what errors you
are likely to see and how to fix them.

Error: ***** ERROR: Unable to find user in "Subject changed to " Re: [Req #16982] can't get ssh to work on Mac for Eudora or NS         Communicator".
Error: ***** ERROR: Unable to find subject in "Subject changed to " Re: [Req #16982] can't get ssh to work on Mac for Eudora or NS         Communicator".
Reason: This usually means that a newline somehow got put into the
	subject line so it has got broken up into 2 separate lines in
	the REQ file.
Fix: Remove the newline.

Error: ***** ERROR: No known action "notified by user.".
Reason: This means for some reason that the user being modified is
	blank.
Fix: Add in a user that makes sense.

Error: ***** ERROR: REQ number or date not found.
Reason: A merge that got munged.
Fix: Good luck. Start at the top work your way down trying to figure
	out where numbers got switched around.

I'm sure I'm forgetting other possible errors, but all have helpful
error message and usually easy fixes. The 'REQ number ...' errors are
by far the hardest to deal with, hopefully you won't get any.

I found it helpful to plug in these couple of lines before the
@directory definition when I was trying to fix a problem REQ:

	$DEBUG = 1;
	&parseREQ( "/net/request/active/19023" );
	exit;

Basically this will help pinpoint the section of the REQ that caused
it to bail.

################################################################
# Step 3
################################################################

Once you have gone through and fixed as many errors as possible, try running it again just to make sure. The program will exit() if it gets an error when importing into RT, which is why making sure no errors are generated is crucial before doing the real import.

################################################################
# Step 4
################################################################

Cool, this means you have a REQ queue that now parses correctly
without any errors and you have optionally setup user mapping. Now to
actually import things into RT.

Note 1: Make sure the queue you are importing to has all notification
features turned off. Otherwise this process will generate TONS of
eMail.

Note 2: Even with the above setting, I needed to make the following
change in order to prevent any eMail from getting sent. I strongly
recommend you do the same:

*** lib/rt/support/mail.pm.orig Tue Nov 30 14:12:31 1999
--- lib/rt/support/mail.pm      Thu Dec 16 14:47:22 1999
***************
*** 100,105 ****
--- 100,106 ----
      #remove leading space
      $in_subject =~ s/^(\s*)//;
  
+     return( "done" );
      open (MAIL, "|$rt::mailprog $rt::mail_options");
      
      print  MAIL "Subject: [$rt::rtname \#". $in_serial_num . "] ($in_queue_id) $in_subject

Note 3: You might need to set root's ACLs in RT for the queue you are
importing into. If you try an import and it reports root can't edit
the queue, that's the most likely cause.

You should have the following settings in the script:

	my( $USER_WARN ) = 0;
	my( $COMMIT ) = 1;
	my( $VERBOSE ) = 1;
	my( $DEBUG ) = 0;

You should now just be able to start the program and watch the REQs
get imported into RT. This will take a long time if you have a lot of
REQs. I suspect that the DB code isn't that optimized as REQ importing
will get slower the more populate the RT database get (this is only a
speculation, not backed up by anything besides my unscientific
observations at this point).

Some problems I ran into at this point:

1) I had some kind of memory leak happen when importing into RT. I
haven't looked to figure out where the leak is occurring, but I ended
up having to import by queue in lots of 5000 so that the leak wouldn't
use up all of the available memory on the machine.

2) RT will silently fail if their isn't enough disk space to write
the body of a transaction to the disk. Make sure you have plenty of
disk space for storing the transaction directory. In general you
should have as much disk space as you current REQ resolved and active
queues are taking up plus each for all of the directory structure
information RT will create (think 365 directory entries per year).

3) If yar2r should produce and ERROR during import it will halt
completely. Unfortunately, this will usually mean you will need to
start the process again from the beginning, unless you are willing to
have the REQ # that is bailed on be in an undefined state in the
queue. If that's the case, just modify the source to skip all REQs
below that number and continue importing after the REQ it bailed on.

If you do want to get a complete clean import, after it fails you will
need to delete the RT transactions directory and remove all rows in
the 'each_req' and 'transactions' table in the MySQL RT DB.

4) If you didn't run your import as the RT user, after the import has
finished, you will need to change the user and group of all of the
files under the transactions directory. Something like: "chown -R
rt.rt $RT_HOME/transactions/" should do the trick.

################################################################
# Thanks
################################################################

For those that have helped with the development and improvement of YaR2R.

Marion Hakanson <hakanson@cse.ogi.edu> - various patches
Jon Stearley <jstear@cs.unm.edu> - documentation suggestions

################################################################
# Other Stuff
################################################################

At this point I'm not sure if this code will work for anyone else and
if it does, how well it would. I mainly put it together for importing
our importing our REQ queue with the added benefit of keeping more
information about a REQ when it was imported versus what the other
existing tools where doing.

I'd still like to hear feedback if other people are using it and how
it is working for them.

- Daniel R. <danielr@ccs.neu.edu> [http://www.ccs.neu.edu/home/danielr/]

