The Five Things Meme

You probably know the drill by now. The good Dr. Bob tagged me with this infectious Five Things Meme and now I'm supposed to tell five things about myself that you probably didn't know. Oh - the excitement! ;) If you fall asleep halfway through the list - come back tomorrow. Ok - here we go:

1. In my childhood and youth I have had part-time jobs as a paperboy, flower delivery, painter (on walls, not canvas), secretary (writing letters based on dictation tapes) and programmer.
   
2. I had my first computer experience at age 11 - my uncle had bought a Sinclair ZX-81 with 1K RAM (later upgraded to 16K). I was awed after playing chess against the machine and quickly wrote my first "game" - Guess the Number. I've been bitten by the programming bug ever since.
   
3. At age 19 I participated in a youth exchange program organized by Lions. To keep costs down I traveled with two other Norwegian youths by train (Interrail) from Oslo to Istanbul - a three-and-a-half day journey. It was very interesting to live in an "ordinary" Turkish family (albeit a relatively rich one) and to get to know other youths from throughout Europe.
   
4. I served my military service in the Navy as a radio-operator. We performed the work deep inside the nuclear bomb-safe mountain Jåttanuten outside Stavanger, Norway.
   
5. I'm an active member of a local community group called GLA'Nordstrand. We are lobbying to improve the traffic conditions and reduce the number of pass-through cars that drive our local resident roads to get away from the queues at the national roads. It is a struggle the be heard and noticed by the municipality politicians and bureaucrats. One day I was interviewed by the regional TV news station about the traffic conditions, when two cars crashed just behind me - almost hitting a child on his way to school.

Wasn't that a thrill? Now you know why I stick to technical issues related to programming and Delphi in mt blog ;-).

Eh? I'm supposed to tag some more people to keep the ball rolling? Well, are there anyone left that hasn't been tagged yet? I dunno - but I'll have a go at Jon Shemitz, Dan Miser, Chuck Jazdzewski, Lars Fosdal and Ingvar Nilsen.

The history of the CodeGear name

A while back, Michael Swindell (Borland CodeGear), posted the story about how the CodeGear name was picked to the delphi.nontech newsgroup. First it was established that Michael had recently bought the domain name codegear.com. Then someone asked:

Can you tell us how much cost you that ?

And Michael's reply was an interesting read of the story behind the name:

one BILLION dollars.... <g> heh... no, actually it was a bargain for several thousand dollars... a great deal IMO.... I was really just blown away that  it was even available for purchase. When I thought of the name, Allen [Bauer -ed] was sitting across the table and I told him, "but of course there's no way the domain will be available"... and sure enough it was. I have go back and research who the original suggestion actually came from, but the name stemmed from one of the names submitted by customers from my Blog post - DelphiGear.... we were browsing thru hundreds of submissions (most were ummm.... how shall I put it... "Intersting" is the word I'm looking for <g>) but I really liked the idea of DelphiGear. Like Climbing Gear or Photo Gear but for Delphi programmers. Of course Delphi in the name is just way too narrow a focus for our new company name, and I just swapped Delphi with Code... and wahlah... there it is :o) funny how those things work sometimes. So thanks to the Delphi developer who submitted DelphiGear as a suggestion - you are a part of history! -m

Thanks for the nice story Michael! And sorry if you intended to blog about this yourself - not intending to steal your thunder! :-)

Don't miss the Oslo Delphi Day, Dec 15th 2006!

In a joined operation between Alfacode AS (and Thomas Fjeld), CodeGear Stockholm (represented by Fredrik Haglund) and the Oslo Delphi Club, we are arranging the Oslo Delphi Day on Friday December 15th.

The key attraction is three presentations held by tech Speaker Extraordinaire, Chad Z. Hower (aka Kudzu) of Indy and IntraWeb fame. In addition there will be few sessions by Fredrik and myself. Thomas has just made the agenda available here.

If you are reading this and are within travelling distance of Oslo, make sure to sign up for the event here!

See you there!

DN4DP#3: Nesting habits

This post continues the series of The Delphi Language Chapter teasers from Jon Shemitz’ .NET 2.0 for Delphi Programmers book. Last time we learned about the new class member visibility specifiers that are available, abstract classes and final methods. This time we will look at the syntax and semantics of nested classes.

Note that I do not get any royalties from the book and I highly recommend that you get your own copy – for instance at Amazon.

"Nesting habits

The compiler now allows you to declare nested types (including classes) and constants inside another class or record declaration. The implementation of a nested class method must have its name preceded by both the outer and inner class names, like this

type
  TOuter = class
  public
    const 
      MeaningOfLife = 42;
    type
      TResult = integer;
      TInner = class
      public
        function  Method: TResult;
      end;
  end;
 
implementation
 
function TOuter.TInner.Method;
begin
  // or just MeaningOfLife 
  Result := TOuter.MeaningOfLife; 
end;
 
procedure Test;
var
  Inner: TOuter.TInner;
  Result: TOuter.TResult;
begin
  Inner := TOuter.TInner.Create;
  Result := Inner.Method;
end;

Note that a nested class can reference all types and constants nested within its parent class – the TOuter prefix in the above example is optional. Any code outside of a class must use an explicit type prefix – for example, the external test code in the Chapter10\NestingHabits project must use the TOuter prefix to get at the types and constants declared in the TOuter class.

Remember, while inner classes can see all types declared in their outer class(es), they have no special access to their outer class’ fields, methods, or properties. (Similarly, outer classes cannot see private or protected members of their inner types). The main purpose of nested classes is to reduce namespace clutter by keeping (often private) helper classes within the public classes that use them. This also makes it easier to hide implementation details. (See Chapter 2’s Nested Classes section.)

Tip   Delphi’s global level constants are not CLS-compliant (or rather, the name of the class they end up in is an implementation detail generated by the compiler, and should not be relied upon), so you should declare constants you want to export to other languages inside a class or record declaration. The same advice goes for global routines and variables - declare them as static methods and fields."

Borland and CodeGear

As most of you will already know, Borland has now moved its developer tools group into a separate company called CodeGear. CodeGear, which will still be wholly owned by Borland, will continue to develop, sell and support developer tools like Delphi, C++Builder, JBuilder and Interbase. There are also indications that they are moving into the webspace with support for languages such as PHP, Pyhon, Ruby and Ajax technology.

While some people have been waiting anxiously for a complete separation from Borland in a operation to sell the IDE group to a separate entity, I think that the current situation is better for the stability and long-term viability of CodeGear in general and the future of Delphi in particular.

The CodeGear management will be in a position to grow the IDE business and pour the revenue stream back into the development of new versions and products. I think this is good news, but as always we should keep our heads calm and keep our options open ;).

Good thing we called our user group Oslo Delphi Club and not NoBUG (Norwegian Borland User Group) or something similar ;)).

Hack#13: Access globals faster ($ImportedData)

There is an aura of magic around Delphi packages. Packages allow you to share Delphi code at a much higher level than is possible with plain old DLLs. Writing a DLL-based library API involves writing flat non-OOP global routines and avoiding any data types more advanced than integers, doubles, static arrays, PChar and records. You cannot share or exchange classes, objects, global variables or let alone strings (unless the client is a Delphi app and both the DLL and client use ShareMem).

Delphi Package Magic
Enter the magic of Delphi packages. Now you no longer have to worry about what you can share between module boundaries, because you can share everything! Packages is just a way of dividing up the logical code into physical deployment modules. Everything just works - somehow. The somehow is the (mainly undocumented) magic.

We're going to look at a small part of that magic - how global variables can be accessed across the application/package boundary. Lets start with the simplest possible case first .

Lets say we have a Unit1 that defines a GlobalVar: integer and some code in the same unit that sets it. Here is the generated assembly and machine code.

Unit1.GlobalVar := 13;
004081B1 C705989240000D00 mov [GlobalVar],$0000000d

That is a single assembly instruction with no indirections or pointer access going on. Notice that both the address of the global variable and the immediate value 13 (encoded as a 16-bit shortint, $000D) is encoded directly inside the machine code instruction. When I evaluate the address of the global variable in the Ctrl+F7 evaluator, @GlobalVar, I get $409298. This is the big-endian value I have highlighted in yellow above.

Then let's look at the package magic situation. If we have a PackageA containing a Unit1 that defines a GlobalVar: integer and a DemoApplication that links to PackageA and with code that access the GlobalVar, the compiler will compile the code into something like this:

Unit1.GlobalVar := 42;
00403389 A1A4404000   mov eax,[$004040a4]
0040338E C7002A000000 mov [eax],$0000002a

Notice the indirection there? Global variables are accessed through a pointer that is fixed up to point to the right address inside the package when the package is loaded. That is cool and a good thing. The global variable itself resides inside the package/DLL module, while the application has a magic, "invisible" global pointer variable that points to the actual package variable. This pointer is fixed up during the magic static loading of packages (it's value is probably set to the result of a GetProcAddress API call).

Global variables in standalone apps
Now lets step back a little and look at another common situation. We have looked at the intra-unit situation and the inter-module situation - the two most extreme conditions. But what happens when you access a global variable between units inside the same application? Surely, then the indirection should not be needed? Lets test it.

We write a simple console application consisting of two units, Unit1 and Unit2, and the main program.

unit Unit1;

interface

var
  GlobalVar: integer = 42;

procedure SetGlobalLocally;

implementation

procedure SetGlobalLocally;
begin
  asm int 3 end; // will break here, look in CPU view
  Unit1.GlobalVar := 13;
end;  

end.

unit Unit2;
  
interface

procedure SetGlobalIndirect;

implementation

uses Unit1;

procedure SetGlobalIndirect;
begin
  asm int 3 end; // will break here, look in CPU view
  Unit1.GlobalVar := 42;
end;  

end.

program TestImportedData;
{$APPTYPE CONSOLE}
uses
  Unit1 in 'Unit1.pas',
  Unit2 in 'Unit2.pas';
begin
  SetGlobalLocally;
  SetGlobalInDirect;
end.

Notice the cute little hand-coded breakpoints? Debugger breakpoints are actually implemented by overwriting a byte of your code with the software breakpoint instruction (int 3) which is encoded in a machine code byte $CC. When you run this program with debugging enabled from the IDE, it will break on the two int 3 instructions. Neat, huh? ;)

The first breakpoint is inside Unit1 just above the code that modifies the global variable. This is the case we showed first above. If you run it and follow the instructions to look at the code in the CPU view, you will see that there is no indirection.

Unit1.GlobalVar := 13;
004081B1 C705989240000D00 mov [GlobalVar],$0000000d

The second breakpoint is in Unit2 where we are modifying the global variable from an "external" unit (from a different unit than were the global was declared). Notice that we are still in a single EXE file here, so there should be no need for indirection, as it was in the package case. Run the code again and when it hits the second breakpoint, look at the machine code in the CPU again.

Unit1.GlobalVar := 42;
00403389 A1A4404000   mov eax,[$004040a4]
0040338E C7002A000000 mov [eax],$0000002a

Looks like we still have indirection here...! What is going on? The reason for this (small) inefficiency is that a unit can be moved into or out of a package without being recompiled. So the compiler has to be conservative and generate the unit's code to support the "worst" case that it will be exported for use from a package. The important thing to learn from this is:

By default, all cross-unit global variable access is performed indirectly via a pointer

The implication is that if you have performance critical code that access a global variable, you should cache it in a local variable inside the loop. You can only do this if you do not need to "see" updates to the global variable while the loop is running (think multithreaded code).

For standalone applications that do not use packages at all, it is a bit irritating to know that the compiler generates sub-optimal code for global variable access. While both the performance and size overhead of the indirection should be negligible in most situations, it would still be nice to force the compiler to get rid of this indirection.

$ImportedData Off
And as it happens there is a compiler directive that does exactly this. Enter the $ImportedData (aka $G) directive. This is what the Delphi help file has to say:

"Type Switch
Syntax {$G+} or {$G-}
{$IMPORTEDDATA ON} or {$IMPORTEDDATA OFF}
Default {$G+} {$IMPORTEDDATA ON}
Scope Local
Remarks

The {$G-} directive disables creation of imported data references. Using {$G-} increases memory-access efficiency, but prevents a packaged unit where it occurs from referencing variables in other packages."

By default the setting is {$ImportedData On} - this enables the pointer indirection in code that access unit-external global variables. By using the {$ImportedData On} directive in a unit, you turn off this indirection, resulting in tighter faster code for global variable access. Notice that you have to use it in every unit referencing global variables, not just in the unit declaring it.

Lets add a third unit to our test bench program.

unit Unit3;
 
interface

procedure SetGlobalDirect;

implementation

uses Unit1;

{$IMPORTEDDATA OFF}

procedure SetGlobalDirect;
begin
  asm int 3 end; // will break here, look in CPU view
  Unit1.GlobalVar := 42; 
end;

end.

The code here is identical to the code in Unit2, but now we have included the $ImportedData Off directive to tell the compiler generate optimized code for globals. Run the program again and when it stops at the breakpoint in Unit3, look in the CPU view.

Unit1.GlobalVar := 42; 
004033D1 C705A04040002A00 mov [GlobalVar],$0000002a

Hurrah! We got rid of the indirection.

The {$ImportedData Off} compiler directive forces the compiler to generate efficient global variable access

This works fine in Delphi 6 and 7. As it happens it no longer works in Delphi 2006 (and probably not 2005). Looks like a bug has sneaked into the compiler here. The directive is effectively ignored, still producing the indirection as we saw for Unit2 above. Hopefully Borland/DevCo will be able to fix this little issue in a future version of the compiler[1].

IDE Compiler Options
Knowing that $ImportedData Off generates better code for global variable access in a standalone application, we would of course like to set it for all the units in the application as a whole. Unfortunately that is not so easy to do. You see the Project | Options | Compiler Options dialog page does not give access to this option... :(.

Because of this issue you have to use one of the following solutions:

• Manually insert {$IMPORTEDDATA OFF} at the top of all units in the project
  
• Manually insert an {$ MyAppSettings.Inc} file in all units. The .inc file would contain settings such as {$IMPORTEDDATA OFF}
  
• Compile using Dcc32.exe setting the option from the command-line

In other words it is a bit awkward to enable this for a given application.

It is hard to set {$ImportedData Off} from the IDE.

I hope that they will be able to make this option available from the compiler options in the IDE in a future version. In fact, I went ahead and reported this as a enhancement request in QC - (please vote for it!):

"Can we please have the $G switch (AKA $IMPORTEDDATA) promoted to appear in the Project Options | Compiler dialog? It makes it easier to switch it off globally for those of us who never use run-time packages."

While it can be argued that introducing this IDE option runs the danger of users producing unusable packages, I don't think this is reason to deny the large number of people writing standalone applications easy access to this option.

The solution could be to only enable this option for projects that are compiled with run-time packages turned off (on the Project Options, Packages page).

Still, if a unit is compiled with IMPORTEDDATA OFF and then moved to a package, it has to be recompiled (i.e. the package project must be rebuilt).

To get around this problem you could:

a) Have the compiler detect this case and recompile all units with the wrong IMPROTEDDATA setting

b) Document the issue and require the user to rebuild himself

c) Combined with b) make the option less obvious and make the IDE compiler options future proof by adding a free-text field for additional compiler options. The user must manually type IMPORTEDDATA OFF in this field, supposedly this makes him qualified to know what he is doing.

The following Dcc32.exe compiler options are currently unavailable from the IDE Compiler Options:

G+ Use imported data references
M- Runtime type info
W- Generate stack frames
Z1 Minimum size of enum types"

If you think this is worthwhile to implement, please vote on QC#34650 here!

Score: Delphi 7 vs. Delphi 2006: 1-1 ;)

[1] I found that this issue has indeed been discovered independently and reported by Frederic Vanmol in Quality Central:

Report No: 33464 Status: Open
{$IMPORTEDDATA OFF} does not seem to work
http://qc.borland.com/wc/qcmain.aspx?d=33464

PS. Just as I had finished this piece, I started experimenting some more...;) In Delphi 7 it is actually possible (but still awkward) to compile an application with $ImportedData Off for all units without messing with each unit or the command line compiler.

You only need to

• close the project
  
• manually edit the project's .dof file
  
• change "-G+" to "-G-"
  
• save and close the .dof file
  
• reopen the project
  
• rebuild the application

If you compile from the command line you can update the .cfg file in a similar manner.

It is probably possible to do the same in Delphi 2006 by manually editing the XML in the .bdsproj file, changing

<Compiler Name="G">1</Compiler>

to

<Compiler Name="G">0</Compiler>

But it is a little hard to check, because of the Delphi 2006 bug of ignoring the directive. In both cases, the manually set compiler options are preserved even when changing compiler options from the IDE.

Hack#12: Create smaller .EXE files ($SetPEFlags)

No, this post is not about so-called EXE-compressors - I don't believe in using them. And it is not a pure hack in the sense that we're breaking any rules - its just about documenting an undocumented and unknown feature of the Delphi 2006 Win32 compiler (it is not implemented in D7 - I don't know about D2005 yet as I don't have it installed on this laptop anymore).

Background info
When you compile a DLL (or a package which is a Delphi-specific DLL in disguise), the linker includes what is known as a relocation table. This table includes information about what addresses must be fixed up by the OS loader in the (likely) event that the DLL must be loaded at a different address than its intended-at-compile/link-time base address. You see all DLLs come with a base address that is the "ideal" loading address of that module. The OS will try to load the DLL at this address to avoid the overhead of runtime rebasing (patching the in-memory pages of the DLL forces it to be paged in from disk and prevents cross-process sharing of the DLL pages). That's why you should set the Image base option in the Linker page of the project options of DLL and package projects. The default Image base that Delphi uses is $00400000 for both applications, DLLs and packages - and thus by default all DLLs and packages will need to be rebased - as that address is reserved for the process' EXE file.

The implication is that an EXE file will always be loaded at the fixed virtual address $00400000 and that it will never need to be rebased. Alas, it doesn't really need its relocation table and we can safely remove it, shrinking the size of the .EXE without affecting its behavior or performance. It is basically a free lunch!

Up until now Delphi users have had to resort to external stripping tools like Jordan Russell's excellent StripReloc tool. It basically takes the name of an .EXE file as a parameter and removes the relocation table from it. I normally don't bother to strip the reloc table during development, but I do run Jordan's tool before deploying our application. This strips a hefty 350 kB from our 7 MB EXE file - a nice, free 5% reduction right there.

As far as I know, most Microsoft tools produces reloc-free EXE files, while Delphi always includes it for both DLLs and EXEs. Up until now. Delphi 2006 actually has a unknown, undocumented compiler directive feature that allows you to turn off the relocation table for EXE files.

To test it out I first opened the ResXplor project from the Demos\DelphiWin32\VCLWin32\ResXplor folder (a pretty neat application, btw). I compiled it as-is. Project | Information told me that the size of the EXE file was 614400 bytes (and Windows Explorer agreed). Then I added the following line to the top of the project file:

{$SetPEFlags 1}

(We'll look at the source of the mysterious value 1 in a moment).

Then I recompiled the project (there is no need to rebuild it, as the unit files does not need to be recompiled - only the linker must be reinvoked). And the size of the EXE file had shrunk to 577536 bytes - that is a reduction of 36864 bytes or 6 %. Not bad for a one-liner, eh? ;).

The Delphi 2006 help file has the following to say about the $SetPEFlags compiler directive:

"Type: Flag
Syntax: {$SetPEFlags <integer expression>} {$SetPEOptFlags <integer expression>}
Scope: Local

Microsoft relies on PE (portable executable) header flags to allow an application to indicate compatiblity with OS services or request advanced OS services. These directives provide powerful options for tuning your applications on high-end NT systems.

These directives allow you to set flag bits in the PE file header Characteristics field and PE file optional header DLLCharacteristics field, respectively. Most of the Characteristics flags, set using $SetPEFlags, are specifically for object files and libraries. DLLCharacteristics, set using $SetPEOptFlags, are flags that describe when to call a DLL's entry point.

The <integer expression> in these directives can include Delphi constant identifiers, such as the IMAGE_FILE_xxxx constants defined in Windows.pas. Multiple constants should be OR'd together."

So it (and the related SetOptPEFlags directive) is a way of informing the OS about the certain aspects of the application. For instance, it can be used to tell the OS that it is ready and compatible with getting access to virtual memory addresses above the 2 GB boundary. Lets look at the IMAGE_FILE constants defined in the Windows.pas unit.

const
  { Relocation info stripped from file. }
  IMAGE_FILE_RELOCS_STRIPPED               = $0001;
  { File is executable  (i.e. no unresolved externel references)}
  IMAGE_FILE_EXECUTABLE_IMAGE              = $0002;
  { Line nunbers stripped from file. }
  IMAGE_FILE_LINE_NUMS_STRIPPED            = $0004;
  { Local symbols stripped from file. }
  IMAGE_FILE_LOCAL_SYMS_STRIPPED           = $0008;
  { Agressively trim working set }
  IMAGE_FILE_AGGRESIVE_WS_TRIM             = $0010;
  { App can handle >2gb addresses }
  IMAGE_FILE_LARGE_ADDRESS_AWARE           = $0020;
  { Bytes of machine word are reversed. }
  IMAGE_FILE_BYTES_REVERSED_LO             = $0080;
  { 32 bit word machine. }
  IMAGE_FILE_32BIT_MACHINE                 = $0100;
  { Debugging info stripped from file in .DBG file }
  IMAGE_FILE_DEBUG_STRIPPED                = $0200;
  { If Image is on removable media, copy and run from the swap file}
  IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP       = $0400;
  { If Image is on Net, copy and run from the swap file. }
  IMAGE_FILE_NET_RUN_FROM_SWAP             = $0800;
  { System File. }
  IMAGE_FILE_SYSTEM                        = $1000;
  { File is a DLL. }
  IMAGE_FILE_DLL                           = $2000;
  { File should only be run on a UP machine }
  IMAGE_FILE_UP_SYSTEM_ONLY                = $4000;
  { Bytes of machine word are reversed. }
  IMAGE_FILE_BYTES_REVERSED_HI             = $8000;

That's a lot of values, but for the purposes of this article, we're really only interested in the first one.

const
  { Relocation info stripped from file. }
  IMAGE_FILE_RELOCS_STRIPPED               = $0001;

That's where our magic number 1 comes from. If we add Windows to the project uses clause, and move the SetPEFlags directive below the uses clause, we can use this more self-documenting directive.

{$SetPEFlags IMAGE_FILE_RELOCS_STRIPPED}

So this tells the OS that the file does not contain any relocation information, and thus it cannot be rebased at runtime. In addition, the Delphi compiler now also picks up on this information and uses it as an instruction to do as you say - leaving out reloc information from the generated EXE file!

Note: The compiler allows using the same directive to remove reloc info from packages and DLLs, but this is not a recommended practice. It will generate slightly smaller DLLs, but the DLL will fail to load unless it can be loaded at its given base address.

Score: Delphi 7 vs. Delphi 2006: 0 - 1 ;)